Video and audio multimodal search system

Abstract

A multimodal search system using video queries is described. The system can receive video data captured by the user device's camera. The video data may consist of a sequence of image frames. Additionally, the system may receive audio data associated with the video data captured by the user device. Furthermore, the system may use one or more machine learning models to process the sequence of image frames to generate video embeddings associated with the sequence of image frames. The video embeddings may consist of multiple image embeddings associated with the sequence of image frames. Furthermore, the system may determine one or more video results based on the video embeddings and audio data. The system may then send one or more video results to the user device.

Description

Technical Field

[0001] Priority Claim This application claims priority based on U.S. Patent Application No. 18 / 326,496, having a filing date of May 31, 2023, the disclosure of which is hereby incorporated herein by reference.

[0002] This disclosure generally relates to processing video and audio data within a search query to provide search results. More specifically, this disclosure relates to multimodal search by processing video embeddings extracted from video data along with voice commands.

Background Art

[0003] In conventional systems, a search query can include text input or audio data to search for a particular item or a particular piece of knowledge. However, when search requests are limited to only text and audio data, it can be difficult to understand the context and intent. In some instances, the text and audio data may not be sufficiently descriptive to generate the desired results that the user is seeking. The search results based on only text or audio data can be limited because the user is restricted by the mode of input for expressing their request. As a result, in conventional systems, if the search results are not satisfactory to the user, the user may post the content on social media or discussion forums to crowdsource answers from other users.

[0004] Video can provide additional insights regarding the user's intent through various visual cues and context information. For example, the context information can be derived from visual actions and gestures within the video. However, determining the user's intent from video is often a complex task that requires computer vision techniques and machine learning algorithms.

Summary of the Invention

[0005] Aspects and advantages of the embodiments of this disclosure are partially described in the following description, can be learned from the description, or can be learned through the practice of the embodiments.

[0006] One exemplary aspect of the present disclosure relates to a computer-implemented method for multimodal retrieval of video results. The method may include receiving video data captured by a user device's camera by a computing system including one or more processors. The video data may have a sequence of image frames. Additionally, the method may include receiving audio data associated with the video data captured by the user device. Furthermore, the method may include processing the sequence of image frames using one or more machine learning models to generate video embeddings associated with the sequence of image frames. The video embeddings have multiple image embeddings associated with the sequence of image frames. Furthermore, the method may include determining one or more video results based on the video embeddings and audio data. The method may then include transmitting one or more video results to the user device.

[0007] In some examples, the method may further involve processing a sequence of image frames using one or more machine learning models to generate temporal information associated with the sequence of image frames. The decision of one or more video results may further be based on the temporal information associated with the sequence of image frames.

[0008] In some examples, the method may further include processing audio data to generate text queries using automatic speech recognition technology. The determination of one or more video results may further be based on the text queries. In some examples, the method may further include processing the text queries and video embeddings concurrently to determine one or more video results using one or more machine learning models.

[0009] In some examples, processing a sequence of image frames to generate a video embedding may involve using a frame selection algorithm to select a subset of image frames from the sequence. Additionally, the method may involve processing each image frame within the subset of image frames to generate multiple image embeddings. Multiple image embeddings may have an image embedding for each image frame within the subset of image frames. Furthermore, the method may involve determining a video embedding based on the multiple image embeddings. The video embedding may be determined by averaging each of the multiple image embeddings. The frame selection algorithm may be a uniform random sampling of frames, and the subset of image frames may be selected at regular intervals, such as every n frames. Additionally or alternatively, the frame selection algorithm may be based on the camera position and camera orientation.

[0010] In some examples, processing a sequence of image frames to generate a video embedding may involve detecting a target object within the video data. Additionally, the frame selection algorithm is based on the spatial relationship between the camera and the target object. Furthermore, the spatial relationship may include a translation vector representing the camera's position in 3D space relative to the target object. The translation vector may specify the distance the camera is displaced relative to the target object. Furthermore, the spatial relationship may include a rotation matrix representing the camera's orientation in 3D space relative to the target object. The orientation matrix may specify the degree of rotation of the camera relative to the target object.

[0011] In some examples, the frame selection algorithm may be running on the user device, and the received video data is a subset of images from a sequence of image frames. Video embeddings can be generated by processing the subset of image frames. Additionally, video embeddings may have temporal information associated with the subset of image frames.

[0012] In some examples, the method may include processing audio data to generate an input audio signature. The determination of one or more video results may further be based on the input audio signature. Additionally, the determination of one or more search results may include detecting a target object within the video data. Furthermore, the method may include accessing multiple known audio signatures associated with the target object from an audio signature database. Furthermore, the method may include selecting a matching audio signature from the known audio signatures, where the comparison score of the matching audio signature exceeds a threshold. The matching score of the matching audio signature may be calculated by comparing the input audio signature with the matching audio signature. One or more video results may further be determined based on the matching audio signature.

[0013] In some examples, processing a sequence of image frames to generate a video embedding may involve processing each image frame in the sequence to generate a video embedding using one or more machine learning models. Additionally, the method may involve mapping the generated video embeddings to a video embedding index. One or more video results may be determined based on the mapping of the generated video embeddings to the video embedding index. Furthermore, the mapping of the generated video embeddings to the video embedding index may involve ranking each video embedding in the video embedding index based on a comparison with the generated video embeddings. Each video embedding may be associated with a video result. One or more video results may be determined based on the ranking of each video embedding in the video embedding index.

[0014] In some examples, one or more machine learning models may include polymass models. Additionally, one or more machine learning models may be trained using tutorial videos published on online video sharing platforms.

[0015] In some examples, one or more machine learning models may include a multimodal, multitask unified model trained to understand information from multiple formats, including video and text.

[0016] In some examples, the method may further include determining one or more web results based on a video embed. Additionally, the method may include sending one or more web results to a user device.

[0017] In some examples, the method may further involve using one or more machine learning models to process audio data along with a sequence of image frames to generate a video embedding.

[0018] Other exemplary aspects of this disclosure relate to a computer-implemented method for multimodal retrieval of video results. The method may include capturing video data having a sequence of image frames by the camera of a user device. Additionally, the method may include capturing audio data associated with the video data by the microphone of a user device. Furthermore, the method may include processing the audio data and the sequence of image frames to generate a video embedding using one or more machine-trained models stored on the user device, the video embedding being derived by processing the video data in parallel with the audio data. Furthermore, the method may include transmitting the video embedding to a server. In response to the transmission of the video embedding and audio data, the method may include receiving one or more video results from the server. The method then presents one or more video results on the display of the user device.

[0019] Other exemplary aspects of this disclosure relate to a computing system. The system may include one or more processors. Additionally, the system may include one or more non-temporary computer-readable media that, when executed by one or more processors, collectively store instructions causing the computing system to perform an action. An action may include receiving video data captured by a camera of a user device by a computing system including one or more processors, wherein the video data has a sequence of image frames. Furthermore, the action may include receiving audio data associated with the video data captured by the user device. Furthermore, the action may include processing a sequence of image frames using one or more machine-trained models to generate video embeddings related to the sequence of image frames, wherein the video embeddings have multiple image embeddings associated with the sequence of image frames. The action may include determining one or more video results based on the video embeddings and audio data. The action may then include transmitting one or more video results to the user device.

[0020] Other exemplary aspects of the present disclosure relate to one or more non-temporary computer-readable media that, when executed by one or more computing devices, collectively store instructions causing one or more computing devices to perform an action. The action may include receiving video data captured by a user device's camera, the video data having a sequence of image frames, by a computing system including one or more processors. Additionally, the action may include receiving audio data associated with the video data captured by the user device. Furthermore, the action may include processing the sequence of image frames using one or more machine-trained models to generate video embeddings relating to the sequence of image frames, the video embeddings having multiple image embeddings associated with the sequence of image frames. Furthermore, the action may include determining one or more video results based on the video embeddings and audio data. The action may then include transmitting one or more video results to the user device.

[0021] Other aspects of this disclosure cover a variety of systems, apparatus, non-temporary computer-readable media, user interfaces, and electronic devices.

[0022] These and other features, aspects and advantages of the various embodiments of this disclosure will be better understood by referring to the following description and the appended claims. The appended drawings incorporated herein and forming part of this specification illustrate exemplary embodiments of this disclosure and, together with the description, serve to illustrate the relevant principles.

[0023] A detailed description of embodiments intended for those skilled in the art is given herein with reference to the accompanying drawings. [Brief explanation of the drawing]

[0024] [Figure 1]Shows a block diagram of an exemplary multimodal search system according to an exemplary embodiment of the present disclosure. [Figure 2] Shows a block diagram of an exemplary user interface of a multimodal search system according to an exemplary embodiment of the present disclosure. [Figure 3A] Shows a block diagram of an exemplary multimodal search system in which a machine-learned model is stored in a server according to an exemplary embodiment of the present disclosure. [Figure 3B] Shows a block diagram of an exemplary multimodal search system in which a machine-learned model is stored in a user device according to an exemplary embodiment of the present disclosure. [Figure 4A] Shows an example of an exemplary use case of using a multimodal search system according to an exemplary embodiment of the present disclosure. [Figure 4B] Shows an example of an exemplary use case of using a multimodal search system according to an exemplary embodiment of the present disclosure. [Figure 4C] Shows an example of an exemplary use case of using a multimodal search system according to an exemplary embodiment of the present disclosure. [Figure 4D] Shows an example of an exemplary use case of using a multimodal search system according to an exemplary embodiment of the present disclosure. [Figure 5] Shows a flowchart diagram of an exemplary method for performing multimodal search on a server having a machine-learned model according to an exemplary embodiment of the present disclosure. [Figure 6] Shows a flowchart diagram of an exemplary method for performing multimodal search on a user device having a machine-learned model according to an exemplary embodiment of the present disclosure. [Figure 7A] Shows a block diagram of an exemplary computing system for performing multimodal search according to an exemplary embodiment of the present disclosure. [Figure 7B] Shows a block diagram of an exemplary computing system for performing multimodal search according to an exemplary embodiment of the present disclosure. [Modes for carrying out the invention]

[0025] The repeated reference numbers across multiple drawings are intended to identify the same features in various embodiments.

[0026] overview Currently, users have many complex questions, such as how to fix a noisy home appliance, requests for tutorial videos to learn specific dance moves, or information about objects in real-time video scenes. These types of questions can be difficult or impossible to query using traditional search systems that only use text and audio data. However, many of these complex questions can be easily expressed by users using video and audio data. By analyzing the visual and contextual aspects of the video, the system can gain valuable insights into the user's intent and behavior in order to provide the correct answers.

[0027] Generally, this disclosure covers systems and methods for improving search results by analyzing video data in parallel with audio data as part of a search query. In some examples, the system can process video data in parallel with audio data to understand complex needs, including nuances and context, and help the user receive the correct answer. Alternatively, in some examples, the audio data is converted to text data, and then the video data is processed with the text data to determine the search results. In particular, the systems and methods disclosed herein can leverage video data, audio data, and / or text data to provide multimodal (e.g., search combining audio and visual inputs) search capabilities and multimodal outputs.

[0028] According to some embodiments, the techniques described herein enable an increase in the number of informational queries to a search system by allowing the user to perform multimodal searches (e.g., video input and voice commands). The system may use a machine learning model to understand the video content, including the video context in the audio within the video, the user's voice within the video, and the video frames.

[0029] The system can enable users to perform multimodal searches in a manner as natural as seeking assistance from an expert. Multimodal search refers to the process of retrieving information or executing search queries using multiple modalities, such as video, audio, or other forms of data. By incorporating different types of media, the system can provide more comprehensive and intuitive search results. Traditional search engines rely primarily on text-based queries and keyword matching to retrieve relevant information. However, multimodal search allows users to use various modalities to express their queries, enabling a more natural and expressive interaction with the search system. For example, a multimodal search may involve submitting voice commands along with video clips captured in real time. The system can analyze the voice commands and video clips to provide responses. By combining video and audio, the system enhances the search experience, improves the accuracy of results, and enables users to explore and discover information that was previously impossible.

[0030] For some complex questions, it may be a natural way for users to express intent and ask complex questions by executing multimodal queries (e.g., video queries). In some cases, video data may include visual action and gesture data. The system may analyze actions and gestures performed by objects in the video to infer intent. For example, intent may be inferred when the user is seen expressing emotion based on hand gestures. Additionally, video data may include facial expression and emotion data. For example, facial expressions shown by the user in the video (e.g., dissatisfaction, confusion) can reveal the user's emotional state and provide information about the user's intent (e.g., seeking help or explanation). Furthermore, video data may include eye-tracking data. For example, the system may use eye-tracking technology to determine where the user is looking in the video in order to determine a target object (e.g., an object of interest). The system may analyze gaze patterns to gain insights into areas of interest and understand the user's intent and preferences. Furthermore, video data may include contextual information. The system may analyze video data to generate contextual information. For example, a system may analyze the surrounding context of a video to determine user intent. For example, a system may analyze the audio data of a video to identify keywords or phrases that indicate topics of interest to the user or user-specific queries. A system may determine user intent from video data by processing the video data with voice commands using a machine learning model. Additionally, using the techniques described herein, a machine learning model may be trained to analyze the visual and contextual aspects of video data to better understand the content of the video data.

[0031] In some embodiments, the systems described herein enhance the multimodal experience by providing a novel way for users to formulate questions simultaneously in multiple modalities (e.g., video and audio). The systems can enable the expansion of query streams seeking information and introduce new types of queries. Furthermore, the systems can utilize large-scale language models (LLMs) to improve search results related to queries seeking opinions and actions. The systems can utilize machine learning models to present artificial intelligence (AI)-generated answers and outputs. The techniques described herein significantly reduce friction in formulating multimodal queries, resulting in an improved user experience.

[0032] Additionally, by enabling new ways of inputting data for search requests, the system increases the search space for potential search results. By incorporating video data in combination with audio data, the system can increase the search space for potential search results, resulting in an improved user experience. In some embodiments, the system solves use cases where the system needs to correctly understand audio along with video. For example, if a user's vehicle is making a particular sound, it may be difficult, if not impossible, to describe that sound or problem in words. The system described herein can receive audio data related to the sound along with video of a vehicle in motion and return search results for repairing the vehicle. The system can determine the type and model of the vehicle from video embeddings extracted from the video data. Additionally, the system may determine that the audio data is related to a problem associated with a particular type and model. Continuing this example, the search results may include a video tutorial on how to fix this problem. In some embodiments, the search results may include augmented reality instructions for fixing the vehicle problem.

[0033] A multimodal model may receive video as a query to examine results from a video index. The system may include a video acquisition benchmark. A video acquisition benchmark can be a standardized evaluation framework or dataset used to assess the performance of an acquisition model or algorithm. Video acquisition benchmarks may be specific to various information acquisition tasks, such as video acquisition, question answering, and recommendation systems. A video acquisition benchmark may include a set of queries or inputs, a set of answers, and performance metrics to measure the quality of the acquisition results. A video acquisition benchmark may provide benchmark datasets and metrics for video acquisition tasks, such as content-based video search and video summarization, to train a multimodal model. Evaluation metrics used in a video acquisition benchmark may include precision, recall, mean precision, and normalized discounted cumulative gain. These metrics quantify the relevance, ranking, and overall effectiveness of the acquisition results.

[0034] The systems and methods of this disclosure offer several technical effects and advantages. For example, the systems and methods can improve search results by enabling multimodal search capabilities. Additionally, by using video and audio data in parallel to determine video embeddings and then mapping the video embeddings, the systems can provide more accurate search results by augmenting queries with additional signals that provide useful context for the search. In some examples, video embeddings may be real-valued vectors encoding the meaning of the video and can be searched in parallel in a multidimensional vector space to provide more accurate search results. In exemplary embodiments, the system may use a frame selection algorithm to select a subset of image frames from a sequence of image frames. Since a video may have a large number of image frames, the selected subset of image frames may be important for improved video search results, as the system can optimally determine which subset of image frames to select based on the frame selection algorithm. Once a subset of image frames is selected, the system can process each image frame in the subset to generate multiple image embeddings, each having an image embedding for each image frame in the subset. The system can then determine video embeddings based on the multiple image embeddings.

[0035] Furthermore, by enabling multimodal search capabilities, the system increases the search space for potential search results by enabling searches that might not have been possible before. The search space can be increased in part based on a known signature database. For example, the system may create a database of audio signatures and / or video signatures from publicly available videos. The system can then determine a matching score between the received video data and the known audio signatures and / or video signatures.

[0036] Additionally, the search space can be expanded by allowing the system to process each image frame in a sequence of image frames to generate video embeddings using one or more machine learning models. The system can map the generated video embeddings to a video embedding index. The video embedding index may be created and / or generated based on publicly available videos. One or more video results are then determined based on the mapping of the generated video embeddings to the video embedding index. For example, in the dance tutorial example in Figure 4C, the video embedding index may include videos of a first dance style, and the generated video embeddings may match video embeddings of videos of the first dance style (e.g., waltz). Thus, if a user queries how they can learn the dance style of a user-created video, the system may determine that the user-created video is associated with a first dance style and provide a video tutorial on how to dance the first dance style (e.g., waltz).

[0037] As highlighted by the vehicle example above, users can perform searches that may have been previously impossible, and the system can now provide tutorials on how to fix vehicle problems by analyzing audio and video data in parallel. In some examples, search results are delivered to the user faster by reducing the number of user interactions. In particular, the systems and methods disclosed herein can leverage interactive user interfaces that enable users to use video data to provide better, faster, and more accurate search results. Furthermore, by analyzing video data in parallel with audio data to generate video embeddings, search queries can be executed faster than in conventional systems that typically convert audio data to text and then feed that text into a machine learning model.

[0038] Other technical effects and benefits relate to improved computational efficiency and improvements in the functionality of the computing system. For example, the systems and methods disclosed herein can leverage a multimodal search system to provide more comprehensive multimodal search queries that reduce the use of additional searches and the viewing of additional search result pages, thereby saving time, processing resources, energy, and computational effort.

[0039] Figure 1 shows a block diagram of an exemplary multimodal search system 100 according to an exemplary embodiment of the present disclosure. The multimodal search system 100 may receive a multimodal input 101 to generate a multimodal output 125. For example, the multimodal search system 100 may be a video and audio search system that processes video data 102 and audio data 104 to generate video results 116, web results 118, and / or augmented reality output 120. The video data 102 may be captured by a camera on a user device. The audio data 104 may be captured by a microphone on a user device. Additionally, the audio data 104 may be associated with the video data 102. For example, a user may record a video of a scene and provide an audio command associated with that scene in the video.

[0040] In some examples, the multimodal search system 100 may receive a multimodal input 101 (e.g., video data 102 and audio data 104) and process the data to generate a multimodal output 125. For example, the video data 102 may include a video of a person dancing, and the audio data may be a voice command saying, "How can I learn these dance moves?" The multimodal search system 100 may input the video data 102 and audio data 104 into a machine learning model 106 to generate a video embedding 108 and temporal information 112. Subsequently, the multimodal search system 100 may use a video search module 110 having a video index 114 to search for a tutorial video that teaches how to dance in a similar way to the person dancing in the video data 102. The multimodal search system 100 may then present the tutorial video as part of the video results 116.

[0041] According to some embodiments, a user can record a video to perform a query (e.g., a query on how to do something). In some examples, visual information and spoken language are input to the multimodal search system 100. The multimodal search system 100 may have one or more machine learning models 106 that can process and understand both text and visual information simultaneously. One or more machine learning models 106 (e.g., multimodal models) enable the integration of language and video data, enabling tasks such as video captioning, visual question answering, and cross-modal search.

[0042] According to some embodiments, the audio data 104 may be voice input from the user, such as a search query, question, inquiry, command, action to be performed, scene exploration, and / or response to a prompt. The system 100 may determine the multimodal output 125 based on the category of the audio data. For example, if the audio data 104 is a question (e.g., "Find a tutorial video for this dance move"), the multimodal output 125 may be a video result 116 of a tutorial video. Alternatively, if the particular category is a scene exploration (e.g., "Where do I put the coolant in the vehicle?"), the multimodal output 125 may be an augmented reality output 120. Yet another alternative output is if the particular category is an image search query (e.g., "Find a dress similar to this dress in this video"), the multimodal output 125 may be a web result 118.

[0043] According to some embodiments, the multimodal search system may include one or more machine-trained models 106 that can process video data 102 and audio data 104 in parallel to generate video embeddings 108. Alternatively, in other embodiments, one or more machine-trained models 106 may process text data derived from the video data 102 and audio data 104 to generate video embeddings 108.

[0044] In some examples, one or more machine learning models 106 may be trained using encoder and decoder tasks. One or more machine learning models 106 may be multimodal models capable of processing both video and audio data in parallel. Assuming that encoder and decoder tasks are used during training, the multimodal search system 100 may obtain video embeddings 108 by using average pooling of encoder outputs. Additionally, to improve embedding quality, one or more machine learning models 106 may be fine-tuned using adapters. Adapters may be used to fine-tune one or more machine learning models 106 by enabling efficient and targeted modifications to specific parts of the model architecture. The system may define an adapter architecture, which may be a lightweight module that can be added to one or more machine learning models 106. The system may initialize the adapter parameters randomly or using transfer learning techniques. The system may freeze the parameters of one or more machine learning models 106 and train only the adapter. This intensive training helps the adapter specialize for a target task while retaining the knowledge taken up by one or more machine learning models 106. The system may prepare task-specific datasets containing relevant multimodal examples for fine-tuning purposes. The system may feed multimodal data (e.g., video and audio data) through one or more pre-trained models 106 to which adapters have been added. Subsequently, the system may calculate the loss between the predicted output and the ground truth label, and update the adapter parameters by backpropagating the gradient. The system may optimize the performance of one or more pre-trained models 106 by tuning hyperparameters such as the learning rate, batch size, or regularization technique. By using adapters in the fine-tuning process, one or more pre-trained models 106 can be efficiently adapted to new multimodal tasks or domains without requiring large-scale retraining from scratch.The adapter enables more intensive updates, reducing the risk of catastrophic forgetting and accelerating the fine-tuning process.

[0045] In some examples, one or more machine learning models 106 may be trained using pre-training data and a mixture of pre-training tasks may be introduced to prepare the models for various downstream applications. For example, the system may enable knowledge sharing between video tasks and language tasks by casting all tasks to a single general-purpose application programming interface (API) having video and audio data with inputs that solve a wide variety of tasks related to different use cases. The purpose used for pre-training can be input into the API as a weighted mixture targeted at training one or more machine learning models 106 to perform new tasks (e.g., split captioning for video descriptions, optical character recognition (OCR) prediction for scene text understanding, visual question answering (VQA) prediction). For example, one or more machine learning models 106 may be trained on both video data 102 and audio data 104 using an open-source framework. For audio data 104, the system may concatenate dense token embeddings with patch embeddings generated by video data 102, together as input to a multimodal encoder decoder. During the training of one or more machine learning models 106, the weights of the multimodal encoder decoder are updated.

[0046] According to some embodiments, the multimodal search system 100 may input video embeddings 108 generated by one or more machine learning models 106 into a video search module 110. The video search module 110 may determine a multimodal output 125 based on the mapping of the video embeddings 108 to a video index 114. For example, the video search module 110 may be a web-based tool and / or a mobile app-based tool that enables users to search, discover, and access videos across the internet. Specifically, the video search module 110 indexes and retrieves video data. The video search module 110 may discover video data by crawling and scanning the internet. The video search module 110 may index video content within a video index 114. Additionally, the video search module 110 may use an algorithm (e.g., a machine learning algorithm) to determine the most relevant videos for a query. The algorithm considers various factors such as video metadata, popularity, and the degree of matching with the video embeddings 108.

[0047] In some examples, the multimodal search system 100 may construct a video index 114 with video embeddings. The video index 114 may contain a structured representation of content and metadata within a collection of videos, enabling efficient searching, retrieval, and analysis. Firstly, the multimodal search system 100 may collect and index video data (e.g., videos). For example, the video data may be a large dataset of millions of narrated videos with an emphasis on instructional videos (e.g., the HowTo100M dataset). Each video in the database of the video index 114 may be processed using a machine learning model 106 to generate video embeddings for each video. Video embeddings stored in the video index 114 may be vectors resulting from videos that can be compared to video embeddings 108 generated based on the multimodal input 101. The multimodal search system 100 may extract embeddings (e.g., features, video features, video embeddings) from videos that can be used for indexing and searching. Video embeddings may incorporate different modalities of video data. The multimodal search system 100 can annotate each video with metadata such as title, description, tags, timestamp, and any other relevant information. The metadata can provide additional context and facilitate efficient retrieval and filtering. The multimodal search system 100 can utilize an index structure based on the specific requirements of the machine learning model 106 and / or the video embedding 108. The video index structure may include an inverted index, a hash-based method, or a content-based retrieval method such as video fingerprinting. These structures enable efficient storage and retrieval of video data. The multimodal search system 100 can index preprocessed videos by storing extracted video embeddings, features, metadata, and any required index structure in a structured manner by mapping video features and metadata to their corresponding identifiers or keys.The multimodal search system 100 can implement machine learning-based search algorithms to enable efficient searching and retrieval of videos based on video embeds 108, keywords, similarity, time range, or other relevant parameters. The multimodal search system 100 can construct a video index 110 using a combination of domain-specific knowledge, data preprocessing techniques, feature extraction methods, and indexing algorithms.

[0048] Additionally, the machine-trained model 106 may generate (e.g., make decisions) temporal information 112 based on the input video data 102 and audio data 104. The machine-trained model 106 may generate temporal information 112 using recurrent neural network (RNN), long-shortened memory (LSTM) network, and 3D convolutional neural network (3D-CNN) techniques. Temporal information 112 may include changes and progression of content (e.g., video data 102 and audio data 104) over time. Unlike still images, video data 102 is dynamic, which allows the machine-trained model 106 to capture movement and transitions that unfold as time progresses. Temporal information 112 may include motion data, continuity data, changes in scene data, changes in audio data, and data on other changes in content over time. Motion data may represent the movement of objects within video frames or the camera itself. Continuity data may include understanding sequences of events or actions within the video to determine the story development or action flow within the video. Scene data changes may include changes in the properties of the scene, lighting, or objects over time. For example, a transition from daytime to nighttime, an object changing color, or a change in facial expression from happy to sad. Audio data changes may include sounds that provide temporal information, such as the progression of speech, changes in music, or ambient sounds occurring at a particular moment in the video. Temporal information 112 can be utilized by the video retrieval module 110 in tasks such as action recognition, event detection, video summarization, and anomaly detection. A machine learning model 106 may determine the temporal information 112 using recurrent neural networks (RNNs), long-shortened memory (LSTM) networks, and 3D convolutional neural networks (3D-CNNs) techniques.

[0049] The video search module 110 may perform query refinement based on temporal information 1112. Query refinement may include a process of reformulating a given query to improve the performance of information retrieval operations, particularly in contextual understanding of queries based on temporal information. Query refinement may include determining the user's intent by evaluating the temporal information 112 and refining the search query based on the temporal information 112.

[0050] Figure 2 shows a block diagram of an exemplary user interface 200 of a multimodal search system according to an exemplary embodiment of the present disclosure. User interface 200 may include a first user interface 210 that allows the user to select a category button to obtain video results 116, web results 118, or augmented reality output 120. For example, the category buttons may be a “Modify” button 212, a “Search” button 214, or an “Augmented Reality” button. Once a button is selected, a second user interface 220 allows the user to record a video using a record button 222. The record button may include a rotating ring that limits the length of the video recording. In some examples, the video recording may be limited to 30 seconds or less. After the video recording is complete, a third user interface 230 may display the video recording 232 and also provide a confirmation button 234 for performing a search. Subsequently, a fourth user interface 240 may present video results 242, 244, 246, and 248 to the user.

[0051] Figure 3A shows a block diagram of an exemplary multimodal search system 300 according to an exemplary embodiment of the present disclosure, in which machine learning models are stored on a server. In this system 300, machine learning models 325 (e.g., machine learning models 106) are part of a server 320. In this embodiment, in 302, a user device 310 may capture both voice and audio data and send it to a server 320. A server 320 may include machine learning models 325 and a search server (e.g., a video search module 110). In 304, a machine learning model 325 stored on a server 320 may generate a video embedding using video data and voice commands received from the user device 310. Additionally, in 304, a machine learning model 325 may send the video embedding to a search server 330. In step 306, the search server 330 can return the search results to the user device 310.

[0052] Figure 3B shows a block diagram of another exemplary multimodal search system 350 according to an exemplary embodiment of the present disclosure, in which a machine learning model is stored in a user device. In this system 350, the machine learning model(s) 370 (e.g., machine learning model(s) 106) is part of the user device 355. In this embodiment, in 352, the input device 360 ​​of the user device 355 may capture voice and audio data and transmit it to the machine learning model(s) 370. In 354, the machine learning model(s) 370 stored in the user device 355 may generate a video embedding using the video data and voice commands received from the input device 360. Additionally, in 304, the machine learning model(s) 325 may transmit the video embedding to the search server 330. In 306, the search server 330 may return the search results to the user device 355.

[0053] Exemplary use cases According to several embodiments, the multimodal search system 100 can be used in multiple use cases. The types of use cases may include, but are not limited to, the information query shown in Figure 4A, the scene search shown in Figure 4B, the motion tutorial query shown in Figure 4C, and the augmented reality experience shown in Figure 4D. The multimodal output 125 described in Figure 1 may depend on the type of use case and / or the category buttons. Additionally, in some embodiments, as described in the first user interface 210 in Figure 2, the user can select category buttons 212, 214 associated with a use case.

[0054] Figure 4A shows an information query use case 400 according to an exemplary embodiment of the present disclosure. In the information query use case 400, the system may assist a user in obtaining information. For example, a user may film a video clip of a vehicle with the hood open and provide a voice command, "Where do I put the coolant?" In this use case, the system may present information about the scene, such as the location of the oil dipstick 402, the engine 404, the battery 406, and contextual information 408, and highlight the answer, which is the location of the radiator reservoir 412 for adding coolant.

[0055] In other information query use cases (not shown), a user might film a video of a noisy electric fan and record a voice command such as, "Please help me fix it. It's making a clicking sound." The video clip could be of the electric fan while it's powered on, with the video clip in question (e.g., the clicking noise). This type of information query can be very difficult, if not impossible, for a user to express this intent in a single modality and obtain an answer. Audio, visual, and context from what the user is trying to do can be input into the system to generate an answer.

[0056] Figure 4B shows a scene exploration use case 420 according to an exemplary embodiment of the present disclosure. In the scene exploration use case 420, in a first user interface 422, the user may provide a voice command 424, “Show me the sugar content of each product,” while recording a video clip of an aisle in a grocery store. In a second user interface 426, the system may determine several target objects 428, 430, 432 in the video clip and determine the nutritional value of each target object. In a third user interface 434, the system may either directly display the nutritional value of each target object on the mobile device, or select a target object 436 based on an analysis of the nutritional value 438 of each target object.

[0057] Figure 4C shows a motion tutorial use case 460 according to an exemplary embodiment of the present disclosure. A user may request the system to learn dance movements from a dance video clip having a sequence of image frames. The sequence of image frames may include a first image frame 462, a second image frame 464, and a third image frame 466. A machine-trained model may determine video embeddings and temporal information by analyzing the sequence of image frames. In this use case 460, the system may determine different motions by analyzing the video and provide the user with a tutorial video for performing the dance movements.

[0058] Figure 4D shows a use case 480 of an augmented reality experience according to an exemplary embodiment of the present disclosure. The user may ask the system to change the color 492 of an object 494 in a video or to add an object to the video. In this example, the attribute (e.g., color) of the target object (e.g., rag) is changed to a different attribute 496.

[0059] Exemplary embodiments of this disclosure will now be described in more detail with reference to the drawings.

[0060] Exemplary Method Figure 5 shows a flowchart of an exemplary method for performing a multimodal search using a server with a machine learning model, according to an exemplary embodiment of the present disclosure. While Figure 5 shows steps performed in a specific order for illustrative and explanatory purposes, the method of the present disclosure is not limited to the order or arrangement described in detail. Various steps of Method 500 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of the present disclosure.

[0061] Method 500 can be performed by a computing system such as the multimodal search system 100, user device 310, server(s) 320, server computing system 730, sensor processing system 60, or output determination system 80 shown in Figure 1.

[0062] In 502, the computing system (e.g., multimodal search system 100, server(s) 320, server computing system 730) may receive video data captured by the user device's camera. The video data may consist of a sequence of image frames. In some examples, the user device in 502 may be user device 310 in Figure 3A, user computing system 702 in Figure 7A, or user computing system 52 in Figure 7B.

[0063] In 504, the computing system may receive audio data associated with video data captured by the user device.

[0064] In some examples, the method may further include processing audio data to generate text queries using automatic speech recognition technology. The determination of one or more video results may further be based on the text queries. In some examples, the method may further include processing the text queries and video embeddings in parallel to determine one or more video results using one or more machine learning models.

[0065] In some examples, the method may include processing audio data to generate an input audio signature. The determination of one or more video results may further be based on the input audio signature. Additionally, the determination of one or more search results may include detecting a target object within the video data. Furthermore, the method may include accessing multiple known audio signatures associated with the target object from an audio signature database. Furthermore, the method may include selecting a matching audio signature from the known audio signatures, where the comparison score of the matching audio signature exceeds a threshold. The matching score of the matching audio signature may be calculated by comparing the input audio signature with the matching audio signature. One or more video results may further be determined based on the matching audio signature.

[0066] In 506, the computing system may use one or more machine learning models to process a sequence of image frames in order to generate a video embedding associated with the sequence of image frames. The video embedding may have multiple image embeddings associated with the sequence of image frames.

[0067] In some examples, processing a sequence of image frames to generate a video embedding may involve using a frame selection algorithm to select a subset of image frames from the sequence. Additionally, the method may involve processing each image frame within the subset of image frames to generate multiple image embeddings. Multiple image embeddings may have an image embedding for each image frame within the subset of image frames. Furthermore, the method may involve determining a video embedding based on the multiple image embeddings. The video embedding may be determined by averaging each of the multiple image embeddings. The frame selection algorithm may be a uniform random sampling of frames, and the subset of image frames may be selected at regular intervals, such as every n frames. Additionally or alternatively, the frame selection algorithm may be based on the camera position and camera orientation.

[0068] In some examples, processing a sequence of image frames to generate a video embedding may involve detecting a target object within the video data. Additionally, the frame selection algorithm is based on the spatial relationship between the camera and the target object. Furthermore, the spatial relationship may include a translation vector representing the camera's position in 3D space relative to the target object. The translation vector may specify the distance the camera is displaced relative to the target object. Furthermore, the spatial relationship may include a rotation matrix representing the camera's orientation in 3D space relative to the target object. The orientation matrix may specify the degree of rotation of the camera relative to the target object.

[0069] In some examples, the frame selection algorithm may be running on the user device, and the received video data is a subset of images from a sequence of image frames. Video embeddings can be generated by processing the subset of image frames. Additionally, video embeddings may have temporal information associated with the subset of image frames.

[0070] In some examples, the method may further involve processing a sequence of image frames using one or more machine learning models to generate temporal information associated with the sequence of image frames. The decision of one or more video results may further be based on the temporal information associated with the sequence of image frames.

[0071] In some examples, processing a sequence of image frames to generate a video embedding may involve processing each image frame in the sequence to generate a video embedding using one or more machine learning models. Additionally, the method may involve mapping the generated video embeddings to a video embedding index. One or more video results may be determined based on the mapping of the generated video embeddings to the video embedding index. Furthermore, the mapping of the generated video embeddings to the video embedding index may involve ranking each video embedding in the video embedding index based on a comparison with the generated video embeddings. Each video embedding may be associated with a video result. One or more video results may be determined based on the ranking of each video embedding in the video embedding index.

[0072] In some examples, one or more machine learning models may include a polymastic model. A polymastic model refers to a machine learning model designed to demonstrate broad knowledge and skills across multiple domains. LLMs can leverage polymastic models to understand and generate content across different domains. Polymastic models can be used for a variety of tasks, such as natural language understanding, question answering, language conversion, text summarization, and creative writing. One or more machine learning models can be trained with vast amounts of data from diverse sources, enabling the models to acquire knowledge from a wide range of domains. For example, one or more machine learning models could be trained using tutorial videos published on online video sharing platforms.

[0073] In some examples, one or more machine learning models may include a multimodal, multitask unified model trained to understand information from multiple formats, including video and text.

[0074] In 508, the computing system may determine one or more video results based on video embedding and audio data.

[0075] In 510, the computing system may transmit one or more video results to a user device.

[0076] In some examples, the method may further include determining one or more web results based on a video embed. Additionally, the method may include sending one or more web results to a user device.

[0077] In some examples, the method may further involve using one or more machine learning models to process audio data along with a sequence of image frames to generate a video embedding.

[0078] In an exemplary embodiment, the system receives video data captured by a user device's camera via a computing system including one or more processors, the video data having a sequence of image frames. The system may receive audio data associated with the video data captured by the user device. The system may process the sequence of image frames using one or more machine learning models to generate video embeddings associated with the sequence of image frames, the video embeddings having multiple image embeddings associated with the sequence of image frames. The system may determine one or more video results based on the video embeddings and audio data. Additionally, the system may transmit one or more video results to the user device. In this exemplary embodiment, processing the sequence of image frames to generate video embeddings may include: selecting a subset of image frames from the sequence of image frames using a frame selection algorithm; processing each image frame in the subset of image frames to generate multiple image embeddings, the multiple image embeddings having an image embedding for each image frame in the subset of image frames; and determining a video embedding based on the multiple image embeddings. Additionally, in this exemplary embodiment, processing a sequence of image frames to generate a video embedding may include processing each image frame in the sequence of image frames to generate a video embedding using one or more machine learning models, and mapping the generated video embedding to a video embedding index, where one or more video results are determined based on the mapping of the generated video embedding to the video embedding index.

[0079] Figure 6 shows a flowchart of an exemplary method for performing multimodal search using a user device with a machine learning model, according to an exemplary embodiment. While Figure 6 shows steps performed in a specific order for illustrative and explanatory purposes, the methods of this disclosure are not limited to the order or arrangement shown. Various steps of Method 600 may be omitted, rearranged, combined, and / or adapted in various ways without departing from the scope of this disclosure.

[0080] Method 600 can be carried out by a computing system such as the multimodal search system 100 in Figure 1, the user device 355 in Figure 3B, the user computing system 702, the sensor processing system 60, or the output determination system 80.

[0081] In 602, the computing system may use a camera to capture video data having a sequence of image frames.

[0082] In 604, the computing system may use a microphone to capture audio data associated with video data. In some examples, the method may further include processing the audio data to generate a text query using automatic speech recognition technology. The determination of one or more video results may further be based on the text query. In some examples, the method may further include processing the text query and video embedding in parallel to determine one or more video results using one or more machine-trained models.

[0083] In some examples, the method may include processing audio data to generate an input audio signature. The determination of one or more video results may further be based on the input audio signature. Additionally, the determination of one or more search results may include detecting a target object within the video data. Furthermore, the method may include accessing multiple known audio signatures associated with the target object from an audio signature database. Furthermore, the method may include selecting a matching audio signature from the known audio signatures, where the comparison score of the matching audio signature exceeds a threshold. The matching score of the matching audio signature may be calculated by comparing the input audio signature with the matching audio signature. One or more video results may further be determined based on the matching audio signature.

[0084] In 606, a computing system may use one or more machine-trained models stored on a user device to process a sequence of audio data and image frames to generate a video embedding. The video embedding may be derived by processing the video data in parallel with the audio data. In some examples, the method may further include using one or more machine-trained models to process the audio data together with a sequence of image frames to generate a video embedding.

[0085] In some examples, processing a sequence of image frames to generate a video embedding may involve using a frame selection algorithm to select a subset of image frames from the sequence. Additionally, the method may involve processing each image frame within the subset of image frames to generate multiple image embeddings. Multiple image embeddings may have an image embedding for each image frame within the subset of image frames. Furthermore, the method may involve determining a video embedding based on the multiple image embeddings. The video embedding may be determined by averaging each of the multiple image embeddings. The frame selection algorithm may be a uniform random sampling of frames, and the subset of image frames may be selected at regular intervals, such as every n frames. Additionally or alternatively, the frame selection algorithm may be based on the camera position and camera orientation.

[0086] In some examples, processing a sequence of image frames to generate a video embedding may involve detecting a target object within the video data. Additionally, the frame selection algorithm is based on the spatial relationship between the camera and the target object. Furthermore, the spatial relationship may include a translation vector representing the camera's position in 3D space relative to the target object. The translation vector may specify the distance the camera is displaced relative to the target object. Furthermore, the spatial relationship may include a rotation matrix representing the camera's orientation in 3D space relative to the target object. The orientation matrix may specify the degree of rotation of the camera relative to the target object.

[0087] In some examples, the frame selection algorithm may be running on the user device, and the received video data is a subset of images from a sequence of image frames. Video embeddings can be generated by processing the subset of image frames. Additionally, video embeddings may have temporal information associated with the subset of image frames.

[0088] In some examples, the method may further involve processing a sequence of image frames using one or more machine learning models to generate temporal information associated with the sequence of image frames. The decision of one or more video results may further be based on the temporal information associated with the sequence of image frames.

[0089] In some examples, processing a sequence of image frames to generate a video embedding may involve processing each image frame in the sequence to generate a video embedding using one or more machine learning models. Additionally, the method may involve mapping the generated video embeddings to a video embedding index. One or more video results may be determined based on the mapping of the generated video embeddings to the video embedding index. Furthermore, the mapping of the generated video embeddings to the video embedding index may involve ranking each video embedding in the video embedding index based on a comparison with the generated video embeddings. Each video embedding may be associated with a video result. One or more video results may be determined based on the ranking of each video embedding in the video embedding index.

[0090] In some examples, one or more machine learning models may include polymass models. Additionally, one or more machine learning models may be trained using tutorial videos published on online video sharing platforms.

[0091] In some examples, one or more machine learning models may include a multimodal, multitask unified model trained to understand information from multiple formats, including video and text.

[0092] In 608, the computing system may transmit a video embed to the server.

[0093] In 610, in response to the transmission of video embedding and audio data, the computing system may receive one or more video results from the server.

[0094] In 612, the computing system may present one or more video results on the display of a user device. In some examples, the method may further include receiving one or more web results based on a video embedding. Additionally, the method may include transmitting one or more web results to the user device.

[0095] Figure 7A shows a block diagram of an exemplary computing system 700 that performs multimodal search according to an exemplary embodiment of the present disclosure. The system 700 includes a user computing system 702, a server computing system 730, and / or a third computing system 750, which are communicably coupled via a network 780.

[0096] The user computing system 702 may include any type of computing device, such as a personal computing device (e.g., a laptop or desktop), a mobile computing device (e.g., a smartphone or tablet), a game console or controller, a wearable computing device, an embedded computing device, or any other type of computing device.

[0097] The user computing system 702 includes one or more processors 712 and memory 714. The one or more processors 712 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 714 may include one or more non-temporary computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 714 may store data 716 and instructions 718 executed by the processors 712 to cause the user computing system 702 to perform an operation.

[0098] In some embodiments, the user computing system 702 may store or include one or more machine learning models 720. For example, the machine learning models 720 may be or include various machine learning models such as neural networks (e.g., deep neural networks), or other types of machine learning models including nonlinear and / or linear models. Neural networks may include feedforward neural networks, recurrent neural networks (e.g., long-short-term memory recurrent neural networks), convolutional neural networks, or other forms of neural networks. The machine learning models 720 may be examples of machine learning models used in query refinement 108 in Figure 1, query refinement 210 in Figure 2, and / or methods 400, 500, and 600.

[0099] In some embodiments, one or more machine learning models 720 can be received from a server computing system 730 via a network 780, stored in a user computing device memory 714, and then used or implemented by one or more processors 712. In some embodiments, the user computing system 702 may implement multiple parallel instances of a single machine learning model 720 (for example, to perform parallel machine learning model processing across multiple instances of input data and / or detected features).

[0100] More specifically, one or more machine learning models 720 may include one or more detection models, one or more classification models, one or more segmentation models, one or more augmentation models, one or more generative models, one or more natural language processing models, one or more optical character recognition models, and / or one or more other machine learning models. One or more machine learning models 720 may include one or more transformer models. One or more machine learning models 720 may include one or more neural radiance field models, one or more diffusion models, and / or one or more autoregressive language models.

[0101] One or more machine learning models 720 may be used to detect one or more object features. The detected object features may be classified and / or embedded. The classification and / or embedding may then be used to perform a search and determine one or more search results. Alternatively and / or additionally, one or more detected features may be used to decide to provide an indicator (e.g., a user interface element indicating a detected feature) to show that a feature has been detected. The user may then select the indicator to perform classification, embedding, and / or searching of the feature. In some embodiments, classification, embedding, and / or searching may be performed before the indicator is selected.

[0102] In some embodiments, one or more machine-trained models 720 can process image data, text data, audio data, and / or latent encoded data to generate output data which may include image data, text data, audio data, and / or latent encoded data. One or more machine-trained models 720 can perform optical character recognition, natural language processing, image classification, object classification, text classification, audio classification, context determination, action prediction, image correction, image augmentation, text augmentation, sentiment analysis, object detection, error detection, inpainting, video stabilization, audio correction, audio augmentation, and / or data segmentation (e.g., mask-based segmentation).

[0103] Additionally or alternatively, one or more machine learning models 740 may be included in, or stored and implemented by, a server computing system 730 that communicates with a user computing system 702 according to a client-server relationship. For example, a machine learning model 740 may be implemented by the server computing system 740 as part of a web service (e.g., a viewfinder service, a visual search service, an image processing service, an ambient computing service, and / or an overlay application service). Thus, one or more models 720 may be stored and implemented in the user computing system 702, and / or one or more models 740 may be stored and implemented in the server computing system 730.

[0104] The user computing system 702 may also include one or more user input components 722 that receive user input (e.g., video data 102, audio data 104). For example, a user input component 722 may be a touch-sensitive component (e.g., a touch-sensitive display screen or touchpad) that is sensitive to the touch of a user input object (e.g., a finger or stylus). The touch-sensitive component may serve to implement a virtual keyboard. Other exemplary user input components include a microphone, a conventional keyboard, or other means by which the user can provide user input.

[0105] In some embodiments, a user computing system may store and / or provide one or more user interfaces 724 that may be associated with one or more applications. One or more user interfaces 724 may be configured to receive input and / or provide data for display (e.g., image data, text data, audio data, one or more user interface elements, augmented reality experiences, virtual reality experiences) and / or other data for display. A user interface 724 may be associated with one or more other computing systems (e.g., a server computing system 730 and / or a third-party computing system 750). A user interface 724 may include a viewfinder interface, a search interface, a generative model interface, a social media interface, and / or a media content gallery interface.

[0106] The user computing system 702 may include and / or receive data from one or more sensors 726 (e.g., image data 202, audio data 204). One or more sensors 726 may be housed in a housing component that houses one or more processors 712, memory 714, and / or one or more hardware components, the housing component may store and / or execute one or more software packets. One or more sensors 726 may include one or more image sensors (e.g., cameras), one or more lidar sensors, one or more audio sensors (e.g., microphones), one or more inertial sensors (e.g., inertial measurement units), one or more biosensors (e.g., heart rate sensors, pulse sensors, retinal sensors, and / or fingerprint sensors), one or more infrared sensors, one or more position sensors (e.g., GPS), one or more touch sensors (e.g., conductive touch sensors and / or mechanical touch sensors), and / or one or more other sensors. One or more sensors may be used to acquire data associated with the user's environment (e.g., images of the user's environment, records of the environment, and / or the user's location).

[0107] The user computing system 702 may include and / or be part of a user computing device 704. The user computing device 704 may include a mobile computing device (e.g., a smartphone or tablet), a desktop computer, a laptop computer, a smart wearable, and / or a smart appliance. Additionally and / or alternatively, the user computing system may acquire data from and / or generate data using one or more user computing devices 704. For example, a smartphone camera may be used to capture image data describing the environment, and / or an overlay application on a user computing device 704 may be used to track and / or process data provided to the user. Similarly, one or more sensors associated with a smart wearable may be used to acquire data about the user and / or data about the user's environment (e.g., image data may be acquired by a camera housed in the user's smart glasses). Additionally and / or alternatively, data may be acquired and uploaded from other user devices that may be specialized in data acquisition or generation.

[0108] The server computing system 730 includes one or more processors 732 and memory 734. The one or more processors 732 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.) and may be a single processor or multiple operably connected processors. The memory 734 may include one or more non-temporary computer-readable storage media such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, etc., and combinations thereof. The memory 734 may store data 736 and instructions 738 executed by the processors 732 to cause the server computing system 730 to perform operations.

[0109] In some embodiments, the server computing system 730 includes or is implemented by one or more server computing devices. In examples where the server computing system 730 includes multiple server computing devices, such server computing devices may operate according to a sequential computing architecture, a parallel computing architecture, or some combination thereof.

[0110] As described above, the server computing system 730 may store or include one or more machine learning models 740. For example, the model 740 may be a variety of machine learning models, or may otherwise include a variety of machine learning models. Exemplary machine learning models include neural networks or other multilayer nonlinear models. Exemplary neural networks include feedforward neural networks, deep neural networks, recurrent neural networks, and convolutional neural networks. Exemplary model 740 is illustrated with reference to Figure 7B. Machine learning model 740 may be an example of a machine learning model used in query refinement 108 in Figure 1, query refinement 210 in Figure 2, and / or methods 400, 500, and 600.

[0111] Additionally and / or alternatively, the server computing system 730 may include and / or be communicably connected to a search engine 742 which can be used to crawl one or more databases (and / or resources). The search engine 742 may process data from the user computing system 702, the server computing system 730, and / or third-party computing system 750 to determine one or more search results associated with input data. The search engine 742 may perform term-based search, label-based search, Boolean-based search, image search, embedding-based search (e.g., nearest neighbor search), multimodal search, and / or one or more other search techniques.

[0112] The server computing system 730 may store and / or provide one or more user interfaces 744 for acquiring input data and / or providing output data to one or more users. One or more user interfaces 744 may include one or more user interface elements, which may include input fields, navigation tools, content chips, selectable tiles, widgets, data display carousels, dynamic animations, information popups, image augmentation, text-to-speech, speech-to-text, augmented reality, virtual reality, feedback loops, and / or other interface elements.

[0113] The user computing system 702 and / or the server computing system 730 may train models 720 and / or 740 through interaction with a third-party computing system 750 that is communicatively coupled via the network 780. The third-party computing system 750 may be separate from the server computing system 730 or may be part of the server computing system 730. Alternatively and / or additionally, the third-party computing system 750 may be associated with one or more web resources, one or more web platforms, one or more other users, and / or one or more contexts.

[0114] The third-party computing system 750 may include one or more processors 752 and memory 754. The one or more processors 752 may be any suitable processing device (e.g., a processor core, microprocessor, ASIC, FPGA, controller, microcontroller, etc.), and may be a single processor or multiple operably connected processors. The memory 754 may include one or more non-temporary computer-readable storage media, such as RAM, ROM, EEPROM, EPROM, flash memory devices, magnetic disks, and combinations thereof. The memory 754 may store data 756 and instructions 758 executed by the processors 752 to cause the third-party computing system 750 to perform operations. In some embodiments, the third-party computing system 750 includes one or more server computing devices or is otherwise implemented by them.

[0115] Network 780 may be any type of communication network, such as a local area network (e.g., an intranet), a wide area network (e.g., the Internet), or a combination thereof, and may include any number of wired or wireless links. In general, communication over Network 780 can be transmitted over any type of wired and / or wireless connection using a wide variety of communication protocols (e.g., TCP / IP, HTTP, SMTP, FTP), encoding or formatting (e.g., HTML, XML), and / or protection methods (e.g., VPN, Secure HTTP, SSL).

[0116] The machine learning models described herein may be used in a variety of tasks, applications, and / or use cases.

[0117] In some embodiments, the input to the machine learning model(s) of this disclosure may be image data (e.g., image data 202). The machine learning model(s) may process the image data to produce an output. For example, the machine learning model(s) may process the image data to produce an image recognition output (e.g., recognition of image data, latent embedding of image data, encoded representation of image data, hash of image data, etc.). As another example, the machine learning model(s) may process the image data to produce an image segmentation output. As yet another example, the machine learning model(s) may process the image data to produce an image classification output. As yet another example, the machine learning model(s) may process the image data to produce an image data modification output (e.g., modification of image data, etc.). As yet another example, the machine learning model(s) may process the image data to produce an encoded image data output (e.g., encoded representation and / or compressed representation of image data, etc.). As yet another example, the machine learning model(s) may process the image data to produce an upscaled image data output. As another example, machine learning models can process image data and generate predictive outputs.

[0118] In some embodiments, the input to the machine learning model(s) of this disclosure may be text or natural language data. The machine learning model(s) can process the text or natural language data to produce an output. For example, the machine learning model(s) can process natural language data to produce a language coding output. As another example, the machine learning model(s) can process text or natural language data to produce a latent text embedding output. As yet another example, the machine learning model(s) can process text or natural language data to produce a translation output. As yet another example, the machine learning model(s) can process text or natural language data to produce a classification output. As yet another example, the machine learning model(s) can process text or natural language data to produce a text segmentation output. As yet another example, the machine learning model(s) can process text or natural language data to produce a semantic intent output. As another example, a machine learning model(s) can process text or natural language data to produce upscaled text or natural language output (e.g., text or natural language data of higher quality than the input text or natural language). As yet another example, a machine learning model(s) can process text or natural language data to produce predictive output.

[0119] In some embodiments, the input to the machine learning model(s) of this disclosure may be speech data (e.g., audio data 204). The machine learning model(s) can process the speech data to produce an output. For example, the machine learning model(s) can process the speech data to produce a speech recognition output. As another example, the machine learning model(s) can process the speech data to produce a speech translation output. As yet another example, the machine learning model(s) can process the speech data to produce a latent embedding output. As yet another example, the machine learning model(s) can process the speech data to produce an encoded speech output (e.g., an encoded representation and / or compressed representation of the speech data). As yet another example, the machine learning model(s) can process the speech data to produce an upscaled speech output (e.g., speech data of higher quality than the input speech data). As yet another example, the machine learning model(s) can process the speech data to produce a text representation output (e.g., a text representation of the input speech data). As another example, a machine learning model(s) can process speech data and generate predictive outputs.

[0120] In some embodiments, the input to the machine learning model(s) of this disclosure may be sensor data (e.g., image data 202, audio data 204). The machine learning model(s) can process the sensor data to generate an output. For example, the machine learning model(s) can process the sensor data to generate a recognition output. As another example, the machine learning model(s) can process the sensor data to generate a prediction output. As yet another example, the machine learning model(s) can process the sensor data to generate a classification output. As yet another example, the machine learning model(s) can process the sensor data to generate a segmentation output. As yet another example, the machine learning model(s) can process the sensor data to generate a segmentation output. As yet another example, the machine learning model(s) can process the sensor data to generate a visualization output. As yet another example, the machine learning model(s) can process the sensor data to generate a diagnostic output. As yet another example, the machine learning model(s) can process the sensor data to generate a detection output.

[0121] In some cases, the input includes visual data (e.g., image data 202), and the task is a computer vision task. In other cases, the input includes pixel data from one or more images, and the task is an image processing task. For example, the image processing task may be image classification, and the output is a set of scores, each corresponding to a different object class, representing the likelihood that one or more images depict an object belonging to that object class. The image processing task may be object detection, and the image processing output identifies one or more regions within one or more images, and for each region, the likelihood that the region depicts an object of interest. As another example, the image processing task may be image segmentation, and the image processing output defines, for each pixel in one or more images, the likelihood for each category within a given set of categories. For example, the set of categories may be foreground and background. As yet another example, the set of categories may be object classes. As yet another example, the image processing task may be depth estimation, and the image processing output defines, for each pixel in one or more images, the respective depth value. As another example, an image processing task could be motion estimation, where the network input includes multiple images, and the image processing output determines the motion of the scene depicted in the pixels between the images in the network input, with each pixel in one of the input images representing the motion of the scene.

[0122] A user computing system may include several applications (e.g., applications 1 through N). Each application may include its own machine learning library and its own trained model(s). For example, each application may include a trained model. Illustrative applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, and browser applications.

[0123] Each application can communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, each application can communicate with each device component using an API (e.g., a public API). In some embodiments, the API used by each application is specific to that application.

[0124] The user computing system 702 may include several applications (e.g., applications 1 to N). Each application communicates with the central intelligence layer. Exemplary applications include text messaging applications, email applications, dictation applications, virtual keyboard applications, and browser applications. In some embodiments, each application may communicate with the central intelligence layer (and the model(s) stored within it) using an API (e.g., a common API across all applications).

[0125] The central intelligence layer may include several machine learning models. For example, each machine learning model (e.g., Model) may be provided for each application and managed by the central intelligence layer. In other embodiments, two or more applications may share a single machine learning model. For example, in some embodiments, the central intelligence layer may provide a single model (e.g., Single Model) for all applications. In some embodiments, the central intelligence layer may be contained within the operating system of the computing system 700 or implemented by the operating system of the computing system 700.

[0126] The central intelligence layer can communicate with the central device data layer. The central device data layer may be a centralized repository of data for the computing system 700. The central device data layer may communicate with several other components of the computing device, such as one or more sensors, a context manager, a device state component, and / or additional components. In some embodiments, the central device data layer may communicate with each device component using an API (e.g., a private API).

[0127] Figure 7B shows a block diagram of an exemplary computing system 50 performing multimodal search according to an exemplary embodiment of the present disclosure. In particular, the exemplary computing system 50 may include one or more computing devices 52 that can be used to acquire and / or generate one or more datasets which can be processed by a sensor processing system 60 and / or an output determination system 80, and to provide feedback to a user which can provide information about the features of one or more acquired datasets. One or more datasets may include image data, text data, audio data, multimodal data, latent encoded data, etc. One or more datasets may be acquired via one or more sensors associated with one or more computing devices 52 (e.g., one or more sensors of computing device 52). Additionally and / or alternatively, one or more datasets may be stored data and / or acquired data (e.g., data acquired from web resources). For example, images, text, and / or other content items may be interacted with by the user. Interaction with the content items can then be used to generate one or more decisions.

[0128] One or more computing devices 52 can acquire and / or generate one or more datasets based on image capture, sensor tracking, data storage acquisition, content download (e.g., downloading images or other content items from web resources via the Internet), and / or via one or more other techniques. One or more datasets can be processed by a sensor processing system 60. The sensor processing system 60 can perform one or more processing techniques using one or more machine learning models, one or more search engines, and / or one or more other processing techniques. One or more processing techniques can be performed in any combination and / or individually. One or more processing techniques can be performed sequentially and / or in parallel. In particular, one or more datasets may be processed by a context determination block 62, which may determine the context associated with one or more content items. The context determination block 62 may identify and / or process metadata, user profile data (e.g., preferences, user search history, user browsing history, user purchase history, and / or user input data), previous interaction data, global trend data, location data, time data, and / or other data in order to determine a specific context associated with a user. The context may be associated with an event, a determined trend, a specific action, a specific type of data, a specific environment, and / or other contexts associated with the user and / or the searched or retrieved data.

[0129] The sensor processing system 60 may include an image preprocessing block 64. The image preprocessing block 64 may be used to adjust one or more values ​​of the acquired and / or received images to prepare the images for processing by one or more machine learning models and / or one or more search engines 74. The image preprocessing block 64 may resize the images, adjust saturation values, adjust resolution, remove and / or add metadata, and / or perform one or more other operations.

[0130] In some embodiments, the sensor processing system 60 may include one or more machine learning models, which may include a detection model 66, a segmentation model 68, a classification model 70, an embedding model 72, and / or one or more other machine learning models. For example, the sensor processing system 60 may include one or more detection models 66 that can be used to detect specific features of a processed dataset. In particular, one or more images may be processed by one or more detection models 66 to generate one or more bounding boxes associated with features detected in one or more images.

[0131] Additionally and / or alternatively, one or more segmentation models 68 can be used to segment one or more parts of a dataset from one or more datasets. For example, one or more segmentation models 68 can use one or more segmentation masks (e.g., one or more segmentation masks generated manually and / or based on one or more bounding boxes) to segment parts of an image, parts of an audio file, and / or parts of text. Segmentation may include isolating one or more detected objects and / or removing one or more detected objects from an image.

[0132] One or more classification models 70 can be used to process image data, text data, audio data, latent encoded data, multimodal data, and / or other data to generate one or more classifications. One or more classification models 70 may include one or more image classification models, one or more object classification models, one or more text classification models, one or more audio classification models, and / or one or more other classification models. One or more classification models 70 can process the data to determine one or more classifications.

[0133] In some embodiments, data can be processed by one or more embedding models 72 to generate one or more embeddings (e.g., text embedding 112, image embedding 114). For example, one or more images can be processed by one or more embedding models 72 to generate one or more image embeddings (e.g., image embedding 114) in the embedding space. The one or more image embeddings may be associated with one or more image features of one or more images. In some embodiments, one or more embedding models 72 may be configured to process multimodal data to generate multimodal embeddings. The one or more embeddings can be used for classification, retrieval, and / or learning of embedding space distributions.

[0134] The sensor processing system 60 may include one or more search engines 74 that can be used to perform one or more searches. One or more search engines 74 may crawl one or more databases (e.g., one or more local databases, one or more global databases, one or more private databases, one or more public databases, one or more dedicated databases, and / or one or more general-purpose databases) to determine one or more search results. One or more search engines 74 may perform feature matching, text-based search, embedding-based search (e.g., k-nearest neighbor search), metadatabase search, multimodal search, web resource search, image search, text search, and / or application search.

[0135] Additionally and / or alternatively, the sensor processing system 60 may include one or more multimodal processing blocks 76 that can be used to assist in the processing of multimodal data. One or more multimodal processing blocks 76 may include generating multimodal queries and / or multimodal embeddings that are processed by one or more machine learning models and / or one or more search engines 74.

[0136] The output(s) of the sensor processing system 60 are then processed by the output determination system 80 to determine one or more outputs to be provided to the user. The output determination system 80 may include heuristic-based determination, machine learning model-based determination, user selection-based determination, and / or context-based determination.

[0137] The output determination system 80 may determine how and / or where to provide one or more search results in the search results interface 82. Additionally and / or alternatively, the output determination system 80 may determine how and / or where to provide one or more machine learning model outputs in the machine learning model output interface 84. In some embodiments, one or more search results and / or one or more machine learning model outputs may be provided for display via one or more user interface elements. One or more user interface elements may be overlaid on the displayed data. For example, one or more detection indicators may be overlaid on detected objects in the viewfinder. One or more user interface elements may be selectable to perform one or more additional searches and / or one or more additional machine learning model processes. In some embodiments, user interface elements may be provided as user interface elements dedicated to a particular application and / or uniformly provided across different applications. One or more user interface elements may include pop-up displays, interface overlays, interface tiles and / or chips, carousel interfaces, audio feedback, animations, interactive widgets, and / or other user interface elements.

[0138] Additionally and / or alternatively, data associated with the output(s) of the sensor processing system 60 may be used to generate and / or provide an augmented reality experience and / or a virtual reality experience 86. For example, processing one or more acquired datasets may generate one or more augmented reality rendering assets and / or one or more virtual reality rendering assets, which can then be used to bring an augmented reality experience and / or a virtual reality experience 86 to the user. The augmented reality experience may render information associated with the environment into its respective environment. Alternatively and / or additionally, objects associated with the processed dataset(s) may be rendered within the user's environment and / or virtual environment. Generating rendering datasets may involve training one or more neural radiiance field models to learn three-dimensional representations of one or more objects.

[0139] In some embodiments, one or more action prompts 88 may be determined based on the output(s) of the sensor processing system 60. For example, a search prompt, a purchase prompt, a generation prompt, a reservation prompt, a call prompt, a redirect prompt, and / or one or more other prompts may be determined to be associated with the output(s) of the sensor processing system 60. The one or more action prompts 88 may then be provided to the user via one or more selectable user interface elements. In response to the selection of one or more selectable user interface elements, the respective actions of the respective action prompts may be performed (e.g., a search may be performed, a purchase application programming interface may be made available, and / or other applications may be opened).

[0140] In some embodiments, one or more datasets and / or outputs of the sensor processing system 60 can be processed by one or more generative models 90 to generate model-generated content items, which can then be provided to the user. Generation may be prompted based on user selection and / or may be performed automatically (for example, automatically based on one or more conditions, which may be associated with search results of unidentified threshold amounts).

[0141] The output determination system 80 can process one or more datasets and / or the output(s) of the sensor processing system 60 in the data augmentation block 92 to generate augmented data. For example, one or more images can be processed in the data augmentation block 92 to generate one or more augmented images. Data augmentation can include data correction, data cropping, removal of one or more features, addition of one or more features, resolution adjustment, lighting adjustment, saturation adjustment, and / or other augmentations.

[0142] In some embodiments, one or more datasets and / or outputs of the sensor processing system 60 may be stored based on a decision in the data storage block 94.

[0143] Next, the output(s) of the output determination system 80 may be provided to the user via one or more output components of the user computing device 52. For example, one or more user interface elements associated with one or more outputs may be provided for display via the visual display of the user computing device 52.

[0144] Processing can be performed iteratively and / or sequentially. One or more user inputs provided to user interface elements may coordinate and / or influence the sequential processing loop.

[0145] The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed and information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described herein can be performed using a single device or component, or multiple devices or components working together. Databases and applications can be run on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0146] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art will readily be able to create modifications, variations, and equivalents to such embodiments, understanding the foregoing. Therefore, the disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, which will be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used together with other embodiments to create yet another embodiment. Therefore, the disclosure is intended to cover such modifications, variations, and equivalents.

[0147] The technologies described herein refer to servers, databases, software applications, and other computer-based systems, as well as actions performed and information transmitted to and from such systems. The inherent flexibility of computer-based systems allows for a wide variety of feasible configurations, combinations, and divisions of tasks and functions between components. For example, the processes described herein can be performed using a single device or component, or multiple devices or components working together. Databases and applications can be run on a single system or distributed across multiple systems. Distributed components can operate sequentially or in parallel.

[0148] While the subject matter has been described in detail with respect to various specific exemplary embodiments, each example is provided for illustrative purposes only and does not limit the disclosure. Those skilled in the art will readily be able to create modifications, variations, and equivalents to such embodiments, understanding the foregoing. Therefore, the disclosure does not exclude the inclusion of such modifications, variations, and / or additions to the subject matter, which will be readily apparent to those skilled in the art. For example, features illustrated or described as part of one embodiment may be used together with other embodiments to create yet another embodiment. Therefore, the disclosure is intended to cover such modifications, variations, and equivalents.

Claims

1. A computer-implemented method for multimodal searching of video results, Receiving video data captured by a user device's camera using a computing system including one or more processors, wherein the video data has a sequence of image frames. Receiving audio data associated with the video data captured by the user device, Processing a sequence of image frames to generate a video embedding associated with the sequence of image frames using one or more machine learning models, wherein the video embedding has a plurality of image embeddings associated with the sequence of image frames. Determining one or more video results based on the aforementioned video embedding and the aforementioned audio data, Transmitting one or more video results to the user device, Methods that include...

2. The method further includes processing the sequence of image frames to generate temporal information relating to the sequence of image frames using one or more machine learning models, The method according to claim 1, wherein the determination of the one or more video results is further based on the temporal information relating to the sequence of the image frames.

3. The process further includes using automatic speech recognition technology to process the audio data in order to generate text queries. The method according to claim 1, wherein the determination of the one or more video results is further based on the text query.

4. Processing the sequence of image frames to generate the aforementioned video embedding is, Selecting a subset of image frames from the sequence of image frames using a frame selection algorithm, Processing each image frame within a subset of the image frames in order to generate multiple image embeddings, wherein each of the multiple image embeddings has an image embedding for each image frame within the subset of the image frames. Determining the video embedding based on the aforementioned plurality of image embeddings, The method according to claim 1, including the method described in claim 1.

5. The method according to claim 4, wherein the video embedding is determined by averaging each of the multiple image embeddings.

6. The method according to claim 4, wherein the frame selection algorithm is uniform random sampling of frames, and a subset of the image frames is selected at regular intervals.

7. The method according to claim 4, wherein the frame selection algorithm is based on the position and orientation of the camera.

8. The aforementioned method, Further includes detecting a target object in the video data, The method according to claim 4, wherein the frame selection algorithm is based on the spatial relationship between the camera and the target object.

9. The method according to claim 8, wherein the spatial relationship includes a translation vector representing the position of the camera in three-dimensional space relative to the target object, and the translation vector specifies the distance the camera is displaced relative to the target object.

10. The method according to claim 8, wherein the spatial relationship includes a rotation matrix representing the orientation of the camera in three-dimensional space relative to the target object, and the orientation matrix specifies the degree of rotation of the camera relative to the target object.

11. The frame selection algorithm operates on the user device, and the received video data is a subset of images from the sequence of image frames. The method according to claim 4, wherein the video embedding is generated by processing a subset of the image frames, and the video embedding has temporal information associated with the subset of the image frames.

12. The aforementioned method, The process further includes processing the audio data to generate an input audio signature, The method according to claim 1, wherein the determination of the one or more video results is further based on the input audio signature.

13. The determination of the one or more search results is, To detect the target object in the aforementioned video data, Accessing multiple known audio signatures associated with the target object from the audio signature database, The selection includes selecting a matching audio signature from the known audio signatures, wherein the comparison score of the matching audio signature exceeds a threshold, and the matching score of the matching audio signature is calculated by comparing the input audio signature with the matching audio signature. The method according to claim 12, wherein the one or more video results are further determined based on the matching audio signature.

14. Processing the sequence of image frames to generate the aforementioned video embedding is, Using one or more machine learning models, process each image frame in the sequence of image frames to generate the video embedding; This includes mapping the generated video embedding to a video embedding index, The method according to claim 1, wherein the one or more video results are determined based on the mapping of the generated video embeddings to the index of the video embeddings.

15. The mapping of the generated video embedding to the index of the video embedding is, The process involves ranking each video embedding in the video embedding index based on a comparison with the generated video embedding, wherein each video embedding is associated with a video result, and the ranking is performed accordingly. The method according to claim 14, wherein the one or more video results are determined based on the ranking of each video embedding in the index of the video embeddings.

16. The method according to claim 1, wherein the one or more machine-trained models include a polymass model trained using tutorial videos published on an online video sharing platform, or a multimodal multitask unified model trained to understand information from multiple formats, the multiple formats including video and text.

17. Determining one or more web results based on the aforementioned video embedding, The method according to claim 1, further comprising transmitting the one or more web results to the user device.

18. The method according to claim 1, further comprising processing the audio data together with the sequence of image frames to generate a video embedding using one or more machine-trained models.

19. A computer-implemented method for multimodal searching of video results, The user device's camera captures video data containing a sequence of image frames, The microphone of the user device captures the audio data associated with the video data, Processing the audio data and the sequence of image frames to generate a video embedding using one or more machine learning models stored on the user device, wherein the video embedding is derived by processing the video data in parallel with the audio data. Sending the aforementioned video embed to the server, In response to the aforementioned video embedding and transmission of the aforementioned audio data, one or more video results are received from the server, To display the one or more video results on the display of the user device, Methods that include...

20. A computing system, One or more processors, The system comprises one or more non-temporary computer-readable media that, when executed by the one or more processors, collectively store instructions that cause the computing system to perform an operation, and the operation is Receiving video data captured by a user device's camera using a computing system including one or more processors, wherein the video data has a sequence of image frames. Receiving audio data associated with the video data captured by the user device, Processing a sequence of image frames to generate a video embedding associated with the sequence of image frames using one or more machine learning models, wherein the video embedding has a plurality of image embeddings associated with the sequence of image frames. Determining one or more video results based on the aforementioned video embedding and the aforementioned audio data, Transmitting one or more video results to the user device, A computing system that includes this.

Images