1191results about "Metadata video data retrieval" patented technology

The invention discloses a video content intelligent management method and system based on multi-modal analysis, and belongs to the field of video content management.The method comprises the steps that multi-modal features of video content are extracted; performing fine structure segmentation on the video content based on the multi-modal features, and segmenting the video into fine segments related to semantics; the fragments obtained after fine structure segmentation are subjected to tagging, and metadata is generated for each fragment; constructing a multi-modal semantic index of the video content based on the generated metadata; according to the method, a semantic retrieval function is provided by utilizing multi-modal semantic indexing, fine fragments required by a user are accurately recalled in combination with text label matching, measurement numerical value scoring, cross-modal vector retrieval and filtering based on a shot language and a shooting technology label, and the user queries and positions a specific time point or the fine fragments in a video through a natural language. According to the invention, the accuracy of video content analysis is improved, the user experience is optimized, and the operation complexity is reduced.

An aspect relates to determining if a request received from a user device is sufficiently similar to a cached request, wherein the request requesting an identification of streamable content meeting one or more criteria. The determination comprises comparing a vector corresponding to the received request with vectors of previously received, cached requests. If the vector corresponding to the received request is sufficiently similar to a first vector of a first previously received, cached request, then a previously generated response corresponding to the first previously received, cached request may be accessed and transmitted to the user device. If a sufficiently similar vector is not identified, then at least a portion of the received request may be transmitted, in association with an identification of content items in a library, to an artificial intelligence learning engine. The response from the artificial intelligence learning engine may be transmitted to the user device.

The invention provides a multi-modal-based data retrieval enhancement method, which solves the problems of semantic gaps and outdated information among different modal data through cross-modal semantic alignment and real-time data source integration. And secondly, by constructing a multi-modal feature space and introducing a dynamic retrieval layer, the retrieval speed and accuracy are improved. In addition, by combining sparse retrieval and dense retrieval methods, the accuracy and the recall rate are balanced, and the retrieval result is optimized. And finally, through the steps of post-processing and result optimization, the correlation of the retrieval result and the user experience are further improved. The method is not only suitable for data retrieval of common modes such as texts, images and videos, but also can be expanded to any multi-mode scene, and provides powerful technical support for cross-mode retrieval.

The invention relates to the technical field of disaster monitoring, in particular to a mine inrush water image processing system based on AI recognition, which comprises a fissure water flow tracking module, a voiceprint disturbance extraction module, an acceleration mapping module, a form analysis module and a risk output module. According to the method, a fine voiceprint perturbation sequence is formed by extracting crack water flow boundary pixel coordinates from continuous frame images, accurately capturing water flow micro-displacement and generating a deviation track in combination with an inter-frame difference between a gray gradient amplitude and a direction angle, and synchronously extracting a liquid level perturbation wave crest position and a voiceprint primary and secondary frequency amplitude difference; the acceleration is calculated according to the pixel displacement speed and mapped to a time axis, so that the boundary movement trend and morphological change are synchronously analyzed in the time dimension, risk frame locking is completed by multi-source signals in the same time period under double-threshold judgment, and three-dimensional presentation of the water flow dynamic state is achieved. The accuracy and reliability of sudden water gushing risk judgment in a complex mine environment are remarkably improved, and the probability of missing detection and misjudgment is reduced.

Techniques for performing a machine learning model based spatial-temporal adaptive shift for end-to-end text-video retrieval are described. According to some examples, a computer-implemented method includes receiving a video comprising a plurality of frames at a content delivery service; generating, by the content delivery service, a set of embeddings for each of a plurality of sections of each frame of the plurality of frames; determining, by a candidate selector machine learning model of the content delivery service, a proper subset of the plurality of sections of each frame of the plurality of frames for a time shift based on the set of embeddings; time shifting, by the content delivery service, the proper subset of the plurality of sections of each frame of the plurality of frames to generate time shifted frames; generating, by the content delivery service, an updated set of embeddings based on the time shifted frames; receiving a search request comprising input text from a user device; determining the video is a match for the search request based on the input text and the updated set of embeddings for the time shifted frames; and sending the video to the user device based on the match.

Disclosed herein are methods, systems, and computer-readable media for prompting a machine learning model to generate answer data based on a recording. Some embodiments involve preprocessing a prompt corresponding to a query for a first system by receiving the prompt and a timestamp corresponding to a time position of the query in a recording, acquiring a text transcript based on the recording, and selecting, based on the timestamp and the text transcript, a first data domain from the text transcript. Some embodiments involve transmitting at least one of the prompt, the text transcript, and the first data domain to a second system, the second system including a machine learning model. Some embodiments involve generating answer data corresponding to the prompt by querying the machine learning model with the prompt, receiving answer data from the machine learning model, and transmitting the answer data to the first system.

The invention discloses a universal scene retrieval analysis method and system based on multi-modal feature fusion, the method comprises a video analysis step and an application service step, and the application service step comprises the steps of receiving a user input request, describing a multi-dimensional standardized video tag based on a video summary, and obtaining a multi-dimensional standardized video tag; the steps of cross-modal video retrieval, dynamic knowledge enhancement question answering and interactive enhancement analysis can be executed, efficient video preprocessing is achieved by constructing an offline feature library, the retrieval precision is improved by adopting a cross-modal feature fusion technology, the analysis authority is enhanced in combination with a dynamic knowledge base, and the interactive enhancement analysis is supported to achieve abnormal early warning. The method has the advantages that the offline video processing efficiency is improved, cross-modal feature fusion retrieval is realized, and the authority of an analysis result is enhanced.

A method, computer system, and computer program product are provided for real-time video searching based on augmented knowledge graphs that are generated using machine learning models. Multimedia data is obtained comprising an image portion and an audio portion, and a user query with respect to the multimedia data is obtained. A knowledge graph of the multimedia data is generated using one or more machine learning models based on the image portion and the audio portion, wherein the knowledge graph includes a plurality of entities and relationships between entities. An augmented knowledge graph is generated, wherein the augmented knowledge graph augments the knowledge graph with additional entities and additional relationships between the additional entities using additional data that is obtained from a source external to the multimedia data. A response to the user query is provided based on the augmented knowledge graph.

The invention discloses a multi-modal video question-answering method and device based on large language model optimization, and the method employs a CoVGT basic framework, optimizes video feature extraction and text input, and adds a video question-answering model output optimizer. The method comprises the steps of video feature extraction, text feature extraction, feature fusion, preliminary answer prediction, related sample generation, Prompt text generation and final answer generation. According to the method, the answer optimizer is designed by using the learning characteristics in GPT-3 analogy, related sample generation, Prompt text generation and final answer generation, and compared with a basic model, the model accuracy is greatly improved.

The invention discloses a monitoring video event analysis method, system and device for monitoring equipment, and relates to the technical field of video abstraction. Obtaining a monitoring video recorded by monitoring equipment, and screening out a moving video clip from the monitoring video; extracting a video frame feature and an audio feature from the video clip, determining a plurality of event clips in the video clip according to the video frame feature and the audio feature, and determining an abstract of each event clip; storing the abstract of each event segment and the corresponding timestamp in a database; and determining event segments recorded by the monitoring equipment through the abstracts in the database and the corresponding timestamps. According to the method, the monitoring video of the monitoring equipment can be effectively analyzed, so that the time for finding the related video clip of the target event is shortened.

The invention relates to the technical field of video content understanding, in particular to a video content understanding method of a multi-mode advertisement inventory intelligent matching system, and the method comprises the steps: taking a video frame of an original advertisement video as a benchmark, and in a shot where a timestamp of each video frame is located, obtaining a video frame of the original advertisement video; analyzing the similarity of adjacent video frames and the dispersion degree of feature matrix elements corresponding to the voice segments, and determining a first search window; meanwhile, analyzing the occurrence frequency and duration of the text fragment in unit time, and confirming a second search window; based on the search window, analyzing the feature similarity degree of the voice segment and the text segment with the video frame in the corresponding window, and obtaining an optimal alignment path; and aligning the timestamps of the voice segments and the text segments with the timestamps of the video frames, and realizing multi-level semantic analysis through a multi-modal fusion technology and a deep learning model. The invention aims to synchronize the sequential relationship among the audio, the text and the video frame and prompt the accuracy of video content understanding.

In other embodiments, a method to perform image-based searching may include receiving a group of images from a content owner; creating an index using the group of images; receiving a further image from a mobile computing device of a user; determining a correlation between the further image and the index; and / or directing the mobile computing device of the user to content of the content owner, responsive to the determining of the correlation.

The invention discloses a video clip label generation method and system for scene change detection, and the method comprises the steps: carrying out the segmentation of a video clip, and carrying out the preliminary recognition of a scene change point in a video through scene change detection; dividing the video into segments with independent scenes, and extracting representative key frames from each scene segment; using a highlight detection algorithm to identify a highlight area in the video frame; and searching a minimum point of a function of the time and the scene change score, so that the video is divided into segments with non-repeated time, and static information and dynamic information in the segmented video segments are extracted and marked, so that the generation of video segment labels is realized. The invention further discloses a video clip label generation system for scene change detection. According to the method, the video can be accurately segmented into fragments which are not repeated in time, so that the management and retrieval efficiency of the video content is greatly improved, and a user can more quickly find related video fragments.

The invention relates to the field of movies, and discloses a movie recommendation method and system based on big data, and the method comprises the steps: collecting the multi-dimensional behavior data of a user to form an initial behavior data set; based on the initial behavior data set, calculating preference scores of the user for different categories of short play contents to obtain a user interest distribution matrix; dividing user groups and labeling user category labels; forming an initial short episode content feature set; constructing a content semantic feature matrix; if the content semantic feature matrix and the user category label have the matching requirement, judging the integrating degree of the short episode content and the user preference, and outputting a preliminarily recommended short episode content list; and on the basis of the preliminarily recommended episode content list, sorting and optimizing the recommended episode content by using click rate and complete playing rate indexes in the historical distribution data to obtain a final recommended episode content list. The method has the following effect that the accuracy and the individuation degree of short play recommendation are improved.

The invention belongs to the technical field of medical treatment and public health, and particularly relates to a hypertension group knowledge recommendation method and system based on a large language model multi-agent. Hypertension knowledge question-answer pair information, medical short video information and user personal health portraits are finely extracted by constructing a hypertension knowledge question-answer pair library; according to the method, video content quality evaluation and matching degree evaluation are performed, and user platform interaction data adjustment is combined, so that personalized and accurate knowledge recommendation services can be provided for hypertension users, the requirements of hypertension patients for health knowledge are met, the accuracy and effectiveness of knowledge recommendation are improved, and self-health management of the patients is facilitated.

The invention relates to a weak supervision online video moment positioning method and system based on memory perception, and belongs to the technical field of artificial intelligence, and the method comprises the steps: carrying out the multi-modal feature fusion of a given video and a text query thereof, and obtaining the unified frame level representation at each stage; inputting the fused features into an offline module and an online module in an integral and frame-by-frame manner by using an offline guide online model architecture; in the off-line module, generating a Gaussian mask to reconstruct query of a covered part of words, and obtaining a proposal of an action starting moment; in the on-line module, the long-term historical memory in the window is used for enhancing the score, the attention weight of the score in the window is dynamically generated, and the score of the current frame is calculated in a weighted mode; taking the proposal obtained by the offline module as a pseudo tag, and providing supervision information for the score sequence of the online module; and high-performance weak supervision on-line moment positioning can be completed only by independently deducing the on-line module. The expansion capability and the application value of the model are remarkably improved.

The invention discloses a video content creation method and device based on new media data analysis, and a server side method comprises the steps: receiving a video content creation request sent by a client side, wherein the video content creation request carries a content keyword; through a preset BI system, retrieving new media accounts related to the content keywords and comment area contents corresponding to videos published by the new media accounts as new media data, and retrieving e-commerce platform data corresponding to products related to the content keywords; generating user preference information and market trend information corresponding to the content keywords according to the new media data and the e-commerce platform data; and generating a target video based on the user preference information and the market trend information, and sending the target video to the client for playing. Therefore, by adopting the embodiment of the invention, the limitation of the traditional manual creation and single automatic editing technology is overcome, so that the automation, individuation and high-quality output of video content creation are realized.

The invention relates to the technical field of computer vision, in particular to a video time retrieval and highlight detection method based on double semantic alignment, which comprises the following steps: respectively extracting visual features and text features by using a visual encoder and a text encoder; performing cross attention operation on the visual features and the text features to obtain joint features; the joint features are processed through an encoder and a decoder, the output of the encoder is used for highlight detection, and the output of the decoder is used for time retrieval; fragment-level semantic alignment is realized based on significance contrast learning; time-level semantic alignment is realized based on the time center distance; carrying out binary matching by adopting a Hungary algorithm, and establishing an optimal corresponding relation between a prediction moment and a real moment; and jointly optimizing the highlight detection loss and the time retrieval loss to update the parameters of the encoder and the decoder. According to the method, double semantic alignment is carried out through a significance contrast learning method and a center distance regression method, so that more accurate highlight detection is realized, and more accurate time retrieval is output.

The invention provides a video generation method and device, and relates to the technical field of computers, in particular to the technical fields of large models, deep learning, image processing, voice recognition and the like. According to the specific implementation scheme, semantic partitioning and position marking are conducted on a to-be-processed material, and a semantic text block set comprising at least one semantic text block is obtained; based on the key event information, classifying semantic text blocks in the semantic text block set to obtain a key text block set and a non-key text block set; generating a key image sequence based on the key text block set, the to-be-processed material and a video material library; generating a non-key image sequence based on the non-key text block set and a video material library; the target video is generated based on the key image sequence and the non-key image sequence, and the video generation effect is improved.

The invention relates to the technical field of data processing, and discloses a film and television supervision analysis method and system based on data mining and a storage medium. The method comprises the following steps: collecting film and television data in a panoramic manner, and carrying out cross-media transcoding processing to obtain a multi-dimensional data set; performing video semantic extraction, audio emotion recognition and text tendency analysis on the data set to generate a time sequence mark list; scoring the key elements according to social influence, propagation depth and audience acceptability to form a dynamic score table; analyzing the propagation path based on the score table, and constructing an influence map; sorting supervision points according to the atlas, and making an intelligent supervision scheme; the execution deviation is analyzed through effect feedback, and the supervision parameters are optimized. According to the method, multi-dimensional content feature extraction, social influence evaluation and propagation path analysis of the film and television works are realized in a mass multimedia content environment, a precise and differentiated intelligent supervision scheme is formed, and the method has self-optimization and adjustment capabilities at the same time.

The invention discloses a film and television content label processing method and terminal based on a large model, and belongs to the technical field of artificial intelligence and film and television content processing, and the method comprises the steps: pre-integrating to form a multi-source film and television data set; performing fine adjustment on a plurality of specified artificial intelligence large models in advance; pre-deploying a task automatic triggering module and constructing a cue word engineering library; acquiring input film and television content, automatically selecting unrechecked content for the film and television content to trigger a task, calling corresponding cue words through the cue word engineering library, and respectively outputting error correction results of film and television content labels through each specified artificial intelligence large model after fine tuning; judging and correcting the error correction result of the output film and television content label by adopting a preset artificial intelligence model, and mapping the correction result to a label library; and after AI auditing is carried out, the identification recheck authentication state of the film and television content is updated. The accuracy, efficiency and diversity of film and television label processing are improved, and the requirements of scenes such as film and television content management, recommendation and analysis are met.

The invention discloses an information retrieval method, a related device, equipment and a storage medium, and is applied to the field of artificial intelligence. The method comprises the steps of obtaining to-be-queried information and target instruction information; based on the to-be-queried information and the target instruction information, obtaining a first token sequence through a feature extraction layer included in an information retrieval model; performing bidirectional attention coding on the first token sequence through a backbone network included in the information retrieval model to obtain a second token sequence; pooling the second token sequence to obtain a target embedded vector; and according to the target embedding vector, obtaining a preset number of candidate information with the highest similarity as a retrieval result. According to the method, the complexity and the calculation cost of model deployment are reduced, and the target embedding vector generated by the model is more discriminative, so that the accuracy of a retrieval result is improved.

The invention relates to the field of information retrieval, and discloses a cross-modal retrieval method for a text video. In the training stage, candidate pairs are input in batches, and the similarity of the candidate pairs is calculated respectively; then, taking the similarity of each candidate pair as a classification probability, and based on Dirichlet distribution, carrying out modeling on the uncertainty of the classification probability; and taking the probability of the classification probability of the candidate pair corresponding to the sample pair as 100% and the probability of other candidate pairs as 0%, constructing a label, and completing the training of the model. In the retrieval stage, the candidate pairs are input in batches, uncertainty modeling is aligned while the similarity is obtained, the similarity is corrected through uncertainty, and matching is conducted through the corrected similarity. Therefore, the method provided by the invention can quantify the uncertainty in the text video pair so as to carry out more reliable similarity evaluation, effectively improve the retrieval performance, and solve the reliability problem caused by only dependence on similarity in the prior art.

The invention discloses a long video positioning method, system and device and a storage medium, which are one-to-one corresponding schemes, in the scheme, a step-by-step positioning mode is adopted, firstly, a highlight video frame is identified through video and text guidance, so that a video clip aligned with a query text is obtained, and the problem that in an existing method, a clip is directly extracted from a long video, so that the video clip cannot be directly extracted is avoided. The problem of lack of overall perception of a long video on the text is solved; further positioning on the basis of obtaining the video clip aligned with the query text to obtain an accurate moment; in the step-by-step positioning mode, each step of positioning further strengthens the relation between the video and the query text, so that a more accurate positioning result can be obtained.

The invention discloses a media information processing method and device, a storage medium and electronic equipment. The method comprises the steps that to-be-retrieved source media information and prompt template information are obtained, the source media information and the prompt template information are input into a target retrieval model, target media information is determined, and the target retrieval model represents a model obtained through joint training according to at least two retrieval task types; a loss function used in the training process of the target retrieval model is a joint loss function, the joint loss function is determined by at least two sub-loss functions in one-to-one correspondence with the at least two retrieval task types, and the target media information represents media information obtained by retrieving the source media information according to the prompt template information. According to the method and the device, the technical problem of relatively low retrieval efficiency of the media information caused by single multi-modal retrieval task is solved.

The embodiment of the invention provides an information processing method and device, and the method comprises the steps: collecting hotspot information related to a target service, and obtaining a hotspot video related to the hotspot information and a reference video copywriting of the hotspot video through a target interface; inputting the hotspot information, the hotspot video and the reference video copywriting into a large language model for processing to obtain a target video copywriting matched with a target platform content publishing style; generating a target audio and a target image corresponding to the target video copywriting, and constructing a target video according to the target audio and the target image; and publishing the target video on the target platform according to a preset publishing strategy.

The invention relates to a label generation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a video identifier and description data of a target video; obtaining classification reference data of the target video according to the video identifier; generating a candidate label set based on the description data and the classification reference data; and inputting the description data and the candidate label set into a preset classification model to obtain a classification label of the target video. According to the embodiment of the invention, the classification label of the target video can be automatically generated, furthermore, the classification label closer to the description data of the target video is selected from the expanded richer candidate label set, the label generation accuracy is improved, and the classification label generation efficiency is improved due to the fact that the classification label is generated based on the same standard. Errors caused by individual differences in manual labeling are avoided, and the consistency of the labels is improved.

Methods, systems, and apparatuses are described to implement voice search in media content for requesting media content of a video clip of a scene contained in the media content streamed to the client device; for capturing the voice request for the media content of the video clip to display at the client device wherein the streamed media content is a selected video streamed from a video source; for applying a NLP solution to convert the voice request to text for matching to a set of one or more words contained in at least close caption text of the selected video; for associating matched words to close caption text with a start index and an end index of the video clip contained in the selected video; and for streaming the video clip to the client device based on the start index and the end index associated with matched closed caption text.

In other embodiments, a method to perform image-based searching may include receiving a group of images from a content owner; creating an index using the group of images; receiving a further image from a mobile computing device of a user; determining a correlation between the further image and the index; and / or directing the mobile computing device of the user to content of the content owner, responsive to the determining of the correlation.

A method for performing a video content search is provided. The method includes receiving a search query associated with one or more videos and obtaining subtitle files associated with the one or more videos. The subtitle files each include a corresponding transcript having a plurality of transcript passages and video timestamps for the plurality of transcript passages. The method further includes determining query responses based on contents of video segments of the one or more videos. The video segments correspond to passage time ranges relating to the search query. The passage time ranges each include video timestamps associated with a transcript passage in the subtitle files that correlates with the search query. The method further includes causing a query result to be presented to a user on a graphical user interface, the query result being based on the determined query responses.