437results about "Multimedia data querying" patented technology

The invention discloses an intelligent file classification and retrieval method and system, and the method comprises the steps: carrying out the unified format conversion and feature extraction of text, image and audio data according to the original data format of a file through a multi-modal data fusion algorithm, and obtaining a standardized file feature data set; based on a deep learning adaptive classification model, performing classification processing on the standardized archive feature data set to generate multi-level archive classification labels; analyzing the semantic relationship between the archive classification labels through an enhanced semantic network construction technology, generating a semantic association map, and optimizing the semantic association map to obtain an optimized semantic association map; and according to a retrieval request input by a user, performing semantic matching and correlation sorting by utilizing the optimized semantic correlation graph, and outputting archive information most related to the retrieval request. By utilizing the embodiment of the invention, the relevance and accuracy of retrieval can be improved, and modernization and intellectualization of archive management are promoted.

The invention discloses a multi-modal enhanced retrieval method and system, and the method comprises the steps: obtaining heterogeneous data, and executing data processing to obtain a meta-information set; encoding the meta-information set into a high-dimensional meta-information vector, and constructing a vector index; receiving a query statement of a user, calculating the similarity between the query statement and the meta-information vector, and obtaining a meta-information query result; and calling a large language model based on the obtained meta-information to generate a summary result, and merging and outputting the summary result and heterogeneous data corresponding to the meta-information. According to the method, multi-modal data is processed into unified semantic vector representation, efficient vectorization indexing and semantic generation and expansion capacity of a large-scale language model are combined, high-value information can be rapidly extracted from multi-source heterogeneous data, and real-time, cross-modal and high-precision retrieval requirements are supported. A reordering and traceability mechanism introduced in the scheme further enhances the correlation and reliability of the information, and provides comprehensive and accurate knowledge support for the generation of query results.

The invention relates to the technical field of information retrieval and generation, in particular to a retrieval enhancement method based on multi-modal data fusion and modal perception. According to the method, firstly, a dual-channel architecture is adopted to perform feature extraction and coding on a text and an image respectively, and mutually independent embedded representation spaces are constructed, so that high-quality collaboration and matching of cross-modal representation are realized; and a pseudo-pairing generation mechanism is introduced to effectively mine and reconstruct the existing non-paired data in the knowledge base. And designing a query modal perception and dynamic weighting mechanism for accurately controlling the fusion proportion of the image-text bimodal information in the retrieval stage so as to match the modal demand difference of different query contents. And further executing aggregation retrieval and reordering of the cross-modal information by using dynamic weighted fusion retrieval to generate a candidate set of multi-modal responses. According to the method, accurate matching and dynamic weight adjustment of the image-text content are realized, and the accuracy and expression integrity of the generated content are improved.

The invention is applicable to the technical field of artificial intelligence, and provides a cross-modal retrieval method and device, electronic equipment and a storage medium, and the method comprises the following steps: encoding an image modal through an image encoder, encoding a text modal through a text encoder, and generating a vector representation of a high-dimensional space; inputting the vector of the image mode and the vector of the text mode into the same shared self-attention layer for alignment, and extracting features through respective feedforward network layers; performing cross-modal cross fusion processing on the vector of the image modal and the vector of the text modal through a cross attention mechanism to obtain a cross-modal representation vector of the image modal and a cross-modal representation vector of the text modal; and based on the cross-modal representation vector of the image modality and the cross-modal representation vector of the text modality, outputting the similarity of the image modality and the text modality. According to the method, the cross-modal retrieval precision can be improved.

The embodiment of the invention provides an information retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that obtained intention description information is input into a preset intention recognition model, and an intention map is obtained; performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation; performing modal feature extraction processing on the intention graph to obtain graph structure feature representation; inputting the modal feature representation and the graph structure feature representation into a pre-trained multi-modal fusion model to obtain a multi-modal semantic representation; generating a query statement through a preset large language model according to the multi-mode semantic representation and mode information of a database to be queried; and retrieving the data in the database according to the query statement, and outputting a retrieval result. According to the embodiment of the invention, the accuracy of an information retrieval result and the use experience of a user can be improved.

The invention relates to a cross-modal document information extraction method based on space-semantic alignment, and belongs to the field of artificial intelligence, computer vision and natural language processing. According to the method, the spatial feature and semantic information bidirectional alignment model is designed, by constructing the spatial feature and semantic feature bidirectional alignment model, the document layout information can dynamically adjust attention distribution of text semantic features, meanwhile, semantic information reversely optimizes the spatial features, collaborative modeling of spatial layout and semantic information is achieved, and the document layout efficiency is improved. Therefore, the accuracy and robustness of complex document information extraction are improved. According to the method, a hierarchical cross-modal information extraction model is designed, through the hierarchical cross-modal information extraction model, the overall structure of a document is recognized on the global level, local key content is focused on the regional level, fine modeling is conducted on fine-grained texts and visual elements on the entity level, and accurate recognition of a cross-modal entity and the semantic relation of the cross-modal entity is achieved; and the generalization ability and applicability of information extraction are enhanced.

A method, apparatus, non-transitory computer readable medium, and system for media processing includes obtaining a variation parameter and a number of variations, identifying a first variation input and a second variation input for the variation parameter, and obtaining a first media content item and a second media content item based on the first variation input and the second variation input, respectively. The first media content item and the second media content item vary from each other with respect to the variation parameter. The method, apparatus, non-transitory computer readable medium, and system for media processing further includes displaying the first media content item and the second media content item in a grid comprising a grid size based on the number of variations.

The invention relates to the technical field of computers, and discloses a nuclear power safety report examination method and system, electronic equipment and a storage medium, and the method comprises the steps: receiving an examination request, carrying out the intention recognition and entity extraction based on a nuclear power field large language model, and disassembling an examination task into a structured subtask sequence based on an intention type and an entity attribute; performing multi-modal retrieval based on the structured sub-task sequence, and outputting a multi-source retrieval result; calling time-space association information of the evolutionary knowledge graph, and outputting knowledge support data in combination with the time sequence model; fusing a multi-source retrieval result and the knowledge support data, performing risk value calculation and uncertainty quantification through a deep learning model, and outputting a risk assessment result; and based on the risk assessment result, carrying out federal learning model parameter adjustment, carrying out uplink evidence storage on the review conclusion, the risk assessment result and the evidence chain, and outputting the review result. The method has the advantages of being efficient, accurate, dynamic, safe and closed-loop during examination.

The invention relates to the technical field of knowledge graph and content management, in particular to an intelligent education content management method and system based on a knowledge graph, and the method comprises the steps: obtaining multi-modal education content data which comprises text data and audio data; based on the multi-modal education content data, executing multi-modal tensor decomposition to obtain unified feature representation; according to the unified feature representation, topological continuous coherence feature extraction is executed to obtain topological features; based on the topological features, quantum entanglement feature enhancement is executed, and enhanced features are obtained; executing non-exchangeable geometric feature fusion according to the enhanced features to obtain fusion features; based on the fusion features, chaotic dynamics knowledge extraction is executed, and extracted knowledge is obtained; according to the method, the extracted knowledge is constructed into the knowledge graph, different forms of education contents such as texts and audios are integrated through a multi-modal tensor decomposition technology, complementary information among different modals can be captured, and comprehensiveness and accuracy of knowledge representation are improved.

Methods, systems, and media for processing queries relating to presented media content are provided. In some implementations, a method comprises: receiving a request to associate with a media playback device that is presenting media content to a user of the mobile device, wherein a mobile application executing on the mobile device and a media application executing on the media playback device exchange media playback information; activating a microphone associated with the mobile device to receive ambient sounds in response to associating with the media playback device; converting the received ambient sounds to one or more text inputs; determining whether the text inputs include a trigger term that corresponds to a request to initiate a query relating to the presented media content and the query; in response to determining that the trigger term has been included in the text inputs, determining the media playback information from the media application that includes timing information corresponding to when during the presentation of the media content the query was received and media content identification information; causing a search to be performed that includes the query, the timing information, and the media content identification information; obtaining a search result that is responsive to the query; and presenting at least a portion of the search result to the query on a mobile display associated with the mobile device.

A combined image retrieval method based on a picture mapper and a text self-covering mapper comprises the following steps: firstly, carrying out operations such as zooming, noise adding and the like on an input picture, then inputting the picture into a frozen pre-training image encoder to obtain an image feature # imgabs0 #, and converting the image feature # imgabs1 # into a pseudo word mark s1 by using the image mapper; and converting the image feature # imgabs2 into a pseudo word mark s2 by using a text self-covering mapper, respectively replacing a mark symbol in a pre-provided prompt text with the pseudo word mark s2, and splicing the input modified text to the prompt text to obtain two new texts x1 and x2. And inputting the text x1 and the text x2 into the frozen pre-training text encoder to obtain two features, performing mean value fusion on the two features, and finally comparing the obtained mean value feature with a target image. According to the method, the defect that an expensive triple is used for training a combined image retrieval model is overcome, and the generalization ability of the model to a combined image retrieval downstream task is also improved.

Provided are systems, methods, and apparatuses for systems and methods of memory efficient multi-vector information retrieval based on embeddings from a storage pipelined network. In one or more examples, the systems, devices, and methods include performing a first portion of a nearest neighbor search on a first subset of a set of media sources; performing a fetch process on the first subset that include identifying a first set of highest matching media sources from the first subset, transferring multi-vector representations of the first set of highest matching media sources from a storage drive to a memory, and performing a second ranking of the first set highest matching media sources. The systems, devices, and methods include performing, in parallel with the fetch process, a second portion of the nearest neighbor search on a second subset of the set of media sources.

Embodiments are disclosed for one-shot document snippet search. A method of one-shot document snippet search may include obtaining a query snippet and a target document. A multi-modal snippet detection model combines first multi-modal features from the query snippet and second multi-modal features from the target document to create a feature volume. The multi-modal snippet detection model identifies one or more matching snippets from the target document based on the feature volume.

The invention relates to the technical field of multi-modal information retrieval, and particularly discloses a news scene multi-level image-text retrieval method based on mask guidance information fusion. According to the method, a mask-based cross attention module is introduced, and information of two modalities of an image and a text is guided to be better fused by optimizing mask language model loss; by selecting the text and image features with the highest correlation degree, performing finer-grained and efficient alignment between the two modals; according to the method, data characteristics of news data, namely weak alignment relationships between pictures in news articles and all text segments, are fully considered, and parameters are updated by fitting real distribution of data and model output distribution with reference to the idea of comparative learning, so that hierarchical association relationships between news images and texts are better learned.

The invention discloses a model training method, electronic equipment and a computer readable storage medium, and relates to the technical field of data processing and large models. The method comprises the steps of obtaining a training data set, wherein the training data set comprises image annotation data of various types of concerned objects and description text annotation data associated with the image annotation data; training the initial multi-modal retrieval model by adopting the training data set to generate a learnable prompt; carrying out image coding training on the initial multi-modal retrieval model by adopting the training data set and the learnable prompt to generate an intermediate multi-modal retrieval model; the training data set is adopted to conduct text and image alignment training on the middle multi-modal retrieval model, a target multi-modal retrieval model is generated, and the target multi-modal retrieval model is used for conducting multi-modal retrieval on the target retrieval request to obtain a target retrieval result. According to the method and the device, the technical problems that a multi-modal retrieval model in related technologies cannot adapt to variability and large-scale data requirements, and the model performance is relatively poor are solved.

The invention discloses a multi-mode convergence media content auxiliary creation method based on an AI technology, and the method comprises the steps: extracting key features according to a creation demand text inputted by a user, carrying out the retrieval in a knowledge base through employing a knowledge graph retrieval mode according to the key features, obtaining a creation material, and generating a first draft; carrying out cross-modal feature mapping on the first draft by adopting a multi-modal alignment model, aligning text-image-video embedded vectors through contrastive learning, and dynamically adjusting the correlation of multi-modal contents by utilizing an attention mechanism to obtain a multi-modal content packet; duplicate checking is carried out on the multi-modal content packet in multiple modes, and the multi-modal content packet after duplicate checking is sent to a user for manual editing. Through multi-mode processing methods such as AI auxiliary writing, AI illustration and video generation and whole-network duplicate checking and propagation value evaluation, the content production efficiency and quality are improved, the propagation effect is optimized, and the original content is protected.

The invention provides a long-range visual question and answer and multi-modal reasoning task agent construction method. For input images and natural language questions, generating answers through orderly calling of an external tool set; the construction process of the intelligent agent is divided into three stages: a global planning stage: analyzing user requirements through a global planning navigator, and screening tool subsets; in the tool autonomous execution stage, visual details are supplemented, background knowledge is retrieved and accurate operation is carried out in the multi-step process by surrounding three types of tools through a tool autonomous executor, and a returned tool result is used for assisting the next step of decision making; in the response synthesis stage, the reasoning process is combed through a model response synthesizer, and user-friendly and clear-format responses are sorted; and finally, the model response synthesis module extracts and obtains answers from the reasoning track and outputs the answers. The problems of global planning insufficiency and visual context forgetting generally existing in complex task reasoning of an existing multi-mode large language model are solved.

The invention discloses a pedestrian re-identification method based on multi-modal auxiliary information guide learning, which belongs to the field of computer vision pedestrian re-identification, and comprises the following steps: constructing a model MMFI to carry out pedestrian re-identification; the model MMFI comprises a multi-modal information extraction module, an auxiliary information guide learning module and a knowledge distillation network; the method specifically comprises the following steps: step 1, acquiring an RGB image of a pedestrian as the input of a multi-modal information extraction module, and generating three modal features carrying different information by the input image through three different networks; 2, constructing an auxiliary information guide learning module and an implicit guide loss function; 3, constructing a knowledge distillation network and an explicit guide loss function; 4, constructing an overall loss function for overall back propagation of the model MMFI; and 5, constructing a reasoning module, and reducing the complexity of the model MMFI. The method is high in pedestrian re-identification accuracy and high in calculation efficiency.

The invention relates to a static and dynamic feature fusion video and text cross-modal retrieval method. The method comprises the following steps: constructing a noun, preposition and verb set for a text; converting nouns and prepositions into text static feature representation Tsf through a text encoder, and converting verbs into text dynamic feature representation Tdf; encoding a video through a video encoder to obtain video static feature representation Vsf; calculating and normalizing an inter-frame difference sequence of adjacent frames of the video, and encoding through a video encoder to obtain a video dynamic feature representation Vdf; calculating the static similarity between the Tsf and the Vsf, calculating the dynamic similarity between the Tdf and the Vdf, and carrying out weighted average on the Tsf and the Vsf to obtain a final similarity score; for a plurality of given texts and videos, k text-video pairs with the highest similarity scores are selected as retrieval results. According to the method and the system, the accuracy and the efficiency of video and text cross-modal retrieval can be improved, and videos related to text description can be effectively retrieved from a large amount of video data.

The present disclosure provides a character recognition model training method and apparatus, a character recognition method and apparatus, a device and a medium, relating to the technical field of artificial intelligence, and specifically to the technical fields of deep learning, image processing and computer vision, which can be applied to scenarios such as character detection and recognition technology. The specific implementing solution is: partitioning an untagged training sample into at least two sub-sample images; dividing the at least two sub-sample images into a first training set and a second training set; where the first training set includes a first sub-sample image with a visible attribute, and the second training set includes a second sub-sample image with an invisible attribute; performing self-supervised training on a to-be-trained encoder by taking the second training set as a tag of the first training set, to obtain a target encoder.

The present disclosure relates to the fields of large language models and machine learning. Disclosed are a multimedia information processing method, a system, and an electronic device. The method is applied to a generative question-answering system, and comprises: receiving inquiry information; rewriting the inquiry information into a candidate query used for executing customized querying for multimedia information, wherein the multimedia information is content uploaded to the generative question-answering system in advance; retrieving from a question-answering database a data chunk associated with the candidate query, wherein at least one set of data chunks is stored in the question-answering database, and a data source of the data chunks is the multimedia information; and using a large language model to process the data chunk to generate at least one piece of reply content matching the inquiry information.

The present disclosure provides a network model training method, a data processing method, and an apparatus. The network model training method comprises: acquiring target sample data, wherein the target sample data comprises text sample data and image sample data; inputting the target sample data into a network model to be trained to obtain a sample recognition result; and adjusting a parameter of a text encoder on the basis of a text recognition result and first supervision data corresponding to the text recognition result, adjusting a parameter of an image encoder on the basis of an image recognition result and second supervision data corresponding to the image recognition result, and a hybrid image-text recognition result and third supervision data corresponding to the hybrid image-text recognition result, and adjusting a parameter of a hybrid encoder on the basis of the hybrid image-text recognition result and the third supervision data corresponding to the hybrid image-text recognition result to obtain the trained network model formed by the text encoder, the image encoder, and the hybrid encoder.

The invention relates to a multi-dimensional knowledge base construction and query method based on a multi-modal large model, and belongs to the technical field of AI large model application. The method comprises the following steps of: analyzing a text file containing text, table and picture contents; selecting a pre-trained large language model and a multi-modal large model, generating text abstracts and picture abstracts by utilizing the models, and enabling storage paths of the picture abstracts to correspond to storage paths of pictures one by one; selecting a text embedding model to construct a vector database and a retriever; and constructing a retrieval chain based on the multi-modal large model. According to the method, construction from a single text knowledge base to a multi-dimensional knowledge base is achieved, a multi-dimensional information retrieval method is provided, the problem of construction of the multi-dimensional knowledge base based on the vectorization technology is solved to a certain extent, and the large model output quality based on the retrieval enhancement generation technology is remarkably improved.

The invention relates to a search processing method and device and a computer readable storage medium, and relates to the technical field of artificial intelligence. The search processing method comprises the following steps: receiving a question input by a user; according to the question, an answer of the question and a plurality of answer videos related to the question are displayed in a search result page, and the answer is a statement which is generated by carrying out machine understanding on the answer videos and is described in a natural language; the description information comprises description information of each answer video generated by performing machine understanding on each answer video in the plurality of answer videos.

The invention belongs to the technical field of advertisement putting management, and discloses an intelligent polling method for advertisement putting, which comprises the following steps of: automatically pulling advertisement putting data from a plurality of media platforms through a timed task; analyzing an advertisement plan name, and extracting key information such as a vehicle type, a delivery position, a material number, a document number and a page number according to a preset rule; vehicle type matching inspection is carried out on the advertisement materials and the copywriting, vehicle type information in the materials and the copywriting is compared with vehicle type information in the advertisement plan name, and early warning is triggered when the materials and the copywriting are inconsistent with the vehicle type information in the advertisement plan name; according to the invention, through integration of multiple platforms, originally dispersed media are integrated in one platform for unified management, employees are prevented from repeatedly skipping to the platform for checking and operation, the rule is flexible, the inspection rule can be customized and configured by the employees, different inspection requirements of different users are met, and a large number of advertisement plans are subjected to in-project and timed inspection. Manual inspection is replaced by a machine, so that the efficiency of staff is greatly improved.

The invention discloses a convergence media manuscript propagation effect prediction method based on an AI technology, and the method comprises the steps: reading a to-be-predicted manuscript, carrying out the classification and processing of the to-be-predicted manuscript according to a text manuscript, a video manuscript and an audio manuscript, and carrying out the normalization processing of the feature data obtained through the processing, and obtaining a structural feature vector; according to the structured feature vector, retrieving the same type of manuscripts based on theme similarity in a knowledge base, and performing analysis to obtain a spatial-temporal feature tensor; constructing a four-dimensional propagation element incidence matrix, establishing a feature cross validation mechanism, and generating a propagation potential comprehensive evaluation vector; and by constructing a dynamic prediction model, analyzing the structured feature vector, the spatial-temporal feature tensor and the propagation potential comprehensive evaluation vector to obtain a prediction result, and generating a structured propagation value analysis report based on the prediction result. According to the method, a multi-modal propagation value prediction method combining an AI kernel and a knowledge base is used, a multi-dimensional propagation effect evaluation system is constructed, and propagation value prediction analysis is effectively provided.

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal data retrieval, generation and synthesis method and system based on artificial intelligence driving, and the method comprises the steps of multi-modal feature extraction, cross-modal alignment, feature fusion, multi-modal retrieval and sorting and multi-modal generation. Different deep learning models are adopted to carry out feature extraction on multi-modal data and convert the multi-modal data into vectors, through cross-modal alignment, different modal feature vectors are mapped to the same vector space, through feature fusion, correlation weights are calculated through an attention mechanism, weighted summation is carried out on fusion features, and through multi-modal retrieval and sorting, multi-modal data are obtained. According to the method, the similarity between a query vector and candidate data is calculated and sorted, and finally, through multi-modal generation, retrieval knowledge is used as external knowledge to be fused into a generation model, and a multi-modal synthesis answer is generated, so that information in multi-modal data can be effectively integrated and utilized.

The invention discloses a multi-modal intelligent question answering system and method based on a large model. The system comprises a signaling processing module, a document preprocessing module, a voice preprocessing module, a retrieval recall module, an RAG processing chain module and a knowledge base module. The method comprises the following steps: (1) a knowledge base offline construction step; (2) a knowledge base online construction step; (3) a real-time question and answer step; according to the system and the method provided by the invention, personalized intelligent question and answer service can be provided for audiences by combining materials which are uploaded by a speaker in advance and contain rich contents such as characters, pictures and charts and real-time presentation contents and utilizing multi-mode presentation information such as characters, pictures and voice, so that instant questions of the audiences can be answered, and the audiences can be more interested. Extensible knowledge supplementation can be provided according to the presentation content and materials.

The invention discloses a cross-modal conference information association retrieval method and system and a medium, and relates to the technical field of artificial intelligence, and the method comprises the following steps: carrying out feature extraction on obtained multi-source heterogeneous data to obtain multi-modal features, and uniformly mapping the multi-modal features to a first feature space of a preset dimension; in the first feature space, cross-modal deep fusion processing is performed on the multi-modal features, and a joint embedding space with consistent semantics is constructed according to the cross-modal deep fusion processing; constructing a vector index database based on a multi-modal feature vector in the joint embedding space, receiving a natural language query and mapping the natural language query to the joint embedding space, executing two-stage retrieval, and then obtaining a semantic fusion score based on calculated semantic fusion scores; multiplying a time sequence reward value based on the query time deviation and a dynamic reward value based on the core word matching degree to obtain a dynamic fusion score, and performing fusion sorting on the candidate set to obtain a final sorting result and an associated retrieval result; according to the method, refined sorting of the retrieval results is realized.

Example embodiments of the present disclosure provide a method, a device, and a computer program product for data query. The method includes selecting, according to a type of input data, a target pre-trained model from a deep network pool including a plurality of pre-trained models; performing, by using the selected target pre-trained model, feature extraction on the input data to determine text descriptors for the input data; and generating, based on the text descriptors, a query table for query. The method according to the present disclosure can select, according to different input data, different target pre-trained models from the deep network pool including the plurality of pre-trained models to process (e.g., compress) the input data. The method according to the present disclosure assembles a plurality of deep networks into a pool to automatically process data to obtain text descriptors for data retrieval, thereby achieving efficient data compression and retrieval.