315results about "Multimedia data querying" patented technology

The invention discloses an intelligent file classification and retrieval method and system, and the method comprises the steps: carrying out the unified format conversion and feature extraction of text, image and audio data according to the original data format of a file through a multi-modal data fusion algorithm, and obtaining a standardized file feature data set; based on a deep learning adaptive classification model, performing classification processing on the standardized archive feature data set to generate multi-level archive classification labels; analyzing the semantic relationship between the archive classification labels through an enhanced semantic network construction technology, generating a semantic association map, and optimizing the semantic association map to obtain an optimized semantic association map; and according to a retrieval request input by a user, performing semantic matching and correlation sorting by utilizing the optimized semantic correlation graph, and outputting archive information most related to the retrieval request. By utilizing the embodiment of the invention, the relevance and accuracy of retrieval can be improved, and modernization and intellectualization of archive management are promoted.

The invention discloses a multi-modal enhanced retrieval method and system, and the method comprises the steps: obtaining heterogeneous data, and executing data processing to obtain a meta-information set; encoding the meta-information set into a high-dimensional meta-information vector, and constructing a vector index; receiving a query statement of a user, calculating the similarity between the query statement and the meta-information vector, and obtaining a meta-information query result; and calling a large language model based on the obtained meta-information to generate a summary result, and merging and outputting the summary result and heterogeneous data corresponding to the meta-information. According to the method, multi-modal data is processed into unified semantic vector representation, efficient vectorization indexing and semantic generation and expansion capacity of a large-scale language model are combined, high-value information can be rapidly extracted from multi-source heterogeneous data, and real-time, cross-modal and high-precision retrieval requirements are supported. A reordering and traceability mechanism introduced in the scheme further enhances the correlation and reliability of the information, and provides comprehensive and accurate knowledge support for the generation of query results.

The invention relates to the technical field of information retrieval and generation, in particular to a retrieval enhancement method based on multi-modal data fusion and modal perception. According to the method, firstly, a dual-channel architecture is adopted to perform feature extraction and coding on a text and an image respectively, and mutually independent embedded representation spaces are constructed, so that high-quality collaboration and matching of cross-modal representation are realized; and a pseudo-pairing generation mechanism is introduced to effectively mine and reconstruct the existing non-paired data in the knowledge base. And designing a query modal perception and dynamic weighting mechanism for accurately controlling the fusion proportion of the image-text bimodal information in the retrieval stage so as to match the modal demand difference of different query contents. And further executing aggregation retrieval and reordering of the cross-modal information by using dynamic weighted fusion retrieval to generate a candidate set of multi-modal responses. According to the method, accurate matching and dynamic weight adjustment of the image-text content are realized, and the accuracy and expression integrity of the generated content are improved.

The invention relates to a cross-modal document information extraction method based on space-semantic alignment, and belongs to the field of artificial intelligence, computer vision and natural language processing. According to the method, the spatial feature and semantic information bidirectional alignment model is designed, by constructing the spatial feature and semantic feature bidirectional alignment model, the document layout information can dynamically adjust attention distribution of text semantic features, meanwhile, semantic information reversely optimizes the spatial features, collaborative modeling of spatial layout and semantic information is achieved, and the document layout efficiency is improved. Therefore, the accuracy and robustness of complex document information extraction are improved. According to the method, a hierarchical cross-modal information extraction model is designed, through the hierarchical cross-modal information extraction model, the overall structure of a document is recognized on the global level, local key content is focused on the regional level, fine modeling is conducted on fine-grained texts and visual elements on the entity level, and accurate recognition of a cross-modal entity and the semantic relation of the cross-modal entity is achieved; and the generalization ability and applicability of information extraction are enhanced.

A method, apparatus, non-transitory computer readable medium, and system for media processing includes obtaining a variation parameter and a number of variations, identifying a first variation input and a second variation input for the variation parameter, and obtaining a first media content item and a second media content item based on the first variation input and the second variation input, respectively. The first media content item and the second media content item vary from each other with respect to the variation parameter. The method, apparatus, non-transitory computer readable medium, and system for media processing further includes displaying the first media content item and the second media content item in a grid comprising a grid size based on the number of variations.

The invention relates to the technical field of computers, and discloses a nuclear power safety report examination method and system, electronic equipment and a storage medium, and the method comprises the steps: receiving an examination request, carrying out the intention recognition and entity extraction based on a nuclear power field large language model, and disassembling an examination task into a structured subtask sequence based on an intention type and an entity attribute; performing multi-modal retrieval based on the structured sub-task sequence, and outputting a multi-source retrieval result; calling time-space association information of the evolutionary knowledge graph, and outputting knowledge support data in combination with the time sequence model; fusing a multi-source retrieval result and the knowledge support data, performing risk value calculation and uncertainty quantification through a deep learning model, and outputting a risk assessment result; and based on the risk assessment result, carrying out federal learning model parameter adjustment, carrying out uplink evidence storage on the review conclusion, the risk assessment result and the evidence chain, and outputting the review result. The method has the advantages of being efficient, accurate, dynamic, safe and closed-loop during examination.

The invention relates to the technical field of knowledge graph and content management, in particular to an intelligent education content management method and system based on a knowledge graph, and the method comprises the steps: obtaining multi-modal education content data which comprises text data and audio data; based on the multi-modal education content data, executing multi-modal tensor decomposition to obtain unified feature representation; according to the unified feature representation, topological continuous coherence feature extraction is executed to obtain topological features; based on the topological features, quantum entanglement feature enhancement is executed, and enhanced features are obtained; executing non-exchangeable geometric feature fusion according to the enhanced features to obtain fusion features; based on the fusion features, chaotic dynamics knowledge extraction is executed, and extracted knowledge is obtained; according to the method, the extracted knowledge is constructed into the knowledge graph, different forms of education contents such as texts and audios are integrated through a multi-modal tensor decomposition technology, complementary information among different modals can be captured, and comprehensiveness and accuracy of knowledge representation are improved.

Provided are systems, methods, and apparatuses for systems and methods of memory efficient multi-vector information retrieval based on embeddings from a storage pipelined network. In one or more examples, the systems, devices, and methods include performing a first portion of a nearest neighbor search on a first subset of a set of media sources; performing a fetch process on the first subset that include identifying a first set of highest matching media sources from the first subset, transferring multi-vector representations of the first set of highest matching media sources from a storage drive to a memory, and performing a second ranking of the first set highest matching media sources. The systems, devices, and methods include performing, in parallel with the fetch process, a second portion of the nearest neighbor search on a second subset of the set of media sources.

The invention discloses a model training method, electronic equipment and a computer readable storage medium, and relates to the technical field of data processing and large models. The method comprises the steps of obtaining a training data set, wherein the training data set comprises image annotation data of various types of concerned objects and description text annotation data associated with the image annotation data; training the initial multi-modal retrieval model by adopting the training data set to generate a learnable prompt; carrying out image coding training on the initial multi-modal retrieval model by adopting the training data set and the learnable prompt to generate an intermediate multi-modal retrieval model; the training data set is adopted to conduct text and image alignment training on the middle multi-modal retrieval model, a target multi-modal retrieval model is generated, and the target multi-modal retrieval model is used for conducting multi-modal retrieval on the target retrieval request to obtain a target retrieval result. According to the method and the device, the technical problems that a multi-modal retrieval model in related technologies cannot adapt to variability and large-scale data requirements, and the model performance is relatively poor are solved.

The invention discloses a multi-mode convergence media content auxiliary creation method based on an AI technology, and the method comprises the steps: extracting key features according to a creation demand text inputted by a user, carrying out the retrieval in a knowledge base through employing a knowledge graph retrieval mode according to the key features, obtaining a creation material, and generating a first draft; carrying out cross-modal feature mapping on the first draft by adopting a multi-modal alignment model, aligning text-image-video embedded vectors through contrastive learning, and dynamically adjusting the correlation of multi-modal contents by utilizing an attention mechanism to obtain a multi-modal content packet; duplicate checking is carried out on the multi-modal content packet in multiple modes, and the multi-modal content packet after duplicate checking is sent to a user for manual editing. Through multi-mode processing methods such as AI auxiliary writing, AI illustration and video generation and whole-network duplicate checking and propagation value evaluation, the content production efficiency and quality are improved, the propagation effect is optimized, and the original content is protected.

The invention provides a long-range visual question and answer and multi-modal reasoning task agent construction method. For input images and natural language questions, generating answers through orderly calling of an external tool set; the construction process of the intelligent agent is divided into three stages: a global planning stage: analyzing user requirements through a global planning navigator, and screening tool subsets; in the tool autonomous execution stage, visual details are supplemented, background knowledge is retrieved and accurate operation is carried out in the multi-step process by surrounding three types of tools through a tool autonomous executor, and a returned tool result is used for assisting the next step of decision making; in the response synthesis stage, the reasoning process is combed through a model response synthesizer, and user-friendly and clear-format responses are sorted; and finally, the model response synthesis module extracts and obtains answers from the reasoning track and outputs the answers. The problems of global planning insufficiency and visual context forgetting generally existing in complex task reasoning of an existing multi-mode large language model are solved.

The invention discloses a pedestrian re-identification method based on multi-modal auxiliary information guide learning, which belongs to the field of computer vision pedestrian re-identification, and comprises the following steps: constructing a model MMFI to carry out pedestrian re-identification; the model MMFI comprises a multi-modal information extraction module, an auxiliary information guide learning module and a knowledge distillation network; the method specifically comprises the following steps: step 1, acquiring an RGB image of a pedestrian as the input of a multi-modal information extraction module, and generating three modal features carrying different information by the input image through three different networks; 2, constructing an auxiliary information guide learning module and an implicit guide loss function; 3, constructing a knowledge distillation network and an explicit guide loss function; 4, constructing an overall loss function for overall back propagation of the model MMFI; and 5, constructing a reasoning module, and reducing the complexity of the model MMFI. The method is high in pedestrian re-identification accuracy and high in calculation efficiency.

The present disclosure provides a character recognition model training method and apparatus, a character recognition method and apparatus, a device and a medium, relating to the technical field of artificial intelligence, and specifically to the technical fields of deep learning, image processing and computer vision, which can be applied to scenarios such as character detection and recognition technology. The specific implementing solution is: partitioning an untagged training sample into at least two sub-sample images; dividing the at least two sub-sample images into a first training set and a second training set; where the first training set includes a first sub-sample image with a visible attribute, and the second training set includes a second sub-sample image with an invisible attribute; performing self-supervised training on a to-be-trained encoder by taking the second training set as a tag of the first training set, to obtain a target encoder.

The present disclosure relates to the fields of large language models and machine learning. Disclosed are a multimedia information processing method, a system, and an electronic device. The method is applied to a generative question-answering system, and comprises: receiving inquiry information; rewriting the inquiry information into a candidate query used for executing customized querying for multimedia information, wherein the multimedia information is content uploaded to the generative question-answering system in advance; retrieving from a question-answering database a data chunk associated with the candidate query, wherein at least one set of data chunks is stored in the question-answering database, and a data source of the data chunks is the multimedia information; and using a large language model to process the data chunk to generate at least one piece of reply content matching the inquiry information.

The present disclosure provides a network model training method, a data processing method, and an apparatus. The network model training method comprises: acquiring target sample data, wherein the target sample data comprises text sample data and image sample data; inputting the target sample data into a network model to be trained to obtain a sample recognition result; and adjusting a parameter of a text encoder on the basis of a text recognition result and first supervision data corresponding to the text recognition result, adjusting a parameter of an image encoder on the basis of an image recognition result and second supervision data corresponding to the image recognition result, and a hybrid image-text recognition result and third supervision data corresponding to the hybrid image-text recognition result, and adjusting a parameter of a hybrid encoder on the basis of the hybrid image-text recognition result and the third supervision data corresponding to the hybrid image-text recognition result to obtain the trained network model formed by the text encoder, the image encoder, and the hybrid encoder.

The invention relates to a multi-dimensional knowledge base construction and query method based on a multi-modal large model, and belongs to the technical field of AI large model application. The method comprises the following steps of: analyzing a text file containing text, table and picture contents; selecting a pre-trained large language model and a multi-modal large model, generating text abstracts and picture abstracts by utilizing the models, and enabling storage paths of the picture abstracts to correspond to storage paths of pictures one by one; selecting a text embedding model to construct a vector database and a retriever; and constructing a retrieval chain based on the multi-modal large model. According to the method, construction from a single text knowledge base to a multi-dimensional knowledge base is achieved, a multi-dimensional information retrieval method is provided, the problem of construction of the multi-dimensional knowledge base based on the vectorization technology is solved to a certain extent, and the large model output quality based on the retrieval enhancement generation technology is remarkably improved.

The invention discloses a convergence media manuscript propagation effect prediction method based on an AI technology, and the method comprises the steps: reading a to-be-predicted manuscript, carrying out the classification and processing of the to-be-predicted manuscript according to a text manuscript, a video manuscript and an audio manuscript, and carrying out the normalization processing of the feature data obtained through the processing, and obtaining a structural feature vector; according to the structured feature vector, retrieving the same type of manuscripts based on theme similarity in a knowledge base, and performing analysis to obtain a spatial-temporal feature tensor; constructing a four-dimensional propagation element incidence matrix, establishing a feature cross validation mechanism, and generating a propagation potential comprehensive evaluation vector; and by constructing a dynamic prediction model, analyzing the structured feature vector, the spatial-temporal feature tensor and the propagation potential comprehensive evaluation vector to obtain a prediction result, and generating a structured propagation value analysis report based on the prediction result. According to the method, a multi-modal propagation value prediction method combining an AI kernel and a knowledge base is used, a multi-dimensional propagation effect evaluation system is constructed, and propagation value prediction analysis is effectively provided.

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal data retrieval, generation and synthesis method and system based on artificial intelligence driving, and the method comprises the steps of multi-modal feature extraction, cross-modal alignment, feature fusion, multi-modal retrieval and sorting and multi-modal generation. Different deep learning models are adopted to carry out feature extraction on multi-modal data and convert the multi-modal data into vectors, through cross-modal alignment, different modal feature vectors are mapped to the same vector space, through feature fusion, correlation weights are calculated through an attention mechanism, weighted summation is carried out on fusion features, and through multi-modal retrieval and sorting, multi-modal data are obtained. According to the method, the similarity between a query vector and candidate data is calculated and sorted, and finally, through multi-modal generation, retrieval knowledge is used as external knowledge to be fused into a generation model, and a multi-modal synthesis answer is generated, so that information in multi-modal data can be effectively integrated and utilized.

The invention discloses a multi-modal intelligent question answering system and method based on a large model. The system comprises a signaling processing module, a document preprocessing module, a voice preprocessing module, a retrieval recall module, an RAG processing chain module and a knowledge base module. The method comprises the following steps: (1) a knowledge base offline construction step; (2) a knowledge base online construction step; (3) a real-time question and answer step; according to the system and the method provided by the invention, personalized intelligent question and answer service can be provided for audiences by combining materials which are uploaded by a speaker in advance and contain rich contents such as characters, pictures and charts and real-time presentation contents and utilizing multi-mode presentation information such as characters, pictures and voice, so that instant questions of the audiences can be answered, and the audiences can be more interested. Extensible knowledge supplementation can be provided according to the presentation content and materials.

The invention discloses a cross-modal conference information association retrieval method and system and a medium, and relates to the technical field of artificial intelligence, and the method comprises the following steps: carrying out feature extraction on obtained multi-source heterogeneous data to obtain multi-modal features, and uniformly mapping the multi-modal features to a first feature space of a preset dimension; in the first feature space, cross-modal deep fusion processing is performed on the multi-modal features, and a joint embedding space with consistent semantics is constructed according to the cross-modal deep fusion processing; constructing a vector index database based on a multi-modal feature vector in the joint embedding space, receiving a natural language query and mapping the natural language query to the joint embedding space, executing two-stage retrieval, and then obtaining a semantic fusion score based on calculated semantic fusion scores; multiplying a time sequence reward value based on the query time deviation and a dynamic reward value based on the core word matching degree to obtain a dynamic fusion score, and performing fusion sorting on the candidate set to obtain a final sorting result and an associated retrieval result; according to the method, refined sorting of the retrieval results is realized.

Example embodiments of the present disclosure provide a method, a device, and a computer program product for data query. The method includes selecting, according to a type of input data, a target pre-trained model from a deep network pool including a plurality of pre-trained models; performing, by using the selected target pre-trained model, feature extraction on the input data to determine text descriptors for the input data; and generating, based on the text descriptors, a query table for query. The method according to the present disclosure can select, according to different input data, different target pre-trained models from the deep network pool including the plurality of pre-trained models to process (e.g., compress) the input data. The method according to the present disclosure assembles a plurality of deep networks into a pool to automatically process data to obtain text descriptors for data retrieval, thereby achieving efficient data compression and retrieval.

The application discloses a kind of PDF intelligent retrieval and generation method and system based on RAG, by obtaining the document data of input, using the classification model established in advance to parse document data, extract text content and image content to form first data set;Using deep learning model to the image content in first data set carries out feature extraction, while the text content in first data set applies natural language processing technology to carry out semantic analysis, obtains multimodal feature set;According to multimodal feature set, application information integration algorithm is uniformly encoded and is handled to generate second data set, if detecting the integrity of fusion feature vector in second data set is lower than preset threshold value, then supplementary context semantic analysis fills in missing information;Using preset index construction mechanism to the clustering processing of fusion feature vector in second data set, generates the retrieval index library containing classification index structure.The application improves the accuracy and comprehensiveness of document retrieval.

The invention provides an object processing method, which comprises the following steps: processing a to-be-processed object to obtain key feature information and initial feature information of the to-be-processed object, the initial feature information comprising local feature information and global feature information; performing fusion processing on the key feature information, the local feature information and the global feature information to obtain target feature information of the to-be-processed object, the key feature information being target frame information or text feature information based on the to-be-processed object of different modalities; and determining that an object meeting a similarity condition with the target feature information of the to-be-processed object is a result object corresponding to the to-be-processed object, the modality of the to-be-processed object is different from the modality of the result object, and the modality comprises at least one of a text, an image, a video and voice. The invention further provides an object processing device and electronic equipment.

A search data processing method includes: obtaining a media search graph including a query node, a media node, and an association node; obtaining first training sample pairs, including a positive node pair and a negative node pair, from the media search graph, sampling the media search graph based on meta-paths respectively corresponding to the query and the media data, to obtain sampling sub-graphs; inputting the sampling sub-graphs to an initial graph neural network to obtain respective initial semantic features and form a semantic feature pair corresponding to the first training sample pair; and training the initial graph neural network based on a difference between semantic feature pairs of the positive node pair and the negative node pair, to obtain a target graph neural network that is configured to determine a target semantic feature corresponding to a query node or a media node.

The embodiment of the invention provides a media file processing method and device, relates to the technical field of chips, and reduces the calculation overhead of a softmax link in an attention layer and the memory access overhead in the calculation process. According to the specific scheme, a query matrix, a key value matrix and a value matrix can be obtained based on an input matrix obtained from a media file, block calculation is carried out on the query matrix, the key value matrix and the value matrix, and softmax channel associated calculation is carried out on row blocks of the query matrix and column blocks of a transpose matrix of the key value matrix. The embodiment of the invention is suitable for the calculation process of the attention layer.

The invention relates to a multi-mode multi-parameter intelligent operation and maintenance platform with real-time front-end sensing and cloud side end cooperation, and belongs to the field of operation and maintenance systems. The terminal equipment layer is used for collecting and transmitting bottom layer data in real time; the edge computing layer is used for enhancing real-time performance and expandability; the cloud platform layer is used for global analysis, optimization and coordination of the system; the operation and maintenance platform comprises a multi-modal data classification module which is used for dividing data into time sequence data, relational data and coded data. The multi-modal data model under the cloud side-end collaborative architecture not only can support flexible storage of various types of data, but also can ensure efficient management and utilization of the data through accurate query and update operations.

The invention discloses a document visual compression and retrieval method and device, equipment and a storage medium, and relates to the technical field of computers, the method comprises the following steps: obtaining a to-be-processed document page image, segmenting the image into a plurality of image blocks, and determining the structure category and the structure importance score of each image block; obtaining a plurality of structure regions based on structure category and spatial position aggregation, and distributing a preset number of compression tokens for each region in combination with structure category weights and importance scores; generating structure anchor point tokens corresponding to the regions by the compressed tokens to form a set; receiving a query request, converting the query request into a query vector, and performing retrieval in the anchor point token set to obtain a target structure region; and performing local decoding reconstruction based on the compressed token of the target region, and outputting a region image or a structure mask. According to the method, through structure-guided self-adaptive compression and fine-grained retrieval, the long document processing efficiency is greatly improved, and the compression effect and the retrieval accuracy are both considered.

The invention provides a multi-modal information interaction method and device, an intelligent agent, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of large models and human-computer interaction. According to the specific implementation scheme, intention recognition is conducted on a media resource request from a terminal, an intention recognition result is obtained, and the intention recognition result represents whether the media resource request hits a preset processing mode or not; calling a first multi-mode processing module to determine a media resource address corresponding to the media resource request in response to the media resource request hitting a preset processing mode; and calling a second multi-mode processing module to render the media resource in the media resource address, and outputting the rendered media stream to the terminal.

The invention relates to the technical field of electronic data processing, and discloses a hydraulic engineering archive retrieval method and system.The hydraulic engineering archive retrieval method comprises the steps that a hydraulic engineering archive containing text data and image data is obtained, and the text data and the image data are mapped into an initial text feature vector and an initial image feature vector; the initial text feature vector and the initial image feature vector are projected to a shared semantic subspace, cross-modal representation is obtained, a joint loss function comprises a local feature alignment loss item and a semantic structure maintenance loss item, for any query request, the query request is converted into a query vector, and in the shared semantic subspace, the local feature alignment loss item and the semantic structure maintenance loss item are obtained. And sorting by calculating the cosine similarity of the query vector and the cross-modal representation of all archives in the archive library, and outputting a sorting result as retrieval. According to the method, fine-grained semantic alignment between the local key information of the image and the core terms of the text is realized, the understanding depth and accuracy of the core associated content of the image and text are improved, and the file retrieval precision is improved.

The invention provides a resource auditing method, device and equipment based on a large model and a storage medium, and relates to the technical field of artificial intelligence such as data security and large models. The specific implementation scheme is as follows: obtaining target image-text data of a to-be-audited resource; based on the target image-text data, respectively retrieving a target positive sample and a target negative sample most related to the target image-text data from a pre-created positive sample knowledge base and a pre-created negative sample knowledge base; and based on the target image-text data, the target positive sample and the target negative sample, adopting a pre-trained large model to audit whether the to-be-audited resource is illegal. According to the technology disclosed by the invention, the content security auditing accuracy and auditing efficiency can be effectively improved.