231results about "Multimedia data querying" patented technology

The invention relates to the technical field of information retrieval and generation, in particular to a retrieval enhancement method based on multi-modal data fusion and modal perception. According to the method, firstly, a dual-channel architecture is adopted to perform feature extraction and coding on a text and an image respectively, and mutually independent embedded representation spaces are constructed, so that high-quality collaboration and matching of cross-modal representation are realized; and a pseudo-pairing generation mechanism is introduced to effectively mine and reconstruct the existing non-paired data in the knowledge base. And designing a query modal perception and dynamic weighting mechanism for accurately controlling the fusion proportion of the image-text bimodal information in the retrieval stage so as to match the modal demand difference of different query contents. And further executing aggregation retrieval and reordering of the cross-modal information by using dynamic weighted fusion retrieval to generate a candidate set of multi-modal responses. According to the method, accurate matching and dynamic weight adjustment of the image-text content are realized, and the accuracy and expression integrity of the generated content are improved.

The invention relates to a cross-modal document information extraction method based on space-semantic alignment, and belongs to the field of artificial intelligence, computer vision and natural language processing. According to the method, the spatial feature and semantic information bidirectional alignment model is designed, by constructing the spatial feature and semantic feature bidirectional alignment model, the document layout information can dynamically adjust attention distribution of text semantic features, meanwhile, semantic information reversely optimizes the spatial features, collaborative modeling of spatial layout and semantic information is achieved, and the document layout efficiency is improved. Therefore, the accuracy and robustness of complex document information extraction are improved. According to the method, a hierarchical cross-modal information extraction model is designed, through the hierarchical cross-modal information extraction model, the overall structure of a document is recognized on the global level, local key content is focused on the regional level, fine modeling is conducted on fine-grained texts and visual elements on the entity level, and accurate recognition of a cross-modal entity and the semantic relation of the cross-modal entity is achieved; and the generalization ability and applicability of information extraction are enhanced.

The invention relates to the technical field of computers, and discloses a nuclear power safety report examination method and system, electronic equipment and a storage medium, and the method comprises the steps: receiving an examination request, carrying out the intention recognition and entity extraction based on a nuclear power field large language model, and disassembling an examination task into a structured subtask sequence based on an intention type and an entity attribute; performing multi-modal retrieval based on the structured sub-task sequence, and outputting a multi-source retrieval result; calling time-space association information of the evolutionary knowledge graph, and outputting knowledge support data in combination with the time sequence model; fusing a multi-source retrieval result and the knowledge support data, performing risk value calculation and uncertainty quantification through a deep learning model, and outputting a risk assessment result; and based on the risk assessment result, carrying out federal learning model parameter adjustment, carrying out uplink evidence storage on the review conclusion, the risk assessment result and the evidence chain, and outputting the review result. The method has the advantages of being efficient, accurate, dynamic, safe and closed-loop during examination.

The invention discloses a model training method, electronic equipment and a computer readable storage medium, and relates to the technical field of data processing and large models. The method comprises the steps of obtaining a training data set, wherein the training data set comprises image annotation data of various types of concerned objects and description text annotation data associated with the image annotation data; training the initial multi-modal retrieval model by adopting the training data set to generate a learnable prompt; carrying out image coding training on the initial multi-modal retrieval model by adopting the training data set and the learnable prompt to generate an intermediate multi-modal retrieval model; the training data set is adopted to conduct text and image alignment training on the middle multi-modal retrieval model, a target multi-modal retrieval model is generated, and the target multi-modal retrieval model is used for conducting multi-modal retrieval on the target retrieval request to obtain a target retrieval result. According to the method and the device, the technical problems that a multi-modal retrieval model in related technologies cannot adapt to variability and large-scale data requirements, and the model performance is relatively poor are solved.

The invention discloses a multi-mode convergence media content auxiliary creation method based on an AI technology, and the method comprises the steps: extracting key features according to a creation demand text inputted by a user, carrying out the retrieval in a knowledge base through employing a knowledge graph retrieval mode according to the key features, obtaining a creation material, and generating a first draft; carrying out cross-modal feature mapping on the first draft by adopting a multi-modal alignment model, aligning text-image-video embedded vectors through contrastive learning, and dynamically adjusting the correlation of multi-modal contents by utilizing an attention mechanism to obtain a multi-modal content packet; duplicate checking is carried out on the multi-modal content packet in multiple modes, and the multi-modal content packet after duplicate checking is sent to a user for manual editing. Through multi-mode processing methods such as AI auxiliary writing, AI illustration and video generation and whole-network duplicate checking and propagation value evaluation, the content production efficiency and quality are improved, the propagation effect is optimized, and the original content is protected.

The invention provides a long-range visual question and answer and multi-modal reasoning task agent construction method. For input images and natural language questions, generating answers through orderly calling of an external tool set; the construction process of the intelligent agent is divided into three stages: a global planning stage: analyzing user requirements through a global planning navigator, and screening tool subsets; in the tool autonomous execution stage, visual details are supplemented, background knowledge is retrieved and accurate operation is carried out in the multi-step process by surrounding three types of tools through a tool autonomous executor, and a returned tool result is used for assisting the next step of decision making; in the response synthesis stage, the reasoning process is combed through a model response synthesizer, and user-friendly and clear-format responses are sorted; and finally, the model response synthesis module extracts and obtains answers from the reasoning track and outputs the answers. The problems of global planning insufficiency and visual context forgetting generally existing in complex task reasoning of an existing multi-mode large language model are solved.

The invention discloses a pedestrian re-identification method based on multi-modal auxiliary information guide learning, which belongs to the field of computer vision pedestrian re-identification, and comprises the following steps: constructing a model MMFI to carry out pedestrian re-identification; the model MMFI comprises a multi-modal information extraction module, an auxiliary information guide learning module and a knowledge distillation network; the method specifically comprises the following steps: step 1, acquiring an RGB image of a pedestrian as the input of a multi-modal information extraction module, and generating three modal features carrying different information by the input image through three different networks; 2, constructing an auxiliary information guide learning module and an implicit guide loss function; 3, constructing a knowledge distillation network and an explicit guide loss function; 4, constructing an overall loss function for overall back propagation of the model MMFI; and 5, constructing a reasoning module, and reducing the complexity of the model MMFI. The method is high in pedestrian re-identification accuracy and high in calculation efficiency.

The present disclosure provides a character recognition model training method and apparatus, a character recognition method and apparatus, a device and a medium, relating to the technical field of artificial intelligence, and specifically to the technical fields of deep learning, image processing and computer vision, which can be applied to scenarios such as character detection and recognition technology. The specific implementing solution is: partitioning an untagged training sample into at least two sub-sample images; dividing the at least two sub-sample images into a first training set and a second training set; where the first training set includes a first sub-sample image with a visible attribute, and the second training set includes a second sub-sample image with an invisible attribute; performing self-supervised training on a to-be-trained encoder by taking the second training set as a tag of the first training set, to obtain a target encoder.

The present disclosure provides a network model training method, a data processing method, and an apparatus. The network model training method comprises: acquiring target sample data, wherein the target sample data comprises text sample data and image sample data; inputting the target sample data into a network model to be trained to obtain a sample recognition result; and adjusting a parameter of a text encoder on the basis of a text recognition result and first supervision data corresponding to the text recognition result, adjusting a parameter of an image encoder on the basis of an image recognition result and second supervision data corresponding to the image recognition result, and a hybrid image-text recognition result and third supervision data corresponding to the hybrid image-text recognition result, and adjusting a parameter of a hybrid encoder on the basis of the hybrid image-text recognition result and the third supervision data corresponding to the hybrid image-text recognition result to obtain the trained network model formed by the text encoder, the image encoder, and the hybrid encoder.

The invention relates to a multi-dimensional knowledge base construction and query method based on a multi-modal large model, and belongs to the technical field of AI large model application. The method comprises the following steps of: analyzing a text file containing text, table and picture contents; selecting a pre-trained large language model and a multi-modal large model, generating text abstracts and picture abstracts by utilizing the models, and enabling storage paths of the picture abstracts to correspond to storage paths of pictures one by one; selecting a text embedding model to construct a vector database and a retriever; and constructing a retrieval chain based on the multi-modal large model. According to the method, construction from a single text knowledge base to a multi-dimensional knowledge base is achieved, a multi-dimensional information retrieval method is provided, the problem of construction of the multi-dimensional knowledge base based on the vectorization technology is solved to a certain extent, and the large model output quality based on the retrieval enhancement generation technology is remarkably improved.

The invention discloses a convergence media manuscript propagation effect prediction method based on an AI technology, and the method comprises the steps: reading a to-be-predicted manuscript, carrying out the classification and processing of the to-be-predicted manuscript according to a text manuscript, a video manuscript and an audio manuscript, and carrying out the normalization processing of the feature data obtained through the processing, and obtaining a structural feature vector; according to the structured feature vector, retrieving the same type of manuscripts based on theme similarity in a knowledge base, and performing analysis to obtain a spatial-temporal feature tensor; constructing a four-dimensional propagation element incidence matrix, establishing a feature cross validation mechanism, and generating a propagation potential comprehensive evaluation vector; and by constructing a dynamic prediction model, analyzing the structured feature vector, the spatial-temporal feature tensor and the propagation potential comprehensive evaluation vector to obtain a prediction result, and generating a structured propagation value analysis report based on the prediction result. According to the method, a multi-modal propagation value prediction method combining an AI kernel and a knowledge base is used, a multi-dimensional propagation effect evaluation system is constructed, and propagation value prediction analysis is effectively provided.

The invention relates to the technical field of artificial intelligence, in particular to a multi-modal data retrieval, generation and synthesis method and system based on artificial intelligence driving, and the method comprises the steps of multi-modal feature extraction, cross-modal alignment, feature fusion, multi-modal retrieval and sorting and multi-modal generation. Different deep learning models are adopted to carry out feature extraction on multi-modal data and convert the multi-modal data into vectors, through cross-modal alignment, different modal feature vectors are mapped to the same vector space, through feature fusion, correlation weights are calculated through an attention mechanism, weighted summation is carried out on fusion features, and through multi-modal retrieval and sorting, multi-modal data are obtained. According to the method, the similarity between a query vector and candidate data is calculated and sorted, and finally, through multi-modal generation, retrieval knowledge is used as external knowledge to be fused into a generation model, and a multi-modal synthesis answer is generated, so that information in multi-modal data can be effectively integrated and utilized.

The invention discloses a multi-modal intelligent question answering system and method based on a large model. The system comprises a signaling processing module, a document preprocessing module, a voice preprocessing module, a retrieval recall module, an RAG processing chain module and a knowledge base module. The method comprises the following steps: (1) a knowledge base offline construction step; (2) a knowledge base online construction step; (3) a real-time question and answer step; according to the system and the method provided by the invention, personalized intelligent question and answer service can be provided for audiences by combining materials which are uploaded by a speaker in advance and contain rich contents such as characters, pictures and charts and real-time presentation contents and utilizing multi-mode presentation information such as characters, pictures and voice, so that instant questions of the audiences can be answered, and the audiences can be more interested. Extensible knowledge supplementation can be provided according to the presentation content and materials.

The invention discloses a cross-modal conference information association retrieval method and system and a medium, and relates to the technical field of artificial intelligence, and the method comprises the following steps: carrying out feature extraction on obtained multi-source heterogeneous data to obtain multi-modal features, and uniformly mapping the multi-modal features to a first feature space of a preset dimension; in the first feature space, cross-modal deep fusion processing is performed on the multi-modal features, and a joint embedding space with consistent semantics is constructed according to the cross-modal deep fusion processing; constructing a vector index database based on a multi-modal feature vector in the joint embedding space, receiving a natural language query and mapping the natural language query to the joint embedding space, executing two-stage retrieval, and then obtaining a semantic fusion score based on calculated semantic fusion scores; multiplying a time sequence reward value based on the query time deviation and a dynamic reward value based on the core word matching degree to obtain a dynamic fusion score, and performing fusion sorting on the candidate set to obtain a final sorting result and an associated retrieval result; according to the method, refined sorting of the retrieval results is realized.

The application discloses a kind of PDF intelligent retrieval and generation method and system based on RAG, by obtaining the document data of input, using the classification model established in advance to parse document data, extract text content and image content to form first data set;Using deep learning model to the image content in first data set carries out feature extraction, while the text content in first data set applies natural language processing technology to carry out semantic analysis, obtains multimodal feature set;According to multimodal feature set, application information integration algorithm is uniformly encoded and is handled to generate second data set, if detecting the integrity of fusion feature vector in second data set is lower than preset threshold value, then supplementary context semantic analysis fills in missing information;Using preset index construction mechanism to the clustering processing of fusion feature vector in second data set, generates the retrieval index library containing classification index structure.The application improves the accuracy and comprehensiveness of document retrieval.

The embodiment of the invention provides a media file processing method and device, relates to the technical field of chips, and reduces the calculation overhead of a softmax link in an attention layer and the memory access overhead in the calculation process. According to the specific scheme, a query matrix, a key value matrix and a value matrix can be obtained based on an input matrix obtained from a media file, block calculation is carried out on the query matrix, the key value matrix and the value matrix, and softmax channel associated calculation is carried out on row blocks of the query matrix and column blocks of a transpose matrix of the key value matrix. The embodiment of the invention is suitable for the calculation process of the attention layer.

The invention discloses a document visual compression and retrieval method and device, equipment and a storage medium, and relates to the technical field of computers, the method comprises the following steps: obtaining a to-be-processed document page image, segmenting the image into a plurality of image blocks, and determining the structure category and the structure importance score of each image block; obtaining a plurality of structure regions based on structure category and spatial position aggregation, and distributing a preset number of compression tokens for each region in combination with structure category weights and importance scores; generating structure anchor point tokens corresponding to the regions by the compressed tokens to form a set; receiving a query request, converting the query request into a query vector, and performing retrieval in the anchor point token set to obtain a target structure region; and performing local decoding reconstruction based on the compressed token of the target region, and outputting a region image or a structure mask. According to the method, through structure-guided self-adaptive compression and fine-grained retrieval, the long document processing efficiency is greatly improved, and the compression effect and the retrieval accuracy are both considered.

The invention relates to the technical field of electronic data processing, and discloses a hydraulic engineering archive retrieval method and system.The hydraulic engineering archive retrieval method comprises the steps that a hydraulic engineering archive containing text data and image data is obtained, and the text data and the image data are mapped into an initial text feature vector and an initial image feature vector; the initial text feature vector and the initial image feature vector are projected to a shared semantic subspace, cross-modal representation is obtained, a joint loss function comprises a local feature alignment loss item and a semantic structure maintenance loss item, for any query request, the query request is converted into a query vector, and in the shared semantic subspace, the local feature alignment loss item and the semantic structure maintenance loss item are obtained. And sorting by calculating the cosine similarity of the query vector and the cross-modal representation of all archives in the archive library, and outputting a sorting result as retrieval. According to the method, fine-grained semantic alignment between the local key information of the image and the core terms of the text is realized, the understanding depth and accuracy of the core associated content of the image and text are improved, and the file retrieval precision is improved.

Techniques include minting a genesis token on a notarized ledger, the genesis token including a first pointer to an off-chain location. The techniques further include receiving new digital content related to the genesis token. The techniques further include generating a new digital token representing the new digital content. The techniques further include storing new digital token data related to the new digital token at the off-chain location. The techniques further include updating the off-chain location to include a second pointer to the new digital token. The techniques further include forming an expandable token set by linking the new digital token with the genesis token.

The invention discloses an image-text retrieval method and system based on multi-modal fusion and depth spectral clustering, and relates to the technical field of data retrieval, and the method comprises the steps: extracting image training features and text training features; splicing to obtain a global feature; constructing local image training features and local text training features; performing alignment processing on the local image training features and the local text training features; fusing the global features and the aligned local image training features and local text training features; performing clustering analysis on the normalized features after fusion feature normalization to obtain a plurality of clustering centers; and extracting query features of the query sample, and determining a retrieval result according to the cosine similarity. According to the method, the characterization capability and robustness of the model to the multi-modal data are improved through data-level fusion and local structure constraint, self-supervised alignment of the image and text modal data is realized through the positive sample pair and the negative sample pair, the clustering center of the image-text data is learned through depth spectral clustering, and new sample data are quickly adapted.

A multimodal information interaction method, an intelligent agent, an electronic device, and a storage medium are provided, which relate to a field of artificial intelligence technology, and in particular, to fields of large model and human-computer interaction technology. The method includes: performing intention recognition on a media resource request from a terminal to obtain an intention recognition result, where the intention recognition result represents whether the media resource request hits a predetermined processing mode; in response to the media resource request hitting the predetermined processing mode, determining a media resource address corresponding to the media resource request; and rendering a media resource in the media resource address, and outputting the rendered media stream to the terminal.

The invention relates to a multimedia aggregation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-aggregated multimedia set, wherein the to-be-aggregated multimedia set comprises a plurality of to-be-aggregated multimedia; obtaining correlation among the plurality of to-be-aggregated multimedia, and dividing the to-be-aggregated multimedia set into a plurality of to-be-aggregated multimedia subsets based on the correlation; and performing clustering processing on the multimedia to be aggregated in each subset of the multimedia to be aggregated to obtain an aggregation result of the plurality of multimedia to be aggregated. In this way, preliminary aggregation of the to-be-aggregated multimedia is achieved through the correlation between the to-be-aggregated multimedia, and accurate aggregation of the multiple to-be-aggregated multimedia in the to-be-aggregated set can be achieved by further performing clustering processing on the obtained to-be-aggregated multimedia subsets.

The invention discloses an index recovery method and device for streaming media data and a storage medium, and the method comprises the steps: persistently storing a data frame to a data storage area, caching an index to an index cache area, and sending the index to a slave storage device for backup storage; if the preset index flashing condition is met, persistently storing the cached index into an index storage area; after the main storage device is abnormal, the main storage device returns to normal, a backup index in the slave storage device is obtained, and the backup index is persistently stored in an index storage area; and if the index corresponding to the streaming media in the index storage area is still not completely recovered, reading the index of the data frame in the data storage area, and persistently storing the read index in the index storage area. The index recovery is carried out according to the backup index in the slave storage device, the index recovery speed can be increased, the information of the recovered index is more complete, then the index of the data frame in the data storage area is read for index recovery, and the index integrity of each data frame is ensured.

Embodiments of the present application provide a query method and an electronic device. The query method comprises: in response to querying a document by using a first query statement, obtaining a query word library and a document library; the query word library comprises an original query statement and a rewritten query statement; the document library comprises original documents and rewritten documents associated with each query statement; matching the first query statement with the query word library; in the case that the query word library contains the first query statement, obtaining a target document associated with the first query statement from the original documents and / or the rewritten documents; in the case that the query word library does not contain the first query statement, rewriting the first query statement by using a pre-trained online query statement rewriting model to obtain a second query statement; and obtaining a target document associated with the second query statement from the original documents and / or the rewritten documents. The present application can enhance the processing capability of the query system for complex query statements, significantly expand the document recall dimension, and optimize the query effect.

Systems and methods are provided herein for automatically arranging media assets that are contextually related to a requested media asset for access prior to a start time of the requested media asset. In some embodiments, user input circuitry may receive a user request to access a media asset. Control circuitry may identify, by consulting a database, a first set of media assets that share a context with the requested media asset and were released prior to the release date of the requested media asset. Control circuitry may determine a subset of media assets of the first set of media assets by performing a comparison of each media asset of the first set of the media assets against a viewing history of the user. Control circuitry may access a user profile to determine a user constraint, and generate a recommendation for display based on a first aspect of the user constraint.

The invention provides a multi-modal information retrieval and analysis method for internet real-time information streams, which comprises the following steps of: acquiring multi-modal data in the internet, including texts, images, videos and audios, and performing timestamp calibration on each modal data to form a multi-modal data set; based on the multi-modal data set, respectively extracting semantic features of each modal through a specific feature extractor, including semantic vectors of texts, visual features of images, spatial-temporal features of videos and audio features of audios; according to the multi-modal information retrieval and analysis method and system for the internet real-time information flow, multi-modal feature alignment is achieved by building a cross-modal joint embedding space, the retrieval precision is improved by adopting a three-stage retrieval assembly line, the calculation performance is optimized by combining model quantification and a heterogeneous calculation platform, and the multi-modal information retrieval and analysis method and system for the internet real-time information flow are achieved. The method has the advantages that the cross-modal retrieval accuracy is improved, the real-time processing capability is enhanced, and the calculation efficiency is optimized.

The present application discloses a media control method, a vehicle-mounted terminal, a storage medium, and a computer program product. The media control method is applied to a first fusion media application system; interfaces of a plurality of media sources are integrated in the first fusion media application system. The media control method comprises: obtaining a content identifier of target content and a media source identifier of a target media source, wherein the target media source is a media source of the target content; and obtaining the target content from a target media server according to the content identifier and by means of a target interface, wherein the target interface is an interface corresponding to the media source identifier, and the target media server is a media server of the target media source.

The present disclosure provides a multimedia resource loading method, comprising: in response to a trigger operation of requesting to display a target page, obtaining a resource mapping table corresponding to the target page; determining an identifier of a target multimedia resource to be loaded on the target page; determining a storage path of the target multimedia resource according to the resource mapping table corresponding to the target page based on the identifier of the target multimedia resource; and obtaining and loading the target multimedia resource based on the storage path of the target multimedia resource. Based on the above multimedia resource loading method, the present disclosure further provides a multimedia resource loading device, an electronic device, a storage medium and a program product.

Disclosed are systems and methods that provide a novel framework for censoring digital content prior to the display of such content and / or at runtime of such rendering of the digital content. The framework can automatically detect and censor particular types of content from rendered content, in real time, via computerized techniques employed to monitor user activity, intercept content concurrent with such activity, and then, perform modifications at the time of display to ensure that any unwanted, unnecessary and / or inappropriate content is censored. The framework operates to securely modify and / or obfuscate viewable content to ensure that the appropriate viewing status / mode of the content is performed. Such content censoring can be at the device-level, application level and / or server-level, in that the content is curated for display upon request, but prior to its display.

A content reproducing device is provided that can appropriately reproduce content associated with a predetermined road. [Solution] The content playback device includes a content acquisition means that requests a content distribution server to distribute content associated with a specified road and acquires the content from the content distribution server, and a playback means that plays back the content acquired by the content acquisition means.