Image-text data retrieval method and device, electronic equipment and readable storage medium

Abstract

The present disclosure relates to the technical field of data retrieval, and provides a kind of graphic text data retrieval method, device, electronic equipment and readable storage medium.The method comprises: based on the content index data of pre-set, query graphic text data is carried out cascaded content retrieval processing, obtains candidate semantic text data;Based on the image semantic index data of pre-set, candidate semantic text data is carried out semantic association retrieval processing, obtains associated image data;Query graphic text data, candidate semantic text data and associated image data are carried out graphic text evidence integration processing, obtain multimodal target retrieval result, so as to improve the semantic association accuracy of graphic text retrieval result, improve the response precision of multimodal retrieval result to user query intent, improve the content matching degree of graphic text retrieval result, improve the semantic integrity and consistency of multimodal information in retrieval result, improve the accuracy and robustness of graphic text data retrieval.

Description

Technical Field

[0001] This disclosure relates to the field of data retrieval technology, and in particular to a method, apparatus, electronic device, and readable storage medium for retrieving image and text data. Background Technology

[0002] In the field of image and text data retrieval, text and images in documents are typically indexed independently, and queries are responded to using a single vector retrieval method. However, this approach ignores the deep semantic relationships between text and images, resulting in fragmented image and text content in the retrieval results. This makes it difficult to accurately match the user's query intent, especially when the query requires understanding the image and text context, leading to low retrieval accuracy. Existing technologies can enhance image descriptions and generate richer image tags using visual models, but these tags still lack association with the specific context of the document. Furthermore, while joint embedding maps image and text pairs to the same vector space for retrieval, it has weak modeling capabilities for non-strictly paired image and text data (such as images and scattered explanatory text), making it difficult to guarantee the semantic consistency of the target results.

[0003] It is evident that existing technologies suffer from problems such as low content matching degree, poor semantic consistency, and low accuracy in understanding user query intent due to the fragmentation of semantic association between images and text, the simplistic indexing logic, and the lack of image association retrieval mechanisms. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, electronic device and readable storage medium for image and text data retrieval, in order to solve the problems in the prior art that, due to the fragmented semantic association between images and text, the simple indexing logic, and the lack of image association retrieval mechanisms, the content matching degree of image and text retrieval results is low, the semantic consistency is poor, and the accuracy of understanding the user's query intent is low.

[0005] A first aspect of this disclosure provides a method for retrieving image and text data, comprising: performing cascaded content retrieval processing on query image and text data based on preset content index data to obtain candidate semantic text data; performing semantic association retrieval processing on candidate semantic text data based on preset image semantic index data to obtain associated image data; and performing image and text evidence integration processing on query image and text data, candidate semantic text data, and associated image data to obtain multimodal target retrieval results.

[0006] In some embodiments, cascading content retrieval processing is performed on query image and text data based on preset content index data to obtain candidate semantic text data, including: semantic encoding processing of query image and text data to obtain query vector data; similarity measurement processing of query vector data and preset content index data to obtain semantic matching data; and candidate extraction processing of preset content index data based on semantic matching data to obtain candidate semantic text data.

[0007] In some embodiments, semantic association retrieval processing is performed on candidate semantic text data based on preset image semantic index data to obtain associated image data, including: performing semantic association encoding processing on candidate semantic text data to obtain text association vector data; performing association similarity measurement processing on text association vector data and preset image semantic index data to obtain association matching data; and performing association recall processing on preset image semantic index data based on association matching data to obtain associated image data.

[0008] In some embodiments, query image and text data, candidate semantic text data, and associated image data are processed to integrate image and text evidence to obtain multimodal target retrieval results, including: performing evidence alignment processing on query image and text data, candidate semantic text data, and associated image data to obtain aligned evidence data; performing evidence fusion processing on the aligned evidence data to obtain fused evidence data; and performing response generation processing on the fused evidence data to obtain multimodal target retrieval results.

[0009] In some embodiments, before performing cascaded content retrieval processing on the query image and text data based on preset content index data to obtain candidate semantic text data, the method further includes: parsing and segmenting the initial document data to obtain text fragment data and initial image data; performing dual description generation processing on the initial image data to obtain content description data and associated description data; and performing index construction processing on the text fragment data, content description data, and associated description data to obtain preset content index data and preset image semantic index data.

[0010] In some embodiments, semantic encoding processing is performed on query text and image data to obtain query vector data, including: performing query type identification processing on query text and image data to obtain query type data; performing modality conversion processing on query text and image data based on query type data to obtain standardized query text and image data; and performing semantic mapping processing on standardized query text and image data to obtain query vector data.

[0011] In some embodiments, the text association vector data and the preset image semantic index data are subjected to association similarity measurement processing to obtain association matching data, including: performing association weight allocation processing on the text association vector data to obtain weighted vector data; performing association distance measurement processing on the weighted vector data and the preset image semantic index data to obtain association distance data; and performing sorting and filtering processing on the association distance data to obtain association matching data.

[0012] A second aspect of this disclosure provides a text and image data retrieval device, comprising: a first processing module, configured to perform cascaded content retrieval processing on query text and image data based on preset content index data to obtain candidate semantic text data; a second processing module, configured to perform semantic association retrieval processing on candidate semantic text data based on preset image semantic index data to obtain associated image data; and a third processing module, configured to perform text and image evidence integration processing on query text and image data, candidate semantic text data, and associated image data to obtain multimodal target retrieval results.

[0013] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0014] A fourth aspect of this disclosure provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0015] The beneficial effects of this disclosed embodiment compared with the prior art are as follows: Through cascaded content retrieval processing, the query image and text data is semantically matched with preset content index data to locate candidate semantic text data most relevant to the query intent; through semantic association retrieval processing, the candidate semantic text data is used as a retrieval condition and associated with preset image semantic index data to recall associated image data that highly matches the candidate semantic text data at the semantic level; furthermore, through image and text evidence integration processing, the query image and text data, candidate semantic text data, and associated image data are fused into a logically consistent image and text evidence package to generate multimodal target retrieval results. This improves the semantic association accuracy of image and text retrieval results, enhances the logical self-consistency of the image and text evidence package in the retrieval stage, improves the response accuracy of multimodal retrieval results to user query intent, enhances the deep mining capability of cross-modal semantic association, improves the content matching degree of image and text retrieval results, improves the semantic integrity and consistency of multimodal information in retrieval results, and improves the accuracy and robustness of image and text data retrieval. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure; Figure 2 This is a schematic flowchart of a text and image data retrieval method provided in an embodiment of this disclosure; Figure 3 This is a flowchart illustrating another image and text data retrieval method provided in this embodiment of the disclosure; Figure 4 This is a schematic diagram of the structure of a text and image data retrieval device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0019] It should be noted that the user information (including but not limited to terminal device information, user personal information, etc.) and data (including but not limited to data used for display, data used for analysis, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties.

[0020] A method and apparatus for retrieving image and text data according to an embodiment of the present disclosure will now be described in detail with reference to the accompanying drawings.

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.

[0022] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays and supporting communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the aforementioned electronic devices. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not limit this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0023] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This disclosure embodiment does not limit this.

[0024] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This disclosure does not limit the scope of the embodiments.

[0025] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This disclosure does not limit the scope of the network.

[0026] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, server 4 can obtain query image and text data through terminal devices 1, 2, and 3, and perform semantic matching between the query image and text data and preset content index data through cascaded content retrieval processing to locate candidate semantic text data most relevant to the query intent; through semantic association retrieval processing, the candidate semantic text data is used as a retrieval condition and matched with preset image semantic index data to recall associated image data that highly matches the candidate semantic text data at the semantic level; and then through image and text evidence integration processing, the query image and text data, candidate semantic text data, and associated image data are merged into a logically consistent image and text evidence package to generate multimodal target retrieval results.

[0027] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any restrictions on this.

[0028] Figure 2 This is a schematic flowchart of a text and image data retrieval method provided in an embodiment of this disclosure. Figure 2 The image and text data retrieval method can be derived from Figure 1 The server executes this. For example... Figure 2 As shown, the image and text data retrieval method includes: S201, based on the preset content index data, perform cascading content retrieval processing on the queried image and text data to obtain candidate semantic text data.

[0029] Specifically, the preset content index data can be a structured retrieval database formed by preprocessing, extracting information, and building an index from the original text and image data. This preset content index data can be used to store text fragment indexes and image semantic indexes. The query text and image data can be a retrieval request containing text and / or image data input by the user. This query text and image data can be used to represent the user's retrieval intent.

[0030] Cascaded content retrieval processing can be a data processing flow that executes retrieval in stages and sequentially. This cascaded content retrieval processing can be used to use the retrieval results of the previous stage as the query input of the next stage, thereby realizing the progressive decomposition of the retrieval task and obtaining candidate semantic text data.

[0031] Among them, candidate semantic text data can be a set of text fragments that are semantically related to the query image and text data obtained through the first stage of the retrieval process. This candidate semantic text data can be used to provide a semantic basis for subsequent image association retrieval.

[0032] Furthermore, the cascaded content retrieval process may specifically include semantic encoding of the query image and text data to obtain query vector data; similarity measurement of the query vector data and the preset content index data to obtain semantic matching data; and candidate extraction processing of the preset content index data based on the semantic matching data to obtain candidate semantic text data.

[0033] This application embodiment performs cascaded content retrieval processing on query image and text data based on preset content index data, semantically encodes the query image and text data to obtain query vector data, and measures the similarity between the query vector and the content index to obtain semantic matching data. Candidate semantic text data can be extracted from the content index based on the matching results, thereby enhancing the semantic matching capability between multimodal query content and structured index; improving the accuracy and efficiency of candidate text fragment retrieval; and providing a semantic foundation for subsequent image association retrieval.

[0034] For example, in the intelligent customer service knowledge base retrieval, the query image and text data can be image and text data containing product fault description text and related screenshots. The query image and text data can be semantically encoded to obtain query vector data. The query vector data and the preset content index data can be similarly measured to obtain semantic matching data. Based on the semantic matching data, the technical document fragments most relevant to the fault description semantics can be extracted from the content index as candidate semantic text data. The candidate semantic text data can be used for subsequent related retrieval of the corresponding product structure diagram.

[0035] S202, based on the preset image semantic index data, perform semantic association retrieval processing on the candidate semantic text data to obtain associated image data.

[0036] Specifically, the preset image semantic index data can be a structured retrieval database obtained by generating context descriptions and building indexes from the original image and text data. This preset image semantic index data can be used to store image data and its associated semantic description data, thereby establishing a mapping relationship between the image and its semantic role in the original document.

[0037] Furthermore, semantic association retrieval processing can be a data processing flow that searches for related entries in an index database based on semantic similarity. This semantic association retrieval processing can be used to match the semantic representation of candidate semantic text data with the image semantic description stored in the preset image semantic index data, thereby locating the most semantically relevant image and obtaining associated image data.

[0038] Among them, the associated image data can be a set of images that are highly semantically related to the candidate semantic text data, obtained through semantic association retrieval processing. This associated image data can be used as visual evidence to support and interpret the candidate semantic text data, providing an image basis for subsequent image and text evidence integration processing.

[0039] Furthermore, the semantic association retrieval process may specifically include performing semantic association encoding on candidate semantic text data to obtain text association vector data; performing association similarity measurement on the text association vector data and preset image semantic index data to obtain association matching data; and performing association recall processing on the preset image semantic index data based on the association matching data to obtain associated image data.

[0040] Among them, text association vector data can be a high-dimensional feature representation obtained by vectorization of candidate semantic text data. This text association vector data can be used to measure the association similarity with the image semantic index in the vector space.

[0041] The association matching data can be the similarity calculation results between each image entry in the text association vector data and the preset image semantic index data. This association matching data can be used to characterize the degree of association between the text semantics and the image semantic description.

[0042] In addition, the association recall process can be a data processing procedure that filters the most relevant image entries from the preset image semantic index data based on the association matching data. This association recall process can be used to locate the associated image data that best matches the candidate semantic text data.

[0043] This application embodiment obtains text association vector data by performing semantic association encoding on candidate semantic text data; obtains association matching data by performing association similarity measurement on the text association vector data and preset image semantic index data; and obtains associated image data by performing association recall processing on the preset image semantic index data based on the association matching data. This enhances the semantic association accuracy of image retrieval, improves the consistency between retrieval results and text context, and enhances the logical self-consistency of the multimodal retrieval evidence package.

[0044] For example, in online education Q&A, candidate semantic text data can be textbook text fragments about the steps of the light reaction in photosynthesis. The candidate semantic text data can be semantically associated with the encoding process to obtain text association vector data. The text association vector data and the preset image semantic index data can be associated with similarity measurement to obtain association matching data. Based on the association matching data, a schematic diagram showing the electron transport chain of the thylakoid membrane, the absorption spectrum of photosynthetic pigments, and the structure of adenosine triphosphate (ATP) synthase can be extracted from the image semantic index as associated image data. This associated image data will be used to integrate with the candidate semantic text data to generate illustrated teaching responses.

[0045] S203, integrate the query image and text data, candidate semantic text data and associated image data into image and text evidence to obtain multimodal target retrieval results.

[0046] Specifically, the image and text evidence integration processing can be a data processing flow that combines multi-source data in a structured way. This image and text evidence integration processing can be used to construct a unified data structure that includes text and image evidence, thereby generating a complete evidence package that is adapted to the input of downstream models and obtaining multimodal target retrieval results.

[0047] Among them, the multimodal target retrieval results can be retrieval results that include text and image evidence obtained through the integration of text and image evidence. These multimodal target retrieval results can be used to characterize the retrieval system's complete response to the user's query.

[0048] Furthermore, the image and text evidence integration processing can specifically include aligning the query image and text data, candidate semantic text data, and associated image data to obtain aligned evidence data; fusing the aligned evidence data to obtain fused evidence data; and generating responses from the fused evidence data to obtain multimodal target retrieval results.

[0049] This application embodiment aligns query textual data, candidate semantic text data, and associated image data to obtain aligned evidence data; it then fuses the aligned evidence data to obtain fused evidence data; and finally, it generates a response from the fused evidence data to obtain multimodal target retrieval results. This enhances the intrinsic correlation between textual and image evidence, improves the information richness and structure of the retrieval results, and enhances the accuracy and understandability of the multimodal retrieval response.

[0050] For example, in new energy technology knowledge retrieval, the query image and text data can be an image and text query about the working principle of solar panels, the candidate semantic text data can be text fragments describing the photovoltaic effect and the basic structure (Positive-Negative Junction, PN) in semiconductor devices, and the associated image data can be a photovoltaic effect diagram and a PN junction structure diagram. Evidence alignment processing can be performed on the query image and text data, candidate semantic text data, and associated image data, pairing the text fragments describing the photovoltaic effect with the photovoltaic effect diagram, and pairing the text fragments describing the PN junction with the PN junction structure diagram to obtain aligned evidence data. Evidence fusion processing can be performed on the aligned evidence data, encapsulating the paired image and text evidence into a unified data structure to obtain fused evidence data. Response generation processing can be performed on the fused evidence data to output multimodal target retrieval results containing complete image and text evidence.

[0051] According to the technical solution provided in this disclosure, cascaded content retrieval processing is performed on query text and image data based on preset content index data. Candidate semantic text data is obtained through semantic encoding, similarity measurement, and candidate extraction. Semantic association retrieval processing is then performed on the candidate semantic text data based on preset image semantic index data. Associated image data is obtained through semantic association encoding, association similarity measurement, and association recall. The query text and image data, candidate semantic text data, and associated image data are then integrated into a text and image evidence system. Through evidence alignment, evidence fusion, and response generation, multimodal target retrieval results are obtained, thereby enhancing… It enhances the semantic matching capability between multimodal query content and structured indexes; improves the semantic association accuracy and information richness of image retrieval results; enhances the accuracy and understandability of multimodal retrieval responses; improves the semantic association accuracy of image and text retrieval results; enhances the logical self-consistency of image and text evidence packages during the retrieval stage; improves the response accuracy of multimodal retrieval results to user query intent; enhances the ability to deeply mine cross-modal semantic associations; improves the content matching degree of image and text retrieval results; improves the semantic integrity and consistency of multimodal information in retrieval results; and improves the accuracy and robustness of image and text data retrieval.

[0052] In some embodiments, cascading content retrieval processing is performed on query image and text data based on preset content index data to obtain candidate semantic text data, including: semantic encoding processing of query image and text data to obtain query vector data; similarity measurement processing of query vector data and preset content index data to obtain semantic matching data; and candidate extraction processing of preset content index data based on semantic matching data to obtain candidate semantic text data.

[0053] Specifically, semantic encoding can be a data processing procedure that converts unstructured query data into vector representations. This semantic encoding can be used to generate machine-computable vectorized representations, thereby achieving a unified representation of multimodal query information and obtaining query vector data.

[0054] The query vector data can be a numerical vector representing the semantic information of the query image and text data. This query vector data can be used as a benchmark for similarity calculation in the vector space and can be obtained by semantically encoding the query image and text data.

[0055] Furthermore, the semantic encoding process may specifically include performing query type identification processing on the query text and image data to obtain query type data; performing modality conversion processing on the query text and image data based on the query type data to obtain standardized query text and image data; and performing semantic mapping processing on the standardized query text and image data to obtain query vector data.

[0056] Furthermore, similarity measurement processing can be a data processing procedure for calculating the degree of semantic association between query vectors and index entries. This similarity measurement processing can be used to generate quantitative results that characterize the degree of semantic association between queries and document content, thus obtaining semantic matching data.

[0057] The semantic matching data can be the similarity calculation result between the query vector data and each item in the preset content index data. This semantic matching data can be used to characterize the degree of semantic association between the query intent and the index content, and can be obtained by performing similarity measurement processing on the query vector data and the preset content index data.

[0058] Furthermore, the similarity measurement process may specifically include calculating the distance between the query vector data and each entry vector in the preset content index data to obtain the raw distance data; normalizing the raw distance data to obtain the standardized distance data; and quantifying the degree of association based on the standardized distance data to obtain the semantic matching data.

[0059] In addition, candidate extraction processing can be a data processing procedure that filters relevant entries from the index based on the degree of semantic relevance. This candidate extraction processing can be used to locate the most relevant candidate content for querying from massive index data.

[0060] Furthermore, the candidate extraction process may specifically include sorting based on semantic matching data to obtain sorted result data; threshold filtering based on sorted result data to obtain filtered result data; and item extraction based on filtered result data to obtain candidate semantic text data.

[0061] For example, in new energy technology knowledge retrieval, the query for graphic data can be a graphic query about the working principle of solar panels. The query graphic data can be semantically encoded to obtain query vector data. The query vector data and the preset content index data can be similarly measured to obtain semantic matching data. Based on the semantic matching data, technical document fragments related to photovoltaic effect and PN junction structure can be extracted from the content index as candidate semantic text data. The candidate semantic text data will be used for subsequent association retrieval of corresponding photovoltaic effect diagrams and PN junction structure diagrams.

[0062] According to the technical solution provided in this disclosure, semantic encoding is performed on the query image and text data. Query vector data is obtained through query type identification, modality conversion, and semantic mapping. The query vector data and preset content index data are then subjected to similarity measurement. Semantic matching data is obtained through distance calculation, normalization, and correlation quantification. Candidate extraction processing is performed on the preset content index data based on the semantic matching data. Candidate semantic text data is obtained through sorting, threshold filtering, and item extraction. This enhances the semantic matching capability between multimodal query content and structured index; improves the accuracy and efficiency of candidate text fragment retrieval; and provides a semantic foundation for subsequent image association retrieval.

[0063] In some embodiments, semantic association retrieval processing is performed on candidate semantic text data based on preset image semantic index data to obtain associated image data, including: performing semantic association encoding processing on candidate semantic text data to obtain text association vector data; performing association similarity measurement processing on text association vector data and preset image semantic index data to obtain association matching data; and performing association recall processing on preset image semantic index data based on association matching data to obtain associated image data.

[0064] Specifically, semantic association coding can be a data processing process that converts candidate semantic text data into vector representations. This semantic association coding process can be used to extract deep semantic features of text, thereby generating machine-understandable vector representations, realizing the mapping of text semantics to vector space, and obtaining text-related vector data.

[0065] Among them, the text association vector data can be a numerical vector representing the semantic information of the candidate semantic text data. This text association vector data can be used as input for similarity calculation with the image semantic index, thereby realizing cross-modal semantic association retrieval. The text association vector data can be obtained by performing semantic association encoding on the candidate semantic text data.

[0066] Furthermore, the semantic association encoding process may specifically include performing semantic feature extraction on candidate semantic text data to obtain initial semantic feature data; and performing vector mapping on the initial semantic feature data to obtain text association vector data.

[0067] Furthermore, the association similarity measurement process can be a data processing procedure for calculating the degree of semantic association between text association vectors and image semantic index entries. This association similarity measurement process can be used to quantify the association strength between candidate text semantics and the semantics of each image context, thereby generating matching results that can be directly compared and ranked, i.e., association matching data.

[0068] Among them, the association matching data can be the similarity calculation result between text association vector data and each image entry in the preset image semantic index data. This association matching data can be used to characterize the degree of association between text semantics and image semantic description, providing a data basis for subsequent association recall. The association matching data can be obtained by performing association similarity measurement processing on text association vector data and preset image semantic index data.

[0069] Furthermore, the association similarity measurement process may specifically include calculating the distance between the text association vector data and each entry vector in the preset image semantic index data to obtain the original association distance data; normalizing the original association distance data to obtain standardized association distance data; and quantifying the degree of association based on the standardized association distance data to obtain association matching data.

[0070] Furthermore, the association recall process can be a data processing procedure that filters relevant image entries from the image semantic index based on the degree of semantic association. This association recall process can be used to locate the image content most relevant to the candidate semantic text from a massive image index, thereby obtaining a set of images with highly consistent semantics and obtaining associated image data.

[0071] Furthermore, the association recall process may specifically include sorting based on association matching data to obtain association sorting result data; threshold filtering based on association sorting result data to obtain association filtering result data; and image extraction based on association filtering result data to obtain association image data.

[0072] For example, in product document knowledge retrieval, candidate semantic text data can be text fragments about network connection parameter configuration. Semantic association encoding processing can be performed on the candidate semantic text data to obtain text association vector data. The text association vector data and preset image semantic index data can be processed to obtain association matching data through association similarity measurement. Based on the association matching data, a screenshot of the Internet Protocol Address (IP) setting area of ​​the network configuration interface can be extracted from the image semantic index as association image data. This association image data will be used to integrate with the candidate semantic text data to generate a configuration guidance reply with both text and images.

[0073] According to the technical solution provided in this disclosure, by performing semantic association encoding on candidate semantic text data, obtaining text association vector data through semantic feature extraction and vector mapping, performing association similarity measurement processing on the text association vector data and preset image semantic index data, obtaining association matching data through distance calculation, normalization, and association degree quantification, and performing association recall processing on the preset image semantic index data based on the association matching data, obtaining associated image data through sorting, threshold filtering, and image extraction, thereby improving the semantic association accuracy and recall efficiency of image retrieval; enhancing the consistency between retrieval results and text context; and improving the logical self-consistency of multimodal retrieval evidence packages.

[0074] In some embodiments, query image and text data, candidate semantic text data, and associated image data are processed to integrate image and text evidence to obtain multimodal target retrieval results, including: performing evidence alignment processing on query image and text data, candidate semantic text data, and associated image data to obtain aligned evidence data; performing evidence fusion processing on the aligned evidence data to obtain fused evidence data; and performing response generation processing on the fused evidence data to obtain multimodal target retrieval results.

[0075] Specifically, evidence alignment processing can be a data processing procedure for performing structured association analysis on heterogeneous modal data. This evidence alignment processing can be used to establish semantic correspondences between different modal evidence, thereby generating a structured data set that records the relationship between text and images, and obtaining aligned evidence data.

[0076] Among them, the aligned evidence data can be a structured data set obtained through evidence alignment processing, which can be used to record the semantic correspondence between different modal evidence. This aligned evidence data can be used to characterize the reference, explanation or illustration relationship between text fragments and images, providing a unified contextual framework for subsequent evidence fusion. The aligned evidence data can be obtained by performing evidence alignment processing on query image and text data, candidate semantic text data and associated image data.

[0077] Furthermore, the evidence alignment process can specifically include extracting key information from candidate semantic text data to obtain key text information data; performing visual content parsing on associated image data to obtain visual object data; and mapping and associating the key text information data and visual object data to obtain aligned evidence data.

[0078] Furthermore, evidence fusion processing can be a data processing procedure that integrates and complements multimodal evidence with established alignment relationships. This evidence fusion processing can be used to combine the abstract semantic information of text and the specific visual information of images to generate a unified evidence representation with richer content and more complete semantics, thus obtaining fused evidence data.

[0079] Among them, fused evidence data can be a unified multimodal evidence representation that integrates detailed text descriptions and intuitive image displays, obtained through evidence fusion processing. This fused evidence data can be used to support the generation of target retrieval results. Fused evidence data can also be obtained by performing evidence fusion processing on aligned evidence data.

[0080] Furthermore, the evidence fusion process may specifically include cross-modal attention calculation based on aligned evidence data to obtain attention weight data; and dynamic weighted fusion processing based on attention weight data to obtain fused evidence data.

[0081] Furthermore, the response generation process can be a data processing procedure for generating natural language responses based on the multimodal evidence obtained through fusion. This response generation process can be used to transform structured multimodal evidence into answers or summaries that are easy for users to understand, thereby outputting richly illustrated search results and obtaining multimodal target search results.

[0082] Furthermore, the response generation process may specifically include assembling the query image and text data and the fused evidence data into the model input data; performing inference generation processing on the model input data to obtain the initial response data; and performing format normalization processing on the initial response data to obtain the multimodal target retrieval results.

[0083] For example, in product document knowledge retrieval, querying text and image data can be a text and image query about network device configuration steps, candidate semantic text data can be text fragments describing the configuration process, and associated image data can be a collection of images displaying screenshots of the configuration interface. Evidence alignment processing can be performed on the query text and image data, candidate semantic text data, and associated image data to establish a correspondence between the text fragments describing IP address configuration and the screenshots displaying the IP settings interface, resulting in aligned evidence data. Evidence fusion processing can be performed on the aligned evidence data to dynamically enhance the weight of the configuration interface screenshots in the fused evidence, resulting in fused evidence data. Response generation processing can be performed on the fused evidence data to output multimodal target retrieval results containing textual descriptions of the configuration steps and references to corresponding interface screenshots. These multimodal target retrieval results can be used to provide users with illustrated configuration guidance.

[0084] According to the technical solution provided in this disclosure, by aligning query text and image data, candidate semantic text data, and associated image data, and obtaining aligned evidence data through key information extraction, visual content parsing, and mapping association, the aligned evidence data is then fused. Evidence fusion processing is performed on the aligned evidence data, and fused evidence data is obtained through cross-modal attention calculation and dynamic weighted fusion. Response generation processing is then performed on the fused evidence data, and multimodal target retrieval results are obtained through input assembly, inference generation, and format standardization. This enhances the intrinsic correlation between textual and image evidence, improves the information richness and structure of the retrieval results, and enhances the accuracy and understandability of the multimodal retrieval response.

[0085] In some embodiments, before performing cascaded content retrieval processing on the query image and text data based on preset content index data to obtain candidate semantic text data, the method further includes: parsing and segmenting the initial document data to obtain text fragment data and initial image data; performing dual description generation processing on the initial image data to obtain content description data and associated description data; and performing index construction processing on the text fragment data, content description data, and associated description data to obtain preset content index data and preset image semantic index data.

[0086] Specifically, the initial document data can be original electronic files containing mixed text and images. This initial document data can be used as the input source for the retrieval system to provide the original text and image information to be retrieved and processed.

[0087] In addition, the parsing and segmentation process can specifically include parsing the initial document data to obtain structured text data and the original image set data; performing logical segmentation on the structured text data to obtain text fragment data; and performing image extraction on the original image set data to obtain the initial image data.

[0088] Among them, text fragment data can be a logical unit with relatively complete semantics formed by segmenting a complete text. This text fragment data can be used as the basic building unit of content indexing. Initial image data can be a set of undescribed original images extracted from initial document data. This initial image data can be used as the original material for generating image description information. Both text fragment data and initial image data can be obtained by parsing and segmenting the initial document data.

[0089] Furthermore, dual description generation processing can be a data processing procedure that generates two different types of descriptive information in parallel for the same image. This dual description generation processing can be used to construct a comprehensive image description that includes objective visual content and contextual semantic association, thereby achieving a separation of the image content dimension and semantic function dimension, and obtaining content description data and association description data.

[0090] The content description data can be text data that describes the objective visual content of the image, and this content description data can be used to characterize the visual content information of the image; the association description data can be text data that characterizes the role or purpose of the image in the document context, and this association description data can be used to characterize the semantic association and functional intent of the image with the surrounding text; the content description data and the association description data can be obtained by performing a dual description generation process on the initial image data.

[0091] Furthermore, the dual description generation process can specifically include performing visual content description generation processing on the initial image data to obtain content description data; and performing contextual semantic description generation processing on the initial image data to obtain associated description data.

[0092] In addition, the index building process can be a data processing process that vectorizes and stores text fragment data, content description data and related description data to form a retrieval database. This index building process can be used to generate a structured index set that supports similarity search, thereby realizing the independent storage and separate retrieval of text content and image semantic description, and obtaining preset content index data and preset image semantic index data.

[0093] Furthermore, the index construction process may specifically include performing a first vectorization process on the text fragment data and content description data to obtain content vector data; performing a second vectorization process on the associated description data to obtain semantic vector data; performing a first index storage process on the content vector data to obtain preset content index data; and performing a second index storage process on the semantic vector data to obtain preset image semantic index data.

[0094] For example, in the construction of a technical document knowledge base, the initial document data can be a Portable Document Format (PDF) file containing product description text and structural diagrams. This initial document data can be parsed and segmented to obtain text fragments describing product functions and initial image data displaying the product structure diagram. The initial image data can undergo dual description generation processing to obtain content description data representing the appearance of the product structure and product structure function description data representing the explanatory role of the structure diagram in the document. The text fragment data, content description data, and associated description data can be indexed to obtain preset content index data and preset image semantic index data. This preset content index data and preset image semantic index data will be used in subsequent cascading retrieval processes to respond to user queries about the product structure.

[0095] According to the technical solution provided in this disclosure, the initial document data is parsed and segmented. Through document parsing, logical segmentation, and image extraction, text fragment data and initial image data are obtained. The initial image data undergoes dual description generation processing, through visual content description generation and contextual semantic description generation, to obtain content description data and associated description data. The text fragment data, content description data, and associated description data are indexed and constructed. Through vectorization and index storage, preset content index data and preset image semantic index data are obtained. This enhances the structured parsing capability of mixed text and image documents; improves the comprehensiveness of image description and the accuracy of semantic association; and provides a structured index foundation for subsequent cascaded content retrieval and semantic association retrieval.

[0096] In some embodiments, semantic encoding processing is performed on query text and image data to obtain query vector data, including: performing query type identification processing on query text and image data to obtain query type data; performing modality conversion processing on query text and image data based on query type data to obtain standardized query text and image data; and performing semantic mapping processing on standardized query text and image data to obtain query vector data.

[0097] Specifically, query type identification processing can be a data processing process that analyzes the modal composition attributes of query text and image data to generate classification labels. This query type identification processing can be used to determine the dominant information modal type of the query request, thereby determining the subsequent modal transformation processing strategy and obtaining query type data.

[0098] Among them, query type data can be the classification result representing the modal composition attributes of the query text and image data. This query type data can be used to provide a decision basis for subsequent modality conversion processing. Query type data can be obtained by performing query type identification processing on the query text and image data.

[0099] Furthermore, the query type identification process may specifically include performing modal component analysis on the query text and image data to obtain modal component data; and performing type determination processing based on the modal component data to obtain query type data.

[0100] In addition, modality conversion processing can be a data processing process that converts heterogeneous modal queries into standardized text representations according to the query type. This modality conversion processing can be used to eliminate the heterogeneity between different modal query data, thereby generating a unified input format that adapts to subsequent semantic mapping, and obtaining standardized query text and image data.

[0101] The standardized query graph data can be structured query data obtained through a unified format and modality transformation. This standardized query graph data can be used to provide a unified input interface for semantic mapping processing. The standardized query graph data can also be obtained by performing modality transformation on the query graph data based on the query type data.

[0102] Furthermore, the modality transformation process can specifically include performing transformation path selection based on query type data to obtain transformation strategy data; and performing modality adaptation processing on the query text and image data based on the transformation strategy data to obtain standardized query text and image data.

[0103] Furthermore, semantic mapping can be a data processing procedure that maps standardized text data to a high-dimensional vector space. This semantic mapping can be used to extract deep semantic features and generate fixed-dimensional vector representations, thereby realizing a numerical representation of query intent and obtaining query vector data.

[0104] Furthermore, the semantic mapping process can specifically include extracting semantic features from the standardized query text and image data to obtain initial semantic feature data; and performing vector projection on the initial semantic feature data to obtain query vector data.

[0105] For example, in intelligent product knowledge retrieval, the query image and text data can be a pure image query containing screenshots of product malfunctions. The query image and text data can be processed by query type identification to obtain image-type query type data. Based on the query type data, the query image and text data can be processed by modality conversion to obtain standardized query image and text data describing the malfunction phenomenon. The standardized query image and text data can be processed by semantic mapping to obtain query vector data. This query vector data will be used to retrieve relevant troubleshooting text in the content index later.

[0106] According to the technical solution provided in this disclosure, query type data is obtained by performing query type identification processing on query text and image data, modal component analysis and type determination, modal transformation processing is performed on query text and image data based on query type data, and standardized query text and image data is obtained through transformation path selection and modal adaptation. Semantic mapping processing is performed on standardized query text and image data, and query vector data is obtained through semantic feature extraction and vector projection. This enhances the unified representation capability of multimodal query data, improves the accuracy of query type identification and the flexibility of modal transformation, and provides a standardized vector input basis for subsequent similarity measurement.

[0107] In some embodiments, the text association vector data and the preset image semantic index data are subjected to association similarity measurement processing to obtain association matching data, including: performing association weight allocation processing on the text association vector data to obtain weighted vector data; performing association distance measurement processing on the weighted vector data and the preset image semantic index data to obtain association distance data; and performing sorting and filtering processing on the association distance data to obtain association matching data.

[0108] Specifically, the association weight allocation process can be a data processing procedure that assigns differentiated importance to each dimension of the text association vector based on preset rules. This association weight allocation process can be used to highlight semantic features related to the image context retrieval task, thereby generating an optimized query representation and obtaining weighted vector data.

[0109] Among them, the weighted vector data can be vector data obtained by assigning associated weights and adjusting each dimension according to its importance. This weighted vector data can be used as an optimized query representation for accurate similarity matching with image semantic index; the weighted vector data can also be obtained by assigning associated weights to text associated vector data.

[0110] Furthermore, the association weight allocation process may specifically include performing feature importance analysis on the text association vector data to obtain feature weight data; and performing weighted adjustment on the text association vector data based on the feature weight data to obtain weighted vector data.

[0111] Furthermore, the associated distance metric processing can be a data processing procedure for calculating the spatial distance between the weighted vector and each entry of the image semantic index. This associated distance metric processing can be used to quantify the degree of association between the text query and the semantic context of the image, thereby generating distance results that can be directly compared and sorted, and obtaining associated distance data.

[0112] Among them, the association distance data can be the similarity calculation result between the weighted vector data and each image entry in the preset image semantic index data. This association distance data can be used to characterize the degree of association between text semantics and image semantic description. The association distance data can be obtained by performing association distance measurement processing on the weighted vector data and the preset image semantic index data.

[0113] Furthermore, the associated distance measurement processing can specifically include performing distance calculation processing between the weighted vector data and each entry vector in the preset image semantic index data to obtain the original distance data; normalizing the original distance data to obtain standardized distance data; and performing association degree quantification processing based on the standardized distance data to obtain associated distance data.

[0114] In addition, the sorting and filtering process can be a data processing procedure that prioritizes and filters image entries based on their degree of relevance. This sorting and filtering process can be used to locate the image content most relevant to the candidate semantic text from a massive image index, thereby outputting image matching results with high semantic fit and obtaining associated matching data.

[0115] Furthermore, the sorting and filtering process may specifically include sorting the data in descending order based on the correlation distance data to obtain the sorted result data; performing threshold filtering based on the sorted result data to obtain the filtered result data; and extracting items based on the filtered result data to obtain the correlation matching data.

[0116] For example, in technical document knowledge retrieval, text association vector data can be a vector representing the semantics of network configuration steps. This text association vector data can be processed by association weight allocation to obtain weighted vector data that highlights the semantics related to configuration parameters. The weighted vector data and preset image semantic index data can be processed by association distance measurement to obtain association distance data representing the degree of semantic association with various configuration screenshots. The association distance data can be sorted and filtered to obtain association matching data that displays screenshots of router configuration interfaces and IP address settings. This association matching data will be used to extract associated image data to generate illustrated configuration guides.

[0117] According to the technical solution provided in this disclosure, by performing association weight allocation processing on text association vector data, and obtaining weighted vector data through feature importance analysis and weight adjustment, the weighted vector data and preset image semantic index data are subjected to association distance measurement processing. Through distance calculation, normalization, and association degree quantification, association distance data is obtained. The association distance data is sorted and filtered. Through descending order sorting, threshold filtering, and item extraction, association matching data is obtained. This enhances the association matching accuracy between text query semantics and image index; improves the consistency between retrieval results and text context; and enhances the logical self-consistency of multimodal retrieval evidence package.

[0118] All of the above-mentioned optional technical solutions can be combined in any way to form optional embodiments of this disclosure, and will not be described in detail here.

[0119] Figure 3 This is a schematic diagram of another image and text data retrieval method provided in this embodiment of the disclosure. For example... Figure 3 As shown, the image and text data retrieval method includes: 1. Data preprocessing and multimodal information extraction: In one specific embodiment, a raw document containing text and images (querying text and image data) can be received, such as a PDF file, an editable document (Word document), or a Hypertext Markup Language (HTML) webpage. This document is then processed by a document parsing module.

[0120] Text Extraction: This module is responsible for extracting all plain text content from the document, preserving its original logical structure and reading order as much as possible, such as chapters, paragraphs, and headings. The extracted plain text content can be candidate semantic text data (text).

[0121] Image Extraction: This module can simultaneously identify and extract all image resources in a document and save them as independent image files. The extracted image set can be associated image data (pics).

[0122] 2. Dual description generation of text and images: Text Segmentation: The extracted complete text is divided into smaller, more easily searchable logical units, namely text_segments, according to preset rules.

[0123] Image Content Description Generation: For each image in the image collection `pics`, a Visual Language Model (VLM) can be used to generate an objective content description. The input to this process is the image itself.

[0124] Image Contextual Description Generation: For each image in the image collection `pics`, the following operations can be performed to generate its contextual description: Locating contextual text: Based on the recorded image location information, the text segments immediately preceding and following the image in the original text can be extracted. For example, extracting the paragraph before and / or after the image.

[0125] Language model generation: Input the image and its associated context text into a Large Language Model (LLM) or Visual Language Model (VLM). Guide the model to analyze and summarize the image's role, purpose, or core message within that context.

[0126] 3. Building a dual-index database: Build a content index (Segment Index): The content index is responsible for retrieving the overall content of a document.

[0127] Building an image context index: The image context index is specifically responsible for retrieving images with specific functions based on text context.

[0128] 4. Query response and generation based on cascading retrieval: When a user query is received, the following two-stage cascading retrieval is performed: Phase 1: Content retrieval (the process of turning a query into a text response).

[0129] The user's query can be vectorized using the same embedding model as when the index was built.

[0130] Then, a similarity search is performed in the segment index to retrieve the K text fragments most similar to the query vector. These K fragments are the text content most directly related to the user's question, and we call them "candidate context text." A text response is then generated.

[0131] Phase Two: Context-Related Image Retrieval (the process of returning text to Pics).

[0132] The first-stage text response can be used as a new query condition. After vectorization, a similarity search is performed in the Pic Context Index to retrieve the N images most similar to the context text vector. Since this index stores descriptions of the "function" and "purpose" of the images, it can obtain the images that best explain, support, or illustrate the text content found in the previous step.

[0133] The third stage: integration and generation of the final response: the text response from the first stage and the N images retrieved in the second stage can be integrated into a context-sensitive "evidence package".

[0134] The user's original query, along with this "evidence package," can be submitted to a backend visual language model (VLM).

[0135] After receiving highly matching text and image materials, VLM can perform in-depth understanding and reasoning to generate an accurate, detailed, and richly illustrated natural language response.

[0136] According to the technical solution provided in this disclosure, by generating two completely different and functionally independent descriptive texts—content description and context description—for each image in parallel, the integrity and retrieval of the semantic information of the image in the document context are improved. Based on the dual descriptions, two independent and functionally specialized vector index libraries—content index and image context index—are constructed, enhancing the retrieval architecture's ability to adapt to different query intentions. Through a cascaded two-stage retrieval process, the content index locates the core text paragraphs and generates text responses. The text responses are then used as new query conditions to retrieve the best matching image in the image context index, improving the inherent correlation and logical consistency of the recalled image-text combinations.

[0137] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0138] Figure 4 This is a schematic diagram of a text and image data retrieval device provided in an embodiment of this disclosure. Figure 4 As shown, the image and text data retrieval device includes: The first processing module 401 is used to perform cascaded content retrieval processing on the queried image and text data based on preset content index data to obtain candidate semantic text data. The second processing module 402 is used to perform semantic association retrieval processing on candidate semantic text data based on preset image semantic index data to obtain associated image data. The third processing module 403 is used to integrate the query image and text data, candidate semantic text data and associated image data into image and text evidence to obtain multimodal target retrieval results.

[0139] According to the technical solution provided in this disclosure, cascaded content retrieval processing is performed on query text and image data based on preset content index data. Candidate semantic text data is obtained through semantic encoding, similarity measurement, and candidate extraction. Semantic association retrieval processing is then performed on the candidate semantic text data based on preset image semantic index data. Associated image data is obtained through semantic association encoding, association similarity measurement, and association recall. The query text and image data, candidate semantic text data, and associated image data are then integrated into a text and image evidence system. Through evidence alignment, evidence fusion, and response generation, multimodal target retrieval results are obtained, thereby enhancing… It enhances the semantic matching capability between multimodal query content and structured indexes; improves the semantic association accuracy and information richness of image retrieval results; enhances the accuracy and understandability of multimodal retrieval responses; improves the semantic association accuracy of image and text retrieval results; enhances the logical self-consistency of image and text evidence packages during the retrieval stage; improves the response accuracy of multimodal retrieval results to user query intent; enhances the ability to deeply mine cross-modal semantic associations; improves the content matching degree of image and text retrieval results; improves the semantic integrity and consistency of multimodal information in retrieval results; and improves the accuracy and robustness of image and text data retrieval.

[0140] In some embodiments, the first processing module 401 is specifically used to: perform semantic encoding processing on the query image and text data to obtain query vector data; perform similarity measurement processing on the query vector data and preset content index data to obtain semantic matching data; and perform candidate extraction processing on the preset content index data based on the semantic matching data to obtain candidate semantic text data.

[0141] In some embodiments, the second processing module 402 is specifically used to: perform semantic association encoding processing on the candidate semantic text data to obtain text association vector data; perform association similarity measurement processing on the text association vector data and the preset image semantic index data to obtain association matching data; and perform association recall processing on the preset image semantic index data based on the association matching data to obtain associated image data.

[0142] In some embodiments, the third processing module 403 is specifically used to perform evidence alignment processing on the query image and text data, candidate semantic text data and associated image data to obtain aligned evidence data; perform evidence fusion processing on the aligned evidence data to obtain fused evidence data; and perform response generation processing on the fused evidence data to obtain multimodal target retrieval results.

[0143] In some embodiments, the above-described image and text retrieval device is further configured to: parse and segment the initial document data to obtain text fragment data and initial image data; perform dual description generation processing on the initial image data to obtain content description data and associated description data; and perform index construction processing on the text fragment data, content description data, and associated description data to obtain preset content index data and preset image semantic index data.

[0144] In some embodiments, semantic encoding of query text and image data to obtain query vector data is specifically used for: query type identification processing of query text and image data to obtain query type data; modality conversion processing of query text and image data based on query type data to obtain standardized query text and image data; and semantic mapping processing of standardized query text and image data to obtain query vector data.

[0145] In some embodiments, the process of performing association similarity measurement on text association vector data and preset image semantic index data to obtain association matching data specifically involves: performing association weight allocation processing on text association vector data to obtain weighted vector data; performing association distance measurement processing on weighted vector data and preset image semantic index data to obtain association distance data; and performing sorting and filtering processing on association distance data to obtain association matching data.

[0146] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its correlation and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0147] Figure 5 This is a schematic diagram of the electronic device 5 provided in an embodiment of this disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various method embodiments described above. Alternatively, when the processor 501 executes the computer program 503, it implements the association of the modules / units in the various device embodiments described above.

[0148] Electronic device 5 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 5 may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or different components.

[0149] The processor 501 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0150] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM of the electronic device 5. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 5. The memory 502 can also include both internal and external storage units of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device.

[0151] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of related units and modules is used as an example. In practical applications, the above-described associations can be assigned to different related units and modules as needed, that is, the internal structure of the device can be divided into different related units or modules to complete all or part of the associations described above. The related units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in software as a related unit.

[0152] If integrated modules / units are implemented as software-associated units and sold or used as independent products, they can be stored in a readable storage medium (e.g., a computer-readable storage medium). Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable storage medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0153] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method for retrieving image and text data, characterized in that, include: Based on the preset content index data, the query image and text data is processed by cascading content retrieval to obtain candidate semantic text data; Based on the preset image semantic index data, semantic association retrieval processing is performed on the candidate semantic text data to obtain associated image data; The query image and text data, the candidate semantic text data, and the associated image data are integrated and processed as image and text evidence to obtain multimodal target retrieval results.

2. The image and text data retrieval method according to claim 1, characterized in that, The cascading content retrieval processing of the queried text and image data based on preset content index data yields candidate semantic text data, including: The query image and text data are semantically encoded to obtain query vector data; The query vector data and the preset content index data are subjected to similarity measurement processing to obtain semantic matching data; Based on the semantic matching data, the preset content index data is subjected to candidate extraction processing to obtain the candidate semantic text data.

3. The image and text data retrieval method according to claim 1, characterized in that, The semantic association retrieval processing of the candidate semantic text data based on the preset image semantic index data to obtain associated image data includes: The candidate semantic text data is subjected to semantic association encoding processing to obtain text association vector data; The text association vector data and the preset image semantic index data are subjected to association similarity measurement processing to obtain association matching data; Based on the associated matching data, the preset image semantic index data is subjected to association recall processing to obtain the associated image data.

4. The image and text data retrieval method according to claim 1, characterized in that, The step of integrating the query image and text data, the candidate semantic text data, and the associated image data into image and text evidence to obtain multimodal target retrieval results includes: The query image and text data, the candidate semantic text data, and the associated image data are subjected to evidence alignment processing to obtain aligned evidence data. The aligned evidence data is subjected to evidence fusion processing to obtain fused evidence data; The fused evidence data is processed to generate a response, resulting in the multimodal target retrieval results.

5. The image and text data retrieval method according to claim 1, characterized in that, Before performing cascaded content retrieval processing on the queried image and text data based on preset content index data to obtain candidate semantic text data, the process further includes: The initial document data is parsed and segmented to obtain text fragment data and initial image data; The initial image data is subjected to dual description generation processing to obtain content description data and association description data; The text fragment data, the content description data, and the associated description data are indexed and constructed to obtain the preset content index data and the preset image semantic index data.

6. The image and text data retrieval method according to claim 2, characterized in that, The semantic encoding process performed on the query image and text data to obtain query vector data includes: The query image and text data is processed to identify the query type, resulting in query type data. Based on the query type data, the query image and text data are subjected to modal conversion processing to obtain standardized query image and text data; The standardized query image and text data are semantically mapped to obtain the query vector data.

7. The image and text data retrieval method according to claim 3, characterized in that, The step of performing association similarity measurement processing on the text association vector data and the preset image semantic index data to obtain association matching data includes: The text association vector data is processed by association weight allocation to obtain weighted vector data; The weighted vector data and the preset image semantic index data are subjected to association distance measurement processing to obtain association distance data; The associated distance data is sorted and filtered to obtain the associated matching data.

8. A text and image data retrieval device, characterized in that, include: The first processing module is used to perform cascaded content retrieval processing on the queried image and text data based on preset content index data to obtain candidate semantic text data. The second processing module is used to perform semantic association retrieval processing on the candidate semantic text data based on preset image semantic index data to obtain associated image data. The third processing module is used to integrate the query image and text data, the candidate semantic text data and the associated image data into image and text evidence to obtain multimodal target retrieval results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images