Image-text retrieval generation method and device based on reference semantics and electronic equipment

Abstract

The invention relates to the technical field of intelligent information retrieval, in particular to an image-text retrieval generation method and device based on reference semantics and electronic equipment. The method comprises the steps of performing semantic segmentation on a document in a document knowledge base to obtain a plurality of paragraphs, performing semantic similarity judgment, and merging similar paragraphs; extracting semantic features of the merged paragraphs to form semantic features of lower-layer paragraphs; abstract description is carried out on the merged paragraphs, semantic features are extracted from abstract description, upper layer description semantic features are formed, and text level semantic features are constructed; image semantic features in the image knowledge base are extracted, normalization processing is carried out on the text level semantic features and the image semantic features, an inverted product quantitative index is constructed, and construction of an image-text reference semantic feature index is achieved; and processing the input text based on the image-text reference semantic feature index to obtain a retrieval result. According to the method, the problems of lack of previous knowledge, insufficient detail information and poor image-text relevance in image-text retrieval are solved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent information retrieval, and in particular to a method, device and electronic device for generating image and text retrieval based on reference semantics. Background Art

[0002] With the growth of multimodal data such as images and text, traditional label-based image and text retrieval methods have problems such as inconsistent context information, low image and text relevance, and low relevance of retrieved information. Therefore, based on current advanced artificial intelligence technology, achieving semantic alignment and complementarity of different forms of information is an important research direction of intelligent information retrieval.

[0003] For data in the two modalities of images and text, due to the different distribution spaces, it is impossible to directly measure the similarity between the two. It is necessary to use the current multimodal pre-training model of images and text based on the similarity of reference semantics. However, there is a problem of overall and local mismatch during retrieval, and the method based on the generative model has the problem of local knowledge loss. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for image and text retrieval and generation based on reference semantics, so as to solve the problem of overall and local mismatch in retrieval of multimodal pre-trained models of images and texts based on similar reference semantics in the prior art, and the problem of local knowledge loss in methods based on generative models, so as to achieve efficient retrieval and generation of images and texts.

[0005] In order to achieve the purpose of the present invention, the following technical solutions are adopted:

[0006] The first aspect of the present invention provides a method for generating image-text retrieval based on reference semantics, comprising:

[0007] Semantically segment the documents in the document knowledge base to obtain multiple paragraphs;

[0008] Determining semantic similarity among the multiple paragraphs and merging similar paragraphs;

[0009] Extracting semantic features of the merged paragraphs to form lower-level paragraph semantic features of the document; summarizing the merged paragraphs, extracting semantic features from the summary description to form upper-level description semantic features of the document; wherein the lower-level paragraph semantic features and the upper-level description semantic features are jointly constructed into the text-level semantic features of the document;

[0010] Extracting image semantic features from the image knowledge base, normalizing the text-level semantic features and the image semantic features, and constructing an inverted product quantization index to achieve the construction of an image-text citation semantic feature index;

[0011] The input text is processed based on the semantic feature index of image and text citations to obtain retrieval results.

[0012] A further improvement is that the specific method of semantically segmenting the document in the document knowledge base to obtain multiple paragraphs includes:

[0013] Encode multiple sentences in a document simultaneously, calculate sentence vectors for the encoded sentences, and based on the calculation results of the sentence vectors, evaluate the similarity and correlation between sentences to determine whether it is necessary to segment the sentences as paragraph boundaries.

[0014] A further improvement is that the specific method of determining the semantic similarity of the multiple paragraphs and merging similar paragraphs includes:

[0015] A text encoder of a multimodal pre-trained model of images and texts is used to extract semantic features from the multiple paragraphs, and then the similarity between the paragraphs is calculated using cosine similarity, and similar paragraphs are merged.

[0016] A further improvement is that the specific method of summarizing the merged paragraphs and extracting semantic features from the summary description to form the semantic features of the upper-level description of the document includes:

[0017] The text generation model is used to summarize the merged paragraphs, and the text encoder is used to extract semantic features from the summary description to form the upper-level description semantic features of the document.

[0018] A further improvement is that the normalization process of the text-level semantic features and the image semantic features and the construction of the inverted product quantization index are performed, and the specific method for constructing the image-text citation semantic feature index includes:

[0019] Normalizing the text-level semantic features and the image semantic features;

[0020] Based on the inner product similarity, the KMeans algorithm is used to roughly quantize the normalized semantic features to obtain a specific number of cluster centers, and an inverted index of each semantic feature relative to its cluster center is established;

[0021] Based on the coarse cluster center, the residual between each semantic feature and the cluster center is calculated;

[0022] The residual vector is divided into column dimensions, and then each segment of the residual is clustered separately to obtain the cluster centers corresponding to different column divisions, thereby mapping the dense semantic vector features into vectors represented by cluster identifiers, and realizing the construction of the semantic feature index of image and text citations.

[0023] A further improvement lies in that when initially constructing the text-level semantic features and the image-text reference semantic feature index, they are implemented in a unified offline manner, and then the semantic features are locally updated in an incremental manner. The process uses a vector database to achieve unified storage, indexing, updating and querying of image-text reference semantic features.

[0024] A further improvement is that the specific method of processing the input text based on the semantic feature index of the image-text reference to obtain the search results includes:

[0025] The text encoder based on the image-text multimodal pre-training model encodes the input text to obtain the semantic features of the input text;

[0026] Based on the constructed semantic feature index of image and text citations and combined with the semantic features of the input text, similarity calculation is used to retrieve images and texts, and sorting is performed based on the calculated similarity scores;

[0027] According to the calculated similarity score and the set judgment threshold, the retrieved overall image and text and local image and text are integrated. Among them, for the part with a similarity score lower than the judgment threshold, the text generation model and the image generation model are used for generation processing to form a retrieval result that combines retrieval and generation.

[0028] A second aspect of the present invention provides a device for generating image-text retrieval based on reference semantics, comprising:

[0029] The segmentation module is used to semantically segment the documents in the document knowledge base into multiple paragraphs;

[0030] a merging module, configured to determine semantic similarity of the plurality of paragraphs and merge similar paragraphs;

[0031] a text-level semantic feature generation module, configured to extract semantic features of the merged paragraphs to form lower-level paragraph semantic features of the document, and to perform a summary description of the merged paragraphs, extract semantic features from the summary description, and form upper-level description semantic features of the document; wherein the lower-level paragraph semantic features and the upper-level description semantic features are jointly constructed to form the text-level semantic features of the document;

[0032] A module for constructing an image-text citation semantic feature index is used to extract image semantic features from an image knowledge base, normalize the text-level semantic features and the image semantic features, and construct an inverted product quantization index to implement the construction of an image-text citation semantic feature index.

[0033] The retrieval module is used to process the input text based on the semantic feature index of image and text references to obtain retrieval results.

[0034] The third aspect of the present invention proposes an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a method for generating image and text retrieval based on reference semantics as described in any one of the first aspects.

[0035] The fourth aspect of the present invention proposes a computer-readable storage medium, which includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute a reference semantics-based image and text retrieval generation method as described in any one of the first aspects.

[0036] Beneficial effects of the present invention:

[0037] The present invention constructs an index of text-level semantic features and image-text citation semantic features, combines an image-text multimodal pre-training model with a citation semantic similarity method, and realizes efficient retrieval of images and texts based on input text. At the same time, it combines a text generation model with an image generation model to further improve the relevance of images and texts, thereby obtaining a retrieval result that combines the whole and the part and is unified. The use of quantitative indexing in the process reduces the consumption of storage and computing resources, and solves the problems of lack of contextual knowledge, insufficient detailed information, and weak relevance between images and texts in image-text retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 This is a flow chart of a method for generating image and text retrieval based on reference semantics according to the present invention;

[0040] Figure 2 A schematic diagram of the image and text retrieval process in the present invention;

[0041] Figure 3 A schematic diagram of the process of constructing text-level semantic features in the present invention;

[0042] Figure 4 A schematic diagram of the process of constructing a semantic feature index for image-text citations according to the present invention;

[0043] Figure 5 A schematic diagram of the search result generation process in the present invention;

[0044] Figure 6 This is a structural diagram of a graphic and text retrieval generation device based on reference semantics of the present invention;

[0045] Figure 7 The figure is a schematic diagram of an electronic device according to the present invention. DETAILED DESCRIPTION

[0046] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0047] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0048] Please refer to the attached Figure 1 -Attached Figure 5 In the first aspect of the embodiment of the present invention, a method for image-text retrieval and generation based on reference semantics is proposed to solve the problem of overall and local mismatch in retrieval of multimodal image-text pre-training models based on similar reference semantics in the prior art, as well as the problem of local knowledge loss in methods based on generative models, thereby achieving efficient retrieval and generation of images and texts. Specifically, Figure 1 As shown, the method includes the following steps:

[0049] Step S1: semantically segment the document in the document knowledge base to obtain multiple paragraphs.

[0050] It is understandable that the semantic segmentation model is used to adaptively segment sentences and paragraphs. Semantic segmentation is the process of dividing text content according to semantic relevance. The purpose is to split the document into more detailed and semantically coherent paragraphs to prepare for subsequent processing and analysis.

[0051] Step S2: Determine the semantic similarity of the multiple paragraphs and merge similar paragraphs.

[0052] It is understandable that by comparing the semantic similarities between paragraphs and merging them, it is possible to reduce repeated and redundant information, reduce the storage and computing consumption of semantic features, and improve retrieval relevance.

[0053] Step S3: extracting semantic features of the merged paragraphs to form lower-level paragraph semantic features of the document; performing a summary description on the merged paragraphs, extracting semantic features from the summary description, and forming upper-level description semantic features of the document; wherein the lower-level paragraph semantic features and the upper-level description semantic features are jointly constructed into the text-level semantic features of the document.

[0054] It can be understood that the summary description is a brief description that summarizes and concludes the content of the merged paragraphs. By combining the semantic features at the paragraph level and the semantic features at the overall description level, the semantic information of the document can be more comprehensively captured, making the semantic expression of the document richer and more accurate.

[0055] Step S4: Use the image encoder of the image-text multimodal pre-training model to extract image semantic features in the image knowledge base, normalize the text-level semantic features and the image semantic features, and construct an inverted product quantization index based on inner product measurement and quantization coding to realize the construction of the image-text citation semantic feature index.

[0056] It can be understood that normalization is performed to ensure that they have similar scale and distribution characteristics, which facilitates subsequent similarity calculation. Inverted Product Quantization (IPQ) is an efficient image indexing technology. Inverted product quantization indexing can achieve rapid retrieval of the above semantic features.

[0057] Step S5: Process the input text based on the image-text citation semantic feature index to obtain retrieval results.

[0058] like Figure 2 The figure shows a schematic diagram of the image-text retrieval generation process in the present invention. By comprehensively considering the semantic associations, detailed information and overall description features of the document's internal paragraphs, and combining the semantic features of the image, a powerful image-text retrieval is achieved, improving the accuracy and efficiency of the retrieval.

[0059] In one feasible solution of the present invention, the specific method of semantically segmenting the document in the document knowledge base to obtain multiple paragraphs in step S1 includes:

[0060] Encode multiple sentences in a document simultaneously to model the dependencies between longer contexts, calculate sentence vectors for the encoded sentences, and based on the calculation results of the sentence vectors, evaluate the similarity and correlation between sentences to determine whether sentences need to be segmented as paragraph boundaries.

[0061] More specifically, a word segmenter is first used to segment the sentence, and then Token Embedding, Position Embedding, and Segment Embedding are used to encode the sentence as a representation of the input. The Transformer structure is further used to obtain the encoding of the sentence. Finally, within the maximum allowed backward step size, a binary classifier (such as a logistic regression model) is used to adaptively determine whether each sentence is a paragraph boundary.

[0062] It is understandable that by combining technologies such as word segmentation, embedding, Transformer structure and binary classifier, adaptive segmentation of sentences and paragraphs can be achieved, which improves the understanding and analysis capabilities of document structure, helps to better process the content in the document knowledge base, thereby improving the efficiency and accuracy of document processing and information retrieval. The Transformer structure can effectively capture the semantic information and contextual dependencies in sentences, and helps to represent sentences in a more semantic vector form.

[0063] In one feasible solution of the present invention, the specific method of determining the semantic similarity of the multiple paragraphs and merging similar paragraphs in step S2 includes:

[0064] A text encoder of a multimodal pre-trained model of images and texts is used to extract semantic features from the multiple paragraphs, and then the similarity between the paragraphs is calculated using cosine similarity, and similar paragraphs are merged.

[0065] Among them, the image-text multimodal pre-training model is based on an image-text multimodal dataset. It uses deep learning to learn the associations between different modal data and generate a model with general knowledge transfer capabilities. It belongs to existing technologies such as CLIP (Contrastive Language-Image Pre-training).

[0066] In one feasible solution of the present invention, the specific method of performing summary description on the merged paragraphs and extracting semantic features from the summary description to form the upper-level description semantic features of the document in step S3 includes:

[0067] The text generation model is used to summarize the merged paragraphs, and the text encoder is used to extract semantic features from the summary description to form the upper-level description semantic features of the document.

[0068] It can be understood that by forming the upper-level descriptive semantic features of the document, more contextual information can be provided in the retrieval results.

[0069] like Figure 3Shown is a schematic diagram of the text-level semantic feature construction process of the present invention.

[0070] In one feasible solution of the present invention, the specific method of normalizing the text-level semantic features and the image semantic features and constructing an inverted product quantization index in step S4 to implement the construction of the image-text citation semantic feature index includes:

[0071] Normalization processing is performed on the text-level semantic features and the image semantic features.

[0072] Based on the inner product similarity, the KMeans algorithm is used to coarsely quantize the normalized semantic features to obtain a specific number of cluster centers. These cluster centers represent some important feature points in the data space. An inverted index is established for each semantic feature relative to its cluster center. These inverted indexes can accelerate the search and retrieval process of features.

[0073] In order to reduce the index storage volume and speed up the retrieval process, based on the coarse cluster center, the residual between each semantic feature and the cluster center is calculated. The residual represents the difference between the actual feature and the cluster center.

[0074] The residual vector is divided into column dimensions, and then each segment of the residual is clustered separately to obtain the cluster centers corresponding to different column divisions, thereby mapping the dense semantic vector features into vectors represented by cluster identifiers, and realizing the construction of the semantic feature index of image and text citations.

[0075] It can be understood that the inverted product quantization index uses clustering and residual calculation methods to convert dense semantic features into vectors represented by cluster identifiers, thereby achieving efficient indexing and retrieval of semantic features, improving retrieval efficiency and reducing storage space usage.

[0076] like Figure 4 Shown is a schematic diagram of the process of constructing the semantic feature index of image and text citations of the present invention.

[0077] In a feasible solution of the present invention, when initially constructing the text-level semantic features and the image-text reference semantic feature index, an offline method is used for unified implementation, and then an incremental method is used to locally update the semantic features. The process uses a vector database to achieve unified storage, indexing, updating and querying of the image-text reference semantic features.

[0078] It's easy to understand that vector databases can efficiently store and index large amounts of semantic features related to image and text citations, supporting incremental updates and fast queries. Vector databases can store semantic features of text and images in vector form and provide efficient similarity calculation and retrieval capabilities, providing powerful support for image and text retrieval. By combining offline construction and incremental updates with the storage and indexing advantages of vector databases, unified management and real-time updates of semantic features related to image and text citations are achieved. This design continuously optimizes and improves the image and text retrieval system, enhancing retrieval efficiency and accuracy.

[0079] In one feasible solution of the present invention, the specific method of processing the input text based on the semantic feature index of the image-text reference in step S5 to obtain the search results includes:

[0080] The text encoder based on the image-text multimodal pre-training model encodes the input text to obtain the semantic features of the input text.

[0081] Based on the constructed semantic feature index of image and text citations and combined with the semantic features of the input text, similarity calculation is used to retrieve images and texts, and sorting is performed based on the calculated similarity scores.

[0082] According to the calculated similarity score and the set judgment threshold, images and texts with a similarity score not lower than the threshold will be considered as results that meet the retrieval requirements. Therefore, for the parts with a similarity score not lower than the judgment threshold, the retrieved overall images and texts and local images and texts are integrated. Among them, for the images and text parts with a similarity score lower than the judgment threshold, the text generation model and the image generation model are used for generation processing to form a retrieval result that combines retrieval and generation, that is, the high-scoring parts are integrated with the generated parts to generate the final retrieval results. This can improve the quality and completeness of the retrieval results, allowing users to more accurately obtain image and text content that meets the retrieval requirements.

[0083] like Figure 5 Shown is a schematic diagram of the search result generation process in the present invention.

[0084] The present invention constructs an index of text-level semantic features and image-text citation semantic features, combines an image-text multimodal pre-training model with a citation semantic similarity method, and realizes efficient retrieval of images and texts based on input text. At the same time, it combines a text generation model with an image generation model to further improve the relevance of images and texts, thereby obtaining a retrieval result that combines the whole and the part and is unified. The use of quantitative indexing in the process reduces the consumption of storage and computing resources, and solves the problems of lack of contextual knowledge, insufficient detailed information, and weak relevance between images and texts in image-text retrieval.

[0085] The second aspect of the embodiment of the present invention proposes a graphic and text retrieval generation device based on reference semantics, please refer to the attached Figure 6, is a structural diagram of a graphic and text retrieval generation device based on reference semantics provided in accordance with an embodiment of the present invention, which corresponds to a graphic and text retrieval generation method based on reference semantics provided in the above-mentioned embodiment of the present invention. Since a graphic and text retrieval generation device based on reference semantics provided in an embodiment of the present invention corresponds to a graphic and text retrieval generation method based on reference semantics provided in the above-mentioned embodiment of the present invention, the implementation method of the above-mentioned graphic and text retrieval generation method based on reference semantics is also applicable to a graphic and text retrieval generation device based on reference semantics provided in this embodiment.

[0086] Specifically, if Figure 6 As shown, a graphic and text retrieval generation device based on reference semantics includes:

[0087] The segmentation module 100 is used to perform semantic segmentation on the documents in the document knowledge base to obtain multiple paragraphs.

[0088] The merging module 200 is configured to determine semantic similarity of the multiple paragraphs and merge similar paragraphs.

[0089] The text-level semantic feature generation module 300 is used to extract the semantic features of the merged paragraphs to form the lower-level paragraph semantic features of the document, and to summarize the merged paragraphs, extract semantic features from the summary description, and form the upper-level description semantic features of the document; wherein the lower-level paragraph semantic features and the upper-level description semantic features are jointly constructed to form the text-level semantic features of the document.

[0090] The image-text citation semantic feature index construction module 400 is used to extract image semantic features from the image knowledge base, normalize the text-level semantic features and the image semantic features, and construct an inverted product quantization index to achieve the construction of the image-text citation semantic feature index.

[0091] The retrieval module 500 is used to process the input text based on the semantic feature index of the image-text reference to obtain the retrieval results.

[0092] See also Figure 7 , an embodiment of the present invention also provides an electronic device and a computer-readable storage medium.

[0093] like Figure 7 FIG2 is a schematic diagram of an electronic device provided by one embodiment of the present invention. The electronic device of this embodiment includes: a processor 11, a memory 12, and a computer program stored in the memory and executable by the processor 11. When the processor 11 executes the computer program, the steps of the aforementioned embodiment of a method for generating image-text retrieval based on reference semantics are implemented. Alternatively, when the processor 11 executes the computer program, the functions of the modules / units in the aforementioned device embodiments are implemented.

[0094] Exemplarily, the computer program may be divided into one or more modules / units, which are stored in the memory and executed by the processor 11 to implement the present invention. The one or more modules / units may be a series of computer program instruction segments capable of implementing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device.

[0095] The electronic device may include, but is not limited to, a processor and a memory. Those skilled in the art will appreciate that the schematic diagram is merely an example of an electronic device and does not limit the electronic device. The electronic device may include more or fewer components than shown, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.

[0096] The processor 11 may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc. The processor is the control center of the electronic device and connects various parts of the entire electronic device using various interfaces and lines.

[0097] The memory 12 can be used to store the computer programs and / or modules. The processor implements various functions of the electronic device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory. The memory can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system 121, an application 122 required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created based on the use of the mobile phone (such as audio data, a phone book, etc.). In addition, the memory can include a high-speed random access memory and can also include a non-volatile memory, such as a hard disk, a memory, a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), at least one disk storage device, a flash memory device, or other volatile solid-state storage device.

[0098] Wherein, if the module / unit integrated in the electronic device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the process in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when the computer program is executed by the processor, it can implement the steps of the above-mentioned various method embodiments. Wherein, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form, etc. The computer-readable medium may include: any entity or device that can carry the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc.

[0099] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed across multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment. In addition, in the drawings of the device embodiments provided by the present invention, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines. A person of ordinary skill in the art can understand and implement the present invention without inventive effort.

[0100] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0101] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present application, and should all be included in the scope of protection of the present application.

Claims

1. A method for generating image and text retrieval based on citation semantics, characterized in that: include: Semantically segment the documents in the document knowledge base to obtain multiple paragraphs; Determining semantic similarity among the multiple paragraphs and merging similar paragraphs; Extract the semantic features of the merged paragraphs to form the lower-level paragraph semantic features of the document; Performing summary descriptions on the merged paragraphs, extracting semantic features from the summary descriptions, and forming upper-level description semantic features of the document; wherein the lower-level paragraph semantic features and the upper-level description semantic features are jointly constructed into the text-level semantic features of the document; Extracting image semantic features from the image knowledge base, normalizing the text-level semantic features and the image semantic features, and constructing an inverted product quantization index to achieve the construction of an image-text citation semantic feature index; The input text is processed based on the semantic feature index of image and text citations to obtain retrieval results.

2. The method for generating image-text retrieval based on reference semantics according to claim 1, characterized in that: The specific method of semantically segmenting the document in the document knowledge base to obtain multiple paragraphs includes: Encode multiple sentences in a document simultaneously, calculate sentence vectors for the encoded sentences, and based on the calculation results of the sentence vectors, evaluate the similarity and correlation between sentences to determine whether it is necessary to segment the sentences as paragraph boundaries.

3. The method for generating image-text retrieval based on reference semantics according to claim 1, characterized in that: The specific method of judging the semantic similarity of the multiple paragraphs and merging similar paragraphs includes: A text encoder of a multimodal pre-trained model of images and texts is used to extract semantic features from the multiple paragraphs, and then the similarity between the paragraphs is calculated using cosine similarity, and similar paragraphs are merged.

4. The method for generating image-text retrieval based on reference semantics according to claim 1, characterized in that: The specific method of performing summary description on the merged paragraphs, extracting semantic features from the summary description, and forming the upper-layer description semantic features of the document includes: The text generation model is used to summarize the merged paragraphs, and the text encoder is used to extract semantic features from the summary description to form the upper-level description semantic features of the document.

5. The method for generating image-text retrieval based on reference semantics according to claim 1, characterized in that: The specific method of normalizing the text-level semantic features and the image semantic features and constructing an inverted product quantization index to implement the construction of an image-text citation semantic feature index includes: Normalizing the text-level semantic features and the image semantic features; Based on the inner product similarity, the KMeans algorithm is used to roughly quantize the normalized semantic features to obtain a specific number of cluster centers, and an inverted index of each semantic feature relative to its cluster center is established; Based on the coarse cluster center, the residual between each semantic feature and the cluster center is calculated; The residual vector is divided into column dimensions, and then each segment of the residual is clustered separately to obtain the cluster centers corresponding to different column divisions, thereby mapping the dense semantic vector features into vectors represented by cluster identifiers, and realizing the construction of the semantic feature index of image and text citations.

6. The method for generating image-text retrieval based on reference semantics according to claim 1, characterized in that: When initially constructing the text-level semantic features and the image-text citation semantic feature index, an offline approach is used for unified implementation. Subsequently, an incremental approach is used to locally update the semantic features. The process uses a vector database to achieve unified storage, indexing, updating, and querying of image-text citation semantic features.

7. The method for generating image and text retrieval based on reference semantics according to claim 1, characterized in that: The specific method of processing the input text based on the semantic feature index of the image-text reference to obtain the search results includes: The text encoder based on the image-text multimodal pre-training model encodes the input text to obtain the semantic features of the input text; Based on the constructed semantic feature index of image and text citations and combined with the semantic features of the input text, similarity calculation is used to retrieve images and texts, and sorting is performed based on the calculated similarity scores; According to the calculated similarity score and the set judgment threshold, the retrieved overall image and text and local image and text are integrated. Among them, for the part with a similarity score lower than the judgment threshold, the text generation model and the image generation model are used for generation processing to form a retrieval result that combines retrieval and generation.

8. A graphic and text retrieval generation device based on reference semantics, characterized in that: include: The segmentation module is used to semantically segment the documents in the document knowledge base into multiple paragraphs; a merging module, configured to determine semantic similarity of the plurality of paragraphs and merge similar paragraphs; a text-level semantic feature generation module, configured to extract semantic features of the merged paragraphs to form lower-level paragraph semantic features of the document, and to perform a summary description of the merged paragraphs, extract semantic features from the summary description, and form upper-level description semantic features of the document; wherein the lower-level paragraph semantic features and the upper-level description semantic features are jointly constructed to form the text-level semantic features of the document; A module for constructing an image-text citation semantic feature index is used to extract image semantic features from an image knowledge base, normalize the text-level semantic features and the image semantic features, and construct an inverted product quantization index to implement the construction of an image-text citation semantic feature index. The retrieval module is used to process the input text based on the semantic feature index of image and text references to obtain retrieval results.

9. An electronic device, characterized in that: It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and when the processor executes the computer program, it implements a reference semantics-based image and text retrieval generation method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored computer program, wherein when the computer program is running, the device where the computer-readable storage medium is located is controlled to execute the image-text retrieval generation method based on reference semantics as described in any one of claims 1 to 7.

Images