Construction method and device of image-text search database, database and storage medium

Abstract

The invention relates to an image-text search database construction method and device, a database and a storage medium. The method comprises the following steps: analyzing an obtained document, and extracting to obtain image information and text information of the document; based on the image information and the text information, constructing at least one image-text pair of the literature, the image-text pair comprising an image and a corresponding image annotation text; generating an image feature vector of the image based on the image and the image annotation text; key text information of the literature is extracted from the text information, and a text feature vector of the literature is generated based on the key text information; and based on the image feature vector and the text feature vector of each document, constructing an image-text search database. By adopting the method, literature feature extraction of multi-modal data such as fusion texts and images is realized, deep professional association among literatures is mined, and the accuracy of carrying out image-text search on related literatures in the astronomical field can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of astronomical image and text recommendation, and in particular to a method, device, database and storage medium for constructing an image and text search database. Background Art

[0002] With the rapid development of astronomical research, the number of astronomical scientific research papers has shown explosive growth. According to statistics, the number of papers published by mainstream astronomical journals and preprint platforms each year has exceeded hundreds of thousands, and is increasing at a high rate every year.

[0003] These documents not only contain traditional text content, but also include a large amount of multimodal scientific data such as observational images, spectral data, radio data, etc. For astronomical researchers, how to quickly and accurately find important documents related to their research direction from the massive amount of multimodal documents has become a difficult problem that needs to be solved urgently. Summary of the invention

[0004] Based on this, it is necessary to provide a document feature extraction framework that can integrate multimodal data such as text, images, spectra, etc., and a method, device, database and storage medium for constructing a graphic search database that comprehensively depicts the scientific content of the document to address the above technical problems.

[0005] In a first aspect, the present application provides a method for constructing a picture and text search database, the method comprising: Parsing the acquired documents to extract image information and text information of the documents; Based on the image information and the text information, construct at least one image-text pair of the document, the image-text pair including an image and a corresponding caption text; based on the image and the caption text, generate an image feature vector of the image; Extracting key text information of the document from the text information, and generating a text feature vector of the document based on the key text information; Based on the image feature vector and the text feature vector of each document, a picture-text search database is constructed.

[0006] In one embodiment, generating an image feature vector of the image based on the image and the caption text comprises: Performing image feature recognition on the image to extract a first feature vector of the image; Performing legend feature recognition on the legend text to extract a second feature vector of the legend text; An average value of the first feature vector and the second feature vector is calculated to obtain an average feature vector of the image-text pair; the image feature vector of the image is the average feature vector of the image-text pair.

[0007] In one embodiment, extracting key text information of the document from the text information and generating a text feature vector of the document based on the key text information includes: Set up key text prompt word template; The text information is passed through the key text prompt word template to extract the key text information of the document; Performing structured typesetting on the key text information to generate core information of the document, wherein the core information includes the document subject, document method and document conclusion of the document; The core information of the document is subjected to text feature recognition to generate a text feature vector of the document; the text feature vector includes a topic feature vector, a method feature vector and a conclusion feature vector.

[0008] In one embodiment, constructing a picture-text search database based on the image feature vector and the text feature vector of each document includes: Obtain the image feature vectors and the text feature vectors of all documents, and calculate the cosine similarities between the image feature vectors, between the text feature vectors, and between the image feature vectors and the text feature vectors; Establish knowledge connections for the feature vectors whose cosine similarity is greater than a set threshold, and obtain the inherent relationships of the documents corresponding to the feature vectors to construct a knowledge graph of all documents; the inherent relationships include collaborator relationships, citation relationships, and document subject method conclusion relationships; The image-text search database is generated based on the knowledge graph, the image feature vectors of each of the documents, and the text feature vectors.

[0009] In one embodiment, the image-text search database also includes basic information of each of the documents, and the method further includes: Parsing the acquired documents to extract basic information of the documents; the basic information includes title, author, abstract, keywords, title level, paragraph division and reference list; The extraction accuracy of the basic information of each of the documents is evaluated, a basic index is established for the documents whose extraction accuracy is greater than a set threshold, and the basic information is stored in the image-text search database.

[0010] In a second aspect, the present application also provides a device for constructing a picture and text search database, the device comprising: An information extraction module is used to parse the acquired documents and extract image information and text information of the documents; A multimodal feature extraction module, configured to construct at least one image-text pair of the document based on the image information and the text information, wherein the image-text pair includes an image and a corresponding caption text; generate an image feature vector of the image based on the image and the caption text; extract key text information of the document from the text information, and generate a text feature vector of the document based on the key text information; The multimodal feature fusion module is used to construct a picture-text search database based on the image feature vector and the text feature vector of each document.

[0011] In a third aspect, the present application further provides a picture-text search database, which is constructed by the picture-text search database construction method as described in the first aspect above.

[0012] In a fourth aspect, the present application further provides a search recommendation method for a picture-text search database, which is applied to the picture-text search database as described in the third aspect, and the method comprises: Obtaining a user's query request, wherein the query request includes text or a picture; Performing vectorization processing on the query request to generate a query vector; The query vector is input into the image-text search database for vector search, and a candidate document list is output.

[0013] In a fifth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps corresponding to the method described in the first aspect above are implemented.

[0014] The above-mentioned method, device, database and storage medium for constructing a picture and text search database parse the acquired documents to extract the image information and text information of the documents; construct at least one picture and text pair of the document based on the image information and the text information, the picture and text pair including an image and a corresponding caption text; generate an image feature vector of the image based on the image and the caption text; extract key text information of the document from the text information, and generate a text feature vector of the document based on the key text information; construct a picture and text search database based on the image feature vector and the text feature vector of each document, realize the document feature extraction that integrates multimodal data such as text and image, establish a recommendation mechanism assisted by knowledge graph in the field of astronomy, explore the deep professional associations between documents, and improve the accuracy of picture and text search for related documents in the field of astronomy. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0016] Figure 1 An application environment diagram of a method for constructing a picture-text search database in one embodiment; Figure 2 A schematic diagram of a process for constructing a picture-text search database in one embodiment; Figure 3 A schematic flow chart of the steps of generating the image feature vector of the image in one embodiment; Figure 4 A schematic flow chart of the steps of generating a text feature vector of the document in one embodiment; Figure 5 A schematic diagram of a process for constructing a picture and text search database in one embodiment; Figure 6 A schematic diagram of a flow chart of a search recommendation method for a picture and text search database in one embodiment; Figure 7 It is a structural block diagram of a device for constructing a picture-text search database in one embodiment; Figure 8 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0018] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantity limitation, and may indicate the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0019] Based on the astronomical literature retrieval problem mentioned in the background technology, the inventor found that the existing astronomical literature retrieval system only recommends based on text content, and fails to make full use of multimodal information such as images contained in the literature, resulting in insufficient accuracy in recommending literature related to specific celestial body morphological characteristics, spectral characteristics, etc. In addition, the existing astronomical literature recommendation system uses a single collaborative filtering or content-based recommendation algorithm, which is difficult to accurately capture the professionalism and subject characteristics of astronomical research. Therefore, this application proposes a method for constructing a picture and text search database, which can well solve the above problems.

[0020] The method for constructing a picture-text search database provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the terminal 102 communicates with the server 104 through a network. The data storage system can store data that the server 104 needs to process. The data storage system can be integrated on the server 104, or placed on the cloud or other network servers.

[0021] On the server 104, the acquired document is parsed to extract the image information and text information of the document; based on the image information and the text information, at least one image-text pair of the document is constructed, the image-text pair including an image and a corresponding caption text; based on the image and the caption text, an image feature vector of the image is generated; key text information of the document is extracted from the text information, and based on the key text information, a text feature vector of the document is generated; based on the image feature vectors and the text feature vectors of each of the documents, a picture-text search database is constructed.

[0022] The terminal 102 may be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, etc. The server 104 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0023] In an exemplary embodiment, Figure 2 As shown, a method for constructing a picture-text search database is provided, and the method is applied to Figure 1 The server side in the example is used to illustrate, including the following steps 201 to 204. Among them: Step 201: parse the acquired document to extract image information and text information of the document.

[0024] The documents may be various astronomical journal articles, books, etc.

[0025] Specifically, the document format is converted into markdown format, and the document in markdown format is parsed, including the overall layout recognition and logical structure analysis of the document. By recognizing the overall layout of the document, the single-column and multi-column layout, header and footer, margin information, charts and other regional location information of the document are determined. By parsing the logical structure of the document, the document's title, author, abstract, keywords, title hierarchy, paragraph division, reference list and other structural content information are determined. Graphics and texts are extracted based on the identified and divided regional location information and content structure information to obtain the image information and text information of the document.

[0026] Step 202: construct at least one image-text pair of the document based on the image information and the text information, wherein the image-text pair includes an image and a corresponding caption text; and generate an image feature vector of the image based on the image and the caption text.

[0027] Specifically, with each document as the center, an image-text network of the document is constructed based on the image information and the text information, and the image-text network includes at least one image-text pair of the document.

[0028] A multimodal feature extraction model is constructed and trained, wherein the multimodal feature extraction model includes at least two feature extraction channels, namely, an image channel and a text channel. The image-text pair is input into the multimodal feature extraction model, and the image and the corresponding caption text are respectively input into the image channel and the text channel of the multimodal feature extraction model, and finally an image feature vector of the image is generated.

[0029] Step 203: extract key text information of the document from the text information, and generate a text feature vector of the document based on the key text information.

[0030] Specifically, the text information is divided into blocks according to a preset number of unit words to obtain multiple text modules of the text information, wherein there are overlapping partial words between each adjacent text module to ensure the contextual coherence of the divided text modules. Based on each text module, the key text information of the document is extracted. The key text information is input into the text channel of the multimodal feature extraction model to generate a text feature vector of the document.

[0031] Step 204: constructing a picture-text search database based on the image feature vector and the text feature vector of each document.

[0032] Specifically, since the feature vectors extracted from all documents come from the same multimodal feature extraction model, the image feature vectors and the text feature vectors of all documents are integrated to generate a knowledge graph. The image-text search database includes the knowledge graph and the image feature vectors and text feature vectors of all documents.

[0033] In the above-mentioned method for constructing a picture-text search database, the acquired documents are parsed to extract the image information and text information of the document; based on the image information and the text information, at least one picture-text pair of the document is constructed, and the picture-text pair includes an image and a corresponding caption text; based on the image and the caption text, an image feature vector of the image is generated; the key text information of the document is extracted from the text information, and based on the key text information, a text feature vector of the document is generated; based on the image feature vector and the text feature vector of each document, a picture-text search database is constructed, which realizes the extraction of document features that integrates multimodal data such as text and image, establishes a recommendation mechanism assisted by knowledge graphs in the field of astronomy, explores deep professional connections between documents, and improves the accuracy of picture-text searches for related documents in the field of astronomy.

[0034] More preferably, the image-text search database also includes basic information of each of the documents. When converting the document format into markdown format and parsing the document in markdown format, the basic information of the document needs to be extracted. The basic information includes title, author, abstract, keywords, title level, paragraph division and reference list. Evaluate the extraction accuracy of the basic information of each document, establish a basic index for the documents with extraction accuracy greater than a set threshold, and store the corresponding basic information in the image-text search database, thereby obtaining a basic index set of all documents.

[0035] Among them, taking the evaluation of the extraction accuracy of the basic information of each document as an example, the title extraction accuracy requirement is set to be >95%, the author information extraction accuracy requirement is >90%, and the reference extraction accuracy requirement is >85%. For documents that are below the extraction accuracy requirement or failed to parse, a retry mechanism is adopted and the reason for the retry is marked. For documents that still fail to meet the requirements or cannot be parsed after retry, manual review is adopted.

[0036] Furthermore, when converting the document format into markdown format and parsing the markdown format document, you can also use the Llama-3.1-70B large language model to verify the format integrity of each paragraph, perform context-based fuzzy matching and term correction assisted by professional dictionaries, and feedback the revised content for manual review and optimization, and correct the document in markdown format after parsing.

[0037] In an exemplary embodiment, Figure 3 As shown, step 202 generates an image feature vector of the image based on the image and the caption text, specifically including the following steps 301 to 303. Among them: Step 301: perform image feature recognition on the image to extract a first feature vector of the image.

[0038] Step 302: perform legend feature recognition on the legend text to extract a second feature vector of the legend text.

[0039] Specifically, the image-text pair is input into the multimodal feature extraction model, the image and the corresponding caption text are respectively input into the image channel and text channel of the multimodal feature extraction model, and the first feature vector of the image and the second feature vector of the caption text are output.

[0040] Step 303, calculating the average value of the first feature vector and the second feature vector to obtain the average feature vector of the image-text pair; the image feature vector of the image is the average feature vector of the image-text pair.

[0041] Specifically, the average value of the first feature vector and the second feature vector of each image-text pair is calculated to obtain an average feature vector of the image-text pair, and the average feature vector is used as the image feature vector of the image.

[0042] In this embodiment, because the scientific research literature images are mainly result analysis diagrams, it is extremely difficult to explore their potential meanings without text descriptions. Therefore, by taking the average of the first eigenvector of the image and the second eigenvector of the caption text as the image feature vector of the image, the comprehensiveness of the image feature vector of the astronomical literature image is improved, thereby improving the accuracy of image feature vector search and related literature search using the image and text search database.

[0043] In an exemplary embodiment, Figure 4 As shown, step 203 extracts the key text information of the document from the text information, and generates a text feature vector of the document based on the key text information, which specifically includes the following steps 401 to 404. Among them: Step 401, setting a key text prompt word template.

[0044] Among them, the key text prompt word template includes modules such as research object, core method, key discovery, equipment used, key parameters and important conclusions. The prompt words of the research object include celestial body type, observation target, etc. The prompt words of the core method include observation, theory, numerical simulation, etc. The prompt words of the key discovery include core scientific conclusions, etc. The prompt words of the equipment used include telescope, instrument, etc. The prompt words of the key parameter include redshift, mass, luminosity, etc. The prompt words of the important conclusion include astronomical observation, astronomical theory, etc.

[0045] Step 402: traverse the key text prompt word template for the text information to extract the key text information of the document.

[0046] Specifically, the StarWhisper4-7B large language model is used to traverse the key text prompt word template to extract the key text information of the document.

[0047] Step 403: structure the key text information to generate the core information of the document.

[0048] Specifically, after the key text information of the document is extracted, the key text information is filtered and structured, and finally the core information of the document is generated. The core information includes the document subject, document method and document conclusion of the document.

[0049] Step 404: Perform text feature recognition on the core information of the document to generate a text feature vector of the document.

[0050] Specifically, the core information of the document is input into the text channel of the multimodal feature extraction model to generate a text feature vector of the document, wherein the text feature vector includes a topic feature vector, a method feature vector and a conclusion feature vector.

[0051] In this embodiment, the text feature vector extraction of the document is achieved by performing core information extraction, structured typesetting and vectorization on the text information.

[0052] In an exemplary embodiment, Figure 5 As shown, step 204 constructs a picture-text search database based on the image feature vector and the text feature vector of each document, and specifically includes the following steps 501 to 503. Among them: Step 501 : Obtain the image feature vectors and the text feature vectors of all documents, and calculate the cosine similarity between the image feature vectors, between the text feature vectors, and between the image feature vectors and the text feature vectors.

[0053] Step 502, establish knowledge connections for the feature vectors whose cosine similarity is greater than a set threshold, and simultaneously obtain the inherent relationships of the documents corresponding to the feature vectors to construct a knowledge graph of all documents; the inherent relationships include collaborator relationships, citation relationships, and document subject method conclusion relationships.

[0054] Step 503: Generate the image-text search database based on the knowledge graph, the image feature vector of each document, and the text feature vector.

[0055] In this embodiment, by calculating the cosine similarity between the feature vectors of all documents, knowledge connections are established for the feature vectors whose cosine similarity is greater than a set threshold. At the same time, the original inherent information of the documents is introduced, the deep professional connections between the documents are mined, and a knowledge graph of all documents is constructed. This realizes the construction of a multimodal fusion recommendation framework and improves the knowledge association capability of the image and text search database.

[0056] In an exemplary embodiment, after extracting the key text information of the document from the text information in step 203 and generating the text feature vector of the document based on the key text information, the following contents are also included: Set up a literature timeliness scoring mechanism and introduce a time decay factor: , Among them, μ is the decay weight, and the timeliness weight of each text feature vector is calculated, and the feature update trigger conditions are set, such as checking regular documents once every thirty days, updating documents in hot fields once every seven days, and the number of citations has changed by more than 10, etc., to achieve incremental calculation of feature vectors and maintain feature version control. The updated feature vectors are checked for consistency, including type consistency and relationship integrity, and conflicting feature vectors are resolved, such as retaining the latest information and storing old information and change records in logs for manual verification.

[0057] In one embodiment, a graph-text search database is provided, which is constructed by the graph-text search database construction method described in each of the above embodiments. The graph-text search database includes a knowledge graph, image feature vectors of all documents, text feature vectors, and basic information of each document.

[0058] In one embodiment, Figure 6 As shown, a search recommendation method for a picture and text search database is provided, which is applied to the picture and text search database, and the method includes the following steps 601 to 603. Among them: Step 601: Obtain a user's query request, where the query request includes text or a picture.

[0059] Step 602: vectorize the query request to generate a query vector.

[0060] Step 603: input the query vector into the image-text search database to perform vector search, and output a candidate document list.

[0061] Specifically, the query request input by the user is vectorized using a multimodal feature extraction model to generate a query vector. The query vector is input into the image-text search database, and a vector search is performed using a hierarchical quantitative index and an adaptive search radius adjustment strategy, and a candidate document list is output. The candidate document list is prioritized according to the multimodal feature similarity, knowledge graph path similarity, and document citation structure similarity of each candidate document and the query vector.

[0062] The hierarchical quantitative index specifically includes the following contents: using Kmeans coarse-grained clustering to divide the vector space of the entire image-text search database into n_clusters subspaces, performing fine-grained product quantization in each subspace, establishing a reverse index table, and recording the mapping relationship from feature vector ID to subspace.

[0063] The adaptive search radius adjustment strategy specifically includes the following: dynamically adjust the search radius in the hierarchical quantitative indexed image and text search database according to the distribution characteristics of the query vector. Use an early stopping mechanism to terminate the search when enough candidates are found. During the entire search process, calculate the quality evaluation index of each candidate to dynamically balance the query speed and recall rate.

[0064] A multi-relation aggregation mechanism is designed to perform relationship-specific matrix transformation and attention-based relationship importance calculation, and multi-hop message transmission is performed to achieve priority sorting of the candidate documents. The relationship importance calculation is performed for the searched candidate documents, and the formula is as follows: ; in, Represents the multimodal feature similarity, represents the knowledge graph path similarity, It represents the similarity of literature citation structure, and a, b, c are weight coefficients.

[0065] In one of the embodiments, the search recommendation method of the image and text search database also includes: implementing a cache strategy and performance optimization, and when the number of search accesses to a document is greater than a set access threshold or the number of accesses within a recent preset time period exceeds a preset access threshold, the document is added to the cache.

[0066] In one embodiment, the search recommendation method of the image-text search database also includes: using a large language model to generate a multi-dimensional recommendation explanation of the candidate document. Specifically, it includes the analysis of the correlation between the recommended document and the query, the summary of the innovation and contribution of the recommended document, the prediction of the future development potential of the research direction of the recommended document, etc.; by integrating multi-source information such as user portraits and behavioral data, personalized recommendation adjustments are made, such as adjusting the use density of professional terms according to the user's professional level, adjusting the explanation focus based on the user research stage, and optimizing the explanation style considering the user's historical feedback, etc.; setting the appropriate number of explanatory word units and the score of the explanatory word unit to ensure the diversity of the explanation mode within the controllable quality range.

[0067] In detail, for example, set the number of explanatory tokens top_k to 3. Use the last token "matches" in the document to calculate the similarity score, select the three tokens with the highest scores: ["the", ".", "it"], and the corresponding weights are: [0.91, 0.84, 0.72]. Use this weight to randomly sample the document to obtain a new token "the"; use "the" to calculate the similarity score, select the three tokens with the highest scores: ["topic", "research", "reason"], and the corresponding weights are: [0.89, 0.65, 0.72]. Use this weight to randomly sample the document to obtain a new token "in"; and so on, the final interpretable output sequence of the recommended document is: "The reason for recommending this article is that it matches the topic of astronomicalresearch.".

[0068] The explanatory word score Temperature aims to regulate the similarity score y of the candidate word by controlling the softmax function j , for the similarity score y j Scaling and adjusting the sampling distribution: ; Add parameters The value of will increase the randomness of the output distribution. When , the distribution becomes uniform, and the number of explanatory words is completely random. The value of will reduce the randomness of the output distribution. , which is equivalent to the number of explanatory word units top_k=1.

[0069] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0070] Based on the same inventive concept, the embodiment of the present application also provides a device for constructing a picture-text search database for implementing the method for constructing a picture-text search database involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of one or more picture-text search database construction devices provided below can refer to the limitations of the picture-text search database construction method above, and will not be repeated here.

[0071] In an exemplary embodiment, Figure 7 As shown, a device for constructing a picture-text search database is provided, comprising: an information extraction module 701, a multimodal feature extraction module 702 and a multimodal feature fusion module 703, wherein: The information extraction module 701 is used to parse the acquired document and extract the image information and text information of the document.

[0072] The multimodal feature extraction module 702 is used to construct at least one image-text pair of the document based on the image information and the text information, wherein the image-text pair includes an image and a corresponding caption text; generate an image feature vector of the image based on the image and the caption text; extract key text information of the document from the text information, and generate a text feature vector of the document based on the key text information.

[0073] The multimodal feature fusion module 703 is used to construct a picture-text search database based on the image feature vector and the text feature vector of each document.

[0074] In one embodiment, the multimodal feature extraction module 702 is also used to: perform image feature recognition on the image to extract a first feature vector of the image; perform caption feature recognition on the caption text to extract a second feature vector of the caption text; calculate the average value of the first feature vector and the second feature vector to obtain an average feature vector of the image-text pair; the image feature vector of the image is the average feature vector of the image-text pair.

[0075] In one embodiment, the multimodal feature extraction module 702 is also used to: set a key text prompt word template; traverse the key text prompt word template to extract the key text information of the document; perform structured typeset on the key text information to generate the core information of the document, the core information includes the document subject, document method and document conclusion of the document; perform text feature recognition on the core information of the document to generate a text feature vector of the document; the text feature vector includes a subject feature vector, a method feature vector and a conclusion feature vector.

[0076] In one embodiment, the multimodal feature fusion module 703 is also used to: obtain the image feature vectors and the text feature vectors of all documents, calculate the cosine similarity between the image feature vectors, between the text feature vectors, and between the image feature vectors and the text feature vectors; establish knowledge connections between the feature vectors whose cosine similarity is greater than a set threshold, and at the same time obtain the inherent relationship of the document corresponding to the feature vector to construct a knowledge graph of all documents; the inherent relationship includes collaborator relationship, citation relationship, document subject method conclusion relationship; based on the knowledge graph, the image feature vectors and the text feature vectors of each document, generate the image-text search database.

[0077] In one of the embodiments, the image-text search database also includes basic information of each of the documents, and the information extraction module 701 is also used to: parse the acquired documents to extract basic information of the documents; the basic information includes title, author, abstract, keywords, title level, paragraph division and reference list; evaluate the extraction accuracy of the basic information of each of the documents, establish a basic index for the documents with an extraction accuracy greater than a set threshold, and store the basic information in the image-text search database.

[0078] Each module in the apparatus for constructing the above-mentioned image-text search database can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0079] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 8 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store relevant data of the document. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for constructing a graphic and text search database is implemented.

[0080] Those skilled in the art will understand that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0081] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented: Parsing the acquired documents to extract image information and text information of the documents; Based on the image information and the text information, construct at least one image-text pair of the document, the image-text pair including an image and a corresponding caption text; based on the image and the caption text, generate an image feature vector of the image; Extracting key text information of the document from the text information, and generating a text feature vector of the document based on the key text information; Based on the image feature vector and the text feature vector of each document, a picture-text search database is constructed.

[0082] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: Performing image feature recognition on the image to extract a first feature vector of the image; Performing legend feature recognition on the legend text to extract a second feature vector of the legend text; An average value of the first feature vector and the second feature vector is calculated to obtain an average feature vector of the image-text pair; the image feature vector of the image is the average feature vector of the image-text pair.

[0083] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: Set up key text prompt word template; The text information is passed through the key text prompt word template to extract the key text information of the document; Performing structured typesetting on the key text information to generate core information of the document, wherein the core information includes the document subject, document method and document conclusion of the document; The core information of the document is subjected to text feature recognition to generate a text feature vector of the document; the text feature vector includes a topic feature vector, a method feature vector and a conclusion feature vector.

[0084] In one embodiment, when the processor executes the computer program, the processor further implements the following steps: Obtain the image feature vectors and the text feature vectors of all documents, and calculate the cosine similarities between the image feature vectors, between the text feature vectors, and between the image feature vectors and the text feature vectors; Establish knowledge connections for the feature vectors whose cosine similarity is greater than a set threshold, and obtain the inherent relationships of the documents corresponding to the feature vectors to construct a knowledge graph of all documents; the inherent relationships include collaborator relationships, citation relationships, and document subject method conclusion relationships; The image-text search database is generated based on the knowledge graph, the image feature vectors of each of the documents, and the text feature vectors.

[0085] In one embodiment, when the processor executes the computer program, the following steps are further implemented: parsing the acquired document to extract basic information of the document; the basic information includes title, author, abstract, keywords, title level, paragraph division and reference list; The extraction accuracy of the basic information of each of the documents is evaluated, a basic index is established for the documents whose extraction accuracy is greater than a set threshold, and the basic information is stored in the image-text search database.

[0086] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps corresponding to the methods described in the above embodiments are implemented.

[0087] In one embodiment, a computer program product is provided, including a computer program, which, when executed by a processor, implements the steps corresponding to the methods described in the above embodiments.

[0088] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0089] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0090] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for constructing a picture and text search database, characterized in that: The method comprises: Parsing the acquired documents to extract image information and text information of the documents; Based on the image information and the text information, construct at least one image-text pair of the document, the image-text pair including an image and a corresponding caption text; based on the image and the caption text, generate an image feature vector of the image; Extracting key text information of the document from the text information, and generating a text feature vector of the document based on the key text information; Based on the image feature vector and the text feature vector of each document, a picture-text search database is constructed.

2. The method for constructing a picture-text search database according to claim 1, characterized in that: The step of generating an image feature vector of the image based on the image and the caption text comprises: Performing image feature recognition on the image to extract a first feature vector of the image; Performing legend feature recognition on the legend text to extract a second feature vector of the legend text; An average value of the first feature vector and the second feature vector is calculated to obtain an average feature vector of the image-text pair; the image feature vector of the image is the average feature vector of the image-text pair.

3. The method for constructing a picture-text search database according to claim 1, characterized in that: The step of extracting key text information of the document from the text information and generating a text feature vector of the document based on the key text information includes: Set up key text prompt word template; The text information is passed through the key text prompt word template to extract the key text information of the document; Performing structured typesetting on the key text information to generate core information of the document, wherein the core information includes the document subject, document method and document conclusion of the document; The core information of the document is subjected to text feature recognition to generate a text feature vector of the document; the text feature vector includes a topic feature vector, a method feature vector and a conclusion feature vector.

4. The method for constructing a picture-text search database according to claim 1, characterized in that: The image feature vector and the text feature vector of each document are used to construct a picture-text search database, which includes: Obtain the image feature vectors and the text feature vectors of all documents, and calculate the cosine similarities between the image feature vectors, between the text feature vectors, and between the image feature vectors and the text feature vectors; Establish knowledge connections for the feature vectors whose cosine similarity is greater than a set threshold, and obtain the inherent relationships of the documents corresponding to the feature vectors to construct a knowledge graph of all documents; the inherent relationships include collaborator relationships, citation relationships, and document subject method conclusion relationships; The image-text search database is generated based on the knowledge graph, the image feature vectors of each of the documents, and the text feature vectors.

5. The method for constructing a picture-text search database according to any one of claims 1 to 4, characterized in that: The image-text search database also includes basic information of each of the documents, and the method further includes: Parsing the acquired documents to extract basic information of the documents; the basic information includes title, author, abstract, keywords, title level, paragraph division and reference list; The extraction accuracy of the basic information of each of the documents is evaluated, a basic index is established for the documents whose extraction accuracy is greater than a set threshold, and the basic information is stored in the image-text search database.

6. A device for constructing a picture and text search database, characterized in that: The device comprises: An information extraction module is used to parse the acquired documents and extract image information and text information of the documents; A multimodal feature extraction module, configured to construct at least one image-text pair of the document based on the image information and the text information, wherein the image-text pair includes an image and a corresponding caption text; generate an image feature vector of the image based on the image and the caption text; extract key text information of the document from the text information, and generate a text feature vector of the document based on the key text information; The multimodal feature fusion module is used to construct a picture-text search database based on the image feature vector and the text feature vector of each document.

7. A picture and text search database, characterized in that: The image-text search database is constructed by the image-text search database construction method according to any one of claims 1 to 5.

8. A search recommendation method for a picture and text search database, characterized in that: Applied to the image-text search database as claimed in claim 7, the method comprises: Obtaining a user's query request, wherein the query request includes text or a picture; Performing vectorization processing on the query request to generate a query vector; The query vector is input into the image-text search database for vector search, and a candidate document list is output.

9. The search recommendation method for a picture and text search database according to claim 8, characterized in that: The candidate document list is prioritized according to the multimodal feature similarity, knowledge graph path similarity and document citation structure similarity between each candidate document and the query vector.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images