Information retrieval method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an information retrieval method and device, electronic equipment and a storage medium, and relates to the technical field of data processing.The method comprises the steps that obtained intention description information is input into a preset intention recognition model, and an intention map is obtained; performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation; performing modal feature extraction processing on the intention graph to obtain graph structure feature representation; inputting the modal feature representation and the graph structure feature representation into a pre-trained multi-modal fusion model to obtain a multi-modal semantic representation; generating a query statement through a preset large language model according to the multi-mode semantic representation and mode information of a database to be queried; and retrieving the data in the database according to the query statement, and outputting a retrieval result. According to the embodiment of the invention, the accuracy of an information retrieval result and the use experience of a user can be improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an information retrieval method and device, an electronic device and a storage medium. Background Art

[0002] In today's digital age, information technology is booming, the amount of data in the database is showing an explosive growth trend, and the data types are also rapidly evolving in a diversified direction. In daily information retrieval application scenarios, it is not difficult for us to detect the difficulties. For the majority of non-technical users, the traditional database query method mainly relies on structured query languages, such as SQL. This has a high threshold for non-technical users, making it difficult for many people to efficiently obtain the required data. At the same time, as time goes by, the scale of data continues to expand, and the data types it covers are becoming more and more diverse. Multimodal data, such as text, images, audio and video, account for an increasing proportion in the database. Under this circumstance, users' query needs become more and more complex. Existing query systems have obvious deficiencies when dealing with these complex query needs, especially when facing multimodal data. It has limitations in understanding user intentions and cannot quickly and accurately locate the information required by users, resulting in users often spending a lot of time and energy to obtain the data they want, and sometimes even failing to obtain accurate information. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to propose an information retrieval method and device, an electronic device and a storage medium, aiming to improve the accuracy of information retrieval results and the user experience.

[0004] To achieve the above object, a first aspect of an embodiment of the present application proposes an information retrieval method, the method comprising:

[0005] Input the acquired intent description information into a preset intent recognition model to obtain an intent graph;

[0006] Performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation;

[0007] Performing modal feature extraction processing on the intention graph to obtain a graph structure feature representation;

[0008] Inputting the modal feature representation and the graph structure feature representation into a pre-trained multimodal fusion model to obtain a multimodal semantic representation;

[0009] Generate a query statement through a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried;

[0010] The data in the database is searched according to the query statement, and the search results are output.

[0011] In some embodiments, inputting the acquired intent description information into a preset intent recognition model to obtain an intent graph includes:

[0012] The intention description information is parsed by the intention recognition model to obtain entity information, grammatical structure and semantic combination;

[0013] According to the entity information, the grammatical structure and the semantic combination, a core entity, relationship information and condition information are obtained;

[0014] Based on the core entity, the relationship information and the condition information, an intent graph is constructed.

[0015] In some embodiments, the intention description information is text intention information, or a combination of text intention information and image intention information; the modal feature representation includes text feature representation and image feature representation;

[0016] The performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation includes:

[0017] Performing modal feature extraction processing on the text intention information through a preset language representation model to obtain text feature representation;

[0018] The image intention information is subjected to modal feature extraction processing by a preset convolutional neural network to obtain an image feature representation.

[0019] In some embodiments, the step of inputting the modal feature representation and the graph structure feature representation into a preset multimodal fusion model to obtain a multimodal semantic representation includes:

[0020] Performing cross-modal feature fusion processing on the text feature representation and the image feature representation to obtain a first multimodal representation;

[0021] Performing cross-modal feature fusion processing on the text feature representation and the graph structure feature representation to obtain a second multimodal representation;

[0022] A multimodal semantic representation is obtained according to the first multimodal representation and the second multimodal representation.

[0023] In some embodiments, the schema information includes field information, table information, and constraint information; and generating a query statement through a preset large language model according to the multimodal semantic representation and the schema information of the database to be queried includes:

[0024] generating a first query clause through the large language model according to the multimodal semantic representation and the field information;

[0025] generating a second query clause through the large language model according to the multimodal semantic representation and the table information;

[0026] generating a third query clause through the large language model according to the multimodal semantic representation and the constraint information;

[0027] The first query clause, the second query clause and the third query clause are integrated and processed by the large language model to obtain a query statement.

[0028] In some embodiments, after the step of generating a query statement by using a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried, the method further includes:

[0029] Get adjustment request;

[0030] Adjusting the intent map according to the adjustment request;

[0031] According to the adjusted intent graph, the query statement is updated through the large language model.

[0032] In some embodiments, searching the data in the database according to the query statement and outputting the search results includes:

[0033] Retrieve the data in the database according to the query statement and output preliminary results;

[0034] According to the preliminary results and the schema information of the database, filtering out erroneous sub-results from the preliminary results;

[0035] The erroneous sub-result is deleted from the preliminary result to obtain the search result.

[0036] In some embodiments, the multimodal fusion model is trained according to the following steps:

[0037] Obtaining sample description information and expected results corresponding to the sample description information;

[0038] Inputting the sample description information into a preset intent recognition model to obtain a sample map;

[0039] Performing modal feature extraction processing on the sample description information to obtain at least one training modal feature representation;

[0040] Performing modal feature extraction processing on the sample graph to obtain a training graph structure feature representation;

[0041] Inputting the training modality feature representation and the training graph structure feature representation into the multimodal fusion model to obtain a training multimodal semantic representation;

[0042] Generate a training query statement through a preset large language model according to the training multimodal semantic representation and the mode information of the database to be queried;

[0043] Outputting a prediction result according to the training query statement and the database;

[0044] According to the predicted result and the expected result, the parameters of the multimodal fusion model are adjusted until the loss value between the predicted result and the expected result meets a preset threshold.

[0045] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an information retrieval device, the device comprising:

[0046] The first generation module is used to input the acquired intention description information into a preset intention recognition model to obtain an intention map;

[0047] A first extraction module, configured to perform modal feature extraction processing on the intention description information to obtain at least one modal feature representation;

[0048] A second extraction module is used to perform modal feature extraction processing on the intention graph to obtain a graph structure feature representation;

[0049] A fusion module, used for inputting the modal feature representation and the graph structure feature representation into a preset multimodal fusion model to obtain a multimodal semantic representation;

[0050] A second generation module is used to generate a query statement through a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried;

[0051] The output module is used to output the search results according to the query statement and the database.

[0052] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0053] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0054] The information retrieval method and device, electronic device and storage medium proposed in the present application obtain the user's intention description information, and then input the intention description information into a preset intention recognition model. This model can parse the user's intention from the intention description information and output the intention map. The original intention description information is subjected to modal feature extraction processing. If the intention description contains text and possibly implied image information, multiple modal feature representations can be extracted. For example, for the text part, features such as keywords and semantic relationships can be extracted; for implicit image information, relevant modal features such as color and shape can be extracted. At the same time, the generated intention map is also subjected to modal feature extraction processing to obtain a graph structure feature representation. The graph structure feature representation can capture the features contained in each node and edge in the intention map. The modal feature representation extracted from the intention description information and the graph structure feature representation extracted from the intention map are input into the pre-trained multimodal fusion model together, and the features of different modalities are effectively fused to output a multimodal semantic representation. Subsequently, based on the multimodal semantic representation and the schema information of the database to be queried, a query statement is generated using a preset large language model to generate a query statement that conforms to the database query syntax and accurately expresses the user's intention. Finally, the data in the database is retrieved based on this query statement, and the retrieval results are output. For non-technical users, users only need to express their intentions in natural language or simple descriptions. The system automatically generates query statements through a series of complex models and processing steps, which improves the user experience of the entire information retrieval system. In addition, the information retrieval method can extract modal features of intent description information and intent maps, as well as multimodal fusion processing. Whether it is a query requirement involving multiple types of data such as text and images, it can be accurately processed to improve the accuracy of the retrieval results. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a flow chart of the information retrieval method provided in an embodiment of the present application;

[0056] Figure 2 yes Figure 1 Flow chart of step S110 in FIG.

[0057] Figure 3 yes Figure 1 Flow chart of step S120 in FIG.

[0058] Figure 4 yes Figure 1 Flow chart of step S140 in FIG.

[0059] Figure 5 yes Figure 1 Flow chart of step S150 in FIG.

[0060] Figure 6 yes Figure 1 The flowchart after step S150 in FIG.

[0061] Figure 7 yes Figure 1 Flow chart of step S160 in FIG.

[0062] Figure 8 yes Figure 1 Flowchart of the training method of the multimodal fusion model in;

[0063] Fig. 9 is a schematic diagram of the structure of an information retrieval device provided in an embodiment of the present application;

[0064] Fig.10 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] First, some nouns involved in this application are analyzed:

[0069] SQL (Structured Query Language): is a special-purpose programming language, a database query and programming language used to access data and query, update, and manage relational database systems. Structured Query Language is a high-level non-procedural programming language that allows users to work on high-level data structures. It does not require users to specify how to store data, nor does it require users to understand the specific data storage method, so different database systems with completely different underlying structures can use the same structured query language as an interface for data input and management. Structured query language statements can be nested, which makes it extremely flexible and powerful.

[0070] Large Language Model (LLM): A large language model is a deep learning model trained on massive text data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks such as text summarization, question answering, translation, etc.

[0071] BERT model (Bidirectional Encoder Representations from Transformers): is a pre-trained language model based on the Transformer architecture. It captures deep bidirectional representations of language by pre-training on large-scale text data, and then fine-tunes for different natural language processing tasks. The BERT model has a wide range of applications in the field of natural language processing and provides a powerful pre-training model foundation for a variety of language understanding tasks.

[0072] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0073] Information Extraction: A text processing technology that extracts specified types of entity, relationship, event and other factual information from natural language text and forms structured data output. Information extraction is a technology that extracts specific information from text data. Text data is composed of some specific units, such as sentences, paragraphs, and chapters. Text information is composed of some small specific units, such as characters, words, phrases, sentences, paragraphs, or a combination of these specific units. Extracting noun phrases, names, place names, etc. from text data are all text information extraction. Of course, the information extracted by text information extraction technology can be various types of information.

[0074] Image caption generates a natural language description for an image and uses the generated description to help applications understand the semantics expressed in the image visual scene. For example, image captioning can convert image retrieval into text retrieval, which is used to classify images and improve image retrieval results. People usually only need to take a quick glance to describe the details of the image visual scene, while automatically adding descriptions to images is a comprehensive and arduous computer vision task that requires converting the complex information contained in the image into a natural language description. Compared with ordinary computer vision tasks, image captioning not only requires identifying objects from images, but also requires associating the identified objects with natural semantics and describing them in natural language. Therefore, image description requires people to extract deep features of images, associate them with semantic features, and convert them for generating descriptions.

[0075] In today's digital age, information technology is booming, the amount of data in the database is showing an explosive growth trend, and the data types are also rapidly evolving in a diversified direction. In daily information retrieval application scenarios, it is not difficult for us to detect the difficulties. For the majority of non-technical users, the traditional database query method mainly relies on structured query languages, such as SQL. This has a high threshold for non-technical users, making it difficult for many people to efficiently obtain the required data. At the same time, as time goes by, the scale of data continues to expand, and the data types it covers are becoming more and more diverse. Multimodal data, such as text, images, audio and video, account for an increasing proportion in the database. Under this circumstance, users' query needs become more and more complex. Existing query systems have obvious deficiencies when dealing with these complex query needs, especially when facing multimodal data. It has limitations in understanding user intentions and cannot quickly and accurately locate the information required by users, resulting in users often spending a lot of time and energy to obtain the data they want, and sometimes even failing to obtain accurate information.

[0076] Based on this, the embodiments of the present application provide an information retrieval method and device, an electronic device and a storage medium, aiming to improve the accuracy of information retrieval results and the user experience.

[0077] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0078] The information retrieval method provided in the embodiment of the present application relates to the field of artificial intelligence technology. The information retrieval method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the information retrieval method, etc., but is not limited to the above forms.

[0079] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0080] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with the relevant laws, regulations, and standards of the relevant countries and regions. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0081] Figure 1 is an optional flowchart of the information retrieval method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S110 to S160.

[0082] Step S110, input the acquired intent description information into a preset intent recognition model to obtain an intent graph.

[0083] Step S120, performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation.

[0084] Step S130, performing modal feature extraction processing on the intent graph to obtain a graph structure feature representation.

[0085] Step S140, inputting the modal feature representation and the graph structure feature representation into a pre-trained multimodal fusion model to obtain a multimodal semantic representation.

[0086] Step S150, generating a query statement through a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried.

[0087] Step S160, searching the data in the database according to the query statement and outputting the search result.

[0088] In the steps S110 to S160 shown in the embodiment of the present application, the user's intention description information is obtained, and then the intention description information is input into a preset intention recognition model. This model can parse the user's intention from the intention description information and output an intention map. The original intention description information is subjected to modal feature extraction processing. If the intention description contains text and possibly implied image information, a variety of modal feature representations can be extracted. For example, for the text part, features such as keywords and semantic relationships can be extracted; for implicit image information, relevant modal features such as color and shape can be extracted. At the same time, the generated intention map is also subjected to modal feature extraction processing to obtain a graph structure feature representation, which can capture the features contained in each node and edge in the intention map through the graph structure feature representation. The modal feature representation extracted from the intention description information and the graph structure feature representation extracted from the intention map are input into the pre-trained multimodal fusion model together, and the features of different modalities are effectively fused to output a multimodal semantic representation. Subsequently, based on the multimodal semantic representation and the schema information of the database to be queried, a query statement is generated using a preset large language model to generate a query statement that conforms to the database query syntax and accurately expresses the user's intention. Finally, the data in the database is retrieved based on this query statement, and the retrieval results are output. For non-technical users, users only need to express their intentions in natural language or simple descriptions. The system automatically generates query statements through a series of complex models and processing steps, which improves the user experience of the entire information retrieval system. In addition, the information retrieval method can extract modal features of intent description information and intent maps, as well as multimodal fusion processing. Whether it is a query requirement involving multiple types of data such as text and images, it can be accurately processed to improve the accuracy of the retrieval results.

[0089] In step S110, the most direct way is to receive user input through a user interface (UI) to obtain intent description information. This can be a text box in which users enter their intent descriptions, or draw sketches of intent descriptions, or a speech recognition system in which users can enter their intents by voice. In addition, for users who are not familiar with the operating system, a question-and-answer system can be designed to guide users to describe their intentions through a series of questions. Intent description information can also be extracted from existing text materials such as documents, emails, chat records, etc. The present application does not specifically limit the method and type of obtaining intent description information.

[0090] In step S150, the schema information of the database refers to the logical structure design of the database, which is the data storage structure information in the database, an abstract expression of the organization and relationship of the data in the database, which defines the basic structure of the database and plans the overall architecture of the database, including how to store data, the relationship between data, etc. In one embodiment, in a database about an e-commerce platform, the data storage structure information corresponding to its schema information may include multiple main tables for storing different types of data, such as user tables, product tables, order tables, evaluation tables, etc. There are various associations between these tables, which together constitute the basic framework of the data storage of this platform.

[0091] In step S110 to step S160, for example, a user wants to plan a seaside vacation trip and hopes to find a tourist attraction with a beautiful beach, suitable for surfing, and seafood restaurants around and with good reviews. After the user enters such intention description information, the system inputs it into the preset intention recognition model. The intention map generated after model analysis may contain the following nodes and relationships: "tourism type-seaside vacation", "attraction features-beautiful beach", "activity-surfing", "surrounding facilities-seafood restaurant", "review-good", and these nodes are interconnected by different edges, which clearly show the structure of the user's intention. Modal feature extraction is performed on the intention description information entered by the user. For the text part, keywords such as "seaside vacation", "beautiful beach", "surfing", "seafood restaurant", and "good review" are extracted as modal feature representations. At the same time, these keywords may imply some visual-related modal features, such as "beautiful beach" may involve image-related features such as the color (gold, white, etc.) and shape (wide, crescent, etc.) of the beach, and the shape of the waves; "seafood restaurant" may involve features such as the appearance of the restaurant and the appearance of seafood dishes. Modal feature extraction processing is performed on the intention map to obtain a graph structure feature representation. For example, the correlation strength between the node "tourism type - seaside vacation" and the node "attraction features - beautiful beach" is high, which is reflected in the large weight of the corresponding edge in the graph structure feature representation, indicating that the two elements are closely related in the user's intention. The modal feature representation extracted from the intention description information and the graph structure feature representation extracted from the intention map are input into the pre-trained multimodal fusion model. After the model fuses these features, a multimodal semantic representation is obtained. This multimodal semantic representation integrates the various needs of users for tourist attractions, including the content of text descriptions and potential visual related requirements. According to the multimodal semantic representation and the mode information of the tourism information database, the preset large language model generates a query statement, and then searches the database based on this query statement. The output search results may be several qualified seaside tourist attractions. Each attraction contains relevant information such as attraction introduction text, beach photos, photos of tourists surfing, photos and reviews of surrounding seafood restaurants, etc., which meets the user's information needs for planning this seaside vacation trip.

[0092] See also Figure 2 It is understandable that step S110 may include but is not limited to steps S210 to S230:

[0093] Step S210: parsing the intention description information through the intention recognition model to obtain entity information, grammatical structure and semantic combination;

[0094] Step S220: According to the entity information, grammatical structure and semantic combination, core entity, relationship information and condition information are obtained;

[0095] Step S230: Construct an intent graph based on core entities, relationship information and condition information.

[0096] In step S210, after obtaining the user's intention description information, it is input into the preset intention recognition model, and the model first parses the information. For example, for the user input "find information about a beach bathing beach with beach volleyball and jet ski projects, it is required that there are seafood restaurants around the bathing beach and the reviews are good". The model will identify the entity information therein, such as "bathing beach", "beach volleyball", "jet ski", "seafood restaurant", "evaluation", etc.; in terms of grammatical structure, the model can analyze the subject, predicate, object and other structures of the sentence, such as "find... information" is the main action, and "there are... projects" is a modification of "bathing beach"; in terms of semantic combination, the model understands that "beach volleyball and jet ski projects" are the characteristics of "bathing beach", and "there are seafood restaurants around and the reviews are good" is a limiting condition for "bathing beach".

[0097] In step S220, core entities, relationship information and conditional information are further extracted based on the parsed entity information, grammatical structure and semantic combination. In the above example, the core entity is "bathing beach", and the relationship information includes the inclusion relationship between "bathing beach" and "beach volleyball" and "jet ski", as well as the location relationship between "bathing beach" and "seafood restaurant"; the conditional information is "good evaluation". These relationship information and conditional information are developed around the core entity, clarifying how different entities are related to each other and the limitation of the core entity.

[0098] In step S230, an intent graph is constructed based on the extracted core entities, relationship information, and condition information. In the graph, "bathing beach" is the central node, "beach volleyball" and "jet ski" are sub-nodes directly connected to "bathing beach", and the edges represent the inclusion relationship between them; "seafood restaurant" is another related node, connected to "bathing beach" through an edge representing the position relationship; at the same time, there is a label representing "good evaluation" associated with "bathing beach", forming an intent graph that can clearly show the user's intent structure.

[0099] In steps S210 to S230, different users may express the same demand in a variety of ways. By analyzing the intent description information in detail and deeply exploring the entities, grammar, and semantic content therein, it is possible to more accurately grasp the user's true intention, thereby accurately extracting key information to construct an intent graph, providing clear guidance for subsequent steps such as modal feature extraction, multimodal fusion, and query statement generation.

[0100] Specifically, use dependency parsing DependencyParse(T) to extract the grammatical structure, use named entity recognition NER(T) to identify entity V, and use semantic rules SemanticRules(S) for semantic combination. Where T represents the user's natural language input, and the system generates the corresponding intent graph through parsing. Identify the core entities, relationships, and conditions in the query and build an intent graph. The node V represents the core entity in the user's query, the edge E represents the relationship information between entities, and C is the additional conditional information. The generation formula of the intent graph G is as follows:

[0101] G=IntentGraphBuilder(V,E,C).

[0102] See also Figure 3 It can be understood that the intention description information is text intention information, a combination of text intention information and image intention information; the modal feature representation includes text feature representation and image feature representation. In step S120, the following steps may be included but not limited to:

[0103] Step S310: performing modal feature extraction processing on the text intention information through a preset language representation model to obtain text feature representation;

[0104] Step S320: Perform modal feature extraction processing on the image intent information through a preset convolutional neural network to obtain an image feature representation.

[0105] It should be noted that the intent description information can be text intent information, or a combination of text intent information and image intent information. For example, a user may enter a text description "looking for tourist attractions with ancient buildings and beautiful gardens" and upload a picture that may be a tourist attraction of similar style as supplementary information.

[0106] In step S310, a preset language representation model is used to extract modal features from the text intent information. This language representation model can be a natural language processing model based on deep learning, such as using BERT or a similar language model to extract text feature representation from the text intent information:

[0107] H query =BERT(M query );

[0108] Among them, M query is the textual intention information, H query is the extracted text feature representation.

[0109] Taking "find tourist attractions with ancient buildings and beautiful gardens" as an example, the model will process this text through word segmentation and vectorization, and extract keywords such as "ancient buildings", "beautiful gardens", "tourist attractions", etc. as text feature representation. At the same time, it may also extract more advanced features such as semantic relationships and emotional tendencies to better understand user intentions.

[0110] In step S320, a preset convolutional neural network is used to perform modal feature extraction processing on the image intent information. Convolutional neural networks have powerful capabilities in image recognition and feature extraction, such as using a convolutional neural network (CNN) to extract image feature representation from image intent information:

[0111] H image =CNN(M image );

[0112] Among them, M image is the input image intent information, H image is the extracted image feature representation.

[0113] If a user uploads a picture of a tourist attraction, the network will extract the image's color features, texture features, shape features, etc. For example, if the image contains a lot of green and floral patterns, features related to "garden" may be extracted; if there are unique building outlines, features associated with "ancient buildings" may be extracted.

[0114] Similarly, in step S130, the node features and relationship information of the intent graph can be extracted by using a graph convolutional network (GCN) to form a graph structure feature representation.

[0115] H graph =GCN(G);

[0116] Among them, G is the intent graph, H graph It is a graph structure feature representation extracted from the intent graph. GCN can capture the dependency and pattern information between nodes in the intent graph.

[0117] See also Figure 4 It is understandable that step S140 may include but is not limited to the following steps:

[0118] Step S410: performing cross-modal feature fusion processing on the text feature representation and the image feature representation to obtain a first multimodal representation;

[0119] Step S420: performing cross-modal feature fusion processing on the text feature representation and the graph structure feature representation to obtain a second multimodal representation;

[0120] Step S430: Obtain a multimodal semantic representation according to the first multimodal representation and the second multimodal representation.

[0121] In steps S410 to S430, the cross-modal attention mechanism is used to dynamically weight the features of different modalities, and to capture the interdependencies between the modalities to ensure that the information of each modality can be correctly aligned and interacted with the information of other modalities. The core of the cross-modal attention mechanism is the query-key-value structure, which adjusts the flow of information based on the similarity between the text intent information (or other modalities) and other modalities. For the three modalities (text intent information, images, and intent graphs), we can use two cross-modal attention mechanisms to process the fusion of text intent information and images, as well as text intent information and graph structures respectively. The formula of the attention mechanism is as follows:

[0122]

[0123] QK T represents the similarity between the query vector Q and the key vector K, is a scaling factor to prevent the value from being too large, and the softmax operation is used to normalize the attention weights. In this example, the text feature representation is used as the query vector Q, Q = H query , the image feature representation and the graph structure feature representation are used as the key vector K and the value vector V. The value vector V is weighted and summed through the attention mechanism to generate a fused representation. This representation will contain the interaction information between the text intent information and other modalities. Through the cross-modal attention mechanism, the features of the text intent information will interact with the image features to generate an image representation based on the text intent information. It can also interact with the nodes and edges in the intent graph through the cross-modal attention mechanism to generate structured query semantics.

[0124] F image-query =Attention(H query ,H image ,H image )

[0125] F graph-query =Attention(H query ,H graph ,H graph )

[0126] F final =F image-query +F graph-query

[0127] Among them, F image-query is the first multimodal representation, F graph-query is the second multimodal representation, F finalIt is a multimodal semantic representation. This representation contains the multi-level relationship between text intent information, image intent information and intent graph, which can fully express the user's query intention and provide semantic support for subsequent large language model understanding and SQL generation. Through the cross-modal attention mechanism, the system can dynamically adjust the fusion process of text intent information and other modalities (image intent information and intent graph). The cross-modal attention mechanism can not only capture the dependencies between modalities, but also weight multimodal information in a shared representation space, thereby generating accurate multimodal semantic representations for subsequent large language model understanding and SQL generation.

[0128] In step S410, after obtaining the text feature representation and the image feature representation, they are input into the multimodal fusion model for cross-modal feature fusion processing. For the keywords "ancient building" and "beautiful garden" in the text feature representation, and the features such as the shape of the ancient building, the color and layout of the garden extracted from the image feature representation, the model calculates the correlation and mapping relationship between them. The method based on the attention mechanism is used to align and fuse the text and image features at the semantic level. For example, the attention mechanism will focus on the word "ancient building" in the text, find the corresponding building outline features in the image, and fuse them into a new feature representation, which is part of the first multimodal representation. Through this fusion processing of all text and image features, a complete first multimodal representation is obtained.

[0129] In step S420, at the same time, the text feature representation and the graph structure feature representation are also input into the multimodal fusion model for cross-modal fusion. Taking the search for tourist attractions as an example, if the text feature representation contains keywords such as "ancient buildings" and "beautiful gardens", the graph structure feature representation contains "tourist attractions" as the core entity and the relationship information (such as inclusion relationship, dependency relationship, etc.) between "ancient buildings" and "beautiful gardens". The model fuses the text features and graph structure features by analyzing this information. For example, according to the importance of the keywords in the text and their position and relationship in the graph structure, the two are fused through the fusion layer in the neural network. If "ancient buildings" are strongly associated with the core entity "tourist attractions" in the graph structure, the model will give it corresponding weights during fusion to form a second multimodal representation.

[0130] In step S430, a multimodal semantic representation is further generated based on the obtained first multimodal representation and the second multimodal representation. Specifically, in this step, the features in the first multimodal representation and the second multimodal representation may be comprehensively considered, common key information may be extracted, and integrated according to certain rules to form a comprehensive and integrated multimodal semantic representation, which completely covers the semantic information contained in the text, image and graph structure, and more accurately reflects the user's intention.

[0131] See also Figure 5 It is understandable that the schema information of the database includes field information, table information and constraint information. In step S150, the following steps may be included but not limited to:

[0132] Step S510: Generate a first query clause through a large language model according to the multimodal semantic representation and field information;

[0133] Step S520: Generate a second query clause through a large language model according to the multimodal semantic representation and the table information;

[0134] Step S530: Generate a third query clause through a large language model according to the multimodal semantic representation and constraint information;

[0135] Step S540: integrating the first query clause, the second query clause and the third query clause through a large language model to obtain a query statement.

[0136] In step S510, the first query clause is a SELECT clause, which is used to specify the fields to be retrieved from the database. The large language model combines the query intent and database schema information to ensure that the selected fields exist in the table structure of the database. Through multimodal fusion features, the model generates a SELECT clause that matches the database fields. For example, when querying the user's name and age, the SELECT clause generated by the model is: "SELECTname, age". By combining the multimodal semantic representation with the database field information to generate a select clause, it can be ensured that the fields returned in the query results are exactly what the user cares about. This precise match avoids returning too many irrelevant fields, reduces the amount of data transmission and the complexity of subsequent processing, while ensuring that users can directly obtain useful information and improves the availability of information.

[0137] In step S520, the second query clause is a FROM clause, which is used to specify the database table to be accessed in the SQL query. The large language model selects an appropriate table structure based on the database schema information to ensure that the FROM clause generated by the model can retrieve data from the correct database table. For example, when retrieving data from the user information table users, the FROM clause generated by the model is: "FROMusers". The from clause is generated based on the table information of the query determined according to the multimodal semantic representation, which can effectively narrow the query scope. In a complex database structure, data is only obtained from related tables, avoiding unnecessary traversal of the entire database, greatly improving query efficiency. Especially when the database contains a large number of different types of data tables, this targeted query can significantly reduce the query time and resource consumption.

[0138] In step S530, the third query clause is a WHERE clause, which is used to specify the filter conditions of the query and filter the records that meet the conditions. The large language model generates a WHERE clause based on the user's query conditions and in combination with the constraint rules of the fields in the database to ensure that the conditional filtering is consistent with the field restrictions of the database model. For example: If the user wants to query users who are older than 30 years old, the WHERE clause generated by the model is: "WHERE age>30". The where clause is generated using constraint information and multimodal semantic representation to strictly filter the query results. This can exclude data that does not meet user requirements and improve the accuracy of the query results. Whether it is based on field value range restrictions, keyword matching or other forms of constraints, it can ensure that the returned results are highly consistent with user expectations, reduce the workload of users to further filter and identify results, and improve user experience.

[0139] In step S540, finally, the large language model integrates the generated select, from, and where clauses, thereby improving the quality of the query statement as a whole, enabling the information retrieval system to more accurately and efficiently meet the query needs of users in complex multimodal scenarios, and enhancing the practicality and reliability of the system.

[0140] See also Figure 6 It is understandable that after step S150, the following steps may also be included but not limited to:

[0141] Step S610: obtaining an adjustment request;

[0142] Step S620: adjusting the intent map according to the adjustment request;

[0143] Step S630: Update the query statement through the large language model according to the adjusted intent graph.

[0144] In step S610 to step S630, after the user receives the initial search results, he or she may be dissatisfied with the results or wish to further optimize the query. The user can make an adjustment request through the interactive interface provided by the system. For example, the user may feel that the search results contain some less relevant content, or wish to add specific filtering conditions. The adjustment request may be a description in text form. The query statement is updated through a large language model based on the obtained adjustment request. By obtaining the adjustment request and updating the query statement, the system can promptly meet the dynamic needs of the user during the search process, so that the search results are more in line with the user's actual expectations and improve the user experience.

[0145] Specifically, the reinforcement learning framework Q-Learning can be used to dynamically optimize the intent graph, with the optimization goal of maximizing user satisfaction. The optimization formula is:

[0146]

[0147] Among them, Q(s,a) is the value of performing action a in state s, α is the learning rate, r is the reward value of feedback, and γ is the discount factor.

[0148] See also Figure 7 It is understood that, in step S160, the following steps may be included but not limited to:

[0149] Step S710: Output preliminary results based on the query statement in the database;

[0150] Step S720: Filter out erroneous sub-results from the preliminary results according to the preliminary results and the schema information of the database;

[0151] Step S730: Delete the erroneous sub-results from the preliminary results to obtain the search results.

[0152] In step S710 to step S730, based on rule analysis and semantic matching technology, the calibration module identifies potential errors in the query and corrects them. For example, if an erroneous table or field appears in the query, the system will automatically correct it through semantic similarity measurement. By screening and deleting erroneous sub-results, it can ensure that the output search results are more accurate and reliable, so that users will not be misled by erroneous data, improve the quality of information retrieval, and enhance the user experience.

[0153] See also Figure 8 , it can be understood that the multimodal fusion model can be trained according to the following steps:

[0154] Step S810: Obtain sample description information and expected results corresponding to the sample description information;

[0155] Step S820: inputting the sample description information into a preset intent recognition model to obtain a sample map;

[0156] Step S830: performing modal feature extraction processing on the sample description information to obtain at least one training modal feature representation;

[0157] Step S840: performing modal feature extraction processing on the sample graph to obtain a training graph structure feature representation;

[0158] Step S850: inputting the training modality feature representation and the training graph structure feature representation into the multimodal fusion model to obtain the training multimodal semantic representation;

[0159] Step S860: Generate a training query statement through a preset large language model according to the training multimodal semantic representation and the mode information of the database to be queried;

[0160] Step S870: output prediction results according to the training query statement and the database;

[0161] Step S880: According to the prediction result and the expected result, the parameters of the multimodal fusion model are adjusted until the loss value between the prediction result and the expected result meets the preset threshold.

[0162] In step S810 to step S880, sample description information is first obtained. These samples can be various actual user query descriptions, covering different fields and different types of needs. The sample description information is input into the preset intent recognition model to generate a sample map. This process is similar to generating an intent map in an actual retrieval method. By analyzing the entity information, grammatical structure and semantic combination of the sample description information, core entities, relationship information and condition information are obtained, and then the sample map is constructed. Subsequently, the sample description information is subjected to modal feature extraction processing to obtain at least one training modal feature representation. If the sample description information contains text and image information, the text feature representation and the image feature representation are extracted by a preset language representation model and a convolutional neural network respectively. Then, the sample map is subjected to modal feature extraction processing to obtain a training graph structure feature representation. The training modal feature representation and the training graph structure feature representation are input into a multimodal fusion model to obtain a training multimodal semantic representation. This fusion process is a model learning the association and complementary information between different modal features to better understand the meaning of the sample description. According to the training multimodal semantic representation and the pattern information of the database to be queried, a training query statement is generated by a preset large language model. This step is similar to the generation of query statements in actual retrieval. The language generation ability and semantic understanding of the large language model are used to generate query statements in combination with database schema information. According to the training query statement, the database is searched and the prediction result is output. This prediction result is the retrieval result generated by the system based on the current model parameters. The prediction result is compared with the expected result, and the loss value between the two is calculated. According to the loss value, the optimization algorithm is used to adjust the parameters of the multimodal fusion model so that the model updates the parameters in the direction of reducing the loss value. Repeat the above process and continuously adjust the model parameters until the loss value between the prediction result and the expected result meets the preset threshold. This threshold is usually set according to actual needs, indicating that the prediction result of the model is close enough to the expected result and the model training has achieved good results. By using a large amount of sample data for training, the parameters of the multimodal fusion model can be continuously optimized to better understand user intent and integrate multimodal information. The trained model can generate query statements more accurately in actual applications, improving the accuracy and comprehensiveness of retrieval results.

[0163] See also Fig. 9The present application also provides an information retrieval device 900, which can implement the above-mentioned information retrieval method, and the device includes:

[0164] The first generation module 910 is used to input the acquired intention description information into a preset intention recognition model to obtain an intention map;

[0165] A first extraction module 920 is used to perform modal feature extraction processing on the intention description information to obtain at least one modal feature representation;

[0166] The second extraction module 930 is used to perform modal feature extraction processing on the intent graph to obtain a graph structure feature representation;

[0167] A fusion module 940 is used to input the modal feature representation and the graph structure feature representation into a preset multimodal fusion model to obtain a multimodal semantic representation;

[0168] A second generating module 950 is used to generate a query statement through a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried;

[0169] The output module 960 is used to output the search results according to the query statement and the database.

[0170] The specific implementation of the information retrieval device is substantially the same as the specific implementation of the above-mentioned information retrieval method, and will not be described in detail here.

[0171] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above information retrieval method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0172] See also Fig.10 , Fig.10 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0173] The processor 1001 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0174] The memory 1002 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1002 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1002, and the processor 1001 calls and executes the information retrieval method of the embodiment of this application;

[0175] Input / output interface 1003, used to implement information input and output;

[0176] The communication interface 1004 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0177] A bus 1005 , which transmits information between various components of the device (e.g., the processor 1001 , the memory 1002 , the input / output interface 1003 , and the communication interface 1004 );

[0178] The processor 1001 , the memory 1002 , the input / output interface 1003 and the communication interface 1004 are connected to each other in communication within the device via the bus 1005 .

[0179] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned information retrieval method is implemented.

[0180] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0181] The information retrieval method, device, electronic device and storage medium provided in the embodiments of the present application obtain the user's intention description information, and then input the intention description information into a preset intention recognition model. This model can parse the user's intention from the intention description information and output an intention map. The original intention description information is subjected to modal feature extraction processing. If the intention description contains text and possibly implied image information, multiple modal feature representations can be extracted. For example, for the text part, features such as keywords and semantic relationships can be extracted; for implicit image information, relevant modal features such as color and shape can be extracted. At the same time, the generated intention map is also subjected to modal feature extraction processing to obtain a graph structure feature representation, which can capture the features contained in each node and edge in the intention map. The modal feature representation extracted from the intention description information and the graph structure feature representation extracted from the intention map are input into the pre-trained multimodal fusion model together, and the features of different modalities are effectively fused to output a multimodal semantic representation. Subsequently, based on the multimodal semantic representation and the schema information of the database to be queried, a query statement is generated using a preset large language model to generate a query statement that conforms to the database query syntax and accurately expresses the user's intention. Finally, the database is searched based on this query statement and the search results are output. For non-technical users, users only need to express their intentions in natural language or simple descriptions. The system automatically generates query statements through a series of complex models and processing steps, which improves the user experience of the entire information retrieval system. In addition, the information retrieval method can extract modal features of intent description information and intent maps, as well as multimodal fusion processing. Whether it is a query requirement involving multiple types of data such as text and images, it can be accurately processed to improve the accuracy of the retrieval results.

[0182] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0183] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0184] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0185] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0186] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0187] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0188] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0189] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0190] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0191] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0192] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An information retrieval method, characterized in that: The method comprises: Input the acquired intent description information into a preset intent recognition model to obtain an intent graph; Performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation; Performing modal feature extraction processing on the intention graph to obtain a graph structure feature representation; Inputting the modal feature representation and the graph structure feature representation into a pre-trained multimodal fusion model to obtain a multimodal semantic representation; Generate a query statement through a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried; The data in the database is searched according to the query statement, and the search results are output.

2. The information retrieval method according to claim 1, characterized in that: The step of inputting the acquired intention description information into a preset intention recognition model to obtain an intention graph includes: The intention description information is parsed by the intention recognition model to obtain entity information, grammatical structure and semantic combination; According to the entity information, the grammatical structure and the semantic combination, a core entity, relationship information and condition information are obtained; Based on the core entity, the relationship information and the condition information, an intent graph is constructed.

3. The information retrieval method according to claim 1, characterized in that: The intention description information is text intention information, a combination of text intention information and image intention information; the modal feature representation includes text feature representation and image feature representation; The performing modal feature extraction processing on the intention description information to obtain at least one modal feature representation includes: Performing modal feature extraction processing on the text intention information through a preset language representation model to obtain text feature representation; The image intention information is subjected to modal feature extraction processing by a preset convolutional neural network to obtain an image feature representation.

4. The information retrieval method according to claim 3, characterized in that: The step of inputting the modal feature representation and the graph structure feature representation into a preset multimodal fusion model to obtain a multimodal semantic representation includes: Performing cross-modal feature fusion processing on the text feature representation and the image feature representation to obtain a first multimodal representation; Performing cross-modal feature fusion processing on the text feature representation and the graph structure feature representation to obtain a second multimodal representation; A multimodal semantic representation is obtained according to the first multimodal representation and the second multimodal representation.

5. The information retrieval method according to claim 1, characterized in that: The schema information includes field information, table information and constraint information; the query statement is generated by a preset large language model according to the multimodal semantic representation and the schema information of the database to be queried, including: generating a first query clause through the large language model according to the multimodal semantic representation and the field information; generating a second query clause through the large language model according to the multimodal semantic representation and the table information; generating a third query clause through the large language model according to the multimodal semantic representation and the constraint information; The first query clause, the second query clause and the third query clause are integrated and processed by the large language model to obtain a query statement.

6. The information retrieval method according to claim 1, characterized in that: After the step of generating a query statement by using a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried, the method further includes: Get adjustment request; Adjusting the intent map according to the adjustment request; According to the adjusted intent graph, the query statement is updated through the large language model.

7. The information retrieval method according to claim 1, characterized in that: The step of searching the data in the database according to the query statement and outputting the search result comprises: Retrieve the data in the database according to the query statement and output preliminary results; According to the preliminary results and the schema information of the database, filtering out erroneous sub-results from the preliminary results; The erroneous sub-result is deleted from the preliminary result to obtain the search result.

8. The information retrieval method according to claim 1, characterized in that: The multimodal fusion model is trained according to the following steps: Obtaining sample description information and expected results corresponding to the sample description information; Inputting the sample description information into a preset intent recognition model to obtain a sample map; Performing modal feature extraction processing on the sample description information to obtain at least one training modal feature representation; Performing modal feature extraction processing on the sample graph to obtain a training graph structure feature representation; Inputting the training modality feature representation and the training graph structure feature representation into the multimodal fusion model to obtain a training multimodal semantic representation; Generate a training query statement through a preset large language model according to the training multimodal semantic representation and the mode information of the database to be queried; Outputting a prediction result according to the training query statement and the database; According to the predicted result and the expected result, the parameters of the multimodal fusion model are adjusted until the loss value between the predicted result and the expected result meets a preset threshold.

9. An information retrieval device, characterized in that: include: The first generation module is used to input the acquired intention description information into a preset intention recognition model to obtain an intention map; A first extraction module, configured to perform modal feature extraction processing on the intention description information to obtain at least one modal feature representation; A second extraction module is used to perform modal feature extraction processing on the intention graph to obtain a graph structure feature representation; A fusion module, used for inputting the modal feature representation and the graph structure feature representation into a preset multimodal fusion model to obtain a multimodal semantic representation; A second generation module is used to generate a query statement through a preset large language model according to the multimodal semantic representation and the mode information of the database to be queried; The output module is used to output the search results according to the query statement and the database.

10. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the information retrieval method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the information retrieval method according to any one of claims 1 to 8 is implemented.

Images