Hybrid retrieval method, device and equipment based on industrial large model and medium

Abstract

The invention relates to a hybrid retrieval method and device based on an industrial large model, equipment and a medium. The method comprises the following steps: acquiring a query text and a query image input by a user; respectively encoding the query text and the query image to obtain a query text vector and a query image vector; respectively searching a target text vector and a first target image vector related to the query text vector and the query image vector from a vector database, and generating and displaying a first retrieval result to the user in combination with the target text vector and the first target image vector, a plurality of text vectors and a plurality of image vectors are stored in the vector database. According to the method, the user not only can directly express the retrieval intention by using the characters, but also can adopt the images and the characters to indicate the retrieval requirements of the user, so that the situation that the user cannot adopt accurate languages and characters to describe the content to be retrieved when describing the requirements of the user is avoided.

Description

Technical Field

[0001] The present application belongs to the field of intelligent retrieval technology, and specifically relates to a hybrid retrieval method, device, equipment and medium based on industrial large models. Background Art

[0002] With the widespread application of large-model artificial intelligence technology, retrieval-augmented generation (RAG) technology is becoming increasingly popular. It has been successfully applied to various scenarios, including question-answering systems, chatbots, document processing, and information retrieval.

[0003] RAG technology combines the advantages of information retrieval and language generation models, and can retrieve relevant information from massive data to enrich and optimize the output of the language model, thereby improving the accuracy of answers and the comprehensiveness of content.

[0004] However, when users describe their search needs, in certain specific scenarios they are unable to accurately use language and words to describe the content they want to search for, that is, the user's intention description is inaccurate, which will cause RAG technology to inaccurately understand the user's intention, and the retrieved results will not match the user's actual search needs. Summary of the invention

[0005] The present application provides a hybrid retrieval method, device, equipment and medium based on an industrial big model, which is used to solve the problem in the prior art that when the user cannot accurately use text to describe the retrieval requirements, the retrieval results do not match the user's actual needs.

[0006] In a first aspect, the present application provides a hybrid retrieval method based on an industrial big model, comprising:

[0007] Obtain the query text and query image input by the user;

[0008] Encoding the query text and the query image respectively to obtain a query text vector and a query image vector;

[0009] A target text vector and a first target image vector related to the query text vector and the query image vector are searched for respectively from a vector database, and a first retrieval result is generated and displayed to the user in combination with the target text vector and the first target image vector, wherein the vector database stores a plurality of text vectors and a plurality of image vectors.

[0010] In a possible design, after combining the target text vector and the first target image vector to generate and present the first search result to the user, the method further includes:

[0011] Acquire a local query image manually selected by the user on the query image to obtain a feature image;

[0012] Performing encoding processing on the feature image to obtain a feature image vector;

[0013] A second target image vector related to the feature image vector is searched from the vector database, and a second search result is generated and displayed to the user in combination with the target text vector, the first target image vector and the second target image vector.

[0014] In a possible design, obtaining the local query image manually selected by the user on the query image to obtain the feature image includes:

[0015] Based on the first search result, automatically perform partial frame selection on the query image;

[0016] Determine whether the automatic local box selection result is consistent with the manual local box selection result of the user;

[0017] If they are consistent, the manual local box selection result of the user on the query image is used as the feature image;

[0018] If they are inconsistent, the manual local box selection result of the user on the query image is deleted, and the automatic local box selection result is used as the feature image.

[0019] In a possible design, before encoding the feature image, the method further includes:

[0020] Removing the color of the query image and converting the query image with the color removed into a black and white line image to obtain a processed query image;

[0021] Recording the diagonal coordinate information of the feature image on the query image;

[0022] Based on the diagonal coordinate information, a processed feature image is cut out from the processed query image.

[0023] In a possible design, before encoding the feature image, the method further includes:

[0024] The color of the characteristic image is removed, and the characteristic image with the color removed is converted into a black and white line image to obtain a processed characteristic image.

[0025] In a possible design, the background color of the black-and-white line image is white, and the line color is black.

[0026] In a possible design, before obtaining the query text and query image input by the user, the method further includes:

[0027] Crawling network data, wherein the network data includes text data and image data;

[0028] Encoding the text data and the image data respectively to obtain a text vector and an image vector;

[0029] storing the text vector and the image vector in the vector database respectively;

[0030] Before encoding the text data and the image data respectively, the method further includes:

[0031] The color of the image data is removed, and the image data after the color is removed is converted into black and white line image data to obtain processed image data.

[0032] In a second aspect, the present application provides a hybrid retrieval device based on an industrial large model, comprising:

[0033] An acquisition module, used to acquire the query text and query image input by the user;

[0034] An encoding module, used to encode the query text and the query image respectively to obtain a query text vector and a query image vector;

[0035] A processing module is used to search for a target text vector and a first target image vector related to the query text vector and the query image vector from a vector database, respectively, and generate and display a first retrieval result to the user in combination with the target text vector and the first target image vector, wherein the vector database stores a plurality of text vectors and a plurality of image vectors.

[0036] In a third aspect, the present application provides a hybrid retrieval device based on an industrial large model, comprising: a processor, and a memory in communication connection with the processor;

[0037] The memory stores computer-executable instructions;

[0038] The processor executes the computer-executable instructions stored in the memory to implement the hybrid retrieval method based on industrial large models as described in the first aspect.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions, and when the computer execution instructions are executed by a computer, the hybrid retrieval method based on the industrial big model as described in the first aspect.

[0040] The hybrid retrieval method, device, equipment and medium based on industrial large models provided in the present application obtain query text and query image input by a user; encode the query text and the query image respectively to obtain a query text vector and a query image vector; search for a target text vector and a first target image vector related to the query text vector and the query image vector respectively from a vector database, and generate and display a first retrieval result to the user in combination with the target text vector and the first target image vector, wherein the vector database stores a plurality of text vectors and a plurality of image vectors.

[0041] In the above method, by obtaining the query text and query image input by the user, images are used with the aid of language to clearly and accurately express the user's search needs, and then the query text and query image are encoded to obtain query text vectors and query image vectors, and then the target text vectors and target image vectors related to the above query text vectors and query image vectors are searched from the vector database respectively, and the search results are generated and displayed to the user accordingly to meet the user's search needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] Figure 1 A schematic diagram of an application process provided for an embodiment of the present application;

[0044] Figure 2.1 A hybrid retrieval method based on an industrial large model provided in the embodiment of the present application Figure 1 ;

[0045] Figure 2.2 for Figure 2.1 A schematic diagram of a step;

[0046] Figure 2.3 for Figure 2.1 Another step diagram of

[0047] Figure 3 Flow chart 2 of a hybrid retrieval method based on an industrial large model provided in an embodiment of the present application;

[0048] Figure 4.1 A hybrid retrieval method based on an industrial large model provided in the embodiment of the present application Figure 3 ;

[0049] Figure 4.2 for Figure 4.1 Schematic diagram of the steps;

[0050] Figure 5.1Upload the schematic diagram of the robot arm to be operated for the user;

[0051] Figure 5.2 Schematic diagram of the effect of returning images for the first retrieval result;

[0052] Figure 5.3 A schematic diagram of the selection result of a user performing a partial selection on a query image;

[0053] Figure 5.4 The following is a schematic diagram of the effect of a black and white line image obtained after color information is removed from the query image, where (a) is the feature image used for retrieval, (b) is the local detail image of the corresponding position of the illustration image returned by the first retrieval, and (c) is the schematic diagram of the retrieved robotic arm;

[0054] Figure 5.5 A schematic diagram of the effect of feature extraction on the selected feature image, where (a) is the robotic arm image uploaded by the user, (b) is the robotic arm image obtained after removing the color information, and (c) is the black and white line image of the robotic arm obtained after color inversion processing;

[0055] Figure 5.6 Another schematic diagram of the effect of feature extraction on the selected feature image, where (a) is the selected feature image with color information, (b) is the feature image obtained after the color information is removed, and (c) is the black and white line image obtained after the feature image is color-inverted;

[0056] Figure 6 A schematic diagram of the structure of a hybrid retrieval device based on an industrial large model provided by an embodiment of the present invention;

[0057] Figure 7 A hardware schematic diagram of a hybrid retrieval device based on an industrial large model provided in an embodiment of the present invention.

[0058] Through the above-mentioned drawings, clear embodiments of the present application have been shown, which will be described in more detail later. These drawings and textual descriptions are not intended to limit the scope of the concept of the present application in any way, but to illustrate the concept of the present application for those skilled in the art by reference to specific embodiments. The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application. DETAILED DESCRIPTION

[0059] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0060] The terms "first", "second", "third", "fourth", etc. (if any) in the description and claims of the present invention and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in sequences other than those illustrated or described herein, for example.

[0061] In the embodiments of the present application, the words "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0062] First, the terms involved in this application are explained:

[0063] RAG is a natural language processing technology that combines information retrieval and text generation. It assists the generation model in answering questions by retrieving information related to the input question from a large external database, greatly improving the accuracy and relevance of text generation.

[0064] The working principle of RAG technology mainly includes three steps:

[0065] Indexing

[0066] Retrieval

[0067] Generation

[0068] First, the documents in the external knowledge base are divided into blocks, encoded into vector form, and stored in the vector database. When the user enters a query, the system will also encode the query into a vector representation and retrieve the top-k chunks most relevant to the query from the vector database. These chunks will serve as additional contextual information when generating text. Finally, the original query and the retrieved data chunks are input into the pre-trained Transformer model together, and the model combines this information to generate the final answer or text. Vectorization of data is a general technology known in the art and will not be elaborated in this application.

[0069] The advantages of RAG technology are improved accuracy, alleviation of hallucination problems, fast knowledge update, enhanced traceability, flexibility and scalability. It has a wide range of applications in question-answering systems, dialogue generation, text summarization, content generation and other fields. With the continuous advancement of technology, RAG technology will play an increasingly important role in many fields and become an important force in promoting the development of natural language processing.

[0070] Large Language Model (LLM) refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text. Large language models can handle a variety of natural language tasks, such as text classification, question answering, and dialogue, and are an important path to artificial intelligence. Currently, large language models use a Transformer architecture and pre-training objectives (such as Language Modeling) similar to small models. The difference from small models is that they increase model size, training data, and computing resources.

[0071] The principle of the large language model is based on deep learning, which uses a large amount of data and computing resources to train a neural network model with a large number of parameters. By continuously adjusting the model parameters, the model can achieve the best performance in various tasks.

[0072] With the widespread application of large-model artificial intelligence technology, retrieval enhancement generation technology has gradually attracted the attention of more and more users. They are widely used in various scenarios such as question-answering systems, chatbots, document processing and information retrieval.

[0073] However, when users are unable to accurately use language to describe their search needs in certain specific scenarios, that is, the user's intention description is inaccurate, the above technology may not be able to accurately understand the user's intention, which will lead to a very low match between the search results and the user's actual search needs.

[0074] Therefore, the present application proposes a retrieval method based on a mixture of text vectors and image vectors. The method mainly obtains the query text and query image input by the user at the same time when the user searches, further explains the query text through the query image, and improves the accuracy of the retrieval system's understanding of the user's language and text. The query text and query image are then encoded separately to obtain the query text vector and query image vector, and then the target text vector and target image vector related to the query text vector and query image vector are searched for respectively from a vector database storing multiple text vectors and multiple image vectors, and the target text vector and the first target image vector are combined to answer the user's question. This answer can also make full use of the above-mentioned query image, for example, highlighting the part related to the retrieval result on the query image.

[0075] Combine the following Figure 1 , the application scenarios of this application are described.

[0076] Figure 1 A schematic diagram of an application process provided by an embodiment of the present application. Figure 1 As shown in the figure, the process includes three main steps: data preparation, data retrieval and LLM generation. Among them, data preparation mainly involves network data crawling and data vectorization processing of the previous vector database; data retrieval mainly involves obtaining the query text and query image input by the user, vectorizing the query text and query image, and querying related vectors from the vector database; LLM generation mainly involves the generation of retrieval results, and the retrieval results are obtained based on the vectors related to the above query text and query image.

[0077] (1) Data preparation stage:

[0078] Crawling network data, the network data includes text data and image data; encoding the text data and image data respectively to obtain text vectors and image vectors; storing the text vectors and image vectors in a vector database respectively.

[0079] Considering that color has a significant impact on images, which will interfere with subsequent retrieval results and reduce retrieval accuracy, the color of images in network data can be removed when establishing a vector database to obtain images with RGB information removed. For example, the color of image data can be removed and the color-removed image data can be converted into black and white line image data to obtain processed image data.

[0080] (2) Data retrieval stage:

[0081] The query text and query image input by the user are obtained; the query text and the query image are encoded respectively to obtain a query text vector and a query image vector; and a target text vector and a target image vector related to the query text vector and the query image vector are searched respectively from a vector database.

[0082] (3) LLM generation stage:

[0083] The target text vector and the target image vector are combined with the large language model to generate search results and display the search results to the user. The large language model generates search results for natural language text based on the understanding of the target text vector and the target image vector, which is not the content that this application intends to explore and will not be elaborated here.

[0084] Based on the mutual coordination between the above processes, the user's search intention can be presented in a more intuitive way. At the same time, the retrieval system's understanding of pure language and text can be improved through images, ensuring that the retrieval system can accurately understand and match these precise needs, thereby improving the relevance of the retrieval results.

[0085] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments may exist independently or in combination with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0086] Figure 2.1 A hybrid retrieval method based on an industrial large model provided in the embodiment of the present application Figure 1 .like Figure 2.1 As shown, the method includes:

[0087] S201: Acquire the query text and query image input by the user.

[0088] In the above scheme, the query text uses language to express the user's search needs. However, in some scenarios, the user may know what his needs are, but cannot express them accurately in language. This makes the pure language expression unsatisfactory. At this time, images are needed to assist in expressing the user's search needs. Figure 2.2 As shown in , users can directly enter the query text and query image at the beginning to express their needs; on this basis, Figure 2.3 As shown, the user can also make a local selection on the query image to further express the specific location of the image that he wants to know. At this time, the query image and the local image obtained by the local selection can be combined for subsequent processing operations, or only the local image can be processed to simplify the image retrieval work.

[0089] S202: Encode the query text and the query image respectively to obtain a query text vector and a query image vector.

[0090] In this step, the data can be vectorized to compress the storage space. Existing retrieval systems generally store basic data after vectorization and compression. Therefore, after obtaining the user's query text and query image, vectorization processing is required to facilitate subsequent search operations.

[0091] S203. Search for a target text vector and a first target image vector related to the query text vector and the query image vector respectively from a vector database, and generate and display a first retrieval result to a user by combining the target text vector and the first target image vector, wherein a plurality of text vectors and a plurality of image vectors are stored in the vector database.

[0092] In the above scheme, a large number of basic text vectors and basic image vectors are stored in the vector database. After the user's query text and query image are vectorized, a matching search can be performed from the vector database to find the text vectors and image vectors related to the above query text and query image, and then the retrieval results are generated based on the above text vectors and image vectors, and the retrieval results are fed back to the user.

[0093] In an embodiment of the present application, by obtaining the query text and query image input by the user, images are used with the aid of language to clearly and accurately express the user's search needs, and then the query text and query image are encoded to obtain a query text vector and a query image vector, and then the target text vector and target image vector related to the above query text vector and query image vector are respectively searched from the vector database, and the search results are generated and displayed to the user accordingly to meet the user's search needs.

[0094] Figure 3 Flowchart 2 of a hybrid retrieval method based on an industrial large model provided in an embodiment of the present application. Figure 3 As shown, based on the previous embodiment, this embodiment further includes performing a color culling operation on the query image. Specifically, the method includes:

[0095] S301: Acquire the query text and query image input by the user.

[0096] S302 , removing the color of the query image, and converting the query image after the color removal into a black and white line image to obtain a processed query image.

[0097] In the above scheme, the aforementioned embodiment mentioned that color information has a great impact on the image. In the prior art, when searching, if the colors of two images are inconsistent, even if the image features such as texture, structure and shape are highly similar, the two images are likely to be considered unrelated. Therefore, before vectorizing the query image, the RGB color information of the query image can be removed to obtain a query image composed only of lines, thereby improving the retrieval accuracy.

[0098] In order to form a sharp contrast effect, illustratively, the black and white line image may have a white background and black lines.

[0099] S303 , respectively encode the query text and the query image to obtain a query text vector and a query image vector.

[0100] S304. Search for a target text vector and a first target image vector related to the query text vector and the query image vector respectively from the vector database, and generate and display a first search result to the user by combining the target text vector and the first target image vector, wherein the vector database stores a plurality of text vectors and a plurality of image vectors.

[0101] For the same steps as those in the previous embodiment, please refer to the description of the previous embodiment, which will not be repeated here.

[0102] In the embodiment of the present application, before the query image is vectorized, the color information of the query image is eliminated so that the query image is converted into a black and white line image to avoid the interference of color on the retrieval results.

[0103] Figure 4.1 A hybrid retrieval method based on an industrial large model provided in the embodiment of the present application Figure 3 .like Figure 4.1 As shown, the method includes:

[0104] S401: Acquire the query text and query image input by the user.

[0105] S402: Encode the query text and the query image respectively to obtain a query text vector and a query image vector.

[0106] S403. Search for a target text vector and a first target image vector related to the query text vector and the query image vector respectively from a vector database, and generate and display a first search result to the user by combining the target text vector and the first target image vector, wherein a plurality of text vectors and a plurality of image vectors are stored in the vector database.

[0107] S404: Obtain a local query image manually selected by the user on the query image to obtain a feature image.

[0108] In this step, if Figure 4.2 As shown, when the user obtains the first search result, he may need to perform further detailed search. At this time, the user can manually select a frame on the query image to obtain a feature image, which is a local image of the query image.

[0109] The user's understanding of the first search result may not be correct, so the manual selection operation performed by the user on the query image may be based on a wrong understanding. In order to shorten the search time and effectively improve the search efficiency, the retrieval system can actively perform local selection based on the first search result, and judge whether the user's manual selection result is correct or not based on this, and correct the user's selection operation when the user has a wrong understanding. For example, based on the first search result, the query image can be automatically locally selected; it is determined whether the automatic local selection result is consistent with the user's manual local selection result; if they are consistent, the user's manual local selection result on the query image is used as a feature image; if they are inconsistent, the user's manual local selection result on the query image is deleted, and the automatic local selection result is used as a feature image.

[0110] S405: Encode the feature image to obtain a feature image vector.

[0111] In the above solution, similarly, after obtaining the user's feature image, vectorization processing is required to facilitate subsequent search operations.

[0112] Similar to the previous embodiment, the color information of the feature image may be removed before the image vectorization process to reduce the interference of color on the retrieval results.

[0113] In one embodiment, the entire query image may be first subjected to color culling, and then a feature image after the color culling may be extracted. For example, the color of the query image may be removed, and the query image after the color culling may be converted into a black and white line image to obtain a processed query image; the diagonal coordinate information of the feature image on the query image may be recorded; and based on the diagonal coordinate information, the processed feature image may be extracted from the processed query image.

[0114] In another implementation, the feature image may be directly color-removed without involving color removal of the query image, for example, the color of the feature image is removed and the color-removed feature image is converted into a black and white line image to obtain a processed feature image.

[0115] S406: Search the vector database for a second target image vector related to the feature image vector, and generate and display a second search result to the user by combining the target text vector, the first target image vector and the second target image vector.

[0116] In this step, after the feature image is vectorized, a matching search can be performed in the vector database to find the image vector related to the above feature image, and then another search result is generated based on the above image vector and the first search result, and the search result is fed back to the user.

[0117] For the same steps as those in the previous embodiment, please refer to the description of the previous embodiment, which will not be repeated here.

[0118] In order to better explain the solution shown in this embodiment, a simple example is given by a robotic arm (the pictures are all grayscale pictures, and the specific color information of the pictures refers to the text description):

[0119] First, suppose a user asks, "The gripper of the robot arm is loose and the gripper is not stable. What should I do?" and uploads a photo of the robot arm that needs to be repaired and inspected. Figure 5.1 shown.

[0120] Second, the retrieval system vectorizes the user's questions and uploaded images and searches in the vector database. For example, it finds the manufacturer, product model, and other information of the robot arm, and then finds the operating manual of the robot arm. After the first search of the manual, the first search results are as follows:

[0121] The detected manual provides an illustration of the loose adjustment of the robot arm gripper, such as Figure 5.2 The arm in the picture appears to be an industrial or professional-grade robot, perhaps used in an assembly line, manufacturing environment, or other automated tasks. The red arm and gray base indicate that it is designed for durability and functionality. To adjust the arm, especially if the gripper is loose, you may want to check the following parts:

[0122] Joint Screws: Make sure all joint screws are tight. Loose screws may cause inaccurate movement.

[0123] Connectors and Ports: Check that the connectors between the various parts of the arm are properly aligned and securely fastened.

[0124] Safety mechanisms: Make sure all safety mechanisms, such as locking mechanisms, are in working order to prevent accidental movement of the jaws.

[0125] Lubricant: Check that the joints and other moving parts have adequate lubrication, which helps keep the robotic arm running smoothly.

[0126] Since the manual and the illustration returned by the first search do not mark the parts that need to be adjusted, the user needs to manually select a part of the query image.

[0127] Third, the user refers to the text instructions and manual images retrieved in the first search results and marks the query image, such as Figure 5.3 As shown, you can ask the question again: "Is it this part?"

[0128] like Figure 5.3 As shown in the figure, the part of the robot head selected by the user belongs to the RGB color image (the main color is red, followed by gray), but the image information retrieved from the vector database cannot completely match it. The difference is as follows Figure 5.4 As shown: Figure 5.4 (a) is the feature image used for retrieval. Figure 5.4 (b) is the local detail image of the corresponding position of the illustration returned by the first search. Figure 5.4 (c) is a schematic diagram of a possible retrieved robotic arm. The two have different grippers and different colors (the main color is yellow, followed by gray).

[0129] Whether it is the "structured drawings" in the instructions or the illustrations of grippers of different colors, they are directly used for local image retrieval selected by the user. Due to the influence of color information, accurate retrieval cannot be achieved. Therefore, before entering the next step of image vectorization retrieval, the local query image can be color preprocessed.

[0130] Fourth, feature extraction is performed on the selected feature image to remove the RGB color information and retain only the features about texture and structural shape.

[0131] One way to deal with this is, Figure 5.5 As shown, the robot arm image to be repaired uploaded by the user ( Figure 5.5 (a) performs feature extraction and removes RGB color information ( Figure 5.5 In (b), Figure 5.5 (c) is a black and white line drawing generated after color inversion processing is performed on the image with extracted features. In order to obtain more effective image retrieval, the diagonal coordinate information of the local feature image in the query image is recorded, and the frame selection coordinates are superimposed with the extracted image features to obtain the feature image with RGB information removed.

[0132] Another way to deal with this is, Figure 5.6 As shown, first cut out the feature image with color information ( Figure 5.6 (a)), and then only perform feature extraction on this part ( Figure 5.6 In (b), the final feature image is obtained (wherein, the image for extracting features is subjected to color inversion, i.e., a black and white line drawing is generated, in order to retrieve the image more effectively. Figure 5.6 Middle (c).

[0133] Fifth, the retrieval system vectorizes the feature image with RGB information removed and searches it in the vector database to obtain the second retrieval result. The retrieval result can be displayed to the user with the query image superimposed with a local selection box.

[0134] In an embodiment of the present application, after the first search is completed, the user can perform a manual local box selection operation on the query image input by himself for further search. At this time, the feature image obtained based on the local box selection will be vectorized, and the image vector related to the feature image vector will be searched from the vector database, and then the user's question will be answered in combination with the first search result and the image vector obtained by the second search.

[0135] To summarize the above embodiments, the hybrid retrieval method based on industrial large models provided in the embodiments of the present application simultaneously obtains the query text and query image input by the user when the user searches, further explains the query text through the query image, and improves the accuracy of the retrieval system's understanding of the user's language and text, and then encodes the query text and query image respectively to obtain a query text vector and a query image vector, and then searches for target text vectors and target image vectors related to the query text vector and the query image vector from a vector database storing multiple text vectors and multiple image vectors, and answers the user's question in combination with the target text vector and the first target image vector.

[0136] Figure 6 A structural diagram of a hybrid retrieval device based on an industrial large model provided by an embodiment of the present invention, such as Figure 6 As shown, the hybrid retrieval device based on industrial large models may include various functional modules for implementing the aforementioned hybrid retrieval method based on industrial large models, and any functional module may be implemented by software and / or hardware.

[0137] For example, the hybrid retrieval device based on the industrial large model may include: an acquisition module 601, an encoding module 602 and a processing module 603;

[0138] The acquisition module 601 is used to acquire the query text and query image input by the user;

[0139] An encoding module 602 is used to encode the query text and the query image respectively to obtain a query text vector and a query image vector;

[0140] Processing module 603 is used to search for a target text vector and a first target image vector related to the query text vector and the query image vector from the vector database respectively, and generate and display a first retrieval result to the user in combination with the target text vector and the first target image vector, wherein the vector database stores multiple text vectors and multiple image vectors.

[0141] Optionally, the acquisition module 601 can also be specifically used to crawl network data before acquiring the query text and query image input by the user, and the network data includes text data and image data; encode the text data and image data respectively to obtain text vectors and image vectors; store the text vectors and image vectors respectively in a vector database; wherein, before encoding the text data and image data respectively, remove the color of the image data, and convert the decolored image data into black and white line image data to obtain processed image data.

[0142] Optionally, the encoding module 602 can also be used to remove the color of the query image before encoding the feature image, and convert the query image with the color removed into a black and white line image to obtain a processed query image; record the diagonal coordinate information of the feature image on the query image; and based on the diagonal coordinate information, extract the processed feature image from the processed query image.

[0143] Optionally, the encoding module 602 may also be specifically configured to: before encoding the feature image, remove the color of the feature image, and convert the feature image without color into a black and white line image to obtain a processed feature image.

[0144] Optionally, the processing module 603 can also be used to generate and display a first search result to the user by combining the target text vector and the first target image vector, obtain a local query image manually selected by the user on the query image, and obtain a feature image; encode the feature image to obtain a feature image vector; search for a second target image vector related to the feature image vector from a vector database, and generate and display a second search result to the user by combining the target text vector, the first target image vector, and the second target image vector.

[0145] Optionally, the processing module 603 may also be specifically configured to: before encoding the feature image, remove the color of the feature image, and convert the feature image without color into a black and white line image to obtain a processed feature image.

[0146] Optionally, the processing module 603 can also be used to automatically perform local box selection on the query image based on the first retrieval result; determine whether the automatic local box selection result is consistent with the user's manual local box selection result; if they are consistent, use the user's manual local box selection result on the query image as the feature image; if they are inconsistent, delete the user's manual local box selection result on the query image, and use the automatic local box selection result as the feature image.

[0147] The hybrid retrieval device based on industrial large models is used to execute the technical solution provided by the aforementioned hybrid retrieval method embodiment based on industrial large models. Its implementation principle and technical effect are similar to those in the aforementioned method embodiment and will not be repeated here.

[0148] The present application also provides a hybrid retrieval device based on an industrial large model, comprising: at least one processor and a memory;

[0149] Memory stores computer-executable instructions;

[0150] At least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs a hybrid retrieval method based on an industrial large model.

[0151] Figure 7 The hardware diagram of the hybrid retrieval device based on the industrial large model provided by the embodiment of the present invention. Figure 7 As shown, the hybrid retrieval device 70 based on industrial large models provided in this embodiment includes: at least one processor 701 and a memory 702. The device 70 also includes a communication component 703. The processor 701, the memory 702 and the communication component 703 are connected via a bus 704.

[0152] In the specific implementation process, at least one processor 701 executes the computer execution instructions stored in the memory 702, so that at least one processor 701 executes the above hybrid retrieval method based on the industrial big model.

[0153] The specific implementation process of the processor 701 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0154] In the above Figure 7 In the illustrated embodiment, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0155] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (NVM), such as at least one disk storage.

[0156] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of this application is not limited to only one bus or one type of bus.

[0157] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the hybrid retrieval method based on the industrial large model as described above is implemented.

[0158] The computer-readable storage medium mentioned above may be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium may be any available medium that can be accessed by a general or special-purpose computer.

[0159] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0160] The division of the units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, which may be electrical, mechanical or other forms.

[0161] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0162] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0163] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0164] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0165] So far, the technical solution of the present application has been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present application is obviously not limited to these specific embodiments, and the above embodiments are only used to illustrate the technical solution of the present application rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A hybrid retrieval method based on industrial large models, characterized in that: include: Obtain the query text and query image input by the user; Encoding the query text and the query image respectively to obtain a query text vector and a query image vector; A target text vector and a first target image vector related to the query text vector and the query image vector are searched for respectively from a vector database, and a first retrieval result is generated and displayed to the user in combination with the target text vector and the first target image vector, wherein the vector database stores a plurality of text vectors and a plurality of image vectors.

2. The method according to claim 1, characterized in that After combining the target text vector and the first target image vector to generate and present a first search result to the user, the method further includes: Acquire a local query image manually selected by the user on the query image to obtain a feature image; Performing encoding processing on the feature image to obtain a feature image vector; A second target image vector related to the feature image vector is searched from the vector database, and a second search result is generated and displayed to the user in combination with the target text vector, the first target image vector and the second target image vector.

3. The method according to claim 2, characterized in that The step of obtaining a local query image manually selected by the user on the query image to obtain a feature image includes: Based on the first search result, automatically perform partial frame selection on the query image; Determine whether the automatic local box selection result is consistent with the manual local box selection result of the user; If they are consistent, the manual local box selection result of the user on the query image is used as the feature image; If they are inconsistent, the manual local box selection result of the user on the query image is deleted, and the automatic local box selection result is used as the feature image.

4. The method according to claim 3, characterized in that Before encoding the feature image, the method further includes: Removing the color of the query image and converting the query image with the color removed into a black and white line image to obtain a processed query image; Recording the diagonal coordinate information of the feature image on the query image; Based on the diagonal coordinate information, a processed feature image is cut out from the processed query image.

5. The method according to claim 3, characterized in that: Before encoding the feature image, the method further includes: The color of the characteristic image is removed, and the characteristic image with the color removed is converted into a black and white line image to obtain a processed characteristic image.

6. The method according to claim 4 or 5, characterized in that: The background color of the black and white line image is white, and the line color is black.

7. The method according to claim 1, characterized in that Before obtaining the query text and query image input by the user, the method further includes: Crawling network data, wherein the network data includes text data and image data; Encoding the text data and the image data respectively to obtain a text vector and an image vector; storing the text vector and the image vector in the vector database respectively; Before encoding the text data and the image data respectively, the method further includes: The color of the image data is removed, and the image data after the color is removed is converted into black and white line image data to obtain processed image data.

8. A hybrid retrieval device based on industrial large models, characterized in that: include: An acquisition module, used to acquire the query text and query image input by the user; An encoding module, used to encode the query text and the query image respectively to obtain a query text vector and a query image vector; A processing module is used to search for a target text vector and a first target image vector related to the query text vector and the query image vector from a vector database, respectively, and generate and display a first retrieval result to the user in combination with the target text vector and the first target image vector, wherein the vector database stores a plurality of text vectors and a plurality of image vectors.

9. A hybrid retrieval device based on industrial large models, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor executes the hybrid retrieval method based on industrial large models as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the hybrid retrieval method based on industrial large models as described in any one of claims 1 to 7 are implemented.

Images