Vehicle retrieval question and answer method, controller and vehicle

Abstract

The invention provides a vehicle retrieval question and answer method, a controller and a vehicle, and the method comprises the steps: carrying out the analysis of a vehicle specification document through a pre-trained layout analysis model, and obtaining target information; the target information comprises a text and a picture; constructing a text vector database and an image vector database according to the target information; constructing a question proposed by a user, and retrieving the constructed question based on the text vector database and the image vector database to obtain an initial answer; reasoning the initial answer by using a multi-modal large model to obtain a target answer; the target answer comprises a picture and text content; therefore, after the question asked by the user is obtained, in order to improve the retrieval quality, the question needs to be constructed, so that the question better fits the understanding of the vehicle; and then the constructed questions are retrieved in the text vector database and the image vector database at the same time, and text answers and image answers are given after the initial answers are reasoned, so that the accuracy of vehicle reply is improved, a user can conveniently understand related operation steps, and the intelligence of the vehicle is improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent automobile technology, and in particular to a vehicle search and answering method, a controller and a vehicle. Background Art

[0002] With the advent of the era of intelligent cars, the functions of vehicles are increasing day by day, but users cannot successfully learn all the functions of the vehicle in a short time. Therefore, most vehicles are equipped with intelligent voice systems to answer questions about vehicle functions for users.

[0003] In the related technology, the vehicle can respond to questions in simple scenarios, but the user's usage scenarios are varied and complex, resulting in the vehicle being unable to give accurate responses, thus affecting the vehicle's intelligence. Summary of the invention

[0004] In response to the problems existing in the prior art, the embodiments of the present invention provide a vehicle retrieval question and answer method, a controller and a vehicle to solve or partially solve the technical problem in the prior art that the vehicle cannot give an accurate response when performing intelligent voice inquiries, thereby affecting the intelligence of the vehicle.

[0005] A first aspect of the present invention provides a vehicle search question answering method, the method comprising:

[0006] Analyze the vehicle manual document using a pre-trained layout analysis model to obtain target information; the target information includes text and pictures;

[0007] Constructing a text vector database and an image vector database according to the target information;

[0008] Constructing a question raised by a user, and searching the constructed question based on the text vector database and the image vector database to obtain an initial answer;

[0009] The initial answer is inferred using a multimodal large model to obtain a target answer; the target answer includes images and text content.

[0010] In the above scheme, the vehicle manual document is parsed using the pre-trained layout analysis model to obtain target information, including:

[0011] The vehicle manual document is divided into different types of content blocks using the pre-trained layout analysis model; the content blocks include: text, tables, formulas and pictures; the text includes headers, footers, titles, paragraphs and specific text descriptions;

[0012] Parse each type of content block to obtain corresponding content information;

[0013] Obtaining the original position of each content information in the vehicle manual document;

[0014] The content information is restored in layout according to the original position to obtain the vehicle manual information in the target format; the vehicle manual information in the target format is the target information.

[0015] In the above solution, constructing a text vector database according to the target information includes:

[0016] Acquire text content in the target information, and construct a plurality of standard text blocks according to the text content;

[0017] Convert the plurality of standard text blocks into text vectors using a pre-trained text vector embedding model;

[0018] The text vector is stored in a database to obtain the text vector database.

[0019] In the above solution, the step of constructing an image vector database according to the target information includes:

[0020] Obtaining the image in the target information;

[0021] Using a pre-trained image vector embedding model to segment the image into multiple sub-blocks;

[0022] Inputting the plurality of sub-image blocks into a pre-trained deep learning architecture model to obtain image vectors corresponding to the sub-image blocks;

[0023] The image vector is stored in a database to obtain the image vector database.

[0024] In the above scheme, the construction of the question raised by the user includes:

[0025] Extract semantic information from questions raised by users to obtain key information of the questions;

[0026] Searching for corresponding description parameters in the vehicle manual document based on the key information of the problem;

[0027] Regenerate the question based on the description parameters.

[0028] In the above solution, searching the constructed question based on the text vector database and the image vector database to obtain an initial answer includes:

[0029] Based on the constructed question, searching the text vector database and the image vector database simultaneously to obtain multiple search results; the search results include text results and image results;

[0030] Determining the relevance of each search result to the constructed question respectively;

[0031] The search result with the highest relevance is determined as the initial answer.

[0032] In the above solution, the use of a multimodal large model to reason about the initial answer to obtain a target answer includes:

[0033] Obtaining a first parameter value in the initial answer;

[0034] Determine a corresponding reference result value according to the first parameter value;

[0035] Obtain a second parameter value contained in the question raised by the user;

[0036] The reference result value is compared with the second parameter value to obtain the target answer.

[0037] A second aspect of the present invention provides a vehicle search question and answer controller, the controller comprising:

[0038] An analysis unit, used to analyze the vehicle manual document using a pre-trained layout analysis model to obtain target information; the target information includes text and pictures;

[0039] A construction unit, used to construct a text vector database and an image vector database according to the target information;

[0040] A retrieval unit, used to construct a question raised by a user, and to retrieve the constructed question based on the text vector database and the image vector database to obtain an initial answer;

[0041] The reasoning unit is used to use the multimodal large model to reason about the initial answer to obtain a target answer; the target answer includes pictures and text content.

[0042] In the above solution, the analysis unit is specifically used for:

[0043] The vehicle manual document is divided into different types of content blocks using the pre-trained layout analysis model; the content blocks include: text, tables, formulas and pictures; the text includes headers, footers, titles, paragraphs and specific text descriptions;

[0044] Parse each type of content block to obtain corresponding content information;

[0045] Obtaining the original position of each content information in the vehicle manual document;

[0046] The content information is restored in layout according to the original position to obtain the vehicle manual information in the target format; the vehicle manual information in the target format is the target information.

[0047] In the above scheme, the construction unit is specifically used for:

[0048] Acquire text content in the target information, and construct a plurality of standard text blocks according to the text content;

[0049] Convert the plurality of standard text blocks into text vectors using a pre-trained text vector embedding model;

[0050] The text vector is stored in a database to obtain the text vector database.

[0051] In the above scheme, the construction unit is specifically used for:

[0052] Obtaining the image in the target information;

[0053] Using a pre-trained image vector embedding model to segment the image into multiple sub-blocks;

[0054] Inputting the plurality of sub-image blocks into a pre-trained deep learning architecture model to obtain image vectors corresponding to the sub-image blocks;

[0055] The image vector is stored in a database to obtain the image vector database.

[0056] In the above scheme, the construction unit is specifically used for:

[0057] Extract semantic information from questions raised by users to obtain key information of the questions;

[0058] Searching for corresponding description parameters in the vehicle manual document based on the key information of the problem;

[0059] Regenerate the question based on the description parameters.

[0060] In the above solution, the retrieval unit is specifically used for:

[0061] Based on the constructed question, searching the text vector database and the image vector database simultaneously to obtain multiple search results; the search results include text results and image results;

[0062] Determining the relevance of each search result to the constructed question respectively;

[0063] The search result with the highest relevance is determined as the initial answer.

[0064] In the above solution, the reasoning unit is specifically used for:

[0065] Obtaining a first parameter value in the initial answer;

[0066] Determine a corresponding reference result value according to the first parameter value;

[0067] Obtain a second parameter value contained in the question raised by the user;

[0068] The reference result value is compared with the second parameter value to obtain the target answer.

[0069] A third aspect of the present invention provides a vehicle, comprising the vehicle search and answering controller according to any one of the first aspects.

[0070] The present invention provides a vehicle retrieval question and answer method, a controller and a vehicle, the method comprising: using a pre-trained layout analysis model to analyze a vehicle manual document to obtain target information; the target information includes text and pictures; constructing a text vector database and an image vector database according to the target information; constructing a question raised by a user, and retrieving the constructed question based on the text vector database and the image vector database to obtain an initial answer; using a multimodal large model to infer the initial answer to obtain a target answer; the target answer includes pictures and text content; in this way, after obtaining the question asked by the user, in order to improve the retrieval quality, the question needs to be constructed so that the question is more in line with the vehicle's understanding; then the constructed question is simultaneously retrieved in the text vector database and the image vector database, and a text answer and a picture answer are given after the initial answer is inferred, which not only improves the accuracy of the vehicle's reply, but also facilitates the user to understand the relevant operation steps, thereby improving the intelligence of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0071] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0072] Figure 1 A schematic diagram of the process flow of a vehicle search question and answer method according to an embodiment of the present invention is shown;

[0073] Figure 2 The content of one page of a vehicle manual document according to one embodiment of the present invention is shown;

[0074] Figure 3 A vehicle trunk image and a size diagram retrieved according to an embodiment of the present invention are shown;

[0075] Figure 4 A schematic diagram of the structure of a vehicle retrieval question and answer controller according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0076] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0077] The present invention provides a vehicle search question and answer method, such as Figure 1 As shown, the method mainly includes the following steps:

[0078] S110, using a pre-trained layout analysis model to analyze the vehicle manual document to obtain target information; the target information includes text and pictures.

[0079] The present invention needs to use a pre-trained layout analysis model to analyze the vehicle manual document to obtain target information, providing a data basis for the subsequent construction of a text vector database and an image vector database; the target information includes text and pictures.

[0080] In one implementation, the vehicle manual document is parsed using a pre-trained layout analysis model to obtain target information, including:

[0081] The vehicle manual document is divided into different types of content blocks using the pre-trained layout analysis model; content blocks include: text, tables, formulas and pictures; text includes headers, footers, titles, paragraphs and specific text descriptions;

[0082] Parse each type of content block to obtain corresponding content information;

[0083] Obtain the original location of each content information in the vehicle manual document;

[0084] The content information is restored in layout according to the original position to obtain the vehicle manual information in the target format; the vehicle manual information in the target format is the target information.

[0085] Specifically, if Figure 2 As shown, the vehicle manual contains text, pictures, titles and other contents, so the present invention can use a pre-trained layout analysis model (such as the LayoutLMv3 model) to divide the manual document into different types of content blocks, such as text, tables, formulas, pictures, etc.; wherein the text includes: headers, footers, titles, paragraphs, page numbers and other contents.

[0086] Optical Character Recognition (OCR) is used to recognize text, tables, formulas, and picture rows to obtain the content information contained in the corresponding content blocks, and the pictures are stored separately.

[0087] Then, the layout analysis model is used to restore the layout of all content information, and finally the target information in MarkDown format is formed. The order and position of text, title, formula and picture in the target information are consistent with the order in the original vehicle manual document.

[0088] Among them, the present invention trains the layout analysis model by collecting a certain number (such as 500) of PDF files of vehicle manuals to obtain a trained layout analysis model. The layout analysis model can also be understood as a pre-trained document artificial intelligence multimodal deep learning architecture model transformer, including a mask language model, a mask image modeling model, a discrete token reconstruction target, and a word matching target. Among them, token is the basic unit in natural language processing, usually referring to a word or symbol. In text, a token can be a word, punctuation mark, number or other special character.

[0089] S111, constructing a text vector database and an image vector database according to the target information.

[0090] After obtaining the target information, a text vector database and an image vector database can be constructed according to the target information.

[0091] In one embodiment, constructing a text vector database according to target information includes:

[0092] Acquire text content in the target information, and construct multiple standard text blocks according to the text content;

[0093] Use a pre-trained text vector embedding model to convert multiple standard text blocks into text vectors;

[0094] The text vectors are stored in a database to obtain a text vector database.

[0095] Specifically, since the vector embedding model has an input limit of 512 or 1024 tokens, the text content of the target information needs to be segmented. In one embodiment, multiple standard text blocks are constructed according to the text content, including:

[0096] Get the standard character length corresponding to the standard text block;

[0097] For any text content, if the character length of the text content is less than or equal to the standard character length, the text content is directly determined as a standard text block;

[0098] If the character length of the text content is greater than the standard character length, the text content is recursively divided into blocks based on the standard character length to obtain standard text blocks.

[0099] Recursive chunking means: assuming the standard character length is 1024, if the character length of the text content is greater than 1024, set the content of the current text to 1024 characters, overlap adjacent texts to 256 characters, and then slide the window in sequence with a length of 768 characters to obtain a text block of 1024 characters. Although the final standard text block size may not be exactly the same, they will approach the required size, thereby maintaining a certain uniformity. After the text chunking process, many standard text blocks less than or equal to 1024 characters will be formed.

[0100] Then, a vector embedding model (such as a text embedding model) is used to extract text vectors from each standard text block to obtain multiple text vectors, and the text vectors are stored in a database to obtain a text vector database.

[0101] Among them, the Embedding model is a text embedding vector model, which specifically generates a vector representation of the sentence by performing self-attention calculation on each word in the sentence through the Transformer model.

[0102] When training the text vector embedding model, for each text block, a large model (such as the Qwen model) can be used to generate one or more questions related to the current text block, and then the generated questions are paired with the corresponding text blocks to form positive sample pairs; the questions are paired with irrelevant text blocks to form negative sample pairs, and the positive sample pairs and negative sample pairs are used as the training set of the text vector embedding model to train the text vector embedding model. The goal of the text vector embedding model is to learn the vector representation of text (questions and text blocks) so that in the vector space, related texts (questions and text blocks in positive sample pairs) are close to each other, and irrelevant texts (questions and text blocks in negative sample pairs) are far away from each other.

[0103] In one implementation, constructing an image vector database according to target information includes:

[0104] Get the image in the target information;

[0105] Using a pre-trained image vector embedding model to segment the image into multiple sub-blocks;

[0106] Input multiple sub-blocks into the pre-trained deep learning architecture transformer model to obtain the image vector corresponding to the sub-block;

[0107] The image vectors are stored in a database to obtain an image vector database.

[0108] The present invention can use two methods to construct an image vector database. The first method is to use a pre-trained image vector embedding model (such as the CLIP Embedding model) to segment the image to obtain multiple sub-block patches, and then input the multiple patches into a pre-trained transformer model to output the corresponding image vectors.

[0109] The second method is to use the open source Qwen-VL multimodal model to extract the text content in the image, and then use the text embedding model to extract vector data from the text summary content of the image and store it in the database to obtain an image vector database.

[0110] S112, constructing the question raised by the user, and searching the constructed question based on the text vector database and the image vector database to obtain an initial answer.

[0111] In order to accurately find relevant content of the vehicle manual according to the questions raised by the user and improve the accuracy of the search, the present invention also needs to construct the questions raised by the user.

[0112] In one implementation, the question raised by the user is constructed to include:

[0113] Extract semantic information from questions raised by users to obtain key information of the questions;

[0114] Find the corresponding description parameters in the vehicle manual document based on the key information of the problem;

[0115] Regenerates the question based on the description parameters.

[0116] For example, suppose the question asked by the user is "Can a 2.3-meter fishing rod be put into the trunk?", then the key information of the question is the length of the fishing rod, and the descriptive parameter that should be found in the vehicle manual document is the size of the trunk (the length, width, and height of the trunk). Then the question regenerated based on the descriptive parameters can be "Can a trunk with dimensions of XX in length, width, and height fit a 2.3-meter-long fishing rod?"

[0117] In this way, the reconstructed questions are more consistent with the content in the vehicle manual, thereby improving the accuracy of the retrieval.

[0118] Then, the constructed question is searched based on the text vector database and the image vector database to obtain an initial answer, including:

[0119] Based on the constructed question, search is performed in the text vector database and the image vector database at the same time to obtain multiple search results; the search results include text results and image results;

[0120] Determine the relevance of each search result to the constructed question;

[0121] The search result with the highest relevance is determined as the initial answer.

[0122] Specifically, in order to enable users to understand the vehicle's response more intuitively, the present invention uses a multi-vector searcher to search in the text vector database and the image vector database based on the constructed questions. In order to improve the search efficiency, the texts and images with correlation in the text vector database and the image vector database can be bound by ID in advance (each text vector and each image vector has a unique ID). In this way, when any one of the image vectors or text vectors is retrieved, if the image vector or text vector has an ID binding relationship, the corresponding text vector or image vector can be directly output together.

[0123] In this way, the multi-vector retriever transforms text and image data of different modalities into a unified vector space.

[0124] After the search, there may be multiple search results related to the constructed question. In order to further improve the search effect, the present invention needs to determine the relevance between each search result and the constructed question, and then sort them from high to low according to the relevance, and determine the search result with the highest relevance as the initial answer. In this way, the subsequent large model can focus on the search result with the highest relevance to reduce noise interference.

[0125] When determining the relevance, since the constructed question is a text content, the similarity between the text content in the search results and the constructed question can be determined, and the highest similarity represents the highest relevance.

[0126] S113, using a multimodal large model to infer the initial answer to obtain a target answer; the target answer includes pictures and text content.

[0127] After the initial answer is determined, the user cannot directly determine the desired result from the initial answer because the initial answer only has some intermediate parameter values. Therefore, it is necessary to use a multimodal large model to infer the initial answer and obtain the target answer.

[0128] In one embodiment, the multimodal large model is used to reason the initial answer to obtain the target answer, including:

[0129] Get the first parameter value in the initial answer;

[0130] Determine a corresponding reference result value according to the first parameter value;

[0131] Get the second parameter value contained in the question asked by the user;

[0132] Compare the reference result value with the second parameter value to get the target answer.

[0133] Specifically, after the initial answer is determined, the initial answer includes text content and image content. The text content can be input as known information into the pre-created prompt (for the language model, the prompt can be a short question, a complete paragraph, or a set of instructions), and the multimodal large model is added to the prompt. The picture is used as known information and is input into the multimodal large model Qwen-VL in the form of a URL, and the multimodal large model is used to reason about the answer.

[0134] When the final answer is generated, if there are related pictures, the pictures need to be inserted in the correct position and returned in Markdown format to achieve the arrangement of text and pictures in the target answer.

[0135] When the multimodal large model is inferring the answer, it can first obtain the first parameter value in the initial answer. Continuing with the above example, suppose the constructed question is "Can a trunk with a length, width and height of XX fit a 2.3-meter-long fishing rod?" Then the initial answer will carry the length, width and height of the trunk, and the first parameter value is the length, width and height of the trunk.

[0136] Then, the multimodal large model will perform mathematical operations based on the length, width and height of the trunk, converting the problem into a mathematical problem of the spatial diagonal corresponding to the length, width and height, and determine the length of the spatial diagonal based on the length, width and height. The length of the spatial diagonal is the reference result value.

[0137] Then, the second parameter value contained in the question raised by the user is obtained, and the second parameter value is the length of the fishing rod 2.3m; finally, the diagonal length of the space and the length of the fishing rod are compared to obtain the target answer.

[0138] If the diagonal length of the space is greater than or equal to the length of the fishing rod, then the target answer is "it can be put in the trunk"; if the diagonal length of the space is less than the length of the fishing rod, then the target answer is "it cannot be put in the trunk".

[0139] The implementation code of the entire reasoning process is as follows:

[0140] REACT_PROMPT="""Answer the following questions as best you can. You have access to the following

[0141] tools:

[0142] {Code Interpreter}:Call this tool to interact with the{code_interpreter}API.What is the{code_interpreter}

[0143] API useful for? {Code interpreter that can execute Python code.Enclosethe code within triple backticks

[0144] (`)at the beginning and end of the code.",#If you expect`Action Input`to be a Markdown code block.}

[0145] Parameters:{"name":"code","description":"Code to be executed",

[0146] "required":True,"schema":{"type":"string"}}Format the arguments as aJSON object.

[0147] The known information is as follows:

[0148] <image_url>

[0149] The image shows the interior of a vehicle's rear compartment. Figure 3 As can be seen from the picture, the car has a spacious and neat space design. There is a small illustration in the upper left corner showing a bottle of water and some items in the car, which may mean that this is a car model for family or business use, focusing on comfort and convenience.

[0150] In addition, there are three labels below the image that indicate the dimensions: Height: 1000mm (millimeter), Depth: 570-1520mm (millimeter), which indicates the distance from the deepest part of the luggage compartment to the edge of the opening. Width: 1240mm (millimeter), which indicates the maximum distance between the left and right sides of the luggage compartment.

[0151] These data provide some basic information about the vehicle's trunk capacity and show that it has enough space to accommodate various sizes of cargo or travel necessities.

[0152] The trunk can be loaded with larger items. To ensure the vehicle's driving stability and safety, when loading luggage, the items should be placed as evenly as possible, and heavy items should be placed at the front of the trunk.

[0153] Use the following format:

[0154] Question:the input question you must answer

[0155] Thought:you should always think about what to do

[0156] Action:the action to take,should be one of tool_names

[0157] Action Input: the input to the action

[0158] Observation: the result of the action

[0159] ...(this Thought / Action / Action Input / Observation can be repeated zeroor more times)

[0160] Thought:I now know the final answer

[0161] Final Answer: the final answer to the original input question. If the final answer is related to the picture, please output the corresponding image address, and arrange the text and picture of the final result, and the answer will be returned in Markdown format.

[0162] Begin!

[0163] Question: Can a trunk with dimensions of XX in length, width and height fit a 2.3 meter long fishing rod?

[0164] In the above code, CodeInterpreter is a Python execution tool used for mathematical calculations (such as calculating the diagonal distance of a space); the known information is the retrieved initial answer, including the URL of the text and image; format is used to prompt the big model to reply according to the preset format and to prompt the big model to layout the final answer.

[0165] In this way, by effectively analyzing the text, pictures, tables and other information of the vehicle manual document, the complex questions raised by the user are reconstructed and multiple inferences are performed in combination with the multimodal large model, so that the questions are more in line with the vehicle's understanding; then the constructed questions are searched in the text vector database and the image vector database at the same time, and the text answer and the picture answer are given after the initial answer is inferred, which not only improves the accuracy of the vehicle's reply, but also facilitates the user to understand the relevant operation steps and improves the intelligence of the vehicle. In addition, the present invention does not need to establish a complex standard answer process, thereby reducing the implementation cost.

[0166] Based on the same inventive concept as in the above-mentioned embodiment, this embodiment also provides a vehicle search question and answer controller, such as Figure 4 As shown, the controller includes:

[0167] The analysis unit 41 is used to analyze the vehicle manual document using a pre-trained layout analysis model to obtain target information; the target information includes text and pictures;

[0168] A construction unit 42, configured to construct a text vector database and an image vector database according to the target information;

[0169] A retrieval unit 43 is used to construct a question raised by a user, and to retrieve the constructed question based on the text vector database and the image vector database to obtain an initial answer;

[0170] The reasoning unit 44 is used to use the multimodal large model to reason about the initial answer to obtain a target answer; the target answer includes pictures and text content.

[0171] In one implementation, the analysis unit 41 is specifically configured to:

[0172] The vehicle manual document is divided into different types of content blocks using the pre-trained layout analysis model; the content blocks include: text, tables, formulas and pictures; the text includes headers, footers, titles, paragraphs and specific text descriptions;

[0173] Parse each type of content block to obtain corresponding content information;

[0174] Obtaining the original position of each content information in the vehicle manual document;

[0175] The content information is restored in layout according to the original position to obtain the vehicle manual information in the target format; the vehicle manual information in the target format is the target information.

[0176] Since the controller introduced in the embodiment of the present invention is the controller used to implement the vehicle search question-answering method of the embodiment of the present invention, based on the method introduced in the embodiment of the present invention, the person skilled in the art can understand the specific structure and deformation of the controller, so it is not repeated here. All controllers used in the method of the embodiment of the present invention belong to the scope of protection of the present invention.

[0177] Based on the same inventive concept as the above embodiment, the embodiment of the present invention further provides a vehicle, comprising the above-mentioned vehicle search and answer controller. The implementation logic and specific structure of the vehicle search and answer controller can be referred to the corresponding description above, so it will not be repeated here.

[0178] Through one or more embodiments of the present invention, the present invention has the following beneficial effects or advantages:

[0179] The present invention provides a vehicle retrieval question and answer method, a controller and a vehicle, the method comprising: using a pre-trained layout analysis model to analyze a vehicle manual document to obtain target information; the target information includes text and pictures; constructing a text vector database and an image vector database according to the target information; constructing a question raised by a user, and retrieving the constructed question based on the text vector database and the image vector database to obtain an initial answer; using a multimodal large model to infer the initial answer to obtain a target answer; the target answer includes pictures and text content; in this way, after obtaining the question asked by the user, in order to improve the retrieval quality, the question needs to be constructed so that the question is more in line with the vehicle's understanding; then the constructed question is simultaneously retrieved in the text vector database and the image vector database, and a text answer and a picture answer are given after the initial answer is inferred, which not only improves the accuracy of the vehicle's reply, but also facilitates the user to understand the relevant operation steps, thereby improving the intelligence of the vehicle.

[0180] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0181] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A vehicle search question-answering method, characterized in that: The method comprises: Analyze the vehicle manual document using a pre-trained layout analysis model to obtain target information; the target information includes text and pictures; Constructing a text vector database and an image vector database according to the target information; Constructing a question raised by a user, and searching the constructed question based on the text vector database and the image vector database to obtain an initial answer; The initial answer is inferred using a multimodal large model to obtain a target answer; the target answer includes images and text content.

2. The method according to claim 1, characterized in that The vehicle manual document is parsed using the pre-trained layout analysis model to obtain target information, including: The vehicle manual document is divided into different types of content blocks using the pre-trained layout analysis model; the content blocks include: text, tables, formulas and pictures; the text includes headers, footers, titles, paragraphs and specific text descriptions; Parse each type of content block to obtain corresponding content information; Obtaining the original position of each content information in the vehicle manual document; The content information is restored in layout according to the original position to obtain the vehicle manual information in the target format; the vehicle manual information in the target format is the target information.

3. The method according to claim 1, characterized in that The step of constructing a text vector database according to the target information includes: Acquire text content in the target information, and construct a plurality of standard text blocks according to the text content; Convert the plurality of standard text blocks into text vectors using a pre-trained text vector embedding model; The text vector is stored in a database to obtain the text vector database.

4. The method according to claim 1, characterized in that The step of constructing an image vector database according to the target information includes: Obtaining the image in the target information; Using a pre-trained image vector embedding model to segment the image into multiple sub-blocks; Inputting the plurality of sub-image blocks into a pre-trained deep learning architecture model to obtain image vectors corresponding to the sub-image blocks; The image vector is stored in a database to obtain the image vector database.

5. The method according to claim 1, characterized in that The construction of the questions raised by the user includes: Extract semantic information from questions raised by users to obtain key information of the questions; Searching for corresponding description parameters in the vehicle manual document based on the key information of the problem; The question is regenerated based on the described parameters.

6. The method according to claim 1, characterized in that The step of searching the constructed question based on the text vector database and the image vector database to obtain an initial answer includes: Based on the constructed question, searching the text vector database and the image vector database simultaneously to obtain multiple search results; the search results include text results and image results; Determining the relevance of each search result to the constructed question respectively; The search result with the highest relevance is determined as the initial answer.

7. The method according to claim 1, characterized in that The method of using the multimodal large model to reason the initial answer to obtain the target answer includes: Obtaining a first parameter value in the initial answer; Determine a corresponding reference result value according to the first parameter value; Obtain a second parameter value contained in the question raised by the user; The reference result value is compared with the second parameter value to obtain the target answer.

8. A vehicle retrieval question-answering controller, characterized in that: The controller comprises: An analysis unit, used to analyze the vehicle manual document using a pre-trained layout analysis model to obtain target information; the target information includes text and pictures; A construction unit, used to construct a text vector database and an image vector database according to the target information; A retrieval unit, used to construct a question raised by a user, and to retrieve the constructed question based on the text vector database and the image vector database to obtain an initial answer; The reasoning unit is used to use the multimodal large model to reason about the initial answer to obtain a target answer; the target answer includes pictures and text content.

9. The controller according to claim 8, characterized in that The analysis unit is specifically used for: The vehicle manual document is divided into different types of content blocks using the pre-trained layout analysis model; the content blocks include: text, tables, formulas and pictures; the text includes headers, footers, titles, paragraphs and specific text descriptions; Parse each type of content block to obtain corresponding content information; Obtaining the original position of each content information in the vehicle manual document; The content information is restored in layout according to the original position to obtain the vehicle manual information in the target format; the vehicle manual information in the target format is the target information.

10. A vehicle, characterized in that: The vehicle includes the vehicle retrieval question and answer controller as described in any one of claims 8-9.

Images