Question guidance interaction method, device, equipment and storage medium

Abstract

The present application provides a method, device, equipment and storage medium for interactive question guidance. In one example, the method includes: obtaining user input information; when the user input information includes textual question description information and question example information, extracting features from the textual question description information to obtain question description feature information; and performing intelligent statistics on the question example information, determining suggestion prompt information based on the intelligent statistical results, and extracting features from the suggestion prompt information to obtain suggestion prompt feature information; based on the question description feature information and the suggestion prompt feature information, using a multimodal large model, performing similarity retrieval in the knowledge base to determine answer guidance. This method can improve the accuracy of answering questions in intelligent question and answering.

Description

Technical Field

[0001] The present application relates to the field of intelligent analysis technology, and in particular to a question-guided interaction method, device, equipment and storage medium. Background Art

[0002] Intelligent question guidance refers to the use of artificial intelligence technology to automatically understand and analyze questions raised by users and provide question guidance suggestions. This technology combines methods from fields such as natural language processing (NLP) and multimodal knowledge representation, aiming to achieve smarter and more accurate interaction methods.

[0003] Currently, existing intelligent question guidance solutions generally provide more answers to questions but they are not targeted enough. Users need to try them one by one, which is very costly.

[0004] How to help users get guidance on their questions more accurately has become a hot research topic in recent years. Summary of the invention

[0005] In view of this, the present application provides a question guidance interaction method, device, equipment and storage medium.

[0006] Specifically, the present application is implemented through the following technical solutions:

[0007] According to a first aspect of an embodiment of the present application, a question-guided interaction method is provided, comprising:

[0008] Get user input information;

[0009] In the case where the user input information includes textual problem description information and problem example information, feature extraction is performed on the textual problem description information to obtain problem description feature information; and intelligent statistics are performed on the problem example information, suggestion prompt information is determined according to the intelligent statistical result, and feature extraction is performed on the suggestion prompt information to obtain suggestion prompt feature information; wherein the suggestion prompt information is used to assist in determining the source of the problem;

[0010] Based on the characteristic information of the question description and the characteristic information of the suggestion prompt, a similar search is performed in the knowledge base using a multimodal large model to determine the answer guidance.

[0011] According to a second aspect of an embodiment of the present application, a question guidance interaction device is provided, including:

[0012] An acquisition unit, used for acquiring user input information;

[0013] A feature extraction unit, configured to extract features from the textual question description information to obtain question description feature information when the user input information includes textual question description information and question example information;

[0014] An intelligent analysis unit, used for performing intelligent statistics on the problem example information, and determining suggestion prompt information according to the intelligent statistical results;

[0015] The feature extraction unit is further used to extract features from the suggestion prompt information to obtain suggestion prompt feature information; wherein the suggestion prompt information is used to assist in determining the source of the problem;

[0016] The guidance unit is used to perform similarity search in the knowledge base based on the feature information of the question description and the feature information of the suggestion prompt using a multimodal large model to determine the answer guidance.

[0017] According to a third aspect of an embodiment of the present application, there is provided an electronic device, comprising a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method provided in the first aspect.

[0018] According to a fourth aspect of an embodiment of the present application, a machine-readable storage medium is provided, wherein the machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method provided in the first aspect is implemented.

[0019] The technical solution provided by this application can at least bring the following beneficial effects:

[0020] In the process of intelligent question guidance, user input information may include not only textual question description information but also question example information. Thus, on the one hand, feature extraction may be performed on the textual question description information to obtain question description feature information. On the other hand, intelligent statistics may be performed on the question example information, suggestion prompt information may be determined based on the intelligent statistical results, and feature extraction may be performed on the suggestion prompt information to obtain suggestion prompt feature information. Furthermore, based on the question description feature information and the suggestion prompt feature information, a multimodal large model may be used to perform similar searches in the knowledge base to determine answer guidance. By performing intelligent statistics on the question example information, rich information about the question may be obtained to assist in obtaining more accurate answer guidance, thereby improving the accuracy of answering questions in intelligent question and answering. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 is a flowchart of a question-guided interaction method shown in an exemplary embodiment of the present application;

[0022] Figure 2It is a schematic diagram of an overall process of question-guided interaction shown in an exemplary embodiment of the present application;

[0023] Figure 3 is a structural diagram of a question guidance interaction device provided in an embodiment of the present application;

[0024] Figure 4 is a structural diagram of another question guidance interaction device provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0027] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings.

[0028] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0029] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0030] See also Figure 1 , which is a flow chart of a question-guided interaction method provided in an embodiment of the present application, such as Figure 1 As shown, the question-guided interaction method may include the following steps:

[0031] Step S100: Obtain user input information.

[0032] In the embodiment of the present application, the user input information may include text problem description information, and the text problem description information describes the problem to be solved in text form.

[0033] For example, the textual problem description information may be “how to solve the problem of misdetection of objects in images”.

[0034] In one example, in order to optimize the interactive effect of question guidance and improve the accuracy of answer guidance output by the multimodal large model, the user input information may also include question example information, which may include examples of problems that the user needs to solve.

[0035] Exemplarily, the question example information may include image information related to the text question description.

[0036] For example, assuming that in a target detection task, the text problem description information is "How to solve the problem of false detection of targets in images", then the problem example information may include wrong examples (such as images where false detection of targets occurs, as well as information such as false detection boxes and false alarm categories in the images) and training and test data, etc.

[0037] It can be seen that the question-guided interaction system provided in the embodiment of the present application is not limited to a single text input or image input, but can support both text input and image input, that is, it can support multimodal input, thereby improving the scalability of the solution.

[0038] Step S110: When the user input information includes textual problem description information and problem example information, feature extraction is performed on the textual problem description information to obtain problem description feature information; and, intelligent statistics are performed on the problem example information, suggestion prompt information is determined based on the intelligent statistical results, and feature extraction is performed on the suggestion prompt information to obtain suggestion prompt feature information; wherein the suggestion prompt information is used to assist in determining the source of the problem.

[0039] In an embodiment of the present application, when the user input information includes textual question description information and question example information, on the one hand, feature extraction can be performed on the textual question description information included in the user input information to obtain corresponding textual feature information (which can be referred to as question description feature information).

[0040] On the other hand, intelligent statistics can be performed on the problem example information included in the user input information, and suggestion prompt information can be determined based on the intelligent statistical results; wherein, the suggestion prompt information can be used to assist in determining the source of the problem.

[0041] That is, intelligently analyze the problem examples, mine the rich information of the problem itself, help locate the problem more accurately and give reasonable and executable suggestions.

[0042] Combining textual question description information and suggestion prompt information can achieve more accurate and comprehensive question guidance.

[0043] For the determined suggestion prompt information, feature extraction may be performed on it to obtain corresponding feature information (which may be referred to as suggestion prompt feature information).

[0044] Step S120: Based on the feature information of the question description and the feature information of the suggestion prompt, a similar search is performed in the knowledge base using a multimodal large model to determine the answer guidance.

[0045] In an embodiment of the present application, when the problem description feature information and the suggestion feature information are obtained in the above manner, a similar search can be performed in the knowledge base based on the problem description feature information and the suggestion feature information using a multimodal large model to determine the answer guidance.

[0046] Exemplarily, the problem description feature information and the suggestion feature information can be input into the multimodal big model, and the multimodal big model can perform similarity search in the knowledge base based on the problem description feature information and the suggestion feature information, and determine the answer guidance based on the search results.

[0047] Exemplarily, the knowledge base may be pre-built and may include frequently asked questions and suggestions corresponding to the questions (eg, answers corresponding to the questions).

[0048] For example, one question may correspond to one or more suggestions.

[0049] For example, {How to solve the problem of missed target detection? Suggestion 1: There are missed targets in the training set, modify the training set annotation; Suggestion 2: There are no similar samples, please add such samples...}.

[0050] For example, the initial knowledge base can be constructed by summarizing questions and corresponding answers. In the subsequent process, the initial knowledge base can be combined with intelligent analysis to obtain answer guidance.

[0051] It should be noted that when the answer guidance obtained by processing in the above manner accumulates to a certain number, it can be fed back to improve and expand the initially constructed knowledge base, that is, new questions and corresponding answer guidance are added to the knowledge base.

[0052] Among them, the newly added questions and the answer guidance in the corresponding answer guidance may be the answer guidance confirmed by manual means.

[0053] In one example, in order to further improve the accuracy of answering questions in intelligent question and answering, for the above-mentioned multimodal big model, the multimodal big model can be fine-tuned and trained to achieve the update of the multimodal big model when the big model update conditions are met.

[0054] Exemplarily, sample data for fine-tuning training of the multimodal large model can be constructed based on user input information and feedback information for answer guidance; during the fine-tuning training of the multimodal large model, training can be performed using a joint comparative training method based on feature information corresponding to the user input information and feature information corresponding to the feedback information.

[0055] Exemplarily, the specific implementation of fine-tuning training of the multimodal large model can be found in the relevant description below, and the embodiments of the present application will not be described in detail here.

[0056] It can be seen that in Figure 1 In the method flow shown, during the intelligent question-answering process, user input information may include not only textual question description information but also question example information. Thus, on the one hand, feature extraction may be performed on the textual question description information to obtain question description feature information. On the other hand, intelligent statistics may be performed on the question example information, suggestion prompt information may be determined based on the intelligent statistical results, and feature extraction may be performed on the suggestion prompt information to obtain suggestion prompt feature information. Furthermore, based on the question description feature information and the suggestion prompt feature information, a multimodal large model may be used to perform similarity searches in the knowledge base to determine answer guidance. By performing intelligent statistics on the question example information, rich question information may be obtained to assist in obtaining more accurate answer guidance, thereby improving the accuracy of answering questions in the intelligent question-answering process.

[0057] In some embodiments, the feature extraction of the textual problem description information to obtain the problem description feature information may include:

[0058] Use natural language processing to extract key information from the text problem description information to obtain the key information in the text problem description information;

[0059] The multimodal large model is used to extract the key information and obtain the feature information of the problem description.

[0060] Exemplarily, for the text problem description information input by the user, natural language processing can be used to extract key information from the text problem description information to obtain the key information in the text problem description information, and a multimodal large model can be used to extract features from the key information to obtain problem description feature information.

[0061] In some embodiments, the question example information includes image information associated with the textual question description.

[0062] The above intelligent statistics of the problem example information may include:

[0063] Performing basic information statistics on the problem example information to obtain basic information statistics results; wherein the basic information statistics results include image basic information statistics results, which include one or more of image size distribution information, image brightness distribution information, and image color distribution information; when there is a specified target in the image, the basic information statistics results also include target basic information statistics results, which include one or more of target size distribution information, target brightness distribution information, and target color distribution information;

[0064] and / or,

[0065] Perform scene / point distribution statistics on the problem example information to obtain scene / point distribution statistics results;

[0066] and / or,

[0067] Extended information statistics are performed on the question example information to obtain extended information statistical results; wherein the extended information statistical results include information obtained by performing information mining on the image based on the image content understanding and reasoning ability.

[0068] For example, the example question information includes image information related to the text question description. The image information may include but is not limited to the example question image, and images in the training and test data.

[0069] Exemplarily, the problem example images, training and test data sets, etc. input by the user can be processed. For example, intelligent statistics can be performed on the training data set and the test data set, such as the target size (which can be the width and height of the detection box, etc.), the diagnosis results of missed or mislabeled items, the scene / point distribution, etc., and text suggestion descriptions (i.e., suggestion prompt information) can be given to assist in determining the source of the problem.

[0070] For example, assuming that based on the problem example input by the user, similar missed targets in the training set are found in the missed and mislabeled diagnosis results through model feature similarity matching, a prompt message can be output: "There are similar missed targets in the training set", and relevant examples can be output to guide the user to make modifications.

[0071] In one example, performing intelligent statistics on the question example information may include performing basic information statistics on the question example information to obtain basic information statistics results.

[0072] For example, it is considered that the distribution of basic information of images in the training data set and the test data set may affect the processing performance of the model.

[0073] For example, if the proportion of images in a certain resolution range in the training data set and the test data set is too high, the model will have better processing performance for images in this resolution range, but will have relatively low processing performance for images in other resolution ranges.

[0074] For another example, if the proportion of images of a certain color in the training data set and the test data set is too low (or even no images of that color exist), the model's processing performance for images of that color will be relatively poor.

[0075] Therefore, basic information statistics can be performed on the problem example information (such as test data set, training data set).

[0076] Exemplarily, the basic information may include but is not limited to basic image information.

[0077] The basic image information may include, but is not limited to, one or more of image size information (such as image width and height), image brightness information, and image color information.

[0078] Accordingly, the basic information statistical results may include but are not limited to the image basic information statistical results.

[0079] The statistical results of basic image information may include, but are not limited to, one or more of image size distribution information (such as image width and height), image brightness distribution information, and image color distribution information.

[0080] In addition, considering tasks such as target detection or target recognition, the basic information distribution of targets in the training data set and the test data set will also affect the processing performance of the model.

[0081] For example, if the proportion of objects of a certain size in the training data set and the test data set is too high, the model will have better processing performance for objects of that size, but will have relatively low processing performance for objects of other sizes.

[0082] Accordingly, when there is a designated target in the image, for example, the image is a target that needs to be detected, and the designated target is a target to be detected, the basic information may also include target basic information, and the basic information statistical results may also include target basic information statistical results.

[0083] Exemplarily, the target basic information statistics may include, but are not limited to, one or more of target size (such as target detection box width and height) distribution information, target brightness distribution information, and target color distribution information.

[0084] In one example, intelligent statistics of question example information may include:

[0085] Perform scene / point distribution statistics on the problem example information to obtain scene / point distribution statistics results.

[0086] For example, the distribution of image scenes / points in the training data set and the test data set will also affect the model processing performance.

[0087] For example, when the image scenes in the training data set and the test data set are unevenly distributed, the processing performance of images of scenes with a smaller proportion will be poor.

[0088] For example, taking vehicle detection as an example, if the proportion of samples of vehicles on the road in the training data set is too large, the model's performance in identifying vehicles in parking lots may be relatively poor.

[0089] Accordingly, scene / point distribution statistics can be performed on the problem example information to obtain scene / point distribution statistics results to assist in determining the source of the problem.

[0090] In one example, intelligent statistics of question example information may include:

[0091] Extended information statistics are performed on the question example information to obtain extended information statistical results; wherein the extended information statistical results include information obtained by performing information mining on the image based on the image content understanding and reasoning ability.

[0092] For example, in order to provide more accurate and comprehensive question guidance, it is also possible to perform statistics on extended information (also called high-order information) of the question example information based on the image content understanding and reasoning capabilities.

[0093] For example, for a picture of fruit, the pre-trained model for understanding the content of the picture (a large model with a certain level of understanding that is trained using relevant picture knowledge data) can be used to infer high-level information such as the climate in which the fruit grows, where it is mostly grown, and how sweet it is. This can help provide more accurate answers based on the questions asked by the user.

[0094] In some embodiments, the above-mentioned description feature information of the question and the suggestion feature information, performing similar searches in the knowledge base, and determining answer guidance may include:

[0095] Based on the feature information of the problem description and the feature information of the suggestion prompt, a similar search is performed in the knowledge base to obtain the search results;

[0096] Obtain the top N search results in descending order of similarity, and output answer guidance based on the top N search results; where N≥1;

[0097] Among them, for any answer guidance, an example of the reason for the guidance is outputted in association.

[0098] Exemplarily, when the problem description feature information and the suggestion feature information are obtained in the above manner, a similar search can be performed in the knowledge base based on the problem description feature information and the suggestion feature information to obtain a search result.

[0099] Exemplarily, for the obtained search results, answer guidance can be output according to the order of similarity (similarity between the search results and the search conditions (i.e., question description feature information, and suggestion prompt feature information)) from high to low, based on the top N (N≥1) search results, i.e., the Top N search results.

[0100] Exemplarily, in order to enable the user to understand the answer guidance more intuitively, for any output answer guidance, an example of the reason for the output guidance may also be associated.

[0101] For example, assuming the problem is target false detection, the answer guidance determined in the above manner includes "adding training data for a certain scenario to optimize the training of the model", and the corresponding guidance reason example may be "the proportion of training data for this scenario in the training set is too low".

[0102] For another example, assuming the problem is target missed detection, the answer guidance determined in the above manner includes "there are missed labels in the training set, modify the training set annotations", and the corresponding guidance reason example may include "missing labeled sample images in the training set".

[0103] In some embodiments, the problem-guided interaction solution provided by the embodiments of the present application may also include:

[0104] Get feedback on answer guidelines;

[0105] When it is determined based on the feedback information that the update conditions of the multimodal large model are met, the multimodal large model is fine-tuned and adversarial training is performed on the multimodal large model to obtain an updated multimodal large model, and the updated multimodal large model is used to replace the multimodal large model before the update.

[0106] Exemplarily, a user feedback function may also be provided in the embodiments of the present application to collect user feedback information regarding answer guidance.

[0107] Exemplarily, the feedback information may include whether the user has solved the described problem (positive feedback) or has not solved the described problem (negative feedback) through the suggested solution.

[0108] Exemplarily, it is possible to determine whether the conditions for updating the multimodal large model are met based on the feedback information obtained regarding the answer guidance, and if it is determined that the conditions for updating the multimodal large model are met, perform fine-tuning adversarial training on the multimodal large model to obtain an updated multimodal large model, and use the updated multimodal large model to replace the multimodal large model before the update, so as to improve the understanding ability of the multimodal model, make it closer to the user's intention, and help solve the problem more accurately.

[0109] In some embodiments, determining whether the multimodal large model update condition is satisfied according to the feedback information may include:

[0110] When the amount of feedback information obtained reaches a first amount, it is determined that the multimodal large model update condition is met.

[0111] Exemplarily, when the amount of feedback information obtained reaches a preset amount (which may be referred to as a first amount), it can be determined that the multimodal large model update condition is met, and the multimodal large model is fine-tuned for adversarial training.

[0112] In some other embodiments, the above-mentioned determination based on the feedback information that the multimodal large model update condition is satisfied may include:

[0113] When the amount of negative feedback information obtained reaches a second amount, it is determined that the multimodal large model update condition is met.

[0114] For example, considering that there is a large amount of negative feedback information, it usually means that the accuracy of the answer guidance given by the current multimodal large model needs to be improved. Therefore, it can be determined whether update training of the multimodal large model is needed based on the amount of negative feedback information obtained.

[0115] Accordingly, when the amount of negative feedback information obtained reaches a preset amount (which may be referred to as a second amount), it can be determined that the multimodal large model update conditions are met, and the multimodal large model can be fine-tuned for adversarial training.

[0116] In some other embodiments, the above-mentioned determination based on the feedback information that the multimodal large model update condition is satisfied may include:

[0117] When the proportion of negative feedback information in the obtained feedback information exceeds a preset proportion threshold and the amount of obtained feedback information reaches a third amount, it is determined that the multimodal large model update condition is met.

[0118] For example, considering that there is a lot of negative feedback information, it may be caused by the fact that the total amount of feedback information obtained is particularly large. In this case, the proportion of negative feedback information may be very low, which does not indicate that there are obvious problems with the multimodal large model.

[0119] In addition, when the amount of feedback information obtained is small, there may be a situation where the amount of negative feedback information is extremely small, but still accounts for a relatively high proportion.

[0120] For example, if 5 pieces of feedback are obtained, including 1 negative piece of feedback, the negative feedback accounts for 20%. However, due to the small number of feedback pieces, the actual significance of this percentage is low.

[0121] Therefore, in order to more reasonably implement the update training of the multimodal large model, when the proportion of negative feedback information in the obtained feedback information exceeds the preset proportion threshold and the number of obtained feedback information reaches a preset number (which can be called the third number), it can be determined that the update conditions of the multimodal large model are met, and the multimodal large model can be fine-tuned and adversarial training can be performed.

[0122] In one example, the user input information includes text question description information and question example information; the question example information includes image information related to the text question description.

[0123] The above-mentioned fine-tuning adversarial training of the multimodal large model may include:

[0124] Constructing a sample data set based on the acquired user input information, the suggestion prompt information corresponding to the user input information, the answer guidance corresponding to the user input information, and the acquired feedback information; wherein, for any sample, it includes the associated user input information, suggestion prompt information, and feedback information; a sample including positive feedback information is a positive sample, and a sample including negative feedback information is a negative sample;

[0125] Based on the sample dataset, the multimodal large model is fine-tuned and adversarially trained.

[0126] For example, in the process of fine-tuning adversarial training of a large multimodal model, a sample data set can be constructed based on the accumulated user input information, the suggestion prompt information corresponding to the user input information, the answer guidance corresponding to the user input information, and the obtained feedback information.

[0127] Among them, different from the previous question-guided data format (input, output), in the embodiment of the present application, the data format is a triple form: (input, inter-reason, output).

[0128] Input is user input information, which can include text problem description information and problem example information.

[0129] For example, iput may include: the question "How to solve this missed target?" and the corresponding missed target image.

[0130] Inter-reason is the intermediate state knowledge associated through the intelligent statistical module, such as the above-mentioned suggestion prompt information.

[0131] For example, the inter-reason may include relevant information such as the target size and target brightness obtained through intelligent statistics, and obtain certain guiding descriptions, such as text descriptions such as the target is too small or the target is too dark.

[0132] For example, a text description library corresponding to the image target distribution can be constructed based on experience.

[0133] For example, if the brightness of the target is lower than a preset brightness threshold, the corresponding text description "the target is too dark" may be used.

[0134] Exemplarily, by combining input information and intermediate intelligent analysis information, more accurate answer guidance can be output.

[0135] The output may include answer instructions corresponding to the user input information and user feedback information.

[0136] The (input, inter-reason, output) triplet data will be used as input for fine-tuning adversarial training to further learn and enhance the model's cognitive understanding ability.

[0137] Exemplarily, during the fine-tuning training process, the incremental data can be preprocessed to a certain extent. For example, natural language processing tools such as word segmentation can be used to perform text segmentation, main content extraction, manual calibration and correction (to prevent users from filling in feedback results arbitrarily or incorrectly) on the incremental data.

[0138] Fine-tuning adversarial training is a knowledge transfer technology that can quickly learn new knowledge and adapt to specific tasks without forgetting the original knowledge.

[0139] Exemplarily, in the process of fine-tuning adversarial training of a large multimodal model, if the user input information includes image information (such as the above-mentioned question example information), that is, there is an example image as supplementary information for the question-answer guidance, then both the image encoding branch and the text encoding branch can be fine-tuned; if the user input information does not include image information (such as the above-mentioned question example information), that is, there is no new image information and only the accumulation of question-answer guidance is included, then the image-encoder branch can be fixed and the text-encoder branch can be fine-tuned.

[0140] As an example, the fine-tuning adversarial training of the multimodal large model based on the sample data set may include:

[0141] For any sample, the text encoding branch of the multimodal large model is used to extract features of the text problem description information included in the sample data to obtain embedded features of the problem description, and the text encoding branch of the multimodal large model is used to extract features of the suggestion prompt information included in the sample data to obtain embedded features of the suggestion prompt information;

[0142] The embedded features of the problem description and the embedded features of the suggestion information are fused to obtain the representative features of the problem;

[0143] Align the problem representative features and image feature maps in the same feature space to obtain the problem description representative features and embedded image features; wherein the image features are obtained by extracting features of the image in the problem example information included in the sample using the image encoding branch of the multimodal large model;

[0144] The representative features of the problem description and the embedded image features are fused to obtain new representative features of the problem;

[0145] According to the new question representative features corresponding to each sample, as well as the answer guidance text features, a joint comparative training method is adopted to perform fine-tuning adversarial training on the multimodal large model; among which, the answer guidance text features corresponding to the sample are obtained by extracting the answer guidance included in the sample using the text encoding branch of the multimodal large model.

[0146] Exemplarily, in the process of fine-tuning adversarial training of the multimodal large model, for any sample in the constructed sample data set, the text encoding branch of the multimodal large model can be used to perform feature extraction on the text problem description information included in the sample data to obtain the embedded features of the problem description, and the text encoding branch of the multimodal large model can be used to perform feature extraction on the suggestion prompt information included in the sample data to obtain the embedded features of the suggestion prompt information, and the embedded features of the problem description and the embedded features of the suggestion prompt information are feature fused to obtain the problem representative features.

[0147] Exemplarily, the method of fusing the embedded features of the problem description and the embedded features of the suggestion prompt information may include but is not limited to concatenation, addition, weighted fusion or cross-attention mechanism fusion.

[0148] For example, the embedded features of the problem description and the embedded features of the suggestion information can be concatenated to obtain the representative features of the problem.

[0149] When the problem representative feature is obtained, the problem representative feature and the image feature map can be aligned in the same feature space to obtain the problem description representative feature and the embedded image feature, and the problem description representative feature and the embedded image feature are feature fused to obtain a new problem representative feature.

[0150] Exemplarily, the method of fusing the problem description representative features and the embedded image features may include but is not limited to concatenation, addition, weighted fusion or cross-attention mechanism fusion.

[0151] For example, the problem description representative features and the embedded image features can be weighted fused to obtain new problem representative features.

[0152] Exemplarily, for any sample, on the one hand, new representative features of the question can be determined in the above manner, and on the other hand, the text encoding branch of the multimodal large model can be used to extract features of the answer guidance included in the sample to obtain the answer guidance text features corresponding to the sample.

[0153] Furthermore, based on the new question representative features corresponding to each sample and the answer guidance text features, a joint comparative training method can be used to perform fine-tuning adversarial training on the multimodal large model.

[0154] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application are described below with reference to specific examples.

[0155] In this embodiment, considering that the traditional problem guidance model usually has a relatively simple input, and only supports a single input form of text description or image data, the effect is poor for more complex and professional tasks. For example, for the analysis of detection results, more data information is needed to analyze specific performance issues, and the traditional problem guidance model cannot be simply migrated and used.

[0156] In addition, considering that the answer guidance given in the traditional problem guidance model usually has a large amount of data and is not specific enough, it is difficult to obtain a quick solution to a specific problem, and the cost of trial and error one by one is very high.

[0157] In view of the above problems, the present application embodiment provides a question guidance interaction solution, and the specific implementation process is as follows:

[0158] S1: Build an initial question suggestion library (i.e. the above-mentioned knowledge base).

[0159] Exemplarily, in the initially constructed knowledge base, each question may include one or more suggestions and corresponding examples for subsequent question retrieval.

[0160] For example, for the question "How to solve the problem of missed target detection?", the corresponding suggestions may include: Suggestion 1, there are missed targets in the training set, modify the training set annotation; Suggestion 2, there are no similar samples, please add such samples.

[0161] S2. Receive user input information, which may include text question description information and question example information.

[0162] For example, assuming that the text problem is described as "How to solve the problem of false detection of targets in images", the problem example information may include: false examples (such as images where false detection of targets occurs, as well as information such as false detection boxes and false alarm categories in the images) and training and testing data, etc.

[0163] It should be noted that in the embodiments of the present application, the user input information may not include question example information. In this case, the user input information may be a single input of text question description information. Feature extraction may be performed on the text question description information to obtain question description feature information, and the question description feature information may be used as the input of the multimodal large model. Based on the question description feature information, the multimodal large model may be used to perform similar searches in the knowledge base to determine answer guidance.

[0164] S3. Processing of user input information.

[0165] Exemplarily, the user input information processing may include text question description information processing and question example information processing.

[0166] S3-1. Text problem description information processing.

[0167] Exemplarily, natural language processing technology and a pre-trained multimodal model may be used to extract features from textual question description information.

[0168] For example, natural language processing can be used to extract key information from textual question descriptions, and pre-trained multimodal models can be used to extract features from key information to facilitate subsequent question guidance.

[0169] S3-2. Problem example information processing.

[0170] For example, the problem example information can be intelligently analyzed to mine the rich information of the problem itself, help locate the problem more accurately and give reasonable and executable suggestions.

[0171] Exemplarily, the question example images, training and testing data sets, etc. input by the user can be processed.

[0172] For example, intelligent statistics are performed on the training data set and the test data set, such as target size, missed or mislabeled diagnosis results, scene distribution, etc., and text suggestion descriptions are given to assist in determining the source of the problem in the input problem example image.

[0173] For example, for the problem of missed target detection, similar missed targets can be found in the missed and mislabeled diagnosis results through model feature similarity matching, and a prompt message can be output: "There are similar missed targets in the training set", and relevant examples can be output to guide users to make modifications.

[0174] For another example, suppose the user inputs a regional map and the question description is "What is fun in this place?", we can analyze the regional schematic diagram input by the user to obtain a number of intelligent statistical information such as the schematic diagram area, seasonal average temperature, and various tourist attractions in the area, to assist the question description and provide users with accurate and comprehensive question guidance.

[0175] The following is a detailed description of the intelligent statistics implementation process.

[0176] Exemplarily, intelligent statistics may include image / target basic information statistics, image scene / point distribution statistics, and high-order information statistics (ie, extended information statistics).

[0177] 1) Basic information statistics of images / targets.

[0178] Exemplarily, the basic image information may include image size, image clarity / brightness, and image color, etc.

[0179] The basic target information may include target size, target clarity / brightness, and target color.

[0180] For example, statistics may be collected on the size, clarity / brightness, and color of the image / object.

[0181] Exemplarily, the above statistics can be implemented by using operators such as the Laplace operator, a custom-designed image brightness discrimination operator, etc.

[0182] The statistics of these basic information can provide certain auxiliary guidance for subsequent problem analysis.

[0183] For example, if a target is missed, the statistical target size can be used to determine whether the target is too small or the target clarity can be used to determine whether the target is too blurry.

[0184] For example, the color of the target can be used to help determine the type and other attributes of some objects. For example, the colors of different types of fruits may also vary, and the color of river water can help determine the water quality.

[0185] 2) Scene / point distribution statistics of the image.

[0186] Exemplarily, a pre-trained multimodal large model can be used to extract features from images, and then clustering technology can be used to cluster the features into scenes / points to obtain the scene / point distribution of the image, thereby obtaining conclusions on whether the scene / point distribution is uneven and which scene / point the example image in the question belongs to, to assist in answering subsequent questions.

[0187] 3) High-level information statistics.

[0188] For example, the implicit information in a picture may not be obtained by simple statistical methods. It is necessary to combine a large pre-trained model for understanding picture content (with certain picture content understanding and reasoning capabilities) to mine some intrinsic information.

[0189] For example, for a picture of fruit, basic information such as color, appearance, size, etc. can be obtained through basic information. The pre-trained model for image content understanding (a large model with a certain understanding ability trained using relevant image knowledge data) can be used to infer high-level information such as the climate it is suitable for growing in, where it is mostly grown, and how sweet it is. This can help to provide more accurate recommended answers based on the questions asked by the user.

[0190] S4. Question retrieval.

[0191] For example, a similar search may be performed in the constructed knowledge base based on the problem description feature information obtained in S3-1 and the suggestion prompt information obtained in S3-2.

[0192] For example, assuming there is a problem of missed detection of a target, based on the "text description of the missed detection problem" and the inferred "similar missed labels in the training set", the feature similarity can be matched to the "similar missed labels in the training set, and it is necessary to supplement the similar missed labels in the training set to solve this type of missed detection problem". At the same time, the location of the suspected missed labels in the data set can be given, which is convenient for users to quickly locate the missed labels and modify them.

[0193] For another example, for the question of "recommending tourist attractions", combined with the corresponding pictures, intelligent statistics can be used to come up with "the average temperature of this place in May is 25 degrees. According to the number of tourists or other information, the popular tourist attraction is XXX". The similar features can be matched to the top results in the knowledge base such as "when is the climate at this tourist attraction pleasant and popular among tourists? It is recommended to visit this tourist attraction".

[0194] S5. Question suggestions (i.e. the answer guidelines mentioned above) are integrated and output.

[0195] For example, the search results may be sorted according to the similarity to obtain the top N.

[0196] For example, for each question suggestion, an example of the reason for the output guidance can be associated to facilitate the user's intuitive understanding.

[0197] S6, feedback iteration trigger.

[0198] Exemplarily, an automatic feedback collection pool (which may be called an opinion pool) may be set up. For example, if the user solves the described problem / is not solved through the suggested solution, the feedback opinions may be automatically collected through manual interactive feedback.

[0199] Exemplarily, a trigger threshold can be set. When the number of automatically collected feedback opinions reaches a preset value, the feedback opinions in the current opinion pool are automatically triggered to perform multimodal large model fine-tuning adversarial training. Regularly iterating and updating the multimodal large model will help improve the understanding ability of the multimodal large model, be closer to user intentions, and help solve problems more accurately.

[0200] The following is an explanation of the implementation of fine-tuning adversarial training for a large multimodal model.

[0201] 1) Fine-tune the training data source.

[0202] Exemplarily, the source of fine-tuning training data may include accumulated user-input questions and corresponding images (i.e., textual question description information and question example information), as well as the guiding answers currently given by the guidance system and the corresponding feedback from users (used to distinguish whether the guiding answers given by the system are reasonable, so as to select appropriate positive and negative answer adversarial samples).

[0203] For example, unlike the previous question-guided data format (input, output), the data in the embodiment of the present application is in the form of a triple (input, inter-reason, output).

[0204] Input is user input information, which can include text problem description information and problem example information.

[0205] For example, iput may include: the question "How to solve this missed target?" and the corresponding missed target image.

[0206] Inter-reason is the intermediate state knowledge associated through the intelligent statistical module, such as the above-mentioned suggestion prompt information.

[0207] For example, the inter-reason may include relevant information such as the target size and target brightness obtained through intelligent statistics, and obtain certain guiding descriptions, such as text descriptions such as the target is too small or the target is too dark.

[0208] For example, a text description library corresponding to the image target distribution can be constructed based on experience.

[0209] For example, if the brightness of the target is lower than a preset brightness threshold, the corresponding text description "the target is too dark" may be used.

[0210] Exemplarily, by combining input information and intermediate intelligent analysis information, more accurate answer guidance can be output.

[0211] The output may include answer instructions corresponding to the user input information and user feedback information.

[0212] The (input, inter-reason, output) triplet data will be used as input for fine-tuning adversarial training to further learn and enhance the model's cognitive understanding ability.

[0213] 2) Fine-tuning training (which can be called incremental supervised fine-tuning training) process.

[0214] Exemplarily, during the fine-tuning training process, the incremental data may be preprocessed to a certain extent. For example, natural language processing tools such as word segmentation may be used to perform text segmentation, main content extraction, manual calibration and correction on the incremental data.

[0215] Fine-tuning adversarial training is a knowledge transfer technology that can quickly learn new knowledge and adapt to specific tasks without forgetting the original knowledge.

[0216] Exemplarily, in the process of fine-tuning adversarial training of a large multimodal model, if the user input information includes image information (such as the above-mentioned question example information), that is, there is an example image as supplementary information for the question-answer guidance, then both the image encoding branch and the text encoding branch can be fine-tuned; if the user input information does not include image information (such as the above-mentioned question example information), that is, there is no new image information and only the accumulation of question-answer guidance is included, then the image-encoder branch can be fixed and the text-encoder branch can be fine-tuned.

[0217] Exemplarily, in the process of fine-tuning adversarial training of the multimodal large model, for any sample in the constructed sample data set, the text encoding branch of the multimodal large model can be used to perform feature extraction on the text problem description information included in the sample data to obtain the embedded feature text_f1 of the problem description, and the text encoding branch of the multimodal large model can be used to perform feature extraction on the suggestion prompt information included in the sample data to obtain the embedded feature text_f2 of the suggestion prompt information, and feature fusion of text_f1 and text_f2, such as feature splicing, can be performed to obtain the problem representative feature text_fm.

[0218] For example, the features of text_f1 and text_f2 can be fused as follows:

[0219] text_fm=concat(text_f1, text_f2)

[0220] Among them, concat is the concatenation function.

[0221] When the problem representative feature text_fm is obtained, the problem representative feature text_fm and the image feature img_f mapping can be aligned in the same feature space to obtain the problem description representative feature text_fm_emb and the embedded image feature img_f_emb.

[0222] For example, a fully connected layer may be used to align the problem representative feature text_fm and the image feature img_f mapping in the same feature space.

[0223] The problem description representative feature text_fm_emb and the embedded image feature img_f_emb can be fused to obtain a new problem representative feature qur_f_emb.

[0224] For example, the feature fusion of the problem description representative feature text_fm_emb and the embedded image feature img_f_emb can be performed as follows:

[0225] qur_f_emb=a2* text_fm_emb + b2*img_f_emb

[0226] Among them, a2 and b2 are weighting coefficients.

[0227] Exemplarily, the answer guidance corresponding to the question can also be extracted through the text-encoder branch to extract its text feature ans_f_emb.

[0228] The new question representative feature qur_f_emb and the feature ans_f_emb corresponding to the answer to the question are jointly trained and learned by comparison, that is, similarity comparison loss is constructed for the embedding information of the two:

[0229] l_sim=f(qur_f_emb,ans_f_emb)

[0230] Among them, for positive samples, the higher the similarity, the smaller the loss; for negative samples, the higher the similarity, the greater the loss.

[0231] The network parameters are optimized based on the determined loss back-propagation, achieving cognitive understanding of the model enhanced by incremental data.

[0232] Exemplarily, the fine-tuned trained multimodal large model is used for subsequent problem guidance. As incremental data accumulates and reaches a certain amount, steps 1) and 2) are iterated again to gradually increase the model's understanding ability.

[0233] For example, the overall flow chart of the question-guided interaction can be as follows: Figure 2 shown.

[0234] The method provided by the present application is described above. The device provided by the present application is described below:

[0235] See also Figure 3 , is a structural diagram of a question guidance interaction device provided in an embodiment of the present application, such as Figure 3 As shown, the question-guiding interaction device may include:

[0236] The acquisition unit 310 is used to acquire user input information;

[0237] A first feature extraction unit 320 is used to extract features from the textual question description information to obtain question description feature information when the user input information includes textual question description information and question example information;

[0238] An intelligent analysis unit 330 is used to perform intelligent statistics on the problem example information and determine the suggestion prompt information according to the intelligent statistical results;

[0239] The second feature extraction unit 340 is further used to extract features from the suggestion prompt information to obtain suggestion prompt feature information; wherein the suggestion prompt information is used to assist in determining the source of the problem;

[0240] A guidance unit 350 is used to perform similarity search in the knowledge base based on the question description feature information and the suggestion prompt feature information using a multimodal large model to determine an answer guide;

[0241] Among them, the multimodal large model is updated by performing fine-tuning training under the condition of satisfying the large model update conditions, and the sample data for fine-tuning training of the multimodal large model is constructed based on the user input information and the feedback information of the answer guidance; during the fine-tuning training of the multimodal large model, the training is performed by joint comparative training based on the feature information corresponding to the user input information and the feature information corresponding to the feedback information.

[0242] In some embodiments, the first feature extraction unit 320 extracts features from the textual question description information to obtain question description feature information, including:

[0243] Extract key information from the problem description using natural language processing to obtain key information in the text problem description information;

[0244] The key information is feature extracted using a multimodal large model to obtain problem description feature information.

[0245] In some embodiments, the question example information includes image information related to the text question description;

[0246] The intelligent analysis unit 330 performs intelligent statistics on the problem example information, including:

[0247] Performing basic information statistics on the problem example information to obtain basic information statistics results; wherein the basic information statistics results include image basic information statistics results, and the image basic information statistics results include one or more of image size distribution information, image brightness distribution information, and image color distribution information; when there is a designated target in the image, the basic information statistics results also include target basic information statistics results, and the target basic information statistics results include one or more of target size distribution information, target brightness distribution information, and target color distribution information;

[0248] and / or,

[0249] Perform scene / point distribution statistics on the problem example information to obtain scene / point distribution statistics results;

[0250] and / or,

[0251] Perform extended information statistics on the question example information to obtain extended information statistical results; wherein the extended information statistical results include information obtained by performing information mining on the image based on image content understanding and reasoning capabilities.

[0252] In some embodiments, the guidance unit 350 performs similarity search in the knowledge base based on the question description feature information and the suggestion prompt feature information to determine the answer guidance, including:

[0253] Based on the problem description feature information and the suggestion prompt feature information, a similar search is performed in the knowledge base to obtain a search result;

[0254] Obtain the top N search results in descending order of similarity, and output answer guidance based on the top N search results; wherein N≥1;

[0255] Among them, for any answer guidance, an example of the reason for the guidance is outputted in association.

[0256] In some embodiments, the acquisition unit 310 is further used to acquire feedback information for the answer guidance;

[0257] like Figure 4 As shown, the device also includes:

[0258] The enhanced training unit 360 is used to perform fine-tuning adversarial training on the multimodal large model when it is determined based on the feedback information that the multimodal large model update conditions are met to obtain an updated multimodal large model, and use the updated multimodal large model to replace the multimodal large model before the update.

[0259] In some embodiments, the enhanced training unit 360 determines that the multimodal large model update condition is met according to the feedback information, including:

[0260] When the amount of feedback information obtained reaches a first amount, determining that a multimodal large model update condition is met;

[0261] or,

[0262] When the amount of negative feedback information obtained reaches a second amount, determining that a multimodal large model update condition is met;

[0263] or,

[0264] When the proportion of negative feedback information in the obtained feedback information exceeds a preset proportion threshold and the amount of obtained feedback information reaches a third amount, it is determined that the multimodal large model update condition is met.

[0265] In some embodiments, the user input information includes textual question description information and question example information; the question example information includes image information related to the textual question description;

[0266] The enhanced training unit 360 performs fine-tuning adversarial training on the multimodal large model, including:

[0267] Constructing a sample data set based on the acquired user input information, the suggestion prompt information corresponding to the user input information, the answer guidance corresponding to the user input information, and the acquired feedback information; wherein, for any sample, it includes the associated user input information, suggestion prompt information, and feedback information; a sample including positive feedback information is a positive sample, and a sample including negative feedback information is a negative sample;

[0268] Based on the sample data set, fine-tuning adversarial training is performed on the multimodal large model.

[0269] In some embodiments, the enhanced training unit 360 performs fine-tuning adversarial training on the multimodal large model based on the sample data set, including:

[0270] For any sample, the text encoding branch of the multimodal large model is used to perform feature extraction on the text problem description information included in the sample data to obtain embedded features of the problem description, and the text encoding branch of the multimodal large model is used to perform feature extraction on the suggestion prompt information included in the sample data to obtain embedded features of the suggestion prompt information;

[0271] Performing feature fusion on the embedded features of the problem description and the embedded features of the suggestion prompt information to obtain a representative feature of the problem;

[0272] Align the problem representative features and image feature maps in the same feature space to obtain problem description representative features and embedded image features; wherein the image features are obtained by extracting features of the image in the problem example information included in the sample using the image encoding branch of the multimodal large model;

[0273] Performing feature fusion on the problem description representative features and the embedded image features to obtain new problem representative features;

[0274] Based on the new question representative features corresponding to each sample, as well as the answer guidance text features, a joint comparative training method is adopted to perform fine-tuning adversarial training on the multimodal large model; wherein, the answer guidance text features corresponding to the sample are obtained by extracting the features of the answer guidance included in the sample using the text encoding branch of the multimodal large model.

[0275] An embodiment of the present application provides an electronic device, including a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the question-guided interaction method described above.

[0276] See also Figure 5 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501 and a memory 502 storing machine executable instructions. The processor 501 and the memory 502 may communicate via a system bus 503. In addition, by reading and executing the machine executable instructions corresponding to the question guidance interaction logic in the memory 502, the processor 501 may execute the question guidance interaction method described above.

[0277] The memory 502 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the machine-readable storage medium may be: RAM (RadomAccess Memory, random access memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0278] In some embodiments, a machine-readable storage medium is also provided. Figure 5 The memory 502 in the machine readable storage medium stores machine executable instructions, and when the machine executable instructions are executed by the processor, the problem-guided interaction method described above is implemented. For example, the storage medium can be ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

Claims

1. A question-guided interaction method, characterized in that: include: Get user input information; In a case where the user input information includes textual question description information and question example information, performing feature extraction on the textual question description information to obtain question description feature information; and, performing intelligent statistics on the problem example information, determining suggestion prompt information based on the intelligent statistical results, and performing feature extraction on the suggestion prompt information to obtain suggestion prompt feature information; wherein the suggestion prompt information is used to assist in determining the source of the problem; the problem example information includes examples of problems that the user needs to solve, including image information related to the text problem description; Based on the characteristic information of the question description and the characteristic information of the suggestion prompt, a similar search is performed in the knowledge base using a multimodal large model to determine an answer guide; Among them, when the user input information includes textual question description information but does not include question example information, feature extraction is performed on the textual question description information to obtain question description feature information, and based on the question description feature information, a multimodal large model is used to perform similarity search in the knowledge base to determine answer guidance.

2. The method according to claim 1, characterized in that The step of extracting features from the textual problem description information to obtain problem description feature information includes: Extract key information from the problem description using natural language processing to obtain key information in the text problem description information; The multimodal large model is used to extract features of the key information to obtain problem description feature information.

3. The method according to claim 1, characterized in that The question example information includes image information related to the text question description; The intelligent statistics of the problem example information includes: Performing basic information statistics on the problem example information to obtain basic information statistics results; wherein the basic information statistics results include image basic information statistics results, and the image basic information statistics results include one or more of image size distribution information, image brightness distribution information, and image color distribution information; when there is a designated target in the image, the basic information statistics results also include target basic information statistics results, and the target basic information statistics results include one or more of target size distribution information, target brightness distribution information, and target color distribution information; and / or, Perform scene / point distribution statistics on the problem example information to obtain scene / point distribution statistics results; and / or, Perform extended information statistics on the question example information to obtain extended information statistical results; wherein the extended information statistical results include information obtained by performing information mining on the image based on image content understanding and reasoning capabilities.

4. The method according to claim 1, characterized in that The method of performing similar search in the knowledge base based on the problem description feature information and the suggestion prompt feature information to determine the answer guidance includes: Based on the problem description feature information and the suggestion prompt feature information, a similar search is performed in the knowledge base to obtain a search result; Obtain the top N search results in descending order of similarity, and output answer guidance based on the top N search results; wherein N≥1; Among them, for any answer guidance, an example of the reason for the guidance is outputted in association.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining feedback information regarding the answer guidance; When it is determined based on the feedback information that the update conditions of the multimodal large model are met, fine-tuning adversarial training is performed on the multimodal large model to obtain an updated multimodal large model, and the updated multimodal large model is used to replace the multimodal large model before the update.

6. The method according to claim 5, characterized in that The determining, based on the feedback information, whether a multimodal large model update condition is satisfied includes: When the amount of feedback information obtained reaches a first amount, determining that a multimodal large model update condition is met; or, When the amount of negative feedback information obtained reaches a second amount, determining that a multimodal large model update condition is met; or, When the proportion of negative feedback information in the obtained feedback information exceeds a preset proportion threshold and the amount of obtained feedback information reaches a third amount, it is determined that the multimodal large model update condition is met.

7. The method according to claim 5, characterized in that The user input information includes text question description information and question example information; The question example information includes image information related to the text question description; The fine-tuning adversarial training of the multimodal large model includes: Constructing a sample data set based on the acquired user input information, the suggestion prompt information corresponding to the user input information, the answer guidance corresponding to the user input information, and the acquired feedback information; wherein, for any sample, it includes the associated user input information, suggestion prompt information, and feedback information; a sample including positive feedback information is a positive sample, and a sample including negative feedback information is a negative sample; Based on the sample data set, fine-tuning adversarial training is performed on the multimodal large model.

8. The method according to claim 7, characterized in that The step of fine-tuning the multimodal large model for adversarial training based on the sample data set includes: For any sample, the text encoding branch of the multimodal large model is used to perform feature extraction on the text problem description information included in the sample data to obtain embedded features of the problem description, and the text encoding branch of the multimodal large model is used to perform feature extraction on the suggestion prompt information included in the sample data to obtain embedded features of the suggestion prompt information; Performing feature fusion on the embedded features of the problem description and the embedded features of the suggestion prompt information to obtain a representative feature of the problem; Align the problem representative features and image feature maps in the same feature space to obtain problem description representative features and embedded image features; wherein the image features are obtained by extracting features of the image in the problem example information included in the sample using the image encoding branch of the multimodal large model; Performing feature fusion on the problem description representative features and the embedded image features to obtain new problem representative features; Based on the new question representative features corresponding to each sample, as well as the answer guidance text features, a joint comparative training method is adopted to perform fine-tuning adversarial training on the multimodal large model; wherein, the answer guidance text features corresponding to the sample are obtained by extracting the features of the answer guidance included in the sample using the text encoding branch of the multimodal large model.

9. A question-guided interactive device, characterized in that: include: An acquisition unit, used for acquiring user input information; A first feature extraction unit is used to extract features from the textual question description information to obtain question description feature information when the user input information includes textual question description information and question example information; The problem example information includes an example of a problem that the user needs to solve, including image information related to the text problem description; An intelligent analysis unit, used for performing intelligent statistics on the problem example information, and determining suggestion prompt information according to the intelligent statistical results; The second feature extraction unit is further used to extract features from the suggestion prompt information to obtain suggestion prompt feature information; wherein the suggestion prompt information is used to assist in determining the source of the problem; A guidance unit, configured to perform similarity search in a knowledge base using a multimodal large model based on the feature information of the question description and the feature information of the suggestion prompt, and determine an answer guidance; The multimodal large model is updated by performing fine-tuning training under the condition that the large model update condition is met, and the sample data for fine-tuning training of the multimodal large model is constructed based on the user input information and the feedback information for the answer guidance; in the process of fine-tuning training of the multimodal large model, the joint contrast training method is adopted for training based on the feature information corresponding to the user input information and the feature information corresponding to the feedback information; The first feature extraction unit is further configured to extract features from the textual question description information to obtain question description feature information when the user input information includes textual question description information but does not include question example information; The guidance unit is also used to perform similarity search in the knowledge base based on the feature information of the question description and use a multimodal large model to determine the answer guidance.

10. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor is used to execute the machine executable instructions to implement the method according to any one of claims 1 to 9.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores machine-executable instructions, and when the machine-executable instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images