Session processing method based on large model and program product

Abstract

The embodiment of the invention provides a session processing method based on a large model and a program product, and relates to the technical field of large language models. The method comprises the following steps: receiving a session request, under the condition that the session request comprises text content and picture content, inputting the text content into an intention classification large model to obtain a first intention classification result, and inputting the picture content and the text content into an image recognition large model to obtain a picture analysis result, and determining whether the first intention classification result is a first preset intention, if not, taking the picture analysis result as the answer content of the session request, and if yes, integrating the picture analysis result and the text content and inputting the integrated result into a text large model to obtain the answer content of the session request. Therefore, the answer content can be given after the session request is comprehensively analyzed in combination with the image recognition large model and the text large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of large language model, in particular to a large model-based conversation processing method and program product. BACKGROUND

[0002] In a large model dialogue, two types of large models are usually involved, one is an image recognition large model, and the other is a text large model. Currently, when using multi-modal interaction, the two large models are usually used separately, that is, this conversation uses either an image recognition large model or a text large model.

[0003] In related technologies, if a user uploads a picture in this conversation, an image recognition large model is used, otherwise a text large model is used.

[0004] However, when a user uploads both a picture and text, the image recognition large model cannot well complete the processing and generation of the text. SUMMARY

[0005] Therefore, the embodiments of the present application aim to provide a large model-based conversation processing method and program product to at least partially improve the above problems.

[0006] To achieve the above-mentioned purpose, the technical solutions adopted by the embodiments of the present application are as follows: In a first aspect, the embodiments of the present application provide a large model-based conversation processing method, comprising: receiving a conversation request, in the case that the conversation request includes text content and picture content, inputting the text content into an intent classification large model to obtain a first intent classification result; inputting the picture content and the text content into an image recognition large model to obtain a picture analysis result; determining whether the first intent classification result is a first preset intent; If not, the picture analysis result is taken as the answer content of the conversation request; If yes, the picture analysis result and the text content are integrated and input into a text large model to obtain the answer content of the conversation request.

[0007] Optionally, the picture analysis result and the text content are integrated and input into a text large model to obtain the answer content of the conversation request, comprising: According to the text content and the picture analysis result, a corresponding target knowledge segment is retrieved in a vector knowledge base; wherein the vector knowledge base includes a plurality of knowledge segments; The target knowledge segment, the text content and the picture analysis result are integrated and input into a text large model to obtain the answer content of the conversation request.

[0008] Optionally, the picture analysis result comprises a list of information contained in the picture content, the list of information comprising a list of recipes, a list of ingredients, and a list of products, and the vector knowledge base further comprises a vector representation corresponding to each of the knowledge segments; the retrieving, according to the text content and the picture analysis result, of the corresponding target knowledge segment from the vector knowledge base comprises: combining the text content and the list of information to construct a comprehensive query vector; calculating a similarity between the comprehensive query vector and each of the vector representations to obtain a similarity score of each of the knowledge segments; taking a preset number of the knowledge segments with the highest similarity scores as the target knowledge segment.

[0009] Optionally, the picture analysis result comprises picture analysis complete content, the picture analysis complete content being an unstructured natural language description generated after the image recognition large model comprehensively understands the picture content; the inputting, after the integrating of the target knowledge segment, the text content, and the picture analysis result into a text large model, of answer content of the conversation request comprises: assembling the target knowledge segment, the text content, and the picture analysis complete content into a prompt word; based on the text large model, taking the text content as a main question, taking the text content and the picture analysis complete content as references, and performing comprehensive understanding analysis to obtain answer content of the conversation request.

[0010] Optionally, after receiving the conversation request comprising picture content, the method further comprises: inputting the picture content into the image recognition large model to obtain visual semantic information corresponding to the picture content; performing structured processing on the visual semantic information to generate a structured visual summary comprising object categories, attribute features, and spatial relationships; encoding the structured visual summary into a visual feature vector, and storing the visual feature vector, a current conversation identifier, and corresponding picture analysis result into a historical conversation record.

[0011] Optionally, the method further comprises: in a case where the conversation request only comprises text content, inputting the text content into the intent classification large model to obtain a second intent classification result; determining whether the second intent classification result is a second preset intent; if not, inputting the text content into the text large model to obtain answer content of the conversation request; If yes, based on the text content and the second intent classification result, an answer content of the conversation request is acquired.

[0012] Optionally, the second intent classification result includes a result of relying on the image recognition large model and a result of not relying on the image recognition large model; and the inputting the text content into the intent classification large model to obtain the second intent classification result includes: determining whether there is a target historical conversation record using the image recognition large model in historical conversation records of a preset number of rounds before the current round; if not, taking the result of not relying on the image recognition large model as the second intent classification result; if yes, acquiring the picture analysis result and a visual feature vector in each of the target historical conversation records; inputting the text content, the visual feature vector and the picture analysis result into the intent classification large model to obtain the second intent classification result.

[0013] Optionally, the second intent classification result further includes historical picture content; and the acquiring the answer content of the conversation request based on the text content and the second intent classification result includes: inputting the text content into the intent classification large model to obtain a first intent classification result, and inputting the historical picture content and the text content into the image recognition large model to obtain a picture analysis result; determining whether the first intent classification result is a first preset intent; if not, taking the picture analysis result as the answer content of the conversation request; if yes, inputting the picture analysis result and the text content into the text large model after integration to obtain the answer content of the conversation request.

[0014] Optionally, the intent classification large model is used to identify whether the conversation request has an intent of querying a recipe and an intent of relying on the image recognition large model; the first intent classification result includes an intent of having a recipe query and an intent of not having a recipe query; the first preset intent is the intent of having a recipe query; and the second preset intent is the intent of relying on the image recognition large model.

[0015] In a second aspect, an embodiment of the present application provides a computer program product, characterized by comprising instructions, which, when executed on an electronic device, cause the electronic device to implement the large model based conversation processing method in any of the above aspects.

[0016] The embodiment of the application provides a conversation processing method and program product based on a large model, in the case that a conversation request comprises text content and picture content, whether the conversation request has a first preset intention is judged by an intention classification large model, if not, the text content and the picture content are directly processed by an image recognition large model to obtain answer content of the conversation request, if yes, the content obtained by the image recognition large model and the text content are further processed by a text large model to obtain the answer content of the conversation request. Therefore, the image recognition large model and the text large model can be combined to give the answer content after comprehensive analysis of the conversation request.

[0017] In order to make the above objectives, features and advantages of the present application more apparent, the following will specifically describe a preferred embodiment in conjunction with the accompanying drawings, and make a detailed description as follows. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0019] Figure 1 A flowchart of a conversation processing of the prior art provided by the embodiment of the present application; Figure 2 A schematic structural block diagram of an electronic device provided by the embodiment of the present application; Figure 3 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 4 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 5 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 6 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 7 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 8 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 9 A flowchart of a conversation processing method based on a large model provided by the embodiment of the present application; Figure 10 FIG. 7 is a flowchart of a method for processing a conversation based on a large model according to an embodiment of the present disclosure; Figure 11 FIG. 8 is another flowchart of a method for processing a conversation based on a large model according to an embodiment of the present disclosure.

[0020] Icon: 100 - electronic device; 101 - memory; 102 - communication interface; 103 - processor; 104 - communication bus. DETAILED DESCRIPTION

[0021] In order to make the objectives, technical solutions, and advantages of the embodiments of the present disclosure clearer, the following will be combined with the accompanying drawings for the embodiments of the present disclosure to make a clear and complete description of the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are some but not all of the embodiments of the present disclosure. The components of the embodiments of the present disclosure described and shown in the accompanying drawings can be arranged and designed in various different configurations.

[0022] Therefore, the following detailed description of the embodiments of the present disclosure provided in the accompanying drawings is not intended to limit the scope of the claimed present disclosure, but only represents selected embodiments of the present disclosure. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in the present disclosure without creative labor are within the scope of protection of the present disclosure.

[0023] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present disclosure, the terms “first”, “second”, and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0024] It should be noted that: similar reference numerals and letters represent similar items in the following drawings, therefore, once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. Meanwhile, in the description of the present disclosure, the terms “first”, “second”, and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0025] With the rapid development of artificial intelligence technology, large models have been widely applied in multiple fields, especially in multi-modal interaction scenarios, image recognition large models and text large models have become core technologies. Image recognition large models are mainly used for processing and analyzing image data, and can identify objects, scenes and features in images, while text large models focus on natural language processing tasks such as text generation, question answering and semantic understanding. In the intelligent kitchen appliance industry, the application of these two large models is particularly important. For example, users can upload pictures to identify kitchen appliance products or dishes, and obtain usage instructions or cooking methods through text interaction.

[0026] In the prior art, the implementation of multi-modal interaction usually separates the use of image recognition large models and text large models. Specifically, referring to Figure 1 When a user uploads a picture in a session, the system calls an image recognition large model for picture analysis; if the user only inputs text, a text large model is called for text processing. Although this isolated use scheme is simple and easy to implement, it has obvious limitations in actual application. For example, in the intelligent kitchen appliance scenario, after a user uploads a picture, the system not only needs to identify the product or dish in the picture, but also needs to recall knowledge in combination with product manuals or recipes, and finally generate detailed text answers. This process requires simultaneous calling of image recognition large models and text large models, and the existing scheme cannot realize the cooperative work of the two models in one session, resulting in the inability to meet the actual needs of users.

[0027] Based on the above, the embodiment of the present application provides a large model-based session processing method and program product. The intention classification large model is used to determine whether the session request has a first preset intention. If so, the content obtained by processing the image recognition large model and the text content are integrated, and then input into the text large model for processing to obtain the answer content of the session request. Thus, the image recognition large model and the text large model can be cooperated to give the answer content after comprehensive analysis of the session request.

[0028] To realize the flow steps and functions of various examples of the present application, please refer to Figure 2 , Figure 2 A schematic structural block diagram of an electronic device provided by the embodiment of the present application is shown in FIG. 1. The electronic device 100 includes a memory 101 and a processor 103, which are directly or indirectly electrically connected to each other to realize data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses 104 or signal lines. The memory 101 can be used to store software programs and modules, and the processor 103 executes the software programs and modules stored in the memory 101 to perform various functional applications and data processing.

[0029] The electronic device 100 can be, but is not limited to, a personal computer (PC), a server, a computer in a distributed deployment, and the like. It can be understood that the electronic device 100 is not limited to a physical server, but can also be a virtual machine on a physical server, a virtual machine built on a cloud platform, and the like, which can provide a computer having the same function as the server or the virtual machine. The operating system of the electronic device 100 can be, but is not limited to, a Windows system, a Linux system, and the like.

[0030] The memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), and the like.

[0031] The communication connection between the electronic device 100 and an external device is realized through at least one communication interface 102 (which can be wired or wireless).

[0032] The processor 103 can be an integrated circuit chip having a processing capability of a signal. In the implementation process, each step of the embodiments of the present application can be completed by an integrated logic circuit or an instruction in the form of software in the processor 103. The processor 103 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), and the like; can also be a digital signal processor (Network Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a ready programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0033] It can be understood that, Figure 2 The structure shown is only schematic, and the electronic device 100 can further include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 2 The structure shown is only schematic, and the electronic device 100 can further include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 2 The structure shown is only schematic, and the electronic device 100 can further include more or fewer components than those shown in the figure, or have a different configuration from that shown in the figure. Figure 2The components shown in the middle can be implemented in hardware, software, or a combination thereof.

[0034] The following will be described in conjunction with the large model-based conversation processing method provided by the present application. Referring to Figure 3 , the method execution subject can be the above Figure 2 The electronic device 100 shown in the figure, the method includes the following steps as Figure 3 described: S210: receiving a conversation request, in the case of including text content and picture content in the conversation request, inputting the text content into an intent classification large model to obtain a first intent classification result; inputting the picture content and the text content into an image recognition large model to obtain a picture analysis result.

[0035] S220: determining whether the first intent classification result is a first preset intent.

[0036] S230: if not, taking the picture analysis result as the answer content of the conversation request.

[0037] S240: if yes, integrating the picture analysis result and the text content and inputting them into a text large model to obtain the answer content of the conversation request.

[0038] First, a conversation request from a user is received. When the conversation request is a multi-modal input containing text content (for example: “How to make the dish in this picture?”) and picture content (for example: a picture of a dish or a picture of kitchen appliances), the two parts of content are processed in parallel: On the one hand, only the text content is extracted and input into a pre-trained intent classification large model to analyze the core intent type of the user's question this time. The result output by the model is called “first intent classification result”, which can take values such as “recipe query intent” and “product function consultation intent”.

[0039] The intent classification large model can be a lightweight neural network model (such as based on BERT architecture fine-tuning) specially trained for the kitchen appliance vertical field, which can accurately identify whether the user wants to obtain information related to dish making process, food matching suggestion, kitchen appliance operation instruction, etc.

[0040] On the other hand, the picture content and the original text content are jointly input into an image recognition large model (such as a multi-modal large model based on CLIP or BLIP architecture) to obtain a deep understanding and structured analysis of the picture content.

[0041] The “picture analysis result” returned by the image recognition large model can include two parts: Picture analysis complete content: a piece of natural language form of unstructured description, for example: the picture shows a dish of braised pork, with bright color and garnished with green onions, served in a white porcelain plate.

[0042] Information list: a list of structured entities automatically extracted from the image, including: recipe list (such as braised pork), ingredient list (such as pork belly, soy sauce, sugar, ginger slices), and kitchen appliance product list (such as smart electric pressure cooker P5 model).

[0043] After obtaining the first intent classification result, logical judgment is made: whether the result belongs to the preset key knowledge service class intent, i.e. "first preset intent", for example, "recipe query intent": the user wants to know the method, cooking skills, and required seasonings of a dish; "product query intent": the user wants to know the function, use method, and cleaning and maintenance method of a kitchen appliance product.

[0044] When it is determined that the user does not have the first preset intent, it indicates that this request is only a simple visual description requirement, and there is no need to call the knowledge base or text large model for deep reasoning.

[0045] For example, the user asks: "How does this dish look?" or "Is this pot stainless steel?" Such questions only need to be answered based on the image itself. At this time, the "picture analysis complete content" returned by the image recognition large model is slightly formatted and returned to the user as the final answer.

[0046] When it is determined that the user has the "first preset intent", it indicates that the request involves deep knowledge understanding and comprehensive generation tasks, and the image-text fusion enhanced generation process needs to be started.

[0047] In this case, the image recognition result is not directly returned, but the picture analysis result and the text content are integrated and input into the text large model for further processing to obtain the answer content of the conversation request.

[0048] This method uses the intent classification large model to deploy the image recognition large model and the text large model to work together, which can better process the text content and picture content of the conversation request, and thus can better generate the answer content required by the conversation request.

[0049] In an optional implementation, referring to Figure 4 , step S240 can include the following steps: S241: retrieving the corresponding target knowledge fragment in the vector knowledge base according to the text content and the picture analysis result; wherein the vector knowledge base includes a plurality of knowledge fragments.

[0050] S242: input the integrated target knowledge fragments, text content and picture analysis results into a text large model to obtain answer content of the conversation request.

[0051] When the intent classification large model determines that the current conversation request has a first preset intent (for example, a kitchen appliance related professional intent such as "recipe query" or "product use consultation"), the system enters a knowledge recall stage.

[0052] The knowledge recall process includes: using the original text content input by the user and the list of structured information in the picture analysis result returned from the image recognition large model to jointly construct a semantic-rich query request, which is used for similarity matching in a pre-constructed vector knowledge base, so as to detect the most relevant knowledge fragments.

[0053] Among them, the picture analysis result can include a list of information contained in the picture content, and the list of information includes a recipe list, an ingredient list and a product list. The vector knowledge base can also include a vector representation corresponding to each knowledge fragment. See Figure 5 , step S241 can include the following steps: S2411: combine the text content and the list of information to construct a comprehensive query vector.

[0054] S2412: calculate the similarity of the comprehensive query vector and each vector representation to obtain a similarity score of each knowledge fragment.

[0055] S2413: take a preset number of knowledge fragments with the highest similarity score as target knowledge fragments.

[0056] Knowledge recall is not only dependent on the text content input by the user, but also utilizes the structured information list in the text semantic information and image recognition result to improve the accuracy of retrieval.

[0057] Among them, the text content is the question or instruction input by the user, such as "how to make this dish?" and "how to defrost this microwave oven?"; the list of information is extracted and formatted by the image recognition large model, and at least includes one or more of the following: a recipe list (such as ["red cooked pork"]), an ingredient list (such as ["pork belly", "soy sauce", "sugar", "ginger"]), and a product list (such as ["microwave oven"]).

[0058] In order to fully utilize the entity-level information of the above-mentioned visual end, the present embodiment proposes to construct a comprehensive query vector. The construction process is as follows: The user input text content is input into a pre-trained embedding model to generate a corresponding text vector; the recipe list, ingredient list and / or product list in the picture analysis result are spliced into a structured sentence (for example: "detected dish: braised pork; main ingredients: pork belly, soy sauce, sugar; equipment model: microwave oven"), which is then converted into a visual semantic vector by the same embedding model; and the two vectors are weighted and fused to obtain a comprehensive query vector.

[0059] The vector knowledge base is a pre-constructed professional knowledge database that stores a large number of document slices in the kitchen electrical field and their corresponding vector representations. These knowledge pieces come from: official manuals of intelligent kitchen electrical products, standardized electronic recipe datasets, cooking techniques and safety operation manuals, and user FAQ libraries.

[0060] Each knowledge piece is converted into a fixed-dimensional vector representation by the same embedding model and indexed in a vector database that supports efficient approximate nearest neighbor search.

[0061] In step S2412, the cosine similarity between the comprehensive query vector and all knowledge piece vectors in the vector knowledge base is calculated, and each knowledge piece is assigned a similarity score.

[0062] After obtaining all the similarity scores, the scores are sorted from high to low, and the top K most relevant knowledge pieces are selected as the target knowledge piece set, where K is a configurable parameter (for example, K=3) that can be dynamically adjusted according to actual performance requirements.

[0063] After completing the knowledge piece recall, the final answer generation stage is entered. The core task of this stage is to effectively fuse multi-source heterogeneous information and guide the text large model to generate professional, coherent and user demand-compliant answers based on all contexts. The picture analysis result can also include the picture analysis complete content, which is an unstructured natural language description generated by the image recognition large model after comprehensively understanding the picture content. Referring to Figure 6 , step S242 can include the following steps: S2421: Assemble the target knowledge piece, text content and picture analysis complete content into a prompt word.

[0064] S2422: Based on the text large model, take the text content as the main question and the text content and picture analysis complete content as the reference to perform comprehensive understanding analysis to obtain the answer content of the conversation request.

[0065] First, the prompt word is assembled, and the following three types of information are integrated into a structured prompt (Prompt) for the text large model: user original question (text content), "picture analysis complete content" in the image recognition result, and "target knowledge piece" retrieved.

[0066] For example, the prompt word construction example is as follows

user question

Picture description

Related knowledge

[0067] Then, comprehensive understanding and generation are performed, and the above prompt words are input into the text large model to guide it to generate a natural, fluent, accurate and detailed answer with the user question as the core, referring to the text and image information and professional knowledge.

[0068] Example answer: According to the picture you uploaded, this dish of braised pork looks almost ready. If you want to recreate this dish, follow these steps: first, blanch the five-spice pork to remove the smell……finally, reduce the sauce on high heat.

[0069] The method is further described below, see Figure 7 , by asking the user how to make the dish in the picture, the text content in the conversation request is "how to make the dish in this picture", and the picture content is a dish. First, get the text content in the conversation request and the picture content, input the text content into the intent classification large model for semantic recognition, and the intent classification large model identifies whether the user's conversation request this time is the intent of recipe query. At the same time, input the text content and the picture content into the image recognition large model, and the image recognition large model returns the complete content of the analysis of the text content and the picture content, and extracts the recipe list and the list of ingredients in the picture. When the intent classification large model identifies that the user's conversation request this time does not have the intent of recipe query, directly return the complete content of the analysis of the text content and the picture content returned by the image recognition large model. If this conversation request is a recipe query intent, retrieve the relevant knowledge fragments from the vector knowledge base according to the recipe list and the list of ingredients extracted by the image recognition large model. At this time, multiple text information has been obtained, including the original text content, the complete content of the analysis returned by the image recognition large model, and the relevant knowledge fragments. Assemble these three into prompt words and input them into the text large model. The text large model will comprehensively understand and analyze these information, and finally give the final fused answer content.

[0070] A series of context modeling and state storage operations are performed to support multi-round cross-modal question answering, and image information is converted into a structured intermediate representation that can be long-term preserved and easily analyzed for subsequent semantic association analysis, thereby providing a technical basis for triggering image-text fusion responses when the user asks follow-up questions in pure text. Accordingly, in one possible implementation, after receiving a session request including picture content, the method can further include the steps of: S310: inputting the picture content into an image recognition large model to obtain visual semantic information corresponding to the picture content.

[0071] S320: structurally processing the visual semantic information to generate a structured visual summary containing object categories, attribute features, and spatial relationships.

[0072] S330: encoding the structured visual summary into a visual feature vector, and storing the visual feature vector, the current session identifier, and the corresponding picture analysis result in a historical session record.

[0073] After inputting the picture content uploaded by the user into the image recognition large model, a set of rich visual semantic information is outputted after comprehensive analysis of the picture, including the following contents: object categories appearing in the image (such as “stewed pork”, “air fryer”, “electric rice cooker display”), key attribute features of each object (such as “color is reddish-brown”, “surface has oil”, “display number ‘15’”), spatial relationships between objects (such as “spatula is located to the right of the dish”, “microwave oven door is in an open state”), and overall scene semantic description (unstructured natural language description, such as “this is a dish of stewed pork being cooked using a microwave oven”).

[0074] To further improve the readability and retrievability of visual information and avoid the subsequent modules directly relying on natural language descriptions that are difficult to analyze, the application introduces a generation mechanism of structured visual summary. Through a lightweight semantic analysis module (which can be a rule engine, a template matching system, or a small sequence labeling model), the visual semantic information obtained in the previous step is structured and organized to generate a standardized data format, such as JSON or triple form. The structured visual summary explicitly contains three core elements: 1. object category: the main entity identified; 2. attribute feature: observable attributes such as color, state, and numerical value of each object; 3. spatial relationship: relative position or action association between objects.

[0075] Compared with the original natural language description, the structured summary has stronger machine readability, higher information density, and lower ambiguity, which is beneficial to the subsequent modules to quickly locate key visual clues.

[0076] To realize cross-round, cross-modal information memory and rapid comparison, the structured visual summary is further converted into a visual feature vector to facilitate semantic similarity calculation in a high-dimensional space. Subsequently, the visual feature vector, a unique identifier of the current session, and the complete result of the current picture analysis are written as an entry into the historical session record.

[0077] When the user's conversation request inputs both pictures and text, we have a solution, but when the user only inputs text, the related art only walks the text large model, but it is possible that the user is asking about the previous picture. At this time, if we only walk the text large model, we cannot accurately answer the user's question. In order to solve the problem of pure text asking about the picture, the embodiment of the application proposes a multi-round semantic association mechanism based on historical visual context, which can intelligently judge whether the user relies on the previously uploaded picture content under the condition that the user only inputs text, and dynamically activate the picture-text fusion processing link accordingly, thereby realizing cross-modal continuous conversation. In one possible implementation, see Figure 8 The method can further include the following steps: S250: In the case where the conversation request only includes text content, input the text content into the intent classification large model to obtain a second intent classification result.

[0078] When a conversation request containing only text content is received, it is first input into the trained intent classification large model to analyze the semantic intent behind the request.

[0079] Unlike traditional solutions, the intent classification large model used in this embodiment not only identifies the user's explicit query intent (such as "recipe query" and "product operation consultation"), but also has the ability to determine whether the request relies on historical image information. Therefore, the "second intent classification result" output by it contains at least one of the following two types of labels: depends on image recognition large model: indicates that the current question needs to be combined with the previously uploaded picture to accurately answer; does not depend on image recognition large model: indicates that the current question can be independently completed by text understanding to respond.

[0080] S260: Determine whether the second intent classification result is a second preset intent.

[0081] Further determine whether the second intent classification result is a "second preset intent", i.e., whether it is determined as "depends on image recognition large model". This determination is a key branching point that determines the subsequent processing path.

[0082] S270: If not, input the text content into the text large model to obtain the answer content of the conversation request.

[0083] If it is determined that the current request does not depend on image information, the regular text question and answer process is entered: the text content input by the user is directly sent to the text large model to generate a natural language answer. This path is suitable for completely independent new problems or general consultation without reference to historical images.

[0084] S280: If yes, based on the text content and the second intent classification result, the answer content of the session request is obtained.

[0085] If it is determined that the current request depends on historical image information, the system is no longer limited to the current input, but starts a cross-turn image-text fusion response mechanism to comprehensively call historical visual data and current semantic information to generate more accurate answers.

[0086] The second intent classification result includes a dependence on an image recognition large model and a non-dependence on an image recognition large model. Step S250 can be based on the dependence judgment of the historical session record, see Figure 9 , step S250 can include the following steps: S251: Determine whether there is a target historical session record using an image recognition large model in the historical session record of the previous preset number of turns.

[0087] S252: If not, the non-dependence on the image recognition large model is taken as the second intent classification result.

[0088] S253: If yes, the picture analysis result and the visual feature vector in each target historical session record are obtained.

[0089] S254: The text content, the visual feature vector, and each picture analysis result are input into the intent classification large model to obtain the second intent classification result.

[0090] First, the historical session record corresponding to the current session identifier is retrieved, and it is checked whether there is an interaction record that has called an image recognition large model in the past N turns of dialogue (N is a configurable parameter, for example, N=10). If no session record involving image recognition is found within the past N turns, it means that there is no available visual information in the current session context.

[0091] If not, even if the user's question may have ambiguity, it cannot be associated with effective visual context, so it is safely determined as "non-dependence on the image recognition large model" and transferred to the pure text processing path.

[0092] If one or more target historical session records (i.e., turns that have uploaded pictures and completed image analysis) are detected, the key information in these records will be extracted: picture analysis result: including unstructured description and structured list; visual feature vector: vector representation encoded by structured visual summary, used for semantic matching.

[0093] The text content, the visual feature vector and the picture analysis result are input into the intention classification large model. The model first calculates the semantic similarity between the text content input by the current user and the visual feature vector in each historical session record. For example, if the user asks "How does the last dish collect juice?", the text vector is embedded and the cosine similarity is calculated with the visual feature vector corresponding to the "stewed pork" image in the history. If the similarity exceeds the preset threshold, it is preliminarily determined that there is a semantic association. Further analyze the language reference features in the text, such as "just now", "that", "this picture", "the dish above" and other indicative words. Once such context-dependent expressions are detected, and the time or space range they point to matches a certain historical visual record, the confidence of "image-dependent" is enhanced. Combined with the content of the current question (such as involving cooking steps, equipment operation, food processing, etc.), it is determined whether the question belongs to the type of task that needs to combine visual information to accurately answer. For example, "How to defrost this microwave?" The "this" strongly suggests a reference to a specific device, and without image context, the model cannot accurately locate the model or status. Based on the above three analysis results, the intention classification large model performs weighted fusion or logical reasoning, and finally outputs the classification result of "image-dependent recognition large model" or "non-image-dependent recognition large model".

[0094] When it is determined in step S260 that the second intention classification result is "image-dependent recognition large model", it indicates that the question raised by the current user has significant association with the picture content uploaded before, and accurate and complete answers cannot be generated by relying on pure text understanding alone. At this time, if the traditional single text processing path is still used, it will lead to context breakage, information loss, and thus affect user experience and service professionalism.

[0095] Therefore, the embodiment of the present application further designs a response generation mechanism based on historical visual context backtracking, that is, without the need for the user to repeatedly upload pictures, the most relevant historical picture content of the question is automatically retrieved and re-injected into the picture-text fusion processing flow, realizing continuous reasoning and answer generation across rounds and modalities. The second intention classification result further includes historical picture content. Referring to Figure 10 , step S280 can include the following steps: S281: input the text content into the intention classification large model to obtain the first intention classification result; input the historical picture content and the text content into the image recognition large model to obtain the picture analysis result; S282: determine whether the first intention classification result is a first preset intention; S283: if not, the picture analysis result is taken as the answer content of the session request; S284: if yes, the picture analysis result and the text content are integrated and input into the text large model to obtain the answer content of the session request.

[0096] The text content input by the current user is input again into the same or another instantiated intent classification large model to obtain whether it belongs to a task type that needs to be combined with domain knowledge, and a "first intent classification result" is output. The result is used for subsequent judgment of whether to trigger the knowledge recall and text large model comprehensive generation process.

[0097] The historical picture content (i.e., the original image data uploaded by the user previously) taken out from the historical session record is input into the image recognition large model together with the current text to re-perform visual analysis and obtain the latest picture analysis result.

[0098] Steps S282 to S284 are consistent with the processing logic of steps S220 to S240, and will not be described again here.

[0099] The intent classification large model is used to identify whether the session request has a recipe query intent and whether it depends on the image recognition large model; the first intent classification result includes the recipe query intent and the non-recipe query intent; the first preset intent is the recipe query intent; and the second preset intent is the intent that depends on the image recognition large model.

[0100] The case where only text is input will be further described below with reference to Figure 11 When only text is input this time, the text content is input into the intent classification large model for identification. If the intent classification large model determines that the image recognition large model does not need to be depended on, the text content is directly input into the text large model to obtain the answer content. If the intent classification large model determines that the image recognition large model needs to be depended on, the relevant picture found by the intent classification large model is taken as the picture content, and the text content is processed in combination with the picture content as described above in steps S210 to S240.

[0101] In summary, the session processing method and program product based on large models provided by the embodiments of the present application can, in the case where the session request includes text content and picture content, determine whether the session request has a first preset intent through an intent classification large model. If not, the text content and the picture content are directly processed by an image recognition large model to obtain the answer content of the session request. If yes, the content obtained by the image recognition large model and the text content are further processed by a text large model to obtain the answer content of the session request. Thus, the image recognition large model and the text large model can be combined to comprehensively analyze the session request and give the answer content. Further, when the user asks a pure text question about the picture, the relevant image can be found from the historical session record, and then a multi-modal answer can be well given.

[0102] In the embodiments of the present application, it should be understood that the disclosed apparatus and method can also be implemented in other manners. The embodiments described above are merely exemplary for describing the present application. For example, the flowcharts and block diagrams in the accompanying drawings show the possible implementation architectures, functions and operations of the apparatus, method and computer program product according to the embodiments of the present application. In this regard, each block in the flowcharts and block diagrams can represent a module, a program segment or a part of code, which contains one or more executable instructions for implementing the specified logic function. It should also be noted that, in some alternative implementations, the functions noted in the blocks can occur in different orders from those described in the accompanying drawings. For example, two consecutive blocks can actually be executed in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for executing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0103] In addition, each functional module in the various embodiments of the present application can be integrated together to form a separate part, or each module can exist independently, or two or more modules can be integrated to form a separate part.

[0104] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art, or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a computer readable storage medium, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0105] The above merely provides the preferred embodiments of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0106] It will be apparent to those skilled in the art that the application is not limited to the details of the above-exemplified embodiments and that the present application can be implemented in other particular forms without departing from the spirit or essential characteristics of the present application. The embodiments should therefore be considered in all respects as illustrative and not restrictive, the scope of the application being indicated by the appended claims rather than by the above description, and all changes which come within the meaning and range of equivalency of the claims are therefore intended to be embraced therein. No reference signs in the claims should be considered as limiting the scope of the claims with respect to the figures of the patent document.

Claims

1. A session processing method based on a large model, characterized in that, include: Upon receiving a session request, if the session request includes both text content and image content, the text content is input into the intent classification model to obtain a first intent classification result. The image content and the text content are input into a large image recognition model to obtain the image analysis results. Determine whether the first intent classification result is the first preset intent; If not, the image analysis result will be used as the response to the session request; If so, the image analysis results and the text content are integrated and input into the large text model to obtain the response to the conversation request.

2. The method according to claim 1, characterized in that, The process of integrating the image analysis results and the text content and inputting them into a large text model yields the response to the conversation request, including: Based on the text content and the image analysis results, the corresponding target knowledge fragments are retrieved from the vector knowledge base; wherein, the vector knowledge base includes multiple knowledge fragments; The target knowledge fragment, the text content, and the image analysis results are integrated and input into the large text model to obtain the response content to the conversation request.

3. The method according to claim 2, characterized in that, The image analysis results include a list of information contained in the image content, which includes a recipe list, an ingredient list, and a product list. The vector knowledge base also includes vector representations corresponding to each knowledge fragment. The step of retrieving the corresponding target knowledge fragment from the vector knowledge base based on the text content and the image analysis results includes: By combining the text content and the information list, a comprehensive query vector is constructed; Calculate the similarity between the comprehensive query vector and each of the vector representations to obtain the similarity score for each knowledge fragment; The knowledge fragments with the highest similarity scores (preset number) are selected as target knowledge fragments.

4. The method according to claim 2, characterized in that, The image analysis results include the complete image analysis content, which is an unstructured natural language description generated by the image recognition big model after it has fully understood the image content. The process of integrating the target knowledge fragment, the text content, and the image analysis results and inputting them into a large text model to obtain the response content to the conversation request includes: The target knowledge fragment, the text content, and the complete image analysis content are assembled into prompt words; Based on the large text model, taking the text content as the main question and using the text content and the complete analysis of the image as references, a comprehensive understanding and analysis is performed to obtain the response content to the conversation request.

5. The method according to claim 1, characterized in that, Upon receiving a session request that includes image content, the method further includes: The image content is input into the image recognition model to obtain the visual semantic information corresponding to the image content. The visual semantic information is processed in a structured manner to generate a structured visual summary containing object categories, attribute features, and spatial relationships; The structured visual summary is encoded into a visual feature vector, and the visual feature vector, along with the current session identifier and the corresponding image analysis results, is stored in the historical session record.

6. The method according to claim 5, characterized in that, The method further includes: If the session request only includes text content, the text content is input into the intent classification model to obtain a second intent classification result; Determine whether the second intent classification result is the second preset intent; If not, input the text content into the text model to obtain the response to the conversation request; If so, based on the text content and the second intent classification result, obtain the response content for the session request.

7. The method according to claim 6, characterized in that, The second intent classification result includes those dependent on the image recognition model and those independent of the image recognition model; the step of inputting the text content into the intent classification model to obtain the second intent classification result includes: Determine whether there are target historical session records that used the image recognition large model in the historical session records of the previous preset number of rounds; If it does not exist, the image recognition model will not be relied upon as the second intent classification result; If they exist, obtain the image analysis results and visual feature vectors from the historical session records of each target; The text content, the visual feature vector, and the analysis results of each image are input into the intent classification model to obtain the second intent classification result.

8. The method according to claim 6, characterized in that, The second intent classification result also includes historical image content; obtaining the response content for the session request based on the text content and the second intent classification result includes: The text content is input into the intent classification model to obtain the first intent classification result; the historical image content and the text content are input into the image recognition model to obtain the image analysis result. Determine whether the first intent classification result is the first preset intent; If not, the image analysis result will be used as the response to the session request; If so, the image analysis results and the text content are integrated and input into the large text model to obtain the response to the conversation request.

9. The method according to claim 6, characterized in that, The intent classification model is used to identify whether the session request has the intent to query a recipe and whether it depends on the intent of the image recognition model. The first intent classification result includes intents to search for recipes and intents not to search for recipes; The first preset intent is to have the intent to search for recipes; The second preset intent is the intent that relies on the large image recognition model.

10. A computer program product, characterized in that, The computer program product includes instructions that, when executed on an electronic device, cause the electronic device to implement the large-model-based session processing method according to any one of claims 1 to 9.

Images