A dynamic intention understanding method based on multi-modal fusion

Abstract

The embodiment of the application discloses a dynamic intention understanding method based on multi-modal fusion, comprising: acquiring multi-modal behavior data of a user in this conversation; rewriting the conversation text according to whether this conversation is the first conversation; providing the rewritten conversation text and other multi-modal behavior data to a large model together, prompting the large model to identify the intention of this conversation; specifically, the role of the large model in the prompt is specified as a user conversation intention understanding expert, the understanding skill in the typical conversation mode is specified, a plurality of intention types to be identified and their descriptions are specified, and the large model is instructed to understand the rewritten conversation and other multi-modal behavior data according to the specified role and information, and determine which one of the plurality of intention types this conversation belongs to; according to the intention recognition result, a corresponding business process is called to provide useful information for this conversation. The embodiment can improve the user intention recognition effect in the online learning scene.

Description

TECHNICAL FIELD

[0001] The embodiment of the application relates to the field of artificial intelligence, in particular to a dynamic intention understanding method based on multi-modal fusion. BACKGROUND

[0002] Intention understanding refers to identifying the purpose behind individual behavior. Introducing intention understanding in intelligent dialogue is more conducive to providing accurate and useful information to users.

[0003] Existing intention recognition technology mainly relies on single-modal (such as text) analysis, which is difficult to effectively fuse multi-dimensional user behavior data, and does not explicitly specify the specific intention type in the online learning scenario. Patent application CN115186076A provides a text understanding intention extraction method, and CN119886352A provides a dialogue intention understanding and explanation text generation method based on deep learning, which cannot solve the above problems. SUMMARY

[0004] The embodiment of the application provides a dynamic intention understanding method based on multi-modal fusion to solve the above technical problems.

[0005] In a first aspect, the embodiment of the application provides a dynamic intention understanding method based on multi-modal fusion, comprising:

[0006] Obtaining multi-modal behavior data of a user in the current dialogue, wherein the multi-modal behavior data includes voice, image and dialogue text;

[0007] Rewriting the current dialogue text according to whether the current dialogue is the first dialogue;

[0008] Providing the rewritten dialogue text and other multi-modal behavior data to a large model together, prompting the large model to identify the intention of the current dialogue; specifically, the role of the large model is specified as a user dialogue intention understanding expert in the prompt, the understanding skill in the typical dialogue mode is specified, the multiple intention types to be identified and their descriptions are specified, and the large model is instructed to understand the rewritten dialogue and other multi-modal behavior data according to the specified role and information, and determine which of the multiple intention types the current dialogue belongs to.

[0009] According to the intention recognition result, a corresponding business process is called to provide useful information for the current dialogue;

[0010] The multiple intention types include question and answer intention, search intention, to-do task intention, learning task intention, teaching plan creation intention and PPT creation intention.

[0011] In a second aspect, the embodiment of the application provides an electronic device, comprising:

[0012] one or more processors;

[0013] a memory for storing one or more programs,

[0014] When the one or more programs are executed by the one or more processors, the one or more processors implement the multi-modal fusion based dynamic intention understanding method described in any embodiment.

[0015] In a third aspect, the embodiments of the present application also provide a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the multi-modal fusion based dynamic intention understanding method described in any embodiment.

[0016] The embodiments of the present application provide a multi-modal fusion based dynamic intention understanding method, which not only fuses multi-modal information and historical dialogues in intention recognition, improves the accuracy of intention recognition, but also realizes automatic recognition of user intention by using a large model, specifies the role of the large model as a user dialogue intention understanding expert through a large model prompt, specifies understanding skills in a typical dialogue mode, specifies a plurality of intention types to be recognized and their descriptions, provides key information for the large model to recognize user intention, and then instructs the large model to understand the rewritten dialogue and other multi-modal behavior data according to the specified role and information, judges which one of the plurality of intention types the current dialogue belongs to, so that the large model can accurately recognize the basic user intention; finally, a corresponding business process is designed for each intention type, which more conveniently provides effective information for the user. BRIEF DESCRIPTION OF DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the specific embodiments or prior art of the present application, the drawings needed in the specific embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0018] Figure 1 is a flowchart of a multi-modal fusion based dynamic intention understanding method provided by the embodiments of the present application;

[0019] Figure 2 is a structural schematic diagram of a reinforcement learning network provided by the embodiments of the present application;

[0020] Figure 3 is a structural schematic diagram of an electronic device provided by the embodiments of the present application. DETAILED DESCRIPTION

[0021] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be clearly and completely described below. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0022] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0023] In the description of the present application, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting" should be understood broadly, for example, it can be fixedly connected, or it can be detachably connected, or integrally connected; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the communication inside two elements. For those of ordinary skill in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0024] Figure 1 is a flowchart of a dynamic intention understanding method based on multi-modal fusion provided by an embodiment of the present application. The method is suitable for an online learning scene and is executed by an electronic device. As shown in Figure 1 , the method specifically comprises:

[0025] S110, acquiring multi-modal behavior data of a user in this conversation, the multi-modal behavior data comprising voice, image and conversation text.

[0026] As described above, the present embodiment is suitable for an online learning scene, such as an online learning platform. The platform provides an intelligent conversation window for users, and users can obtain useful information through intelligent conversation to more conveniently complete various matters in the online learning platform.

[0027] Optionally, the intelligent dialogue first acquires the text data input by the user in the current dialogue, and simultaneously acquires voice data and image data of the user through a camera and an audio device of a learning device (such as a computer or a mobile phone), and collectively takes the three types of data as the multi-modal behavior data of the current dialogue. The voice data can provide the user's voice, tone, and other semantic information not mentioned in the text data, and the image data can provide the user's facial expression, mental state, and the like.

[0028] S120, rewriting the current dialogue text according to whether the current dialogue is a first dialogue.

[0029] Optionally, a "start new dialogue" button is provided in the intelligent dialogue window, and when the user clicks the button or first logs in to the intelligent dialogue window, the current dialogue is considered as a first dialogue.

[0030] Further, if the current dialogue is a first dialogue, the current dialogue text is directly taken as the rewritten dialogue text and is involved in subsequent operations; if the current dialogue is not a first dialogue, the historical dialogue text between the current dialogue and the most recent first dialogue is first extracted, and after the historical dialogue and the current dialogue are combined, they are collectively taken as the rewritten dialogue.

[0031] S130, collectively providing the rewritten dialogue text and other multi-modal behavior data to the large model, and prompting the large model to identify the intention of the current dialogue.

[0032] The embodiment divides a plurality of intention types for an online learning scene, including a question and answer intention, a search intention, a to-do task intention, a learning task intention, a teaching plan creation intention, and a PPT creation intention. The several intention types cover several major basic requirements in the online learning scene.

[0033] The embodiment utilizes the large model to identify the dialogue intention, and the several intention types and their descriptions play a crucial role in the large model prompt. Meanwhile, the prompt also needs to have the following contents to fully guarantee the accuracy of the intention identification: specifying the role of the large model as a user dialogue intention understanding expert; specifying the understanding skills in a typical dialogue mode; instructing the large model to understand the rewritten dialogue and other multi-modal behavior data according to the specified role and information, and determine which one of the plurality of intention types the current dialogue belongs to.

[0034] Exemplarily, a prompt is as follows:

[0035] ## Specifying the role and task

[0036] You are an experienced and professional user question intent understanding and intent judgment expert. Based on the user's question, you will understand the question intent and, based on the given intent classification, feedback the user's true intent in this conversation.

[0037] ## Understanding skills in typical dialogue mode

[0038] 1. If the user's question explicitly mentions a specific intent, return the intent mentioned by the user.

[0039] 2. If the user's question expresses an intent that highly coincides with the meaning of multiple intents in the intent classification description, feedback multiple matching intent classifications.

[0040] 3. If the user's question does not explicitly express an intent, default to the "question and answer intent" return.

[0041] ## Specify output format requirements

[0042] Output the results according to the format requirements (separated by <format requirements> < / format requirements>). Ensure that the output results can be correctly parsed by Python json.loads.

[0043] ## Specify multiple intent types to be recognized and their descriptions

[0044] <Intent Classification>

[0045] Question and answer intent: This intent is mainly for users who want to quickly solve problems and hope you can answer questions professionally.

[0046] Search intent: This intent is mainly for users who want to actively query certain resources and hope you can search for business and give results as required.

[0047] To-do task intent: This intent is mainly for users who want to find their future to-do items / work sets and hope you can help them find the number of tasks to be solved.

[0048] Learning task intent: This intent is mainly for users who want to find their future learning content sets and emphasize "required" learning resource search.

[0049] Teaching plan creation intent: This intent is mainly for users who want to create an article and hope you can create a teaching article for me.

[0050] PPT creation intent: This intent is mainly for users who want to write a PPT and hope you can plan and create a beautiful PPT for me.

[0051] < / Intent Classification>

[0052] ## Specify other parameter variables:

[0053] <user question>

[0054] {question}

[0055] < / user question>

[0056] <format requirement>

[0057] ["Return to this dialogue intent here"]

[0058] < / format requirement>

[0059] Among them, ## is used to mark the note of the prompt, < > and<!-- --> are used to indicate parameter slots, < intent classification> and < / intent classification> are filled with pre-set multiple intent types and their descriptions, and < user question> and < / user question> are filled with modified user dialogue.

[0060] Based on the above prompt, the large language model can more accurately determine which of the multiple intent types this dialogue belongs to.

[0061] S140, according to the intent recognition result, call the corresponding business process to provide useful information for this dialogue.

[0062] In this embodiment, different business processes are designed for different intent types to provide more efficient and accurate personalized services.

[0063] For example, for the question and answer intent, the business process is: first call the large model to analyze and trace the knowledge points, and then search in the knowledge vector library according to the knowledge points and sources to get useful information. For the search intent, the business process is: directly query the knowledge vector library to retrieve matching information; At the same time, execute the business process of the question and answer intent, and provide direct search results and analysis of search results for the user to choose. For the retrieval of to-do task process, call the large model to analyze the to-do object of the dialogue text, and remotely schedule to-do task statistical data according to the to-do object, and feed back the to-do task list of the specific object to the user. For the learning task intent, call the large model to analyze the to-be-learned object of the dialogue text, and remotely schedule learning resource data according to the to-be-learned object, and feed back the specific learning resource to the user. For the lesson plan creation intent, call the large model to analyze the creation points, and search for related texts of the creation points in the knowledge vector library, and feed back to the large model to process into a complete manuscript. For the PPT creation intent: call the large model to analyze the creation points, and at the same time plan the page layout, search for related texts and multimedia materials of the creation points in the knowledge vector library, and feed back to the large model to process into the corresponding PPT.

[0064] The above flow is executed in each conversation. The entire method not only integrates multi-modal information and historical conversations to improve the accuracy of intent recognition, but also uses a large model to automatically identify user intent, specifies the role of the large model as a user conversation intent understanding expert through a large model prompt, specifies understanding techniques in typical conversation modes, specifies various intent types to be identified and their descriptions, provides key information for the large model to identify user intent, and instructs the large model to understand the rewritten conversation and other multi-modal behavior data according to the specified role and information, determines which of the various intent types the current conversation belongs to, so that the large model can accurately identify the basic user intent. Finally, a corresponding business process is designed for each intent type to more conveniently provide effective information for the user.

[0065] Further, as described above, the various intent types preset in the prompt are key to ensuring that the large model accurately identifies intent. In order to adapt to the dynamic changes in user conversation intent and adjust the intent recognition ability of the large model in a timely manner, the various preset intent types can be dynamically adjusted. In a specific embodiment, the process can include the following steps:

[0066] Step one, generating an intent analysis chain of the user according to the user's acceptance of useful information in previous conversations, wherein the intent analysis chain is composed of the user's acceptance of useful information in each conversation or the intent recognition result corresponding to the previous conversation before the termination of the conversation chain. This embodiment reflects the exploration of the large model for the user's true intent through the intent analysis chain. The shorter the intent analysis chain, the more accurate the intent recognition of the large model.

[0067] Optionally, first, identify the chain nodes for dividing the intent analysis chain. Specifically, if the user clicks or copies the useful information of a certain conversation, it is determined that the user accepts the useful information of the certain conversation, and the certain conversation is marked as a chain node; if the user does not input conversation text for more than a set time in a certain conversation, or exits the conversation interface, or clicks the option to start a new conversation, it is determined that the user terminates the conversation chain, and the certain conversation is also marked as a chain node. At this time, there is a special case: after the user clicks or copies the useful information of a certain conversation, any of the following behaviors occurs: not inputting conversation text for more than a set time, exiting the conversation interface, or clicking the option to start a new conversation. The certain conversation is marked as a chain node.

[0068] After the chain node is marked, the intention recognition results corresponding to the previous dialogues between the same user at two chain nodes are arranged in turn, and the intention analysis chain of the same user is formed. It should be noted that the intention analysis chain between the two chain nodes includes the latter chain node in the two chain nodes and does not include the former chain node in the two chain nodes, that is, each chain node is regarded as the end point of the previous intention analysis chain and is not counted in the next intention analysis chain.

[0069] Step two, according to the change of each intention recognition result in the intention analysis chain, adjust the plurality of intention types, and apply the adjusted plurality of intention types to subsequent large model prompts.

[0070] As described above, the embodiment reflects the exploration of the large model to the real intention of the user through the intention analysis chain. The shorter the intention analysis chain is, the more accurate the intention recognition of the large model is, and the real intention is explored through a smaller number of dialogues and useful information is provided. If the intention analysis chain is too long, the real intention is obtained through multiple dialogues and useful information is provided, then it is very likely that the intention classification provided to the large model is not perfect enough, which affects the accuracy of intention analysis, and it is necessary to consider adjusting the intention classification. It should be noted that the real intention here refers to the actual intention of the user, which is not limited to the plurality of intention types set in the large model prompt, but may be an intention type close to, more subdivided, or completely different from the several intention types. As long as one of the plurality of intention types set in the large model prompt is very close to the real intention, the large model can more accurately recognize the real intention, therefore, the purpose of the embodiment is to improve the closeness of the intention types in the prompt to the real intention. Optionally, the embodiment provides the following two optional implementation manners to dynamically adjust the plurality of types set in the large model prompt (hereinafter referred to as “intention classification adjustment”) to achieve this purpose:

[0071] The first optional implementation manner is applicable to the intention analysis chain whose end point is that the user adopts the useful information of the last dialogue. If the length of the intention analysis chain exceeds the set threshold, it indicates that the user intention exploration process is too long, and the useful information that can be adopted is obtained through multiple dialogues. At this time, the intention classification adjustment can be performed in the following two cases:

[0072] In case one, all the intent recognition results in the intent analysis chain are the same, indicating that the large model correctly identifies the user's intent but fails to provide useful information that can be accepted in the early stage. This is likely because the intent classification is not detailed enough, resulting in the user always issuing more in-depth conversations. At this time, if all the intent recognition results in the intent analysis chain are the same, the same intent type corresponding to each intent recognition result is subdivided according to the last two conversations in the intent analysis chain. Optionally, the last two conversations can be respectively analyzed for keywords to determine the keyword that is most different in semantics compared to the second last conversation, and the same intent type in the current intent analysis chain is subdivided according to the keyword. For example, if all the intent recognition results in the intent analysis chain are search intent, the content of the second last conversation is "Please help me search for the content of the rescue points in the anti-flood disaster course", and the content of the last conversation is "Please help me search for the content of the rescue operation in the anti-flood disaster course", the keyword that is most different in semantics compared to the second last conversation is "operation", and "search intent" can be subdivided into "operation search intent" and "theory search intent" according to the keyword, and the description of the subdivided intent and the corresponding business process are increased. In this way, the large model can focus on the information about operation and theory as soon as possible in each conversation, identify the user's more subdivided real intent as soon as possible, or present "operation search intent" and "theory search intent" to the user as soon as possible, and the user can directly select the real subdivided intent. Among them, the various operations from keyword division to business process design described above can be automatically implemented by the large model, or can be implemented by the developer according to experience, or can be implemented by automatic means and human experience, and the present embodiment is not limited in detail.

[0073] In case two, all the intention recognition results in the intention analysis chain are not all the same, indicating that the large model finally recognizes the correct intention after multiple explorations of the user's intention and provides useful information that can be adopted. In this case, the embodiment pays more attention to the turning point from the wrong intention to the correct intention to correct the most basic intention direction. Optionally, the last continuous repetition segment of the last intention recognition result in the intention analysis chain at the end of the intention analysis chain can be extracted; and the intention type of the last dialogue before the continuous repetition segment is subdivided according to the first dialogue in the continuous repetition segment. For example, the intention analysis chain is "question and answer, search, to-do task, learning task, learning task", and the last intention recognition result is learning task, which is repeated twice at the end of the above-mentioned intention analysis chain, and the continuous repetition segment is the last "learning task, learning task" segment; the first dialogue in the segment is the second last dialogue, and the last dialogue before the segment is the third last dialogue, and the intention type "to-do task" of the third last dialogue is subdivided according to the content of the second last dialogue. Optionally, the keywords of the two dialogues can be analyzed respectively to determine the keywords that are most different in semantics compared with the second last dialogue and the third last dialogue, and the "to-do task" intention type is subdivided according to the keywords. For example, the content of the third last dialogue is "please help me find the unfinished course", and the content of the second last dialogue is "please help me preview the content of the unfinished course", and the keywords that are most different in semantics compared with the second last dialogue and the third last dialogue are "preview" and "content", and the "to-do task" intention can be subdivided into "to-do task list" and "to-do task content preview", and the subdivided intention can be added with a corresponding description, such as the description of the to-do task list is "providing a task list to the user", and the description of the to-do task content preview is "providing a task list and a content preview thumbnail of each task to the user", to distinguish the meanings of the two intentions; at the same time, the corresponding business processes can also be subdivided and designed, such as the business process of the to-do task content preview has more processes of to-do task content search and thumbnail making than the business process of the to-do task list, etc. Similarly, the above operations from keyword division to business process design can be automatically realized by means of the large model, can be realized by the developers according to experience, or can be realized by means of automatic means and human experience, and the embodiment is not specifically limited.

[0074] The second optional implementation is suitable for an intention analysis chain whose end is that the user terminates the dialogue chain. If the length of the intention analysis chain exceeds a set threshold, it indicates that the user's exploration process is too long and has not obtained useful information that can be adopted after multiple dialogues, and it is very likely that the intention recognition is inaccurate, resulting in a series of invalid question and answer. At this time, the intention classification can be adjusted in the following two cases:

[0075] Case I: The intent recognition result in the intent analysis chain covers multiple existing intent types, indicating that the existing intent types do not touch the user's real intent. At this time, new intent types can be added according to the user's previous conversation content. Optionally, all conversations in the intent analysis chain can be keyword parsed, and the most frequently co-occurring keywords in all conversations can be determined using association rules. New intent types can be added based on these keywords. For example, the most frequently co-occurring keywords in all conversations are "tender scheme", "technical indicators", "business indicators", etc. A new intent type "tender scheme generation intent" can be added, and the corresponding business process can be designed. Similarly, the above operations from keyword division to business process design can be automatically implemented with the aid of a large model, or can be implemented by developers based on experience, or can be implemented by combining automatic means and human experience. The present embodiment is not specifically limited.

[0076] Case II: The intent recognition result in the intent analysis chain does not cover multiple existing intent types. The user can be asked questions based on the intent types not covered by the intent analysis chain to determine whether the user belongs to the uncovered intent types. For example, if the "PPT creation intent" does not appear in the current intent analysis chain, the user can be asked "Do you want to create a PPT?" If the user answers affirmatively, it is possible that there is useful information that can be adopted.

[0077] The above embodiments focus on intent analysis chains with long user exploration processes, analyze the real intent recognition obstacles that may be caused by intent classification based on different exploration paths in intent analysis, and propose specific ways to eliminate such causes, providing improvement directions for the rationality and accuracy of intent classification.

[0078] However, in the above case of a long intent exploration process, in addition to the real intent recognition that may be hindered by intent classification, factors such as inappropriate user conversation content and uncertainty in user intent may also hinder real intent recognition. Therefore, whether the adjusted intent classification is truly beneficial to real intent analysis after being applied to the large model prompt still needs to be further verified. For this purpose, the present embodiment takes multiple intent types adjusted each time as an intent classification scheme, and constructs a reinforcement learning network to determine the optimal intent classification scheme. For example, in the prompt example of S130, the combination of the question and answer intent, search intent, to-do task intent, learning task intent, teaching plan creation intent, and PPT creation intent is an intent classification scheme. In another embodiment, after the search intent is refined into practical search intent and theoretical search intent, the combination of the question and answer intent, practical search intent, theoretical search intent, to-do task intent, learning task intent, teaching plan creation intent, and PPT creation intent is another intent classification scheme.

[0079] In an embodiment, the reinforcement learning network takes the user's intent analysis chain in the actual dialogue as the state variable, takes various intent classification schemes as the action variable, and constructs a reward function according to the final result and the average length of the user's intent analysis chain to select the optimal intent classification scheme and apply it to subsequent large model prompts. Optionally, the higher the coverage of the final result of the intent analysis chain of all users in a period of time on the intent classification scheme used, the higher the positive reward; the shorter the average length of the intent analysis chain of all users in a period of time, the higher the positive reward. Exemplarily, a reward function can be constructed as follows:

[0080]

[0081] wherein, represents the reward value, represents the number of intent types in the current intent classification scheme being used covered by the final result of the intent analysis chain of all users in a period of time, represents the total number of intent types in the intent classification scheme, represents the average length of the intent analysis chain of all users in a period of time, and are positive proportional adjustment coefficients, respectively.

[0082] Optionally, the user's intent analysis chain in the state variable can be encoded as a vector, the dimension of the vector is greater than the maximum length of the historical intent analysis chain, each intent type corresponds to a non-zero value, the first dimension of the vector is the non-zero value corresponding to the first intent recognition result in the intent analysis chain, the second dimension is the non-zero value corresponding to the second intent recognition result in the intent analysis chain, and so on until all intent recognition results in the intent analysis chain have corresponding vector elements, and the remaining vector elements are filled with 0.

[0083] Optionally, the structure of the reinforcement learning network is as follows: Figure 2As shown, the network takes the intention analysis chain (i.e., state variable) of all students in a period of time as batch input, and the optimal intention classification scheme (i.e., action variable) to be adopted in the next period of time as output. The network includes a feature extraction layer and a Q value calculation layer. The feature extraction layer is used to encode the input features to obtain corresponding encoded features. The Q value calculation layer is used to output a Q value vector according to the encoded features. Each element in the Q value vector is the probability of selecting each intention classification scheme. The intention classification scheme with the maximum probability is the action variable finally output by the reinforcement learning network. Optionally, the feature extraction layer and the Q value calculation layer both adopt a fully connected structure. The number of nodes of the feature extraction layer gradually increases to enrich the feature information layer by layer. The number of nodes of the Q value calculation layer gradually decreases to adapt to the dimension of the Q value vector. The number of layers and nodes can be flexibly set according to needs. The network updates the Q value in the following way:

[0084] First, calculate :

[0085]

[0086] wherein represents the Q value of the action under the state , represents the updated Q value of the action under the state , represents the maximum one among all Q values under the state , represents the reward obtained after the action is taken from the state to , and are adjustable coefficients.

[0087] Then, according to the difference between and , the following loss function item is constructed:

[0088]

[0089] According to the minimization of , the network parameters of the feature extraction layer and the Q value calculation layer can be updated.

[0090] Based on the above basic algorithm, the reinforcement learning network gradually learns the optimal intent classification scheme that can truly improve the intent recognition effect in continuous user dialogue according to the changes in the intent analysis chain brought by each intent classification scheme. The optimal scheme is applied in subsequent large model prompts, which can make the intent recognition effect and user dialogue result of the large model more excellent. With the addition of each new intent classification scheme, only the corresponding node and full connection operation with the previous layer need to be added in the last layer of the Q value calculation layer, and the network will gradually update the parameters according to the dialogue situation.

[0091] Of course, in addition to the above reinforcement learning network, various intent classification schemes can also be evaluated according to user feedback, and the intent classification can be evaluated in time to avoid the effect of adjustment from declining or being unstable.

[0092] Further, in another specific embodiment, considering the difference of users, the above operation of adjusting and evaluating the intent classification scheme can also be performed for each user respectively, that is, the intent analysis chain of each user is generated respectively according to the acceptance of useful information of each user in each dialogue, and the multiple intent types of each user are adjusted respectively according to the change of each intent recognition result in the intent analysis chain of each user, and the multiple intent types of each user after adjustment are applied to the subsequent large model prompts of each user respectively.

[0093] Correspondingly, for each user, the multiple intent types after each adjustment are used as an intent classification scheme, and a reinforcement learning network is constructed for each user, taking the intent analysis chain of the current user as a state variable, taking various intent classification schemes of the current user as action variables, constructing a reward function according to the final result and average length of the intent analysis chain of the current user, to select the optimal intent classification scheme of the current user and apply it to the large model prompt for the current user.

[0094] The advantage of this is that the intent classification scheme can be adjusted for each user respectively to adapt to the personalized learning habits of users, and also avoid the inappropriate influence that may be caused by applying the intent classification adjustment of a single user to all users. The disadvantage is that more storage space and computing resources are needed. In actual application, the implementation mode can be flexibly selected according to the actual situation of the system.

[0095] In addition, it should be noted that the user data (including voice data, image data and dialogue text data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0096] Figure 3A structural schematic diagram of an electronic device provided by an embodiment of the present application is shown in FIG. 1, which includes a processor 60, a memory 61, an input device 62 and an output device 63; the number of processors 60 in the device can be one or more, Figure 3 of which one processor 60 is taken as an example; the processor 60, the memory 61, the input device 62 and the output device 63 in the device can be connected through a bus or other means, Figure 3 of which connection through a bus is taken as an example. Figure 3

[0097] The memory 61, as a kind of computer readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the dynamic intention understanding method based on multi-modal fusion in the embodiment of the present application. The processor 60 executes the software programs, instructions and modules stored in the memory 61, thereby performing various functional applications and data processing of the device, i.e. implementing the dynamic intention understanding method based on multi-modal fusion.

[0098] The memory 61 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system and application programs required by at least one function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory 61 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device or other non-volatile solid-state memory device. In some examples, the memory 61 can further include a memory remotely arranged with respect to the processor 60, which can be connected to the device through a network. Examples of the above network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0099] The input device 62 can be used to receive input digital or character information, and generate key signal input related to user settings and function control of the device. The output device 63 can include a display device such as a display screen.

[0100] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the dynamic intention understanding method based on multi-modal fusion of any embodiment.

[0101] ​The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0102] The computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which the computer-readable program code is contained. Such propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can be used to carry or propagate program code that is used by or in connection with an instruction execution system, apparatus or device.

[0103] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination thereof.

[0104] The computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including an object oriented programming language such as Java, Smalltalk, C++ or the like and conventional procedural programming languages such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments of the present application, electronic mail (email) can be utilized as the insecure digital messaging described above.

[0105] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit the present application; although the present application has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions recorded in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the technical solutions of the embodiments of the present application.

Claims

1. A dynamic intent understanding method based on multimodal fusion, characterized in that, include: Acquire multimodal behavior data of the user in this conversation, including voice, images, and conversation text; Rewrite the text of this dialogue based on whether it is the first dialogue of its kind; The rewritten dialogue text and other multimodal behavioral data are provided to the large model to help it identify the intent of the dialogue. Specifically, the prompt specifies the role of the large model as a user dialogue intent understanding expert, specifies the understanding techniques in typical dialogue patterns, specifies the multiple intent types to be identified and their descriptions, and instructs the large model to understand the rewritten dialogue and other multimodal behavioral data based on the specified role and information, and determine which of the multiple intent types the current dialogue belongs to; wherein, the multiple intent types include question-and-answer intent, search intent, to-do task intent, learning task intent, lesson plan creation intent, and PPT creation intent; Based on the intent recognition results, the corresponding business process is invoked to provide useful information for this dialogue; Based on the user's acceptance of useful information in each conversation, an intent analysis chain is generated. The intent analysis chain consists of the intent recognition results of each conversation before the user accepts useful information in each conversation or terminates the conversation chain. Based on the changes in the intent recognition results in the intent analysis chain, the various intent types are adjusted, and the adjusted intent types are applied to subsequent large model prompts. Specifically, when the endpoint of the intent analysis chain is that the user has accepted the useful information from the last dialogue: if all intent recognition results in the intent analysis chain are the same, keyword parsing is performed on the last two dialogues respectively to determine the keyword with the greatest semantic difference between the last dialogue and the penultimate dialogue, and the same intent type in the current intent analysis chain is further subdivided based on this keyword; otherwise, the continuous repetition segment at the end of the intent analysis chain from the last intent recognition result in the intent analysis chain is extracted; keyword parsing is performed on the first dialogue of the continuous repetition segment and the last dialogue before the continuous repetition segment respectively to determine the keyword with the greatest semantic difference between the first dialogue of the continuous repetition segment and the last dialogue before the continuous repetition segment, and the intent type of the last dialogue before the continuous repetition segment is further subdivided based on this keyword. Each adjusted intent type is treated as an intent classification scheme. A reinforcement learning network is constructed, with the user's intent analysis chain as the state variable and various intent classification schemes as action variables. A reward function is constructed based on the final result and average length of the user's intent analysis chain to select the optimal intent classification scheme and apply it to subsequent large model prompts. The higher the coverage of the final result of all users' intent analysis chains with the used intent classification scheme within a certain period, the higher the positive reward; the shorter the average length of all users' intent analysis chains within a certain period, the higher the positive reward.

2. The method according to claim 1, characterized in that, The rewriting of the dialogue text based on whether this is the first dialogue includes: If this is the first conversation, use this conversation text as the rewritten conversation text. If this is not the first dialogue, the text of this dialogue and the text of the previous dialogue will be combined to form the rewritten dialogue text.

3. The method according to claim 1, characterized in that, The step of generating a user intent analysis chain based on the user's acceptance of useful information in previous conversations also includes: If a user clicks on or copies useful information from a conversation, it is determined that the user has accepted the useful information from that conversation, and that conversation is marked as a chain node. If a user fails to enter dialogue text within the set time limit in a conversation, exits the conversation interface, or clicks the option to start a new conversation, it is determined that the user has terminated the conversation chain, and the conversation is marked as a chain node. The intent recognition results of each dialogue between the same user and the two chain nodes are arranged sequentially to form the intent analysis chain of the same user.

4. The method according to claim 1, characterized in that, The step of adjusting the multiple intent types based on the changes in the intent recognition results in the intent analysis chain further includes: In the case where the intent analysis chain terminates due to the user ending the conversation chain: If the intent recognition results in the intent analysis chain cover the multiple intent types, new intent types are added based on the user's previous dialogue content. Otherwise, the user is prompted with questions based on intent types not covered by the intent analysis chain to determine whether the user belongs to the uncovered intent type.

5. The method according to claim 1, characterized in that, The step of generating a user intent analysis chain based on the user's acceptance of useful information in each dialogue includes: generating an intent analysis chain for each user based on the user's acceptance of useful information in each dialogue. Accordingly, adjusting the multiple intent types based on the changes in the intent recognition results in the intent analysis chain and applying the adjusted multiple intent types to the subsequent large model prompts includes: adjusting the multiple intent types for each user based on the changes in the intent recognition results in the intent analysis chain for each user, and applying the adjusted multiple intent types for each user to the subsequent large model prompts for each user.

6. An electronic device, characterized in that, include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the dynamic intent understanding method based on multimodal fusion as described in any one of claims 1-5.

7. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the dynamic intent understanding method based on multimodal fusion as described in any one of claims 1-5.

Images