Intelligent multimedia interaction method and device and storage medium

Abstract

The invention discloses a multimedia intelligent interaction method and device and a storage medium, and relates to the field of artificial intelligence medical health. Obtaining multi-modal user input information, and obtaining a user input text by using the input analysis model; according to a text input by a user, utilizing the large model to obtain agent reply information; according to the agent reply information, utilizing a multimedia generation model or a retrieval technology to obtain an agent response; and the user interacts with the multimedia content to generate a corresponding result to enter the next round of interaction. On one hand, the intelligent agent not only can identify character information input by a user, but also adds demand analysis combining the user and screen operation, and greatly improves the flexibility and communication efficiency of a communication mode; and on the other hand, the intelligent agent can convey information and explain problems to the user in the modes of graphics, images, videos, animations and the like which are not limited to characters, so that expression of complex connotation and transmission of non-language information are realized in a session, and the service quality of the intelligent agent is improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence medical health, and more specifically to an intelligent multimedia interaction method, device and storage medium. Background Art

[0002] An intelligent agent is a system that can perceive the environment, make decisions, and perform actions. The emergence of intelligent agents marks the progress of artificial intelligence (AI) from simple rule matching and computational simulation to a higher level of autonomous intelligence. With the rapid development of artificial intelligence technology, chat agents have begun to show their power in the field of medical and health consultation. More and more intelligent medical consultation models have sprung up like mushrooms after rain, and are vividly called "chatting doctors". However, most traditional intelligent consultation models are based on large language models and can only interact with users in the form of text or voice. Although this interactive method is convenient, it has limitations in expressing complex connotations, transmitting non-verbal information, and establishing deep user connections. In order to overcome these limitations, it is particularly important to develop an intelligent multimedia interactive system that can communicate with users in the form of multimedia dialogue.

[0003] Existing video generation technologies are mostly focused on the creation of natural scenes, animations or virtual characters, but there are few systems that can respond to user input in real time and generate natural, smooth animations and video content. In addition, effectively combining AI technology with multimedia generation technology to achieve intelligent multimedia interaction will greatly improve the communication efficiency and service quality of the intelligent consultation system, which is also an important direction for current technological development. Summary of the invention

[0004] In view of this, the present invention provides an intelligent multimedia interaction method, device and storage medium, which can increase the amount of information transmitted between the intelligent agent and the user in the interaction in the form of multimodal data, thereby meeting the user's diverse communication needs and realizing flexible and efficient information exchange.

[0005] In order to achieve the above object, the present invention adopts the following technical solution:

[0006] A multimedia intelligent interaction method comprises the following steps:

[0007] S1: Obtain multimodal user input information and obtain the user input text using the input analysis model;

[0008] S2: Based on the user input text, use the large model to obtain the agent's reply information;

[0009] S3: according to the agent reply information, using a multimedia generation model or retrieval technology to obtain an agent response;

[0010] S4: The user interacts with the multimedia content, generates corresponding results and enters the next round of interaction.

[0011] Optionally, in S1, the input analysis model includes three sub-models: a text processing model, a speech-to-text model, and an image instance segmentation model. All three are pre-trained artificial intelligence algorithm models, and their working principle is to collect sample data that meets the ideal input and output of the model, and use the sample data to adjust the preset model parameters with the help of loss functions, back propagation and other methods, so that the model has the function of converting input data into output of a specific pattern. Among them, the input of the text processing model is the original text data that may contain grammatical errors, typos and other problems, and the output is the corrected text data, which mainly functions to correct and polish the text; the input of the speech-to-text model is the voice data recorded by the user through a microphone, and the output is text data, which mainly functions to convert the user's voice data into text data of the corresponding content.

[0012] Optionally, S1 includes the following steps:

[0013] S1.1: Use multimedia input devices to obtain user input, including but not limited to text, voice, image, and video information input;

[0014] S1.2: Preprocessing the input multimedia content, including but not limited to denoising, enhancement, and format conversion;

[0015] S1.3: Inputting the text typed by the user into the text processing model for error correction and polishing to obtain a first text label;

[0016] S1.4: inputting the preprocessed speech data into the speech-to-text model, and obtaining an output text as a second text label;

[0017] S1.5: Input the image / video into the pre-trained image instance segmentation model to obtain classification labels of different regions; obtain the user's intention label according to the action trajectory using the current operation trajectory recognition technology, determine the description of the corresponding image / video region according to the coordinate trajectory of the operation, and combine the user's intention label with the description label of the video / image region as a third text label;

[0018] S1.6 Combine the first text tag, the second text tag and the third text tag together as the user's input sentence.

[0019] Optionally, in S2, the following steps are included:

[0020] S2.1: Formatting the user input sentence and inputting it into the large model I to obtain an output text as an initial reply sentence;

[0021] S2.2: Determine whether the initial response statement contains a question sentence. If so, cache the question sentence for separate processing, including two optional processing methods: Method (1) is to input the question sentence into the large model II to obtain N expected answers (N is a positive integer), combine the answer and the question sentence into a question-answer body, and then replace the question sentence in the initial response statement with the question-answer body to obtain an intelligent response statement; Method (2) is to combine the question sentence and the initial response statement into an intelligent response statement.

[0022] It is worth noting that if S2.2 adopts the first interrogative sentence processing method, the agent's reply sentence may contain two types of sentences: declarative sentences and question-answering sentences. If S2.2 adopts the second interrogative sentence processing method, the agent's reply sentence may contain two types of sentences: declarative sentences and interrogative sentences.

[0023] Optionally, in S3, include the following steps:

[0024] S3.1: Determine whether the reply sentence of the agent contains a declarative sentence. If so, input the declarative sentence into the first type of pre-trained multimedia generation model to obtain a multimedia generation result whose content matches the semantics of the declarative sentence as the first multimedia material; or use a retrieval technology to search and obtain multimedia data that matches the semantics of the declarative sentence as the first multimedia material;

[0025] S3.2: Determine whether the reply sentence of the agent contains a question and answer body. If yes, input the question and answer body into the second type of pre-trained multimedia generation model to obtain a multimedia generation result corresponding to the question and answer body as the second multimedia material; or use a retrieval technology to search and obtain multimedia data corresponding to the question and answer body as the second multimedia material;

[0026] S3.3: combining the first multimedia material and the second multimedia material in a corresponding sentence order to generate content in one or more forms of text, image, video, animation, and display the content to the user;

[0027] S3.4: Determine whether the agent's reply statement contains a question sentence, and if so, output the question sentence to the user.

[0028] Specifically, in S3, the multimedia generation model is a pre-trained artificial intelligence algorithm model, whose input is the question and answer body, that is, text label data, and whose output is multimedia data in one or more forms corresponding to the input text content, including but not limited to text, pictures, videos, animations, etc. The multimedia generation model includes but is not limited to large language models, text video models, text graphs, etc.

[0029] It is worth noting that, optionally, in S3, there are two ways to obtain the first multimedia material, and one of the two can be selected or used in combination. The same is true for the way to obtain the second multimedia material. Optionally, when the interrogative sentence in S2.2 is processed using the method (2), step S3.2 can be omitted; when the interrogative sentence in S2.2 is processed using the method (1), step S3.4 can be omitted.

[0030] Optionally, in S4, the following steps are included:

[0031] S4.1: The user interacts with the multimedia content generated by the multimedia generation model or displayed by the external retrieval technology, and the interactive actions include but are not limited to selection, zooming in, zooming out, moving, positioning, and outlining operations;

[0032] S4.2: The user's interaction with multimedia content is analyzed and screened using the same method as S1.5 to analyze the patient's intention and generate interaction results;

[0033] S4.3: The interaction result enters the next round of interaction in the form of the third text label.

[0034] Optionally, in S2, the agent reply statement includes: one or two statements in the declarative sentence and the question-answer body. The large model described in S2 includes large model I and large model II, which are pre-trained artificial intelligence algorithm models, and the input and output are both text data, including but not limited to large language models. Among them, large model I plays the role of a chat doctor to output answers and suggestions, and large model II plays the role of a user to output the predicted results of the user's answer. The two can use the same model or different models.

[0035] Optionally, in S3, the multimedia generation model is a pre-trained artificial intelligence algorithm model, whose input is the question and answer body, that is, text label data, and whose output is multimedia data in one or more forms corresponding to the input text content, including but not limited to text, pictures, videos, animations, etc. The multimedia generation model includes but is not limited to large language models, Vincent video models, Vincent graph models, etc., and its working principle includes using the collected sample data to train the model with the help of the back propagation algorithm to obtain the optimal parameters and using the reinforcement learning strategy to adjust the model to achieve a certain generation function.

[0036] A multimedia intelligent interactive device, comprising:

[0037] Input analysis module: used to obtain multi-modal user input information and obtain user input sentences using the input analysis model;

[0038] Response generation module: used to obtain the agent response sentence based on the user input sentence using the pre-trained large model;

[0039] Multimedia response module: used to obtain the agent response based on the agent reply statement using multimedia generation model or retrieval technology;

[0040] User interaction module: used for users to freely operate the generated content.

[0041] Optionally, the input analysis module specifically includes:

[0042] The speech input analysis submodule is used to obtain the speech input by the user; pre-process the input speech; input the pre-processed speech data into the pre-trained speech-to-text model to obtain the output text;

[0043] The image / video input analysis submodule is used to obtain the coordinate trajectory of the user's screen operation and the interface image corresponding to the start time of the screen operation by using the screen operation detection technology; input the interactive interface image corresponding to the start time of the screen operation into the pre-trained image / video analysis model to obtain the partition description of different areas in the image / video; obtain the user's intention label according to the coordinate trajectory of the screen operation by using the operation trajectory recognition technology; determine the classification description of the corresponding video image area according to the coordinate trajectory of the screen operation; combine the user's intention label with the description label of the video image area as the image content description;

[0044] The text analysis submodule is used to record typed text and use the pre-trained model to correct and polish the recorded text.

[0045] Optionally, a process control module is also included to pause or end the session.

[0046] A computer storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any one of the steps of a multimedia intelligent interaction method.

[0047] It can be seen from the above technical solutions that, compared with the prior art, the present invention provides an intelligent multimedia interaction method, system and storage medium, which realizes multimedia information exchange in the process of multiple rounds of interaction between the intelligent agent and the user. On the one hand, the intelligent agent can not only recognize the text information typed by the user, but also increase the demand analysis of combining the user and the screen operation, which greatly improves the flexibility and efficiency of the communication method; on the other hand, the intelligent agent can convey information and explain problems to the user in a way that is not limited to text, such as graphics, images, videos, animations, etc., so as to realize the expression of complex connotations and the transmission of non-verbal information in the conversation, and improve the service quality of the intelligent agent. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0049] Figure 1 An example flow chart of the user and multimedia interaction understanding input content of the present invention;

[0050] Figure 2 This is a flowchart of multimedia intelligent interaction based on a large model of the present invention;

[0051] Figure 3 It is a structural schematic diagram of the present invention;

[0052] Figure 4 The interactive effect diagram of an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0054] The embodiment of the present invention discloses a multimedia intelligent interaction method, such as Figure 2 As shown, the following steps are included:

[0055] S1: Obtain multimodal user input information and obtain the user input text using the input analysis model;

[0056] S2: Based on the user input text, use the large model to obtain the agent's reply information;

[0057] S3: Step 3, according to the agent reply information, using multimedia generation model or retrieval technology to obtain the agent response;

[0058] S4: The user interacts with the multimedia content, generates corresponding results and enters the next round of interaction.

[0059] The execution subject of the intelligent interaction method provided in this application is an intelligent interaction device. The intelligent interaction device can be specifically divided into an intelligent interaction system, a terminal device, etc. In this embodiment, the execution subject is collectively referred to as an intelligent agent for explanation.

[0060] Figure 1The figure is a schematic diagram of an implementation scenario of an embodiment disclosed in this specification. The solid arrow represents the conversation process between the agent and the user, and the content pointed to by the hollow arrow represents the result data example generated by each step of the operation of the agent in the conversation. In this embodiment, the user's initial instruction can be guided by the agent first displaying a piece of multimedia data, and the specific content is not limited to a text or video of a self-introduction and instructions for use. When the agent interacts with the user, the acceptable user input information includes voice, image, video, text, screen operation and other forms, and can also be a combination of one or more of them. The screen operation includes but is not limited to "click", "slide", "zoom", etc. The object of the operation can be a computer display screen or a display screen of other electronic products. The operation method includes but is not limited to a mouse, an electronic pen, a finger touch, etc. The agent can provide some buttons on the interactive interface to facilitate the user to issue instructions using screen operations, and can also analyze user needs with the help of gesture recognition, cursor drag trajectory recognition and other technologies. Analyze the user's input data one by one to obtain text labels representing the content of the instruction, and integrate them together to form a user input statement. If one or more data are missing, the default value is retained for it. The integrated user input statement is as follows: Figure 1 As shown in the example. Note that the order of analyzing and integrating multiple forms of data is not limited here, and the integration methods include but are not limited to splicing, fusion, etc. Afterwards, the intelligent body will generate a reply sentence based on the user's input sentence, which may include answers to the user's questions and re-questions to the user. The intelligent body will generate a multimedia response with matching content based on the reply sentence and display it to the user. The multimedia response includes but is not limited to text, images, animations, videos, etc. While the intelligent body displays the multimedia response, it is also waiting for a new round of user input to enter the next round of interaction.

[0061] Figure 2 A flowchart of a multi-round interaction method between an agent and a user according to an embodiment is shown: Figure 2 As shown, the intelligent multimedia interaction method includes the following steps: step S1, obtaining user input information, using the input analysis model to obtain the user input sentence; step S2, according to the user input sentence, using the pre-trained large model to obtain the intelligent agent reply sentence; step S3: according to the intelligent agent reply sentence, using the multimedia generation model or retrieval technology to obtain the intelligent agent response; step S4: the user interacts with the multimedia content generated by the intelligent agent to generate part of the content of the next round of interactive input. The specific execution process of the above steps is described in detail below.

[0062] Specifically, the input analysis model described in step S1 may include three submodules: a pre-trained image / video analysis model, a pre-trained speech-to-text model, and a pre-trained typed text processing model. The three submodules are used to process three forms of user input information respectively. Moreover, all three are pre-trained artificial intelligence algorithm models, and their working principle is to collect sample data that meets the ideal input and output of the model, and use the sample data to adjust the preset model parameters with the help of loss function, back propagation and other methods, so that the model has the function of converting input data into output of a specific mode. Among them, the input of the text processing model is the original text data that may contain grammatical errors, typos and other problems, and the output is the corrected text data, which mainly functions to correct and polish the text; the input of the speech-to-text model is the voice data recorded by the user through the microphone, and the output is text data, which mainly functions to convert the user's voice data into text data of the corresponding content.

[0063] Step S1, obtain multi-modal user input information, and use the input analysis model to obtain the user input sentence. The specific execution process includes:

[0064] S1.1: Use multimedia input devices to obtain user input, including but not limited to input of text, voice, image, video and other information.

[0065] S1.2: Preprocessing the input multimedia content, including but not limited to operations such as denoising, enhancement, and format conversion.

[0066] S1.3: Input the text typed by the user into the text processing model for error correction, polishing and other processing to obtain a first text label.

[0067] S1.4: Input the preprocessed speech data into a dedicated or general speech-to-text model, and obtain the output text as the second text label.

[0068] S1.5: Input the image / video into the pre-trained image instance segmentation model to obtain classification labels for different regions in the image / video. According to the action trajectory, the user's intention label is obtained using the operation trajectory recognition technology. The intention label includes but is not limited to "select", "zoom in", "zoom out", "drag", etc. According to the coordinate trajectory of the operation, the description of the corresponding image / video area is determined, and the user's intention label is combined with the description label of the video / image area as the third text label.

[0069] S1.6 Combine the first text tag, the second text tag and the third text tag together as the user's input sentence.

[0070] Among them, the input analysis model described in S1 includes three sub-models: a text processing model, a speech-to-text model, and an image instance segmentation model. All three are pre-trained artificial intelligence algorithm models. Among them, the input of the text processing model is the original text data that may contain grammatical errors, typos, etc., and the output is the corrected text data, which mainly functions to correct and polish the text; the input of the speech-to-text model is the voice data recorded by the user through a microphone, and the output is text data, which mainly functions to convert the user's voice data into text data of the corresponding content.

[0071] It is worth noting that, when one or several text labels are missing due to the diversity of user input, the missing text labels can be set to default values ​​or directly discarded in step 1.6.

[0072] Step S2, based on the user input sentence, use the pre-trained large model to obtain the agent reply sentence. The specific execution process includes:

[0073] S2.1: Formatting the user input sentence and inputting it into the large model I to obtain an output text as an initial reply sentence;

[0074] S2.2: Determine whether the initial response statement contains a question sentence. If so, cache the question sentence for separate processing, including two optional processing methods: Method (1) is to input the question sentence into the large model II to obtain N expected answers (N is a positive integer), combine the answer and the question sentence into a question-answer body, and then replace the question sentence in the initial response statement with the question-answer body to obtain an intelligent response statement; Method (2) is to combine the question sentence and the initial response statement into an intelligent response statement.

[0075] It is worth noting that the large model I described in S2.1 and the large model II described in S2.2 can be the same model that plays two roles (the former plays the role of the intelligent agent, and the latter plays the role of the user), or they can be two models that are trained separately, and their training data includes but is not limited to dialogue data in the same form as the user input data and a large amount of corpus data in related fields. It is worth noting that if S2.2 adopts the (1) interrogative sentence processing method, the intelligent agent's reply sentence may include two types of sentences: declarative sentences and question-answering sentences. If S2.2 adopts the (2) interrogative sentence processing method, the intelligent agent's reply sentence may include two types of sentences: declarative sentences and interrogative sentences.

[0076] Step S3, based on the agent reply statement, using multimedia generation model or retrieval technology to obtain the agent response. The specific execution process includes:

[0077] S3.1: Determine whether the reply statement of the agent contains the declarative sentence, and if so, input the declarative sentence into the first type of pre-trained multimedia generation model to obtain a multimedia generation result whose content matches the semantics of the declarative sentence as the first multimedia material; or use a retrieval technology to search and obtain multimedia data that matches the semantics of the declarative sentence as the first multimedia material;

[0078] S3.2: Determine whether the reply statement of the agent contains the question and answer body, and if so, input the question and answer body into the second type of pre-trained multimedia generation model to obtain a multimedia generation result corresponding to the question and answer body as the second multimedia material; or use a retrieval technology to search and obtain multimedia data corresponding to the question and answer body as the second multimedia material;

[0079] S3.3: combining the first multimedia material and the second multimedia material in a corresponding sentence order to generate one or more forms of content such as text, image, video, animation, etc. and displaying the content to the user;

[0080] S3.4: Determine whether the agent's reply statement contains a question sentence, and if so, output the question sentence to the user.

[0081] Specifically, in S3, the multimedia generation model is a pre-trained artificial intelligence algorithm model, whose input is the question and answer body, that is, text label data, and whose output is multimedia data in one or more forms corresponding to the input text content, including but not limited to text, pictures, videos, animations, etc. The multimedia generation model includes but is not limited to large language models, text video models, text graphs, etc.

[0082] It is worth noting that, optionally, in S3, there are two ways to obtain the first multimedia material, either of which can be selected or used in combination. The same is true for the way to obtain the second multimedia material. Optionally, when the interrogative sentence in S2.2 is processed using the method (2), step S3.2 can be omitted; when the interrogative sentence in S2.2 is processed using the method (1), step S3.4 can be omitted. Step S4, generates multimedia content based on the intelligent agent, and the user interacts with the generated content to obtain a further response. The specific execution process includes:

[0083] S4.1: Users interact with multimedia content generated by large models or displayed by external retrieval technology. Interactions include but are not limited to selection, zooming in, zooming out, moving, positioning, and outlining.

[0084] S4.2: The user's interaction with multimedia content is analyzed and screened using the same method as S1.5 to analyze the patient's intention and generate interaction results;

[0085] S4.3: The interaction result content enters the next round of interaction in text form as the third text label.

[0086] Among them, it is not difficult to understand that in order to facilitate user selection, in this embodiment, the three sentences (declarative sentences, interrogative sentences, and question-and-answer bodies) that may be included in the intelligent agent reply statement in step 2 are processed separately. Declarative sentences are used to answer user questions, and question-and-answer bodies or interrogative sentences are used to further understand user information, which is conducive to the next step of accurately judging user needs. Step S3.2 The second multimedia material obtained according to the question-and-answer body in the intelligent agent reply statement can provide answer options for users to click and select, thereby improving the information communication effect. The first multimedia material and the second multimedia material may be missing depending on the content of the intelligent agent reply statement. Therefore, in this case, another instance of step S3.3 does not have the combination process of the first multimedia material and the second multimedia material, but directly uses the obtained first multimedia material or the second multimedia material as the intelligent agent response. Moreover, in S3, there are two ways to obtain the first multimedia material, and one of them can be selected or used in combination. The same is true for the way to obtain the second multimedia material. In the specific implementation process, it is not difficult to find that when the question sentence in S2.2 is processed by the method (2), step S3.2 can be omitted; when the question sentence in S2.2 is processed by the method (1), step S3.4 can be omitted.

[0087] It should be further explained that the intelligent multimedia multi-round interaction method is a cyclic process of S1→S2→S3→S4→S1 until the user ends the session and terminates the cyclic process. S1 to S4 are a loop body, that is, a round of the session process. The user ends the session in ways including but not limited to closing the interactive interface, disconnecting the power supply of the interactive device, etc.

[0088] In order to implement the above embodiments, the present application also proposes a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by the server-side processor, the server-side can execute an intelligent interaction method, which includes: first, obtaining multimodal user input information, using the input analysis model to obtain the user input sentence; second, according to the user input sentence, using the pre-trained large model, obtaining the agent reply sentence; finally, according to the agent reply sentence, using the multimedia generation model or retrieval technology, obtaining the agent response. Through the method provided by the embodiment of this specification, multimedia information exchange can be realized in the multi-round interaction process between the agent and the user. On the one hand, the agent can not only recognize the text information typed by the user, but also increase the demand analysis combining the user's voice and screen operation, which greatly improves the flexibility and efficiency of the communication method; on the other hand, the agent can convey information and explain problems to the user in a way that is not limited to text, such as graphics, video, animation, etc., so as to realize the expression of complex connotations and the transmission of non-verbal information in the conversation, and improve the service quality of the agent. Therefore, the embodiments of this specification can increase the amount of information transmitted between the intelligent agent and the user in the interaction in the form of multimodal data, thereby meeting the user's diverse communication needs, achieving flexible and efficient information exchange, and improving the user experience.

[0089] Corresponding to the intelligent interaction method provided in the above-mentioned embodiment, an embodiment of the present application also provides an intelligent interaction device. Since the intelligent interaction device provided in the embodiment of the present application corresponds to the intelligent interaction method provided in the above-mentioned embodiment, the implementation method of the aforementioned intelligent interaction method is also applicable to the intelligent interaction device provided in this embodiment, and will not be described in detail in this embodiment.

[0090] Figure 3 This is a schematic diagram of the structure of an intelligent interactive device provided in an embodiment of the present application. Figure 3 As shown, the intelligent interactive device 400 includes:

[0091] Input analysis module 41: obtains multi-modal user input information and obtains user input sentences using an input analysis model.

[0092] Reply generation module 42: Based on the user input sentence, use the pre-trained large model to obtain the intelligent agent reply sentence.

[0093] Multimedia response module 43: Based on the agent reply statement, a multimedia generation model or retrieval technology is used to obtain the agent response.

[0094] Optionally, as an embodiment, the input analysis module 41 includes:

[0095] The input text analysis submodule has the following specific functions: recording the typed text and using the pre-trained model to correct and polish the recorded text.

[0096] The speech input analysis submodule has the following specific functions: first, the user's voice is acquired using a microphone; second, the recorded voice is preprocessed; and finally, the preprocessed voice data is input into the pre-trained speech-to-text model to obtain output text.

[0097] The video image input analysis submodule has the following specific functions: first, using screen operation detection technology to obtain the coordinate trajectory of the user's screen operation and the interface image corresponding to the start time of the screen operation; secondly, inputting the interactive interface image corresponding to the start time of the screen operation into the pre-trained image analysis model to obtain the classification description of different areas in the image; using the operation trajectory recognition technology according to the coordinate trajectory of the screen operation to obtain the user's intention label; determining the classification description of the corresponding video image area according to the coordinate trajectory of the screen operation; and finally combining the user's intention label with the classification description of the video image area as input.

[0098] Optionally, as an embodiment, the device may further include: a process control module, used to pause or end the session.

[0099] Figure 4 This is a schematic diagram of a multimedia interaction example shown in the implementation process of the present invention. The large model can generate a multimedia image based on the user input view to more clearly and accurately obtain the content that the user wants to express. The user can interact with the model-generated content by activating a certain operation function through the operation icon on the image.

[0100] It should be noted that the multimedia generated content is not limited to text, voice, image, video, animation, etc., and the generated format can be one or more. In addition, the multimedia operation action will change according to the generated multimedia content.

[0101] According to another embodiment, there is also provided a computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute a combination of Figure 2 The method described.

[0102] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0103] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multimedia intelligent interaction method, characterized in that: The following steps are involved: S1: Obtain multimodal user input information and obtain the user input text using the input analysis model; S2: Based on the user input text, use the large model to obtain the agent's reply information; S3: according to the agent reply information, using a multimedia generation model or retrieval technology to obtain an agent response; S4: The user interacts with the multimedia content, generates corresponding results and enters the next round of interaction.

2. A multimedia intelligent interaction method according to claim 1, characterized in that: In S1, the following steps are included: S1.1: Use multimedia input devices to obtain user input, including but not limited to text, voice, image, and video information input; S1.2: Preprocessing the input multimedia content, including but not limited to denoising, enhancement, and format conversion; S1.3: Inputting the text typed by the user into the text processing model for error correction and polishing to obtain a first text label; S1.4: input the preprocessed speech data into a dedicated or general speech-to-text model, and the obtained output text is used as the second text label; S1.5: Input the image / video into a pre-trained image instance segmentation model to obtain classification labels for different regions; obtain the user's intention label according to the action trajectory using the existing operation trajectory recognition technology, determine the description of the corresponding image / video region according to the coordinate trajectory of the operation, and combine the user's intention label with the description label of the video / image region as a third text label; S1.6 Combine the first text tag, the second text tag and the third text tag together as the user's input sentence.

3. A multimedia intelligent interaction method according to claim 1, characterized in that: In S2, the following steps are included: S2.1: Formatting the user input sentence and inputting it into the large model I to obtain an output text as an initial reply sentence; S2.2: Determine whether the initial response statement contains a question sentence. If so, cache the question sentence for separate processing, including two optional processing methods: Method (1) is to input the question sentence into the large model II to obtain N expected answers, where N is a positive integer, and combine the answer and the question sentence into a question-answer body, and then replace the question sentence in the initial response statement with the question-answer body to obtain an intelligent response statement; Method (2) is to combine the question sentence and the initial response statement into an intelligent response statement.

4. A multimedia intelligent interaction method according to claim 1, characterized in that: In S3, the following steps are included: S3.1: Determine whether the reply sentence of the agent contains a declarative sentence. If so, input the declarative sentence into the first type of pre-trained multimedia generation model to obtain a multimedia generation or retrieval result whose content matches the semantics of the declarative sentence as the first multimedia material; or use the retrieval technology to search and obtain multimedia data that matches the semantics of the declarative sentence as the first multimedia material; S3.2: Determine whether the reply sentence of the agent contains a question and answer body. If yes, input the question and answer body into the second type of pre-trained multimedia generation model to obtain a multimedia generation result corresponding to the question and answer body as the second multimedia material; or use a retrieval technology to search and obtain multimedia data corresponding to the question and answer body as the second multimedia material; S3.3: Combine the first multimedia material and the second multimedia material in the corresponding sentence order to generate content in one or more forms of text, image, video, animation, and display them to the user. S3.4: Determine whether the agent's reply statement contains a question sentence, and if so, output the question sentence to the user.

5. A multimedia intelligent interaction method according to claim 1, characterized in that: In S4, the following steps are included: S4.1: Users interact with multimedia content generated by large models or displayed by external retrieval technology. Interactions include but are not limited to selection, zooming in, zooming out, moving, positioning, and outlining operations; S4.2: The user's interaction with multimedia content is analyzed and screened using the same method as S1.5 to analyze the patient's intention and generate interaction results; S4.3: The interaction result enters the next round of interaction in the form of the third text label.

6. A multimedia intelligent interaction method according to claim 3, characterized in that: If S2.2 adopts the question sentence processing method (1), the intelligent agent's reply sentence may include two types of sentences: declarative sentences and question-answer bodies; if S2.2 adopts the question sentence processing method (2), the intelligent agent's reply sentence may include two types of sentences: declarative sentences and question-answer bodies; the large model described in S2 includes large model I and large model II, which are pre-trained artificial intelligence algorithm models, and the input and output are both text data, including but not limited to large language models; among them, large model I plays the role of a chat doctor to output answers and suggestions, and large model II plays the role of a user to output the predicted results of the user's answers, and the two use the same model or different models.

7. A multimedia intelligent interactive device, characterized in that: include: Input analysis module: used to obtain multi-modal user input information and obtain user input sentences using the input analysis model; Response generation module: used to obtain the agent response sentence based on the user input sentence using the pre-trained large model; Multimedia response module: used to obtain the agent response based on the agent reply statement using multimedia generation model or retrieval technology; User interaction module: used for users to freely operate the generated content.

8. A multimedia intelligent interactive device according to claim 7, characterized in that: The input analysis module specifically includes: The speech input analysis submodule is used to obtain the speech input by the user; pre-process the input speech; input the pre-processed speech data into the pre-trained speech-to-text model to obtain the output text; The image / video input analysis submodule is used to obtain the coordinate trajectory of the user's screen operation and the interface image corresponding to the start time of the screen operation by using the screen operation detection technology; input the interactive interface image corresponding to the start time of the screen operation into the pre-trained image / video analysis model to obtain the partition description of different areas in the image / video; obtain the user's intention label according to the coordinate trajectory of the screen operation by using the operation trajectory recognition technology; determine the classification description of the corresponding video image area according to the coordinate trajectory of the screen operation; combine the user's intention label with the description label of the video image area as the image content description; The text analysis submodule is used to record typed text and use the pre-trained model to correct and polish the recorded text.

9. A multimedia intelligent interactive device according to claim 7, characterized in that: A process control module is also included for pausing or ending a session.

10. A computer storage medium, characterized in that: The computer storage medium stores a computer program, and when the computer program is executed by the processor, the steps of a multimedia intelligent interaction method as described in any one of claims 1 to 6 are implemented.

Images