Interaction method and device based on multi-modal communication, equipment and storage medium

Abstract

The invention relates to the technical field of exhibition, and discloses an interaction method and device based on multi-modal communication, equipment and a storage medium, and the method comprises the steps: carrying out the behavior analysis of visitors, so as to obtain multi-modal input information; inputting the multi-modal input information into a multi-modal large model, and enabling the multi-modal large model to independently extract behavior characteristics of each visitor to obtain visitor identity information and visitor intention information; establishing an independent communication link between the multi-modal large model and the visitors according to the identity information of the visitors; generating a virtual interaction role according to the intention information of the visitor; carrying out an interaction activity according to the independent communication link and the virtual interaction role to generate an interaction result; by accurately distinguishing the operation behavior and the problem content of each visitor, the virtual interaction role is controlled to give feedback in time, information transmission disorder and interaction conflicts are avoided, and the problem that a traditional interaction system is difficult to process a large number of concurrent interaction requests during multi-visitor visiting is effectively solved.

Description

Technical Field

[0001] The present invention relates to the field of exhibition technology, and in particular to an interactive method, device, equipment and storage medium based on multimodal communication. Background Art

[0002] In today's digital age, exhibition venues such as science and technology museums are committed to providing visitors with a richer and more immersive experience.

[0003] However, there are many problems with the existing science and technology museum exhibition scenes. One notable problem is that in multi-person visitor scenes, it is difficult for the system to accurately respond to personalized interactions based on the behavior and needs of each visitor. Traditional exhibition interaction systems often adopt a unified interaction mode. When facing multiple interactive objects at the same time, they cannot handle a large number of concurrent interaction requests, nor can they effectively distinguish the intentions of different visitors, resulting in confusion in information transmission and a greatly reduced interactive experience. For example, when multiple visitors are active in different exhibit areas at the same time, the system may not be able to accurately identify each visitor's attention and operation on a specific exhibit, and thus cannot provide targeted explanations and interactions, affecting visitors' understanding of the exhibition content and interest stimulation. This problem seriously restricts the intelligence level of science and technology museum exhibitions and visitor satisfaction, and needs to be solved urgently. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present invention is to provide an interactive method, device, equipment and storage medium based on multimodal communication, by obtaining multimodal input information from each visitor, and generating exclusive exhibition interaction strategies for different visitors based on this information, and then completing interactive activities with exclusive virtual interactive characters through independent communication links; the interactive method based on multimodal communication provided by the present invention can accurately distinguish the operating behavior and question content of each visitor, and give accurate feedback and answers in time through virtual interactive characters, avoiding information transmission confusion and interaction conflicts, ensuring the fluency and efficiency of the interactive experience, and effectively solving the problem that traditional interactive systems are difficult to handle a large number of concurrent interaction requests when multiple people visit.

[0005] The first aspect of the present invention provides an interactive method based on multimodal communication, comprising: performing behavioral analysis on visitors to obtain multimodal input information; inputting the multimodal input information into a multimodal large model, so that the multimodal large model independently extracts the behavioral characteristics of each visitor to obtain visitor identity information and visitor intention information; establishing an independent communication link between the multimodal large model and the visitor based on the visitor identity information; generating a virtual interactive character based on the visitor intention information; and conducting interactive activities based on the independent communication link and the virtual interactive character to generate an interactive result.

[0006] Optionally, in a first implementation method of the first aspect of the present invention, the obtaining of multimodal input information from the visitor includes: capturing the visitor's gestures, expressions and body movements through a camera to generate visual information; collecting the visitor's voice signal through a microphone to generate sound information; calculating the visitor's spatial position through a positioning sensor to generate position information; and integrating the visual information, sound information and position information to obtain multimodal input information.

[0007] Optionally, in a second implementation method of the first aspect of the present invention, the multimodal input information is input into the multimodal big model, so that the multimodal big model independently extracts the behavioral characteristics of each visitor to obtain the visitor's identity information and the visitor's intention information, including: extracting the visitor's voiceprint features and facial features from the multimodal input information, so that the multimodal big model confirms the visitor's identity based on the voiceprint features and facial features to obtain the visitor's identity information; extracting the visitor's voice features and position features from the multimodal input information, so that the multimodal big model infers the visitor's intention based on the voice features and position features to obtain the visitor's intention information.

[0008] Optionally, in a third implementation method of the first aspect of the present invention, the generating of virtual interactive characters according to visitor intention information includes: generating navigation planning information, interaction planning information and exhibit recommendation information according to the visitor intention information; generating a virtual navigator according to the navigation planning information; generating a virtual character according to the interaction planning information; generating an exhibit intelligent body according to the exhibit recommendation information; and allocating the virtual navigator, virtual character and exhibit intelligent body to a preset plurality of interaction links to complete the generation process of the virtual interactive characters.

[0009] Optionally, in a fourth implementation method of the first aspect of the present invention, the interactive activities are carried out according to the independent communication link and the virtual interactive characters to generate interactive results, including: assigning different virtual interactive characters to visitors according to the scenes in which the visitors are located; collecting the visitors' behavior data in real time, and transmitting the behavior data to the multimodal large model through an independent communication link to generate virtual interactive character guidance data; controlling the virtual interactive characters to carry out interactive activities according to the virtual interactive character guidance data, and obtaining real-time multimodal input information from the visitors to generate interactive results.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, different virtual interactive characters are assigned to visitors according to the scene in which the visitors are located, including: when the visitor is in a preset navigation scene, the virtual interactive character is determined to be a virtual navigator, and the virtual navigator is assigned to the visitor; when the visitor is in a preset interaction scene, the virtual interactive character is determined to be a virtual character, and the virtual character is assigned to the visitor; when the visitor is in a preset exhibit display scene, the virtual interactive character is determined to be an exhibit agent, and the exhibit agent is assigned to the visitor.

[0011] Optionally, in a sixth implementation method of the first aspect of the present invention, after conducting interactive activities based on independent communication links and virtual interactive characters to generate interactive results, it also includes: using a multimodal large model to optimize and analyze the interactive results to obtain analysis results; when the analysis results meet preset optimization conditions, generating an optimized interaction strategy based on the interactive results; and controlling the virtual interactive characters to conduct interactive activities based on the optimized interaction strategy.

[0012] The second aspect of the present invention provides an interactive device based on multimodal communication, including: an acquisition module, used to perform behavioral analysis on visitors to obtain multimodal input information; an identification module, used to input the multimodal input information into a multimodal large model, so that the multimodal large model independently extracts the behavioral characteristics of each visitor to obtain visitor identity information and visitor intention information; an establishment module, used to establish an independent communication link between the multimodal large model and the visitor according to the visitor identity information; a generation module, used to generate a virtual interactive character according to the visitor intention information; an interaction module, used to carry out interactive activities based on the independent communication link and the virtual interactive character to generate an interactive result.

[0013] A third aspect of the present invention provides an interactive device based on multimodal communication, the interactive device based on multimodal communication comprising: a memory and at least one processor, the memory storing instructions; at least one of the processors calling the instructions in the memory so that the interactive device based on multimodal communication executes each step of the interactive method based on multimodal communication described above.

[0014] A fourth aspect of the present invention provides a computer-readable storage medium having instructions stored thereon, and when the instructions are executed by a processor, the steps of any of the above-mentioned methods for interaction based on multimodal communication are implemented.

[0015] In the technical solution of the present invention, the behavior of visitors is analyzed to obtain multimodal input information, and the multimodal input information is input into a multimodal large model to extract the identity information and intention information of the visitors; first, an independent communication link between the multimodal large model and the visitors is constructed according to the identity information of the visitors, and then a virtual interactive role is generated according to the intention information of the visitors, and then interactive activities are carried out with the independent communication link and the virtual interactive role to obtain interactive results; the interactive method based on multimodal communication provided by the present invention can accurately distinguish the operating behavior and question content of each visitor, and give accurate feedback and answers in time through the virtual interactive role, avoiding information transmission confusion and interaction conflicts, ensuring the fluency and efficiency of the interactive experience, and effectively solving the problem that the traditional interactive system is difficult to handle a large number of concurrent interactive requests when multiple people visit. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The above and / or additional aspects and advantages of the present invention will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0017] Figure 1 A first flow chart of an interactive method based on multimodal communication provided by an embodiment of the present invention;

[0018] Figure 2 A second flow chart of an interactive method based on multimodal communication provided by an embodiment of the present invention;

[0019] Figure 3 A third flow chart of the interactive method based on multimodal communication provided by an embodiment of the present invention;

[0020] Figure 4 A fourth flow chart of an interactive method based on multimodal communication provided by an embodiment of the present invention;

[0021] Figure 5 A fifth flow chart of an interactive method based on multimodal communication provided by an embodiment of the present invention;

[0022] Figure 6 A sixth flow chart of an interactive method based on multimodal communication provided by an embodiment of the present invention;

[0023] Figure 7 A seventh flow chart of an interactive method based on multimodal communication provided by an embodiment of the present invention;

[0024] Figure 8 A schematic diagram of the structure of an interactive device based on multimodal communication provided by an embodiment of the present invention;

[0025] Fig. 9 A schematic diagram of the structure of an interactive device based on multimodal communication provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The present invention provides an interactive method, device, equipment and storage medium based on multimodal communication. The present invention performs behavioral analysis on visitors to obtain multimodal input information, and inputs the multimodal input information into a multimodal large model to extract the visitor's identity information and intention information; first, an independent communication link between the multimodal large model and the visitor is constructed according to the visitor's identity information, and then a virtual interactive role is generated according to the visitor's intention information, and then interactive activities are carried out with the independent communication link and the virtual interactive role to obtain interactive results; the interactive method based on multimodal communication provided by the present invention can accurately distinguish the operating behavior and question content of each visitor, and give accurate feedback and answers in time through the virtual interactive role, thereby avoiding information transmission confusion and interaction conflict, ensuring the fluency and efficiency of the interactive experience, and effectively solving the problem that the traditional interactive system is difficult to handle a large number of concurrent interactive requests when multiple people visit.

[0027] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" or "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0028] For ease of understanding, the specific process of the embodiment of the present invention is described below. Figure 1 , an embodiment of the interaction method based on multimodal communication in an embodiment of the present invention includes:

[0029] 101. Conduct behavioral analysis on visitors to obtain multimodal input information;

[0030] In this embodiment, by deploying multiple video acquisition devices, sound acquisition devices and space sensing devices in the exhibition venue, the visual information, sound information and location information of the visitors are captured in real time; by collecting, integrating and analyzing these different modal information, multimodal input information is obtained; the multimodal input information covers the identity characteristics and action intentions of the visitors, and based on the multimodal input information, targeted exhibition interaction strategies can be formulated for the visitors.

[0031] 102. Input the multimodal input information into the multimodal large model, so that the multimodal large model independently extracts the behavioral characteristics of each visitor to obtain the visitor's identity information and the visitor's intention information;

[0032] In this embodiment, the acquired multimodal input information is input into a pre-trained multimodal large model. The multimodal large model, with its powerful deep learning architecture and a large amount of training data, can accurately and independently analyze and extract the unique behavioral characteristics of each visitor, thereby identifying the identity information and intention information of different visitors; the visitor identity information is used to confirm the identity of each visitor, thereby establishing an independent communication link for each visitor, thereby meeting the interaction needs of each visitor; the intention information is used to identify and analyze the visiting intention of each visitor, and establish a targeted interaction strategy for each visitor.

[0033] 103. Establish an independent communication link between the multimodal large model and the visitor based on the visitor's identity information;

[0034] In this embodiment, based on the visitor identity information extracted from the multimodal input information, the communication middleware is used to build an independent communication link between the multimodal large model and the visitor. The independent communication link has the characteristics of high bandwidth and low latency, and can transmit multimodal data in real time. Specifically, the communication middleware uses a message queue system (such as RabbitMQ) to implement an asynchronous, decoupled message transmission mechanism. The visitor's multimodal input information is put into the queue as a message, and then the multimodal large model obtains the message from the queue for processing, thereby ensuring that the multimodal large model can efficiently process high-concurrency communication requests; the communication middleware also adopts a message publishing / subscription mechanism, and the visitor device is used as a subscriber to subscribe to related topics according to demand. When the multimodal large model completes the related tasks, the result message is broadcast to the corresponding visitor device through the publishing mechanism, thereby realizing real-time information update; in addition, the communication middleware also uses the load balancing function of data routing to select the best path to transmit data according to the actual state of the multimodal large model, thereby reducing the load of the independent communication link and ensuring the stability and fluency of communication. The independent communication link established by the communication middleware can handle the concurrent interaction needs of multiple visitors, ensure the accurate transmission and interaction of information between the model and the visitor, and ensure the independence and security of the data to meet the interaction needs of each visitor. It should be noted that when a visitor is identified for the first time or his identity information is updated, a unique identity code is generated for him and a communication link is established based on it; if the visitor is an old user, his historical interaction records are retrieved, including previous interaction time, interaction topics, evaluation of virtual characters, etc. Based on these historical data, the cache strategy of the communication link is adjusted to preload the interaction resources that may be used to improve the interaction response speed.

[0035] 104. Generate virtual interactive characters based on visitor intention information;

[0036] In this embodiment, the visitor intention information is first analyzed, and then different virtual interactive roles are established according to different interaction needs and interaction scenarios; specifically, different virtual interactive roles are generated for navigation scenarios, interactive scenarios during exhibition introductions, and exhibit display scenarios, and corresponding knowledge systems and interactive skills are configured for these virtual interactive roles based on the visitor's intention information; an exclusive virtual interactive role is generated for each visitor, so that the visitor can obtain excellent interactive experience in real time during each process of visiting the exhibition.

[0037] 105. Conducting interactive activities based on independent communication links and virtual interactive roles to generate interactive results;

[0038] In this embodiment, after the virtual interactive character is generated and a communication link is established with the visitor, the scene in which the visitor is located is judged, and a corresponding virtual interactive character is assigned to interact with the visitor; because an independent communication link is used, the interaction between the visitor and the virtual character is also independent and undisturbed, thereby avoiding the virtual interactive character from making erroneous interactive behaviors due to interference from other visitors; during the interaction process, the visitor's multimodal input information is collected in real time, so as to adjust the output information of the virtual interactive character in real time to match the visitor's needs; at the same time, it is necessary to completely collect the visitor's multimodal input information, the output information of the virtual interactive character, the interaction timestamp, and the visitor's feedback information, and thereby generate an interaction result containing all the information of the interaction process; the interaction result is used to evaluate the effect of this interaction, and the interaction process of the virtual character can be optimized by analyzing the interaction result.

[0039] In the embodiment of the present invention, the behavior of the visitor is analyzed to obtain multimodal input information, and the multimodal input information is input into the multimodal large model to extract the visitor's identity information and intention information; first, an independent communication link between the multimodal large model and the visitor is constructed according to the visitor's identity information, and then a virtual interactive role is generated according to the visitor's intention information, and then interactive activities are carried out with the independent communication link and the virtual interactive role to obtain interactive results; finally, when the preset optimization conditions are met, an optimized interactive strategy is generated according to the interactive results and the virtual interactive role is controlled to carry out interactive activities accordingly; the interactive method based on multimodal communication provided by the present invention can accurately distinguish the operating behavior and question content of each visitor, and give accurate feedback and answers in time through the virtual interactive role, avoiding information transmission confusion and interaction conflict, ensuring the fluency and efficiency of the interactive experience, and effectively solving the problem that the traditional interactive system is difficult to handle a large number of concurrent interactive requests when multiple people visit.

[0040] See also Figure 2, a second embodiment of the interaction method based on multimodal communication in an embodiment of the present invention includes:

[0041] 201. Capture visitors’ gestures, expressions and body movements through cameras to generate visual information;

[0042] In this embodiment, multiple high-definition cameras are installed in the science exhibition hall, covering the range of visitors' activities, so as to ensure that the dynamic images of visitors can be captured clearly and smoothly; before visual capture, the camera needs to be calibrated so that it can accurately obtain the visitor's posture data in three-dimensional space, and the coordinates of various parts of the visitor's body in space are calculated through the stereoscopic vision technology of multiple cameras, providing a basis for the subsequent accurate analysis of gestures, expressions and body movements. Specifically, the captured images are analyzed and processed using computer vision algorithms to identify the facial features, gestures, facial expressions and body movements of the visitors, and the information is summarized as visual information; the facial features of the visitors can help the multimodal large model identify the identity of the visitors, and the gestures, facial expressions and body movements of the visitors can help the multimodal large model analyze the visitor's intention and satisfaction.

[0043] 202. Collecting the visitor's voice signal through a microphone to generate sound information;

[0044] In this embodiment, a microphone array is installed in the science exhibition hall and covers the activity range of visitors to ensure that the voice signals of visitors can be fully collected; the microphone array can integrate an automatic gain control function, which can automatically adjust the gain according to the environmental noise level and the strength of the voice signal to ensure that the collected voice signal has a suitable volume and signal-to-noise ratio; for the scene where multiple visitors speak at the same time, the microphone array uses beamforming technology to perform weighted summation processing on the sound signals collected by each microphone in the microphone array, so that the pickup direction of the microphone array is focused on the area where the target visitor is located, effectively suppressing noise interference from other directions, thereby distinguishing the voice signal of the target visitor and improving the accuracy of voice recognition; then, the collected voice signal is preprocessed, including noise reduction processing, echo cancellation, voice enhancement and other operations to generate sound information; the multimodal large model can identify the content of the visitor's speech by converting the sound information into text information, and can also analyze the sound information based on acoustic features to judge the emotional state of the visitor.

[0045] 203. Calculate the spatial position of the visitor through a positioning sensor to generate position information;

[0046] In this embodiment, a positioning sensor can be set to detect the range where the visitor is located, such as a Bluetooth positioning sensor, a Wi-Fi positioning sensor, and an infrared positioning sensor. The sensor receives the signal strength and signal arrival time difference of the device carried by the visitor, thereby calculating the visitor's location information; the visual information captured by the camera can also be used to calculate the visitor's coordinates in three-dimensional space, thereby generating location information; the location information can not only be used to determine the specific coordinates of the visitor in space, but can also be combined with the exhibit location information, interactive area location information, etc., to provide visitors with navigation guidance, personalized recommendations and other services.

[0047] 204. Integrate the visual information, the sound information and the position information to obtain multimodal input information;

[0048] In this embodiment, in order to ensure that the visual information, sound information and location information can be accurately matched, an accurate timestamp is added to each piece of data when collecting each modal information; through the timestamp, the visual image frames, voice signal segments and location information data collected at the same time or similar time are associated to form a complete multimodal data sample; and because the visual information, sound information and location information may use different data formats and encoding methods, before the information is integrated, it is necessary to convert the format of each modal data to improve the analysis efficiency of the multimodal large model; secondly, in order to ensure that the numerical range and characteristic scale of different modal data are comparable, it is necessary to normalize them; after the above steps, the integration of visual information, sound information and location information is completed to generate multimodal input information. The multimodal large model analyzes the multimodal input information to generate interaction strategies suitable for different visitors, so as to achieve more accurate and intelligent interaction.

[0049] See also Figure 3 , a third embodiment of the interaction method based on multimodal communication in the embodiment of the present invention includes:

[0050] 301. Extracting the visitor's voiceprint features and facial features from the multimodal input information, and enabling the multimodal large model to confirm the visitor's identity based on the voiceprint features and facial features to obtain the visitor's identity information;

[0051] In this embodiment, the multimodal large model extracts the visitor's voiceprint features and facial features from the multimodal input information respectively, so as to identify the visitor's identity information; specifically, for the voiceprint feature extraction step, the multimodal large model extracts sound information from the multimodal input information, and then performs frame processing on the sound information so as to extract features from each frame of sound signal; then for each frame of sound signal, its Mel frequency cepstral coefficient (MFC C) and fundamental frequency are extracted; the Mel frequency cepstral coefficient is a parameter that can effectively characterize the spectral characteristics of the speech signal, which simulates the perception characteristics of the human auditory system to the sound frequency; the fundamental frequency reflects the periodicity of the speech signal, which plays an important role in distinguishing the voiceprint features of different speakers; the features such as the Mel frequency cepstral coefficient and the fundamental frequency are combined to form a complete voiceprint feature vector. For the facial feature extraction step, the multimodal large model extracts visual information from the multimodal input information, then recognizes the face in the image based on the face detection algorithm, extracts the key features of the face, and uses the active shape model (ASM) or active appearance model (AAM) to accurately locate and describe the shape and texture information of the facial feature points to obtain the facial feature vector. Finally, by learning the voiceprint and facial feature patterns corresponding to different identities, the identity information of the visitor can be accurately identified based on the input voiceprint and facial features.

[0052] 302. Extracting the voice features and location features of the visitor from the multimodal input information, and enabling the multimodal large model to infer the visitor's intention based on the voice features and location features to obtain the visitor's intention information;

[0053] In this embodiment, the multimodal large model extracts the visitor's voice features and location features from the multimodal input information respectively, thereby identifying the visitor's intention information; specifically, for the voice feature extraction step, the multimodal large model extracts sound information from the multimodal input information, then converts the sound information into text information, and then performs lexical analysis on the text information, thereby extracting key information in the text, such as noun phrases (which may represent the name of the exhibit, the name of the place, etc.), verb phrases (indicating the visitor's action intention, such as asking, visiting, searching, etc.), to generate voice features. For the location feature extraction step, the multimodal large model extracts visual information and location information from the multimodal input information, determines the location area where the visitor is located, and thus generates location features. The multimodal large model infers the visitor's intention by combining the visitor's voice features and location features, preliminarily judges the visitor's intention using voice features, and then verifies the inferred intention in combination with the visitor's location features, improves the accuracy of intention inference, and finally outputs the visitor's intention information.

[0054] See also Figure 4A fourth embodiment of the interaction method based on multimodal communication in the embodiment of the present invention includes:

[0055] 401. Generate navigation planning information, interaction planning information and exhibit recommendation information according to visitor intention information;

[0056] In this embodiment, when the visitor's intention information includes navigation needs, a path planning algorithm is used to calculate the optimal or preferred navigation path based on the visitor's current location information and target location information, and navigation planning information is generated; when the visitor's intention information includes interaction needs (such as asking for exhibit details, discussing specific topics, etc.), the interaction process and content framework are determined based on the visitor's interaction needs, and interaction planning information is generated; the visitor's intention information is analyzed for preferences and potential points of interest for exhibits, exhibits that match the visitor's interest are screened from the exhibit database, and exhibit recommendation information is generated.

[0057] 402. Generate a virtual navigator according to the navigation planning information;

[0058] In this embodiment, the image and functions of the virtual navigator are designed based on the navigation planning information. The virtual navigator has the ability to display paths in a three-dimensional virtual environment or a two-dimensional map interface, and adjust the navigation strategy and prompt information in real time according to the visitor's position changes and operation feedback, to ensure that the visitor can reach the destination smoothly.

[0059] 403. Generate a virtual character according to the interaction planning information;

[0060] In this embodiment, the image and function of the virtual character are designed based on the interaction planning information; specifically, different virtual characters are generated according to the scenario and purpose of the interaction; the virtual character is used to provide visitors with background knowledge or topics related to the exhibition, such as answering questions, demonstrating historical scenes or guiding visitors' attention.

[0061] 404. Generate an exhibit intelligent entity according to the exhibit recommendation information;

[0062] In this embodiment, the display and recommendation functions of the exhibit agent are constructed based on the exhibit recommendation information; the exhibit agent is used to introduce the information of the recommended exhibits, so that visitors can have a more comprehensive understanding of the exhibits through multimedia materials such as high-definition pictures, video demonstrations, three-dimensional models, etc. of the exhibits; at the same time, the exhibit agent is also responsible for interacting with visitors, such as answering visitors' questions about the exhibits, collecting visitors' evaluation and feedback information on the exhibits, so as to further optimize the exhibit recommendation and display strategies.

[0063] 405. Allocate the virtual navigator, the virtual character, and the exhibit agent to a plurality of preset interaction links to complete the generation process of the virtual interaction character;

[0064] In this embodiment, the multimodal large model determines the sequence and key content of different interactive links by judging the visitor's current visiting intention and location information, thereby allocating the virtual navigator, virtual character and exhibit agent to multiple preset interactive links, thereby completing the generation process of the virtual interactive role. Specifically, the virtual navigator is allocated to the navigation-related interactive links, such as the initial navigation guidance after the visitor enters the interactive space, the path navigation in the process of finding a specific target, etc.; the virtual character is allocated to the link that requires deep interaction, knowledge explanation and discussion, such as the detailed introduction and theme discussion in the exhibit display area; the exhibit agent is allocated to the exhibit recommendation and display link. In the conversion process of the interactive link, the virtual interactive characters are coordinated with each other. When the virtual navigator guides the visitor to the exhibit display area, the virtual character takes over the interactive leadership and starts to interact deeply with the visitor related to the exhibits; after the interaction is over, if there is an exhibit recommendation link, the exhibit agent is allowed to interact with each other in exhibit recommendation.

[0065] See also Figure 5 A fifth embodiment of the interaction method based on multimodal communication in the embodiment of the present invention includes:

[0066] 501. Assign different virtual interactive roles to visitors according to the scene they are in;

[0067] In this embodiment, the multimodal large model divides the interactive space into scenes, such as navigation scenes, interaction scenes, and exhibit display scenes, etc.; the corresponding virtual interactive roles are assigned to visitors according to the scenes in which the visitors are located. When the visitor is in the preset navigation scene, the virtual interactive role is determined to be a virtual navigator, and the virtual navigator is assigned to the visitor; when the visitor is in the preset interaction scene, the virtual interactive role is determined to be a virtual role, and the virtual role is assigned to the visitor; when the visitor is in the preset exhibit display scene, the virtual interactive role is determined to be an exhibit intelligent agent, and the exhibit intelligent agent is assigned to the visitor; it should be noted that when the visitor needs multiple interactions at the same time, multiple virtual interactive roles serve the visitor at the same time, and the interactive role switching is done based on the transformation of the interactive scene. When the virtual navigator guides the visitor to the exhibit display area, the virtual role takes over the interactive dominance and starts to interact deeply with the visitor related to the exhibits; after the interaction is over, if there is an exhibit recommendation link, the exhibit intelligent agent is allowed to interact with each other in exhibit recommendation.

[0068] 502. Collecting the behavior data of visitors in real time, and transmitting the behavior data to the multimodal large model through an independent communication link to generate virtual interactive role guidance data;

[0069] In this embodiment, the body posture and movement changes of the visitors are continuously monitored by the camera, and the voice communication content of the visitors is collected by the microphone. This information reflects all the feedback of the visitors during the interaction process. This information is marked as the behavior data of the visitors and transmitted to the multimodal large model through an independent communication link; the multimodal large model generates virtual interaction role guidance data based on the received behavior data, combined with the knowledge graph and interaction rules; the virtual interaction role guidance data includes suggestions for adjusting the speed of the virtual character's explanation (such as slowing down the explanation speed when the visitor shows confusion), suggestions for adjusting the depth of the explanation content (such as adding relevant in-depth content when the visitor shows great interest in a certain knowledge point), and suggestions for changing the interaction method (such as when the visitor takes the initiative to initiate a topic, the virtual character changes from a one-way explanation to an interactive dialogue); the virtual interaction role guidance data is used to better control the interaction of the virtual character to better adapt to the real-time needs and status of the visitors.

[0070] 503. Control the virtual interactive character to carry out interactive activities according to the virtual interactive character guidance data, and obtain real-time multimodal input information from the visitor to generate interactive results;

[0071] In this embodiment, the virtual navigator, virtual character and exhibit intelligent body are controlled to carry out interactive activities according to the virtual interactive character guidance data; while the virtual interactive character is carrying out the interactive activities, the multimodal input information of the visitor is continuously collected through the camera, microphone and other interactive devices, and the output information of the virtual interactive character is recorded, and the interaction results are generated according to the multimodal input information and the multimodal output information.

[0072] It should be noted that during the interaction process, the multimodal communication service mechanism is used to flexibly handle the information flow and multimodal data exchange between virtual and reality, and intelligently select the appropriate communication method according to different interaction scenarios to ensure seamless connection between virtual exhibitions and real exhibitions.

[0073] Specifically, according to different interaction modes and interaction needs, the communication service mechanism determines the transmission requirements and selects different communication protocols accordingly. For example, data interaction in voice may use WebRTC or RTSP protocol, data interaction in video may use HLS or RTMP protocol, and data interaction in image and tactile feedback may use MQ TT or WebSocket protocol. These protocols can ensure the reliability, low latency and high bandwidth efficiency of interactive data in different network environments. The communication service mechanism adopts multi-channel synchronization technology to ensure the synchronization between various forms of interaction such as sound, vision, and touch, and ensures that voice interpretation and multi-modal data such as exhibit videos and tactile feedback can be synchronized by marking accurate timestamps for interactive data. At the same time, in response to high concurrent interaction needs, the communication service mechanism optimizes data transmission and signal quality through intelligent algorithms and technical means. QoS (quality of service) and QoE (quality of experience) management are adopted to optimize communication delay, packet loss rate, bandwidth consumption and other indicators, improve communication quality, and enable multiple visitors to have a good interactive experience.

[0074] The interaction between virtual interactive characters and visitors needs to ensure the coordination of multiple synchronization dimensions, including scenario synchronization, time synchronization, space synchronization, theme synchronization and knowledge synchronization; the purpose of scenario synchronization is to ensure that all interactions and events can occur in the same context between different virtual and real scenes; time synchronization is used to ensure that different devices and modules can be executed on the same timeline to avoid inconsistent interactions due to timing differences; space synchronization ensures that the physical location, virtual interactive characters, visitor locations, etc. in the virtual and real environments can be updated synchronously; theme synchronization ensures that all communication events and messages revolve around the same core theme; knowledge synchronization is used to ensure that all knowledge content in the system can flow between different modules and be uniformly managed and updated. By integrating the above multiple synchronization dimensions, the synchronization of multimodal communication is guaranteed; the communication service mechanism presets multiple fields in the communication protocol for the above multiple synchronization dimensions to cooperate with virtual interactive characters to carry out interactive activities.

[0075] Specifically, for scenario synchronization, the simulationDomain field in the communication protocol is used to define the "scenario domain" in which the current communication occurs; for example, when visitors are in the historical and cultural exhibition area, this field is identified as the "historical and cultural" scenario domain; the simulation time is set for each event and behavior through the simulationTime field to ensure that the virtual tour guide's explanation, the display of the exhibit intelligent body and other behaviors are synchronized with the visitor's behavior in the same simulation scenario.

[0076] Specifically, for time synchronization, the timestamp (real timestamp) and simulationTime (simulation time) fields in the communication protocol are used to define the timestamp of the interaction between the virtual space and the real space; for example, when a visitor enters a specific exhibit area, the simulation time may be paused or accelerated according to preset rules. At the same time, the system uses time synchronization algorithms (such as the NTP network time protocol) to ensure that the data inputs of multiple cameras, multiple microphones, and multiple sensors have a unified timestamp; all virtual interactions are based on the simulationTime (simulation time) field. When the virtual interactive character starts to interact, the relevant display information (such as images, videos, voice feedback, etc.) is triggered at the same simulation time to ensure that the multimodal information received by the visitor is coordinated in time.

[0077] Specifically, for spatial synchronization, the data field in the communication protocol is used to transmit the spatial coordinates of visitors, map the activity range of virtual interactive characters to the physical space in the real scene, and synchronously update the positions of visitors, virtual navigators, virtual characters, and exhibit agents based on real-time data input (such as cameras, sensors, and deep learning models) to ensure spatial consistency;

[0078] Specifically, for topic synchronization, the topic field in the communication protocol indicates the topic of the current communication, so that all virtual interactive roles serving the same visitor obtain relevant explanation content from the knowledge base according to the same topic field, ensuring the consistency of the interaction topic;

[0079] Specifically, for knowledge synchronization, the data field in the communication protocol synchronizes knowledge data, and a shared knowledge base system (such as one based on a graph database or knowledge graph) is used to store and manage knowledge data. All virtual interactive roles serving the same visitor will feed back interaction information, behavior data, and display data to the shared knowledge base system in real time based on the same data field. The multimodal large model then generates interaction logic for the virtual interactive roles based on the data in the shared knowledge base system, thereby ensuring that different virtual interactive roles interact based on the same knowledge information.

[0080] While the virtual interactive character is carrying out interactive activities, multimodal input information of visitors is continuously collected through cameras, microphones and other interactive devices, and the output information of the virtual interactive character is recorded, and interaction results are generated based on the multimodal input information and multimodal output information.

[0081] See also Figure 6 A sixth embodiment of the interaction method based on multimodal communication in the embodiment of the present invention includes:

[0082] 601. When the visitor is in a preset navigation scene, the virtual interactive role is determined to be a virtual navigator, and the virtual navigator is assigned to the visitor;

[0083] In this embodiment, the preset navigation scene can be defined in a variety of ways. When the visitor has just entered the interactive space and has not yet determined the specific area to go to, the multimodal large model obtains the visitor's location near the entrance area based on the positioning sensor or camera, and by analyzing the visitor's voice information or behavior, it can be determined that the visitor is in the navigation scene. In addition, when the visitor goes from one exhibition area to another, the system continuously monitors his position changes and whether there are new navigation needs, and it can also be determined that he is in the navigation scene; once it is determined that the visitor is in the navigation scene, the multimodal large model determines the virtual interactive role as a virtual navigator and assigns it to the visitor to carry out interactive activities; the virtual navigator is a virtual interactive role designed specifically to assist visitors in finding the target location in the interactive space.

[0084] 602. When the visitor is in a preset interactive scene, the virtual interactive character is determined to be a virtual character, and the virtual character is assigned to the visitor;

[0085] In this embodiment, the preset interactive scenario covers the situation where visitors stay in the exhibition area and have an in-depth understanding of the display content and the need to discuss it. When the multimodal large model detects that the visitor has stopped in front of a certain exhibition area for more than a certain period of time, or the visitor actively asks questions about the exhibition area through voice, or the visitor operates the interactive equipment in the exhibition area, it can be determined that the visitor is in an interactive scenario; in this scenario, the system determines that the virtual interactive character is a virtual character, and assigns it to the visitor to carry out interactive activities; the virtual character is a virtual interactive character customized according to different exhibition themes and exhibits types. The virtual character has professional knowledge in the corresponding field and can conduct in-depth and targeted knowledge exchange and interaction with visitors.

[0086] 603. When the visitor is in a preset exhibit display scene, the virtual interactive role is determined to be an exhibit agent, and the exhibit agent is assigned to the visitor;

[0087] In this embodiment, when a visitor enters a specific exhibit display area, or the visitor shows a tendency to understand the exhibits, it can be determined that the visitor is in an exhibit display scene; in this scene, the system determines the virtual interactive character as an exhibit agent and assigns it to the visitor to carry out interactive activities; the exhibit agent is a virtual image or information display carrier closely related to a specific exhibit, and its main function is to display and recommend specific exhibits in an all-round way.

[0088] See also Figure 7, a seventh embodiment of the interaction method based on multimodal communication in the embodiments of the present invention includes:

[0089] 701. Optimize and analyze the interaction results using a multimodal large model to obtain analysis results;

[0090] In this embodiment, the interaction results are optimized and analyzed by a multimodal large model to obtain analysis results, which include multiple aspects, such as the accuracy of understanding the visitor's intention, the visitor's satisfaction evaluation, the visitor's potential needs and interests, etc.

[0091] Specifically, by analyzing the interaction between visitors and virtual characters, we can determine whether the virtual interactive characters correctly interpret the needs and intentions of visitors, such as whether they accurately answer visitors' questions, provide navigation instructions or exhibit recommendations that meet visitors' expectations, etc. By analyzing visitors' facial emotions and voice characteristics, we can infer visitors' satisfaction evaluations; by analyzing visitors' stay time and interaction enthusiasm in different interaction scenarios, we can evaluate visitors' potential needs and interests. The performance of the current virtual interactive characters and the effectiveness of the interaction strategy can be evaluated based on the analysis results, and the analysis results can also be used as an important basis for optimizing interaction strategies and updating the behavior patterns of virtual interactive characters.

[0092] 702. When the analysis result satisfies the preset optimization condition, an optimization interaction strategy is generated according to the interaction result;

[0093] In this embodiment, the preset optimization conditions include understanding degree optimization conditions, interest preference change optimization conditions, satisfaction optimization conditions and abnormality optimization conditions; the multimodal large model triggers and judges the preset optimization conditions according to the analysis results, and when these preset optimization conditions are met, the generation process of the optimization interaction strategy is triggered;

[0094] Specifically, when the analysis results show that visitors ask a large number of questions about key knowledge and have a high error rate in answering during the interaction with the virtual interactive characters, it indicates that visitors have a low level of understanding of the relevant knowledge, and the conditions for optimizing the level of understanding are met; at this time, the multimodal large model updates the knowledge content that needs to be interacted with based on the analysis results, or changes other interaction methods, thereby generating an optimized interaction strategy;

[0095] Specifically, when the analysis results show that visitors' stay time in a certain exhibit area suddenly increases significantly, the frequency of interaction with exhibits in a certain area increases significantly, or they actively change their tour routes, the optimization conditions for interest preference changes are met; at this time, the multimodal large model generates an optimized interaction strategy based on the analysis results to adapt to the visitors' interest preferences;

[0096] Specifically, when the analysis results show that there are more negative emotional words in the visitor's voice, or the visitor frowns, shakes his head and other expressions of dissatisfaction last for a long time during the interaction, the satisfaction optimization condition is met; at this time, the multimodal large model optimizes the response speed and interaction method of the virtual interactive role according to the analysis results, and generates an optimized interaction strategy;

[0097] Specifically, when the analysis results show that there is communication interruption or poor communication continuity during the interaction between the virtual interactive character and the visitor, or the visitor makes continuous operational errors, it is judged as an abnormal optimization condition; if there is a communication failure, the multimodal large model needs to repair the independent communication link according to the communication vulnerability feedback from the analysis results; if the interactive equipment fails and causes the visitor to make an operational error, it is necessary to adjust the interaction strategy in time, such as guiding visitors to visit other normal exhibits, and notifying maintenance personnel to repair the faulty equipment as soon as possible.

[0098] 703. Controlling the virtual interactive character to carry out interactive activities according to the optimized interactive strategy;

[0099] In this embodiment, once the optimized interaction strategy is generated, it is immediately applied to the current interaction activities with visitors. By controlling the behavioral parameters of the virtual interactive characters, the characters are made to interact according to the optimized strategy. At the same time, the interaction effect after applying the optimized strategy is continuously monitored, and new interaction result data is collected to further evaluate and adjust the optimized strategy, forming an interaction cycle of continuous iterative optimization, and continuously improving the interaction quality and visitor experience.

[0100] The above describes the interaction method based on multimodal communication in the embodiment of the present invention. The following describes the interaction device based on multimodal communication in the embodiment of the present invention. Figure 8 , an embodiment of an interactive device based on multimodal communication in an embodiment of the present invention includes:

[0101] The acquisition module 801 is used to analyze the behavior of visitors to obtain multimodal input information;

[0102] The identification module 802 is used to input the multimodal input information into the multimodal large model, so that the multimodal large model can independently extract the behavioral characteristics of each visitor to obtain the visitor's identity information and visitor's intention information;

[0103] Establishing module 803, for establishing an independent communication link between the multimodal large model and the visitor according to the visitor's identity information;

[0104] A generating module 804, for generating a virtual interactive character according to the visitor's intention information;

[0105] The interaction module 805 is used to carry out interaction activities according to the independent communication link and the virtual interaction role to generate an interaction result.

[0106] In this embodiment, the acquisition module 801 performs behavior analysis on visitors to obtain multimodal input information, and then uses the identification module 802 to input the multimodal input information into the multimodal large model to extract the visitor's identity information and intention information; then uses the establishment module 803 to build an independent communication link between the multimodal large model and the visitor according to the visitor's identity information, and uses the generation module 804 to generate a virtual interactive role according to the visitor's intention information. Then the interaction module 805 carries out interactive activities with the independent communication link and the virtual interactive role, thereby obtaining an interactive result. The interactive device based on multimodal communication provided by the present invention can accurately distinguish the operation behavior and question content of each visitor, and give accurate feedback and answers in time through the virtual interactive role, avoiding information transmission confusion and interaction conflict, ensuring the fluency and efficiency of the interactive experience, and effectively solving the problem that the traditional interactive system is difficult to handle a large number of concurrent interactive requests when multiple people visit.

[0107] above Figure 8 The interactive device based on multimodal communication in the embodiment of the present invention is described in detail from the perspective of modular functional entities. The interactive device based on multimodal communication in the embodiment of the present invention is described in detail from the perspective of hardware processing.

[0108] Fig. 9 1 is a schematic diagram of the structure of an interactive device based on multimodal communication provided by an embodiment of the present invention. The interactive device 900 based on multimodal communication may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) 910 (for example, one or more processors) and a memory 920, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 933 or data 932. Among them, the memory 920 and the storage medium 930 may be temporary storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the interactive method device 900 based on multimodal communication. Furthermore, the processor 910 may be configured to communicate with the storage medium 930, and execute a series of instruction operations in the storage medium 930 on the interactive method device 900 based on multimodal communication to implement the steps of the interactive method based on multimodal communication provided in the above-mentioned method embodiments.

[0109] The interactive device 900 based on multimodal communication may also include one or more power supplies 940, one or more wired or wireless network interfaces 950, one or more input and output interfaces 960, and / or one or more operating systems 831, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, etc. Those skilled in the art will appreciate that Fig. 9 The illustrated structure of the interactive device based on multimodal communication does not constitute a limitation on the interactive device based on multimodal communication, and may include more or less components than illustrated, or a combination of certain components, or a different arrangement of components.

[0110] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium. Instructions are stored in the computer-readable storage medium. When the instructions are executed on a computer, the computer executes the steps of an interaction method based on multimodal communication.

[0111] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device, or unit can refer to the corresponding process in the aforementioned method embodiment and will not be repeated here.

[0112] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program codes.

[0113] Finally, it should be noted that the above description is only a preferred example of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An interactive method based on multimodal communication, characterized in that: include: Conduct behavioral analysis on visitors to obtain multimodal input information; Input the multimodal input information into the multimodal large model, so that the multimodal large model can independently extract the behavioral characteristics of each visitor to obtain the visitor's identity information and visitor's intention information; Establishing an independent communication link between the multimodal large model and the visitor based on the visitor's identity information; Generate virtual interactive characters based on visitor intention information; Interaction activities are carried out according to independent communication links and virtual interaction roles to generate interaction results.

2. The interactive method based on multimodal communication according to claim 1, characterized in that: The obtaining of multimodal input information from visitors includes: The camera captures the visitor's gestures, expressions and body movements to generate visual information; the microphone collects the visitor's voice signals to generate sound information; The spatial position of the visitor is calculated by positioning sensors to generate position information; Visual information, sound information and position information are integrated to obtain multimodal input information.

3. The interactive method based on multimodal communication according to claim 1, characterized in that: The multimodal input information is input into the multimodal large model, so that the multimodal large model independently extracts the behavioral characteristics of each visitor to obtain the visitor's identity information and visitor intention information, including: Extracting the visitor's voiceprint features and facial features from the multimodal input information, so that the multimodal large model can confirm the visitor's identity based on the voiceprint features and facial features to obtain the visitor's identity information; The voice features and position features of visitors are extracted from the multimodal input information, so that the multimodal large model can infer the visitor's intention based on the voice features and position features to obtain the visitor's intention information.

4. The interactive method based on multimodal communication according to claim 1, characterized in that: The generating of a virtual interactive role according to the visitor's intention information includes: Generate navigation planning information, interaction planning information and exhibit recommendation information based on visitor intention information; Generate a virtual navigator according to navigation planning information; Generate a virtual character based on the interaction planning information; Generate an exhibit agent based on the exhibit recommendation information; The virtual navigator, virtual character and exhibit agent are assigned to multiple preset interaction links to complete the generation process of the virtual interaction character.

5. The interactive method based on multimodal communication according to claim 1, characterized in that: The performing of interactive activities according to the independent communication links and the virtual interactive roles to generate interactive results includes: Different virtual interactive roles are assigned to visitors according to the scene they are in; Collect visitors' behavior data in real time and transmit the behavior data to the multimodal large model through an independent communication link to generate virtual interactive role guidance data; The virtual interactive character is controlled to carry out interactive activities according to the virtual interactive character guidance data, and real-time multimodal input information is obtained from visitors to generate interactive results.

6. The interactive method based on multimodal communication according to claim 5, characterized in that: The method of assigning different virtual interactive roles to visitors according to the scene in which the visitors are located includes: When the visitor is in a preset navigation scene, the virtual interactive role is determined to be a virtual navigator, and the virtual navigator is assigned to the visitor; When the visitor is in a preset interactive scene, the virtual interactive character is determined to be a virtual character, and the virtual character is assigned to the visitor; When the visitor is in a preset exhibit display scene, the virtual interactive role is determined to be an exhibit agent, and the exhibit agent is assigned to the visitor.

7. The interactive method based on multimodal communication according to claim 1, characterized in that: After the interactive activity is carried out according to the independent communication link and the virtual interactive role to generate the interactive result, the method further includes: Use a multimodal large model to optimize and analyze the interaction results to obtain analysis results; When the analysis result satisfies the preset optimization conditions, an optimization interaction strategy is generated according to the interaction result; The virtual interactive characters are controlled to carry out interactive activities according to the optimized interaction strategy.

8. An interactive device based on multimodal communication, characterized in that: include: The acquisition module is used to analyze the visitor's behavior to obtain multimodal input information; The recognition module is used to input the multimodal input information into the multimodal large model, so that the multimodal large model can independently extract the behavioral characteristics of each visitor to obtain the visitor's identity information and visitor's intention information; A building module for building an independent communication link between the multimodal large model and the visitor based on the visitor's identity information; A generation module, used to generate a virtual interactive character according to visitor intention information; The interaction module is used to carry out interaction activities according to the independent communication links and the virtual interaction roles to generate interaction results.

9. An interactive device based on multimodal communication, characterized in that: The interactive device based on multimodal communication includes: a memory and at least one processor, wherein instructions are stored in the memory; At least one of the processors calls the instructions in the memory to enable the multimodal communication-based interaction device to perform each step of the multimodal communication-based interaction method as described in any one of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, characterized in that: When the instructions are executed by the processor, the steps of the interaction method based on multimodal communication as described in any one of claims 1 to 7 are implemented.

Images