A somatic intelligent multi-modal interaction method, device, equipment and storage medium

Abstract

This application discloses an embodied intelligent multimodal interaction method, apparatus, device, and storage medium. The method includes: detecting at least one user in a target scene using a multimodal sensor and continuously sensing and tracking the user to obtain sensing and tracking information; determining the current speaker from the users based on the sensing and tracking information; performing memory hard isolation and context switching when the current speaker is inconsistent with the previous interacting user; and generating and outputting corresponding multimodal response content and listening posture based on the information of the current speaker. This application can solve the problems of chaotic multi-person interaction, memory crosstalk, asynchronous response, and poor privacy protection in related technologies, and achieve a more stable, secure, and human-like multimodal human-computer interaction.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to an embodied intelligent multimodal interaction method, device, electronic device, and readable storage medium. Background Technology

[0002] With the development of embodied intelligence and service robot technology, robots are widely used in scenarios such as home companionship, business reception, and public services. Existing embodied robots typically have functions such as voice interaction, visual recognition, and simple action output, but they still have significant shortcomings in real-world scenarios where multiple people coexist: for example, robots cannot continuously perceive and track users in a scene, making it difficult to reliably distinguish between different users; they cannot accurately identify the current speaker when multiple people are present, easily resulting in misresponses, interruptions, and irrelevant answers; when switching users, they cannot effectively isolate the conversation context and interaction memory, easily leading to crosstalk, audio crosstalk, and user privacy leaks; and the interaction response and physical posture cannot be accurately bound to the current speaker, resulting in unnatural interactions and low levels of anthropomorphism.

[0003] Therefore, achieving accurate speaker recognition, secure isolation of interactive memory, and synchronized output of response and posture in multi-user scenarios has become the key to improving the safety, fluency, and human-like experience of embodied robot interaction. Summary of the Invention

[0004] The purpose of this application is to provide an embodied intelligent multimodal interaction method, device, equipment and storage medium to solve the problems of chaotic multi-person interaction, memory crosstalk, asynchronous response and poor privacy protection in related technologies, and to achieve more stable, secure and human-like multimodal human-computer interaction.

[0005] To achieve the above objectives: In a first aspect, embodiments of this application provide an embodied intelligent multimodal interaction method, including: At least one user in the target scene is detected by a multimodal sensor, and the user is continuously sensed and tracked to obtain sensed tracking information; the current speaker is determined from the users based on the sensed tracking information; when the current speaker is different from the previous interactive user, memory hard isolation and context switching are performed; according to the information of the current speaker, the corresponding multimodal response content and listening posture are generated and output.

[0006] In one embodiment, the step of detecting at least one user in the target scene through a multimodal sensor and continuously sensing and tracking the user to obtain sensing and tracking information includes: simultaneously acquiring at least one of the user's facial features, voiceprint features, spatial orientation, lip movement state, and speaking state as sensing and tracking information through a camera and microphone array.

[0007] In one embodiment, determining the current speaker from the users based on perception tracking information includes: determining the direction of the voice through sound source localization; matching the corresponding user identity through voiceprint recognition; and verifying whether the time of the voice and the lip movement timestamp are synchronized through lip movement detection.

[0008] In one embodiment, the method further includes: when multiple users initiate voice calls simultaneously, determining the user with the highest priority as the current speaker based on a preset priority.

[0009] In one embodiment, when the current speaker is different from the previous interactive user, performing memory hard isolation and context switching includes: unloading the session record of the previous interactive user and resetting the current session state; loading the user profile, historical interaction record and independent memory data of the current speaker.

[0010] In one embodiment, generating and outputting corresponding multimodal response content and listening posture based on the current speaker's information includes: driving the robot's head and torso to turn in real time according to the current speaker's location; generating a voice response with emotion, speech rate, and tone based on the current speaker's independent memory; and simultaneously outputting matching body movements and facial expressions.

[0011] In one embodiment, after generating and outputting the corresponding multimodal response content and listening posture, the method further includes: storing the current dialogue content, emotional state, and action records into the current speaker's independent memory data.

[0012] Secondly, the present invention also provides a multimodal interaction device for an embodied intelligent robot, comprising: The multimodal perception module is used to detect at least one user in the target scene through a multimodal sensor, and to continuously perceive and track the user to obtain perception and tracking information; The speaker determination module is used to determine the current speaker from the users based on the perception tracking information; the memory isolation module is used to perform hard memory isolation and context switching when the current speaker is different from the previous interactive user. The interactive output module is used to generate and output corresponding multimodal response content and listening posture based on the information of the current speaker.

[0013] Thirdly, the present invention also provides an electronic device, including a memory and a processor; the memory stores executable program code, which, when executed by the processor, implements the steps of the embodied intelligent multimodal interaction method as described in any of the above embodiments.

[0014] Fourthly, the present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the embodied intelligent multimodal interaction method as described in any of the above embodiments.

[0015] The embodied intelligent multimodal interaction method, apparatus, device, and storage medium provided in this application include: detecting at least one user in a target scene using a multimodal sensor and continuously sensing and tracking the user to obtain sensing and tracking information; determining the current speaker from the users based on the sensing and tracking information; performing memory hard isolation and context switching when the current speaker is different from the previous interacting user; and generating and outputting corresponding multimodal response content and listening posture based on the information of the current speaker. This application, through continuous multimodal sensing and tracking of users, can stably identify the current speaker in scenarios with multiple users present; through memory hard isolation and context reset during speaker switching, it achieves complete isolation of interaction data between different users, preventing crosstalk and privacy leaks; and through synchronous output of listening posture and multimodal response oriented towards the speaker, it makes the interaction more natural and more human-like, thereby improving the overall interaction fluency, security, and user experience. Attached Figure Description

[0016] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A flowchart illustrating the embodied intelligent multimodal interaction method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the structure of the embodied intelligent multimodal interaction device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0020] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0021] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0022] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0023] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0024] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0025] First Embodiment See Figure 1 This application provides an embodied intelligent multimodal interaction method, which can be executed by an embodied intelligent multimodal interaction device provided in this application. This device can be implemented using software and / or hardware. In this embodiment, the embodied intelligent multimodal interaction device is taken as the executing entity of the method. The embodied intelligent multimodal interaction method provided in this embodiment includes the following steps: Step S101: Detect at least one user in the target scene using a multimodal sensor, and continuously perceive and track the user to obtain perception and tracking information.

[0026] In one embodiment, at least one user in the target scene is detected by a multimodal sensor, and the user is continuously perceived and tracked to obtain perception and tracking information, including: acquiring at least one of the user's facial features, voiceprint features, spatial orientation, lip movement state and speaking state as perception and tracking information through a camera and microphone array.

[0027] Specifically, facial features of the user are acquired through a camera and facial recognition; voiceprint features and spatial orientation are collected through a microphone array; and speaking state is determined through visual lip movement detection. The above information is then fused to form perceptual tracking information.

[0028] Step S102: Determine the current speaker from the users based on the perception tracking information.

[0029] In one embodiment, determining the current speaker from among users based on perception tracking information includes: determining the direction of the voice through sound source localization; matching the corresponding user identity through voiceprint recognition; and verifying whether the time of the voice and the lip movement timestamp are synchronized through lip movement detection.

[0030] It is understandable that when voice activity is detected, a triple verification logic of source localization, voiceprint recognition, and lip movement detection is initiated simultaneously: that is, determining the location of the voice; extracting voice features and matching them with users continuously tracked in the background; and using visual algorithms to verify whether the corresponding user's lip movements and voice timestamps are synchronized at that location. Combining the above three pieces of information, the system determines the unique current speaker in real time and updates the global focus.

[0031] In one embodiment, the method further includes: when multiple users initiate voice calls simultaneously, determining the user with the highest priority as the current speaker based on a preset priority.

[0032] Step S103: When the current speaker is different from the previous interactive user, perform memory hard isolation and context switching.

[0033] In one implementation, when the current speaker is different from the previous interactive user, memory hard isolation and context switching are performed, including: unloading the session record of the previous interactive user and resetting the current session state; loading the user profile, historical interaction record and independent memory data of the current speaker.

[0034] Understandably, the system continuously compares the current speaker with the speaker from the previous interaction. If they are determined to be the same user, the current active session remains unchanged. If they are determined to be different users, the system immediately performs a hard switch, completely unloading and persistently storing the entire session context of the previous user (including temporary dialogue records, emotional states, unfinished intentions, etc.) from memory to ensure that no information of the previous user remains in the current process. Only the independent memory data of the current speaker (including their complete user profile and historical interaction records) is loaded to initialize the current session context and ensure that the interaction of the new user is not affected.

[0035] Furthermore, based on the current speaker's voiceprint and facial features, a match is made in the user database: if the current speaker has a history, their complete long-term profile, historical dialogue content, emotional change trajectory, and preference settings are loaded; if they are a new user, a unique user identifier and independent storage space are automatically created, and profile data is initialized.

[0036] Step S104: Based on the information of the current speaker, generate and output the corresponding multimodal response content and listening posture.

[0037] In one embodiment, based on the information of the current speaker, a corresponding multimodal response content and listening posture are generated and output, including: driving the robot's head and torso to turn in real time according to the current speaker's location; generating a voice response with emotion, speech rate, and tone based on the current speaker's independent memory; and simultaneously outputting matching body movements and facial expressions.

[0038] Understandably, the system executes synchronized physical and digital outputs based on the current speaker's location and interaction state. After determining the sound source's location and visual tracking position, it drives the robot's neck and waist joints in real time, smoothly turning its head and torso towards the current speaker; ensuring the robot's body orientation and gaze focus are bound to the current user, simulating a focused listening posture. If there is no user interacting, it returns to standby patrol mode; based on the emotion tags output by the artificial intelligence model, it generates synchronized facial expressions and gestures, plays the speech through the speaker, and displays the content on the screen. In one embodiment, after generating and outputting the corresponding multimodal response content and listening posture, the method further includes: storing the dialogue content, emotional state, and action records into the current speaker's independent memory data.

[0039] It is understandable that after the interaction is completed, the complete record of this interaction (voice content, emotional state, gestures and actions performed, etc.) will be stored only in the current speaker's independent memory data, and the current speaker's conversation context state will be temporarily stored to ensure that the user can seamlessly resume speaking next time.

[0040] In summary, the implementation method of this application can reliably identify the current speaker in a multi-person scenario by continuously sensing and tracking the user through multimodal means; it can completely isolate the interaction data of different users, prevent cross-contamination, and protect privacy by hard isolation of memory and context reset when the speaker switches; and it can make the robot interaction more natural and more human-like by synchronously outputting listening posture and multimodal response in the face of the speaker, thereby improving the overall interaction fluency, security and user experience.

[0041] Second Embodiment Based on the first embodiment of this application, an embodied intelligent multimodal interaction device is provided in this embodiment, see reference. Figure 2The device includes: The multimodal perception module 21 is used to detect at least one user in the target scene through a multimodal sensor, and to continuously perceive and track the user to obtain perception and tracking information.

[0042] Speaker determination module 22 is used to determine the current speaker from the users based on perception tracking information.

[0043] The memory isolation module 23 is used to perform hard memory isolation and context switching when the current speaker is different from the previous interactive user.

[0044] The interactive output module 24 is used to generate and output corresponding multimodal response content and listening posture based on the information of the current speaker.

[0045] In one embodiment, the multimodal sensing module 21 is further configured to: By using a camera and microphone array, at least one of the user's facial features, voiceprint features, spatial orientation, lip movement state, and speaking state is simultaneously acquired as perceptual tracking information.

[0046] In one embodiment, the speaker determination module 22 is further configured to: Determining the current speaker from among users based on perception tracking information includes: determining the direction of the voice through sound source localization; matching the corresponding user identity through voiceprint recognition; and verifying whether the time of the voice and the lip movement timestamp are synchronized through lip movement detection.

[0047] In one embodiment, the speaker determination module 22 is further configured to: When multiple users initiate a voice call simultaneously, the user with the highest priority is determined as the current speaker based on a preset priority.

[0048] In one embodiment, the memory isolation module 23 is further configured to: When the current speaker is different from the previous user, perform memory hard isolation and context switching, including: unloading the previous user's session record and resetting the current session state; loading the current speaker's user profile, historical interaction records and independent memory data.

[0049] In one embodiment, the interactive output module 24 is further configured to: Based on the current speaker's location, the robot's head and torso turn in real time; based on the current speaker's independent memory, it generates a voice response with emotion, speech rate, and tone; and simultaneously outputs matching body movements and facial expressions.

[0050] It should be noted that the description of the embodied intelligent multimodal interaction device above is similar to the description of the embodied intelligent multimodal interaction method above, and the beneficial effects of the same method will not be repeated. For technical details not disclosed in the embodiments of the embodied intelligent multimodal interaction device of the present invention, please refer to the description of the embodiments of the embodied intelligent multimodal interaction method of the present invention.

[0051] Third Embodiment Based on the same inventive concept as the foregoing embodiments, this application provides an electronic device, such as... Figure 3 As shown, the device includes: a processor 301 and a memory 302 storing a computer program; wherein, Figure 3 The processor 301 shown in the diagram does not indicate that there is only one processor 301, but only indicates the positional relationship of processor 301 relative to other devices. In practical applications, there can be one or more processors 301; similarly, Figure 3 The memory 302 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 302 relative to other devices. In practical applications, there can be one or more memories 302. When the processor 301 runs the computer program, it implements the embodied intelligent multimodal interaction method described above.

[0052] The device may also include at least one network interface 303. The various components of the device are coupled together via a bus system 304. It is understood that the bus system 304 is used to implement communication between these components. In addition to a data bus, the bus system 304 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 3 The general designated all buses as Bus System 304.

[0053] The memory 302 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 302 described in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0054] Fourth embodiment Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program. The computer-readable storage medium can be a magnetic random access memory (FRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic surface memory, an optical disc, or a compact disc read-only memory (CD-ROM), etc.; it can also be various devices including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc. When the computer program stored in the computer-readable storage medium is executed by a processor, it implements the above-described charging current adjustment method. For the specific steps implemented when the computer program is executed by the processor, please refer to [link to relevant documentation]. Figure 2 The description of the illustrated embodiments will not be repeated here.

[0055] In this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions are generally described in detail only when they appear for the first time. When they appear again, they are generally not repeated for the sake of brevity. When understanding the technical solutions and other contents of this application, the same or similar terms, concepts, technical solutions and / or application scenario descriptions that are not described in detail later can be referred to their previous relevant detailed descriptions.

[0056] In this application, the descriptions of the various embodiments have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0057] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0058] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for embodied intelligent multimodal interaction, characterized in that, The method includes: At least one user in the target scene is detected by a multimodal sensor, and the user is continuously sensed and tracked to obtain sensed and tracked information; The current speaker is determined from the users based on the perception tracking information; When the current speaker is different from the previous user, perform hard memory isolation and context switching; Based on the current speaker's information, generate and output the corresponding multimodal response content and listening posture.

2. The method as described in claim 1, characterized in that, The step of detecting at least one user in the target scene using a multimodal sensor and continuously sensing and tracking the user to obtain sensing and tracking information includes: The system uses a camera and microphone array to simultaneously acquire at least one of the user's facial features, voiceprint features, spatial orientation, lip movement state, and speaking state as perceptual tracking information.

3. The method as described in claim 1, characterized in that, The step of determining the current speaker from the users based on perception tracking information includes: Determine the direction of sound emission by sound source localization; Matching the corresponding user identity through voiceprint recognition; The timing of phonation and the lip movement timestamp are synchronized through lip movement detection.

4. The method as described in claim 1, characterized in that, The method further includes: When multiple users initiate a voice call simultaneously, the user with the highest priority is determined as the current speaker based on a preset priority.

5. The method as described in claim 1, characterized in that, When the current speaker is different from the previous user, the process of performing memory hard isolation and context switching includes: Unload the session record of the previous user and reset the current session state; Load the user profile, historical interaction records, and independent memory data of the current speaker.

6. The method as described in claim 1, characterized in that, The step of generating and outputting corresponding multimodal response content and listening posture based on the current speaker's information includes: Based on the current speaker's location, drive the robot's head and torso to turn in real time; Generates voice responses with emotion, speech rate, and tone based on the current speaker's independent memory; Synchronously output matching body movements and facial expressions.

7. The method as described in claim 1, characterized in that, After generating and outputting the corresponding multimodal response content and listening posture, the method further includes: The content of this conversation, emotional state, and action records are stored in the speaker's independent memory data.

8. A embodied intelligent multimodal interaction device, characterized in that, include: The multimodal perception module is used to detect at least one user in the target scene through a multimodal sensor, and to continuously perceive and track the user to obtain perception and tracking information; The speaker determination module is used to determine the current speaker from the users based on perception tracking information; The memory isolation module is used to perform hard memory isolation and context switching when the current speaker is different from the previous interactive user; The interactive output module is used to generate and output corresponding multimodal response content and listening posture based on the information of the current speaker.

9. An electronic device, characterized in that, The electronic device includes: a memory and a processor, wherein the memory stores executable program code, and when the executable program code is executed by the processor, it implements the steps of the embodied intelligent multimodal interaction method as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that, The readable storage medium stores a computer program that, when executed by a processor, implements the steps of the embodied intelligent multimodal interaction method as described in any one of claims 1 to 7.

Images