Man-machine interaction method, device and equipment for multi-modal information identification and storage medium

Abstract

The invention discloses a human-computer interaction method, device and equipment for multi-modal information recognition and a storage medium, multi-modal recognition is realized by fusing information such as human faces, eyeballs, voices and body postures through multiple sensors, the accuracy and diversity of human-computer interaction are improved, meanwhile, the interaction intention of a user is judged through the multi-modal information, and the user experience is improved. According to the method, orientations and voice contents of different speakers are recognized, passive or active continuous dialogues with different users are realized, and the problems that in the prior art, recognition accuracy is low in a noisy environment when a wake-up word mechanism carries out man-machine voice interaction, and the recognition accuracy cannot be realized in a multi-user dialogue scene are solved. And smooth dialogues with multiple persons in the aspects of dialogue contents and dialogue modes can be realized.

Description

Technical Field

[0001] The present application relates to the field of intelligent agent technology, and in particular to a human-computer interaction method, device, equipment and storage medium for multimodal information recognition. Background Art

[0002] Nowadays, human-computer interaction technology has attracted wide attention in various fields such as daily life, work and entertainment. Among them, efficient voice interaction technology, as a natural and existing human-computer interaction method, has received widespread attention and application. The main technology is based on voice recognition technology, which realizes human-computer interaction by recognizing the user's voice commands, such as waking up the smart speaker by issuing a clear wake-up word: After the "Xiao Ai" smart speaker recognizes the wake-up word and responds to humans, it enters a voice interaction dialogue and conducts a single or multiple rounds of dialogue.

[0003] The commonly used human-computer voice interaction technology currently has a wake-up word mechanism, which requires continuous monitoring of the wake-up word and microphone array to initiate further voice interaction. However, the current wake-up mechanism can only respond to one-on-one voice content conversations. From the perspective of the intelligent agent, it cannot be understood as multiple rounds of conversations between different users, so that the intelligent agent cannot accept the content spoken by different speakers, and thus cannot achieve a smooth conversation with multiple people in terms of conversation content and conversation method in a multi-person conversation scenario.

[0004] In addition, in a noisy environment, the difficulty in recognizing or incorrect recognition of the wake-up word will cause the recognition accuracy of voice interaction to drop significantly, affecting the fluency of human-computer interaction.

[0005] Therefore, it is urgent for technicians in this field to provide a human-computer interaction method for multimodal information recognition to solve the above technical problems. Summary of the invention

[0006] The present application provides a human-computer interaction method, apparatus, device and storage medium for multimodal information recognition, which solves the technical problems in the prior art of low recognition accuracy in noisy environments for human-computer voice interaction using a wake-up word mechanism, and the inability to achieve smooth conversations with multiple people in terms of conversation content and conversation method in a multi-person conversation scenario.

[0007] In view of this, the first aspect of the present application provides a human-computer interaction method for multimodal information recognition, the method comprising:

[0008] Step S1: acquiring biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on a sensor module;

[0009] Step S2: performing multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model;

[0010] Step S3: judging and analyzing the number of users in the current environment, speaking users, watching users, intimacy of each user, and speaking content based on the multimodal fusion model;

[0011] Step S4: Determine the current interactive user, interactive voice content, and interactive action based on the judgment and analysis results;

[0012] Step S5: Execute the interactive action for the current interactive user and play the interactive voice content.

[0013] Optionally, the step S1 specifically includes:

[0014] Based on the face acquisition module, the face recognition information, face orientation information, eyeball pupil position information, gaze direction information, blinking action information, lip movement information and body posture information of at least one target object are acquired;

[0015] Acquire user voice content information, sound source localization information, volume information, timbre feature information, and audio feature information based on the microphone array module;

[0016] Get current environment information based on the environment sensor module.

[0017] Optionally, the step S3 specifically includes:

[0018] When a sound is detected, determine whether a preset wake-up word is detected;

[0019] If the preset wake-up word is not detected, determine whether the face recognition information of at least one target object is detected, and determine whether the at least one target object is looking at it based on the face orientation information, eye pupil position information and gaze direction information, blinking action information and body posture information of at least one target object;

[0020] If being watched by at least one target object, when there are two or more watching targets, judging whether each watching target object is speaking according to the lip movement information of each watching target object;

[0021] If two or more gaze targets are speaking, then according to the intimacy with each gaze target, the gaze target with the highest intimacy is turned to identify the speech content of the most intimate gaze target and generate response content and response behavior; if only one gaze target is speaking, the gaze target is turned to identify the speech content of the gaze target and generate response content and response behavior.

[0022] Optionally, step S3 further includes:

[0023] If the preset wake-up word is detected, voice recognition is performed on the acquired voice content information, and the response content is generated and played or the response behavior is executed.

[0024] Optionally, step S3 further includes:

[0025] If the target object is not looking at it, the detected sound content is identified to determine whether there is relevant information of at least one target object in the sound content. If so, the target object with the highest intimacy is turned to according to the intimacy with each target object and the active dialogue behavior mode is executed.

[0026] Optionally, the step of identifying the speech content of the gaze target and generating response content and response behavior specifically includes:

[0027] The large language model (LLM) is used to identify the speech content of the target person, and combined with the understanding of the overall conversation content, the response language content, response eye content and response body content are generated.

[0028] Optionally, it also includes:

[0029] Adaptively adjust the face acquisition module and / or microphone array module according to the current environmental information.

[0030] A second aspect of the present application provides a human-computer interaction device for multimodal information recognition, the device comprising:

[0031] An acquisition unit, used to acquire biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on the sensor module;

[0032] A multimodal fusion unit, used to perform multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model;

[0033] An analysis unit, used to judge and analyze the number of users in the current environment, speaking users, watching users, intimacy of each user, and speaking content based on a multimodal fusion model;

[0034] A processing unit, used to determine the current interactive user, interactive voice content and interactive action according to the judgment and analysis results;

[0035] The execution unit is used to execute the interaction action for the current interactive user and play the interactive voice content.

[0036] A third aspect of the present application provides a human-computer interaction device for multimodal information recognition, the device comprising a processor and a memory:

[0037] The memory is used to store program code and transmit the program code to the processor;

[0038] The processor is used to execute the steps of the human-computer interaction method for multimodal information recognition as described in the first aspect according to the instructions in the program code.

[0039] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the human-computer interaction method for multimodal information recognition described in the first aspect above.

[0040] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0041] In the present application, a human-computer interaction method, device, equipment and storage medium for multimodal information recognition are provided, which use multiple sensors to fuse information such as face, eyeballs, voice, body posture, etc. to achieve multimodal recognition, increase the accuracy and diversity of human-computer interaction, and at the same time judge the user's interaction intention through multimodal information, identify the position and voice content of different speakers, and achieve passive or active continuous dialogue with different users. The technical problems of low recognition accuracy in noisy environments for human-computer voice interaction using a wake-up word mechanism in the prior art and the inability to achieve smooth dialogue with multiple people in terms of dialogue content and dialogue method in a multi-person dialogue scenario are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 A method flow chart of a human-computer interaction method for multimodal information recognition in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of the structure of a human-computer interaction device for multimodal information recognition in an embodiment of the present application;

[0044] Figure 3 Schematic diagram of the structure of a human-computer interaction device for multimodal information recognition in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0046] The present application designs a human-computer interaction method, device, equipment and storage medium for multimodal information recognition, which solves the technical problems in the prior art of low recognition accuracy in noisy environments for human-computer voice interaction using a wake-up word mechanism, and the inability to achieve smooth conversations with multiple people in terms of conversation content and conversation methods in multi-person conversation scenarios.

[0047] For easier understanding, see Figure 1 , Figure 1 is a method flow chart of the human-computer interaction method for multimodal information recognition in an embodiment of the present application, such as Figure 1 As shown, specifically:

[0048] Step S1: acquiring biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on a sensor module;

[0049] Specifically:

[0050] Based on the face acquisition module, the face recognition information, face orientation information, eyeball pupil position information, gaze direction information, blinking action information, lip movement information and body posture information of at least one target object are acquired;

[0051] Acquire user voice content information, sound source localization information, volume information, timbre feature information, and audio feature information based on the microphone array module;

[0052] Get current environment information based on the environment sensor module.

[0053] It should be noted that multiple sensors are used to obtain multimodal information, including:

[0054] Face acquisition module (camera): at least one for capturing face recognition information, face orientation information, eye pupil position information, gaze direction information, blinking action information, lip movement information, and body posture information of the target;

[0055] Microphone array module: used to capture the user's voice content information, sound source location information, volume information, timbre feature information and audio feature information;

[0056] Environmental sensor module: used to detect current environmental information such as light, temperature, and noise.

[0057] Through these sensors, the device can collect various information related to human-computer interaction, providing a basis for subsequent data processing.

[0058] Step S2: performing multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model;

[0059] It should be noted that the acquired biometric information, motion recognition information, voice recognition information and current environmental information are integrated to comprehensively judge whether there is a person, whether someone is speaking, and whether the person is speaking to the device, so as to provide more accurate status judgment of the speaker and listener similar to the interaction between people.

[0060] Step S3: judging and analyzing the number of users in the current environment, speaking users, watching users, intimacy of each user, and speaking content based on the multimodal fusion model;

[0061] Specifically:

[0062] When a sound is detected, determine whether a preset wake-up word is detected;

[0063] If the preset wake-up word is not detected, determine whether the face recognition information of at least one target object is detected, and determine whether the at least one target object is looking at it based on the face orientation information, eye pupil position information and gaze direction information, blinking action information and body posture information of at least one target object;

[0064] If being watched by at least one target object, when there are two or more watching targets, judging whether each watching target object is speaking according to the lip movement information of each watching target object;

[0065] If two or more gaze targets are speaking, then according to the intimacy with each gaze target, the gaze target with the highest intimacy is turned to identify the speech content of the most intimate gaze target and generate response content and response behavior; if only one gaze target is speaking, the gaze target is turned to identify the speech content of the gaze target and generate response content and response behavior.

[0066] Also includes:

[0067] If the preset wake-up word is detected, voice recognition is performed on the acquired voice content information, and the response content is generated and played or the response behavior is executed.

[0068] Also includes:

[0069] If the target object is not looking at it, the detected sound content is identified to determine whether there is relevant information of at least one target object in the sound content. If so, the target object with the highest intimacy is turned to according to the intimacy with each target object and the active dialogue behavior mode is executed.

[0070] It should be noted that the following states in the surrounding environment are judged through the multimodal fusion model:

[0071] Is there anyone around: If no sound is picked up, the device uses visual face detection and millimeter-wave radar-based micro-motion human body detection to determine whether there is anyone around the device and whether the user is a familiar user. If the user is a familiar user, the device will proactively initiate a topic based on the visually recognized user's expression and status;

[0072] Is someone looking at the device: Looking at the device means that someone is talking to the device, and the device should be awakened and enter the voice pickup mode.

[0073] Is someone talking? If someone is detected talking, voice data will be collected and uploaded to LLM for recognition, understanding and feedback.

[0074] Is it a conversation between multiple people? If it is detected that there are multiple people talking, the voice and lip movement detection will be used to determine which speaker is the more familiar one, and the phone will turn to the more familiar speaker, recognize and understand the speaker's speech content through the large model, and respond at the appropriate time.

[0075] Furthermore, the step of identifying the speech content of the gaze target and generating response content and response behavior specifically includes:

[0076] The large language model (LLM) is used to identify the speech content of the target person, and combined with the understanding of the overall conversation content, the response language content, response eye content and response body content are generated.

[0077] It should be noted that the voice data, image data picked up by the microphone array, and other data describing the current state of the device acquired by the sensor are textualized and input into the large language model LLM. The large language model LLM is used to judge the current user's state, understand the content of single-person or multi-person conversations, generate corresponding feedback information based on the contextual content, and insert conversations when appropriate to complete interaction with the user.

[0078] The response content of interactive feedback may include:

[0079] Sound module: plays the text returned by LLM through the speaker through the speech synthesis system, or directly plays the audio data returned by LLM;

[0080] Eyes description module: When the device is speaking, it understands what it is saying and combines it with the context of the user's speech to generate different types of eyes, such as surprise, curiosity, happiness, sadness, and indifference, to enrich the interactive eye expressions.

[0081] Body movement module: After the microphone identifies the direction of the speaker, if the LLM determines that the speaker is the person the device is going to respond to next, the device will automatically turn to the speaker through the body movement module.

[0082] Step S4: Determine the current interactive user, interactive voice content, and interactive action based on the judgment and analysis results;

[0083] Step S5: Execute the interactive action for the current interactive user and play the interactive voice content.

[0084] Furthermore, it also includes:

[0085] Adaptively adjust the face acquisition module and / or microphone array module according to the current environmental information.

[0086] It should be noted that environmental sensors are used to monitor the environmental conditions around the device, including light, temperature, and noise. The device can adjust sensors and technical modules based on environmental information. For example, in a dark environment, the infrared camera is activated; in a noisy environment, the sensitivity of the microphone array is adjusted. This ensures that the device can maintain stable performance under various environmental conditions.

[0087] See also Figure 2 , Figure 2 Schematic diagram of the structure of the human-computer interaction device for multimodal information recognition in the embodiment of the present application, such as Figure 2 As shown, specifically:

[0088] An acquisition unit 201 is used to acquire biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on a sensor module;

[0089] The multimodal fusion unit 202 is used to perform multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model;

[0090] The analysis unit 203 is used to judge and analyze the number of users in the current environment, the speaking users, the watching users, the intimacy of each user, and the speaking content based on the multimodal fusion model;

[0091] The processing unit 204 is used to determine the current interactive user, interactive voice content and interactive action according to the judgment and analysis results;

[0092] The execution unit 205 is used to execute the interaction action for the current interactive user and play the interactive voice content.

[0093] The present application also provides another multi-modal information recognition human-computer interaction device, such as Figure 3For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a pet robot, a virtual person, a mobile phone, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a mobile phone as an example:

[0094] Figure 3 FIG. 1 is a block diagram showing a partial structure of a pet robot related to a terminal provided in an embodiment of the present application. Figure 3 The pet robot includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The pet robot structure shown in the figure does not constitute a limitation of the pet robot, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0095] Combine the following Figure 3 A detailed introduction to the various components of the pet robot:

[0096] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: LowNoiseAmplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0097] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the pet robot by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the pet robot (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0098] The input unit 1030 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the pet robot. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, a stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1080, and can receive and execute the command sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0099] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the pet robot. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the pet robot, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the pet robot.

[0100] The pet robot may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the pet robot moves to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, which can be used for applications that identify the posture of the pet robot (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that the pet robot can also be equipped with, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0101] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the pet robot. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another pet robot through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0102] WiFi is a short-range wireless transmission technology. The pet robot can help users send and receive emails, browse web pages, and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the pet robot and can be omitted as needed without changing the essence of the invention.

[0103] The processor 1080 is the control center of the pet robot. It uses various interfaces and lines to connect various parts of the entire pet robot. By running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, it performs various functions of the pet robot and processes data, thereby monitoring the pet robot as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0104] The pet robot also includes a power source 1090 (such as a battery) for supplying power to each component. Preferably, the power source can be logically connected to the processor 1080 through a power management system, thereby realizing functions such as charging, discharging, and power consumption management through the power management system.

[0105] Although not shown, the pet robot may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0106] In the embodiment of the present application, the processor 1080 included in the pet robot also has the following functions:

[0107] Step S1: acquiring biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on a sensor module;

[0108] Step S2: performing multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model;

[0109] Step S3: judging and analyzing the number of users in the current environment, speaking users, watching users, intimacy of each user, and speaking content based on the multimodal fusion model;

[0110] Step S4: Determine the current interactive user, interactive voice content, and interactive action based on the judgment and analysis results;

[0111] Step S5: Execute the interactive action for the current interactive user and play the interactive voice content.

[0112] An embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementations of the human-computer interaction method for multimodal information recognition described in the aforementioned embodiments.

[0113] In the embodiments of the present application, a human-computer interaction method, apparatus, device and storage medium for multimodal information recognition are provided, which utilize multiple sensors to fuse information such as face, eyeballs, voice, body posture, etc. to achieve multimodal recognition, increase the accuracy and diversity of human-computer interaction, and at the same time judge the user's interaction intention through multimodal information, identify the position and voice content of different speakers, and achieve passive or active continuous dialogue with different users. The technical problems of low recognition accuracy in noisy environments for human-computer voice interaction using a wake-up word mechanism in the prior art and the inability to achieve smooth dialogue with multiple people in terms of dialogue content and dialogue method in a multi-person dialogue scenario are solved.

[0114] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0115] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0116] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0117] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0118] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0120] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0121] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A human-computer interaction method for multimodal information recognition, characterized in that: The following steps are involved: Step S1: acquiring biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on a sensor module; Step S2: performing multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model; Step S3: judging and analyzing the number of users in the current environment, speaking users, watching users, intimacy of each user, and speaking content based on the multimodal fusion model; Step S4: Determine the current interactive user, interactive voice content, and interactive action based on the judgment and analysis results; Step S5: Execute the interactive action for the current interactive user and play the interactive voice content.

2. The human-computer interaction method for multimodal information recognition according to claim 1, characterized in that: The step S1 specifically includes: Based on the face acquisition module, the face recognition information, face orientation information, eyeball pupil position information, gaze direction information, blinking action information, lip movement information and body posture information of at least one target object are acquired; Acquire user voice content information, sound source location information, volume information, timbre feature information, and audio feature information based on the microphone array module; Get current environment information based on the environment sensor module.

3. The human-computer interaction method for multimodal information recognition according to claim 2, characterized in that: The step S3 specifically includes: When a sound is detected, determine whether a preset wake-up word is detected; If the preset wake-up word is not detected, determine whether the face recognition information of at least one target object is detected, and determine whether the at least one target object is looking at it based on the face orientation information, eye pupil position information and gaze direction information, blinking action information and body posture information of at least one target object; If being watched by at least one target, when there are two or more watching targets, judging whether each watching target is speaking according to the lip movement information of each watching target; If two or more gaze targets are speaking, then according to the intimacy with each gaze target, the gaze target with the highest intimacy is turned to identify the speech content of the most intimate gaze target and generate response content and response behavior; if only one gaze target is speaking, the gaze target is turned to identify the speech content of the gaze target and generate response content and response behavior.

4. The human-computer interaction method for multimodal information recognition according to claim 3, characterized in that: The step S3 further comprises: If the preset wake-up word is detected, voice recognition is performed on the acquired voice content information, and the response content is generated and played or the response behavior is executed.

5. The human-computer interaction method for multimodal information recognition according to claim 3, characterized in that: The step S3 further comprises: If the target object is not looking at it, the detected sound content is identified to determine whether there is relevant information of at least one target object in the sound content. If so, the target object with the highest intimacy is turned to according to the intimacy with each target object and the active dialogue behavior mode is executed.

6. The human-computer interaction method for multimodal information recognition according to claim 3, characterized in that: The identifying the speech content of the gaze target and generating the response content and response behavior specifically includes: The large language model (LLM) is used to identify the speech content of the target person, and combined with the understanding of the overall conversation content, the response language content, response eye content and response body content are generated.

7. The human-computer interaction method for multimodal information recognition according to claim 2, characterized in that: Also includes: Adaptively adjust the face acquisition module and / or microphone array module according to the current environmental information.

8. A human-computer interaction device for multimodal information recognition, characterized in that ,,include: An acquisition unit, used to acquire biometric information, motion recognition information and voice recognition information of at least one target object, as well as current environment information based on the sensor module; A multimodal fusion unit, used to perform multimodal fusion processing on the acquired biometric information, action recognition information, voice recognition information and current environment information to generate a multimodal fusion model; An analysis unit, used to judge and analyze the number of users in the current environment, speaking users, watching users, intimacy of each user, and speaking content based on a multimodal fusion model; A processing unit, used to determine the current interactive user, interactive voice content and interactive action according to the judgment and analysis results; The execution unit is used to execute the interaction action for the current interactive user and play the interactive voice content.

9. A human-computer interaction device for multimodal information recognition, characterized in that ,, the device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the human-computer interaction method for multimodal information recognition according to any one of claims 1-7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the human-computer interaction method for multimodal information recognition according to any one of claims 1-7.

Images