Multimodal interaction method, device, storage medium and smart screen device

Abstract

The present application discloses a multimodal interaction method, device, storage medium and smart screen device, which relate to the fields of artificial intelligence technology and multimodal interaction technology. The specific implementation scheme is: identifying user characteristics based on multimodal interaction technology, and obtaining user interaction scene information based on the multimodal interaction technology; analyzing the user's interaction intention based on the user characteristics combined with the interaction scene information; obtaining the target interaction instruction corresponding to the interaction intention, and using the target interaction instruction to interactively control the smart screen device.

Description

Technical Field

[0001] The present application relates to the field of computer technology, specifically to the field of artificial intelligence and multimodal interaction technology, and in particular to a multimodal interaction method, apparatus, storage medium and smart screen device. Background Art

[0002] Multimodal intelligent interaction is the main form of intelligent interaction in the future. With the advancement of artificial intelligence technology, the interaction between people and intelligent devices, and between people and machines tends to be more intelligent and more natural. For example, with smart speakers and smart robots, users can obtain various resources and services through interaction with electronic devices. Summary of the invention

[0003] Provided are a multimodal interaction method, apparatus, storage medium, smart screen device, and computer program product, which can determine the user's interaction intention by combining user characteristics and the user's interaction environment information, and effectively improve the multimodal intelligent interaction effect of the smart screen device.

[0004] According to the first aspect, a multimodal interaction method is provided, which is applied to a smart screen device, including: identifying user characteristics based on multimodal interaction technology, and obtaining user interaction scene information based on the multimodal interaction technology; analyzing the user's interaction intention based on the user characteristics combined with the interaction scene information; obtaining a target interaction instruction corresponding to the interaction intention, and using the target interaction instruction to interactively control the smart screen device.

[0005] The multimodal interaction method of the embodiment of the present application identifies user characteristics based on multimodal interaction technology, obtains user interaction scene information based on multimodal interaction technology, analyzes user interaction intentions based on user characteristics combined with interaction scene information, and obtains target interaction instructions corresponding to the interaction intentions, thereby using target interaction instructions to interactively control the smart screen device. It is possible to combine user characteristics and user interaction environment information to jointly determine the user's interaction intentions, and effectively improve the multimodal intelligent interaction effect of the smart screen device.

[0006] According to the second aspect, a multimodal interaction device is provided, which is applied to a smart screen device, including: an identification module, used to identify user characteristics based on multimodal interaction technology; a first acquisition module, used to obtain the user's interaction scene information based on the multimodal interaction technology; an analysis module, used to analyze the user's interaction intention based on the user characteristics in combination with the interaction scene information; a second acquisition module, used to obtain a target interaction instruction corresponding to the interaction intention, so as to use the target interaction instruction to interactively control the smart screen device.

[0007] The multimodal interaction device of the embodiment of the present application identifies user characteristics based on multimodal interaction technology, obtains user interaction scene information based on multimodal interaction technology, analyzes the user's interaction intention based on the user characteristics and the interaction scene information, and obtains the target interaction instructions corresponding to the interaction intention, so as to use the target interaction instructions to interactively control the smart screen device. It is possible to combine the user characteristics and the user's interaction environment information to jointly determine the user's interaction intention, and effectively improve the multimodal intelligent interaction effect of the smart screen device.

[0008] According to the third aspect, a smart screen device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the multimodal interaction method of an embodiment of the present application.

[0009] The smart screen device of the embodiment of the present application identifies user characteristics based on multimodal interaction technology, obtains user interaction scene information based on multimodal interaction technology, analyzes the user's interaction intention according to the user characteristics combined with the interaction scene information, and obtains the target interaction instructions corresponding to the interaction intention, so as to use the target interaction instructions to interactively control the smart screen device. It is possible to determine the user's interaction intention by combining the user characteristics and the user's interaction environment information, thereby effectively improving the multimodal intelligent interaction effect of the smart screen device.

[0010] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is proposed, wherein the computer instructions are used to enable the computer to execute the multimodal-based interaction method disclosed in the embodiment of the present application.

[0011] According to a fifth aspect, a computer program product is proposed, including a computer program, which, when executed by a processor, implements the multimodal-based interaction method disclosed in an embodiment of the present application.

[0012] The technology according to the present application solves the technical problem in the related technology that simple visual perception or voice perception is far from meeting the current needs of human-computer intelligent interaction. It can combine user characteristics and user interaction environment information to jointly determine the user's interaction intention, and effectively improve the multi-modal intelligent interaction effect of smart screen devices.

[0013] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present application, nor is it intended to limit the scope of the present application. Other features of the present application will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0015] Figure 1 is a schematic diagram according to the first embodiment of the present application;

[0016] Figure 2 is a schematic diagram according to a second embodiment of the present application;

[0017] Figure 3 Schematic diagram of the structure of a multi-modal interaction device in an embodiment of the present application;

[0018] Figure 4 is a schematic diagram according to a third embodiment of the present application;

[0019] Figure 5 is a schematic diagram according to a fourth embodiment of the present application;

[0020] Figure 6 It is a block diagram of a smart screen device used to implement the multimodal interaction method according to an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following is a description of exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0022] Figure 1 It is a schematic diagram according to the first embodiment of the present application. It should be noted that the execution subject of the multimodal interaction method of this embodiment is a multimodal interaction device, which can be implemented by software and / or hardware, and can be configured in a smart screen device, which can be understood as a hardware device with intelligent functional elements such as a human-computer interaction screen, a voice collector, a voice player, and an image acquisition group.

[0023] The embodiments of the present application relate to the fields of artificial intelligence technology and multimodal interaction technology, wherein artificial intelligence (AI) is abbreviated as AI. It is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. Multimodal interaction is a form of interaction that integrates vision, voice and pattern perception, which is proposed with the development of intelligent systems and interactive human-like hardware devices.

[0024] like Figure 1 As shown, the multimodal interaction method may include:

[0025] S101: Identify user features based on multimodal interaction technology, and obtain user interaction scene information based on the multimodal interaction technology.

[0026] Each source or form of information can be called a modality. For example, people have touch, hearing, vision, and smell; the media of information include voice, video, text, etc.; various sensors, such as radar, infrared, accelerometer, etc., each of which can be called a modality. Two different languages ​​can be considered as two modalities, and even data sets collected under two different circumstances can be considered as two modalities.

[0027] Therefore, in this application, the field of multimodal interaction technology can be applied to identify user features based on multimodal interaction technology, and obtain user interaction scene information based on multimodal interaction technology. In this way, the dimension of interaction information can be enriched as much as possible, so that the interaction between users and smart screen devices can effectively simulate the interaction mode between people, and effectively assist in the confirmation of subsequent interaction intentions.

[0028] In some embodiments, the user's facial features, body features, and gesture features can be obtained based on multimodal interaction technology, and the facial features, body features, and gesture features can be used as user features. Each of the above features can be regarded as a "modality". Of course, user features may not be limited to the above, but may also be fingerprint features, voiceprint features, etc., thereby realizing the acquisition of user features in multiple dimensions, so that the user's interaction intention with the smart screen device is not limited to voice interaction and gesture interaction, thereby ensuring the accuracy of subsequent interaction intention confirmation.

[0029] For example, face recognition technology can be used to capture facial images of human faces, and then facial features are obtained by analyzing the facial images. Gesture features and body features can be identified in a similar manner, and there is no limitation to this.

[0030] After identifying user characteristics based on multimodal interaction technology, the user's interaction scene information can also be obtained based on multimodal interaction technology, wherein the interaction scene information is used to describe the interaction environment information of the environment in which the user who is currently interacting with the smart screen device is located, or to describe the interaction information between the user and the smart screen device, and can also describe the user's control information on the environment state, etc. Of course, the interaction scene information can also be some interaction information of any other possible user in the environment, and there is no limitation to this.

[0031] Optionally, in some embodiments, see Figure 2 , Figure 2 is a schematic diagram according to the second embodiment of the present application, which obtains the user's interaction scene information based on the multimodal interaction technology, including:

[0032] S201: Acquire interactive environment information of multiple dimensions of the environment where the user is located, and use the interactive environment information of multiple dimensions as interactive scene information.

[0033] Optionally, in some embodiments, image recognition can be performed on the user's environment to obtain interactive environment information of the visual dimension of the environment based on the recognized image; and / or voice recognition can be performed on the user's environment to obtain interactive environment information of the voice dimension of the environment based on the recognized voice, thereby acquiring interactive scene information of the environmental dimension and enriching the modal dimension of the multimodal interaction logic.

[0034] As an example, when identifying the interactive environment information in the visual dimension, the visual scene currently facing the smart screen device can be continuously sent to the visual analysis module to recognize the image; and when identifying the interactive environment information in the voice dimension, the voice content currently in the smart screen device can be continuously sent to the voice analysis module to recognize the voice.

[0035] In some embodiments, when acquiring the interactive environment information of the visual dimension of the environment based on the recognized image, the recognized image can be specifically analyzed to analyze the content, time, brightness and other features of the image, so as to use these features as the recognized interactive environment information. When acquiring the interactive environment information of the voice dimension of the environment based on the recognized voice, the recognized voice can be subjected to voice analysis processing to obtain the pitch of the voice, the duration of the voice and other features, so as to use these features as the recognized interactive environment information.

[0036] In the embodiment of the present application, it is possible to specifically identify the environmental features related to the environment and the audio features in the environment described in the above-mentioned images and voices as interactive environment information. Specifically, the recognized image can be analyzed to obtain the environmental features of the environment, and the environmental features can be used as the interactive environment information in the visual dimension, and the recognized voice can be analyzed to obtain the audio features in the environment, and the audio features can be used as the interactive environment information in the voice dimension. The implementation is simple, and relatively accurate environmental features and audio features can be obtained, thereby assisting the subsequent output of interactive instructions that are compatible with the environmental features and audio features.

[0037] The above-mentioned environmental characteristics are at least one of the following: the light characteristics of the environment, the distance between the calibrated objects in the environment and the user, and the time characteristics of the environment. The above-mentioned audio characteristics are at least one of the following: the sound characteristics of the environment and the noise characteristics of the environment. Thus, the light characteristics of the environment, the distance between the calibrated objects in the environment and the user, and the time characteristics of the environment can be combined with the sound characteristics of the environment and the noise characteristics of the environment to determine the user's interaction intention, thereby assisting the subsequent output of interaction instructions that are compatible with the environmental characteristics and audio characteristics.

[0038] S202: Acquire interaction information between the user and the smart screen device, and use the interaction information as interaction scene information.

[0039] In some embodiments, the smart screen device is connected to an external input device, which external input device includes but is not limited to a keyboard, a mouse, a remote control, etc., that is, if there is an external input device connected to the smart screen device, the interaction information between the user and the smart screen device is obtained, which can be the interaction information directly input by the user via the smart screen device; and / or the interaction information input by the user to the smart screen device via the external input device is obtained, thereby determining the user's interaction intention by combining user characteristics and the interaction information between the user and the smart screen device, enriching the "modality" in multimodal interaction from the dimension of user and device, thereby assisting the subsequent output of interaction instructions that are compatible with the interaction information between the user and the smart screen device.

[0040] S203: Obtain the user's control information on the environment state, and use the control information as interaction scene information.

[0041] In some embodiments, the above-mentioned interaction scene information may also be, for example, the user's control information on the environmental state. Obtaining the user's control information on the environmental state may be obtaining the user's control information on the environmental state using a third-party device, wherein the third-party device has established a communication connection with the smart screen device. Thus, the user's interaction intention is determined by combining the user's characteristics and the user's control information on the environmental state, enriching the "modality" in the multimodal interaction from the dimension of the user's control over the environmental state, thereby assisting the subsequent output of interaction instructions that are compatible with the user's control information on the environmental state.

[0042] As an example, the third-party device mentioned above can be, for example, an air-conditioning device in the environment. When the user adjusts the temperature and humidity of the environment through the air-conditioning device, the user's adjustment information of the temperature and humidity of the environment can be used as control information, without limitation.

[0043] By acquiring the interactive environment information of multiple dimensions of the user's environment and using the interactive environment information of multiple dimensions as the interactive scene information, acquiring the interactive information between the user and the smart screen device and using the interactive scene information, as well as acquiring the user's control information on the environmental state and using the control information as the interactive scene information, it is possible to realize multimodal collection of the interactive environment information and acquire the interactive environment information of multiple dimensions. Not only can it assist in determining the interactive intention based on the user's own characteristics, but it can also determine the user's interactive intention in combination with a variety of interactive environment information, which can effectively improve the multimodal intelligent interaction effect of the smart screen device.

[0044] S102: Analyze the user's interaction intention based on user characteristics and interaction scenario information.

[0045] After the user characteristics of multiple modalities and the user's interaction scenario information are determined as described above, a preconfigured analysis algorithm can be used to analyze the user's interaction intention based on the user characteristics combined with the interaction scenario information.

[0046] For example, the feature keywords corresponding to the user features can be analyzed, and the scene keywords corresponding to the interaction scene information can be analyzed. Then, based on the feature keywords and scene keywords, the corresponding interaction intentions can be matched from a preconfigured relationship table as the user's interaction intention. The relationship table may include sample feature keywords, sample scene keywords, and sample interaction intentions corresponding to the sample feature keywords and sample scene keywords obtained in advance based on big data analysis. This allows the user's interaction intention to be analyzed directly based on the user features combined with the interaction scene information.

[0047] Of course, any other possible methods may be used to analyze the user's interaction intention based on user characteristics combined with interaction scene information, such as modeling methods, engineering methods, etc., and there is no limitation to this.

[0048] S103: Obtain a target interaction instruction corresponding to the interaction intention, and use the target interaction instruction to interactively control the smart screen device.

[0049] In view of the description of the above embodiments, the present application also provides a specific example, in which the multimodal interaction method can be applied to a multimodal interaction device, and the multimodal interaction device can be used, for example, Figure 3 As shown, Figure 3 It is a structural diagram of the multimodal interactive device in the embodiment of the present application. The multimodal interactive device includes: a multimodal cloud 301 and a multimodal terminal 302, wherein the multimodal terminal 302 can be configured in the smart screen device, and the multimodal cloud 301 can be configured on the cloud server side, that is to say, the determination of the interactive intention and the generation and processing logic of the interactive instruction are processed by the multimodal cloud 301, thereby reducing the hardware resource consumption of the smart screen device and improving the interactive response efficiency of the smart screen device.

[0050] Above Figure 3The multimodal terminal 302 may include a multimodal analysis module 3021, a multimodal interaction decision 3022, a multimodal input 3023, and a multimodal output 3024. The multimodal input 3023 captures user feature input, voice input, visual input, external device input, and screen input of the smart screen device, and sends these multimodal inputs to the multimodal analysis module 3021. The multimodal analysis module 3021 analyzes the user features and the interaction environment information. The multimodal interaction decision 3022 matches the user's interaction intention according to the analysis results of the multimodal analysis module 3021, and sends the interaction intention to the multimodal cloud 301. The multimodal cloud 301 obtains the corresponding target interaction instruction according to the user's interaction intention, and then sends the target interaction instruction to the multimodal output 3024, so as to use the target interaction instruction to interactively control the smart screen device.

[0051] Combined with the above embodiments and Figure 3 , the detailed description of this application can be as follows:

[0052] Multimodal Input 3023:

[0053] It mainly includes four input modes: visual input, voice input, peripheral input, and touch input. They are used to collect the user's current visual scene, voice scene, other peripheral input scene, and screen touch input scene respectively.

[0054] The visual input scenario inputs the visual image currently facing the device into the visual analysis module. The voice input scenario inputs the voice audio currently facing the smart screen device into the voice analysis module. The screen trigger input supports the interaction scenario between the user and the smart screen device. The peripheral input scenario is used to enhance the input efficiency of the smart screen device, including but not limited to keyboard, mouse, remote control, etc.

[0055] Multimodal Analysis Module 3021:

[0056] The multimodal analysis module 3021 is used to perform multi-dimensional analysis on the currently collected visual information, voice information, and sensor peripheral information, and multi-dimensional comprehensive assistance to complete an effective user interaction.

[0057] The visual analysis module analyzes the visual input data in multiple dimensions, including light detection, distance detection, and outputs the visual judgment conditions for the current interactive environment. It also identifies facial features, body analysis, gesture recognition, etc. based on multimodal recognition technology to form user characteristics.

[0058] The speech analysis module performs multi-dimensional analysis on the speech input data, such as environmental sound detection, noise detection, etc., and outputs the speech discrimination conditions for the current interactive environment; and identifies the user's voiceprint features based on multimodal recognition technology to form user features.

[0059] Embedded peripherals are similar to screen triggers and are used to respond to user input for specific application scenarios (fingerprint recognition, gaming scenarios, etc.) and general usage scenarios.

[0060] The multimodal interaction decision 3022 will use the interaction environment information (light brightness, interaction time, distance between user and device, ambient sound of the interaction environment, noise intensity, etc.) and user characteristics (using face recognition, gesture recognition, body recognition, voiceprint recognition, specific wake-up command recognition, etc.) generated by the multimodal analysis module to determine the interaction scenario and user identity. Based on the analysis results, the visual interaction information / voice interaction information / other forms of input information are converted into target interaction instructions, different information results are obtained from the interaction response module for the user, and a certain interaction control effect is produced on the interaction output module.

[0061] The multimodal interaction response module of the multimodal cloud 301 provides interactive services based on the target interaction instructions provided by the multimodal interaction decision module, including but not limited to life services, home control, dialogue communication, children's education, multimedia entertainment, office applications, etc., without limitation.

[0062] The multimodal output 3024 is used to interactively display the results of the multimodal response, and its display form is controlled by the multimodal decision module. The forms of interactive output include but are not limited to screen, sound, embedded peripherals, etc. Based on the decision results of the multimodal decision module (user characteristics, interactive environment information, etc.), the output screen brightness, sound volume, and the current output display form are automatically adjusted.

[0063] In this embodiment, by identifying user characteristics based on multimodal interaction technology, obtaining user interaction scene information based on multimodal interaction technology, and analyzing user interaction intentions based on user characteristics combined with interaction scene information, and obtaining target interaction instructions corresponding to the interaction intentions, the target interaction instructions are used to interactively control the smart screen device, which can realize the combination of user characteristics and user interaction environment information to jointly determine the user's interaction intentions, and effectively improve the multimodal intelligent interaction effect of the smart screen device.

[0064] Figure 4 It is a schematic diagram according to the fourth embodiment of the present application.

[0065] The multimodal-based interaction device 400 is applied to a smart screen device.

[0066] like Figure 4 As shown, the multimodal interaction device 400 includes:

[0067] Identification module 401, used to identify user characteristics based on multimodal interaction technology;

[0068] A first acquisition module 402, configured to acquire user interaction scene information based on a multimodal interaction technology;

[0069] Analysis module 403, used to analyze the user's interaction intention based on user characteristics and interaction scenario information;

[0070] The second acquisition module 404 is used to acquire the target interaction instruction corresponding to the interaction intention, so as to interactively control the smart screen device using the target interaction instruction.

[0071] In one embodiment of the present application, wherein, see Figure 5 , Figure 5 is a schematic diagram according to the fifth embodiment of the present application, the first acquisition module 402 includes:

[0072] The first acquisition submodule 4021 is used to acquire the interaction environment information of multiple dimensions of the environment where the user is located, and use the interaction environment information of multiple dimensions as the interaction scene information; and / or,

[0073] The second acquisition submodule 4022 is used to acquire interaction information between the user and the smart screen device, and use the interaction information as interaction scene information; and / or,

[0074] The third acquisition submodule 4023 is used to acquire the user's control information on the environment state and use the control information as the interaction scene information.

[0075] In one embodiment of the present application, the first acquisition submodule 4021 is specifically used to:

[0076] Performing image recognition on the environment in which the user is located to obtain interactive environment information of the visual dimension of the environment based on the recognized image; and / or,

[0077] Perform speech recognition on the user's environment to obtain interactive environment information of the speech dimension of the environment based on the recognized speech.

[0078] In one embodiment of the present application, the first acquisition submodule 4021 is further used to:

[0079] The recognized image is analyzed to obtain the environmental features of the environment, and the environmental features are used as interactive environmental information in the visual dimension.

[0080] In one embodiment of the present application, the environmental characteristic is at least one of the following: a light characteristic of the environment, a distance between a calibration object in the environment and the user, and a time characteristic of the environment.

[0081] In one embodiment of the present application, the first acquisition submodule 4021 is further used to:

[0082] The recognized speech is analyzed to obtain the audio features in the environment, and the audio features are used as the interactive environment information in the speech dimension.

[0083] In one embodiment of the present application, the audio feature is at least one of the following: a sound feature in an environment and a noise feature in an environment.

[0084] In one embodiment of the present application, the smart screen device is connected to an external input device, wherein the second acquisition submodule 4022 is specifically used to:

[0085] Obtaining interaction information directly input by the user via the smart screen device; and / or,

[0086] Acquire interaction information input by the user to the smart screen device via an external input device.

[0087] In one embodiment of the present application, the third acquisition submodule 4023 is specifically used to:

[0088] Obtain information about the user's control of the environment status using a third-party device, where the third-party device has established a communication connection with the smart screen device.

[0089] In one embodiment of the present application, the identification module 401 is specifically used to:

[0090] The user's facial features, body features, and gesture features are acquired based on multimodal interaction technology, and are used as user features.

[0091] It should be noted that the aforementioned explanation of the multimodal interaction method is also applicable to the multimodal interaction device of this embodiment, and will not be repeated here.

[0092] In this embodiment, by identifying user characteristics based on multimodal interaction technology, obtaining user interaction scene information based on multimodal interaction technology, and analyzing user interaction intentions based on user characteristics combined with interaction scene information, and obtaining target interaction instructions corresponding to the interaction intentions, the target interaction instructions are used to interactively control the smart screen device, which can realize the combination of user characteristics and user interaction environment information to jointly determine the user's interaction intentions, and effectively improve the multimodal intelligent interaction effect of the smart screen device.

[0093] According to an embodiment of the present application, according to an embodiment of the present application, the present application also provides a smart screen device, a readable storage medium and a computer program product.

[0094] like Figure 6, is a block diagram of a smart screen device based on a multimodal interaction method according to an embodiment of the present application. The smart screen device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The smart screen device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0095] like Figure 6 As shown, the smart screen device includes: one or more processors 601, a memory 602, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed in the smart screen device, including instructions stored in or on the memory to display graphical information of the GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple smart screen devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 601 is taken as an example.

[0096] The memory 602 is a non-transient computer-readable storage medium provided in the present application. The memory stores instructions executable by at least one processor to enable the at least one processor to perform the multi-modal interaction method provided in the present application. The non-transient computer-readable storage medium of the present application stores computer instructions, which are used to enable a computer to perform the multi-modal interaction method provided in the present application.

[0097] The memory 602 is a non-transient computer-readable storage medium that can be used to store non-transient software programs, non-transient computer executable programs and modules, such as program instructions / modules corresponding to the multimodal interaction method in the embodiment of the present application (for example, the attached Figure 4 The processor 601 executes various functional applications and data processing of the server by running the non-transient software programs, instructions and modules stored in the memory 602, that is, implements the multi-modal interaction method in the above method embodiment.

[0098] The memory 602 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of a smart screen device that performs a multimodal interaction method, etc. In addition, the memory 602 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 602 may optionally include a memory remotely disposed relative to the processor 601, and these remote memories may be connected to a smart screen device that performs a multimodal interaction method via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0099] The smart screen device that executes the multi-modal interaction method may further include: an input device 603 and an output device 604. The processor 601, the memory 602, the input device 603 and the output device 604 may be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0100] The input device 603 can receive input digital or character information, and generate key signal input related to user settings and function control of the smart screen device based on the multimodal interaction method, such as a touch screen, a keypad, a mouse, a trackpad, a touchpad, an indicator rod, one or more mouse buttons, a trackball, a joystick and other input devices. The output device 604 may include a display device, an auxiliary lighting device (e.g., an LED) and a tactile feedback device (e.g., a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display and a plasma display. In some embodiments, the display device may be a touch screen.

[0101] Various implementations of the systems and techniques described herein can be realized in digital electronic circuit systems, integrated circuit systems, dedicated ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0102] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for programmable processors and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or means (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0104] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0105] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0106] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this application can be executed in parallel, sequentially or in different orders, as long as the expected results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0107] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A multimodal interaction method, applied to a smart screen device, comprising: Acquire the user's current visual scene, voice scene, other peripheral input scenes, and screen touch input scene, identify user features based on multimodal interaction technology, and acquire the user's interaction scene information based on the multimodal interaction technology; Analyzing the user's interaction intention according to the user characteristics and the interaction scenario information; Acquire a target interaction instruction corresponding to the interaction intention, so as to interactively control the smart screen device using the target interaction instruction; The acquiring the user's interaction scene information based on the multimodal interaction technology includes: Acquire the interactive environment information of multiple dimensions of the environment where the user is located, and use the interactive environment information of multiple dimensions as the interactive scene information; and acquire the interactive information between the user and the smart screen device, and use the interactive information as the interactive scene information; and acquire the control information of the user on the environmental state, and use the control information as the interactive scene information; Analyzing the user's interaction intention according to the user characteristics in combination with the interaction scene information, including: analyzing the feature keywords corresponding to the user characteristics, and analyzing the scene keywords corresponding to the interaction scene information, and matching the corresponding interaction intention from a preconfigured relationship table according to the feature keywords and the scene keywords as the user's interaction intention; The acquiring of the interactive environment information of multiple dimensions of the environment where the user is located includes: Performing image recognition on the environment where the user is located to obtain interactive environment information of the visual dimension of the environment according to the recognized image; and performing voice recognition on the environment where the user is located to obtain interactive environment information of the voice dimension of the environment according to the recognized voice; The step of obtaining the interactive environment information of the visual dimension of the environment according to the recognized image includes: Analyze the recognized image to obtain environmental features of the environment, and use the environmental features as interactive environment information of the visual dimension; the environmental features are at least one of the following: light features of the environment, the distance between a calibration object in the environment and the user, and time features of the environment; The step of obtaining the interactive environment information of the voice dimension of the environment according to the recognized voice comprises: parsing the recognized voice to obtain audio features in the environment, and using the audio features as the interactive environment information of the voice dimension, wherein the audio features are at least one of the following: sound features in the environment and noise features in the environment; The identifying of user characteristics based on multimodal interaction technology includes: Based on the multimodal interaction technology, the user's facial features, body features, and gesture features are acquired, and the facial features, body features, and gesture features are used as the user features.

2. According to the multimodal interaction method of claim 1, the smart screen device is connected to an external input device, wherein: The acquiring interaction information between the user and the smart screen device includes: Acquiring interaction information directly input by the user via the smart screen device; and / or, Acquire interaction information input by the user to the smart screen device via the external input device.

3. The multimodal interaction method according to claim 1, wherein: The obtaining the control information of the user on the environment state includes: The control information of the environment state by the user using a third-party device is obtained, wherein the third-party device has established a communication connection with the smart screen device.

4. A multimodal interactive device, applied to a smart screen device, comprising: The input module is used to obtain the user's current visual scene, voice scene, other peripheral input scenes, and screen touch input scenes; Identification module, used to identify user characteristics based on multimodal interaction technology; A first acquisition module, used to acquire user interaction scene information based on the multimodal interaction technology; An analysis module, used to analyze the user's interaction intention according to the user characteristics and the interaction scenario information; A second acquisition module is used to acquire a target interaction instruction corresponding to the interaction intention, so as to interactively control the smart screen device using the target interaction instruction; The first acquisition module includes: A first acquisition submodule is used to acquire the interaction environment information of multiple dimensions of the environment where the user is located, and use the interaction environment information of multiple dimensions as the interaction scene information; A second acquisition submodule is configured to acquire interaction information between the user and the smart screen device, and use the interaction information as the interaction scene information; and A third acquisition submodule is used to acquire the control information of the user on the environment state, and use the control information as the interaction scene information; The analysis module is further used to analyze the user's interaction intention according to the user characteristics in combination with the interaction scene information, including: analyzing the feature keywords corresponding to the user characteristics, and analyzing the scene keywords corresponding to the interaction scene information, and matching the corresponding interaction intention from a preconfigured relationship table according to the feature keywords and the scene keywords as the user's interaction intention; The first acquisition submodule is specifically used for: Performing image recognition on the environment where the user is located to obtain interactive environment information of the visual dimension of the environment according to the recognized image; and performing voice recognition on the environment where the user is located to obtain interactive environment information of the voice dimension of the environment according to the recognized voice; The first acquisition submodule is further used for: Analyze the recognized image to obtain environmental features of the environment, and use the environmental features as interactive environment information of the visual dimension; the environmental features are at least one of the following: light features of the environment, the distance between a calibration object in the environment and the user, and time features of the environment; The first acquisition submodule is further used for: Parsing the recognized speech to obtain audio features in the environment, and using the audio features as interactive environment information of the speech dimension, wherein the audio features are at least one of the following: sound features in the environment and noise features in the environment; The identification module is specifically used for: Based on the multimodal interaction technology, the user's facial features, body features, and gesture features are acquired, and the facial features, body features, and gesture features are used as the user features.

5. The multimodal interaction device according to claim 4, wherein the smart screen device is connected to an external input device, wherein: The second acquisition submodule is specifically used for: Acquiring interaction information directly input by the user via the smart screen device; and / or, Acquire interaction information input by the user to the smart screen device via the external input device.

6. The multimodal interaction device according to claim 4, wherein: The third acquisition submodule is specifically used for: The control information of the environment state by the user using a third-party device is obtained, wherein the third-party device has established a communication connection with the smart screen device.

7. A smart screen device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 3.

9. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 3.

Images