Sweeping robot, voice interaction method and device thereof and storage medium

By obtaining user voice and sweeping robot status image information, and generating multimodal semantic representations, intelligent voice interaction between sweeping robots and users is realized, the problem of single traditional interaction methods is solved, and interaction efficiency and experience are improved.

CN118924197APending Publication Date: 2024-11-12MIDEA ROBOZONE TECH CO LTD
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411036950.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-30
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The traditional sweeping robot has a single interaction method, making it difficult for users to ask about the robot's status or cleaning plan through natural language, and the robot cannot understand the user's intentions, resulting in insufficient intelligence and naturalness of human-computer dialogue and poor user experience.

Method used

By obtaining user voice information, combining image information and status information around the sweeping robot, it is converted into voice text and image description information, multimodal semantic representation is generated, reply information is determined, and intelligent voice interaction between the sweeping robot and the user is realized.

Benefits of technology

It improves the intelligence and nature of the interaction between the sweeping robot and the user, simplifies user operations, and improves interaction efficiency and experience.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118924197A_ABST
    Figure CN118924197A_ABST
Patent Text Reader

Abstract

The invention discloses a sweeping robot and a voice interaction method and device thereof, and a storage medium, and the method comprises the steps: obtaining the voice information of a user, the image information of the surrounding of the sweeping robot, and the state information of the sweeping robot; converting the voice information into voice text information, and converting the image information into image description information; generating a multi-modal semantic representation according to the voice text information, the image description information and the state information of the sweeping robot; determining reply information according to the multi-modal semantic representation; and controlling the sweeping robot to output reply information. Therefore, intelligence and naturalness of voice interaction between the sweeping robot and the user can be realized, operation of the user is simplified, and experience and efficiency between the user and the sweeping robot are improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of sweeping robots, and in particular to a sweeping robot voice interaction method, a computer-readable storage medium, a sweeping robot voice interaction device and a sweeping robot. Background Art

[0002] With the rapid development of artificial intelligence and Internet of Things technologies, smart home devices have been widely used. Among them, sweeping robots, as a typical household service robot, have been favored by consumers. Traditional sweeping robots are controlled by buttons, remote controls, and mobile applications. Users need to actively operate the robot or set cleaning tasks. The interaction method is relatively simple and the operation is cumbersome.

[0003] The voice control schemes in related technologies are mainly limited to some predefined simple voice commands, such as start, pause, return to the charging station, etc. The robot cannot truly understand the user's intentions, and it is difficult to conduct flexible conversations. Users cannot use natural language to ask the robot about its working status, cleaning plan, etc. The robot cannot interpret or respond to user questions based on the current visual information and task status. The lack of understanding of the environment and the integration of contextual information greatly reduces the intelligence and naturalness of human-computer conversations, which in turn leads to a poor user experience. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, the first purpose of the present invention is to propose a voice interaction method for a sweeping robot, which can realize the intelligence and naturalness of the voice interaction between the sweeping robot and the user, simplify the user's operation, and improve the experience and efficiency between the user and the sweeping robot.

[0005] A second object of the present invention is to provide a computer-readable storage medium.

[0006] The third objective of the present invention is to provide a voice interaction device for a sweeping robot.

[0007] The fourth objective of the present invention is to provide a sweeping robot.

[0008] In order to achieve the above-mentioned objectives, the first aspect of the present invention proposes a voice interaction method for a sweeping robot, characterized in that the method includes: obtaining voice information of the user; obtaining image information around the sweeping robot and status information of the sweeping robot based on the voice information; converting the voice information into voice text information, and converting the image information into image description information; generating a multimodal semantic representation based on the voice text information, the image description information and the status information of the sweeping robot; determining reply information based on the multimodal semantic representation; and controlling the sweeping robot to output the reply information.

[0009] According to the robot vacuum cleaner interaction method of the embodiment of the present invention, the user's voice information is first obtained, and the image information around the robot vacuum cleaner and the status information of the robot vacuum cleaner are obtained based on the voice information, and then the voice information is converted into voice text information, and the image information is converted into image description information, and then a multimodal semantic representation is generated based on the voice text information, the image description information and the status information of the robot vacuum cleaner, so as to determine the reply information based on the multimodal semantic representation, so as to control the robot vacuum cleaner to output the reply information. In this way, the intelligence and naturalness of the voice interaction between the robot vacuum cleaner and the user can be achieved, the user's operation is simplified, and the experience and efficiency between the user and the robot vacuum cleaner are improved.

[0010] In addition, the voice interaction method of the sweeping robot according to the above embodiment of the present invention may also include the following additional technical features:

[0011] According to one embodiment of the present invention, a multimodal semantic representation is generated based on the voice text information, the image description information and the state information of the sweeping robot, including: converting the voice text information, the image description information and the state information into a text feature vector, an image feature vector and a state feature vector, respectively; using a self-attention mechanism to respectively construct a text modal feature vector, an image modal feature vector and a state modal feature vector corresponding to the state feature vectors of the text feature vector, the image feature vector and the state feature vector; and fusing the text modal feature vector, the image modal feature vector and the state modal feature vector to generate the multimodal semantic representation.

[0012] According to one embodiment of the present invention, the text modal feature vector, the image modal feature vector and the state modal feature vector are fused, including: using a cross-modal self-attention mechanism to splice the text modal feature vector, the image modal feature vector and the state modal feature vector into a multi-modal feature vector; performing dot product processing and normalization processing on feature vectors of different modalities to obtain cross-modal attention weights; and performing weighted summation of the text modal feature vector, the image modal feature vector and the state modal feature vector according to the cross-modal attention weights to complete the fusion processing.

[0013] According to one embodiment of the present invention, the voice interaction method of a sweeping robot also includes: acquiring scene information of the sweeping robot according to the voice information; and generating a multimodal semantic representation according to the scene information, the voice text information, the image description information and the state information of the sweeping robot.

[0014] According to one embodiment of the present invention, the voice interaction method of the sweeping robot further includes: generating a task plan according to the multimodal semantic representation and the scene information, so that the multimodal semantic representation is combined with the task plan to determine the reply information.

[0015] According to an embodiment of the present invention, the reply information includes voice reply information and text reply information, and the method further includes: performing security filtering processing on the voice reply information and the text reply information.

[0016] According to one embodiment of the present invention, the voice reply information and the text reply information are subjected to security filtering processing, including: obtaining black and white list information; performing preliminary filtering on the voice reply information and the text reply information according to the black and white list information to obtain a first filtering result; performing semantic analysis on the first filtering result to obtain an analysis result; and performing deep filtering on the first filtering result according to the compatibility of the analysis result and the voice and text information to complete the filtering processing.

[0017] In order to achieve the above-mentioned purpose, the second aspect of the embodiment of the present invention proposes a computer-readable storage medium, on which a sweeping robot voice interaction program is stored. When the sweeping robot voice interaction program is executed by a processor, the sweeping robot voice interaction method of the aforementioned embodiment of the present invention is implemented.

[0018] According to the computer-readable storage medium of an embodiment of the present invention, the processor executes the sweeping robot voice interaction program, which can realize the intelligence and naturalness of the voice interaction between the sweeping robot and the user, simplify the user's operation, and improve the experience and efficiency between the user and the sweeping robot.

[0019] In order to achieve the above-mentioned purpose, the third aspect of the embodiment of the present invention proposes a voice interaction device for a sweeping robot, wherein the device includes: a first acquisition module, used to acquire the user's voice information; a second acquisition module, used to acquire the image information around the sweeping robot and the status information of the sweeping robot according to the voice information; a conversion module, used to convert the voice information into voice text information, and convert the image information into image description information; a generation module, used to generate a multimodal semantic representation according to the voice text information, the image description information and the status information of the sweeping robot; a determination module, used to determine the reply information according to the multimodal semantic representation; and an output module, used to control the sweeping robot to output the reply information.

[0020] According to the sweeping robot voice interaction device of the embodiment of the present invention, the user's voice information is first acquired through the first acquisition module, and the image information around the sweeping robot and the state information of the sweeping robot are acquired through the second acquisition module according to the voice information, and then the voice information is converted into voice text information through the conversion module, and the image information is converted into image description information, and then the generation module generates a multimodal semantic representation according to the voice text information, the image description information and the state information of the sweeping robot, so that the reply information is determined according to the multimodal semantic representation through the determination module, and the sweeping robot is controlled to output the reply information through the output module. In this way, the intelligence and naturalness of the voice interaction between the sweeping robot and the user can be realized, the user's operation is simplified, and the experience and efficiency between the user and the sweeping robot are improved.

[0021] In order to achieve the above-mentioned purpose, a fourth aspect of the present invention proposes a sweeping robot, including the sweeping robot voice interaction device of the aforementioned embodiment of the present invention.

[0022] According to the sweeping robot voice interaction device of the embodiment of the present invention, by adopting the sweeping robot voice interaction device of the above-mentioned embodiment of the present invention, the intelligence and naturalness of the voice interaction between the sweeping robot and the user can be realized, the user's operation is simplified, and the experience and efficiency between the user and the sweeping robot are improved.

[0023] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a schematic flow chart of a voice interaction method for a sweeping robot according to an embodiment of the present invention;

[0025] Figure 2 is a flow chart of a voice interaction method of a sweeping robot according to another embodiment of the present invention;

[0026] Figure 3 is a flow chart of a voice interaction method of a sweeping robot according to another embodiment of the present invention;

[0027] Figure 4 is a flow chart of a voice interaction method of a sweeping robot according to another embodiment of the present invention;

[0028] Figure 5 is a flow chart of a voice interaction method of a sweeping robot according to another embodiment of the present invention;

[0029] Figure 6 is a block diagram of a voice interaction device for a sweeping robot according to an embodiment of the present invention;

[0030] Figure 7 is a block diagram of a sweeping robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.

[0032] The following describes a cleaning robot voice interaction method, a computer-readable storage medium, a cleaning robot voice interaction device, and a cleaning robot according to embodiments of the present invention with reference to the accompanying drawings.

[0033] Figure 1 It is a flowchart of a voice interaction method of a sweeping robot according to an embodiment of the present invention.

[0034] Specifically, in some embodiments of the present invention, the voice interaction method of the sweeping robot includes:

[0035] S101, obtaining user's voice information.

[0036] Specifically, in this embodiment, a sound sensor can be installed in the sweeping robot, and then the sweeping robot can obtain the user's voice information through the sound sensor, wherein the voice information includes voice content, intonation, speaking speed, volume, pronunciation characteristics and pronunciation rhythm.

[0037] S102, acquiring image information around the cleaning robot and status information of the cleaning robot according to the voice information.

[0038] Specifically, in this embodiment, a sound sensor can be installed in the sweeping robot, and then the sweeping robot can obtain the user's voice information through the sound sensor, and then obtain the image information around the sweeping robot and the state information of the sweeping robot according to the voice information. For example, when the voice information is "clean the bedroom", after the sweeping robot receives the voice information of "clean the bedroom", it obtains the image information around the sweeping robot and the state information of the sweeping robot according to the voice information, so as to go to the bedroom and complete the task of cleaning the bedroom.

[0039] At least one camera sensor may be installed on the sweeping robot, and image information around the sweeping robot may be acquired through the camera sensor. The present invention does not specifically limit the number of installed camera sensors.

[0040] The status information of the sweeping robot includes power information, connection status information, positioning information, cleaning status information, cleaning record information and fault information, etc., and these status information can be obtained in real time after the sweeping robot is turned on. In addition, the present invention may not specifically limit the types of information contained in the status information.

[0041] S103, converting the voice information into voice text information, and converting the image information into image description information.

[0042] Specifically, in this embodiment, a pre-trained Transformer-Transducer speech recognition model can be used to convert speech information into speech text information, and a computer vision algorithm and a pre-trained visual language model can be used to analyze and describe the image information. The key information in the image is extracted by algorithms such as target detection and semantic segmentation, and then the image description information is generated using visual language models such as CLIP and BLIP. In addition, the present invention may not specifically limit the way of converting speech information into speech text information and converting image information into image description information. For example, ASR (Automatic Speech Recognition) technology can be used to convert speech information into text information. Among them, the visual language model uses contrastive learning and self-supervised learning methods to pre-train on large-scale image and text data, and fine-tunes it for the sweeping robot scene to generate accurate and fluent image description text. In addition, the selection and optimization details of the image processing algorithm, the network structure and training method of the visual language model, the construction method of the image and text representation space, etc. can be protected.

[0043] S104, generating a multimodal semantic representation according to the voice text information, the image description information and the state information of the sweeping robot.

[0044] Specifically, in this embodiment, multimodal large-scale language models such as ViLBERT and LXMERT can be used to perform feature extraction and semantic analysis on information such as speech text information, image description information, and state information. Through the self-attention mechanism and cross-modal attention mechanism, the association between different modal information is established, and then a multimodal semantic representation is generated.

[0045] Among them, voice text information contains the instructions, questions or descriptions input by the user through voice. Voice text information can directly reflect the user's intentions and needs, and is an important basis for the sweeping robot to understand and execute tasks.

[0046] The image description information contains the environmental information around the sweeping robot, for example, there are orange peels next to the sofa, there is hair under the seat, etc. Through the image description information, the sweeping robot can identify obstacles, furniture, ground types, etc., so as to better plan the cleaning path and perform cleaning tasks.

[0047] The status information includes real-time data such as the power, position, and working mode of the sweeping robot. These data reflect the current operating status and capabilities of the robot. Combined with the status information, the sweeping robot can adjust its work plan according to its actual situation, such as returning to the charging station to charge when the power is low, or adjusting the cleaning strategy when encountering a difficult-to-clean area. In addition, the present invention may not specifically limit the types of information included in the voice text information, image description information, and status information.

[0048] Multimodal semantic representation is a unified representation that integrates voice, image, and robot state information. Through this multimodal semantic representation, the sweeping robot can more comprehensively understand the user's intentions and the state of the environment, thereby making more accurate decisions and performing more efficient cleaning tasks.

[0049] It should be noted that the multimodal large language model uses self-attention mechanism and cross-modal attention mechanism to establish the association between different modal information and generate reply text that conforms to the current conversation context. At the same time, it also includes the alignment and fusion strategy of multimodal data, the fine-tuning and domain adaptation method of large language models, the optimization of decoding generation strategy, etc. In this way, the network architecture of the multimodal language model, the design details of the attention mechanism, the construction method of cross-modal representation, the training and fine-tuning process, etc. can be protected.

[0050] S105, determining reply information according to the multimodal semantic representation.

[0051] Specifically, in this embodiment, after generating a multimodal semantic representation based on the speech text information, image description information and the status information of the sweeping robot, the multimodal semantic representation is processed by a large language model to generate text reply information, and fine-tuning and knowledge enhancement techniques for the sweeping robot field are adopted to make the generated text reply information conform to the domain knowledge and dialogue context.

[0052] It should be noted that in this embodiment, the pre-trained large language model can be fine-tuned and knowledge enhanced. First, a high-quality dialogue dataset in the field of sweeping robots is constructed, covering a variety of typical dialogue scenarios and tasks. Then, the general language model is fine-tuned using the dialogue dataset to adapt it to the language style and knowledge in the field of sweeping robots. At the same time, domain knowledge graphs and rules are introduced, and explicit knowledge is injected into the language model through technologies such as knowledge embedding and attention fusion to improve the performance of the model in professional dialogue tasks. In addition, the construction method of the domain dataset, the representation method of the knowledge graph and rules, and the technical details of integrating knowledge into the language model can be protected.

[0053] S106, controlling the sweeping robot to output reply information.

[0054] Specifically, in this embodiment, after the reply information is determined according to the multimodal semantic representation, the sweeping robot is controlled to output the reply information, and the output method can be a voice reply or a text display. The reply information includes a simple status update, a detailed cleaning report, or a confirmation of the user's instructions, such as "cleaning completed", "the living room floor cleanliness reaches 90%", "having received your instructions, we are heading to the designated area for cleaning". In addition, the present invention may not specifically limit the content and output method of the output reply information.

[0055] It should be noted that, in this embodiment, the text reply information can be converted into voice reply information using TTS (Text To Speech) technology.

[0056] Further, in some embodiments of the present invention, Figure 2 As shown in FIG. 1 , a multimodal semantic representation is generated based on the speech text information, image description information and the state information of the sweeping robot, including:

[0057] S201, converting speech text information, image description information and state information into text feature vectors, image feature vectors and state feature vectors respectively.

[0058] Specifically, in this embodiment, the speech text information can be converted into a text feature vector by word embedding technology at the embedding layer of the network architecture, the image description information can be extracted and converted into an image feature vector by a convolutional neural network, and the state information of the sweeping robot can be encoded and converted into a state feature vector. In addition, the present invention may not specifically limit the method of converting speech text information, image description information and state information into a text feature vector, an image feature vector and a state feature vector, respectively.

[0059] S202, using a self-attention mechanism to respectively construct a text modal feature vector, an image modal feature vector and a state modal feature vector corresponding to the state feature vectors of the text feature vector, the image feature vector and the state feature vector.

[0060] Specifically, in this embodiment, a lightweight self-attention mechanism can be used to encode the text feature vector, the image feature vector, and the state feature vector of the state feature vector to generate a text modality feature vector, an image modality feature vector, and a state modality feature vector. For example, the speech text information is encoded by a lightweight model to generate a text feature vector, the image description information is feature extracted by a lightweight model to generate an image feature vector, and the state information of the sweeping robot is encoded to generate a state feature vector.

[0061] S203, fusing the text modality feature vector, the image modality feature vector, and the state modality feature vector to generate a multimodal semantic representation.

[0062] Specifically, in this embodiment, the linear attention mechanism can be used to fuse the text modality feature vector, the image modality feature vector, and the state modality feature vector to generate a multimodal semantic representation. For example, the linear attention mechanism improves computational efficiency by reducing complexity, performs linear transformation on the text modality feature vector, the image modality feature vector, and the state modality feature vector to generate a query vector, a key vector, and a value vector, calculates the dot product of the query and the key, and normalizes it with the Softmax function to obtain the attention weight. In the calculation process, the amount of calculation is reduced by linear transformation, and the value vector is weighted and summed using the linear attention weight to generate a multimodal semantic representation. The formula is as follows:

[0063]

[0064] Among them, Q represents the query vector, K represents the key vector, V represents the value vector, and d k Indicates the length of the word vector.

[0065] Further, in some embodiments of the present invention, Figure 3As shown, the text modality feature vector, the image modality feature vector and the state modality feature vector are fused, including:

[0066] S301, using a cross-modal self-attention mechanism to concatenate the text modality feature vector, the image modality feature vector and the state modality feature vector into a multi-modal feature vector.

[0067] Specifically, in this embodiment, a cross-modal self-attention mechanism can be used to concatenate the extracted text modal feature vector, image modal feature vector, and state modal feature vector to generate a multimodal feature vector, thereby achieving the optimized combination and complementary enhancement of speech text information, image description information, and state information.

[0068] S302, performing dot product processing and normalization processing on feature vectors of different modalities to obtain cross-modal attention weights.

[0069] Specifically, in this embodiment, a linear attention mechanism can be used to perform dot product processing and normalization processing on the feature vectors of different modalities to obtain cross-modal attention weights. For example, the feature vectors of different modalities include a query vector, a key vector, and a value vector. The query vector, the key vector, and the value vector are mapped to a new feature space so that the dot product is calculated in the space. The dot product of the query vector and the key vector is calculated in the new feature space. The dot product of the feature mapping of the query and the key is calculated, and the cross-modal attention weight is obtained after normalization processing.

[0070] S303, performing weighted summation on the text modality feature vector, the image modality feature vector and the state modality feature vector according to the cross-modality attention weight to complete the fusion process.

[0071] Specifically, in this embodiment, a linear attention mechanism can be used to perform weighted summation on the text modality feature vector, the image modality feature vector, and the state modality feature vector according to the cross-modality attention weight to complete the fusion process. The specific formula is as follows:

[0072]

[0073] Among them, Q represents the query vector, K represents the key vector, and V represents the value vector. Represents the feature mapping function, which is used to map the original vector to the new feature space. This can greatly improve the inference speed to complete the fusion process.

[0074] Further, in some embodiments of the present invention, Figure 4 As shown, the method also includes:

[0075] S401, obtaining scene information of the sweeping robot according to the voice information.

[0076] Specifically, in this embodiment, after obtaining the user's voice information, the scene information of the sweeping robot can be obtained according to the voice information. For example, when the voice information is "go to the kitchen", after the sweeping robot receives the voice information of "go to the kitchen", the scene information of the sweeping robot can be obtained according to the voice information. Computer vision and deep learning technology can also be used to analyze and identify the scene in which the sweeping robot is currently located. Through algorithms such as object detection and scene classification, it can be determined whether the robot is in different indoor scenes such as the living room, bedroom, kitchen, etc., and information in the scene can be extracted, such as spatial layout information, ground type information, and obstacle information. In addition, the present invention may not specifically limit the method of obtaining the scene information of the sweeping robot.

[0077] S402, generating a multimodal semantic representation according to the scene information, the voice text information, the image description information and the state information of the sweeping robot.

[0078] Specifically, in this embodiment, ViLBERT, LXMERT and other multimodal large language models can be used to extract features and perform semantic analysis on scene information, speech text information, image description information and state information, and establish associations between different modal information through self-attention mechanisms and cross-modal attention mechanisms, thereby generating multimodal semantic representations. Among them, scene information may include spatial layout information, lighting condition information, ground type information and obstacle information, etc. The environmental perception capability of the sweeping robot can be enhanced to improve the accuracy of task execution. In addition, the present invention may not specifically limit the content of the scene information.

[0079] Furthermore, in some embodiments of the present invention, the voice interaction method of the sweeping robot also includes: generating a task plan based on the multimodal semantic representation and scene information, so that the multimodal semantic representation is combined with the task plan to determine the reply information.

[0080] Specifically, in this embodiment, a task planning module can be set up. The task planning module generates a task plan based on multimodal semantic representation and scene information, and also uses algorithms such as action planning and path planning to generate a series of subtasks and their execution order. These subtasks may include cleaning specific areas, avoiding obstacles, adjusting cleaning modes, etc. After generating these task plans, the sweeping robot will determine the reply information based on the multimodal semantic representation and task planning, wherein the reply information is not only a response to the user's instructions, but also a notification of the current task execution status. For example, when encountering an obstacle, the robot will send a notification of "encountering an obstacle, adjusting the path", and inform the user of the estimated resolution time. When the robot starts to perform the cleaning task, it will send a voice notification of "cleaning has started" to the user. This will improve the user's interactive experience and task execution efficiency.

[0081] Furthermore, in some embodiments of the present invention, the reply information includes voice reply information and text reply information, and the method further includes: performing security filtering on the text reply information.

[0082] Furthermore, in some embodiments of the present invention, the voice reply information and the text reply information are subjected to security filtering, including:

[0083] S501, obtaining blacklist and whitelist information.

[0084] Specifically, in this embodiment, the blacklist and whitelist information includes blacklist information and whitelist information. The blacklist information can be generated within the organization based on historical data, user feedback, or the assessment of the security team. The blacklist information can be obtained through the organization, government agency, or security company. The whitelist information can be obtained based on business needs, user authorization, and default settings. In addition, the present invention may not specifically limit the method of obtaining the blacklist and whitelist information.

[0085] S502: Preliminary filtering of the voice reply information and the text reply information is performed according to the blacklist and whitelist information to obtain a first filtering result.

[0086] Specifically, in this embodiment, after obtaining the black and white list information, the voice reply information and the text reply information can be preliminarily filtered according to the black and white list information to obtain the first filtering result, thereby ensuring the security and accuracy of the information. In addition, on the basis of preliminarily filtering the voice reply information and the text reply information according to the black and white list information, multimodal features can be extracted from the voice, image and state data of the first filtering result through feature extraction, and the extracted multimodal features can be matched using a rule matching algorithm to filter out inputs that do not meet the rules to obtain the first filtering result.

[0087] S503: Perform semantic analysis on the first filtering result to obtain an analysis result.

[0088] Specifically, in this embodiment, a text classification model can be used to perform semantic analysis on the first filtering result to obtain an analysis result. For example, when the voice text information is "Please navigate to the kitchen and wipe the floor", the image description information is that multiple room entrances are detected in the current scene, the sweeping robot recognizes the location of the kitchen entrance, and the state information of the sweeping robot is that the robot is currently located in the living room and is facing the kitchen entrance. The first filtering result can be "Navigate to the kitchen" and "Wipe the floor". The text classification model is used to perform semantic analysis on the first filtering result to obtain an analysis result, and then the instruction type is determined to be "navigation" and "cleaning". In addition, the present invention may not specifically limit the manner in which the semantic analysis of the first filtering result is performed.

[0089] S504: Perform deep filtering on the first filtering result according to the analysis result and the adaptability of the voice text information to complete the filtering process.

[0090] Specifically, in this embodiment, the adaptability of the speech-text information can be obtained through content relevance, context understanding, and emotion and intent matching, and then the first filtering result is deeply filtered according to the analysis result and the adaptability of the speech-text information to complete the filtering process, thereby ensuring the accuracy and effectiveness of the information.

[0091] It should be noted that a security filtering module can also be set up to perform security filtering on text reply information. First, a predefined rule library is used for initial selection to filter out obvious illegal content. Then, a pre-trained text classification model is used for more detailed security judgment. Finally, a manual review channel can be set up to handle difficult and high-risk content, thereby ensuring the security and compliance of the conversation content, creating a safe and reliable voice interaction environment for users, and thus increasing users' trust in the sweeping robot.

[0092] You can also set up a speech synthesis module, using a pre-trained end-to-end ASR (Automatic Speech Recognition) model to convert text reply information into natural and fluent voice reply information. In addition, you can also use the speech synthesis module to set parameters such as the speaking speed, volume, and sound style of the voice reply information to enhance the user experience.

[0093] Furthermore, in some embodiments of the present invention, before converting the voice information into voice text information, the method also includes: preprocessing the voice information, wherein the preprocessing includes operations such as voice endpoint detection and / or noise suppression, thereby improving the accuracy and stability of the acquired voice information.

[0094] In addition, before converting the image information into image description information, the image information may be preprocessed, including operations such as adjusting the image size, color correction, and denoising, thereby ensuring the quality of the acquired image information.

[0095] Furthermore, in some embodiments of the present invention, the cleaning robot includes a display device for displaying reply information in different forms.

[0096] Specifically, in this embodiment, the display device can be used to display the reply information in a variety of different forms, such as text, image or video. In addition, the user can set different display modes, font sizes, colors and other parameters according to their needs to enhance the user experience. The present invention does not specifically limit the form of displaying the reply information.

[0097] Furthermore, in some embodiments of the present invention, a global control and scheduling module is set up to coordinate the execution process of each functional module, and a microservice architecture and standard interface design are adopted to divide functions such as speech recognition, image understanding, semantic understanding, and dialogue generation into independent modules. Each module communicates and exchanges data through a RESTful API. Through a loosely coupled design, each module can be independently developed, deployed, and upgraded, which greatly improves the flexibility and maintainability of the system. At the same time, new functional modules can also be easily connected to the system, giving the sweeping robot more capabilities. At the same time, the global control and scheduling module is also responsible for handling abnormal situations, such as request timeouts, service crashes, etc., thereby ensuring the stability and robustness of the system.

[0098] In summary, by setting up a speech recognition module, an image understanding module, a multimodal semantic understanding module, a dialogue generation module, a security filtering module, a speech synthesis module, a global control and scheduling module, and a human-computer interaction module, first, the user sweeps the robot through the voice environment, triggering the speech recognition module to convert the user's voice information into speech text information, while the image understanding module analyzes the image information around the sweeping robot, and converts the image information into image description information, and then the multimodal semantic understanding module generates a multimodal semantic representation according to the speech text information, the image description information and the state information of the sweeping robot, the dialogue generation module determines the text reply information according to the multimodal semantic representation and combined with the domain knowledge, the security filtering module performs multi-level filtering on the text reply information to ensure that the text reply information is safe and compliant, and then the speech synthesis module converts the filtered text reply information into voice reply information and outputs it to the user, among which the global control and scheduling module is used to coordinate the work between the modules and handle abnormal situations to ensure the stable operation of the system, and the human-computer interaction module is used to process the user's voice input and interface operation, as well as the system's voice and visual output, such as controlling the display device to display the reply information in different forms.

[0099] In addition, a scene understanding module and a task planning module can also be set. First, the user uses the voice environment to sweep the robot, triggering the voice recognition module to convert the user's voice information into voice text information. At the same time, the scene understanding module obtains the scene information of the sweeping robot. The image understanding module analyzes the image information around the sweeping robot and converts the image information into image description information. Then the multimodal semantic understanding module generates a multimodal semantic representation based on the scene information, voice text information, image description information and the state information of the sweeping robot. The task planning module generates a task plan based on the multimodal semantic representation and scene information. The dialogue generation module determines the text reply information based on the multimodal semantic representation and task plan, and converts the task plan into a natural language description. The security filtering module performs multi-level filtering on the text reply information to ensure that the text reply information is safe and compliant. Then the speech synthesis module converts the text reply information into voice reply information and outputs it to the user. Among them, the global control and scheduling module is used to coordinate the work between the modules and handle abnormal situations to ensure the stable operation of the system. The human-computer interaction module is used to process the user's voice input and interface operation, as well as the system's voice and visual output, such as controlling the display device to display the reply information in different forms.

[0100] In summary, according to the robot vacuum cleaner interaction method of the embodiment of the present invention, the user's voice information, the image information around the robot vacuum cleaner and the status information of the robot vacuum cleaner are first obtained, and then the voice information is converted into voice text information, and the image information is converted into image description information, and then a multimodal semantic representation is generated according to the voice text information, the image description information and the status information of the robot vacuum cleaner, so as to determine the reply information according to the multimodal semantic representation, so as to control the robot vacuum cleaner to output the reply information. In this way, the intelligence and naturalness of the voice interaction between the robot vacuum cleaner and the user can be realized, the user's operation is simplified, and the experience and efficiency between the user and the robot vacuum cleaner are improved.

[0101] Based on the sweeping robot voice interaction method proposed in the aforementioned embodiment of the present invention, the embodiment of the present invention further proposes a computer-readable storage medium, on which a sweeping robot voice interaction program is stored. When the sweeping robot voice interaction program is executed by a processor, the sweeping robot voice interaction method of the aforementioned embodiment of the present invention is implemented.

[0102] According to the computer-readable storage medium of an embodiment of the present invention, the processor executes the sweeping robot voice interaction program, which can realize the intelligence and naturalness of the voice interaction between the sweeping robot and the user, simplify the user's operation, and improve the experience and efficiency between the user and the sweeping robot.

[0103] Figure 6 4 is a block diagram of a voice interaction device for a sweeping robot according to an embodiment of the present invention.

[0104] Specifically, Figure 6 As shown, the voice interaction device 100 of a cleaning robot includes a first acquisition module 10 , a second acquisition module 20 , a conversion module 30 , a generation module 40 , a determination module 50 and an output module 60 .

[0105] Among them, the first acquisition module 10 is used to obtain the user's voice information; the second acquisition module 20 is used to obtain the image information around the sweeping robot and the status information of the sweeping robot according to the voice information; the conversion module 30 is used to convert the voice information into voice text information, and convert the image information into image description information; the generation module 40 is used to generate a multimodal semantic representation based on the voice text information, the image description information and the status information of the sweeping robot; the determination module 50 is used to determine the reply information according to the multimodal semantic representation; the output module 60 is used to control the sweeping robot to output the reply information.

[0106] In some embodiments of the present invention, scene information of the sweeping robot is acquired based on voice information; and a multimodal semantic representation is generated based on the scene information, voice text information, image description information, and state information of the sweeping robot.

[0107] In some embodiments of the present invention, a task plan is generated based on the multimodal semantic representation and the scene information, so that the multimodal semantic representation is combined with the task plan to determine the reply information.

[0108] In some embodiments of the present invention, the text reply message is converted into a voice reply message.

[0109] In some embodiments of the present invention, security filtering is performed on the text reply information.

[0110] In some embodiments of the present invention, the speech information is preprocessed, wherein the preprocessing includes speech endpoint detection and / or noise suppression.

[0111] It should be noted that for other specific implementations of the sweeping robot voice interaction device proposed in the embodiment of the present invention, reference can be made to the specific implementations of the sweeping robot voice interaction method in the aforementioned embodiment of the present invention, which will not be described again here to reduce redundancy.

[0112] In summary, according to the sweeping robot voice interaction device of the embodiment of the present invention, the user's voice information is first acquired through the first acquisition module, and the image information around the sweeping robot and the status information of the sweeping robot are acquired through the second acquisition module according to the voice information, and then the voice information is converted into voice text information through the conversion module, and the image information is converted into image description information, and then the generation module generates a multimodal semantic representation according to the voice text information, the image description information and the status information of the sweeping robot, so that the reply information is determined according to the multimodal semantic representation through the determination module, and the sweeping robot is controlled to output the reply information through the output module. In this way, the intelligence and naturalness of the voice interaction between the sweeping robot and the user can be realized, the user's operation is simplified, and the experience and efficiency between the user and the sweeping robot are improved.

[0113] Figure 7 is a block diagram of a sweeping robot according to an embodiment of the present invention.

[0114] like Figure 7 As shown, the cleaning robot 1000 includes the cleaning robot voice interaction device 100 according to the above-mentioned embodiment of the present invention.

[0115] According to the sweeping robot voice interaction device of the embodiment of the present invention, by adopting the sweeping robot voice interaction device of the above-mentioned embodiment of the present invention, the intelligence and naturalness of the voice interaction between the sweeping robot and the user can be realized, the user's operation is simplified, and the experience and efficiency between the user and the sweeping robot are improved.

[0116] In addition, other structures and functions of the sweeping robot according to the embodiment of the present invention are known to those skilled in the art and will not be described in detail here to reduce redundancy.

[0117] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.

[0118] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0119] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0120] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.

[0121] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.

[0122] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0123] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.

[0124] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A voice interaction method for a sweeping robot, characterized in that: The method comprises: Get the user's voice information; Acquire image information around the cleaning robot and status information of the cleaning robot according to the voice information; Converting the voice information into voice text information, and converting the image information into image description information; Generate a multimodal semantic representation according to the voice text information, the image description information and the state information of the sweeping robot; determining reply information according to the multimodal semantic representation; The cleaning robot is controlled to output the reply information.

2. The robot vacuum cleaner voice interaction method according to claim 1, characterized in that: Generating a multimodal semantic representation according to the voice text information, the image description information and the state information of the sweeping robot includes: Converting the speech text information, the image description information and the state information into a text feature vector, an image feature vector and a state feature vector respectively; A self-attention mechanism is used to construct a text modality feature vector, an image modality feature vector and a state modality feature vector corresponding to the text feature vector, the image feature vector and the state feature vector, respectively; The text modality feature vector, the image modality feature vector and the state modality feature vector are fused to generate the multimodal semantic representation.

3. The robot vacuum cleaner voice interaction method according to claim 2, characterized in that: The text modality feature vector, the image modality feature vector and the state modality feature vector are fused, comprising: Using a cross-modal self-attention mechanism to concatenate the text modality feature vector, the image modality feature vector, and the state modality feature vector into a multi-modal feature vector; Perform dot product processing and normalization on feature vectors of different modalities to obtain cross-modal attention weights; The text modality feature vector, the image modality feature vector and the state modality feature vector are weightedly summed according to the cross-modal attention weight to complete the fusion process.

4. The robot vacuum cleaner voice interaction method according to claim 1, characterized in that: The method further comprises: Acquire scene information of the sweeping robot according to the voice information; A multimodal semantic representation is generated according to the scene information, the voice text information, the image description information and the state information of the sweeping robot.

5. The robot vacuum cleaner voice interaction method according to claim 4, characterized in that: The method further comprises: A task plan is generated according to the multimodal semantic representation and the scene information, so that the multimodal semantic representation is combined with the task plan to determine the reply information.

6. The voice interaction method of a sweeping robot according to any one of claims 1 to 5, characterized in that: The reply information includes voice reply information and text reply information, and the method further includes: The voice reply information and the text reply information are subjected to security filtering processing.

7. The robot vacuum cleaner voice interaction method according to claim 6, characterized in that: The voice reply information and the text reply information are subjected to security filtering, including: Get blacklist and whitelist information; Preliminarily filtering the voice reply information and the text reply information according to the blacklist and whitelist information to obtain a first filtering result; Performing semantic analysis on the first filtering result to obtain an analysis result; The first filtering result is deeply filtered according to the adaptability of the analysis result and the voice text information to complete the filtering process.

8. A computer-readable storage medium, characterized in that: A sweeping robot voice interaction program is stored thereon, and when the sweeping robot voice interaction program is executed by the processor, the sweeping robot voice interaction method according to any one of claims 1-7 is implemented.

9. A voice interaction device for a sweeping robot, characterized in that: The device comprises: The first acquisition module is used to acquire the user's voice information; A second acquisition module, used for acquiring image information around the cleaning robot and status information of the cleaning robot according to the voice information; A conversion module, used to convert the voice information into voice text information, and convert the image information into image description information; A generating module, used for generating a multimodal semantic representation according to the voice text information, the image description information and the state information of the sweeping robot; A determination module, configured to determine reply information according to the multimodal semantic representation; The output module is used to control the cleaning robot to output the reply information.

10. A sweeping robot, characterized in that: Including the sweeping robot voice interaction device as described in claim 9.

Citation Information

Cited By

  • Voice processing method and device

    CN120895042A

  • Humanoid robot decision interaction method, system and equipment and medium

    CN120974368A