Robot, interaction method and interaction system thereof, storage medium and controller
Through the multimodal deep learning model, it solves the problem of insufficient perception and interaction capabilities of traditional robots in complex environments, and realizes powerful environmental adaptation and human-computer interaction, which is suitable for intelligent robots. and the field of artificial intelligence.
Patent Information
- Application Number
- CN202410888760.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-03
- Publication Date
- 2024-11-12
AI Technical Summary
Traditional intelligent sweeping robots and indoor service robots are not performing well in dealing with complex and changing environments and emergencies, lacking universality and flexibility, and require a lot of manpower to define perception rules and label training data, making it difficult to cover all kinds of complex environments. combination situation.
Multimodal deep learning model is used to process multimodal environmental data, integrate heterogeneous data from multiple sensors, build environmental cognition through end-to-end learning, and generate structured semantic information for human-computer interaction with target push strategy .
It realizes efficient environmental perception and human-computer interaction of robots in complex environments, has the advantages of strong environmental adaptability, flexible deployment and efficient deployment, and is suitable for intelligent robots and artificial intelligence fields.
Smart Images

Figure CN118939110A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of intelligent robots, and in particular to a robot and an interaction method, an interaction system, a storage medium, and a controller thereof. Background Art
[0002] Smart sweeping robots and indoor service robots have been widely used in home and commercial environments. Traditional robots mainly rely on simple sensors and preset rules to complete environmental perception and task execution. However, as application scenarios become increasingly complex, traditional solutions are unable to cope with changing environments and emergencies. Summary of the invention
[0003] The present invention aims to solve at least one of the technical problems in the related art to a certain extent. To this end, one purpose of the present invention is to propose a robot interaction method, which has the advantages of strong environmental adaptability, flexible and efficient deployment, friendly human-computer interaction, etc., and has broad application prospects in the fields of intelligent robots and artificial intelligence.
[0004] A second object of the present invention is to provide a computer-readable storage medium.
[0005] The third object of the present invention is to provide a controller.
[0006] The fourth object of the present invention is to provide a robot.
[0007] A fifth objective of the present invention is to provide a robot interaction system.
[0008] To achieve the above-mentioned objectives, the first aspect of the present invention proposes a robot interaction method, which includes: acquiring multimodal environmental data of a target scene; inputting the multimodal environmental data into a pre-trained multimodal deep learning model to obtain an environmental perception result; performing semantic extraction on the environmental perception result to obtain structured semantic information; generating a push message according to a target push strategy and the structured semantic information, and pushing the push message to a user terminal bound to the robot.
[0009] According to the robot interaction method of the embodiment of the present invention, the multimodal environmental data is processed using a pre-trained multimodal deep learning model, the environmental perception results obtained by the processing are extracted as structured semantic information, and the structured semantic information is generated as a push message and sent to the user terminal based on the target push strategy. The embodiment of the present invention forms a set of technical solutions for robot universal perception and human-computer interaction by integrating the multimodal deep learning model and the target push strategy, which has the advantages of strong environmental adaptability, flexible and efficient deployment, and friendly human-computer interaction, and has broad application prospects in the fields of intelligent robots and artificial intelligence.
[0010] In addition, the robot interaction method proposed in the above embodiment of the present invention may also have the following additional technical features:
[0011] According to one embodiment of the present invention, the training process of the multimodal deep learning model includes: establishing a multimodal deep learning model; obtaining a multimodal training data set, wherein the multimodal training data set includes multiple groups of multimodal training data, and each group of the multimodal training data includes at least one of image data, audio data, and point cloud data; using the multimodal training data and the target guidance information to train the multimodal deep learning model to obtain the trained multimodal deep learning model.
[0012] According to one embodiment of the present invention, the target push strategy includes: instantly pushing abnormal conditions in the target scene, periodically pushing the status of the target person in the target scene, and the push message includes at least one of text, voice, picture and video.
[0013] According to an embodiment of the present invention, the method further includes: receiving operation information of the user terminal for the push message, and optimizing the target push strategy according to the operation information.
[0014] According to one embodiment of the present invention, the multimodal environment data includes voice instructions, and the voice instructions include a target object and a target action. The method also includes: obtaining a target position of the target object in the target scene based on the structured semantic information, and performing the target action based on the target position.
[0015] According to one embodiment of the present invention, the method further includes: when it is determined that the target task is completed, generating an inquiry dialogue corresponding to the target task; and / or when it is determined that there is a care target based on the structured semantic information, generating a care dialogue.
[0016] According to an embodiment of the present invention, the target scene is provided with a plurality of robots, and the acquiring the multimodal environment data of the target scene comprises: acquiring the multimodal environment data of the target scene through the plurality of robots.
[0017] According to an embodiment of the present invention, the method further includes: when there is a task to be migrated, determining a category of the task to be migrated; determining a target migration robot from the multiple robots according to the category, and controlling the target migration robot to execute the task to be migrated.
[0018] According to one embodiment of the present invention, the target migration robot is determined from the multiple robots according to the category, and the target migration robot is controlled to execute the task to be migrated, including: if the category is a global task, the target migration robot is controlled to execute the unfinished part of the task to be migrated; if the category is a local task, the target migration robot is controlled to execute the task to be migrated, and the execution result is sent to the original execution robot of the task to be migrated, so that the original execution robot continues to execute the global task to which the task to be migrated belongs.
[0019] According to one embodiment of the present invention, before acquiring the multimodal environmental data of the target scene, the method includes: determining whether a periodic trigger instruction, a preset sound feature trigger instruction, a preset image feature trigger instruction or a manual trigger instruction is detected.
[0020] To achieve the above objectives, a second embodiment of the present invention proposes a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the robot interaction method proposed in the first embodiment of the present invention is implemented.
[0021] To achieve the above-mentioned purpose, the third aspect of the present invention proposes a controller, a memory, and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the robot interaction method proposed in the first aspect of the present invention is implemented.
[0022] To achieve the above-mentioned purpose, a fourth aspect of the present invention provides a robot, comprising a controller as provided in the third aspect of the present invention.
[0023] To achieve the above-mentioned purpose, a fifth embodiment of the present invention proposes a robot interaction system, including a user terminal and the robot proposed in the fourth embodiment of the present invention.
[0024] Additional aspects and advantages of the present invention will be given in part in the following description and in part will be obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] Figure 1 is a flow chart of a robot interaction method according to an embodiment of the present invention;
[0026] Figure 2 is a training flow chart of a multimodal deep learning model according to an embodiment of the present invention;
[0027] Figure 3 is a flow chart of a robot interaction method according to a specific embodiment of the present invention;
[0028] Figure 4is a structural block diagram of a controller according to an embodiment of the present invention;
[0029] Figure 5 is a schematic diagram of a robot according to an embodiment of the present invention;
[0030] Figure 6 Schematic diagram of a robot interaction system according to an embodiment of the present invention. DETAILED DESCRIPTION
[0031] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present invention, and should not be construed as limiting the present invention.
[0032] The following is attached to the instruction manual Figure 1-6 And specific implementation methods are described in detail for the robot and its interaction method, interaction system, storage medium, and controller of the embodiments of the present invention.
[0033] Figure 1 FIG. 1 is a flow chart of a robot interaction method according to an embodiment of the present invention. Figure 1 As shown, the robot interaction method may include:
[0034] S101, acquiring multimodal environmental data of a target scene;
[0035] S102, inputting the multimodal environmental data into a pre-trained multimodal deep learning model to obtain an environmental perception result;
[0036] S103, performing semantic extraction on the environmental perception result to obtain structured semantic information;
[0037] S104, generating a push message according to the target push strategy and the structured semantic information, and pushing the push message to the user terminal bound to the robot.
[0038] In the embodiment of the present invention, the robot body is equipped with a multimodal sensor, and the multimodal sensor carried is not limited to an RGB camera, a depth camera, a microphone array, a laser radar, etc. Therefore, the multimodal environmental data collected by the robot body is not limited to image data, audio data, and point cloud data. The camera carried on the robot body can be an ordinary monocular camera, or a binocular camera or a structured light camera that can provide depth information. The microphone array carried on the robot body can be an array composed of multiple microphones, which can perform sound source positioning and speech recognition. The laser radar carried on the robot body can obtain three-dimensional point cloud data of the environment, and can build map navigation and avoid obstacles. The layout and installation position of each sensor are optimized according to the specific model and application requirements of the robot.
[0039] The robot body in the embodiment of the present invention integrates basic functional modules such as drive, control, and power supply. Other functional modules may also be integrated, such as a cleaning module and a service module. Exemplarily, the robot body in the embodiment of the present invention may be a sweeping robot or an indoor service machine. It should be noted that the robot body in the embodiment of the present invention may also be other types of robots, and the embodiment of the present invention does not limit the type of the robot body.
[0040] Since most of the relevant robot perception technologies are targeted at specific tasks and scenarios, they lack versatility and flexibility. In addition, the relevant technologies require a lot of manpower to define perception rules and annotate training data, and it is difficult to cover various combinations in complex environments. The embodiment of the present invention uses an intelligent perception model-a multimodal deep learning model to process multimodal environmental data. The pre-trained multimodal deep learning model in the embodiment of the present invention has excellent generalization ability, can cope with open environmental understanding tasks, and can integrate and process heterogeneous data from multiple sensors, and build a holistic understanding of the environment through end-to-end learning. By introducing the open domain understanding capability of the multimodal large model, the robot is given more powerful environmental perception and human-computer interaction intelligence.
[0041] Specifically, the acquired multimodal environmental data is input into a pre-trained multimodal deep learning model, which understands and infers the multimodal environmental data. The set target guidance information guides the multimodal deep learning model to describe and summarize the environmental information, and generates corresponding environmental perception results, which are presented in natural language. For example, the environmental perception results may be "an old man is sitting on the sofa in the living room, and he seems to be dozing off", "there is an apple and an orange on the dining table", etc.
[0042] In one embodiment of the present invention, a semantic extraction module can be used to convert the environmental perception results into structured representations of subjects, states, attributes, locations, etc. using natural language processing technologies such as named entity recognition, relationship extraction, and semantic role labeling. For example, from the description of "an old man is sitting on the sofa in the living room, and he seems to be dozing off", a triple such as <subject: old man, state: dozing off, location: sofa in the living room> can be extracted. For another example, from "there is an apple and an orange on the dining table", two triples of <object: apple, location: dining table> and <object: orange, location: dining table> can be extracted. It should be noted that the extracted semantic information is encapsulated in JSON format, with the corresponding confidence score and timestamp attached.
[0043] In the embodiment of the present invention, structured semantic information can be stored in the form of multiple groups such as subject, state, attribute, position, etc., and can also be stored in the form of graphs, relational databases, etc. The stored structured semantic information provides a data basis for subsequent information push, retrieval, analysis and reasoning tasks, and also provides support for the robot's knowledge accumulation and decision-making.
[0044] The target push strategy in the embodiment of the present invention is configured by the user according to his needs, taking into account factors such as the importance, timeliness, and user preferences of the information. Therefore, when executing push tasks, push tasks can be dynamically generated and assigned priorities, and push information with high priority is pushed to users first, thus realizing personalized and accurate push.
[0045] In the embodiment of the present invention, the user terminal can be a device with communication and human-computer interaction functions such as a smart phone, a tablet computer, and a smart speaker. The user terminal is installed with an APP supporting the embodiment of the present invention as an interface for receiving push information and feedback of user instructions.
[0046] As a specific implementation method, after triggering the robot body to cruise the target scene, the multimodal environmental data of multiple channels such as vision, sound, and distance collected by the multimodal sensors carried by the robot body during the cruise are obtained. The multimodal environmental data collected by the robot body is input into a pre-trained multimodal deep learning model for processing, and the pre-trained multimodal deep learning model generates corresponding environmental perception results according to the target guidance information. Natural language processing technologies such as named entity recognition, relationship extraction, and semantic role labeling are used to convert the environmental perception results into structured representations of subjects, states, attributes, locations, etc., and obtain structured semantic information about the environmental perception results. According to the target push strategy and structured semantic information, push tasks are dynamically generated and priorities are assigned. Push messages comprehensively consider multiple contextual factors such as user portraits, terminal status, and network environment to achieve personalized and accurate push. The push tasks are converted into push messages suitable for display on different terminals and sent to the user terminal through the mobile network. Users receive push messages and can view and feedback through the APP interactive interface. Compared with the traditional post-query method, the mechanism of pushing according to the target push strategy can remind users of key information in a timely manner, greatly improving the security and interactive experience of the system.
[0047] The robot interaction method of the embodiment of the present invention uses a pre-trained multimodal deep learning model to process multimodal environmental data, extracts the processed environmental perception results as structured semantic information, and generates a push message based on the target push strategy to send the structured semantic information to the user terminal. The embodiment of the present invention forms a set of technical solutions for robot universal perception and human-computer interaction by integrating multimodal deep learning models and target push strategies. It has the advantages of strong environmental adaptability, flexible and efficient deployment, and friendly human-computer interaction. It has broad application prospects in the fields of intelligent robots and artificial intelligence.
[0048] In one embodiment of the present invention, before acquiring the multimodal environment data of the target scene, the robot interaction method may include:
[0049] It is determined that a periodic trigger instruction, a preset sound feature trigger instruction, a preset image feature trigger instruction or a manual trigger instruction is detected.
[0050] Specifically, a combination of multiple trigger mechanisms can be used to flexibly start the robot to perform perception tasks to flexibly respond to different environmental requirements. Among them, the trigger mechanism includes at least a periodic trigger mechanism, a preset sound feature trigger mechanism, a preset image feature trigger mechanism or a manual trigger mechanism.
[0051] It is feasible that the robot starts the perception task in the standby state through periodic awakening or external event triggering (such as a preset sound feature triggering instruction, a preset image feature triggering instruction or a manual triggering instruction), and the multimodal sensor starts to collect multimodal environmental data.
[0052] For example, when an external event is triggered, the user can wake up the robot through natural voice commands such as "Xiaoou, what time is it now?" or "Xiaoou, go to the kitchen" to start the robot's perception task.
[0053] In one embodiment of the present invention, before inputting the multimodal environment data into a pre-trained multimodal deep learning model, the robot interaction method may further include: preprocessing the multimodal environment data.
[0054] In one embodiment of the present invention, preprocessing includes feature extraction, spatiotemporal alignment, distortion correction of images and point clouds, noise reduction and echo cancellation of speech data, etc.
[0055] Specifically, a data processing unit may be used to preprocess the multimodal environment data. Exemplarily, RGB images and depth images are processed through distortion correction, spatiotemporal alignment, feature extraction, etc. to form unified visual features. Audio data is processed through noise reduction, echo cancellation, acoustic feature extraction, etc. to form audio features. Point cloud data is processed through downsampling, voxel filtering, normal estimation, etc. to form geometric features. The preprocessed multimodal environment data can be transmitted to a pre-trained multimodal deep learning model in a unified format.
[0056] In an embodiment of the present invention, the data processing unit may adopt a hardware module such as an embedded processor, a system on a chip or a dedicated integrated circuit, and carry data processing algorithm software to pre-process the multimodal environmental data.
[0057] In one embodiment of the present invention, Figure 2 As shown, the training process of the multimodal deep learning model may include:
[0058] S201, building a multimodal deep learning model;
[0059] S202, obtaining a multimodal training data set, wherein the multimodal training data set includes multiple groups of multimodal training data, and each group of multimodal training data includes at least one of image data, audio data, and point cloud data;
[0060] S203, using the multimodal training data and the target guidance information to train the multimodal deep learning model to obtain a trained multimodal deep learning model.
[0061] Specifically, the multimodal deep learning model can use network architectures such as Multi-Modal Transformer and Cross-ModalAttention Network that integrate visual, speech, and text features to perform end-to-end processing of multimodal environmental data. Large-scale cross-modal data is obtained to obtain a multimodal training dataset, and the multimodal deep learning model is pre-trained using the multimodal training dataset. Self-supervised learning, contrastive learning, distillation learning and other methods are used to learn the general representation of multimodal information such as vision (such as image data, point cloud data), speech (such as audio data), and text. Pre-training on massive graphic, text and audio data obtains a trained multimodal deep learning model with general cross-modal representation capabilities. The trained multimodal deep learning model takes image, audio, and point cloud features as input, and can output natural language text, and realizes the fusion and interaction of cross-modal features through the attention mechanism.
[0062] Adapt to the complex and ever-changing task environment, set target guidance information, so that the multimodal deep learning model can perform different types of perception tasks such as environmental understanding, visual question and answer, anomaly detection, etc., to guide the multimodal deep learning model to describe and summarize environmental information and generate a natural language description of the environmental state. Exemplarily, the target guidance information can be a prompt text of "Please describe the environmental state you observed", guiding the multimodal deep learning model to output the corresponding environmental description. Similarly, you can also set questions such as "What objects are on the dining table?" and "Is there anyone in the living room?" to allow the multimodal deep learning model to perform visual question and answer and anomaly detection.
[0063] Multimodal fusion and open domain understanding enable robots to acquire human-like environmental perception and expression capabilities, and adapt to complex and changing task environments. In the application stage, the trained multimodal deep learning model takes preprocessed multimodal environmental data as input and outputs environmental descriptions in natural language.
[0064] The multimodal deep learning model of the embodiment of the present invention has open cognitive capabilities. Thanks to pre-training on massive data, the multimodal deep learning model can cope with open vocabulary environmental understanding tasks without designing rules and annotating data for specific scenarios and tasks. By setting target guidance information to guide the multimodal deep learning model to summarize environmental information, it can obtain environmental perception and expression capabilities similar to those of humans, and can handle complex and changing environments in a universal and flexible manner to cope with various extreme emergencies.
[0065] In one embodiment of the present invention, the target push strategy may include: instant push of abnormal conditions in the target scene, periodic push of the status of the target person in the target scene, and the form of the push message includes at least one of text, voice, picture and video.
[0066] Specifically, the target push strategy may include strategies such as instant push of abnormal situations in the target scene, periodic push of the status of the target person in the target scene, etc. It should be noted that the target push strategy is customized according to user needs. Push messages can be in various forms such as text, voice, image, video, etc., and can be embedded with interactive controls.
[0067] It is feasible to use an information push platform to receive structured perception information from the semantic extraction module and select high-priority push tasks according to the preset target push strategy. The push strategy is defined in the form of rules, taking into account factors such as the confidence, timeliness, and user preferences of the information. For example, "when an abnormal situation is detected (such as the appearance of a stranger, broken glass, etc.), it will be pushed to the owner immediately"; "the status of the elderly and babies will be summarized every 30 minutes and pushed to family members", etc. The information push platform converts push tasks into push messages in the format of text, voice, pictures, videos, etc., and sends them to the user terminal in real time through the WebSocket protocol.
[0068] For example, the robot body is a sweeping robot and the user terminal is a smart phone. The user terminal is installed with a control APP that matches the sweeping robot. The APP maintains a long connection with the information push platform in the background, receives real-time push messages and reminds users by vibration, ringtones, etc. Users can browse the text description and image annotation of the perceived information on the APP homepage, review the historical push records, and use interactive functions such as likes, comments, and sharing.
[0069] In one embodiment of the present invention, the robot interaction method may further include:
[0070] Receive the user terminal's operation information for push messages, and optimize the target push strategy based on the operation information.
[0071] Specifically, users can browse, play, and save push information in the user terminal APP, and interact with the robot through clicks, voice, etc. When a new push message is received, the APP reminds the user by means of cards, pop-ups, voice broadcasts, etc. Users can view historical pushes in the APP, and interact with messages of interest by liking, commenting, sharing, and other operations. Users can also actively query the robot's working status through the APP, adjust the target push strategy, and remotely control the robot to perform specific tasks. For example, users can remotely control the sweeping robot through the APP control panel, such as returning to charge, fixed-point cleaning, pausing work, etc. Users can receive push information through the terminal APP, view the details of the perception task, and interact with the robot by voice, text, etc. as needed. Based on the user's APP usage behavior and feedback, the information push platform can further optimize the push strategy to improve the accuracy of the push and user satisfaction.
[0072] The embodiment of the present invention generates push tasks according to the importance, timeliness and relevance of the target push strategy and structured semantic information. The push tasks are converted into push messages in various forms and distributed through the user terminal APP. Users browse the push messages in the APP and can comment, like, control tasks and other interactive operations on the push messages. The perceived quality is evaluated based on user feedback, and the pre-trained multimodal deep learning model is optimized, and robot task planning and behavior control are performed at the same time. The optimized pre-trained multimodal deep learning model and control instructions are fed back to the robot to form a closed loop.
[0073] As a specific embodiment, Figure 3 As shown in the figure, the robot's perception task execution is flexibly started through various trigger mechanisms (periodic, feature trigger, manual trigger, etc.). The multimodal sensors carried on the sweeping robot or indoor service robot actively detect the environmental information of the target scene. The sensors carried on the sweeping robot or indoor service robot include but are not limited to RGB cameras, depth cameras, microphone arrays, laser radars, etc. The multimodal environmental data collected by the multi-sensors are pre-processed by time-space alignment, feature extraction, etc. to form a unified multimodal data stream. Based on the open domain environment understanding method of multimodal deep learning, a multimodal deep neural network (such as Multi-Modal Transformer) is used to perform end-to-end processing of the robot's perception data. The model is self-supervised pre-trained on large-scale cross-modal data to learn the general representation of multimodal information such as vision, speech, and text. By setting the natural language prompt guidance model, that is, the target guidance information performs cognitive tasks such as open domain description, question answering, and summary of environmental information. Multimodal fusion and open domain understanding enable the robot to acquire human-like environmental perception and expression capabilities and adapt to complex and changing task environments. The semantic information extraction method for information push performs semantic structured extraction on the environment description text generated by the multimodal model to obtain the structured representation of information elements. Natural language processing technologies such as named entity recognition and relationship extraction are used to extract key semantic information of the environment state. Semantic information is structured and stored, retrieved and reasoned in the form of multiple groups such as subject, state, attribute, and location. The structured semantic representation provides a data basis for intelligent information push and also supports the robot's knowledge accumulation and decision-making. The intelligent information push method for robots dynamically generates push tasks and assigns priorities based on the importance, timeliness, relevance and other attributes of semantic information. The push method comprehensively considers multiple contextual factors such as user portraits, terminal status, and network environment to achieve personalized and accurate push. The push content supports multiple information presentation forms such as text, voice broadcast, and image annotation, and can be optimized in combination with user interaction feedback. Users can receive push information through the terminal APP, view the details of the perception task, and interact with the robot through voice, text, etc. as needed.
[0074] The embodiment of the present invention is a robot intelligent perception system based on multimodal learning and information push, and proposes a robot intelligent perception and information push method, which integrates multimodal perception, deep learning, semantic extraction, intelligent push and other modules. The robot obtains multimodal perception data through environmental detection, inputs the deep learning model for open domain understanding, and extracts semantic information to form a push task. The push platform pushes the perception information to the user terminal according to the push strategy, realizing active information interaction between the robot and the user. The system significantly improves the robot's environmental adaptability, perception and understanding ability, and human-computer interaction experience, and can be widely used in smart home, security monitoring, auxiliary care and other fields.
[0075] In one embodiment of the present invention, the multimodal environment data may include voice instructions, the voice instructions include a target object and a target action, and the robot interaction method further includes:
[0076] The target position of the target object in the target scene is obtained according to the structured semantic information, and the target action is performed according to the target position.
[0077] The multimodal deep learning model (such as BERT, GPT, etc.) introduced in the intelligent perception model realizes the cross-modal fusion understanding of speech, vision, and text information. The multimodal deep learning model establishes deep semantic associations between voice commands, visual observations, and environmental knowledge, which can achieve more accurate and natural speech understanding and interaction. Because the structured semantic information in the embodiment of the present invention is in the form of multi-tuples such as subject, state, attribute, position, or in the form of graph storage, relational database, etc. The stored structured semantic information provides a data basis for subsequent information push, retrieval, analysis and reasoning tasks, and also provides support for the robot's knowledge accumulation and decision-making. Therefore, when using voice commands to trigger the robot body to perform perception tasks, the robot body can obtain the target position of the target object in the target scene based on the structured semantic information, and perform the target action according to the target position.
[0078] As a specific example, when the voice command issued is "Bring me my glasses", the robot body can be controlled to perform the perception task of "Bring me my glasses". The intelligent perception model generates environmental perception results based on "Bring me my glasses" in the multimodal environmental data, and can convert the environmental perception results into structured semantic information. According to the structured semantic information stored in it, the target position of the target object in the target scene can be obtained, and the specific position of "my glasses" can be located in combination with the visual information. According to the specific position of "my glasses", the robot can perform the picking action to bring "my glasses" to the user.
[0079] As another specific example, when the voice command issued is "Put this cup on the table", the robot body is controlled to perform the perception task of "Put this cup on the table". The robot body infers the referent objects of "this cup" and "table" based on the co-occurrence relationship of objects in the environment, and performs the action of "Put this cup on the table".
[0080] In one embodiment of the present invention, to realize the far-field voice interaction function of the robot, an echo cancellation unit may be added to the microphone array to improve the far-field voice interaction performance of the robot, so as to better serve the user when executing the above voice commands.
[0081] In one embodiment of the present invention, in order to push messages in a human-like voice form, a text-to-speech module and a speech synthesizer may be integrated into the information push platform to convert push messages into a human-like voice form in real time.
[0082] In one embodiment of the present invention, the robot interaction method may further include:
[0083] When it is determined that the target task is completed, generating a query dialogue corresponding to the target task; and / or
[0084] When it is determined that there is a care target according to the structured semantic information, a care dialogue is generated.
[0085] Specifically, after the robot completes a complex task, it can actively request user feedback, that is, generate a responsive inquiry dialogue to request user feedback. Active dialogue increases the robot's emotional interaction and social attributes, and enhances the user's sense of companionship and trust.
[0086] As a specific example, the robot completes the task of making coffee, and when the coffee is ready, it can generate an inquiry dialogue such as "Master, your coffee is ready, do you need me to bring it to you?" and can reply "OK, bring it to the study" according to the user's voice to perform the corresponding coffee delivery task.
[0087] As another specific example, after the robot finishes cleaning the bedroom, it can generate an inquiry dialogue such as “Master, I have finished cleaning the bedroom. Are you satisfied with my work?”
[0088] Specifically, the robot can also generate caring dialogues. As a specific example, when the robot generates environmental perception results of the user's low mood based on the multimodal environmental data it collects, after converting the environmental perception results of the user's low mood into structured semantic information, when it determines that the user has been in a low mood for a long time, it can actively initiate a caring dialogue, such as "Master, you look a little tired, do you need me to play some light music for you?"
[0089] It should be noted that the user's feedback on the inquiry dialogue and the caring dialogue can provide an optimization basis for the robot's dialogue strategy and task execution.
[0090] In one embodiment of the present invention, a target scene is provided with a plurality of robots, and obtaining multimodal environmental data of the target scene may include:
[0091] Acquire multimodal environmental data of the target scene through multiple robots.
[0092] With the increasing number of smart devices deployed in home and office scenarios, more than one robot may be deployed in the target scene, and information sharing and task collaboration between different robots become particularly important.
[0093] When multiple robots of different types (such as sweeping robots, window cleaning robots, food delivery robots, etc.) are deployed in the target scene, the robots can communicate with each other through the local area network and connect to the smart home platform in the cloud. Each robot is equipped with a set of multimodal sensors to collect environmental information of the area it is responsible for. The cloud platform aggregates multimodal environmental data from multiple robots and uniformly inputs them into a pre-trained multimodal deep learning model to output the global environmental perception results of the entire scene.
[0094] As a specific example, when a window cleaning robot, a sweeping robot, and a food delivery robot are deployed in the target scenario, the window cleaning robot detects a suspicious person wandering outside the window, the sweeping robot finds glass fragments on the floor indoors, and the food delivery robot hears screams in the kitchen. Through multimodal information fusion, the cloud infers the high-risk scenario of "someone breaks into the house through the window, and the house may be robbed." The early warning information is pushed to the owner and security personnel in a timely manner, and the robots are coordinated to take emergency measures, such as calling the police, recording evidence, and shouting on the spot.
[0095] In one embodiment of the present invention, the robot interaction method further comprises:
[0096] When there are tasks to be migrated, determine the categories of the tasks to be migrated;
[0097] A target migration robot is determined from a plurality of robots according to the category, and the target migration robot is controlled to execute the task to be migrated.
[0098] The embodiment of the present invention introduces a cross-device perception enhancement and task migration mechanism. Different robots dynamically adjust the division of labor and cooperation of perception tasks according to their own perception capabilities and working states.
[0099] Specifically, when there are tasks to be migrated, the category of the tasks to be migrated can be determined, a target migration robot can be determined from multiple robots according to the category, and the target migration robot can be controlled to perform the tasks to be migrated, thereby realizing the division of labor and cooperation in perception tasks.
[0100] In one embodiment of the present invention, determining a target migration robot from a plurality of robots according to categories, and controlling the target migration robot to perform the task to be migrated may include:
[0101] If the category is a global task, the target migration robot is controlled to execute the unfinished part of the task to be migrated;
[0102] If the category is a local task, the target migration robot is controlled to execute the task to be migrated, and the execution result is sent to the original execution robot of the task to be migrated, so that the original execution robot continues to execute the global task to which the task to be migrated belongs.
[0103] As a specific example, when a sweeping robot is unable to patrol due to insufficient power, it can transfer its patrol task to a fully charged window cleaning robot. When the high-definition camera carried by a food delivery robot is conducive to fine-grained object recognition, the sweeping robot's object retrieval task can be completed with the help of the food delivery robot's visual ability. Through task transfer, robots achieve complementary perception and execution capabilities, improving the overall robustness of the system.
[0104] The cloud platform can continuously plan and optimize the global tasks based on the execution results fed back by each robot. For example, the cleaning paths of each robot can be optimized based on the ground pollution distribution map fed back by the sweeping robot, and the meal preparation and delivery process can be optimized based on the user's service evaluation of the meal delivery robot. Through end-cloud collaborative perception and continuous learning optimization, the multi-robot system can provide more efficient, accurate, and thoughtful intelligent services.
[0105] It should be noted that the above embodiments have elaborated on the implementation methods of the robot interaction methods of the embodiments of the present invention from the perspectives of single robot, human-computer interaction, and multi-robot collaboration. Those skilled in the art will appreciate that the above embodiments are only for illustration and cannot be used to limit the scope of protection of the present invention. Without departing from the spirit and scope of the present invention, various modifications or equivalent substitutions may be made to the implementation methods, and these modifications and substitutions also fall within the scope of protection of the present invention. For example, different robot hardware platforms, sensor configurations, model architectures, task forms, interaction methods, etc. may be used. As long as the overall technical ideas of multimodal perception, open domain understanding, semantic extraction, and intelligent push are followed, they are all within the scope of protection of the present invention.
[0106] The embodiments of the present invention can optimize the interaction method adaptively according to the application scenario. For example, functions such as face recognition and pet detection can be added in the home scenario, and functions such as document parsing and meeting record can be added in the office scenario. Through customized perception models and push strategies, the personalized needs of different users can be flexibly adapted.
[0107] The embodiment of the present invention introduces multimodal deep learning, so that the robot acquires the ability to understand the environment with open vocabulary, can flexibly respond to complex and changeable real scenes, and improves the robot's environmental adaptability; the multimodal deep learning model in the embodiment of the present invention can quickly migrate to a new environment through a small number of samples, reduce the cost of data collection and manual annotation, accelerate the deployment of the system, and improve the efficiency of system deployment; the multimodal fusion in the embodiment of the present invention helps to fully and accurately understand the environmental state, and the open domain understanding based on the target guidance information enables the robot to acquire human-like cognitive and descriptive capabilities, and enhances the robot's perception and understanding capabilities; the intelligent information push in the embodiment of the present invention enables the system to actively remind users of key information, and the multi-form information presentation and interactive feedback enhance the user's sense of participation and control, and improve the human-computer interaction experience; the embodiment of the present invention can be widely used in homes, offices, businesses, public places, etc., providing users with multiple value-added services such as security monitoring, abnormal alarms, and behavior analysis, enriching the application scenarios of robots.
[0108] The embodiments of the present invention integrate cutting-edge artificial intelligence technologies such as multimodal deep learning, knowledge representation, and intelligent push, and provide a new solution for the intelligent development of robots. The embodiments of the present invention significantly improve the intelligent level of robot perception and human-computer interaction, and have the advantages of strong environmental adaptability, flexible and efficient deployment, comprehensive perception and understanding, and friendly interactive experience. The embodiments of the present invention not only expand the application boundaries of sweeping robots and indoor service robots, but also provide technical references for the intelligent upgrade of other types of robots, and have broad application prospects in the fields of intelligent robots and artificial intelligence.
[0109] The embodiment of the present invention uses a pre-trained multimodal deep learning model to perform end-to-end processing on the multi-channel perception data collected by the robot, breaking through the bottleneck of traditional isolated perception and realizing the integrated understanding of visual, voice, text and other information. Through pre-training on massive data, the model has learned a common cross-modal representation and has the ability to generalize and understand open vocabulary. By setting target guidance information to describe and summarize environmental information, there is no need to train a dedicated model for each scene. As a result, the robot's understanding and adaptability to complex environments is greatly improved.
[0110] The embodiment of the present invention designs a set of active information interaction mechanisms based on push services. The system extracts the environmental information perceived by the robot into a structured semantic representation, and intelligently generates push tasks based on the importance and timeliness of the information. Through a personalized push platform, information can be pushed to the user's mobile terminal in a timely manner, and multi-form information presentation and interactive feedback further enhance the human-computer interaction experience. Active information push enables the application value of robots in smart homes, security monitoring and other fields to be fully utilized.
[0111] The embodiments of the present invention break through the limitations of traditional robot perception and interaction technologies, and use cutting-edge artificial intelligence models to significantly improve the robot's environmental understanding and information service intelligence, better meeting the needs of practical applications. Compared with the existing technology, the present invention has the advantages of strong perception ability, good generalization, efficient deployment, and friendly interaction, providing new technical ideas and implementation plans for the intelligent development of the robot industry.
[0112] The embodiments of the present invention significantly improve the robot's environmental perception ability and intelligence level. The present invention uses a multimodal deep learning model to integrate and understand the multi-channel perception data collected by the robot, overcoming the limitations of traditional single-modal perception. Through pre-training on massive cross-modal data, the model learns common feature representations and has the ability to generalize and understand open vocabulary. Therefore, the robot can describe and explain the perceived environmental information in the form of natural language like a human, and adapt to complex and changing application scenarios.
[0113] The embodiments of the present invention improve the deployment efficiency and scalability of the robot perception system. Thanks to the transfer learning capability of the multimodal large model, the present invention can quickly adapt to new perception tasks through a small number of samples without the need to train a dedicated model from scratch. This greatly reduces the cost of data collection and annotation and shortens the system development cycle. In addition, by setting different target guidance information, the same perception model can perform different types of tasks such as environment description, anomaly detection, semantic understanding, etc., which improves the reusability and scalability of the model.
[0114] The embodiments of the present invention enhance the initiative, real-time and friendliness of human-computer interaction. The intelligent information push mechanism of the embodiments of the present invention can automatically generate push tasks based on the importance and timeliness of perceived information, and send them to users in real time through the mobile network. Compared with the traditional passive query method, active push enables users to promptly learn about important events in the environment, such as elderly people falling, babies crying and other abnormal situations, which greatly improves the system's service quality and user experience. Push content supports a variety of media forms and interaction methods, and users can easily view perceived details, provide feedback, initiate control, etc.
[0115] The embodiments of the present invention provide support for the continuous learning and knowledge evolution of robots. The present invention extracts semantic structures from the perception information obtained by the robot to form accurate and rich representations of environmental states, such as <subject, state, attribute, location>, etc. These semantic information can be persistently stored as the basis for the robot's long-term memory. As the robot continues to work in different scenarios, perception data continues to accumulate, and the semantic information library is also constantly expanding. This lays a solid foundation for the robot's advanced intelligent behaviors such as continuous learning, knowledge evolution, and experience generalization.
[0116] The embodiment of the present invention provides a universal robot intelligent upgrade solution. The intelligent perception and information push system proposed in the present invention integrates a number of cutting-edge technologies such as environmental perception, multimodal learning, semantic extraction, and intelligent push, forming a relatively complete system architecture. The solution is universal and is not only suitable for sweeping robots and indoor service robots, but can also be extended to other types of robots such as security inspections, industrial inspections, and scene navigation. For robot products that have already been commercialized, the present invention can be used as an open function expansion solution to give them more powerful intelligent perception and human-computer interaction capabilities.
[0117] The embodiments of the present invention promote the integrated development of artificial intelligence technology and the robotics industry. Robots are an important carrier for the implementation of artificial intelligence. Introducing the latest artificial intelligence technology into the robot system in a timely manner is conducive to improving the intelligence level of robot products. The needs of robot application scenarios also provide practical problems and data support for artificial intelligence research. The present invention applies multimodal large models, push services and other technologies in the fields of robot perception and human-computer interaction, opens up a new application model of artificial intelligence in the field of robotics, and promotes the benign interaction of industry, academia, research and use. In the long run, it will surely promote the coordinated development of artificial intelligence and robots.
[0118] In summary, the present invention integrates cutting-edge artificial intelligence technologies such as multimodal deep learning, knowledge representation, and intelligent push, and provides a new solution for improving the robot's intelligent perception and human-computer interaction capabilities. This solution can significantly improve the robot's environmental adaptability, task scalability, and interactive friendliness, while reducing the system development and deployment costs. The present invention represents an important development direction for future household service robots, and is expected to form new product forms and application models, bringing intelligent, caring, and reliable robot service experiences to more families.
[0119] The present invention provides a computer-readable storage medium.
[0120] In this embodiment, a computer program is stored on a computer-readable storage medium, and when the computer program is executed by a processor, the robot interaction method as described above is implemented.
[0121] The invention provides a controller.
[0122] In this embodiment, the controller may include a memory and a processor. The memory stores a computer program. When the computer program is executed by the processor, the robot interaction method as described above is implemented.
[0123] Figure 4 4 is a structural block diagram of a controller according to an embodiment of the present invention.
[0124] like Figure 4As shown, the controller 500 includes: a processor 501 and a memory 503. The processor 501 and the memory 503 are connected, such as through a bus 502. Optionally, the controller 500 may also include a transceiver 504. It should be noted that in actual applications, the transceiver 504 is not limited to one, and the structure of the controller 500 does not constitute a limitation on the embodiments of the present invention.
[0125] Processor 501 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of the present invention. Processor 501 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.
[0126] The bus 502 may include a path to transmit information between the above components. The bus 502 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 502 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.
[0127] The memory 503 is used to store a computer program corresponding to the robot interaction method of the above embodiment of the present invention, and the computer program is controlled and executed by the processor 501. The processor 501 is used to execute the computer program stored in the memory 503 to implement the content shown in the above method embodiment.
[0128] Among them, the controller 500 includes but is not limited to: mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 4 The controller 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.
[0129] The invention provides a robot.
[0130] Figure 5 Schematic diagram of a robot according to an embodiment of the present invention. Figure 5 As shown, the robot 100 includes a controller 500 as described above.
[0131] The invention provides a robot interaction system.
[0132] Figure 6 Schematic diagram of a robot interaction system according to an embodiment of the present invention. Figure 6 As shown, the robot interaction system 200 includes a user terminal 300 and the robot 100 as described above.
[0133] In an embodiment of the present invention, the robot interaction system may further include a sweeping robot body, a multimodal sensor module, a data processing unit, an intelligent perception model, a semantic extraction module, an information push platform and a user terminal.
[0134] The robot vacuum cleaner integrates basic modules such as motion control, power management, and functions. The multimodal sensor module is installed at an appropriate position on the robot body, including a front RGB camera, a depth camera, a microphone array, and a lidar. The front RGB camera and the depth camera provide color images and depth images of the environment for visual perception tasks such as object recognition and scene understanding. The microphone array is used to receive ambient audio to achieve auditory perception tasks such as sound source localization, speech recognition, and abnormal sound detection. The lidar obtains three-dimensional point cloud data of the environment for spatial perception tasks such as mapping and positioning, and obstacle avoidance and navigation.
[0135] The data processing unit uses an embedded processor to perform real-time preprocessing on the data collected by the multimodal sensor. RGB images and depth images are processed through distortion correction, spatiotemporal alignment, feature extraction, etc. to form unified visual features. Audio data is processed through noise reduction, echo cancellation, acoustic feature extraction, etc. to form audio features. Point cloud data is processed through downsampling, voxel filtering, normal estimation, etc. to form geometric features. The processed multimodal features are transmitted to the intelligent perception model through the internal bus.
[0136] The intelligent perception model adopts a multimodal Transformer architecture and is pre-trained on massive image, text and audio data to obtain general cross-modal representation capabilities. The model takes image, audio, and point cloud features as input and natural language text as output, and realizes the fusion and interaction of cross-modal features through the attention mechanism. In the application stage, by setting prompt text such as "Please describe the environmental state you observe", the model is guided to output the corresponding environmental description. Similarly, questions such as "What objects are on the dining table?" and "Is there anyone in the living room?" can also be set to allow the model to perform visual question answering and anomaly detection.
[0137] After the semantic extraction module is connected to the intelligent perception model, it uses natural language processing technology to extract structured information from the generated environment description. For example, from the description of "an old man is sitting on the sofa in the living room, and he seems to be dozing off", we can extract a triple such as <subject: old man, state: dozing off, location: sofa in the living room>. For another example, from "there is an apple and an orange on the dining table", we can extract two triples, <object: apple, location: dining table> and <object: orange, location: dining table>. The extracted semantic information is encapsulated in JSON format, with the corresponding confidence score and timestamp attached.
[0138] The information push platform receives structured perception information from the semantic extraction module and selects high-priority push tasks according to the preset push strategy. The push strategy is defined in the form of rules, taking into account factors such as information confidence, timeliness, and user preferences. For example, "when an abnormal situation is detected (such as the appearance of a stranger, broken glass, etc.), it will be pushed to the owner immediately"; "the status of the elderly and babies will be summarized every 30 minutes and pushed to family members", etc. The platform converts push tasks into push messages in the format of text, voice, pictures, videos, etc., and sends them to the user terminal in real time through the WebSocket protocol.
[0139] Taking a smartphone as an example, the user terminal is installed with a control APP for the sweeping robot. The APP maintains a long connection with the information push platform in the background, receives real-time push messages and reminds users by vibration, ringtones, etc. Users can browse the text description and image annotation of the perceived information on the APP homepage, review historical push records, and use interactive functions such as likes, comments, and sharing. Users can also remotely control the sweeping robot through the APP control panel, such as returning to charge, cleaning at a fixed point, pausing work, etc. In addition, based on the user's APP usage behavior and feedback, the information push platform can further optimize the push strategy to improve the accuracy of the push and user satisfaction.
[0140] The robot interaction system of the embodiment of the present invention integrates multimodal perception, deep learning, semantic extraction, intelligent push and other modules. The robot obtains multimodal perception data through environmental detection, inputs the deep learning model for open domain understanding, and extracts semantic information to form a push task; the push platform pushes the perception information to the user terminal according to the push strategy, realizing active information interaction between the robot and the user; the system significantly improves the robot's environmental adaptability, perception and understanding ability, and human-computer interaction experience, and can be widely used in smart homes, security monitoring, auxiliary care and other fields.
[0141] The embodiment of the present invention realizes the robot's active perception of the environment and intelligent interaction with the user through multimodal perception, intelligent understanding, semantic extraction, push interaction and other steps. The modules of the system work closely together, and the perception results can be used to optimize the model and plan tasks, forming a data-driven closed-loop control, so that the robot can continue to evolve and adapt to environmental changes.
[0142] The robot interaction system of the embodiment of the present invention integrates multiple key technologies such as multimodal perception, deep learning, semantic extraction, and intelligent push, forming a complete set of technical solutions for human-computer interaction. This solution makes full use of the open domain understanding ability of the multimodal large model to give the robot a more comprehensive, accurate, and intelligent environmental perception ability. At the same time, the active information push mechanism makes human-computer interaction more efficient, timely, and friendly. It is expected to significantly improve the practicality and intelligence level of household service robots and bring users a better smart home service experience.
[0143] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection portion with one or more wirings (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or processing in other suitable ways if necessary, and then stored in a computer memory.
[0144] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0145] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.
[0146] In the description of the present invention, it is to be understood that the terms “center”, “longitudinal”, “lateral”, “length”, “width”, “thickness”, “up”, “down”, “front”, “back”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, “clockwise”, “counterclockwise”, “axial”, “radial”, “circumferential”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the referred device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as limiting the present invention.
[0147] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present invention, the meaning of "plurality" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined.
[0148] In the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", "fixed" and the like should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, it can be the internal connection of two elements or the interaction relationship between two elements, unless otherwise clearly defined. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.
[0149] In the present invention, unless otherwise clearly specified and limited, a first feature being "above" or "below" a second feature may mean that the first and second features are in direct contact, or the first and second features are in indirect contact through an intermediate medium. Moreover, a first feature being "above", "above" or "above" a second feature may mean that the first feature is directly above or obliquely above the second feature, or simply means that the first feature is higher in level than the second feature. A first feature being "below", "below" or "below" a second feature may mean that the first feature is directly below or obliquely below the second feature, or simply means that the first feature is lower in level than the second feature.
[0150] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.
Claims
1. A robot interaction method, It is characterized in that The method comprises: Acquire multimodal environmental data of the target scene; Inputting the multimodal environmental data into a pre-trained multimodal deep learning model to obtain an environmental perception result; Performing semantic extraction on the environmental perception result to obtain structured semantic information; A push message is generated according to the target push strategy and the structured semantic information, and the push message is pushed to a user terminal bound to the robot.
2. The robot interaction method according to claim 1, It is characterized in that The training process of the multimodal deep learning model includes: Building multimodal deep learning models; Acquire a multimodal training data set, wherein the multimodal training data set includes multiple groups of multimodal training data, and each group of the multimodal training data includes at least one of image data, audio data, and point cloud data; The multimodal deep learning model is trained using the multimodal training data and the target guidance information to obtain the trained multimodal deep learning model.
3. The robot interaction method according to claim 1, It is characterized in that The target push strategy includes: instantly pushing abnormal situations in the target scene, and periodically pushing the status of target personnel in the target scene. The form of the push message includes at least one of text, voice, picture and video.
4. The robot interaction method according to claim 3, It is characterized in that The method further comprises: Receive operation information of the user terminal for the push message, and optimize the target push strategy according to the operation information.
5. The robot interaction method according to claim 2, It is characterized in that The multimodal environment data includes a voice instruction, the voice instruction includes a target object and a target action, and the method further includes: A target position of the target object in the target scene is obtained according to the structured semantic information, and the target action is performed according to the target position.
6. The robot interaction method according to claim 2, It is characterized in that The method further comprises: When it is determined that the target task is completed, generating a query dialogue corresponding to the target task; and / or When it is determined according to the structured semantic information that a care target exists, a care dialogue is generated.
7. The robot interaction method according to claim 1, It is characterized in that The target scene is provided with a plurality of robots, and the multimodal environment data of the target scene is obtained, including: The multimodal environmental data of the target scene is acquired through the multiple robots.
8. The robot interaction method according to claim 7, It is characterized in that The method further comprises: When there are tasks to be migrated, determining the category of the tasks to be migrated; A target migration robot is determined from the multiple robots according to the category, and the target migration robot is controlled to execute the task to be migrated.
9. The robot interaction method according to claim 8, It is characterized in that The step of determining a target migration robot from the plurality of robots according to the category, and controlling the target migration robot to perform the task to be migrated, comprises: If the category is a global task, controlling the target migration robot to execute the unfinished part of the task to be migrated; If the category is a local task, the target migration robot is controlled to execute the task to be migrated, and the execution result is sent to the original execution robot of the task to be migrated, so that the original execution robot continues to execute the global task to which the task to be migrated belongs.
10. The robot interaction method according to claim 1, It is characterized in that Before acquiring the multimodal environment data of the target scene, the method includes: It is determined that a periodic trigger instruction, a preset sound feature trigger instruction, a preset image feature trigger instruction or a manual trigger instruction is detected.
11. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the robot interaction method according to any one of claims 1 to 10 is implemented.
12. A controller, a memory, and a processor, wherein the memory stores a computer program. It is characterized in that When the computer program is executed by the processor, the robot interaction method according to any one of claims 1 to 10 is implemented.
13. A robot, It is characterized in that Comprising a controller as claimed in claim 12.
14. A robot interaction system, It is characterized in that The device comprises a user terminal and the robot as claimed in claim 13.
Citation Information
Cited By
Multi-type robot collaborative operation method based on dynamic environment perception modeling
CN119376326A
Adaptive behavior planning system for sensing robot
CN119567266A
Robot control method, device and system
CN119772907A
