Robot task reasoning method and system fusing multi-modal information and ontology knowledge

Abstract

The application discloses a robot task reasoning method and system fusing multi-modal information and ontology knowledge, and comprises the following steps: constructing an ontology knowledge base; acquiring voice information, visual information and scene knowledge information; recognizing user behavior information and scene relationship information between the user and the environment according to the visual information, and obtaining visual modal information in a text form through information screening; recognizing voice modal information in a text form according to the voice information; obtaining scene knowledge modal information in a text form by querying the ontology knowledge base according to the scene knowledge information; obtaining scene multi-modal fusion information according to the voice modal information, the visual modal information and the scene knowledge modal information in a text form; generating a service task by using a large language model after training; and controlling a service robot to perform corresponding actions, so that the decision-making capability, the execution capability and the service level of the service robot are improved.

Description

Technical Field

[0001] This invention relates to the field of service robot technology, and in particular to a robot task reasoning method and system that integrates multimodal information and ontology knowledge. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The perception, decision-making, and execution capabilities of service robots determine their service level, and improvements in their decision-making abilities will contribute to enhancing that level. An ontology knowledge base is a crucial means for service robots to manage scene information. Real-time acquisition of environmental information by service robots improves perception efficiency and facilitates better environmental interaction. However, in multimodal environments, multiple modalities such as voice, vision, and scene knowledge coexist. The complexity of environmental information and the variability of user behavior lead to diverse expected service tasks. Traditional methods relying on robot knowledge bases combined with rules for task reasoning are insufficient to flexibly address complex multimodal scenarios.

[0004] Generative large language models have powerful text generation and reasoning capabilities and have shown great potential in human-computer interaction applications in recent years. Large language models can effectively process natural language and directly generate understandable robot instructions, simplifying development work. The transfer application capability based on fine-tuning mode also provides solutions for small-sample learning and low computational performance requirements for different application scenarios.

[0005] However, service robots developed based on large language models often lack the ability to acquire and manage scene knowledge. During the reasoning phase, service robots only reason based on the voice or visual information collected in the current time period, lacking the acquisition and analysis of prior information about users and the environment. This reasoning is based on the condition that "the machine does not know the human", making it difficult to generate the service tasks expected by the user. On the other hand, when service robots interact with the environment during the execution phase, they lack environmental information and need to perceive the current environment from scratch to obtain information about the interaction target, making it difficult to accurately and effectively execute functions and complete service tasks. Summary of the Invention

[0006] To address the aforementioned issues, this invention proposes a robot task reasoning method and system that integrates multimodal information and ontology knowledge. It extracts multimodal information from the current scene and combines it with large language models and knowledge base technology, thereby improving the decision-making ability, execution ability, and service level of service robots.

[0007] To achieve the above objectives, the present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a robot task reasoning method that integrates multimodal information and ontology knowledge, comprising:

[0009] Construct an ontology knowledge base that includes ontology information of users, environment, and service robots;

[0010] Acquire audio information, visual information, and scene knowledge information from the scene;

[0011] User behavior information and scene relationship information between users and the environment are obtained based on visual information recognition, and then visual modal information in text form is obtained through information filtering; wherein, information filtering includes retaining scene relationship information in which users interact with the environment based on user behavior information;

[0012] Speech modality information in text form is obtained from speech information recognition;

[0013] Based on the scene knowledge information, the scene knowledge modality information in text form is obtained by querying the ontology knowledge base;

[0014] Based on the speech modal information, visual modal information, and scene knowledge modal information in text form, scene multimodal fusion information is obtained;

[0015] Based on the multimodal fusion information of the scene, a trained large language model is used to generate service tasks, thereby controlling the service robot to perform corresponding actions, and updating the service tasks based on the feedback from the service robot.

[0016] As an alternative implementation, the ontology knowledge base includes classes, instances, and attributes, wherein a class represents a collection of individuals with the same characteristics, an instance represents an individual, and attributes include object attributes and data attributes. Object attributes represent the relationship between classes and instances and are used for qualitative description, while data attributes represent the relationship between instances and numerical values ​​and are used for quantitative description.

[0017] As an alternative implementation, in the ontology knowledge base, user information includes age and behavioral habits; service robot ontology information includes robot appearance and functions; environmental information includes environmental objects, operation objects, location and scene structure, used to define relevant information about items in the scene, and further subdivided according to different subordinate categories in each category, thereby defining the smallest classification.

[0018] As an alternative implementation method, the scene relationship information between the user and the environment is a text-based triplet relationship pair, represented in the form of subject + predicate + object. The process of obtaining the text-based visual modal information through information filtering includes:

[0019] (1) Retain the triplet of predicates that are user-subject and involve action relationships. Among them, tags that are synonymous with user behavior information tags are merged and one is retained, while tags that are not synonymous are retained and connected at the same time.

[0020] (2) Preserve triples of predicates that are user-subject and involve positional relationships;

[0021] (3) Preserve triples of predicates that have the user as the object and involve positional relationships;

[0022] (4) Using the user as the subject, merge the relationship between the behavior recognition verb and the predicate attributes of (2) and (3).

[0023] As an alternative implementation method, the item information and user behavior information in the ontology knowledge base are queried. The item information includes the item location and whether it can be grasped, which is used to assist the service robot in target grasping. The user behavior information in text form is used as the scene knowledge modal information.

[0024] Secondly, the present invention provides a robot task reasoning system that integrates multimodal information and ontology knowledge, comprising:

[0025] The knowledge base construction module is configured to build an ontology knowledge base containing ontology information of users, environment, and service robots.

[0026] The acquisition module is configured to acquire voice information, visual information, and scene knowledge information in the scene.

[0027] The visual processing module is configured to identify user behavior information and scene relationship information between the user and the environment based on visual information, and then obtain visual modal information in text form through information filtering; wherein, information filtering includes retaining scene relationship information in which the user and the environment have interactive relationships based on user behavior information;

[0028] The speech processing module is configured to recognize speech modal information in text form based on speech information;

[0029] The scene knowledge processing module is configured to obtain scene knowledge modal information in text form by querying the ontology knowledge base based on scene knowledge information.

[0030] The fusion module is configured to obtain scene multimodal fusion information based on text-based speech modal information, visual modal information, and scene knowledge modal information;

[0031] The task generation module is configured to generate service tasks based on multimodal fusion information of the scene and a trained large language model, thereby controlling the service robot to perform corresponding actions and updating the service tasks based on the feedback from the service robot.

[0032] Thirdly, the present invention provides a service robot system, comprising:

[0033] The robot itself is equipped with a binocular depth camera for acquiring visual information;

[0034] The edge computing platform is used to identify user behavior information and scene relationship information between users and the environment based on visual information, and then obtain visual modal information in text form through information filtering. The information filtering includes retaining scene relationship information with interactive relationships between users and the environment based on user behavior information; and obtaining scene knowledge modal information in text form by querying the ontology knowledge base based on scene knowledge information.

[0035] The human-computer interaction terminal is used to acquire voice information in the scene and to recognize the voice modal information in text form based on the voice information;

[0036] The cloud computing platform receives text-based speech modal information, visual modal information, and scene knowledge modal information, and fuses them to obtain scene multimodal fusion information. Based on the scene multimodal fusion information, a trained large language model is used to generate service tasks, thereby controlling the service robot to perform corresponding actions, and updating the service tasks based on the feedback from the service robot.

[0037] Fourthly, the present invention provides an electronic device including a memory and a processor, and computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in the first aspect.

[0038] Fifthly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0039] In a sixth aspect, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0040] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0041] This invention proposes a robot task reasoning method and system that integrates multimodal information and ontology knowledge. It extracts multimodal information from the current scene and combines it with a large language model and knowledge base technology. The advantage of the ontology knowledge base lies in its structured storage and management of scene knowledge information, conveniently providing the necessary scene knowledge. The advantage of the large language model lies in its ability to process complex task data in complex multimodal scenarios, and its excellent text generation capabilities make service task reasoning possible. Therefore, by combining the ontology knowledge base and the large language model, and targeting specific scenarios, service task reasoning is performed based on the general applicability of the large language model in task data processing. Task execution is achieved with the assistance of the ontology knowledge base, thereby improving the decision-making ability, execution ability, and service level of the service robot.

[0042] This invention proposes a robot task reasoning method and system that integrates multimodal information and ontology knowledge, enabling service task reasoning that better meets user expectations. It can better understand user intentions, improve the intelligence level of service robots, and have the ability to execute user-expected service functions and manage scene knowledge information, thus contributing to the entry of service robots into homes.

[0043] Advantages of additional aspects of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0044] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0045] Figure 1 This is an overall framework diagram of the robot task reasoning method that integrates multimodal information and ontology knowledge provided in Embodiment 1 of the present invention;

[0046] Figure 2 This is a topology diagram of the ontology knowledge base provided in Embodiment 1 of the present invention;

[0047] Figure 3 This is a hierarchical structure diagram of the knowledge base class provided in Embodiment 1 of the present invention;

[0048] Figure 4 This is a diagram of object attributes and data attributes of an ontology knowledge base provided in Embodiment 1 of the present invention;

[0049] Figure 5 This is a flowchart of the fall medication delivery process provided in Embodiment 1 of the present invention. Detailed Implementation

[0050] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0051] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0052] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form as well. Furthermore, it should be understood that the terms “comprising” and “including”, and any variations thereof, are intended to cover non-exclusive inclusion, for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such process, method, product, or apparatus.

[0053] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0054] Example 1

[0055] This embodiment provides a robot task reasoning method that integrates multimodal information and ontology knowledge, improving and enhancing the decision-making ability of service robots, such as... Figure 1 As shown, it includes:

[0056] Construct an ontology knowledge base that includes ontology information of users, environment, and service robots;

[0057] Acquire audio information, visual information, and scene knowledge information from the scene;

[0058] User behavior information and scene relationship information between users and the environment are obtained based on visual information recognition, and then visual modal information in text form is obtained through information filtering; wherein, information filtering includes retaining scene relationship information in which users interact with the environment based on user behavior information;

[0059] Speech modality information in text form is obtained from speech information recognition;

[0060] Based on the scene knowledge information, the scene knowledge modality information in text form is obtained by querying the ontology knowledge base;

[0061] Based on the speech modal information, visual modal information, and scene knowledge modal information in text form, scene multimodal fusion information is obtained;

[0062] Based on the multimodal fusion information of the scene, a trained large language model is used to generate service tasks, thereby controlling the service robot to perform corresponding actions, and updating the service tasks based on the feedback from the service robot.

[0063] In this embodiment, taking the home-based elderly care scenario as an example, an ontology knowledge base is constructed based on the home-based elderly care scenario and the home environment.

[0064] Specifically:

[0065] Based on the OWL language and the Protégé ontology knowledge base visualization development software, an ontology knowledge base containing ontology information of users, environment, and service robots is constructed. The ontology knowledge base represents knowledge from three aspects: classes, instances, and attributes. A class represents a set of individuals with the same characteristics, an instance represents a specific individual, and attributes include object attributes and data attributes. Object attributes represent the relationships between classes and instances and are used for qualitative description, while data attributes represent the relationships between instances and numerical values ​​and are used for quantitative description.

[0066] The user information includes age, behavioral habits, etc.; the service robot body information includes robot appearance, functions, etc.; the environmental information is represented from multiple levels such as environment object, operating object, location, and scene structure. It mainly defines the relevant information of items in the scene, such as tables, chairs, cups, medicines, etc. that involve interaction in the scene. It is further subdivided according to different subordinate categories in each category, and then defined down to the smallest classification.

[0067] In this embodiment, the process of acquiring video streams, i.e. visual information, from the scene in real time using the binocular camera equipped on the service robot, and identifying user behavior information and scene relationship information between the user and the environment from the visual information includes:

[0068] The system acquires visual images of each frame of each video segment, identifies user targets using a trained object detection model, obtains user detection boxes in the scene, and then dynamically identifies user behavior based on the user detection boxes using a trained behavior recognition model, obtaining user behavior information labels in text form, represented as "person do".

[0069] Relationship detection: Based on the visual image of the first frame obtained from the video clip, the scene graph generation model trained is used to obtain the scene relationship information between the user and the environment. The scene graph generation model obtains text-based triple relationship pairs, represented in the form of "subject predicate object" ("subject + predicate + object"), and selects the top five triples with the highest confidence.

[0070] Subsequently, user behavior information labels in text form obtained from the behavior recognition model, and triplet relationship pairs in text form obtained from the scene graph generation model, are filtered to obtain visual modal information in text form.

[0071] Specifically:

[0072] The scene graph generation model generates a large number of triplet pairs based on scene images. However, common triples such as "person has hand" are not helpful for scene understanding and need to be removed as redundant information; while triples involving user-object interaction actions and positional relationships between users and objects are retained. Therefore, an information filtering strategy is designed, including:

[0073] (a) Retain the triplet with person as subject and the predicate involving action relationship, including predicates such as “sit on” and “carry”, and remove “has” and “wearing”; merge and retain one of the tags that are synonymous with the user behavior information tags, and retain the tags that are not synonymous with each other and connect them with “and”.

[0074] (b) Triples with person as subject and predicates involving positional relationships, including predicates such as “on”, “in front of”, etc.

[0075] (c) Retain the triplet of predicates with person as object and involving positional relationship, the predicates are the same as described in (b), and remove subordinate relationships such as "of" and "part of".

[0076] (d) Combine the verbs of behavior recognition with the predicates of (b) and (c) preposition attributes, taking "person" as the subject, to obtain recognition results in the form of "person lie near chair".

[0077] Visual modal information in text form is obtained based on information filtering strategies.

[0078] In this embodiment, based on real-time acquisition of voice information in scenarios such as smartphone microphone recording, and combined with a voice API, voice information recognition and processing are realized, including speech-to-text (ASR), Chinese-English translation, etc., to obtain voice modal information in text form.

[0079] In this embodiment, after acquiring scene knowledge information, the service robot system interacts with the ontology knowledge base based on the OwlReady2 interface. The ontology knowledge base is automatically queried and updated in real time using Python. The queried and updated content includes item information and user behavior information. Item information includes item location, whether it can be grasped, etc. The item information is used to help the system achieve target grasping of the service robot. The user behavior information in text form is obtained by querying the ontology knowledge base as scene knowledge modal information.

[0080] In this embodiment, text-based speech modal information, visual modal information, and scene knowledge modal information are combined to generate tagged scene multimodal fusion information. This scene multimodal fusion information is used as input to a large language model for training and testing. Feedback information from the service robot's sensors is added according to the actual application requirements of the scenario.

[0081] The meanings of scene multimodal fusion information labels are shown in Table 1, and examples of scene multimodal fusion information are shown in Table 2.

[0082] Table 1. Meaning of Multimodal Fusion Information Tags in a Scene

[0083]

[0084] Table 2 Examples of Multimodal Fusion Information in Scenarios

[0085]

[0086] In this embodiment, taking the home-based elderly care scenario as an example, a home-based elderly care scenario dataset is constructed, and a large language model is fine-tuned. Through the large language model, multimodal fusion information of the scenario in text form is processed. Relying on the text understanding and generation capabilities of the fine-tuned large language model, service tasks that can be parsed and executed by the service robot are generated. The generated service tasks are represented in text form as "atomic actions (interaction targets); atomic actions (interaction targets); ...". Atomic actions include navigation, grasping, speech, etc., and are stored in the ontology knowledge base.

[0087] Based on the above method, this embodiment is designed, tested, and its effects verified in the following aspects:

[0088] The advantage of an ontology knowledge base lies in its structured storage and management of scenario-specific knowledge, conveniently providing the system with the necessary scenario knowledge. The advantage of a large language model lies in its ability to process complex task data in complex multimodal scenarios, and its excellent text generation capabilities make service task reasoning possible. Therefore, combining an ontology knowledge base and a large language model, targeting specific scenarios, leveraging the general applicability of the large language model for task data processing to perform service task reasoning, and implementing task execution with the assistance of the ontology knowledge base, is a relatively ideal solution.

[0089] This embodiment constructs a family-assisted elderly care scenario dataset based on the family task dataset Activity Progress. The input is labeled text containing three modalities: voice, vision, and scenario knowledge. The output is a standardized service robot task plan, which is used for model fine-tuning.

[0090] To verify that multimodal fusion information and ontology knowledge can be used to reason about service tasks that better meet user expectations, this embodiment constructs three test sets based on a family-based elderly care scenario dataset. These sets contain voice information, voice + visual information, and voice + visual + scenario knowledge information, respectively. The test results are compared, and the results are evaluated based on NLP metrics Bleu and Rouge, as well as subjective metrics accuracy and usability. Tables 3 and 4 show the comparative experimental results. Accuracy indicates that the generated result is basically consistent with the labeled result; usability indicates that it is permissible to allow non-core steps (such as voice interaction) to be missing in the generated task plan, as long as the task can be executed to obtain the expected task result.

[0091] Table 3. Results of the Comparative Experiment on Service Task Reasoning

[0092]

[0093] Table 4. Results of the Comparative Experiment on Service Task Reasoning (Part 2)

[0094]

[0095]

[0096] As can be seen, in the designed home-based elderly care scenario, the decision-making accuracy of simple voice commands is low. However, because voice commands are more explicit, they can still accurately handle more than half of the task scenarios. After introducing visual modal information and ontology knowledge base information, the accuracy of the robot's reasoning for service tasks is improved. The best results are obtained when the voice information, visual information and knowledge base information in the scenario are combined.

[0097] Experimental verification:

[0098] This embodiment sets up a simulated family scenario, designs scenario tasks based on a dataset of family-based elderly assistance scenarios, and conducts system development and real-world testing based on a service robot hardware platform. The experimental scenario uses a family member experiencing a sudden illness requiring the robot's assistance as an example, and demonstrates the execution flow of a fall-dealing medication delivery scenario. The verification process and results are as follows:

[0099] I. System Framework and Execution Process

[0100] Voice interaction is based on the voice information acquisition mechanism. An Android human-computer interaction APP is developed and deployed, which calls the phone's microphone interface and built-in TTS API to realize text-to-speech. An Android command classification database is established based on Text-to-SQL. The voice commands that can be directly executed on the Android side are filtered by SQL fuzzy matching and handed over to the functional modules for execution. Natural language text that needs to be understood and reasoned is forwarded to the edge for information fusion.

[0101] Visual inspection, based on a deep learning-based visual inspection network, processes the video stream acquired by the binocular camera of the service robot to obtain visual semantic information.

[0102] Contextual knowledge information is provided through real-time queries of the ontology knowledge base based on the OwlReady2 API. This includes queries for user behavior and habit information required for reasoning, as well as contextual knowledge queries when the service robot performs its functions.

[0103] II. The topological structure of the ontology knowledge base for home-based elderly care scenarios is as follows: Figure 2 As shown. In a home-based service scenario based on an ontology knowledge base, the architecture is constructed from a "human-machine-environment" framework. Each category is further subdivided according to its different subordinate categories, ultimately defining the smallest possible classification. The class hierarchy is as follows: Figure 3 As shown. Object attributes mainly include the main positional relationships and action interaction relationships defined in the scene graph generation part of visual inspection. Data attributes include attributes that can quantitatively represent or do not point to specific classes or instances in the knowledge base. The lists of object attributes and data attributes are as follows. Figure 4 As shown. Based on the experimental scenario and following the OWL ontology knowledge base specification, some scenario knowledge was pre-written, including user information (mainly user behavior habits), item information, etc., to create instances and define attributes.

[0104] III. Scenario Design. Taking delivering medicine after a user falls as an example, the process is as follows: Figure 5 As shown, the service robot detects that a user has fallen to the ground, approaches to inquire, and if the user is lying unconscious and unresponsive, it executes the emergency treatment procedure; if the user is sitting on the ground and requests the service robot to deliver medicine, the service robot locates the emergency medicine, navigates to the target point, grabs and delivers the medicine to the user.

[0105] IV. The testing process includes three stages: detection, information fusion and instruction generation, and execution. The detection stage includes the acquisition of voice and visual information, and the processing to obtain semantic information. The information fusion and instruction generation stage includes querying user behavior habits based on the ontology knowledge base, filtering visual and semantic information, fusing and generating large language models to input instructions, and reasoning to generate a task plan (medicine delivery). The execution stage includes the service robot parsing the task plan, querying the location information of interactive items in the ontology knowledge base, and executing service tasks, including interaction, navigation, and grasping.

[0106] The testing process includes: The service robot detects a user lying on the ground, filters visual and semantic information, queries the ontology knowledge base to determine the user has no current behavioral habits, and uses a large language model to infer the service task: move to the user's side and ask for assistance. If the user does not respond, the robot infers again to execute the emergency handling procedure and call for emergency medical services. If the user is sitting on the ground, the robot recognizes the user's voice saying "Give me the medicine," queries the ontology knowledge base to determine the user has no current behavioral habits, and uses a large language model to infer the service task "Give the user emergency medication," which is then parsed and executed. The robot finds the location of the medication on the table and its corresponding object detection tag; it navigates to the target location, uses an object detection network to detect and grasp the target, simultaneously grasping both the water bottle and the emergency medication. After grasping the medication, the robot navigates to the user, hands the medication to the user, and completes the service task.

[0107] In summary, the robot task reasoning method integrating multimodal information and ontology knowledge proposed in this embodiment optimizes the decision-making process of service robots. The service robot effectively integrates multimodal information and ontology knowledge to achieve service task reasoning that better meets user expectations. A service robot system was designed and integrated, and its feasibility in a home setting was tested. Through this system, the intelligence level of home service robots is improved. Service robots can better understand user intentions and possess the ability to execute expected service functions and manage scene knowledge information, thus contributing to the integration of service robots into homes.

[0108] Example 2

[0109] This embodiment provides a robot task reasoning system that integrates multimodal information and ontology knowledge, including:

[0110] The knowledge base construction module is configured to build an ontology knowledge base containing ontology information of users, environment, and service robots.

[0111] The acquisition module is configured to acquire voice information, visual information, and scene knowledge information in the scene.

[0112] The visual processing module is configured to identify user behavior information and scene relationship information between the user and the environment based on visual information, and then obtain visual modal information in text form through information filtering; wherein, information filtering includes retaining scene relationship information in which the user and the environment have interactive relationships based on user behavior information;

[0113] The speech processing module is configured to recognize speech modal information in text form based on speech information;

[0114] The scene knowledge processing module is configured to obtain scene knowledge modal information in text form by querying the ontology knowledge base based on scene knowledge information.

[0115] The fusion module is configured to obtain scene multimodal fusion information based on text-based speech modal information, visual modal information, and scene knowledge modal information;

[0116] The task generation module is configured to generate service tasks based on multimodal fusion information of the scene and a trained large language model, thereby controlling the service robot to perform corresponding actions and updating the service tasks based on the feedback from the service robot.

[0117] It should be noted that the above modules correspond to the steps described in Embodiment 1, and the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules, as part of the system, can be executed in a computer system such as a set of computer-executable instructions.

[0118] In further embodiments, the following is also provided:

[0119] A service robot system, comprising:

[0120] The robot itself is a two-handed, humanoid wheeled service robot with mapping, navigation, and grasping capabilities, and is equipped with a binocular depth camera to acquire visual image information.

[0121] The edge computing platform is used to identify user behavior information and scene relationship information between users and the environment based on visual information, and then obtain visual modal information in text form through information filtering. The information filtering includes retaining scene relationship information with interactive relationships between users and the environment based on user behavior information, and obtaining scene knowledge modal information in text form by querying the ontology knowledge base based on scene knowledge information.

[0122] Among them, an edge computing smart box is used to deploy robot control algorithms, visual inspection models, ontology knowledge bases and their interfaces, and chassis control and communication are realized based on wireless networks.

[0123] The human-computer interaction terminal is used to acquire voice information in the scene and to recognize the voice modal information in text form based on the voice information;

[0124] Specifically, a smartphone is used and carried by a service robot to deploy a human-computer interaction APP. It combines a voice API to interact with users, including obtaining voice information and converting it into text (ASR), converting the generated interactive text into speech playback (TTS), and performing functions such as making phone calls, recording videos, and voice interaction. It communicates with the edge based on a TCP long connection.

[0125] The cloud computing platform uses desktop servers and is based on the Flask framework to communicate with edge devices. It is used to deploy large language models, receive text-based speech modal information, visual modal information, and scene knowledge modal information, and fuse them to obtain scene multimodal fusion information. Based on the scene multimodal fusion information, the trained large language model generates service tasks, thereby controlling the service robot to perform corresponding actions, and updating the service tasks based on the feedback from the service robot.

[0126] In further embodiments, the following is also provided:

[0127] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0128] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0129] Memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of memory may also include non-volatile random access memory. For example, memory may also store information about the device type.

[0130] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in Embodiment 1.

[0131] The method in Example 1 can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0132] A computer program product includes a computer program that, when executed by a processor, implements the method described in Embodiment 1.

[0133] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0134] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0135] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0136] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0137] While the specific embodiments of the present invention have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A robot task reasoning method integrating multimodal information and ontology knowledge, characterized in that, include: Construct an ontology knowledge base containing information on users, environment, and service robots. In the ontology knowledge base, user information includes age and behavioral habits; service robot ontology information includes robot appearance and functions; and environmental information includes environmental objects, operational objects, location, and scene structure, which are used to define relevant information about items in the scene. The knowledge base is further subdivided according to different subordinate categories in each category, and then defined down to the smallest classification. Acquire audio information, visual information, and scene knowledge information from the scene; User behavior information and scene relationship information between users and the environment are obtained based on visual information recognition, and then visual modal information in text form is obtained through information filtering; wherein, information filtering includes retaining scene relationship information in which users interact with the environment based on user behavior information; The scene relationship information between the user and the environment is represented by textual triplet relation pairs, in the form of subject + predicate + object. The process of obtaining textual visual modal information through information filtering includes: (1) Retain the triplet of predicates that are user-subject and involve action relationships. Among them, tags that are synonymous with user behavior information tags are merged and one is retained, while tags that are not synonymous are retained and connected at the same time. (2) Preserve triples of predicates that are user-subject and involve positional relationships; (3) Preserve triples of predicates that have the user as the object and involve positional relationships; (4) Taking the user as the subject, merge the predicates of the behavior recognition verbs and the preposition attributes of (2) and (3); Speech modality information in text form is obtained from speech information recognition; Based on the scene knowledge information, the scene knowledge modal information in text form is obtained by querying the ontology knowledge base; among them, the item information and user behavior habit information in the ontology knowledge base are queried. The item information includes the item location and whether it can be grasped, which is used to assist the service robot in target grasping. The user behavior habit information in text form is used as the scene knowledge modal information. Based on the speech modal information, visual modal information, and scene knowledge modal information in text form, scene multimodal fusion information is obtained; Based on the multimodal fusion information of the scene, a trained large language model is used to generate service tasks, thereby controlling the service robot to perform corresponding actions, and updating the service tasks based on the feedback from the service robot.

2. The robot task reasoning method integrating multimodal information and ontology knowledge as described in claim 1, characterized in that, The ontology knowledge base includes classes, instances, and attributes. A class represents a collection of individuals with the same characteristics, an instance represents an individual, and attributes include object attributes and data attributes. Object attributes represent the relationship between classes and instances and are used for qualitative description, while data attributes represent the relationship between instances and numerical values ​​and are used for quantitative description.

3. A robot task reasoning system integrating multimodal information and ontology knowledge, characterized in that, The robot task reasoning method for performing the fusion of multimodal information and ontology knowledge as described in any one of claims 1-2 includes: The knowledge base construction module is configured to build an ontology knowledge base containing ontology information of users, environment, and service robots. The acquisition module is configured to acquire voice information, visual information, and scene knowledge information in the scene. The visual processing module is configured to identify user behavior information and scene relationship information between the user and the environment based on visual information, and then obtain visual modal information in text form through information filtering; wherein, information filtering includes retaining scene relationship information in which the user and the environment have interactive relationships based on user behavior information; The speech processing module is configured to recognize speech modal information in text form based on speech information; The scene knowledge processing module is configured to obtain scene knowledge modal information in text form by querying the ontology knowledge base based on scene knowledge information. The fusion module is configured to obtain scene multimodal fusion information based on text-based speech modal information, visual modal information, and scene knowledge modal information; The task generation module is configured to generate service tasks based on multimodal fusion information of the scene and a trained large language model, thereby controlling the service robot to perform corresponding actions and updating the service tasks based on the feedback from the service robot.

4. A service robot system, characterized in that, The robot task reasoning method for performing the fusion of multimodal information and ontology knowledge as described in any one of claims 1-2 includes: The robot itself is equipped with a binocular depth camera for acquiring visual information; The edge computing platform is used to identify user behavior information and scene relationship information between users and the environment based on visual information, and then obtain visual modal information in text form through information filtering. The information filtering includes retaining scene relationship information with interactive relationships between users and the environment based on user behavior information; and obtaining scene knowledge modal information in text form by querying the ontology knowledge base based on scene knowledge information. The human-computer interaction terminal is used to acquire voice information in the scene and recognize the voice modal information in text form based on the voice information; The cloud computing platform receives text-based speech modal information, visual modal information, and scene knowledge modal information, and fuses them to obtain scene multimodal fusion information. Based on the scene multimodal fusion information, a trained large language model is used to generate service tasks, thereby controlling the service robot to perform corresponding actions, and updating the service tasks based on the feedback from the service robot.

5. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method according to any one of claims 1-2.

6. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-2.

7. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-2.

Images