Automatic construction method for scene knowledge graph of underwater robot

Abstract

The invention provides an underwater robot scene knowledge graph automatic construction method, and relates to the technical field of underwater robots, and the method comprises the following steps: introducing multi-modal scene side writing, appointing a specific underwater robot operation scene S, and obtaining n sections of multi-modal scene side writing sets Ws = {w0, w1,..., wn} about the scene S through keyword and multi-modal retrieval; based on a set Ws = {w0, w1,..., wn}, scene concept mining and ontology design are carried out, and a scene-driven underwater body knowledge graph schema C is obtained; the knowledge type of the underwater robot body knowledge graph is decoupled into general knowledge kg and scene knowledge ks; according to the method, the characteristic that the robot has an intelligent dependence scene is effectively matched, so that the construction of the robot knowledge graph becomes efficient, flexible and low-cost.

Description

Technical Field

[0001] The present invention relates to the technical field of underwater robots, and in particular to a method for automatically constructing a scene knowledge graph of an underwater robot. Background Art

[0002] The schema in the existing (multimodal) knowledge graph construction is mainly realized through manual construction, automatic construction and semi-automatic construction. Manual construction is the construction of knowledge graphs by domain experts based on the knowledge of specific fields. Its advantages are high accuracy and applicability. Automated construction automatically generates schemas by analyzing large-scale data or existing knowledge graphs, such as statistical analysis and clustering, natural language processing to analyze named entities and syntactic structures in structured texts, and automatic generation of implicit embedding space representations based on pre-trained models such as BERT and GPT. Semi-automatic construction methods are a combination of automatic schema construction methods and domain expert screening.

[0003] For robot-related domain knowledge graphs, the main focus is on home service scenarios, mainly involving common sense, environmental entities, and basic skill knowledge. Representative examples include RoboEarth, Robobrain, Open-EASE, etc. In the existing multimodal knowledge graph construction process, schema construction is one of the most important links affecting the quality of the graph. The manual schema construction method consumes a lot of manpower and time; the automated method requires high-quality data, efficient analysis tools or pre-trained models; the semi-automatic method inherits the advantages of the first two to ensure graph quality, but is also limited by their shortcomings. In addition, for the knowledge graph centered on underwater robots, due to the difficulty of reaching the working environment of special robots, the currently disclosed multimodal knowledge graphs and efficient construction methods are in a blank state. Therefore, the present invention proposes an automated construction method for underwater robot scene knowledge graphs to solve the problems existing in the prior art. Summary of the invention

[0004] In view of the above problems, the present invention proposes an automatic construction method of underwater robot scene knowledge graph. The automatic construction method of underwater robot scene knowledge graph is based on scene drive, which effectively matches the characteristics of robot embodied intelligence dependence on scenes, making the robot knowledge graph construction efficient, flexible and low-cost.

[0005] To achieve the purpose of the present invention, the present invention is implemented by the following technical solution: a method for automatically constructing a scene knowledge graph of an underwater robot, comprising the following steps:

[0006] S1: Introduce multimodal scene profiles, specify a specific underwater robot operation scene S, and obtain a set of n multimodal scene profiles W about scene S through keyword and multimodal retrieval.s ={w 0 ,w 1 ,...,wn};

[0007] S2: Based on set W s ={w 0 ,w 1 ,...,w n}, conduct scenario concept mining and ontology design to obtain the scenario-driven underwater embodied knowledge graph schema C;

[0008] S3: Decoupling the knowledge types of the underwater robot embodied knowledge graph into general knowledge κ g and scene knowledgeκ s ,Based on the constructed schema, the existing general knowledge base and scenario-related data are associated and filled with knowledge respectively;

[0009] S4: Evaluate knowledge completeness and consistency and eliminate long tails based on scenario profiling.

[0010] A further improvement is that: S1 comprises the following steps:

[0011] For underwater robot operation scenes, multimodal scene profiling is introduced, including underwater videos, underwater pictures, and underwater text description materials;

[0012] Specify a specific underwater robot operation scenario S, and perform keyword and multimodal retrieval on open world OW Internet data, publicly published data sets, and user private vertical domain data;

[0013] Get a set of n multimodal scene profiles W about the user-specified scene S s ={w 0 ,w 1 ,...,w n}.

[0014] A further improvement is that: S2 comprises the following steps:

[0015] Scene concept mining, using a large language model to generate a list of questions Q about scene information, using a multimodal large model through prompt engineering technology f prompt (·) Retrieve the scene concepts involved in the collected multimodal scene profiles to obtain a preliminary scene concept set C raw =f promot (W s ,S,Q), the scene concept contains entity or abstract node types and attribute relationships;

[0016] Ontology design, for the initially extracted scene concept set C rawThrough the general semantic knowledge base κ, we can query hyponyms and hyponyms to achieve concept expansion C expand , mathematically expressed as follows:

[0017]

[0018] For the expanded scene concept set, the semantic similarity between each concept is calculated to perform concept aggregation and obtain the aggregated scene concept set C as the schema of the final underwater scene embodied knowledge graph.

[0019] A further improvement is that in S3, the general knowledge κ g is the common sense existing in various scenarios; scenario knowledgeκ s Assign scenario-specific knowledge to users.

[0020] A further improvement is that: S3 comprises the following steps:

[0021] Based on the underwater robot scene-related schema designed by S2, the general knowledge related to the scene is retrieved and collected in the public general knowledge base. g , realize the general knowledge filling of underwater scene knowledge graph fil (·):

[0022] κ underwater-KG =f fill (C,κ g ).

[0023] Based on the underwater robot scene-related schema designed by S2, semantic retrieval is performed in public text encyclopedia databases, visual datasets, video datasets, and user private vertical domain data to obtain scene knowledge κ s , and fill in and update knowledge;

[0024] For the acquired scene knowledge κ s Compared with the constructed κ containing only general knowledge underwater-KG Conflict detection is performed. When there is no conflict, the scene knowledge is directly used for knowledge filling; when there is a conflict, the scene knowledge κ is used s Replace general knowledge κ g , to achieve knowledge update:

[0025] κ underwater-KG =f fill (C,deconfict(κ underwater-KG ,κ s )).

[0026] A further improvement is that: said S4 comprises the following steps:

[0027] The scene concept c of each scene profile obtained by S1raw_n ∈C raw As query, the underwater robot multimodal knowledge graph κ constructed from S3 underwater-KG For the retrieved multimodal knowledge κ query Check the integrity and consistency respectively;

[0028] For attribute terms whose attribute frequency is less than the set threshold, we first perform attribute hierarchical decomposition to remove attribute modifiers and obtain an indivisible and unambiguous atomic attribute set A. atomic , then, for the atomic attribute set A atomic , attributes with semantic similarity higher than the threshold γ are merged to alleviate the long-tail problem of the constructed knowledge graph. The specific implementation is as follows:

[0029]

[0030] Further improvements are: for completeness, the retrieved knowledge κ is improved through multimodal large models and hint engineering query The scene relevance is judged, and the scene irrelevant concepts are deleted. At the same time, the success rate of multimodal knowledge retrieval is counted. For the scene concepts that cannot retrieve multimodal knowledge, raw_n , conduct secondary multimodal knowledge injection;

[0031] For consistency, based on multimodal large models and hint engineering techniques, the retrieved knowledge κ query With scene profile n ∈W s Conduct logical common sense tests to check for contradictory knowledge query , from κ underwater-KG Clean and delete.

[0032] The beneficial effects of the present invention are:

[0033] 1. The present invention combines underwater robot operation scenarios to perform multimodal scenario profiling. By using scenario profiling and designed prompt questions, the prompt questions are input into the existing multimodal large model to extract the embodied scenario knowledge of the underwater robot. Based on other related common sense encyclopedia knowledge bases, concept expansion and concept merging are performed through RAG to finally obtain a high-precision underwater robot embodied knowledge graph. The constructed schema is used to construct the graph for mixed data sources such as Internet data, domain public data, and active collection of scenario data. For the constructed graph, prompt engineering and large model technology are used to evaluate and control the quality according to the multimodal scenario profiling. The integrity, consistency, and long-tail problems of the constructed graph are detected, and the graph knowledge is updated to form a closed loop. The characteristics of the robot's embodied intelligence dependence on the scenario are effectively matched, making the robot knowledge graph construction efficient, flexible, and low-cost.

[0034] 2. The present invention implements a closed-loop quality control mechanism for the scene-driven robot embodied knowledge graph, proposes a prompt engineering + large model technology based on scene profiling content, directly realizes closed-loop graph scene knowledge quality inspection and update from the scene perspective, and fills the technical gap in the construction of underwater special robot knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0036] In order to deepen the understanding of the present invention, the present invention will be further described in detail below in conjunction with examples. The examples are only used to explain the present invention and do not constitute a limitation on the protection scope of the present invention.

[0037] Embodiment 1

[0038] according to Figure 1 As shown, this embodiment proposes a method for automatically constructing a scene knowledge graph of an underwater robot, comprising the following steps:

[0039] S1: Introduce multimodal scene profiles, specify a specific underwater robot operation scene S, and obtain a set of n multimodal scene profiles W about scene S through keyword and multimodal retrieval. s ={w 0 ,w 1 ,...,w n};

[0040] S2: Based on set W s ={w 0 ,w 1 ,...,w n}, conduct scenario concept mining and ontology design to obtain the scenario-driven underwater embodied knowledge graph schema C;

[0041] S3: Decoupling the knowledge types of the underwater robot embodied knowledge graph into general knowledge κ g and scene knowledgeκ s ,Based on the constructed schema, the existing general knowledge base and scenario-related data are associated and filled with knowledge respectively;

[0042] S4: Evaluate knowledge completeness and consistency and eliminate long tails based on scenario profiling.

[0043] Multimodal Knowledge Graph (MMKG) is a structured multimodal semantic knowledge base that integrates data from multiple modalities (such as text, vision, video, acoustics, touch, etc.) into a knowledge graph. This knowledge graph not only contains the entity-relationship information described by the traditional knowledge graph, but also contains the multimodal semantics and spatiotemporal attribute information of each entity, so that it can describe the real world more comprehensively and on-the-spot.

[0044] Schema generation technology is mainly used to define the structure of the knowledge graph, including entity types, relationship types, relationship types and their constraints, etc. Schema is the skeleton of the knowledge graph, which determines the organization of data and semantic associations. In the process of building a knowledge graph, schema acquisition mainly includes manual construction, automatic generation, and semi-automatic generation technology.

[0045] Underwater Embodied Intelligence is a robotic intelligence technology that can sense, learn, make decisions and act in underwater environments. The underwater environment is significantly different from the terrestrial environment, and the embodied intelligence system needs to be able to adapt to complex underwater conditions, such as high pressure, low light, strong corrosiveness, and highly dynamic fluid environments, so as to achieve underwater environmental perception, underwater scene understanding, and underwater cognitive decision-making.

[0046] In the present invention, it is necessary to solve the problem of efficient and high-quality automatic schema construction method for underwater special robot scenes, and propose a solution to the quality control of the graph in the process of building the robot field knowledge graph. Specifically, in combination with the underwater robot operation scene, a multimodal scene profile (such as picture + text, video + text, etc.) is proposed, and the scene profile and the designed prompt questions are input into the existing multimodal large model (such as GPT4, Video-LLaVA) to extract the underwater robot embodied scene knowledge. Based on other related common sense encyclopedia knowledge bases, concept expansion and concept merging are performed through RAG, and finally a high-precision underwater robot embodied knowledge graph is obtained. For mixed data sources such as Internet data, field public data, and active collection of scene data, the constructed schema is used to construct the graph. For the constructed graph, the prompt engineering and large model technology are used for quality assessment and control according to the multimodal scene profile, and the integrity, consistency, and long-tail problems of the constructed graph are detected and the graph knowledge is updated to form a closed loop.

[0047] Embodiment 2

[0048] according to Figure 1 As shown, this embodiment proposes a method for automatically constructing a scene knowledge graph of an underwater robot, comprising the following steps:

[0049] Multimodal scene profiling acquisition

[0050] For underwater robot operation scenes, a multimodal scene profile is introduced, which includes underwater videos, underwater pictures, and underwater text description materials. The user specifies a specific underwater robot operation scene S, and retrieves the open world OW Internet data, publicly published data sets, and user private vertical domain data by keywords, multimodal methods, etc., to obtain a set of n multimodal scene profiles W about the user-specified scene S. s ={w 0 ,w 1 ,...,w n}

[0051] Scenario schema design

[0052] Based on the collected multimodal underwater scene profile set W s ={w 0 ,w 1 ,...,w n}, an automated schema design scheme is proposed to obtain the scenario-driven underwater embodied knowledge graph schema C, which mainly includes the following two parts:

[0053] Scene concept mining. First, use the existing large language model to generate a list of questions Q about scene information (called scene prompt in this case); second, use the existing multimodal large model (such as GPT4o, Video-LLaVA, etc.) through prompt engineering technology f prompt (·) Retrieve the scene concepts involved in the collected multimodal scene profiles to obtain a preliminary scene concept set C raw =f promot (W s ,S,Q), the scene concept contains entities or abstract node types and attribute relationships.

[0054] Ontology design. First, for the initially extracted scene concept set C raw Concept expansion C is achieved by querying hypernyms and hyponyms through existing general semantic knowledge bases κ (such as Probase, WordNet) expand , mathematically expressed as follows:

[0055]

[0056] Secondly, for the expanded scene concept set, the semantic similarity between each concept is calculated to perform concept aggregation and obtain the aggregated scene concept set C as the schema of the final underwater scene embodied knowledge graph.

[0057] Construction of multimodal knowledge graph for underwater scenes

[0058] Decouple the knowledge types of the underwater robot embodied knowledge graph into general knowledge κ g and scene knowledgeκ s The former is commonly found in various scenarios, such as common sense; the latter is knowledge unique to user-specified scenarios. Therefore, based on the constructed schema, the existing general knowledge base and scenario-related data are associated and filled with knowledge respectively.

[0059] General knowledge. Based on the designed underwater robot scene-related schema, the scene-related general knowledge κ is retrieved and collected from the existing public general knowledge base. g , realize the general knowledge filling of underwater scene knowledge graph fil (·):

[0060] κ underwater-KG =f fill (C,κ g ).

[0061] Scene knowledge. Based on the designed underwater robot scene-related schema, semantic retrieval is performed in the existing public text encyclopedia database, visual dataset, video dataset, and user private vertical domain data to obtain scene knowledgeκ s , and perform knowledge filling and knowledge updating. For the acquired scene knowledge κ s Compared with the constructed κ containing only general knowledge underwater-KG Conflict detection is performed. When there is no conflict, the scene knowledge is directly used for knowledge filling; when there is a conflict, the scene knowledge κ is used s Replace general knowledge κ g , to achieve knowledge update:

[0062] κ underwater-KG =f fill (C,deconfict(κ underwater-KG ,κ s )).

[0063] Quality control and knowledge maintenance

[0064] A knowledge completeness, consistency assessment and long-tail elimination method based on scenario profiling is proposed.

[0065] Completeness and consistency check. The scenario concept c of each scenario profile obtained in the first part raw_n ∈C raw As a query, from the constructed underwater robot multimodal knowledge graph κ underwater-KG For the retrieved multimodal knowledge κ queryCheck the integrity and consistency respectively:

[0066] For completeness, through existing multimodal large models (such as GPT4, etc.), the retrieved knowledge κ is trained by hint engineering query , judge the scene relevance, delete the scene irrelevant concepts, and count the retrieval success rate of multimodal knowledge. For the scene concepts c that cannot retrieve multimodal knowledge raw_n , conduct secondary multimodal knowledge injection;

[0067] For consistency, based on the existing multimodal large model and prompt engineering technology, the retrieved knowledge κ query With scene profile n ∈W s Conduct logical common sense tests to check for contradictory knowledge query , from κ underwater-KG Clean and delete.

[0068] For the underwater robot multimodal knowledge graph κ underwater-KG To solve the long-tail problem, we propose that for attribute terms with an attribute frequency less than a set threshold, we first perform a hierarchical decomposition of the attributes to remove attribute modifiers, and obtain an indivisible and unambiguous atomic attribute set A. atomic ; Next, for the atomic attribute set A atomic , attributes with semantic similarity higher than the threshold γ are merged, thereby alleviating the long-tail problem of the knowledge graph constructed based on this method. The specific implementation is as follows:

[0069]

[0070] The method for automatically constructing the underwater robot scene knowledge graph combines the underwater robot operation scene to perform multimodal scene profiling. The scene profiling and designed prompt questions are input into the existing multimodal large model to extract the embodied scene knowledge of the underwater robot. Based on other related common sense encyclopedia knowledge bases, concept expansion and concept merging are performed through RAG to finally obtain a high-precision underwater robot embodied knowledge graph. The graph is constructed using the constructed schema for mixed data sources such as Internet data, domain public data, and scene data actively collected. For the constructed graph, prompt engineering and large model technology are used to evaluate and control the quality according to the multimodal scene profiling. The integrity, consistency, and long-tail problems of the constructed graph are detected and the graph knowledge is updated to form a closed loop. The characteristics of the robot's embodied intelligence dependence on the scene are effectively matched, making the construction of the robot knowledge graph efficient, flexible, and low-cost. At the same time, the present invention implements a closed-loop quality control mechanism for the scene-driven robot embodied knowledge graph. Compared with the traditional graph performance evaluation from aspects such as node scale, number of relationships (edges) and downstream tasks, the present invention proposes a prompt engineering + large model technology based on scene profiling content, which directly realizes the closed-loop graph scene knowledge quality inspection and update from the scene perspective, and fills the technical gap in the construction of knowledge graphs for underwater special robots.

[0071] The above shows and describes the basic principles, main features and advantages of the present invention. It should be understood by those skilled in the art that the present invention is not limited to the above embodiments. The above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, which fall within the scope of the present invention. The scope of protection of the present invention is defined by the attached claims and their equivalents.

Claims

1. A method for automatically constructing a scene knowledge graph of an underwater robot, characterized in that: The following steps are involved: S1: Introduce multimodal scene profiles, specify a specific underwater robot operation scene S, and obtain a set of n multimodal scene profiles W about scene S through keyword and multimodal retrieval. s ={w0,w1,...,w n }; S2: Based on set W s ={w0,w1,...,w n }, conduct scenario concept mining and ontology design to obtain the scenario-driven underwater embodied knowledge graph schema C; S3: Decoupling the knowledge types of the underwater robot embodied knowledge graph into general knowledge κ g and scene knowledgeκ s ,Based on the constructed schema, the existing general knowledge base and scenario-related data are associated and filled with knowledge respectively; S4: Evaluate knowledge completeness and consistency and eliminate long tails based on scenario profiling.

2. The method for automatically constructing a scene knowledge graph of an underwater robot according to claim 1, characterized in that: The S1 comprises the following steps: For underwater robot operation scenarios, multimodal scene profiling is introduced, including underwater videos, underwater pictures, and underwater text description materials; Specify a specific underwater robot operation scenario S, and perform keyword and multimodal retrieval on open world OW Internet data, publicly published data sets, and user private vertical domain data; Get a set of n multimodal scene profiles W about the user-specified scene S s ={w0,w1,...,w n }.

3. The method for automatically constructing a scene knowledge graph of an underwater robot according to claim 1, characterized in that: The S2 comprises the following steps: Scene concept mining, using a large language model to generate a list of questions Q about scene information, using a multimodal large model through prompt engineering technology f prompt (·) Retrieve the scene concepts involved in the collected multimodal scene profiles to obtain a preliminary scene concept set C raw =f promot (W s ,S,Q), the scene concept contains entity or abstract node types and attribute relationships; Ontology design, for the initially extracted scene concept set C raw Through the general semantic knowledge base κ, we can query hyponyms and hyponyms to achieve concept expansion C expand , mathematically expressed as follows: For the expanded scene concept set, the semantic similarity between each concept is calculated to perform concept aggregation and obtain the aggregated scene concept set C as the schema of the final underwater scene embodied knowledge graph.

4. The method for automatically constructing a scene knowledge graph of an underwater robot according to claim 3, characterized in that: In S3, the general knowledge κ g is the common sense existing in various scenarios; scenario knowledgeκ s Assign scenario-specific knowledge to users.

5. The method for automatically constructing a scene knowledge graph of an underwater robot according to claim 4, characterized in that: The S3 comprises the following steps: Based on the underwater robot scene-related schema designed by S2, the general knowledge related to the scene is retrieved and collected in the public general knowledge base. g , realize the general knowledge filling of underwater scene knowledge graph fil (·): k underwater-KG =f fill (C,k g ). Based on the underwater robot scene-related schema designed by S2, semantic retrieval is performed in public text encyclopedia databases, visual datasets, video datasets, and user private vertical domain data to obtain scene knowledge κ s , and fill in and update knowledge; For the acquired scene knowledge κ s Compared with the constructed κ containing only general knowledge underwater-KG Conflict detection is performed. When there is no conflict, the scene knowledge is directly used for knowledge filling; when there is a conflict, the scene knowledge κ is used s Replace general knowledge κ g , to achieve knowledge update: k underwater-KG =f fill (C,deconfict(k underwater-KG ,k s )).。 6. The method for automatically constructing a scene knowledge graph of an underwater robot according to claim 5, characterized in that: The S4 comprises the following steps: The scene concept c of each scene profile obtained by S1 raw_n ∈C raw As query, the underwater robot multimodal knowledge graph κ constructed from S3 underwater-KG For the retrieved multimodal knowledge κ query Check the integrity and consistency respectively; For attribute terms whose attribute frequency is less than the set threshold, we first perform attribute hierarchical decomposition to remove attribute modifiers and obtain an indivisible and unambiguous atomic attribute set A. atomic , then, for the atomic attribute set A atomic , attributes with semantic similarity higher than the threshold γ are merged to alleviate the long-tail problem of the constructed knowledge graph. The specific implementation is as follows:

7. The method for automatically constructing a scene knowledge graph of an underwater robot according to claim 6, characterized in that: For completeness, the retrieved knowledge κ is optimized through multimodal large models and hint engineering query The scene relevance is judged, and the scene irrelevant concepts are deleted. At the same time, the success rate of multimodal knowledge retrieval is counted. For the scene concepts that cannot retrieve multimodal knowledge, raw_n , conduct secondary multimodal knowledge injection; For consistency, based on multimodal large models and hint engineering techniques, the retrieved knowledge κ query With scene profile n ∈W s Conduct logical common sense tests to check for contradictory knowledge query , from κ underwater-KG Clean and delete.

Images