Operator arrangement and intelligent annotation-based vertical domain knowledge graph reasoning method

Abstract

The application discloses a vertical field knowledge graph reasoning method based on operator arrangement and intelligent labeling, relates to the technical field of data processing, and comprises the following steps: constructing a field business knowledge model based on a business process; configuring a data introduction and preprocessing process through a visual process arrangement platform, collecting multi-modal data, processing the multi-modal data according to the configured process, and storing the multi-modal data into field knowledge through the field business knowledge model; calling the field knowledge, identifying and labeling text data based on a Qwen large language model, and detecting and labeling image and video frames based on a YOLO model to extract multi-modal knowledge elements; performing entity alignment, entity linking and event fusion processing on the multi-modal knowledge elements, combining rules and a BERT model to realize multi-modal knowledge fusion; and storing the fused knowledge in a form of triplets into a graph database and dynamically rendering and interactively displaying the knowledge through a visual component. The application breaks through the technical bottlenecks of traditional schemes in the aspects of heterogeneous data integration, cross-modal association and field adaptation.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, device, and medium for reasoning about vertical domain knowledge graphs based on operator arrangement and intelligent annotation. Background Technology

[0002] Knowledge graphs are structured semantic knowledge bases used to describe concepts and their relationships in the physical world in symbolic form. Essentially, they construct network knowledge structures through "entity-relationship-entity" triples, providing structured knowledge to users in a graphical form, thus realizing the transformation of the Web from webpage links to concept links.

[0003] The logical structure of a knowledge graph consists of a data layer and a schema layer. In the data layer, knowledge is stored in the graph database in units of facts. The schema layer, located above the data layer, is the core of the knowledge graph and stores refined knowledge.

[0004] Knowledge graphs can be constructed in two ways: top-down and bottom-up. Most current knowledge graphs are constructed using the bottom-up approach. The construction process is an iterative update process, which involves starting from the original data and using a series of automatic or semi-automatic technical means to extract knowledge elements (i.e., facts) from the original data. After processing such as fusion and disambiguation, these elements are stored in the knowledge base data layer and schema layer.

[0005] In the construction of knowledge graphs in vertical domains, existing technologies include methods based on large language models. These approaches first standardize multimodal raw data such as text, images, and videos. Then, they extract feature representations from various data types using word vector models and convolutional neural networks (CNNs). Finally, cross-modal dynamic fusion techniques are used to concatenate or jointly encode features from different modalities, forming a unified multimodal feature representation. This representation is then input into a finely tuned large language model to identify and extract entities and relationships. Ultimately, it is stored in a graph database as triples, completing the initial construction of the domain knowledge graph. While these methods enhance the cross-modal information association capabilities to some extent and optimize some fusion effects with the help of large models, they still have the following limitations:

[0006] First, when processing multimodal data, the approach of "word vector model + convolutional neural network + keyframe feature fusion" is used to extract multimodal data features and realize the idea of ​​customized design in the field of science and technology innovation. However, it is difficult to adapt to the multimodal data types in other fields. For example, some professional fields involve non-standard image formats, tabular data with specific structures, or field-specific symbol systems. As a result, the fragmentation problem of multimodal data processing has not been fundamentally solved, and the cost of cross-domain reuse is high.

[0007] Second, using only large models for entity-relationship extraction and fusion ignores the special terminology and logical relationships of vertical domains, which can easily lead to biases due to the probabilistic output of large models. The robustness of data fusion is weak, and domain knowledge is not fully utilized. Summary of the Invention

[0008] This invention provides a vertical domain knowledge graph reasoning method, device, equipment, and medium based on operator orchestration and intelligent annotation, which solves the problems of insufficient multimodal adaptation capability and insufficient utilization of domain knowledge faced by knowledge graph construction methods based on large models in vertical domain applications.

[0009] To achieve the above objectives, this application adopts the following technical solution:

[0010] Firstly, a vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation is provided, including:

[0011] A domain business knowledge model is built based on business processes to define and store entities and relationships between entities;

[0012] Through a visual process orchestration platform, configure data acquisition and preprocessing processes, collect multimodal data, process the multimodal data according to the configured process, and store it as domain knowledge through the domain business knowledge model;

[0013] The domain knowledge is invoked to identify and annotate text data based on the fine-tuned Qwen large language model, and to detect and annotate targets in images and video frames based on the fine-tuned YOLO model, thereby extracting multimodal knowledge elements.

[0014] The multimodal knowledge elements are processed by entity alignment, entity linking and event fusion, and multimodal knowledge fusion is achieved by combining rules with a pre-trained BERT model;

[0015] The merged knowledge is stored in a graph database in the form of <entity-relationship-entity> triples, and the node-edge relationships are dynamically rendered and interactively displayed through a visualization component.

[0016] Secondly, a vertical domain knowledge graph reasoning device based on operator orchestration and intelligent annotation is provided, including:

[0017] The domain business knowledge model module is used to build a domain business knowledge model based on business processes, and to define and store entities and the relationships between entities;

[0018] The visual process orchestration module is used to configure data acquisition and preprocessing processes through the visual process orchestration platform, collect multimodal data, process the multimodal data according to the configured process, and store it as domain knowledge through the domain business knowledge model.

[0019] The knowledge element extraction module is used to call the domain knowledge, recognize and annotate text data based on the fine-tuned Qwen large language model, perform target detection and annotation on image and video frames based on the fine-tuned YOLO model, and extract multimodal knowledge elements.

[0020] The knowledge fusion module is used to perform entity alignment, entity linking and event fusion processing on the multimodal knowledge elements, and to achieve multimodal knowledge fusion by combining rules with a pre-trained BERT model;

[0021] The knowledge graph generation and visualization module is used to store the fused knowledge in the form of <entity-relationship-entity> triples in the graph database, and to dynamically render and interactively display the node-edge relationships through visualization components.

[0022] Thirdly, an electronic device is provided, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the computer program is executed by the processor, it implements the steps of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation as described in the first aspect.

[0023] Fourthly, a readable storage medium is provided, on which a program or instructions are stored, which, when executed by a processor, implement the steps of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation as described in the first aspect.

[0024] The vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation of the present invention has the following beneficial effects:

[0025] This application, starting from the characteristics of the business domain, proposes a knowledge graph construction scheme covering the entire process of "business modeling - data acquisition - data preprocessing - data extraction - data fusion - visualization representation". This application focuses on the end-to-end collaboration and vertical domain adaptability of multimodal data processing. Its core lies in overcoming the technical bottlenecks of traditional solutions in heterogeneous data integration, cross-modal association, and domain adaptation. It proposes a hybrid strategy including multimodal data initiation and preprocessing, cross-modal data extraction collaboration, and data fusion, as well as a systematic solution with end-to-end domain penetration design, possessing the following advantages:

[0026] The flexibility of data acquisition and processing is enhanced by dynamic adaptation technology using operator-based access. This breaks down the access logic for multimodal data (text, images, video) into reusable operators, constructing a capability for categorizing and automatically batch processing multi-source data. Flexible combination through workflow orchestration solves the problem of traditional fixed interfaces being unable to adapt to diverse data sources in vertical domains, achieving dynamic matching of "data source-operator-processing flow" and improving the flexibility of cross-domain data access. In the preprocessing stage, key content extraction, cleaning, and organization of text, images, and videos are performed, laying the foundation for subsequent annotation and fusion.

[0027] The collaborative nature of data extraction leverages the advantages of the YOLO algorithm in image / video object detection, combined with the deep understanding capabilities of the Qwen large language model for domain text, to achieve complementary cross-modal annotation between YOLO and Qwen. This enables precise cross-modal association of "image / video entities-text semantics," overcoming the semantic fragmentation problem in cross-modal annotation caused by a single model.

[0028] The complementary nature of data fusion, targeting the dual characteristics of "structured norms + unstructured implicit information" in vertical domain knowledge, adopts a layered strategy of "rule fusion + BERT pre-trained model fusion" to ensure the accuracy of rigid domain knowledge and the depth and flexibility of implicit related knowledge, thus solving the limitations of single fusion methods in domain knowledge coverage.

[0029] Domain-specific customization involves embedding vertical domain characteristics into every step from data collection to graph generation, forming a domain-specific closed loop covering the entire process of "collection-processing-extraction-fusion-graph", generating a complete and customizable vertical domain knowledge graph solution.

[0030] The device, electronic equipment, and readable storage medium corresponding to the vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation of this invention can achieve the same technical effect, and will not be described in detail here to avoid repetition. Attached Figure Description

[0031] Figure 1 A schematic diagram of a knowledge graph technology architecture provided in an embodiment of this application;

[0032] Figure 2 A schematic flowchart illustrating the construction of a domain knowledge graph based on a large model, provided for embodiments of this application;

[0033] Figure 3 A schematic flowchart illustrating a vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, provided for an embodiment of this application;

[0034] Figure 4 A knowledge modeling framework diagram provided for embodiments of this application;

[0035] Figure 5 A schematic flowchart illustrating the construction of a process orchestration platform provided in this application embodiment;

[0036] Figure 6 A schematic flowchart illustrating the construction of an operator component provided in an embodiment of this application;

[0037] Figure 7 A schematic flowchart illustrating an entity alignment stage provided for an embodiment of this application;

[0038] Figure 8 This is a schematic flowchart illustrating an event fusion rule provided in an embodiment of this application. Detailed Implementation

[0039] To further illustrate the technical means and effects adopted by the present invention to achieve its intended purpose, the technical solutions in the embodiments of this application are clearly described. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art are within the scope of protection of this application.

[0040] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0041] The steps described in the specification and the flowcharts in the accompanying drawings of this invention are not necessarily strictly executed according to the step numbers; the execution order of the method steps can be changed. Furthermore, certain steps can be omitted, multiple steps can be combined into one step, and / or one step can be broken down into multiple steps.

[0042] See Figure 1 Knowledge graph technology architecture such as Figure 1 As shown, based on the knowledge acquisition logic, each iteration of the knowledge graph construction process generally includes three steps:

[0043] ①Information extraction: This refers to extracting entities (concepts), attributes, and relationships between entities from various types of data sources, and forming ontological knowledge representations based on this.

[0044] ② Knowledge integration: After acquiring new knowledge, it is necessary to integrate it to eliminate contradictions and ambiguities. For example, some entities may have multiple expressions, and a specific name may correspond to multiple different entities.

[0045] ③ Knowledge Processing: New knowledge that has been integrated needs to undergo quality assessment (some parts require manual verification) before qualified portions are added to the knowledge base to ensure its quality. After adding new data, knowledge reasoning can be performed to expand existing knowledge and obtain new knowledge.

[0046] In related technologies, the implementation ideas for applying large models to vertical domains for knowledge graph construction are as follows: Figure 2 As shown, it includes:

[0047] 1. In the data standardization stage, the acquired text, image, and video data are standardized to obtain a standardized multimodal dataset of text, images, and videos.

[0048] 2. In the feature extraction and fusion stage, a word vector model is used to extract word and sentence vectors from text data, a convolutional neural network is used to extract feature vectors from image data, and keyframe feature fusion and time series analysis are used to extract features from video data. The original features from different modalities are concatenated and input into a unified model for joint learning to obtain a multimodal feature representation that comprehensively reflects the information of the multimodal data.

[0049] 3. In the stage of extracting entity-relation pairs based on the large model, a large amount of text and image labeled data is selected to fine-tune the large model. Multimodal feature representation data is input into the fine-tuned large model to extract entities and relations, and obtain entity-relation pairs.

[0050] 4. In the domain knowledge graph construction stage, the entity-relationship pairs extracted from the large model are stored in the graph data in the form of triples of "entity-relationship-entity" or "entity-attribute-attribute value" to construct the initial domain knowledge graph.

[0051] In recent years, various business domains have accumulated a large amount of structured, semi-structured, and unstructured data resources, including text, images, and videos. Since different types of data express the same semantics from different perspectives, cross-referencing information can strengthen the representation of retrieved features, facilitating the integration, summarization, and filtering of various types of data. Therefore, the high-level semantic relevance of heterogeneous data is of great significance for the semantic understanding of massive amounts of data. Unified modeling of data from different modalities, sources, and time periods is the primary task in building knowledge graphs for vertical domains. The construction of vertical domain knowledge graphs faces systemic quantitative contradictions in multimodal data processing, which are mainly manifested in three core issues:

[0052] ① The discreteness of data formats and sources leads to the fragmentation of multimodal data processing. Traditional interfaces are difficult to cover most data sources. At the same time, the processing flow is non-standardized due to modal differences and domain-specific needs. The cross-modal process reuse rate is low, which greatly increases the processing cost.

[0053] ② There are obvious bottlenecks in cross-modal information association. The annotation error rate remains high when processing single modality, and the lack of unified feature representation in multimodal implicit association affects the integrity of the knowledge graph.

[0054] ③ The accuracy of data fusion is limited by the performance constraints of a single method. Pure rule-based or pre-trained models have low accuracy when processing unstructured data and technical terms in vertical domains, making it difficult to meet the domain's accuracy requirements. The core of these problems lies in the conflict between the "heterogeneity-domain specificity-relevance" of multimodal data and the "standardization-generality-unimodality" of traditional processing solutions. Targeted technological innovations are urgently needed to overcome the bottlenecks in efficiency, accuracy, and adaptability.

[0055] In related technologies, domain knowledge graph construction schemes based on large models enhance the accuracy of cross-modal information association and data fusion by utilizing large models, which improves the problem of multimodal data fragmentation to some extent. However, the approach of "word vector model + convolutional neural network + keyframe feature fusion" for multimodal data feature extraction, which is suitable for customized design in the field of science and technology innovation, is difficult to adapt to multimodal data types in other fields (such as images or videos in special formats), lacks universality, and has high costs for cross-domain reuse. At the same time, using only large models for entity-relationship extraction and fusion ignores the special terminology and logical relationships of vertical fields, which is prone to bias due to the probabilistic output of large models, resulting in weak robustness of data fusion.

[0056] To address the challenges of fragmented multimodal data processing, insufficient cross-modal information association, and limited data fusion accuracy in vertical domain knowledge graph construction, this specification provides a vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation. It also includes a vertical domain knowledge graph reasoning device based on operator orchestration and intelligent annotation, a computer device, and a computer-readable storage medium. The method enhances the flexibility of data acquisition and processing through operator-based access and process orchestration. Multimodal data collaborative annotation is achieved based on the Qwen large language model and the YOLO object detection algorithm. The complementary approach of collecting data using "rules + BERT pre-trained model" optimizes the fusion effect. Through domain-specific design throughout the entire process, an end-to-end solution from data acquisition to graph construction is formed, effectively addressing the pain points of complex data types and low standardization in vertical domains. The following detailed descriptions, in conjunction with the accompanying drawings and preferred embodiments, illustrate these challenges.

[0057] Please see Figure 3This application provides a vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, such as... Figure 4 As shown, it includes:

[0058] Step S1: Construct a domain business knowledge model based on the business process to define and store entities and relationships between entities.

[0059] The construction of the domain business knowledge model specifically includes:

[0060] Step S11: Focusing on business activities, extract multiple types of concepts strongly related to the activities, including: entities, time, space, tasks, events, capabilities, activities, and multimedia data objects.

[0061] Step S12: Establish logical relationships between concepts based on business processes;

[0062] Step S13: Construct a schema layer for storing domain-specific knowledge through entity modeling and relation modeling.

[0063] In this step, the knowledge referred to is the knowledge extracted from the domain business knowledge model that is associated with the domain local area network and the domain business system, and the entities are obtained from multimodal data such as natural language text, images, and video data using named entity recognition tools.

[0064] See Figure 4 This application, based on a top-level analysis approach, centers on common business activities within a vertical domain. It extracts strongly related entities, time, space, tasks, events, capabilities, activities, and multimedia data objects, defines logical relationships between these objects that conform to the laws of things, and constructs an ontology framework suitable for the professional domain. This framework connects the various objects, describes the relationships between concepts, and integrates them into an organic top-level business knowledge framework, providing the main framework for knowledge graph construction. For example... Figure 4 As shown. Through comprehensive research on the business areas of interest, a method emphasizing dynamic interactions and using business processes as the link is adopted, combined with a top-level business knowledge framework (see...). Figure 4 Knowledge Model Framework Diagram: Construct entity and relationship models applicable to specific domains.

[0065] Taking the experimental training business domain as an example, a knowledge graph framework covering all elements of experimental training, including "people, equipment, materials, settings, environment, and programming," is constructed around the entire process of "planning → resource allocation → training implementation → evaluation and optimization" to ensure the implementation of business scenarios. Specific entity modeling, relationship modeling, and logical reasoning modeling are shown in Tables 1 and 2.

[0066] Table 1. Examples of Entity Modeling

[0067] Domain Objects Entity / Attribute Type For example personnel Trainees, coaches, support staff, and review experts Trainees, flight instructors, communications officers, and equipment operators equipment Physical equipment, training materials, and simulation devices Flight simulators, flight trainers, airport equipment supplies Training materials, consumables, and supplies Fuel, consumables, training targets, first aid kit facility Training grounds, support facilities, and infrastructure Power supply system for flight simulation training center and training ground environment Natural environment, electromagnetic environment, simulated environment Thunderstorms, satellite communications, mountainous terrain, warning zones prepared by Drafting unit, rules and regulations, task plan Departments, teams, training syllabus, training plans

[0068] Table 2 Example of Entity Relationship Modeling

[0069] Relationship Name Synonyms for relation For example Trainees - Participation - Task Plan Participate, execute, implement... <Zhang San, participated in, flight training course> Trainees - Use - Equipment Use, operation, control... <Zhang San, use, flight simulator> Coaches - Instructors - Trainees Guide, instruct, teach, instruct... <Li Si, supervisor, Zhang San> Support personnel - support facilities Maintenance, repair, overhaul... <Zhang San, Maintenance, Training Ground Power Supply System> Support personnel - preparation - supplies Supply, storage, procurement, distribution... <Zhang San, Procurement, Fuel> Review experts - evaluation - training effectiveness Evaluation, inspection, testing, verification... <Zhang San, Evaluation, Flight Training Course> Review experts - assessment - equipment effectiveness Evaluation, inspection, testing, verification... <Zhang San, Evaluation, Flight Simulator> Equipment - Allocation - Organizational Unit Assignment, affiliation, allocation... <Flight simulator, assigned to, a certain unit> Facilities - Storage - Equipment Storage, equipment, deployment, placement... <Simulation Training Center, Deployment, Simulation Equipment> Equipment-Adaptation-Environment Adaptation, applicability, constraints... <Aircraft, applicable, high visibility meteorology> Materials - Supply - Compilation Unit Supply, allocation, transfer... <Fuel, supply, a certain unit> Materials - Supplies - Trainees Supply, allocation, transfer... <Fuel, supply, Li Si> Mission Planning - Consumption - Resources Loss, expenditure, damage, reduction... <Flight training course, consumption, fuel> Facilities - Management - Materials Storage, placement, preservation, reserve... <Classroom, storage, training materials> Facilities - Ownership - Compilation Unit Classification, affiliation, subordination... <Simulation Training Center, Subordinate to, Training Department> Facilities - Preparation - Mission Plan Preparation, discussion, agreement... <Unit meeting room, preparation, flight training plan> Environment-Impact-Task Plan Restrictions, adjustments, adaptations, ... <Typhoon weather, restrictions, flight training plan> Compilation Unit - Development - Task Plan Formulate, compile, draft, revise... <A certain unit, establishment, flight training plan> Organizing Unit - Subordinate Units - Trainees Management, jurisdiction, control... <Training Department of a certain unit, under which there is Li Si>

[0070] Step S2: Configure the data acquisition and preprocessing process through the visual process orchestration platform, collect multimodal data, process the multimodal data according to the configured process, and store it as domain knowledge through the domain business knowledge model.

[0071] The visual process orchestration platform includes multiple data import operator components and multiple data preprocessing operator components. By obtaining orchestration instructions through the front-end interface, the orchestration components assemble data import and preprocessing processes that adapt to business requirements.

[0072] The multimodal data is data associated with the domain local area network and domain business system, which is preprocessed and then extracted and stored as domain knowledge through the domain business knowledge model; the multimodal data includes: structured data, semi-structured data and unstructured data;

[0073] Structured data includes tabular data, CSV files, and Excel files, etc.

[0074] Semi-structured data includes API data and JSON data, etc.

[0075] Unstructured data includes text data, image data, and video data.

[0076] In practice, this step utilizes the Activity engine to build a flexibly configurable workflow orchestration framework. Data ingress operators and data preprocessing operators are used as needed to orchestrate data ingress processing flows that adapt to business requirements. After the task starts, the system connects to the organization's internal business database, file service system, or local file system according to preset workflow nodes and parameters. It automatically backs up structured data (including table data, CSV files, Excel files, etc.), semi-structured data (including interface data and JSON data), and unstructured data (including text data, image data, and video data) to the system's database. Preprocessing operators are invoked to perform batch cleaning and standardization operations on the imported text, image, and video data. For text data, operations include removing duplicate records, filtering records, value mapping, handling missing values, and handling outliers. For image data, image reading, grayscale conversion, image noise reduction, and size adjustment are performed. For video data, video reading, frame extraction, frame merging, and frame sorting are performed. High-quality standard data is obtained upon completion of the workflow.

[0077] For example, the technical architecture of the visual workflow orchestration platform consists of a front-end interaction layer, a back-end orchestration service layer, an execution engine layer, and a data storage layer to ensure decoupling of the interfaces of each layer and to ensure high scalability and stability. The technology selection is shown in Table 3.

[0078] Table 3 Technology Selection Table

[0079] Module Technical tools Function Front-end visualization Vue3 + ElementPlus + GoJS + D3.js Build a canvas drag-and-drop, workflow editing interface Backend services Spring Boot + Activity7 + MyBatis-Plus Implement process definition, task scheduling, and data persistence. Data storage MySQL (metadata) + MinIO (file storage) Storage process definition, task status and preprocessing results Preprocessing engine OpenCV library (for images) + FFmpeg tool (for videos) Perform image grayscale conversion, video decoding and other processing.

[0080] This application's embodiments use the Activity7 workflow engine as its core, combining visual orchestration technology to build a visual workflow orchestration platform. The main idea is to parse the workflow definition into an executable task sequence, determine the execution order through the connection relationships between task nodes, and provide a configuration file interface to facilitate defining complex workflows via code scripts. This allows for saving and generating executable task flows. During task execution, the system executes orchestrated tasks in the task order, calling corresponding processing logic based on node type, gradually achieving automated processing of the overall business logic. The specific implementation process is as follows: Figure 5 As shown, firstly, a project is created using the Spring Boot framework, and the Activity (workflow engine) dependency is imported, along with a configured MySQL database. Next, the Vue 3 framework and ElementPlus component library are used to build the front-end visual interface, and the GoJS library is imported to implement canvas functionality for drawing and manipulating graphics. Then, a process model is defined, a process definition file is created, and task attributes are set, laying the foundation for subsequent process development. Next, basic components, including process initiation, processing, and judgment, are developed. Further auxiliary components such as data referencing and data cleaning are developed. Finally, visual rules and the interface are configured, and the rules are stored in the database in JSON format, enabling flexible configuration and management of process rules.

[0081] This system incorporates import / processing operators and preprocessing operators, using the Activity workflow engine as its core framework. It integrates text processing tools, the OpenCV library, and FFmpeg technology to construct a workflow-orchestred data cleaning, image processing, and video preprocessing solution. The operator construction process is as follows: Figure 6 As shown,

[0082] Construction of the introductory operator:

[0083] The gateway type is determined based on the data source type, including database gateways, local gateways, and file server gateways;

[0084] Task node configuration: For database access, configure properties and set the execution class to a custom database access processing class; for local access, set the path as a process variable including CSV, JSON, tables, and Excel files; for file server access, configure properties, set the execution class to a custom server interaction class, and specify the path as a process variable.

[0085] Reception class implementation: For database reception, create an interface implementation class, connect to the database, write / execute query statements, and store data to process variables; for local reception, create an interface implementation class, read file content, parse the file according to the format, and store the data value as process variables; for file server reception, create an interface implementation class, connect to the server, download the specified file, parse the data, and store the data as process variables.

[0086] Preprocessing operator construction:

[0087] Determine the data type;

[0088] For text data, the NLTK open-source tool is introduced; for image data, the OpenCV library dependency is introduced; for video data, the FFmpeg tool is introduced.

[0089] Task node configuration: For text data, configure 22 operator attributes; for image data, configure 4 operator attributes; for video data, configure 4 operator attributes.

[0090] Receiving class implementation: Write an interface that implements Activity.

[0091] Finally, integrate all the above components into the Activity task, encapsulate them as components, and set the task node to the fully qualified name of the actual class.

[0092] The above operator construction process has developed various operators specifically for multimodal data acquisition and preprocessing, as detailed in Table 4.

[0093] Table 4 List of constructed operators

[0094] Operator (component) category Operator (component) description Data transfer operator Three operators: local connection, file server connection, and database connection. Data cleaning operators 22 cleaning operators including: removing duplicate records, adding constants, sorting records, cutting strings, filtering records, value mapping, field selection, string replacement, setting field values, setting field values ​​to constants, adding sequences, row flattening, string manipulation, numerical range, calculator, row to column, column to row, splitting fields, splitting columns into multiple rows, unique rows, handling missing values, and handling outliers. Image processing operators Image processing operators (image reading, grayscale conversion, image noise reduction, and resizing); and video processing operators (frame reading, frame extraction, frame merging, and frame sorting).

[0095] The acquisition and preprocessing of multi-source data is achieved through the creation, orchestration, and execution of scheduled tasks. Users edit the task flow canvas in a visual workflow orchestration platform, dragging and dropping components such as the start component, acquisition component, preprocessing component, output component, and end component into the canvas. These components are connected by lines, and parameters for each task component are set to construct a complete data acquisition and processing flow. During flow construction, components can be orchestrated as needed to create batch processing flows where task nodes within the same data source are connected serially, while different data sources are processed in parallel without interference. After a task starts, data is passed through flow variables to achieve the serial or parallel execution of operators, completing the synchronous acquisition and preprocessing of multi-source data. This includes text data acquisition and cleaning, image data acquisition, video reading, frame extraction, frame merging, and frame sorting, and video data acquisition, format conversion, decoding, and frame extraction, providing standard data that meets the model's input requirements for subsequent data annotation.

[0096] Step S3: Invoke the domain knowledge, identify and annotate the text data based on the fine-tuned Qwen large language model, and perform target detection and annotation on the image and video frames based on the fine-tuned YOLO model to extract multimodal knowledge elements.

[0097] Furthermore, for the text data, training and validation datasets are created using manual annotation. These datasets are then used as input information for the Qwen large language model to fine-tune the parameters, ensuring the model's optimal understanding of domain-specific terminology and semantic relationships. After fine-tuning, the model automatically identifies and extracts entity, relation, and attribute information from the text data.

[0098] Leveraging the superior language understanding and generation capabilities of the Qwen large language model, highly accurate annotation is performed on preprocessed text data, including filtering, deduplication, and mapping. Entity, relation, and attribute information is extracted from massive amounts of text data, providing foundational materials for constructing diverse and accurate knowledge graphs. In practice, this process mainly includes four key steps: data preparation, model fine-tuning, annotation execution, and result post-processing.

[0099] ① In the data preparation phase, based on the ontology construction approach and domain characteristics, publicly available internet databases and domain-specific intranets were accessed to obtain data such as industry reports, academic papers, policy and regulatory documents, news, case studies, and community texts. Experts or domain-related personnel were consulted to determine the relevance of the text data, ensuring that the data sources were broad and covered common text types and language expressions in the domain. Based on the domain knowledge modeling framework, rules for data annotation were defined, and text data was manually annotated using the BIO annotation method. The annotation results were cross-validated to resolve inconsistencies in annotation.

[0100] ② In the model fine-tuning stage, a GPU service was used to build the Qwen 2.5 model running and fine-tuning environment, and the PyTorch learning framework was installed to obtain the pre-trained weights of the Qwen model. Simultaneously, the labeled data was divided into training, validation, and test sets in an 8:1:1 ratio according to the principles of complete data types and uniform distribution, to prepare for fine-tuning. During the fine-tuning stage, a fine-tuning training script was built using the PyTorch deep learning framework. The pre-trained weights of the Qwen model were loaded into the script for fine-tuning. The optimal parameters (in this project, the learning rate was 3e-5, the batch size was 64, and the number of training epochs was 10) were determined based on the performance on the validation set to prevent underfitting after overfitting.

[0101] During fine-tuning, the cross-entropy loss function is used as the optimization objective, and the model parameters are updated through the backpropagation algorithm, enabling the model to learn domain-specific language patterns and annotation rules.

[0102] The cross-entropy loss function is as follows:

[0103]

[0104] The input is: real label (one-hot vector), model prediction probability The output is: the average loss value L.

[0105] The backpropagation algorithm is designed as follows:

[0106] Forward propagation: The model calculates the predicted value from the input sample.

[0107] Calculate the loss: L is calculated using the cross-entropy loss function;

[0108] Back-derivative: Output layer is: ,

[0109] Backpropagation of gradients layer by layer: updating the gradients of parameters in each layer using the chain rule. ;

[0110] Parameter update: ).

[0111] ③ Text annotation is performed by using a word segmenter to convert the preprocessed text data into a token sequence, which is then input into the fine-tuned Qwen model. The model uses the learned knowledge to reason and predict the annotation label corresponding to each token in the text. For example, if “a department released new equipment”, the model predicts “a department (D-Department) released (O) new equipment (E-equipment)”. The annotation results are then sorted and organized according to entities, relationships, and attributes, and restored to text.

[0112] ④ In the post-processing stage of text, the annotation results are mainly supplemented and corrected manually, and consistency checks are performed to ensure the quality of annotation.

[0113] For image data, the same method as for text datasets is used to create image datasets, namely, manually annotating training and validation datasets. These datasets are used as input information for the YOLO object detection model to fine-tune parameters, ensuring the model's best understanding of domain terminology and semantic relationships. After fine-tuning, the model performs object recognition and annotation on the input image data to obtain the object category and location.

[0114] In practice, leveraging the efficient and accurate target detection capabilities of YOLO (You Only Look Once) algorithm version 7, target annotation is performed on single frames of images or videos to extract key information that constitutes the knowledge graph, thus achieving image annotation and extraction. This process mainly includes four key steps: data preparation, model training, annotation execution, and result post-processing.

[0115] ① In the data preparation stage, image data is collected in the same way as text data to obtain images covering various scenes, angles, and lighting conditions to ensure the generalization ability of the model.

[0116] ② During the model training phase, professionals should annotate the target objects and their locations in the images and save them as XML files. The V7 version, known for its high detection speed and accuracy, should be selected for target detection. The backbone network structure of the algorithm should be set to CPSDarknet, and the feature pyramid structure should combine FPN (Feature Pyramid Network) and PAN (Path Aggregation Network). The boundary value regression and classification formulas for the loss function are as follows:

[0117]

[0118] In the boundary value regression formula, loU is the orthogonality ratio. α is the Euclidean distance between the center points of the predicted bounding box and the ground truth bounding box, c is the diagonal length of the smallest bounding rectangle containing both boxes, α is the balance parameter, and v is a parameter that measures the consistency of the aspect ratio between the predicted bounding box and the ground truth bounding box.

[0119]

[0120] The classification uses a binary cross-entropy loss function, where y is the true label (0 or 1). It is the probability value predicted by the model.

[0121] During training, the training set, validation set, and test set are divided in a manner similar to that used for text data. The model is trained using a GPU, with appropriate parameters set (the learning rate in this project is 0.06, the batch size is 64, and the number of training epochs is 300). The model weights are updated using the backpropagation algorithm, enabling the model to learn the feature patterns of different target objects.

[0122] ③ During the annotation execution phase, the trained model is loaded into the system for detection. The model extracts image features through the backbone network, detects targets, labels categories, and defines bounding boxes through a feature pyramid structure, and filters prediction results with low confidence based on a confidence threshold (0.5), retaining high-confidence results as annotation information. The annotation style is defined as "target category, bounding box coordinates (x1, y1, x2, y2), confidence".

[0123] ④ In the post-annotation processing stage, the annotation results are manually reviewed, and the annotation information is formatted and standardized to remove duplicate or redundant annotations. The target category in the annotation information is mapped to the entity type in the knowledge graph. For example, a target labeled "person" is mapped to "trainees" in the knowledge graph. Border and confidence information can be added to the knowledge graph as spatial location attributes of entities, and the relationships between the positions of target objects are inferred and mapped to semantic relationships in the knowledge graph.

[0124] To evaluate algorithm performance, recall, precision, and mean precision were used as evaluation metrics. A comprehensive performance evaluation was conducted on the labeled dataset, with recall and precision set at 80% and mean precision at 80%. The formulas for each metric are as follows:

[0125] Recall rate

[0126]

[0127] In this context, TP (True Positive) represents the number of true positives, which is the number of targets correctly detected by the model; FN (False Negative) represents the number of false negatives, which is the number of targets that actually exist but were not detected by the model.

[0128] accuracy

[0129]

[0130] Here, FP (False Positive) represents false positives, which is the number of targets that the model incorrectly detects.

[0131] Mean Precision

[0132]

[0133] The mean precision (AP) is calculated by sorting the prediction results within each category from highest to lowest confidence, calculating the current recall and precision for each prediction result, obtaining a series of recall-precision pairs, and then using 11-point interpolation to find the maximum precision at the 11 points of recall (0, 0.1, 0.2…1.0). The average of these maximum recalls is then used as the AP for that category. As shown in formula (6), C is the total number of categories. It is the average precision of the i-th category.

[0134] For video data, the same method as for image datasets is used to create video datasets, namely, manually annotating training and validation datasets. These datasets are used as input information for the YOLO object detection model to fine-tune parameters, ensuring the model's best understanding of domain terminology and semantic relationships. After fine-tuning, the model identifies the target category and location in a single frame of video image, calls the target tracking algorithm provided by FFmpeg, obtains the relationships between targets, and achieves video information annotation and extraction.

[0135] In practice, the pre-processed video is parsed into single-frame images, and its data annotation is the same as that of the image data, including four key steps: data preparation, model training, annotation execution, and result post-processing.

[0136] ① The data preparation and model fine-tuning stages are the same as those for image annotation. YOLO algorithm version 7 is used for object detection, and the training set, test set and validation set that meet the requirements are obtained in the same way.

[0137] ② During the annotation execution phase, each image frame is sequentially read from the stored image frames and input into the fine-tuned model. Target detection is performed through forward propagation to predict the category, bounding box coordinates, and confidence score of the target object in the image. Low confidence values ​​are filtered out based on the confidence threshold (0.5), and high confidence results are retained. The annotation format is defined as "frame sequence number, target category, bounding box coordinates (x1, y1, x2, y2), confidence score".

[0138] ③ In the post-annotation processing stage, a corresponding timestamp is added to each annotation based on the video frame rate and frame sequence number. A target tracking algorithm is used to establish relationships between targets in consecutive frames of the video. The detected target object categories are mapped to entity types in the knowledge graph, and the bounding box coordinates, timestamps, and relationships of the target objects are mapped to entity attributes and relationships and added to the knowledge graph, completing information extraction. The methods for evaluating video annotation results and algorithm performance are consistent with those for image annotation.

[0139] Furthermore, based on this, information extraction and correction are performed. Information extracted from text, image, and video data is supplemented and corrected through manual annotation to form high-quality knowledge data. The target categories, annotation locations, relationship information, and feature information extracted from text, images, and videos are mapped to entity, relationship, and attribute information of the knowledge graph, completing the knowledge graph information extraction. The corrected dataset is then added to the training dataset for model fine-tuning.

[0140] Step S4: Perform entity alignment, entity linking and event fusion processing on the multimodal knowledge elements, and combine rules with the pre-trained BERT model to achieve multimodal knowledge fusion.

[0141] Furthermore, in the entity alignment stage, the gensim library is used to perform attribute name matching and entity vector similarity calculation to obtain the entity alignment results.

[0142] By introducing the gensim natural language processing library, entity alignment technology is used to solve the problem of knowledge redundancy or attribute missingness caused by inconsistent entity names. First, for each entity in the external structured knowledge, its non-empty attribute names are extracted. Second, the obtained attribute names are matched with all triples under that entity category in the knowledge base, obtaining all triples that match the attribute names, and filtering out entities containing the matched attribute names. Finally, gensim is used to transform the input entities and the filtered entities into entity vectors containing attribute information, and similarity is calculated to obtain the entity alignment result, such as... Figure 7 As shown.

[0143] In the entity linking stage, based on the entity alignment results, the BERT model is used to map entity representations and knowledge base entities to the same semantic space, and disambiguation is achieved through similarity ranking.

[0144] The entity linking stage is mainly achieved through two phases: candidate set generation and candidate entity disambiguation. Candidate set generation involves obtaining a set of relevant entities from the knowledge base for each entity mention in the text. Candidate entity disambiguation involves ranking the candidate entities based on their relevance to obtain the optimal entity link. Since most of the entity mentions to be linked appear in unstructured text and lack attribute values, this embodiment uses a BERT pre-trained language model to transform the context in which the entity mention appears into vectors to represent its semantic information, thus realizing the representation of a single entity. On the other hand, entities in the knowledge base possess attribute triples, and BERT can transform this attribute information into attribute vectors within the same semantic space. After obtaining the semantic vector of the entity mention and the attribute vectors of the entities in the knowledge base, the similarity between the two is calculated to generate an entity candidate set. Finally, by ranking the entities in the entity candidate set according to their similarity, the entity with the highest similarity is selected as the linking entity for the entity mention in the text.

[0145] The event fusion phase includes rule-based preliminary event fusion and BERT model-based event fusion. Fusion metrics include time, space, causal relationship, entity attributes, event sequence, target consistency, and no-match events.

[0146] Event fusion is achieved by combining rule-based event fusion methods with representation learning-based event fusion methods.

[0147] Preliminary event fusion based on rules involves formulating rules for determining the same event based on domain expert knowledge and common event patterns. Time (±5 minutes), space (1-5 meters), causal relationship, entity attributes, event sequence, and target consistency characteristics are set as fusion indicators, and thresholds are set for each. Figure 8 The process will merge data within the thresholds of various indicators to form a complete and consistent event representation, aggregating multi-source event information and eliminating conflicts and duplications.

[0148] Event fusion based on the BERT model includes:

[0149] Step S401: Fine-tune the BERT model using the domain-labeled dataset.

[0150] This step involves domain adaptation of the BERT model. It fine-tunes the BERT model using a domain-labeled dataset, enabling it to learn domain-specific semantics and event models. Simultaneously, it leverages multimodal characteristics by introducing an attention mechanism to enhance feature fusion capabilities. Fine-tuning parameters include: learning rate (3e-5), number of training epochs (10), batch size (32), and number of attention heads (10).

[0151] Step S402: Input the initially fused event data into the fine-tuned BERT model for event semantic fusion.

[0152] Event fusion based on the BERT model involves inputting the initially fused event data into a fine-tuned BERT model, predicting event relationships based on domain knowledge, and then making further decisions and refinements on the initially fused events to improve the fused events.

[0153] Step S403: Verify the model fusion results. Based on the verification results, make feedback adjustments to the fusion rules and the BERT model.

[0154] This step is the verification and optimization stage of the fusion results. The average standard for accuracy, recall, and mean precision is set to 80%. The model fusion results are verified. Based on the verification results, the fusion rules and BERT model are adjusted. For cases of inaccurate fusion, the reasons for the differences are analyzed, and the rules are corrected and supplemented, or the model parameters are adjusted or optimized.

[0155] Step S5: Store the fused knowledge in the form of <entity-relationship-entity> triples in the graph database, and use a visualization component to dynamically render and interactively display the node-edge relationships.

[0156] In practice, the merged structured data is stored in the Neo4j graph database, and the D3.js visualization library provides real-time rendering capabilities of node-edge relationships to generate a visualized topological relationship graph for knowledge graph display.

[0157] Based on the above technical solutions, this application proposes a knowledge graph construction scheme covering the entire process of "business modeling - data acquisition - data preprocessing - data extraction - data fusion - visualization representation," starting from the characteristics of the business domain. This application focuses on the end-to-end collaboration and vertical domain adaptability of multimodal data processing. Its core lies in overcoming the technical bottlenecks of traditional solutions in heterogeneous data integration, cross-modal association, and domain adaptation. It proposes a hybrid strategy including multimodal data initiation and preprocessing, cross-modal data extraction collaboration, and data fusion, as well as a systematic solution with end-to-end domain penetration design, possessing the following advantages:

[0158] The flexibility of data acquisition and processing is achieved through dynamic adaptation technology using operator-based access. The access logic for multimodal data (text, images, video) is broken down into reusable operators. Rich algorithms and functions are obtained using Nltk, OpenCV, and FFmpeg tools to construct the capability for multi-source data classification, access, and automatic batch processing. Flexible combination through workflow orchestration solves the problem of traditional fixed interfaces being difficult to adapt to diverse data sources in vertical domains, achieving dynamic matching of "data source-operator-processing flow" and improving the flexibility of cross-domain data access. In the preprocessing stage, key content extraction, cleaning, and organization of text, images, and videos are performed, laying the foundation for subsequent annotation and fusion.

[0159] The collaborative nature of data extraction leverages the advantages of the YOLO algorithm in image / video object detection, combined with the deep understanding capabilities of the Qwen large language model for domain text, to achieve complementary cross-modal annotation between YOLO and Qwen. This enables precise cross-modal association of "image / video entities-text semantics," overcoming the semantic fragmentation problem in cross-modal annotation caused by a single model.

[0160] The complementary nature of data fusion, targeting the dual characteristics of "structured norms + unstructured implicit information" in vertical domain knowledge, adopts a layered strategy of "rule fusion + BERT pre-trained model fusion" to ensure the accuracy of rigid domain knowledge and the depth and flexibility of implicit related knowledge, thus solving the limitations of single fusion methods in domain knowledge coverage.

[0161] Domain-specific customization involves embedding vertical domain characteristics into every step from data collection to graph generation, forming a domain-specific closed loop covering the entire process of "collection-processing-extraction-fusion-graph", generating a complete and customizable vertical domain knowledge graph solution.

[0162] Corresponding to the above embodiments of the vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation, this application provides a vertical domain knowledge graph reasoning device based on operator arrangement and intelligent annotation, including:

[0163] The domain business knowledge model module is used to build a domain business knowledge model based on business processes, and to define and store entities and the relationships between entities;

[0164] The visual process orchestration module is used to configure data acquisition and preprocessing processes through the visual process orchestration platform, collect multimodal data, process the multimodal data according to the configured process, and store it as domain knowledge through the domain business knowledge model.

[0165] The knowledge element extraction module is used to call the domain knowledge, recognize and annotate text data based on the fine-tuned Qwen large language model, perform target detection and annotation on image and video frames based on the fine-tuned YOLO model, and extract multimodal knowledge elements.

[0166] The knowledge fusion module is used to perform entity alignment, entity linking and event fusion processing on the multimodal knowledge elements, and to achieve multimodal knowledge fusion by combining rules with a pre-trained BERT model;

[0167] The knowledge graph generation and visualization module is used to store the fused knowledge in the form of <entity-relationship-entity> triples in the graph database, and to dynamically render and interactively display the node-edge relationships through visualization components.

[0168] Furthermore, the construction of the domain business knowledge model includes:

[0169] Centered on business activities, extract multiple types of concepts that are strongly related to the activities, including: entities, time, space, tasks, events, capabilities, activities, and multimedia data objects;

[0170] Establish logical relationships between concepts based on business processes;

[0171] By using entity modeling and relation modeling, a schema layer is constructed to store domain-specific knowledge.

[0172] Furthermore, the visual process orchestration platform includes multiple data acquisition operator components and multiple data preprocessing operator components. By obtaining orchestration instructions through the front-end interface, the orchestration components assemble data acquisition and preprocessing processes that adapt to business requirements.

[0173] The multimodal data is data associated with the domain local area network and the domain business system, which is preprocessed and then extracted and stored as domain knowledge by the domain business knowledge model;

[0174] The multimodal data includes structured data, semi-structured data, and unstructured data; the structured data includes table data, CSV files, and Excel files.

[0175] The semi-structured data includes interface data and JSON data;

[0176] The unstructured data includes text data, image data, and video data.

[0177] Furthermore, in the knowledge element extraction module:

[0178] For text data, training and validation datasets are created using manual annotation and used as input information for fine-tuning the parameters of the Qwen large language model. After fine-tuning, the model can extract entity, relation, and attribute information from the text after inputting text data.

[0179] For image data, training and validation datasets are created using manual annotation, which are then used as input information for fine-tuning the parameters of the YOLO object detection model. After fine-tuning, the model can obtain the object category and object location after inputting image data.

[0180] For video data, training and validation datasets are created using manual annotation and used as input information for the YOLO object detection model to fine-tune parameters. After fine-tuning, the model identifies the target category and location in a single frame of video image and obtains the relationship between targets after inputting video data.

[0181] Furthermore, in the aforementioned knowledge fusion module:

[0182] In the entity alignment stage, the gensim library is used to perform attribute name matching and entity vector similarity calculation to obtain entity alignment results.

[0183] In the entity linking stage, based on the entity alignment results, the BERT model is used to map entity representations and knowledge base entities to the same semantic space, and disambiguation is achieved through similarity ranking.

[0184] The event fusion phase includes rule-based preliminary event fusion and BERT model-based event fusion. Fusion metrics include time, space, causal relationship, entity attributes, event sequence, target consistency, and no-match events.

[0185] Furthermore, event fusion based on the BERT model includes:

[0186] Fine-tuning the BERT model using a domain-labeled dataset;

[0187] The initially fused event data is input into the fine-tuned BERT model for event semantic fusion;

[0188] Verify the model fusion results, and adjust the fusion rules and BERT model based on the verification results.

[0189] The above-described vertical domain knowledge graph reasoning device based on operator arrangement and intelligent annotation implements the steps and processes of the above-described vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0190] Corresponding to the above embodiments of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, this application provides an electronic device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps and processes of the above embodiments of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, and achieves the same technical effect. To avoid repetition, it will not be described again here.

[0191] The memory 1009 can be used to store software programs and various data. The memory 1009 may primarily include a first storage area for storing programs or instructions and a second storage area for storing data. The first storage area may store the operating system, application programs or instructions required for at least one function (such as sound playback function, image playback function, etc.). Furthermore, the memory 1009 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct memory bus RAM (DRRAM). The memory 1009 in the embodiments of this application includes, but is not limited to, these and any other suitable types of memory.

[0192] The processor 1010 may include one or more processing units; optionally, the processor 1010 integrates an application processor and a modem processor, wherein the application processor mainly handles operations involving the operating system, user interface, and applications, and the modem processor mainly handles wireless communication signals, such as a baseband processor. It is understood that the aforementioned modem processor may also not be integrated into the processor 1010.

[0193] Corresponding to the above embodiments of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, this application also provides a readable storage medium storing a program or instructions. When the program or instructions are executed by a processor, they implement the steps and processes of the above embodiments of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, and achieve the same technical effect. To avoid repetition, they will not be described again here.

[0194] The processor is the processor in the electronic device described in the above embodiments of this application. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0195] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0196] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a computer software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0197] It is understood that the embodiments of this application have been described above in conjunction with the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. As those skilled in the art will know, various changes or equivalent substitutions can be made to these features and embodiments without departing from the spirit and scope of the invention. Furthermore, those skilled in the art, under the guidance or instruction of this application, can modify these features and embodiments to adapt to specific situations and materials without departing from the spirit and scope of the invention. Therefore, this invention is not limited to the specific embodiments disclosed herein, and all embodiments falling within the scope of the claims of this application are within the protection scope of this invention.

Claims

1. A vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation, characterized in that, include: A domain business knowledge model is built based on business processes to define and store entities and relationships between entities; Through a visual process orchestration platform, configure data acquisition and preprocessing processes, collect multimodal data, process the multimodal data according to the configured process, and store it as domain knowledge through the domain business knowledge model; The domain knowledge is invoked to identify and annotate text data based on the fine-tuned Qwen large language model, and to detect and annotate targets in images and video frames based on the fine-tuned YOLO model, thereby extracting multimodal knowledge elements. The multimodal knowledge elements are processed by entity alignment, entity linking and event fusion, and multimodal knowledge fusion is achieved by combining rules with a pre-trained BERT model; The merged knowledge is stored in a graph database in the form of <entity-relationship-entity> triples, and the node-edge relationship is dynamically rendered and interactively displayed through a visualization component; The construction of the domain business knowledge model includes: Centered on business activities, extract multiple types of concepts that are strongly related to the activities, including: entities, time, space, tasks, events, capabilities, activities, and multimedia data objects; Establish logical relationships between concepts based on business processes; By using entity modeling and relation modeling, a schema layer is constructed to store domain-specific knowledge. The steps described above, including entity alignment, entity linking, and event fusion processing of the multimodal knowledge elements, and combining rules with a pre-trained BERT model to achieve multimodal knowledge fusion, are as follows: In the entity alignment stage, the gensim library is used to perform attribute name matching and entity vector similarity calculation to obtain entity alignment results. In the entity linking stage, based on the entity alignment results, the BERT model is used to map entity representations and knowledge base entities to the same semantic space, and disambiguation is achieved through similarity ranking. The event fusion phase includes rule-based preliminary event fusion and BERT model-based event fusion. Fusion metrics include time, space, causal relationship, entity attributes, event sequence, target consistency, and no-match events.

2. The vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation according to claim 1, characterized in that, The visual process orchestration platform includes multiple data import operator components and multiple data preprocessing operator components. By obtaining orchestration instructions through the front-end interface, the orchestration components assemble data import and preprocessing processes that adapt to business requirements. The multimodal data is data associated with the domain local area network and the domain business system, which is preprocessed and then extracted and stored as domain knowledge by the domain business knowledge model; The multimodal data includes structured data, semi-structured data, and unstructured data; the structured data includes table data, CSV files, and Excel files. The semi-structured data includes interface data and JSON data; The unstructured data includes text data, image data, and video data.

3. The vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation according to claim 1, characterized in that, The steps described above, including invoking domain knowledge, recognizing and labeling text data based on the fine-tuned Qwen large language model, performing target detection and labeling of image and video frames based on the fine-tuned YOLO model, and extracting multimodal knowledge elements, are as follows: For text data, training and validation datasets were created using manual annotation, which were then used as input information for the Qwen large language model to fine-tune the parameters. After fine-tuning, the model extracts entity, relation, and attribute information from the text after inputting text data; For image data, training and validation datasets were created using manual annotation, which were then used as input information for the YOLO object detection model to fine-tune the parameters. After fine-tuning, the model obtains the target category and target location from the input image data; For video data, training and validation datasets were created using manual annotation, which were then used as input information for the YOLO object detection model to fine-tune the parameters. After fine-tuning, the model identifies the target category and location in a single frame of video image and obtains the relationship between targets after inputting video data.

4. The vertical domain knowledge graph reasoning method based on operator arrangement and intelligent annotation according to claim 1, characterized in that, Event fusion based on the BERT model includes: Fine-tuning the BERT model using a domain-labeled dataset; The initially fused event data is input into the fine-tuned BERT model for event semantic fusion; Verify the model fusion results, and adjust the fusion rules and BERT model based on the verification results.

5. A vertical domain knowledge graph reasoning device based on operator arrangement and intelligent annotation, characterized in that, include: The domain business knowledge model module is used to build a domain business knowledge model based on business processes, and to define and store entities and the relationships between entities; The visual process orchestration module is used to configure data acquisition and preprocessing processes through the visual process orchestration platform, collect multimodal data, process the multimodal data according to the configured process, and store it as domain knowledge through the domain business knowledge model. The knowledge element extraction module is used to call the domain knowledge, recognize and annotate text data based on the fine-tuned Qwen large language model, perform target detection and annotation on image and video frames based on the fine-tuned YOLO model, and extract multimodal knowledge elements. The knowledge fusion module is used to perform entity alignment, entity linking and event fusion processing on the multimodal knowledge elements, and to achieve multimodal knowledge fusion by combining rules with a pre-trained BERT model; The knowledge graph generation and visualization module is used to store the fused knowledge in the form of <entity-relationship-entity> triples into the graph database, and to dynamically render and interactively display the node-edge relationships through visualization components; The construction of the domain business knowledge model includes: Centered on business activities, extract multiple types of concepts that are strongly related to the activities, including: entities, time, space, tasks, events, capabilities, activities, and multimedia data objects; Establish logical relationships between concepts based on business processes; By using entity modeling and relation modeling, a schema layer is constructed to store domain-specific knowledge. The steps described above, including entity alignment, entity linking, and event fusion processing of the multimodal knowledge elements, and combining rules with a pre-trained BERT model to achieve multimodal knowledge fusion, are as follows: In the entity alignment stage, the gensim library is used to perform attribute name matching and entity vector similarity calculation to obtain entity alignment results. In the entity linking stage, based on the entity alignment results, the BERT model is used to map entity representations and knowledge base entities to the same semantic space, and disambiguation is achieved through similarity ranking. The event fusion phase includes rule-based preliminary event fusion and BERT model-based event fusion. Fusion metrics include time, space, causal relationship, entity attributes, event sequence, target consistency, and no-match events.

6. An electronic device, characterized in that, The electronic device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, it implements the steps of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation as described in any one of claims 1 to 4.

7. A readable storage medium, characterized in that, The readable storage medium stores a program or instructions that, when executed by a processor, implement the steps of the vertical domain knowledge graph reasoning method based on operator orchestration and intelligent annotation as described in any one of claims 1 to 4.

Images