Method, device and system for constructing multi-modal knowledge graph of military software defects

Abstract

The application discloses a kind of military software defect multimodal knowledge graph construction method, device and system, belong to knowledge graph construction field, comprising: S1, mode layer design;S2, defect knowledge mining: data source is collected, information extraction and data preprocessing;S3, multimodal knowledge graph construction: data entry is carried out in multiple databases, the semantic relationship between nodes is stored in graph database, the multimodal information of graph database node is stored in relational database, and multimodal knowledge graph is constructed.The military software defect knowledge graph of the data driving and knowledge guiding combination constructed by the application changes the traditional knowledge management mode, multidirectional rich semantic information and the defect of other information are shown, the problems of military software defect knowledge resource fragmentation, insufficient utilization, high knowledge acquisition cost and updating difficulty are solved, and foundation is laid for military software defect encyclopedia retrieval, intelligent question and answer, intelligent prediction and intelligent recommendation and the like application.

Description

Technical Field

[0001] This invention relates to the field of knowledge graph construction, and more specifically, to a method, apparatus, and system for constructing a multimodal knowledge graph of military software defects. Background Technology

[0002] Thanks to the tireless efforts of countless predecessors, the military industry has spawned numerous new fields, and related industries have become increasingly developed, leading to more complex military testing. The vast amount of domain knowledge, the ever-increasing number of system functions, the explosive growth of test data, and the bloated typical defect databases have made knowledge retrieval, searching, and learning difficult. However, software defects are an inherent property of software, and discovered software defects constitute a knowledge base rich in lessons learned. This is a valuable resource for developers, testers, and even requirements analysts and system designers.

[0003] The military industry suffers economic and reputational losses annually due to software defects. Reducing the occurrence of defects in military software is a pressing issue that the industry needs to address. Current military software defects exhibit the following problems:

[0004] (1) Defects discovered during internal testing, joint testing, experimentation, and on-site use by users are not fully utilized. Instead, they are buried in a vast amount of documents such as test reports, joint testing records, zeroing reports, quality cases, and lists of field issues, making them extremely inconvenient to use.

[0005] (2) Military software defects: The current storage methods are not uniform, such as: structured data stored in the form of tables, semi-structured data obtained from information systems, and unstructured data stored in text format.

[0006] (3) Each defect exists in isolation. They are not connected to each other and are completely disconnected from the model, software, hardware, equipment, components, and defect phenomena. They are like isolated islands, hidden behind the document and concealed deep in the text.

[0007] (4) In addition to simple text descriptions, each software defect often contains rich multimodal information such as rich text, images, documents, audio, and video. Besides semantically associating defect terms with other expressions, how to comprehensively present the relevant content of military software defects is also one of the key issues of concern in the industry. Summary of the Invention

[0008] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method, apparatus and system for constructing a multimodal knowledge graph of military software defects, which solves the problems of fragmented knowledge resources of military software defects, insufficient utilization, high knowledge acquisition costs and difficulty in updating.

[0009] The objective of this invention is achieved through the following solution:

[0010] A method for constructing a multimodal knowledge graph of military software defects includes the following steps:

[0011] S1, Pattern Layer Design;

[0012] S2, Defect Knowledge Mining: Data collection, information extraction, and data preprocessing from data sources;

[0013] S3, Multimodal Knowledge Graph Construction: Data is entered into multiple databases, semantic relationships between nodes are stored in a graph database, and multimodal information of graph database nodes is stored in a relational database to construct a multimodal knowledge graph.

[0014] Furthermore, in step S1, the pattern layer design adopts a top-down pattern layer design approach. First, domain experts and other industry professionals establish the top-level pattern layer themselves, and then the data layer's descriptive concepts are added to the already defined pattern layer to ensure the accuracy of the pattern system. Moreover, the pattern layer design adopts hierarchical classification to determine the system's label types, relationship sets, and label node sets.

[0015] Furthermore, in step S2, the data collection, information extraction, and data preprocessing from the data source specifically include the following sub-steps:

[0016] S21, Data source acquisition;

[0017] S22, Information Extraction;

[0018] S23, Data Preprocessing: This includes data cleaning and knowledge fusion. Data cleaning is used to remove data that does not meet the requirements. The knowledge fusion step achieves the linking and merging of knowledge graphs by integrating knowledge from data from different sources, using entity alignment, entity disambiguation, and merging of duplicate knowledge.

[0019] Furthermore, in step S3, storing the semantic relationships between nodes in the graph database specifically includes: storing the semantic relationships corresponding to terms in the Neo4J graph database, storing the multimodal information of terms in the SQL database, and connecting the two through the term ID.

[0020] Furthermore, following step S3, a knowledge update step is also included, specifically comprising:

[0021] To address the issue of content addition, a "Add Entry" function has been introduced, allowing users to create new entries, including entry name, category, content, and semantic relationships.

[0022] It records user-input search questions and, based on the update cycle, dynamically updates the graph database by manually extracting triples from all recorded user search questions; it also shows the relationship between questioners and questions, providing big data support for the recommended term function.

[0023] To address content optimization issues, a "Term Version Management" function has been introduced, which includes the person who made changes to each term version, the term operation record of the content changes, and allows administrators to select a specific version as the term content; a "Term Management" function has also been introduced, which includes the management of term content and term information;

[0024] To address the issue of deleted entries, a "delete entry" function has been introduced, with administrators having the necessary permissions.

[0025] Furthermore, in step S21, the data source includes accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing-out reports, quality cases, software testing information system data, and field problems; wherein, the accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing-out reports, and quality cases are all unstructured data, the software testing information system data is semi-structured data, and the field problems are structured data.

[0026] Furthermore, in step S22, for structured data, the triples are stored in the knowledge graph after ensuring their format is correct; for semi-structured data, a wrapper is used to convert it into triples before storing it in the knowledge graph; for unstructured data, the processing specifically includes the following sub-steps:

[0027] Based on the text information description and in accordance with the design of the top-level pattern layer, the triplet relationship is extracted by combining manual extraction and high-order semantic models. That is, the text pre-trained model BERT is used as the text encoder, combined with Bi-LSTM bidirectional long short-term memory network and CRF conditional random field to form BERT+Li-LSTM+CRF semantic model, which is used to identify triplet relationships in unstructured word text. In terms of training data construction, the model is trained directly using manually labeled training set, and the Doccano tool is used to label the expressions.

[0028] Using the representation tagging method, after BERT modeling, input text information and insert special markers at the beginning and end positions of entities. The BIO three-dimensional tagging method is used: each element is labeled as "BX", "IX", or "O". Among them, BX indicates that the fragment in which the element is located belongs to type X and the element is at the beginning of the fragment, IX indicates that the fragment in which the element is located belongs to type X and the element is in the middle of the fragment, and O indicates that it does not belong to any type, so as to determine the category of the text.

[0029] Furthermore, in step S23, for structured and semi-structured data, after data preprocessing, the required triples are extracted directly according to the schema layer design and concept knowledge graph theory; for unstructured data, the text information is first extracted using the constructed Bert+Li-LSTM+CRF semantic model, a manual optimization step is added, and then data preprocessing is performed and triples are extracted according to the schema layer design concept and concept knowledge graph theory.

[0030] A multimodal knowledge graph construction device for military software defects, the device comprising a processor and a memory, the memory storing a computer program which, when loaded by the processor, executes the method described in any of the preceding claims.

[0031] A system for constructing a multimodal knowledge graph of military software defects, comprising the apparatus described above.

[0032] The beneficial effects of this invention include:

[0033] The military software defect knowledge graph constructed using the present invention, which combines data-driven and knowledge-guided approaches, transforms the traditional knowledge management method. It comprehensively displays rich semantic and other information, addressing the problems of fragmented and underutilized military software defect knowledge resources, high knowledge acquisition costs, and difficulties in updating. This lays the foundation for applications such as military software defect encyclopedia retrieval, intelligent question answering, intelligent prediction, and intelligent recommendation.

[0034] The data sources for this invention are diverse. There are eight sources of data for this invention: accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing reports, quality cases, software testing information system data, and field problems.

[0035] This invention employs a top-down schema layer design method, resulting in high-quality knowledge graphs with low knowledge construction barriers. First, domain experts and other industry professionals establish the top-level schema layer. Then, the data layer's conceptual representations are added to the predefined schema layer, ensuring the accuracy of the schema system. The system constructs a multimodal knowledge network of military software defects, tagged with concepts, representations, and term entities. Concepts include defects, development languages, operating platforms, development environments, defect domains, software types, problem phenomena, problem types, root cause analysis, solutions, introduction stages, and discovery timing. This allows data entry personnel to easily add data without needing prior knowledge of the knowledge graph, facilitating later data maintenance. New data can be divided into two parts: the first part is the representation, where each representation can belong to a concept, thus establishing an implicit connection between the new data; the second part is the multimodal information of term entities, enriching the multimodal information of the defects. This method, while ensuring the quality of the knowledge network, lowers the barrier to knowledge graph construction, enabling the collective effort to build richer knowledge graphs.

[0036] This invention constructs a professional knowledge graph through a human-machine collaborative approach to mining professional knowledge, employing diverse methods to acquire semantic relationships. Domain knowledge graphs possess certain professional characteristics, requiring relevant background knowledge to determine corresponding attribute dimensions; otherwise, the system's uncontrollability increases. Therefore, this system adopts different processes for different data sources and data structures. First, structured and semi-structured data are imported into the system after preprocessing. For textual semantic information such as user behavior logs, user additions after expert review, and some unstructured data sources, a pre-trained text model BERT (Bidirectional Encoder Representation from Transformers) is used as the text encoder, combined with a Bi-LSTM (Bidirectional Long Short-Term Memory) network and a CRF (Conditional Random Field) network (hereinafter referred to as BERT+Bi-LSTM+CRF), using natural language processing models for information extraction and manual optimization, before being imported into the system after preprocessing. Simultaneously, after the data is imported into the system, semantic relationships such as synonym / hypernyo relationships, concept attribute relationships, and concept inheritance relationships are fully utilized to establish a more human-like knowledge network.

[0037] This invention preserves the multimodal information inherent in military software deficiencies. Traditional graph embedding methods often overlook the multimodal information within knowledge graphs. For example, in the data bus knowledge domain, the term 1553B data bus contains rich text information, images, PDF learning materials, and other non-simple graph node-type knowledge. Existing graph embedding methods encounter problems when applied to multimodal graph representation because, in multimodal scenarios, the meaning of edges in the graph is no longer simply a semantic reasoning relationship, but rather a relationship that supplements multimodal information.

[0038] Trial results show that the multimodal knowledge graph in the field of military software defects constructed by the method of the embodiments of the present invention has reliable and diverse data sources, accurate semantic relationships, rich modal information, high-quality knowledge network, and low system maintenance cost. It lays the foundation for applications such as military software defect encyclopedia retrieval, intelligent question answering, intelligent prediction, and intelligent recommendation, and can provide reference value for the research and development and testing processes of other military software. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a framework diagram of a deep learning-based method for constructing a multimodal knowledge graph of military software defects, according to an embodiment of the present invention.

[0041] Figure 2 This is a flowchart of the human-machine integrated professional knowledge mining technology according to an embodiment of the present invention;

[0042] Figure 3 This is a data annotation diagram for unstructured text information extraction according to an embodiment of the present invention. Detailed Implementation

[0043] All features disclosed in all embodiments of this specification, or steps in all methods or processes implied in the disclosure, may be combined and / or extended or replaced in any way, except for mutually exclusive features and / or steps.

[0044] In this invention, a graph database establishes relationships between software defects and descriptive tags such as model number, software, hardware, equipment, components, and defect phenomena. The relational data stores multimodal information about software defects, including rich text, images, documents, audio, and video, showcasing a wealth of information associated with software defects from multiple perspectives. By constructing a multimodal knowledge graph in the field of military software defects, it lays the foundation for applications such as military software defect encyclopedia retrieval, intelligent question answering, intelligent prediction, and intelligent recommendation, and can provide valuable reference for the research and development and testing processes of other military software. Simultaneously employing natural language processing and knowledge graph technologies, and fully leveraging semantic relationships such as synonym / hypernyo-hypernyo relationships, concept attribute relationships, and concept succession relationships, a multimodal knowledge network for military software defects was constructed through processes including data collection from eight data sources, information extraction using two methods, data preprocessing using two methods, and data entry into multiple databases with different modalities. This network includes three types of tags: concepts, descriptions, and term entities. It can push military software defects as multimodal information term entities with intricate connections to relevant personnel. Testers can use it to inspire test case design ideas, developers can use it to improve code structure, and requirements personnel can use it to assist in requirements analysis.

[0045] In a further embodiment, this invention provides a method for constructing a multimodal knowledge graph of military software defects based on deep learning, the framework of which is shown in the figure below. Figure 1 As shown, this involves a top-down schema layer design, fully utilizing semantic relationships such as synonym / hyper-hypernyic relationships, concept attribute relationships, and concept succession relationships to construct a conceptual knowledge graph. Then, a human-machine combined professional knowledge mining method is used to obtain semantic relationship data of knowledge graph triples. Finally, multi-database connections are established to construct a multimodal military software defect knowledge graph. The specific methods are described below:

[0046] 1. Pattern Layer Design

[0047] This invention's system uses a top-down approach to construct a knowledge graph. In the schema layer design, a top-level schema layer is first established by domain experts and other industry professionals. Then, the concepts from the data layer are added to the pre-defined schema layer to ensure the accuracy of the schema system. The schema layer concept design in this invention's system adopts a hierarchical classification approach, including categories such as defects, development languages, operating platforms, development environments, defect domains, software types, problem phenomena, defect types, root cause analysis, solutions, introduction stages, and discovery timing. Subcategories within these broad categories are often exhaustive; for example, the defect category includes typical defects, field problems, zero-based reports, and quality cases, while the defect type category includes software problems, process problems, circuit design problems, structural design problems, and component quality problems.

[0048] 2. Conceptual Knowledge Graph

[0049] The construction of a concept knowledge graph specifically includes the following:

[0050] (1) Mining of conceptual synonyms / hyper-hyper-parameters. Semantic synonyms / hyper-hyper-parameters, such as "national standard - synonym - national standard", etc. The concept taxonomy system covering "what is", which is also a kind of hyper-hyper-parameter relationship, is therefore included in the synonym / hyper-hyper-parameter relationship.

[0051] (2) Conceptual Attribute Relationships. Concepts are described and defined from various attribute dimensions, such as defect-discovery time-June 2020, etc. Conceptual attribute relationships include two types. Predefined conceptual attributes: that is, typical attributes under predefined concepts. For example, a defect includes discovery time, discovery timing, repair time, operating platform, development language, development environment, defect domain, software type, problem type, problem phenomenon, cause analysis, solution, integration development path, etc. Open conceptual attributes: in addition to predefined public conceptual attributes, this invention also mines some specific attribute words from the text to supplement some specific attribute words. For example, impact, enlightenment, and suggestions, etc. For example, the text "This problem caused all 30 sets of receiving control boards to be reworked, delaying the progress by several months and causing economic losses of hundreds of thousands of yuan." can be used to mine the impact attribute of the defect.

[0052] (3) Conceptual succession relationship. Based on the top-level model layer established by domain experts and other industry professionals, the triple relationship is mined from the text information.

[0053] 3. Human-machine collaborative professional knowledge mining

[0054] This invention's system constructs a professional knowledge graph through a human-machine collaborative approach to professional knowledge mining. In this human-machine collaborative professional knowledge mining, the domain knowledge graph possesses certain professional characteristics, requiring relevant background knowledge to determine the corresponding attribute dimensions; otherwise, it will increase the system's uncontrollability. Therefore, this invention's system adopts different processes for different data sources and data structures, using a combination of intelligent models, expert review, and manual optimization to construct the knowledge graph. Figure 2 As shown, it specifically includes the following:

[0055] (1) Data source

[0056] The knowledge graph dataset in this embodiment of the invention contains diverse data sources, including structured, semi-structured, and unstructured data, specifically as follows: a) accurate industry data systems or documents provided by domain experts; b) user behavior logs; c) newly added user data; d) typical defect sets; e) zeroing reports; f) quality cases published on the network; g) relevant data obtained from the software testing information management system; h) field problems. Where a)-f) are unstructured data, g) is semi-structured data, and h) is structured data, as shown in Table 1. User-added data requires expert review and approval before processing.

[0057] Table 1

[0058]

[0059] (2) Information extraction

[0060] Of the three data structure types, structured data is the simplest to process; it only requires ensuring the triples are in the correct format before being stored in the knowledge graph. Semi-structured data is transformed into triples using wrappers. Unstructured data accounts for the largest proportion and has the most complex processing procedure.

[0061] Because domain knowledge graphs involve multiple layers and deep, fine-grained concepts, achieving quality requires a high degree of expert involvement and has a low degree of automation. Therefore, in the system of this invention, knowledge mining of unstructured data features mainly involves extracting triplet relationships based on textual information descriptions, following the design of the top-level pattern layer, and combining manual extraction with high-order semantic models. Specifically, the system uses the pre-trained text model BERT (Bidirectional Encoder Representation from Transformers) as the text encoder, combined with a Bi-LSTM (Bidirectional Long Short-Term Memory) network and a CRF (Conditional Random Field) to identify triplet relationships in unstructured term text. In terms of training data construction, since sentences expressing hyponymous relationships are very sparse, manually labeled training sets are directly used to train the model, and Doccano is used to label the expressions.

[0062] Using the representation tagging method, after BERT modeling, input text information is processed by inserting special markers at the beginning and end positions of entities. A three-dimensional tagging method, BIO (B-begin, I-inside, O-outside), is used: each element is labeled as "BX", "IX", or "O". Here, BX indicates that the element belongs to type X and is at the beginning of the segment; IX indicates that the element belongs to type X and is in the middle of the segment; and O indicates that it does not belong to any type, thus determining the text's category. The model structure is as follows... Figure 3 As shown, when the input text message "short circuit between chip pins" is entered, the system identifies the pins as devices and the short circuit as the phenomenon.

[0063] (3) Data preprocessing

[0064] For structured and semi-structured data, data preprocessing can be performed, followed by extraction of the required triples according to schema layer design and concept knowledge graph theory. For unstructured data, a BERT+Li-LSTM+CRF semantic model is first used to extract triple relationships from the text information. To improve the accuracy of relevant knowledge and compensate for model recognition errors or omissions, a manual optimization step is added before further data preprocessing and triple extraction according to schema layer design principles and concept knowledge graph theory.

[0065] The data preprocessing methods used in the system of this invention mainly involve data cleaning and knowledge fusion.

[0066] Data cleaning involves filtering coarse-grained triples, removing data that does not meet the requirements, and generating fine-grained triples.

[0067] Knowledge fusion is a step in the knowledge graph construction process. Through knowledge fusion, entities within a knowledge graph can be simplified, making its operation more efficient. Simultaneously, by fusing knowledge from data from different sources, links and mergers between knowledge graphs can be achieved, thereby constructing a larger-scale knowledge graph system with a wider service scope. Knowledge fusion includes three aspects: entity alignment, entity disambiguation, and merging duplicate knowledge.

[0068] By exploring the data characteristics of different knowledge data sources and extracting effective information from different data structure types, a large number of triple relationships that conform to the design concept of the schema layer and the theory of concept knowledge graphs are generated, which can then form a professional knowledge graph for the field of military software defects.

[0069] 4. Multimodal knowledge graph

[0070] Traditional graph embedding methods often overlook the multimodal information in knowledge graphs. For example, in the knowledge domain of data bus, the term 1553B data bus contains rich text information, images, PDF learning materials, and other non-simple graph node-type knowledge. Existing graph embedding methods have some problems when applied to multimodal graph representation because, in multimodal scenarios, the meaning of edges in the graph is no longer a simple semantic reasoning relationship, but a relationship that supplements multimodal information.

[0071] Therefore, this invention proposes storing semantic relationships corresponding to terms in a Neo4J graph database and multimodal information of terms, such as rich text, images, WORD, PPT, PDF, audio, and video, in an SQL database. The two are linked by the term's ID. That is, the multimodal knowledge graph consists of the following two parts: Neo4J graph database: entities and relationships; SQL database: rich text, images, WORD, PPT, PDF, audio, and video. The information value of the multimodal knowledge graph includes: rich text information of terms, term images, learning materials references, learning videos, attachments, etc.

[0072] 5. Knowledge Update

[0073] The knowledge extraction process relies on experts in the relevant domain. Following the design of the top-level model layer, it combines manual extraction of triples with the capabilities of higher-order models. Therefore, the standardization and completeness of entries are relatively poor, and the types of relationships and attributes are limited, failing to meet actual needs and requiring continuous optimization. Furthermore, some domain entries generate new knowledge over time, such as software defects. Therefore, knowledge graphs are not static but dynamically perceptive and require an update mechanism. This invention's embodiment proposes different solutions for content addition, input recording, content optimization, and entry deletion.

[0074] (1) To address the issue of content addition, the "Add New Entry" function is introduced, allowing users to create new entries, including entry name, entry category, entry content, and semantic relationships.

[0075] (2) Record the search questions entered by users. According to the update cycle, manually extract triples from all the questions searched by all users and dynamically update the graph database content. It has the relationship between the questioner and the question, and provides big data support for the recommended term function.

[0076] (3) Addressing optimization issues. A "Term Version Management" function is introduced, which includes records of term operations for each version, such as who made the changes and the content of the changes. Administrators can freely select a version as the content of a term. A "Term Management" function is also introduced, which includes the management of term content and term information, such as term name and term category.

[0077] (4) Regarding the removal of entries. A "delete entry" function is introduced, which can only be operated by administrators.

[0078] In summary, the solutions of the embodiments of the present invention can effectively improve the update efficiency of knowledge graphs.

[0079] It should be noted that, within the scope of protection defined in the claims of this invention, the following embodiments can be combined and / or extended or replaced in any logical manner from the above specific embodiments, such as the disclosed technical principles, disclosed technical features or implicitly disclosed technical features.

[0080] Example 1

[0081] A method for constructing a multimodal knowledge graph of military software defects includes the following steps:

[0082] S1, Pattern Layer Design;

[0083] S2, Defect Knowledge Mining: Data collection, information extraction, and data preprocessing from data sources;

[0084] S3, Multimodal Knowledge Graph Construction: Data is entered into multiple databases, semantic relationships between nodes are stored in a graph database, and multimodal information of graph database nodes is stored in a relational database to construct a multimodal knowledge graph.

[0085] Example 2

[0086] Based on Example 1, in step S1, the pattern layer design adopts a top-down pattern layer design approach. First, domain experts and other industry professionals establish the top-level pattern layer themselves, and then the data layer's representation concepts are added to the already defined pattern layer to ensure the accuracy of the pattern system. Furthermore, the pattern layer design adopts hierarchical classification to determine the system's label types, relationship sets, and label node sets.

[0087] Example 3

[0088] Based on Example 1, step S2, which involves collecting data from the data source, extracting information, and preprocessing data, specifically includes the following sub-steps:

[0089] S21, Data source acquisition;

[0090] S22, Information Extraction;

[0091] S23, Data Preprocessing: This includes data cleaning and knowledge fusion. Data cleaning is used to remove data that does not meet the requirements. The knowledge fusion step achieves the linking and merging of knowledge graphs by integrating knowledge from data from different sources, using entity alignment, entity disambiguation, and merging of duplicate knowledge.

[0092] Example 4

[0093] Based on Example 1, in step S3, storing the semantic relationships between nodes in the graph database specifically includes: storing the semantic relationships corresponding to terms in the Neo4J graph database, storing the multimodal information of terms in the SQL database, and connecting the two through the term ID.

[0094] Example 5

[0095] Based on Example 1, after step S3, a knowledge update step is also included, specifically including:

[0096] To address the issue of content addition, a "Add Entry" function has been introduced, allowing users to create new entries, including entry name, category, content, and semantic relationships.

[0097] It records user-input search questions and, based on the update cycle, dynamically updates the graph database by manually extracting triples from all recorded user search questions; it also shows the relationship between questioners and questions, providing big data support for the recommended term function.

[0098] To address content optimization issues, a "Term Version Management" function has been introduced, which includes the person who made changes to each term version, the term operation record of the content changes, and allows administrators to select a specific version as the term content; a "Term Management" function has also been introduced, which includes the management of term content and term information;

[0099] To address the issue of deleted entries, a "delete entry" function has been introduced, with administrators having the necessary permissions.

[0100] Example 6

[0101] Based on Example 3, in step S21, the data source includes accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing-out reports, quality cases, software testing information system data, and field problems; wherein, the accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing-out reports, and quality cases are all unstructured data, the software testing information system data is semi-structured data, and the field problems are structured data.

[0102] Example 7

[0103] Based on Example 3, in step S22, for structured data, the triples are stored in the knowledge graph after ensuring that the triple format is correct; for semi-structured data, a wrapper is used to convert it into triple form before storing it in the knowledge graph; for unstructured data, the processing specifically includes the following sub-steps:

[0104] Based on the text information description and in accordance with the design of the top-level pattern layer, the triplet relationship is extracted by combining manual extraction and high-order semantic models. That is, the text pre-trained model BERT is used as the text encoder, combined with Bi-LSTM bidirectional long short-term memory network and CRF conditional random field to form BERT+Li-LSTM+CRF semantic model, which is used to identify triplet relationships in unstructured word text. In terms of training data construction, the model is trained directly using manually labeled training set, and the Doccano tool is used to label the expressions.

[0105] Using the representation tagging method, after BERT modeling, input text information and insert special markers at the beginning and end positions of entities. The BIO three-dimensional tagging method is used: each element is labeled as "BX", "IX", or "O". Among them, BX indicates that the fragment in which the element is located belongs to type X and the element is at the beginning of the fragment, IX indicates that the fragment in which the element is located belongs to type X and the element is in the middle of the fragment, and O indicates that it does not belong to any type, so as to determine the category of the text.

[0106] Example 8

[0107] Based on Example 7, in step S23, for structured and semi-structured data, after data preprocessing, the required triples are extracted directly according to the schema layer design and concept knowledge graph theory; for unstructured data, the text information is first extracted using the constructed Bert+Li-LSTM+CRF semantic model, a manual optimization step is added, and then data preprocessing is performed and triples are extracted according to the schema layer design concept and concept knowledge graph theory.

[0108] Example 9

[0109] A multimodal knowledge graph construction device for military software defects, the device includes a processor and a memory, the memory storing a computer program, which, when loaded by the processor, executes the method described in any one of Examples 1 to 8.

[0110] Example 10

[0111] A system for constructing a multimodal knowledge graph of military software defects, the system comprising the apparatus described in Example 9.

[0112] The units described in the embodiments of the present invention can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0113] According to one aspect of the present invention, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the methods provided in the various optional implementations described above.

[0114] In another aspect, embodiments of the present invention also provide a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.

Claims

1. A method for constructing a multimodal knowledge graph of military software defects, characterized in that, Includes the following steps: S1, Pattern Layer Design; S2, Defect Knowledge Mining: Data collection, information extraction, and data preprocessing of the data source; In step S2, the data collection, information extraction, and data preprocessing of the data source specifically include the following sub-steps: S21, Data source acquisition; S22, Information Extraction; S23, Data Preprocessing: This includes data cleaning and knowledge fusion, where data cleaning is used to remove data that does not meet the requirements; The knowledge fusion step achieves the linking and merging of knowledge graphs by fusing knowledge from data from different sources, using entity alignment, entity disambiguation, and merging of duplicate knowledge. In step S22, for structured data, the triples are stored in the knowledge graph after ensuring they are in the correct format; for semi-structured data, a wrapper is used to convert it into triples before storing it in the knowledge graph; for unstructured data, the processing includes the following sub-steps: Based on the text information description and in accordance with the design of the top-level pattern layer, the triplet relationship is extracted by combining manual extraction and high-order semantic models. That is, the text pre-trained model BERT is used as the text encoder, combined with Bi-LSTM bidirectional long short-term memory network and CRF conditional random field to form BERT+Li-LSTM+CRF semantic model, which is used to identify triplet relationships in unstructured word text. In terms of training data construction, the model is trained directly using manually labeled training set, and the Doccano tool is used to label the expressions. Using the representation tagging method, after BERT modeling, input text information and insert special markers at the beginning and end positions of entities. The BIO three-dimensional tagging method is used: each element is labeled as "BX", "IX" or "O". Among them, BX indicates that the segment in which the element is located belongs to type X and the element is at the beginning of the segment, IX indicates that the segment in which the element is located belongs to type X and the element is in the middle of the segment, and O indicates that it does not belong to any type, so as to determine the category of the text. S3, Multimodal Knowledge Graph Construction: Data is entered into multiple databases, semantic relationships between nodes are stored in a graph database, and multimodal information of graph database nodes is stored in a relational database to construct a multimodal knowledge graph.

2. The method for constructing a multimodal knowledge graph of military software defects according to claim 1, characterized in that, In step S1, the pattern layer design adopts a top-down pattern layer design approach. First, industry professionals establish the top-level pattern layer themselves, and then the data layer's descriptive concepts are added to the predefined pattern layer to ensure the accuracy of the pattern system. Furthermore, the pattern layer design adopts hierarchical classification to determine the system's label types, relationship sets, and label node sets.

3. The method for constructing a multimodal knowledge graph of military software defects according to claim 1, characterized in that, In step S3, storing the semantic relationships between nodes in the graph database specifically includes: storing the semantic relationships corresponding to terms in the Neo4J graph database, storing the multimodal information of terms in the SQL database, and linking the two through the term ID.

4. The method for constructing a multimodal knowledge graph of military software defects according to claim 1, characterized in that, Following step S3, a knowledge update step is also included, specifically: To address the issue of content addition, a "Add Entry" function has been introduced, allowing users to create new entries, including entry name, category, content, and semantic relationships. It records user-input search questions and, based on the update cycle, dynamically updates the graph database by manually extracting triples from all recorded user search questions; it also shows the relationship between questioners and questions, providing big data support for the recommended term function. To address content optimization issues, a "Term Version Management" function has been introduced, which includes the person who made changes to each term version, the term operation record of the content changes, and allows administrators to select a specific version as the term content; a "Term Management" function has also been introduced, which includes the management of term content and term information; To address the issue of term deletion, a "delete term" function has been introduced, with administrators granted the necessary permissions.

5. The method for constructing a multimodal knowledge graph of military software defects according to claim 1, characterized in that, In step S21, the data source includes accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing-out reports, quality cases, software testing information system data, and field problems; wherein, the accurate industry data systems or documents provided by domain experts, user behavior logs, new user data, typical defect sets, zeroing-out reports, and quality cases are all unstructured data, the software testing information system data is semi-structured data, and the field problems are structured data.

6. The method for constructing a multimodal knowledge graph of military software defects according to claim 1, characterized in that, In step S23, for structured and semi-structured data, after data preprocessing, the required triples are extracted directly according to the schema layer design and concept knowledge graph theory; for unstructured data, the text information is first extracted using the constructed Bert+Li-LSTM+CRF semantic model, a manual optimization step is added, and then data preprocessing is performed and triples are extracted according to the schema layer design concept and concept knowledge graph theory.

7. A device for constructing a multimodal knowledge graph of military software defects, characterized in that, The device includes a processor and a memory, the memory storing a computer program that, when loaded by the processor, executes the method as described in any one of claims 1 to 6.

8. A system for constructing a multimodal knowledge graph of military software defects, characterized in that, The system includes the apparatus as described in claim 7.

Images