Detecting data leaks and / or detecting hazard information

Abstract

Aspects relate to computer-implemented methods, computer programs and computer systems for detecting data leaks and / or detecting hazardous information. The method includes receiving a knowledge graph and extracting data from at least one network service. The method also includes identifying a statement in the extracted data. For each identified statement, the method further includes determining, using a knowledge graph, whether the identified statement is public or private, and / or determining, using a knowledge graph, whether the identified statement is true or false.

Description

Technical Field

[0001] The disclosed technical field is information protection. Specifically, aspects disclosed relate to detecting data leaks in the context of information security, where sensitive or confidential data has been transmitted or otherwise distributed to unauthorized destinations. Additional aspects relate to detecting dangerous information that has become available to the public and may be harmful to people. Detecting such dangerous information may contribute to user safety. Background Art

[0002] The leaked data may have been distributed by a trusted agent or another insider. Traditionally, data leak detection is handled by watermarking, for example, embedding a unique code in each distributed copy. If a copy is subsequently found in the hands of an unauthorized party, the leaker can be identified. Another method of detecting leaked data is to use pattern matching in an intrusion detection system. Yet another method may include search operations on encrypted data.

[0003] In some cases, data leakage or dangerous information can be detected by interacting with a language model, such as a large language model based on a generative pre-trained transformer (such as ChatGPT). In some cases, a language model is a probability distribution over sequences of words. A large language model can have at least one million parameters or at least one billion parameters.

[0004] Dangerous information might include incorrect or erroneous instructions on the use of drugs, instructions on how to build a bomb that should not be made public, or incorrect instructions on how to build a machine that could cause harm.

[0005] Such a language model can be trained in an unsupervised manner on a large amount of unlabeled human-readable text. More specifically, the language model can be trained on at least one million, at least ten million, at least 100 million, or at least one billion samples. The samples can be documents or web pages. At least a portion of the samples can be publicly available or otherwise available to unauthorized users.

[0006] Additionally or alternatively, the language model may include a neural network. The neural network may be a deep neural network, for example, a neural network having one or more hidden layers. The neural network may have at least one million parameters (e.g., weights and biases) or at least one billion parameters. The neural network may have been trained on unlabeled (i.e., unannotated) text using unsupervised (i.e., self-supervised) learning.

[0007] Additionally or alternatively, the neural network can include a transformer that uses self-attention to differentially weight the importance of each portion of the input data provided to the neural network. The input to the neural network can be parsed into tokens, and the tokens can be processed simultaneously by calculating the weights of the tokens in successive layers of the neural network. The neural network can be designed to process sequential input data. The neural network can include weights that can be changed during runtime (e.g., soft weights).

[0008] Therefore, it may be desirable to interact with an application programming interface for a social media application or a language model in order to detect leaked data and / or detect dangerous information. Summary of the invention

[0009] According to one aspect, a computer-implemented method for detecting data leaks and / or detecting dangerous information is provided. The method includes receiving a knowledge graph and extracting data from at least one network service. The method also includes identifying statements in the extracted data. The method also includes, for each identified statement: determining using the knowledge graph whether the identified statement is public or private, and / or determining using the knowledge graph whether the identified statement is true (i.e., correct) or false (i.e., incorrect).

[0010] A knowledge graph may be associated with or belong to an enterprise. A knowledge graph may store knowledge about the skills, processes, experiences, capabilities, and insights relied upon in the day-to-day operations of an enterprise. The content of a knowledge graph may also include enterprise-specific acronyms, the enterprise's departments, and product specifications. The knowledge in a knowledge graph may enable an enterprise to react to business situations in a fast, professional, and flexible manner. The construction and maintenance of a knowledge graph may be expensive and labor-intensive.

[0011] A knowledge graph (i.e., a semantic web and / or a web of linked data) may be specified using the Resource Description Framework (RDF) or an extension of the Resource Description Framework such as RDF-Star (i.e., RDF*). RDF-Star enables descriptions to be added to edges in a knowledge graph, where the descriptions may include scores, weights, temporal aspects, and sources. RDF-Star enables the creation of statements about statements in a knowledge graph. Specifically, in addition to attaching statements to nodes in a knowledge graph, metadata may also be attached to describe edges in the knowledge graph. In this disclosure, references to RDF may be understood as references to RDF-Star.

[0012] A knowledge graph may include multiple statements. A knowledge graph may be implemented as a directed graph. Statements may be referred to as triples or triple statements having subject, predicate, and object components. Subjects, objects, and predicates may be referred to as components of their respective statements. The subject and / or predicate may be an RDF resource (e.g., the subject and / or predicate may have resource properties of RDF and may be a type or tag). An object may be a literal (e.g., an RDF literal) with a defined data type, such as a string, integer, Boolean, or double (as defined in the Extensible Markup Language (XML) Schema Definition Language (XSD)). For RDF, see the RDF specification "Concepts and Abstract Syntax", https: / / www.w3.org / TR / rdf11-concepts /

[0013] In some cases, the statements in the knowledge graph can be close to human language.

[0014] The following three examples may be included in the knowledge graph:

[0015] EmissionsManagement isABusinessCapability.

[0016] BusinessCapability rdfs:label "Business Capability".

[0017] EmissionsManagement SequenceID "5".

[0018] Additionally or alternatively, the following four statements may be included in the knowledge graph:

[0019] sap:PrintReceipt rdf:type sap:Task.

[0020] sap:Task rdfs:label "Task"@en.

[0021] sap:Task rdfs:label "Process Task"@en.

[0022] sap:Task rdfs:label "Schritt"@de.

[0023] Additionally or alternatively, the following four statements may be included in the knowledge graph:

[0024] sap:PrintReceipt sap:requires sap:Printer.

[0025] sap:PrintReceipt rdf:type sap:Task.

[0026] sap:PrintReceipt rdfs:label "Print Receipt".

[0027] sap:Printer rdfs:label "Printer".

[0028] In some cases, a knowledge graph includes multiple nodes connected by edges. Nodes can represent real-world entities, and edges can represent relationships between entities or relationships between entities and types of entities (i.e., classes). Therefore, predicates can be distinguished based on whether they connect two entities or connect entities and entity types. Entities can also be referred to as resources. For each statement, a subject can correspond to a node of a knowledge graph, an object can correspond to a different node of the knowledge graph, and an edge of the knowledge graph corresponding to the predicate can connect the subject node to the object node.

[0029] A node may have a corresponding class, such that each of the nodes has a corresponding class. The corresponding class may be part of (or organized in) a schema (i.e., a data model or ontology). The schema may be defined in RDF or a web ontology language. The following are examples of classes:

[0030] :State a rdfs:Class.

[0031] :EuropeanState a rdfs:Class.

[0032] :City a rdfs:Class.

[0033] Thus, ":State" is a resource that is a class (more specifically, an RDF class). The class ":EuropeanState" is another resource that is a class, more specifically, a subclass of ":State". Thus, a hierarchy of classes is possible. Furthermore, multiple inheritance is also possible.

[0034] Additionally or alternatively, the knowledge graph may be labeled and multi-relational. Thus, both nodes and edges may have labels, and edges may have directions. The object of a statement may be a label of the knowledge graph. The knowledge graph may be multi-relational in the sense that edges have different labels. The nodes of the knowledge graph may be subjects or objects, and the edges may be predicates.

[0035] In some cases, a schema includes attributes. Each of the attributes can be applied to at least one of the classes in the schema. At least one of the attributes can have a domain and / or a placement. Each of the attributes can be used by (or applied to) at least one statement. A domain (e.g., rdfs:domain) can specify a class to which a subject belongs, and a range (e.g., rdfs:range) can specify a class to which an object belongs. More specifically, a domain can specify a class to which a subject of a statement belongs, and a range can specify a class to which an object of a statement belongs. For RDF schemas, please refer to the W3C RDF Schema specification https: / / www.w3.org / TR / rdf-schema / .

[0036] The following are examples of properties:

[0037] rdf:type a rdf:Property

[0038] dbo:foundationPlace a rdf:Property.

[0039] :EuropeanState rdfs:subClassOf:State.

[0040] :locatedIn a rdf:Property.

[0041] :capitalOf a rdf:Property.

[0042] :capitalOf rdfs:subPropertyOf:locatedIn.

[0043] Thus, ":locatedIn" and ":capitalOf" are properties. Furthermore, ":capitalOf" is a sub-property of ":locatedIn". Thus, properties can also form a hierarchy. The property ":EuropeanState rdfs:subClassOf:State" indicates that ":EuropeanState" is a sub-class in a class hierarchy that includes the class ":State" and the sub-class ":EuropeanState".

[0044] Thus, a schema can provide a vocabulary for a knowledge graph. A knowledge graph can have predefined property prefixes that can indicate whether a node (i.e., subject or object) is an instance of a class or a class (e.g., if a node has a prefix "dbo" representing a DBPedia ontology, the node can be a class, and if a node has a prefix "dbr" representing a DBPedia resource, the node can be an instance). In some cases, a directed graph can use components of a URI to distinguish between instances and classes.

[0045] A knowledge graph may include statements (explicitly or implicitly) indicating that certain nodes are classes. Thus, whether a particular node represents an instance or a class may depend on the underlying model (i.e., the definition in the schema). For example, one can determine whether a node is a class (and included in the knowledge graph's schema) or an instance (and therefore not included in the knowledge graph's schema) by inspecting the rdf:type property. If rdf:type is owl:class, then the node is a class and is included in the schema; otherwise, the node is an instance (i.e., an instance of a class) and is not included in the schema.

[0046] Additionally or alternatively, statements from a knowledge graph cover multiple topical domains. Each statement can be identified by at least one uniform resource identifier (URI). At least one of a node and an edge can be identified by a URI or an internationalized resource identifier (IRI). More specifically, nodes and edges can each be identified by a URI or an IRI. In some cases, one or more of a subject, an object, and a predicate can be a URI. Some nodes (e.g., nodes corresponding to an object) can be identified via text rather than a URI. RDF can be used to represent a knowledge graph. The graph can have a structure similar to a known knowledge graph (such as DBPedia, Wikidata, BabelNet, DBkWik, Freebase, and DBnaray).

[0047] Compared to relational databases, knowledge graphs have a more flexible data structure because the type of data provided by the knowledge graph can vary. For example, the attributes associated with different instances can be different, even if these instances share the same class (e.g., "SAP_SE" and "BASF_SE" can have different attribute data, although they share the same class "Company"). A relational database can be represented in a knowledge graph format, i.e., a knowledge graph can be a higher-level abstraction of a relational database.

[0048] In some examples, the nodes of the knowledge graph can be organized in a hierarchical structure, where lower-level nodes (representing more specific objects) can be connected to higher-level nodes (representing more general objects) by one or more edges. Lower-level nodes (or lower-level objects they represent) can be descendants of higher-level nodes (or higher-level objects they represent), and higher-level nodes (or higher-level objects they represent) can be ancestors of lower-level nodes (or lower-level objects they represent).

[0049] The method may also include receiving one or more rules corresponding to the knowledge graph. The rules may be reasoning, logic, inference, or RDF pattern rules. The method may also include materializing the knowledge graph by applying the rules to the statements of the knowledge graph to calculate additional statements. Materializing the knowledge graph may be described as adding context data or references to context data to the knowledge graph. Materializing the knowledge graph may be implemented by applying reasoning or applying rules to the knowledge graph. Example numbering of rules is as follows:

[0050] 1. Each object of the predicate "dbo:foundationPlace" is a country

[0051] 2. Every subject of the predicate "capitalOf" is a city

[0052] 3. Every object of the predicate "capitalOf" is a country

[0053] The first rule can be implemented by setting the scope of the "dbo:foundationPlace" predicate so that its object must be an instance of the country class. The second rule can be implemented by setting the domain of the ":capitalOf" predicate so that its subject must be an instance of the city class. Similar to the first rule, the third rule can be implemented by setting the scope of the ":capitalOf" predicate so that its object must be an instance of the country class.

[0054] An example of reification (i.e., reasoning) is as follows. The reification is based on the following statement:

[0055] :Madrid:capitalOf:Spain.

[0056] And the following features:

[0057] :capitalOf rdfs:domain:City.

[0058] :capitalOf rdfs:range:Country

[0059] :capitalOf rdfs:subPropertyOf:locatedIn.

[0060] Thus, reification may include combining statements with one or more attributes. More specifically, reification may include combining statements with attributes that restrict the subject or object of the statement (e.g., attribute restrictions). These combinations may be used to determine further statements, such as classes of which the subject of the statement is an instance and / or classes of which the object of the statement is an instance. Reification may include determining statements that can be implicitly derived from the knowledge graph and adding the determined statements to the knowledge graph. The following are three numbered examples of reasoning:

[0061] 1.:Madrid:capitalOf:Spain.

[0062] :capitalOf rdfs:domain:City.

[0063] →:Madrid a:City.

[0064] 2.:Madrid:capitalOf:Spain.

[0065] :capitalOf rdfs:range:Country

[0066] →:Spain a:Country.

[0067] 3.:Madrid:capitalOf:Spain.

[0068] :capitalOf rdfs:subPropertyOf:locatedIn.

[0069] →:Madrid:locatedIn:Spain.

[0070] Each of the three examples above combines the statement ":Madrid:capitalOf:Spain" with a different property in order to compute (i.e., derive) an additional statement. In the first example, the statement ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:domain:City" to compute ":Madrid a:City", which indicates that the subject of the statement ":Madrid" belongs to the class ":City" (i.e., is an instance of the class ":City"). In the second example, ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:range:Country" to compute ":Spain a:Country", which indicates that the object of the statement ":Spain" is an instance of the class ":Country". In the third example, ":Madrid:capitalOf:Spain" is combined with the property ":capitalOf rdfs:subPropertyOf:locatedIn" to compute ":Madrid:locatedIn:Spain", which indicates that the subject ":Madrid" has the property ":locatedIn" with respect to the object ":Spain".

[0071] Continuing with the example, given the previous statements and properties of the reification, the following SPARQL Protocol and RDF Query Language (SPARQL) query will return FALSE:

[0072] ASK{:Madrid a:City.}

[0073] After materialization, the same SPARQL query will return TRUE. Materializing the knowledge graph may increase the effectiveness and / or usability of the knowledge graph for detecting data leaks and detecting dangerous information. The knowledge graph may be materialized before it is received.

[0074] Additionally or alternatively, the method may further include sorting the knowledge graph so that nodes of the knowledge graph are grouped together with their neighbors. The sorting may be performed after the knowledge graph is materialized. The sorting may include determining a list of nodes in the knowledge graph and adding a randomly selected node to the list of nodes. For each node in the new list of nodes, the connected nodes are determined. For each of the connected nodes, if the corresponding node is in the knowledge graph, the corresponding node is added to the list of nodes. The method may further include removing nodes from the list of nodes.

[0075] The following is pseudocode for the sorting algorithm described in the previous paragraphs:

[0076]

[0077]

[0078] The serialize() function above can represent the corresponding triples in the language ("r"-subject, "n.edge"-object, "n.node"-predicate). In other words, the serialize() function can convert statements from the knowledge graph into a serialization format such as RDF / XML, RDFa, Notation3 (.n3) or Turtle (.ttl), N-Triples, or JSON-LD.

[0079] The sorting algorithm can be called a clustering algorithm and can ensure topicality, i.e., the sorted statements are close to each other in the sense that they refer to similar topics or the same topic. In other words, adjacent statements are semantically similar. This can improve the effectiveness of the knowledge graph in detecting data leaks and detecting dangerous information.

[0080] In some cases, determining whether the identified statement is true or false using the knowledge graph includes determining whether the subject-predicate combination of the identified statement has a maximum cardinality, i.e., a maximum number of objects. When the identified statement has a maximum cardinality and the maximum cardinality of the subject-predicate combination of the identified statement is exceeded, the identified statement is determined to be false. For example, the maximum cardinality can be 1. When the subject-predicate combination has a maximum cardinality of 1, then the subject-predicate combination can be TRUE when it appears in combination with only one object, and can be false when combined with any other object.

[0081] Additionally or alternatively, the at least one network service may include one or more of the following: a web service, a social network service, a service that provides an interface to a language model. The network service may run on a server and listen for network requests on a port. The web service may use the Hypertext Transfer Protocol (HTTP) and may listen for network requests on ports 80 and / or 443.

[0082] The at least one network service may include one or more of: a web service, a social network service, a service that provides an interface to a language model. Extracting data from the at least one network service may include accessing at least one application programming interface via the at least one network service. Extracting data from the at least one network service may include web scraping using HTTP, wherein web scraping may include web crawling.

[0083] In the context of a social networking service, an API may provide access to settings and allow updates to settings. The API may also allow accessing and updating content, creating and getting posts, getting comments or insights, and performing other update actions. The API may also enable searching, filtering, and data extraction.

[0084] Web scraping can include extracting content from a website, particularly the underlying HTML code and / or data stored in a database. Web crawling can include systematically browsing web pages and indexing, updating, and retrieving web page content.

[0085] Examples of social networking services include Facebook, LinkedIn, and Twitter (or X). Examples of language models include ChatGPT, PaLM, Megatron, Titan, and Chinchilla.

[0086] The method may also include calculating a vector for each uniform resource identifier (URI) of the knowledge graph, so that each of the uniform resource identifiers of the knowledge graph has a corresponding vector. The method may also include creating a copy of the knowledge graph including the calculated vectors. The method may also include modifying the copy of the knowledge graph to include a number of true statements (i.e., true facts) equal to the number of false statements (i.e., false facts) in the copy of the knowledge graph. In the context of this disclosure, statements may also be referred to as facts.

[0087] The combination of a URI, vector, and endpoint identifier may be referred to as an embedding. In this context, an endpoint may be a URL that identifies a web service or API that can be used to retrieve information from a knowledge graph. The knowledge graph may be deployed on a graph database server, and the graph database server may have a software layer that exposes the web service or API. Example endpoints are as follows:

[0088] https: / / github.com / blazegraph / API

[0089] There can be a 1-1 correspondence between the knowledge graph and the endpoints.

[0090] The purpose of computing the vector for each URI of the knowledge graph may be to make the vector available for training a machine learning model. Computing the vector may be referred to as training embeddings. Training embeddings may be a machine learning task of learning a low-dimensional representation of the entities and relations (edges) of the knowledge graph while retaining the semantic meaning of the entities and relations. Embeddings may be used for link prediction, triple classification, entity recognition, clustering, and relationship extraction.

[0091] The purpose of modifying a copy of the knowledge graph so that the number of true statements equals the number of false statements may be to make the copy of the knowledge graph usable for training a machine learning model. Specifically, if the knowledge graph has only true statements or there is a significant difference between the number of true statements and the number of false statements, the machine learning model may not be trained correctly.

[0092] Modifying a copy of a knowledge graph to include a number of true statements equal to the number of false statements may include: when the copy of the knowledge graph has more false statements than true statements, randomly deleting false statements. When the copy of the knowledge graph has more true statements than false statements, adding false statements to the copy of the knowledge graph by randomly selecting and combining subjects, predicates, and objects from different statements in the knowledge graph. Each component of a statement in the copy of the knowledge graph may have a corresponding calculated vector. For each statement in the copy of the knowledge graph, the method may include calculating a similarity value using the calculated vector corresponding to the statement, the similarity value measuring the similarity of the statement to each of the other statements in the copy of the knowledge graph. The similarity value may be calculated using a cosine similarity measure.

[0093] Thus, a similarity value may be calculated by dividing the calculated dot product of the vectors by the product of the lengths of the vectors. Thus, a proportional vector may have a cosine similarity of 1, two orthogonal vectors may have a similarity of 0, and two opposite vectors may have a similarity of -1.

[0094] The vectors may be computed such that each relation (i.e., predicate) in the knowledge graph is represented by two vectors, the norm vector of the hyperplane and the translation vector on the hyperplane. The hyperplane may be perpendicular to the embedding space of the vectors. The vectors may be computed using a pure translation model (e.g., TransH, TransD, TransA, or TransE). Further information on pure translation models may be found in "Knowledge Graph Embedding by Translating on Hyperplanes", Wang et al., 2014.

[0095] The benefit of using TransH to calculate the vectors can be that there will be small distances between true statements (e.g., close to 0, where 0 is perfect similarity) and large distances between false statements (e.g., close to 2, where 2 is perfect dissimilarity). The distance can be a cosine distance (i.e., cosine similarity or 1-complement of cosine similarity). Close to 0 can be less than 0.1 or less than 0.2. Close to 2 can be greater than 1.8 or greater than 1.9.

[0096] The method may also include training a statistical model using the calculated similarity values ​​and information indicating whether the similarity values ​​correspond to true or false statements in the copy of the knowledge graph. The statistical model may be a logistic regression model. Determining whether the identified statement is true or false using the knowledge graph may include applying the statistical model to the identified statement. When the statistical model returns a probability of the identified statement greater than a specified threshold, the identified statement is determined to be true. When the statistical model returns a probability of the identified statement less than or equal to a specified threshold, the identified statement is determined to be false.

[0097] The method may also include identifying sentences in the extracted data, including performing a term frequency-inverse sentence frequency with stemming and lemmatization on the extracted data. For further details on the term frequency-inverse sentence frequency with stemming and lemmatization, see "Sentry retrieval using steming and lemmatization with different length of the queries", Boban et al., June 11, 2020. The method may also include identifying entities in the extracted data using named entity recognition. Named entity recognition can be a subtask of information extraction that seeks to locate and classify named entity mentions in unstructured text into predefined categories.

[0098] The method may also include identifying implicit relationships between the identified entities via relationship extraction. Relation extraction may include deriving structured text from the extracted data. The result of named entity recognition and relationship extraction may be one or more entity 1-predicate-entity 2 combinations.

[0099] The method may also include mapping the identified entity to a knowledge graph via entity linking. The mapping may include named entity disambiguation. The method may also include mapping the predicate of the identified entity 1-predicate-entity 2 combination to an attribute of the knowledge graph. The method may also include: based on the mapping of the identified entity to the knowledge graph in the mapping of the predicate to the attribute of the knowledge graph, providing at least one result statement, each result statement including a subject-predicate-object triple, each component of the subject-predicate-object triple including a uniform resource identifier of the knowledge graph.

[0100] Additionally or alternatively, determining whether the identified statement is true or false (eg, 0 or 1) and / or determining whether the identified statement is public or private includes comparing the identified statement to statements in the knowledge graph.

[0101] In some cases, comparing the identified statement to statements in the knowledge graph may include determining (e.g., finding) at least one comparable statement to the identified statement. The comparable statement to the identified statement may have at least one value in common with the identified statement. Specifically, the comparable statement to the identified statement may have at least one of a subject, a predicate, and an object in common with the identified statement. For example, a comparable statement may have the same subject as the identified statement, but a different predicate and a different object than the identified statement. Multiple statements in the knowledge graph may include metadata indicating whether the corresponding statement is (i.e., the statement is marked as) true or false, and / or multiple statements in the knowledge graph may include metadata indicating whether the statement is (i.e., the statement is marked as) public or private.

[0102] When one of the statements in the knowledge graph does not include metadata indicating whether the statement is true or false, the method may also include determining that the statement is true. Determining whether the identified statement is true or false may include identifying a maximum cardinality of at least one comparable statement, and determining whether the identified statement exceeds the maximum cardinality. In some cases, if the identified statement exceeds the maximum cardinality, the identified statement is determined to be false.

[0103] Each statement of the knowledge graph may include a subject-predicate-object triple, wherein each subject-predicate-object triple consists of a subject, a predicate, and an object. The subject, the predicate, and the object may each be represented (ie, identified) by a uniform resource identifier.

[0104] Extracting data from at least one web service may include randomly selecting statements from the knowledge graph. The method may also include constructing a query string based on the subject and predicate of the statement, and calling at least one application programming interface (API) via at least one web service using a uniform resource locator (URL) and using the query string.

[0105] Extracting data from at least one web service may include constructing a natural language processing question based on randomly (ie, pseudo-randomly) selected statements from the knowledge graph. The method may also include issuing a call to at least one application programming interface using the question.

[0106] According to a second aspect, a computer program is provided. The computer program may be tangibly embodied in a computer-readable medium. The computer program includes instructions, which, when executed by a computer, cause the computer to perform any one of the above methods.

[0107] According to a third aspect, a computer system for detecting data leakage and / or detecting dangerous information is provided. The computer system may include a memory and at least one processor. The memory may include instructions that, when executed, cause at least one processor to receive a knowledge graph and extract data from at least one network service. The instructions, when executed, may also cause at least one processor to identify statements in the extracted data. For each identified statement, the instructions, when executed, may cause at least one processor to use the knowledge graph to determine whether the identified statement is public or private, and / or use the knowledge graph to determine whether the identified statement is true or false.

[0108] The subject described in this disclosure may be implemented as a method or on a device, and may be implemented in the form of one or more computer programs (eg, computer program products). Such computer programs may enable a data processing device to perform one or more operations described in this disclosure.

[0109] The subject matter described in this disclosure can be implemented in a data signal or on a machine-readable medium, wherein the medium is embodied in one or more information carriers, such as a CD-ROM, a DVD-ROM, a semiconductor memory or a hard disk. In particular, the disclosed subject matter can be tangibly embodied in a non-transitory machine (computer) readable medium.

[0110] In addition, the subject matter described in this public can be implemented as a system including a processor and a memory coupled to the processor. The memory can encode one or more programs so that the processor performs one or more of the methods described in this application. The other subject matter described in this public can be implemented using various machines.

[0111] The details of one or more implementations are set forth in the exemplary drawings and description below. Other features will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0112] Figure 1 An exemplary knowledge graph is shown.

[0113] Figure 2 The domain and scope of attributes in the schema of the knowledge graph are shown.

[0114] Figure 3 An exemplary SPARQL query and the results of the query are shown.

[0115] Figure 4 Block diagram symbols for subsequent figures are shown.

[0116] Figure 5 Steps of a method that may be performed according to some embodiments are shown.

[0117] Figure 6 Further steps of the method are shown which may be performed according to some embodiments.

[0118] Figure 7 A computer system according to some embodiments is shown.

[0119] Figure 8 A statement inspection engine that may be included in a computer system is shown.

[0120] Fig. 9 Shows Figure 7 Example output of the system.

[0121] Fig.10 A logical data model according to some embodiments is shown.

[0122] Fig.11 A computer system in a cloud configuration is shown.

[0123] Fig.12 is an exemplary computer system configuration for implementing some embodiments. DETAILED DESCRIPTION

[0124] Hereinafter, a detailed description of the examples will be given with reference to the accompanying drawings. Various modifications may be made to the examples. In particular, one or more elements of one example may be combined and used in other examples to form new examples.

[0125] Figure 1 A knowledge graph 100 is shown. The nodes of the knowledge graph 100 are shown as circles, and the edges of the knowledge graph are shown as arrows. Therefore, the knowledge graph 100 is a directed graph. The knowledge graph 100 includes node labels 102, 104, 106, 108, 110, where the node labels (i.e., labels) are URIs and are defined in the Resource Description Framework (RDF). Node labels 102 and 104 are objects, and edge labels 106, 108, and 110 are predicates. The string "1972-01-01" can also be a node label (i.e., object) with an xsd:date type.

[0126] The knowledge graph 100 includes a statement 112 (i.e., a triple statement) having a subject "dbr:SAP_SE", a predicate "dbo:foundationPlace", and an object "dbr:Germany", each of which is a URI defined in RDF. An exemplary serialization of the statement 112 is dbr:SAP_SE dbo:foundationPlace dbr:Germany. The schema of the knowledge graph 100 may be defined via RDF Schema (RDFS) or Web Ontology Language (OWL) from the World Wide Web Consortium (W3C).

[0127] Figure 2 The domain 201 and scope 203 of the attribute ":capitalOf" are shown. The domain 201 and scope 203 may be defined as follows:

[0128] :capitalOf rdfs:domain:City.

[0129] :capitalOf rdfs:range:Country.

[0130] Figure 3 An exemplary SPARQL query 301 of the knowledge graph 100 is shown. The query 301 is configured to determine an answer to the following question: What is the population of Chinese cities with more than 100,000 residents? The results of executing the query 301 are also shown.

[0131] Figure 4 Block diagram symbols used in subsequent figures are shown. System component 401 is shown as a rectangle. System components may also be shown as ovals. Storage device 403 is shown as a cylinder with an oval outline. Requests between system components 405 are shown as arrows including circles. Writes between system components and storage devices are shown at 407. Reads between system components and storage devices are shown at 409. Reads and writes between system components and storage devices are shown at 411.

[0132] Figure 5 Configuration steps that may be taken prior to receiving the knowledge graph 100 are shown. Figure 5 The configuration steps shown can be used as Figure 7 The configuration user interface and / or configuration API shown and discussed in more detail below can be performed. At 501, a knowledge graph 100 can be defined. More specifically, a way to connect to the knowledge graph 100 over a network can be defined. For example, an Internet Protocol (IP) address and a Transmission Control Protocol (TCP) port for connecting to the knowledge graph 100 can be defined. The knowledge graph 100 can be a company or organization knowledge graph. At 503, a connection to a network service can be defined. More specifically, a connection to an API accessible via a network service can be defined. At 505, parameters can be defined. Parameters can determine the periodicity, warning level, or threshold of calls to the API.

[0133] Figure 6 Steps that may be performed according to some embodiments are shown. Figure 6 The steps shown can be performed by a statement checking engine, such as Figure 7 and 8As shown, and discussed in more detail below. At 601, a network service can be accessed (e.g., one of the network services to which the connection is defined at 503), and an API can be called (e.g., one of the APIs defined at 503). The API can be used to extract data via the network service. At 603, statements (i.e., facts) can be identified in the extracted data. At 605, the identified statements can be marked. More specifically, the knowledge graph 100 can be used to determine whether the identified statement is public or private and / or whether the identified statement is true or false. At 607, a warning can be generated for a false statement or a private statement. The warning can enable detection of data leakage, such as via the distribution of private statements and / or detection of dangerous information, such as via the distribution of false statements. Exemplary marked statements are shown in Table 1 below:

[0134] Statement Identifier Correctness Availability 27654398633567 Fake public 17354628935672 Fake public 10243568365493 real public 10243568365494 real private

[0135] Table 1

[0136] Figure 7 700 according to some embodiments. The system 700 may include a configuration user interface that may be used to perform the above-described Figure 5 The configuration user interface may include a computer and a monitor so that a user can interact with the system 700. The system 700 may include a configuration API that may allow remote access to the system 700 to perform configuration steps. The system 700 may include a configuration repository (store) that may enable storage of system configuration properties. The system 700 may include a statement inspection engine, such as regarding Figure 8 100. The system 700 may include a monitoring user interface and a monitoring API. The monitoring user interface may include a computer and a monitor so that a user can determine information about the health of the system 700, for example, to view logs and / or warnings generated by the system 700. The monitoring API may enable a user to remotely access the system 700 to view logs and / or warnings. The warning storage may enable storage of warning information. The knowledge graph 100 may be stored in a KG triple repository.

[0137] Figure 8 Shown in more detail Figure 7 The statement checking engine may include a scheduler. The scheduler may access information about Figure 5The scheduler may be configured to access a web service associated with a Facebook API every 15 minutes to extract data from the web service and identify statements in the extracted data. Each web service and / or API may be associated with a time period (e.g., 30 minutes), and data may be extracted via the web service according to the time period.

[0138] The API crawler can determine for each API whether the API has a standard type or a session type, and can use this information to issue a corresponding request to the API. The session type can correspond to an API for a language model. The standard type can correspond to other APIs, for example, for social networking services, web services, and search engine services. For example, if the API is standard, the API crawler can select statements from the knowledge graph 100 and use the label information of the subject and predicate of the statement to construct a query for the API. Statements can be randomly selected from the knowledge graph 100. A query can be used to call an API (e.g., an API for a social networking service).

[0139] If the API is conversational, the API crawler can construct natural language processing questions from randomly selected statements of the knowledge graph 100. Natural language processing questions can be constructed as described in European patent application entitled "Providing Data from a Directed Graph to a Language Model", Docket No. S15972EU, Application No. 23177538.8, filed with the European Patent Office on June 6, 2023. The API crawler can use natural language processing questions to issue calls to the API (e.g., an API for a language model). It can be based on the above description of Figure 5 The configuration settings provided in the discussion determine the URI of the API. An example call from the API crawler to the session API is as follows:

[0140] curl -X POST https: / / api.openai.com / abc / xyz\

[0141] -H"Content-Type:application / json"\

[0142] -H"Authorization:Bearer API_KEY_HERE"\

[0143] -d'{"prompt":"Is Business Objects a product from Microsoft?","max_tokens":100,"temperature":0.7,"n":

[0144] 1,"stop":"."}'

[0145] The response provided via the session API (e.g., from a language model) may be as follows:

[0146] {"choices":[{"text":"Yes, Microsoft has a product called BusinessObjects.

[0147] It is a business intelligence and reporting tool that was originally developed by the French company Business Objects and is now owned by Microsoft.

[0148] Based on the above responses, the following statements can be identified:

[0149] fact=('Microsoft','product or material produced','Business Objects')

[0150] The identified statements may be compared with statements in the knowledge graph to determine whether the identified statements are false.

[0151] The response obtained by the API crawler can be forwarded to the statement identifier of the statement inspection engine. The statement identifier can include multiple modules. Specifically, the statement identifier can include a sentence identifier module, a named entity recognition module, a relationship extraction module, an entity linking module, and a predicate mapping module. The sentence identifier module can identify sentences in the extracted data, for example, by performing a term frequency-inverse sentence frequency with stemming and semantics on the extracted data. The named entity recognition module can use named entity recognition to identify entities in the extracted data. Named entity recognition can be a subtask of information extraction that seeks to locate and classify named entities mentioned in unstructured text into predefined categories. The relationship extraction module can identify implicit relationships between the identified entities. The relationship extraction module can include deriving structured text from the extracted data. The relationship extraction module can generate multiple entity 1-predicate-entity 2 combinations. The predicate mapping module can map entity 1-predicate-entity 2 combinations to attributes of the knowledge graph. In the following algorithms 1 to 4, statements are referred to as facts.

[0152] The statement identifier can be operated according to Algorithm 1 as follows:

[0153] Input: Text T

[0154] Step 1: Use the sentence identifier module to split T into sentences

[0155] Step 2: For the sentences within the sentence:

[0156] Identify entities using the NER module

[0157] Identifying relations using the RE module

[0158] The result is a set of distinct entity1-predicate-entity2 combinations

[0159] Step 3: For entity 1-predicate-entity 2 in step 2:

[0160] Use the entity linking module to map entities to knowledge graph instances Use the predicate mapping module to map relations to knowledge graph attributes

[0161] The result is a set of facts F; each of type f = (s, p, o), s, p, o ∈ knowledge graph output: F

[0162] The input text of Algorithm 1 may be data extracted from at least one web service, for example, via an API accessible via the web service.The knowledge graph instances may be different or distinct knowledge graphs.

[0163] The statement checking engine may also include a statement tagger. The statement tagger may depend on a machine learning model. Figure 5The machine learning model is then trained after the configuration discussed. The operation of the statement tagger is shown in Algorithm 2 below. The output of Algorithm 2 includes multiple embeddings (one embedding for each resource (i.e., node or edge) in the knowledge graph), and multiple vectors (e.g., TransH vectors).

[0164] The output of Algorithm 2 can be stored for later use in Algorithm 3, which will be described in more detail below. Algorithm 2 can be implemented as follows:

[0165] Input: Image G

[0166] Step 1: Use the true facts of G to train each resource in G TransH Embed

[0167] Step 2: Create a copy G' of G and change it so that it includes an equal number of true and false facts:

[0168] If G' has more false facts than true facts:

[0169] Randomly remove false facts

[0170] otherwise:

[0171] Add fake facts by randomly selecting and combining subjects, predicates, and objects

[0172] Step 3: For each fact in G', F = (s, p, o) and the corresponding embedding (s e ,p e ,o e ),,calculate

[0173] Step 4: Using the cs values ​​calculated for each fact and the information of whether they are 1 – true or 0 – false, train a logistic regression model that predicts the probability of a fact being true or false given cs:

[0174]

[0175] Output: TransH vector, β0 and β1

[0176] As illustrated in step 1 of Algorithm 2, the goal of training embeddings is that, given a subject s and a predicate p, if s and p are added together, they produce a vector that is close to (e.g., a small cosine distance from) the embedding of the corresponding object o.

[0177] The goal of Algorithm 2 is to generate a vector for each resource (i.e., each node and each edge) of the knowledge graph, with some exceptions. For example, if a node or edge only appears in false facts, then the node or edge will not have an embedding. It is possible that only facts with resources that have embeddings will be used in Algorithm 3. In other words, facts with resources that do not have embeddings will not be used in machine learning (e.g., logistic regression) models.

[0178] s, p, and o can refer to subject, predicate, and object respectively. e 、p e , o e May refer to subject embedding, predicate embedding, and object embedding, respectively. cs may refer to cosine similarity. Therefore, step 1 of Algorithm 2 may involve calculating a vector for each uniform resource identifier of the knowledge graph 100, so that each of the uniform resource identifiers of the knowledge graph has a corresponding vector. Step 2 of Algorithm 2 may include creating a copy of the knowledge graph 100 including the calculated vector. Step 2 may also include modifying the copy of the knowledge graph 100 to include a number of true statements equal to the number of false statements. Step 3 of Algorithm 2 may include: for each statement in the copy of the knowledge graph, using the calculated vector corresponding to the statement to calculate a similarity value, which measures the similarity between the subject and predicate of the statement and the object of the statement in the copy of the knowledge graph. In other words, step three of Algorithm 2 may include calculating a similarity value that measures the similarity of ES+EP and EO, where ES is the subject embedding of subject S, EP is the predicate embedding of predicate P, and EO is the object embedding of object O. In the example shown above, the similarity is calculated using the cosine similarity measure.

[0179] Step 4 of Algorithm 2 may include training a statistical model using the calculated similarity values ​​from step 3 and information indicating whether the similarity value corresponds to a true statement or a false statement in the copy of the knowledge graph (i.e., whether the similarity value between the predicted object embedding and the actual object embedding corresponds to a true statement or a false statement in the copy of the knowledge graph). In the example above, the statistical model is implemented as a logistic regression model. Step 4 may output the probability of the identified statement.

[0180] Therefore, vectors β0 and β1 are trained so that, in general, when there is a subject S and a predicate P (for which the corresponding embeddings are ES and EP), then ES+EP should be "mathematically close" to EO (e.g., having a cosine distance close to 0 or a cosine distance less than 0.1 or less than 0.2), where EO is the embedding of O, and S, P, O are true statements (i.e., facts) in the knowledge graph.

[0181] For example:

[0182] S=SAP

[0183] P = MarketLeaderIn

[0184] ES=(1,1)

[0185] EP=(2,3)

[0186] Continuing with the example, the corresponding knowledge graph embedding may include the following.

[0187] Ex=(3.1,3.9)

[0188] Ey=(1013414,1352452)

[0189] Ez=(0,1)

[0190] Continuing with the example further, the knowledge graph resources associated with the embeddings Ex, Ey, and Ez may be as follows:

[0191] X=BusinessSoftware

[0192] Y=Oracle

[0193] Z=BusinessProcessData

[0194] Further operations of the statement tagger can be implemented according to Algorithm 3, for example as follows:

[0195] Input: graph G, fact f = (s, p, o) and corresponding embedding (s e ,p e ,o e ), model parameters β0 and β1

[0196] If F is in G and F is private:

[0197] Return to Private

[0198] If F is in G and F is false:

[0199] Return False

[0200] If there exists f′ in G, and f′=(s,p,x):

[0201] If Power:maxCardinality == 1:

[0202] Return False

[0203] if

[0204] Return true

[0205] Return False

[0206] In the example of the statement tagger above, f is a fact from the knowledge graph G, and f' is another fact from the knowledge graph G. f and f' have the same subject and predicate but different objects.

[0207] Thus, Algorithm 2 may implement machine learning model training to prepare statement labeling, and Algorithm 3 may implement statement labeling based on the machine learning model training performed in Algorithm 2. In Algorithm 3, G may be a knowledge graph (e.g., knowledge graph 100), and F may be an identified statement in data extracted from a web service using an API. β0 and β1 may be logistic regression parameters output by Algorithm 2.

[0208] Algorithm 3 includes two conditional statements that involve comparing the identified statement (i.e., fact F) to a statement in the knowledge graph 100. If the identified statement matches a statement in the knowledge graph, it is determined whether the statement in the knowledge graph is private or false. If the statement in the knowledge graph is private (e.g., the knowledge graph includes metadata indicating that the corresponding statement is private), Algorithm 3 returns "private". If the statement in the knowledge graph is false (e.g., the knowledge graph includes metadata indicating that the corresponding statement is false), Algorithm 3 returns "false". According to Algorithm 3, determining whether the identified statement is true or false includes identifying the maximum cardinality of comparable statements in the knowledge graph (e.g., statements with the same subject and predicate and different objects) and determining whether the identified statement exceeds the maximum cardinality. In the example of Algorithm 3, the maximum cardinality is 1.

[0209] When the probability of the identified statement is greater than a specified threshold (e.g., 0.5 as shown above in Algorithm 3), the identified statement can be determined to be true. When the probability of the identified statement is less than or equal to the specified threshold, the identified statement can be determined to be false.

[0210] Algorithm 3 may return false by default. This may be to increase security in the event of a leak being detected or to improve safety in the event of dangerous information being detected.

[0211] False or private statements detected by Algorithm 3 may result in the generation of corresponding warnings. These warnings can be found in Figure 7 is displayed to the user on the monitor.

[0212] Fig. 9 Shows Figure 7Output of system 700. False or private facts are marked as warnings and identified as work items. Facts can be searched, classified, and checked in further detail (e.g., facts can be provided as links so that they can be connected to the corresponding website for verification). The identified statements appear in the leftmost column under the heading "Fact". An indication of whether the corresponding statement is correct appears in the next column under the heading "Correctness". An indication of whether the corresponding statement is public or private appears under the heading "Availability". The rightmost column provides a date indicating when the data containing the identified statement was extracted.

[0213] Fig.10 1 shows a logical data model according to some embodiments. The logical data model may include multiple API configurations, for example, as described above in conjunction with Figure 5 As discussed. Each of the API configurations can enable data to be extracted from the corresponding network service. Each API configuration can include an API identifier (api_id), a URL (baseurl) that identifies the access point of the API, a type indicating how to communicate with the API (e.g., standard or session), a login (user), a password (pass), and a frequency (e.g., in minutes) indicating how often the API is called to extract data from the corresponding network service (e.g., a frequency of 15 minutes would indicate that the API should be called every 15 minutes). Each API can have at least one configuration. Exemplary API configurations are as follows:

[0214] api_id:chatgpt

[0215] base_url:https: / / api.openai.com / pqr / tuv

[0216] Type: Conversation

[0217] User:dummy_user

[0218] pass:************

[0219] Frequency: Every hour

[0220] In the above example, "user" corresponds to the login id, and "pass" corresponds to the password.

[0221] The logical data model may include multiple connection configurations. There may be exactly one connection configuration for each API configuration, and each connection configuration corresponds to at least one API configuration. Each connection configuration may include an endpoint identifier, an endpoint, a login, a password, and multiple APIs. Each connection configuration may correspond to a network service, and the endpoint may be a network service. The endpoint identifier may be a combination of an Internet Protocol address and a port number.

[0222] An example connection configuration is as follows:

[0223] endpoint_id:BlazeGraph

[0224] Endpoint: https: / / blazegraph.abc123.com / #query

[0225] User:dummy_user

[0226] pass:*******

[0227] API: chatgpt; twitter

[0228] Each connection configuration may correspond to at least one set of embeddings. Each set of embeddings may correspond to exactly one connection configuration. The combination of an endpoint identifier, a URI, and a vector may be referred to as an embedding container, where the vector may be an embedding. Embeddings may be computed based on the identified statements and compared to embeddings computed from the knowledge graph to determine whether the corresponding identified statement is true or false and / or to determine whether the corresponding identified statement is public or private. The embeddings shown in the logical data model may be embeddings that have been computed from the knowledge graph. Examples of embedding containers are as follows:

[0229] endpoint_id:BlazeGraph

[0230] URI:IsMarketLeaderIn

[0231] Vector: (1,2,25,13)

[0232] Each connection configuration can be associated with exactly one model. Each model can include an endpoint identifier and vectors β0 and β1, as discussed above in conjunction with Algorithm 2. There can be one machine learning model (e.g., a logistic regression model) per knowledge graph endpoint.

[0233] Fig.11 Shows Figure 7 Cloud configuration of computer systems. Fig.11 A cloud configuration may be part of a cloud computing environment. Cloud computing is a service delivery model for enabling convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) that can be quickly provisioned and released with minimal management effort or interaction with a service provider.

[0234] A cloud computing environment (i.e., a cloud environment or a cloud) can have one or more of the following characteristics: scalability, multi-tenancy, performance monitoring, virtual resources that can be dynamically allocated to different users based on demand, multiple redundant sites, multiple virtual machines, and network accessibility (e.g., via the Internet) from multiple locations (e.g., via a web browser) and devices (e.g., a mobile device or a PC).

[0235] A cloud computing environment can have a higher ratio of virtual resources to physical resources (e.g., a higher ratio of virtual machines to physical machines) than a local computing environment. For example, in a cloud computing environment, the ratio of virtual resources (e.g., machines) to physical resources can be at least 10 to 1, at least 20 to 1, or at least 30 to 1. In contrast, a local computing environment can have fewer than four virtual resources (e.g., machines) per physical resource.

[0236] The cloud computing environment may include a public cloud 850 (i.e., a public cloud computing infrastructure) and / or a private cloud 852 (i.e., a private cloud computing infrastructure). The public cloud computing infrastructure 850 may involve sharing hardware, storage, and / or network resources between multiple organizations or tenants. Services and / or applications may be accessed and managed using a web browser. The private cloud computing infrastructure 852 may include resources used exclusively by one organization or a group of users. Compared to the public cloud computing infrastructure 850, the private cloud infrastructure 852 may provide more flexibility and control, however, the private cloud infrastructure 852 may be more expensive. In both cases, the public cloud 850 and private cloud 852 computing infrastructures may be hosted by a service provider (e.g., Microsoft (Azure), Amazon (AWS), or SAP Business Technology Platform).

[0237] exist Fig.11 In a cloud configuration, the configuration UI and configuration repository can be located in the public cloud 850 and can be remotely accessed via an API. Similarly, the statement inspection engine, monitor, and alert repository can be located in the public cloud 850. In contrast, the KG triple repository can be located in the private cloud 852.

[0238] Fig.12An exemplary system for implementing the claimed subject matter is shown, which includes a general-purpose computing device in the form of a conventional computing environment 720 (e.g., a personal computer). The conventional computing environment includes a processing unit 722, a system memory 724, and a system bus 726. The system bus couples various system components including the system memory 724 to the processing unit 722. The processing unit 722 can perform arithmetic, logic, and / or control operations by accessing the system memory 724. The system memory 724 can store information and / or instructions for use in conjunction with the processing unit 722. The system memory 724 can include volatile and non-volatile memory, such as random access memory (RAM) 728 and read-only memory (ROM) 730. A basic input / output system (BIOS) containing basic routines such as those that help transfer information between elements within the personal computer 720 during startup can be stored in the ROM 730. The system bus 726 can be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures.

[0239] The personal computer 720 may also include a hard disk drive 732 for reading and writing to a hard disk (not shown), and an external disk drive 734 for reading or writing to a removable disk 736. The removable disk may be a disk for a disk drive or an optical disk (such as a CD-ROM) for an optical disk drive. The hard disk drive 732 and the external disk drive 734 are connected to the system bus 726 via a hard disk drive interface 738 and an external disk drive interface 740, respectively. The drive and its associated computer-readable medium provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the personal computer 720. The data structure may include relevant data for an embodiment of a method for detecting data leakage and / or detecting dangerous information. The relevant data may be organized in a database, such as a relational database or an object database.

[0240] Although the exemplary environment described herein employs a hard disk (not shown) and an external disk 736, those skilled in the art will appreciate that other types of computer-readable media that can store computer-accessible data, such as magnetic tape cassettes, flash memory cards, digital video disks, random access memory, read-only memory, etc., may also be used in the exemplary operating environment.

[0241] A plurality of program modules may be stored on the hard disk, external disk 736, ROM 730 or RAM 728, including an operating system (not shown), one or more application programs 744, other program modules (not shown) and program data 746. The application program may include at least a portion of the functionality shown in FIG. 16 .

[0242] As described below, a user can enter commands and information into the personal computer 720 through input devices such as a keyboard 748 and a mouse 750. Other input devices (not shown) may include a microphone (or other sensor), a joystick, a game controller, a scanner, and the like. These and other input devices may be connected to the processing unit 722 via a serial port interface 752 coupled to the system bus 726, or may be collected by other interfaces such as a parallel port interface 754, a game port, or a universal serial bus (USB). In addition, information may be printed using a printer 756. The printer 756 and other parallel input / output devices may be connected to the processing unit 722 via a parallel port interface 754. A monitor 758 or other type of display device is also connected to the system bus 726 via an interface such as a video input / output 760. In addition to the monitor, the computing environment 720 may include other peripheral output devices (not shown), such as speakers or other audible outputs.

[0243] The computing environment 720 can communicate with other electronic devices such as computers, phones (wired or wireless), personal digital assistants, televisions, etc. For communication, the computer environment 720 can operate in a networked environment using connections to one or more electronic devices. FIG. 16 depicts a computer environment networked with a remote computer 762. The remote computer 762 can be another computing environment, such as a server, a router, a network PC, a peer device, or other public network node, and can include many or all of the elements described above with respect to the computing environment 720. The logical connections depicted in FIG. 16 include a local area network (LAN) 764 and a wide area network (WAN) 766. Such networking environments are common in offices, enterprise-wide computer networks, intranets, and the Internet, and can be particularly encrypted.

[0244] When used in a LAN networking environment, the computing environment 720 can be connected to the LAN 764 via a network I / O 768. In a networking environment, program modules or portions thereof depicted relative to the computing environment 720 may be stored in remote memory storage devices resident on or accessible by the remote computer 762. In addition, other data related to the methods for detecting data leaks and / or detecting hazardous information (as described above) may reside on or be accessible via the remote computer 762. The network connections shown are exemplary, and other means of establishing a communications link between electronic devices may be used.

[0245] The computing system described above is merely one example of the type of computing system that may be used to implement the method for detecting a data leak and / or detecting hazardous information.

Claims

1. A computer-implemented method for detecting data leakage and / or detecting dangerous information, the method comprising: Receive a knowledge graph (100); Extracting data from at least one network service (601); identifying (603) statements in the extracted data; For each identified statement (605): using the knowledge graph (100) to determine whether the identified statement is public or private, and / or Using the knowledge graph (100), it is determined whether the identified statement is true or false.

2. The method according to claim 1, wherein: Determining whether the identified statement is true or false using the knowledge graph (100) includes determining whether a subject-predicate combination of the identified statement has a maximum cardinality; When the maximum cardinality of the subject-predicate combination of the identified statement is exceeded, the identified statement is determined to be false.

3. The method according to claim 1 or 2, wherein: The at least one network service comprises one or more of: a web service, a social network service, a service providing an interface to a language model; wherein extracting data from the at least one network service comprises accessing at least one application programming interface via the at least one network service; Wherein extracting data from the at least one network service comprises web scraping using hypertext transfer protocol, wherein the web scraping may comprise web crawling.

4. The method according to any one of the preceding claims, further comprising: Calculating a vector for each uniform resource identifier of the knowledge graph (100) so that each of the uniform resource identifiers of the knowledge graph (100) has a corresponding vector; creating a copy of the knowledge graph (100) including the calculated vector; The copy of the knowledge graph (100) is modified to include a number of true statements in the copy of the knowledge graph (100) equal to the number of false statements.

5. The method according to claim 4, wherein: The modifications also include: When said copy of said knowledge graph (100) has more false statements than true statements, randomly deleting false statements; When the copy of the knowledge graph (100) has more true statements than false statements, adding false statements to the copy of the knowledge graph (100) by randomly selecting and combining subjects, predicates, and objects from different statements in the knowledge graph (100); wherein each component of a statement in said copy of said knowledge graph (100) has a corresponding computed vector; for each statement in said copy of said knowledge graph (100), calculating a similarity value using the calculated vector corresponding to said statement, said similarity value measuring the similarity of the subject and predicate of said statement to the object of said statement, The similarity value may be calculated using a cosine similarity measurement.

6. The method according to claim 4 or 5, wherein: The vectors are calculated so that each relationship in the knowledge graph (100) is represented by two vectors, the two vectors being a norm vector of a hyperplane and a transformation vector on the hyperplane, Wherein, the hyperplane is perpendicular to the embedding space of the vector, and wherein TransH can be used to calculate the vector.

7. The method according to claim 5 or 6, further comprising: training a statistical model using the calculated similarity values ​​and information indicating whether the similarity values ​​correspond to true or false statements in the copy of the knowledge graph (100), wherein the statistical model is a logistic regression model; Wherein, using the knowledge graph (100) to determine whether the identified statement is true or false comprises applying the statistical model to the identified statement: determining that the identified statement is true when the trained statistical model returns a probability of the identified statement that is greater than a specified threshold; When the trained statistical model returns a probability of the identified statement that is less than or equal to the specified threshold, the identified statement is determined to be false.

8. The method according to any one of the preceding claims, further comprising: identifying sentences in the extracted data, including performing term frequency-inverse sentence frequency with stemming and semantics on the extracted data; Use named entity recognition to identify entities in the extracted data; identifying implicit relationships between the identified entities via relationship extraction, wherein the relationship extraction includes deriving structured text from the extracted data; The results of the named entity recognition and the relationship extraction are one or more entity 1-predicate-entity 2 combinations.

9. The method according to claim 8, further comprising: mapping the identified entities to the knowledge graph (100) via entity linking, the mapping comprising named entity disambiguation; Mapping the predicate of the identified entity 1-predicate-entity 2 combination to an attribute of the knowledge graph (100); Based on the mapping of the identified entities to the knowledge graph (100) and the mapping of predicates to attributes of the knowledge graph (100), at least one result statement is provided, each result statement comprising a subject-predicate-object triple, each component of the subject-predicate-object triple comprising a uniform resource identifier of the knowledge graph (100).

10. A method according to any one of the preceding claims, wherein: Determining whether the identified statement is true or false and determining whether the identified statement is public or private includes comparing the identified statement to statements in the knowledge graph (100).

11. The method according to claim 10, wherein: Comparing the identified statement to the statements in the knowledge graph (100) includes determining at least one comparable statement to the identified statement, wherein the comparable statement to the identified statement has at least one value in common with the identified statement; wherein the plurality of statements in the knowledge graph (100) include metadata indicating whether the statement is true or false, and / or wherein the plurality of statements in the knowledge graph (100) include metadata indicating whether the statements are public or private; Wherein, determining whether the identified statement is true or false includes identifying a maximum cardinality of the at least one comparable statement, and determining whether the identified statement exceeds the maximum cardinality.

12. A method according to any one of the preceding claims, wherein: Each statement of the knowledge graph (100) includes a subject-predicate-object triple, wherein each subject-predicate-object triple consists of a subject, a predicate and an object, wherein the subject, the predicate and the object are each represented by a uniform resource identifier.

13. A method according to any one of the preceding claims, wherein: Extracting data from the at least one web service includes randomly selecting statements from the knowledge graph (100); Wherein, the method further comprises: constructing a query string based on the subject and predicate of the statement; At least one application programming interface is invoked via the at least one web service using the base uniform resource locator and using the query string.

14. A computer program comprising instructions which, when executed by a computer, cause the computer to perform a method according to any one of the preceding claims.

15. A computer system (700) for detecting data leakage and / or detecting dangerous information, the system comprising a memory and at least one processor, the memory comprising instructions, the instructions causing the at least one processor to: Receive a knowledge graph (100); extracting data from at least one web service; Identify statements in the extracted data; For each identified statement: using the knowledge graph (100) to determine whether the identified statement is public or private, and / or The knowledge graph (100) is used to determine whether the identified statement is true or false.

Images