A job safety risk identification method based on a multi-modal knowledge graph

Abstract

The application relates to a kind of job safety risk identification methods based on multi-modal knowledge graph, comprising the following steps: obtaining multiple modal power field operation data and pre-processing;Schema of multi-modal knowledge graph is constructed;The knowledge extraction of different modal power field operation data is carried out respectively, and the label of power field operation data is obtained;The label of different modal power field operation data is fused;Knowledge storage is carried out through the label after fusion and Schema, and the multi-modal power field operation risk cognitive graph is obtained;The multi-modal power field operation risk cognitive graph is inferred;Through the multi-modal power field operation risk cognitive graph, auxiliary decision is carried out.The application solves the problem that multi-source data of power field operation is difficult to be efficiently utilized by constructing multi-modal power field operation risk cognitive graph, improves the comprehensiveness, timeliness and accuracy of operation risk research and judgment, and provides support for on-site operation safety control decision.

Description

TECHNICAL FIELD

[0001] The application relates to a job safety risk identification method based on a multi-modal knowledge graph, and belongs to the technical field of power site job safety. BACKGROUND

[0002] Power site jobs have many aspects, many construction personnel, and a complex environment, and various safety risks and accident hazards are intertwined and superimposed, so safety control is difficult. In view of the dispersion of job sites, insufficient control of personnel and equipment, and insufficient safety monitoring personnel in the safety control of site jobs, site job safety control based on various monitoring terminals is currently widely carried out, automatic identification and alarm of illegal acts in site jobs are carried out based on the joint use of various sensors, video image recognition systems, wearable equipment and tools, and the level of job safety supervision is improved, among which, the use of artificial intelligence, edge computing, sensors, the Internet of Things and other technologies realizes on-site analysis and real-time judgment of various data of site jobs, and relatively low-cost high-accuracy identification of illegal acts is the focus and difficulty of carrying out safety control of power site jobs. Due to various factors such as many interference factors in the site job environment, easy to block, the target to be identified may not be prominent enough, and the corresponding artificial intelligence model training samples are not typical or sufficient, the overall accuracy of the current site job illegal identification device is not high, there are many missed and false detections, and how to innovate the organization and application form of power multi-source heterogeneous data and improve the risk prevention and control level of power job sites is a problem that needs to be considered.

[0003] As one of the key technologies of artificial intelligence, the knowledge graph (KG) contains rich semantic information and topological structure information of the graph, and has become an effective organization and application carrier for multi-source heterogeneous data. The research and application of knowledge graph technology in the power field have achieved certain results, but the data resources involved are mainly structured table data and text data, and in the face of power production job safety control scenarios, there is still a lack of multi-modal data (image, video, voice, text, etc.) linkage cognitive analysis means for core elements (equipment, personnel, environment, process, specifications, etc.). The power site job environment is complex, the control types are various, and various elements such as people, machines and the environment are involved, so it is difficult for a single mode to provide complete information about a phenomenon, which may cause problems such as job risk, illegal false alarm and missed report rate of the artificial intelligence model. In addition, the knowledge bases in the power transmission and transformation and infrastructure fields are scattered, and are supported by respective systems, lack of unified deployment and sharing mechanism, and data and services cannot be effectively integrated, so the knowledge graph technology cannot effectively improve the business processing efficiency and practicality.

[0004] A safety knowledge graph construction method and system for a smart power plant are disclosed in Chinese Patent Application No. CN202110683084.3. The knowledge graph constructed by this method takes into account the application status and needs of safety management in power enterprises, effectively reducing the safety risks in power plant production links, and facilitating intelligent management in power production to ensure the safety of power plant production.

[0005] This method does not take into account the complex environment of power field, the diversity of control types, and the interlaced and superimposed problems of various risk hidden dangers. Using single modal data can hardly provide complete information about a certain phenomenon, which may cause the artificial intelligence model to have operation risks, violations, false positives, and high false negative rates, and the accuracy is not high in actual use. SUMMARY

[0006] To overcome the above problems, the present application provides a job safety risk identification method based on a multi-modal knowledge graph. This method uses multi-modal knowledge graph technology to construct a multi-modal power field job risk cognitive graph. Based on the data organization and causal reasoning capabilities of the knowledge graph, it solves the problem of efficient utilization of multi-source data in power field operations, which is difficult to be discretized and heterogeneous. It integrates unstructured and structured data such as text, job images, and videos to conduct job risk research and judgment, excavate hidden job risks, recommend risk response measures, and improve the comprehensiveness, timeliness, and accuracy of job risk research and judgment, providing support for on-site job safety control decisions.

[0007] The technical solution of the present application is as follows:

[0008] A job safety risk identification method based on a multi-modal knowledge graph, comprising the following steps:

[0009] Obtain multi-modal power field job data;

[0010] Construct a Schema for the multi-modal knowledge graph;

[0011] Extract knowledge from different modal power field job data to obtain labels for the power field job data;

[0012] Fuse the labels of different modal power field job data;

[0013] Store knowledge through the fused labels and Schema to obtain a multi-modal power field job risk cognitive graph;

[0014] Use the multi-modal power field job risk cognitive graph to assist in decision-making and identify power field job risks.

[0015] Further, the data types of the power field job data include text data, image data, and video data.

[0016] Further, the text data includes at least one of a job work ticket, a regulation system, personnel information, equipment information, a work report, and a violation case; and further includes preprocessing of the text data, specifically:

[0017] Obtaining text data A;

[0018] Identifying the file type of the text data A; the file type includes a text file and a non-text file;

[0019] If the file type of the text data A is a text file, output the pure text data of the text data A through a Tika component, otherwise, extract the pure text data of the text data A through an optical character recognition technology and output.

[0020] Further, the knowledge extraction of the power field operation data includes key information extraction of the text data, including the following steps:

[0021] Obtaining power field operation data with a data type of text data;

[0022] Defining entities and relationship types in the power field operation data;

[0023] Extracting part of the power field operation data with a data type of text data, and annotating according to the definition to obtain a training set;

[0024] Constructing a segmented convolutional neural network model based on an attention mechanism;

[0025] Training the segmented convolutional neural network model based on the attention mechanism through the training set;

[0026] Inputting the power field operation data to be extracted, target entities, and relationship types into the segmented convolutional neural network model based on the attention mechanism for entity relationship extraction.

[0027] Further, the knowledge extraction of the power field operation data includes semantic label extraction of image data, including the following steps:

[0028] Constructing a list of common targets contained in the power field operation image;

[0029] Constructing a list of common violation behaviors of the power field operation through the list of common targets;

[0030] Annotating violation behaviors for part of the power field operation data with a data type of image data;

[0031] The common violation list is trained by a deep learning algorithm combined with annotated power field operation data to obtain an identification model of different violations.

[0032] The identification model is used to extract semantic labels from image data of the power field operation data, and the output of the identification model is the semantic label.

[0033] Further, the identification model is used to extract semantic labels from image data of the power field operation data, including the following steps:

[0034] The image data of the power field operation data is input into an encoder, and the convolutional neural network of the decoder extracts n position features x = {x1, x2, x3,..., xn} from the image data. n} x n ∈R D , wherein x n is a D-dimensional vector, D is the number of channels, and n is the size of the feature map.

[0035] The feature x is input into the decoder.

[0036] The Attention mechanism of the decoder is used to assign different weights to the feature x.

[0037] When generating the tth word of the semantic label, the context vector of the recurrent neural network of the decoder is z t , the hidden layer state of the previous stage of the recurrent neural network is g t-1 , and the context vector z t is a weighted average of the feature x, which is obtained by the following formula:

[0038]

[0039] wherein a t,m is a weight measuring the image feature of the mth position when generating the tth word; the weight is a function of the hidden layer state g t-1 of the previous stage and the mth position image feature x m ; and the word is a character of the semantic label of the image data.

[0040] The context vector is input into a long short-term memory network, and a hidden variable of the long short-term memory network is generated to output a model result y t .

[0041] The model result y t is used as the semantic label, and the model result y tis the probability of the t-th word appearing in the power field operation data, k is the size of the text dictionary used by the identification model.

[0042] Further, the knowledge extraction of the power field operation data includes semantic label extraction of video data, including the following steps:

[0043] Obtaining power field operation data of which the data type is video data;

[0044] Extracting key frames in the video;

[0045] Extracting semantic labels of the key frames by the method of semantic label extraction of image data;

[0046] Associating the key frames with the corresponding videos, and taking the semantic labels of the key frames as the attribute values of the corresponding videos.

[0047] Further, the labels of different modalities of power field operation data are fused, including the following steps:

[0048] For the label A of the power field operation data of which the data type is text data, the similarity between the label A and all labels B of the power field operation data of which the data type is image data is calculated by the longest common subsequence algorithm, and the label A is linked with the label B with the highest similarity;

[0049] For the label A of the power field operation data of which the data type is text data, the similarity between the label A and all labels C of the power field operation data of which the data type is video data is calculated by the longest common subsequence algorithm, and the label A is linked with the label C with the highest similarity;

[0050] For the label B of the power field operation data of which the data type is image data, the similarity between the label B and all labels C of the power field operation data of which the data type is video data is calculated by the longest common subsequence algorithm, and the label B is linked with the label C with the highest similarity.

[0051] Further, the multi-modal power field operation risk cognition graph is also inferred to infer the potential relationship between different triple entities, and the multi-modal power field operation risk cognition graph is perfected, including the following steps:

[0052] Mapping the entities and relationships in the multi-modal power field operation risk cognition graph into a low-dimensional vector space by a TransD model, and inferring the entities and relationships mapped into the low-dimensional vector space by a random walk algorithm, the inference process being as follows:

[0053] Calculating the eigenvalue function s of each path h，Q(t) , so as to obtain a plurality of paths; the eigenvalue function sh，Q(t) is the probability of reaching the tail entity vector f from the head entity vector h along the path Q; the path Q consists of several relation vectors:

[0054]

[0055] where T z-1 is the scope of the relation vector r z ;

[0056] When the path Q reaches any intermediate entity vector e, the update method of the eigenvalue function s h，Q(t) is:

[0057]

[0058] In the initial stage of the random walk, if e'∈s h，Q(e，) , the eigenvalue function s h，Q(t) = 1, otherwise the eigenvalue function s h,Q(t) = 0;

[0059] I(r l (e', e)) is an indicator function, if r l (e', e) exists, then I(r l (e', e)) = 1, otherwise I(r l (e', e)) = 0;

[0060] where r l (e', e) is the relation r l between any entity e' and entity e on the random walk path, ran(Q) is the set of entities on the walk path Q, s h,Q(e，) (e, ) is the entity e on the walk path, and the eigenvalue vector;

[0061] A score function f Q (h k , t k ) is established for each head entity and tail entity combination (h n , t k ) under the relation vector through the random walk algorithm to obtain several path characteristics Q1,..., Qn:

[0062]

[0063] where θ k is the score of the training sample under the relation vector on the nth walk path characteristic, taking a floating point number in [0-1].

[0064] The probability of the existence of the relation between the head entity and tail entity combination (h k , t k ) is Pk :

[0065]

[0066] wherein, r k is the relationship of the kth training sample;

[0067] The loss function is:

[0068] minw k [y k lnP k +(1-y k )ln(1-P k )];

[0069] wherein, y k is a label of whether the head entity and tail entity combination (h k , t k ) has the relationship vector r, if the triple vector (h k , r, t k ) exists, then y k =1, otherwise y k =0, w k is the error size of the model predicting the relationship of the kth head entity and tail entity combination and the real combination relationship; the loss function is used to evaluate the degree of inconsistency between the model predicting the head-tail entity combination relationship and the real head-tail entity combination relationship, and guide the model training process to continuously adjust the parameters.

[0070] Further, the Schema of the multi-modal knowledge graph comprises:

[0071] a list of core entities and attributes of the power field operation risk cognition graph;

[0072] a list of core relationships of the power field operation risk cognition graph.

[0073] The second aspect

[0074] A job safety risk identification device based on a multi-modal knowledge graph, comprising a data acquisition unit, a data processing unit and an output unit;

[0075] The data acquisition unit is used to acquire power field operation data of multiple modalities;

[0076] The data processing unit constructs a Schema of a multi-modal knowledge graph;

[0077] Knowledge extraction is performed on power field operation data of different modalities respectively to obtain labels of the power field operation data;

[0078] The labels of the power field operation data of different modalities are fused;

[0079] store knowledge through the fused label and Schema, to obtain a multi-modal power field operation risk cognitive graph;

[0080] make an auxiliary decision through the multi-modal power field operation risk cognitive graph, to identify a power field operation risk;

[0081] The output unit outputs a power field operation risk.

[0082] In an embodiment of the present application, the data types of the power field operation data include text data, image data and video data.

[0083] In an embodiment of the present application, the text data includes at least one of a job work ticket, a regulation, personnel information, equipment information, a work report and a violation case; and further includes preprocessing of the text data, specifically:

[0084] obtain text data A;

[0085] identify a file type of the text data A; the file type includes a text file and a non-text file;

[0086] if the file type of the text data A is a text file, output pure text data of the text data A through a Tika component, otherwise, extract pure text data of the text data A through an optical character recognition technology and output.

[0087] In an embodiment of the present application, knowledge extraction of the power field operation data includes key information extraction of the text data, including the following steps:

[0088] obtain power field operation data of which the data type is text data;

[0089] define entities and relationship types in the power field operation data;

[0090] extract part of the power field operation data of which the data type is text data, and obtain a training set according to defined line annotation;

[0091] construct a segmented convolutional neural network model based on an attention mechanism;

[0092] train the segmented convolutional neural network model based on the attention mechanism through the training set;

[0093] input power field operation data to be extracted, target entities and relationship types into the segmented convolutional neural network model based on the attention mechanism for entity relationship extraction.

[0094] In one embodiment of the present invention, knowledge extraction of the power field operation data, including semantic tag extraction of image data, includes the following steps:

[0095] Construct a list of common targets contained in images of power field operations;

[0096] A list of common violations in power field operations is constructed based on the aforementioned list of common targets;

[0097] Violations were flagged in power field operation data that included some image data.

[0098] The identification models for different violations were obtained by training each violation in the list of common violations using a deep learning algorithm combined with labeled power field operation data.

[0099] The recognition model extracts semantic tags from power field operation data, which is image data. The output of the recognition model is the semantic tag.

[0100] In one embodiment of the present invention, semantic tag extraction is performed on power field operation data, which is of image data type, using the recognition model, including the following steps:

[0101] Power field operation data, which is of image type, is input into the encoder. The decoder's convolutional neural network extracts features x = {x1, x2, x3, ..., xn} at n locations. n}, x n ∈R D , where x n Let n be a D-dimensional vector, where D is the number of channels and n is the size of the feature map.

[0102] Input feature x into the decoder;

[0103] Different weights are assigned to the feature x using the Attention mechanism of the decoder;

[0104] When generating the t-th word of the semantic label, the context vector input to the recurrent neural network of the decoder is z. t The hidden layer state of the previous stage of the recurrent neural network is g. t-1 The context vector z t The weighted average of feature x is obtained using the following formula:

[0105]

[0106] Among them, a t，m This measures the weight of the image feature at position m when generating the t-th word; the weight is the hidden layer state g from the previous stage.t-1 and the m-th location image feature x m ; the word is a literal of a semantic label of the image data;

[0107] The context vector is taken as an input of a long short-term memory network, and a model result y t is generated and output by a hidden variable of the long short-term memory network.

[0108] The model result y t is taken as a semantic label, and the model result y t is a k-dimensional probability, representing a probability of the t-th word appearing in the power field operation data, and k is a size of a text dictionary used by the recognition model.

[0109] In an embodiment of the present application, the knowledge extraction on the power field operation data includes semantic label extraction of video data, including the following steps:

[0110] Obtaining power field operation data of which data type is video data;

[0111] Extracting key frames in the video;

[0112] Extracting semantic labels of the key frames by a method of semantic label extraction of image data;

[0113] Associating the key frames with corresponding videos, and taking the semantic labels of the key frames as attribute values of the corresponding videos.

[0114] In an embodiment of the present application, the labels of power field operation data of different modalities are fused, including the following steps:

[0115] For a label A of power field operation data of which data type is text data, calculating similarities of the label A with all labels B of power field operation data of which data type is image data by a longest common subsequence algorithm, and linking the label A with a label B having the highest similarity;

[0116] For a label A of power field operation data of which data type is text data, calculating similarities of the label A with all labels C of power field operation data of which data type is video data by a longest common subsequence algorithm, and linking the label A with a label C having the highest similarity;

[0117] For a label B of power field operation data of which data type is image data, calculating similarities of the label B with all labels C of power field operation data of which data type is video data by a longest common subsequence algorithm, and linking the label B with a label C having the highest similarity.

[0118] In an embodiment of the present application, the multi-modal power field operation risk cognitive graph is inferred to deduce the potential relationship between different triple entities, and the multi-modal power field operation risk cognitive graph is perfected, comprising the following steps:

[0119] The entities and relationships in the multi-modal power field operation risk cognitive graph are mapped into a low-dimensional vector space by the TransD model, and the entities and relationships mapped into the low-dimensional vector space are inferred by the random walk algorithm, and the inference process is as follows:

[0120] The eigenvalue function s of each path is calculated h,Q(t) , so as to obtain a plurality of paths; the eigenvalue function s h,Q(t) is the probability of reaching the tail entity vector t from the head entity vector h along the path Q; the path Q is composed of a plurality of relationship vectors:

[0121]

[0122] Wherein, T z-1 is the scope of the relationship vector r z ;

[0123] When the path Q reaches any intermediate entity vector e, the updating method of the eigenvalue function s h,Q(t) is:

[0124]

[0125] In the initial stage of random walk, if e' ∈ s h,Q(e) , then the eigenvalue function s h,Q(t) = 1, otherwise the eigenvalue function s h,Q(t) = 0;

[0126] I(r l (e', e)) is an indicator function, if r l (e', e) exists, then I(r l (e', e)) = 1, otherwise I(r l (e', e)) = 0;

[0127] Wherein, r l (e', e) is the relationship r l between any entity e' and entity e on the random walk path, ran(Q) is the entity set on the walk path Q, s h,Q(e，) (e,) is the entity e on the walk path, and the eigenvalue vector;

[0128] A plurality of path characteristics Q1,..., Q z are obtained by the random walk algorithm, and each head entity and tail entity combination (hk , t k ) establishes a score function f Q (h k , t k ):

[0129]

[0130] wherein θ n is the score of the training sample under the relation vector on the n th walk path feature, and the value is a floating point number of [0-1];

[0131] The probability that the head entity and tail entity combination (h k , t k ) has a relationship is P k :

[0132]

[0133] wherein r k is the relationship of the k th training sample;

[0134] The loss function is:

[0135] minw k [y k lnP k +(1-y k )ln(1-P k )];

[0136] wherein y k is the label of whether the head entity and tail entity combination (h k , t k ) has the relation vector r, if the triple vector (h k , r, t k ) exists, y k =1, otherwise y k =0, w k is the error size of the model predicting the relationship of the k th head entity and tail entity combination and the real combination relationship; the loss function is used to evaluate the degree of inconsistency between the model predicting the head-tail entity combination relationship and the real head-tail entity combination relationship, and guide the model training process to continuously adjust the parameters.

[0137] In an embodiment of the present application, the Schema for constructing the multi-modal knowledge graph comprises:

[0138] A list of core entities and attributes for constructing a power field operation risk cognitive graph;

[0139] A list of core relationships for constructing a power field operation risk cognitive graph.

[0140] The third aspect

[0141] An electronic device, which, when in operation, implements the method for identifying work safety risks based on a multi-modal knowledge graph as described in the first aspect to identify power field work risks.

[0142] Fourth aspect

[0143] A storage medium, in which a computer program stored therein, when executed by a computer, implements the method for identifying work safety risks based on a multi-modal knowledge graph as described in the first aspect to identify power field work risks.

[0144] The present application has the following beneficial effects:

[0145] 1. The present application proposes a multi-modal power field work risk cognitive graph construction method. A multi-modal data fusion method based on label alignment is proposed, a deep learning and artificial intervention combined method is used to extract entities and entity relationships in the data, a semantic network is constructed according to the extraction results, and a power field work risk cognitive graph containing text, picture, video and other modal data is constructed, integrating various power work data such as field work control regulations, work tickets, personnel / equipment information, work monitoring videos, work violation images, typical accident cases, etc., solving the problem of efficient utilization of multi-source data discretization and isomerization of power field work.

[0146] 2. The present application proposes a field work risk reasoning method based on the fusion of random walk algorithm and TransD model. The relationship vector of the multi-modal knowledge graph is learned through the TransD model, the multi-modal knowledge graph entity sequence is generated by combining the reasoning model based on random walk, and the rapid identification of power field work risks is realized by relying on the reasoning of the relationship vector similarity rules, entity sequence frequency rules, relationship type conversion rules and other rules. BRIEF DESCRIPTION OF DRAWINGS

[0147] Figure 1 The method flowchart of the embodiment of the present application.

[0148] Figure 2 The data preprocessing flowchart of the embodiment of the present application.

[0149] Figure 3 The power field work data label extraction flowchart of the text data type of the embodiment of the present application.

[0150] Figure 4 The power field work data label extraction network structure schematic diagram of the text data type of the embodiment of the present application.

[0151] Figure 5 The power field work data label extraction flowchart of the image data type of the embodiment of the present application.

[0152] Figure 6 A schematic diagram for power field operation data label extraction of the video data type of the embodiment of the present application.

[0153] Figure 7 A flow chart for power field operation data label fusion of different modalities of the embodiment of the present application.

[0154] Figure 8 An overall framework for power field operation risk cognition graph assisted decision making based on the embodiment of the present application.

[0155] Figure 9 A power field operation risk disposal process of the embodiment of the present application. DETAILED DESCRIPTION

[0156] The present application will be described in detail below in combination with the drawings and specific embodiments.

[0157] First aspect

[0158] Reference Figure 1 A job safety risk identification method based on a multi-modal knowledge graph, comprising the following steps:

[0159] Obtaining power field operation data of multiple modalities;

[0160] Constructing a Schema of a multi-modal knowledge graph;

[0161] Respectively performing knowledge extraction on power field operation data of different modalities to obtain labels of the power field operation data;

[0162] Fusing labels of power field operation data of different modalities;

[0163] Storing knowledge through the fused labels and the Schema to obtain a multi-modal power field operation risk cognition graph; and performing assisted decision making through the multi-modal power field operation risk cognition graph to identify power field operation risks.

[0164] In an embodiment of the present application, the data types of the power field operation data include text data, image data and video data.

[0165] Reference Figure 2 In an embodiment of the present application, the text data includes at least one of a job work ticket, a regulation and system, personnel information, equipment information, a work report and a violation case; and further includes preprocessing of the text data, specifically:

[0166] Obtaining text data A;

[0167] Identify the file type of the text data A; the file type includes text files and non-text files.

[0168] If the file type of the text data A is a text file, then the plain text data of the text data A is output through the Tika component; otherwise, the plain text data of the text data A is extracted through optical character recognition technology and output.

[0169] Specifically, in one embodiment, data is input in the format of Table 1.

[0170] Table 1. Data input format for the JSON-based unstructured data plain text conversion process.

[0171] Field name schema value type remark Data identification id string File name name string File address url string File type suffix string ... ... ... ...

[0172] Example of input data is as follows:

[0173] {

[0174] "id":"1298298a3458ec66f486acc935c06a62",

[0175] "name":"State Grid Corporation of China Operational Safety Risk Early Warning and Control Work Specifications (Trial)",

[0176] "url":" / Users / wuchenzhou / tmp",

[0177] "suffix":"doc"

[0178] }

[0179] Where “id” is the file’s unique identifier, “name” is the file name, “url” is the absolute path to the file, and “suffix” is the file type.

[0180] The data output format is shown in Table 2.

[0181] Table 2 shows the data output format for the plain text conversion process of unstructured data based on JSON.

[0182]

[0183] Examples of output data are as follows:

[0184] {

[0185] "id":"1298298a3458ec66f486acc935c06a62",

[0186] "name":"State Grid Corporation of China Operational Safety Risk Early Warning and Control Work Specifications (Trial)",

[0187] "content":"State Grid Corporation of China Operation Safety Risk Early Warning Management Specification (Trial)

[0188] Chapter I General Provisions

[0189] Article 1 To deepen the operation safety risk early warning management work, and improve the on-site operation safety management capability and accident prevention level. According to the "State Grid Corporation of China Production Operation Safety Management Standardization Work Specification (Trial)",...

[0190] }

[0191] Wherein "id" is the unique identification of the file, "name" is the file name, and "content" is the text content corresponding to the file.

[0192] In specific embodiments of the application, the types of text files include word, pdf, excel, ppt, etc., and non-text files include pictures, pdf scans, etc.

[0193] The preprocessing of the text data of the application uses TiKa components and OCR components, which are as follows:

[0194] 1. TiKa component, TiKa is a java toolkit that can realize the conversion of pure text of word, pdf, ppt, excel, html and other files, and other tool components or system interfaces based on word, pdf and other files can also be used to realize the conversion of pure text of corresponding files.

[0195] 2. OCR service, non-editable image files such as pictures and pdf scans need to use OCR (Optical Character Recognition) technology to extract text content. The application uses the OCR component developed by the company to complete the text content extraction of pictures and pdf scans. Third-party or self-developed components such as baidu paddleocr component can also be used to extract text content.

[0196] In one embodiment, it also includes the preprocessing of image data and video data.

[0197] The preprocessing of image data is:

[0198] Delete images with a long side less than 400 pixels, a short side less than 250 pixels, and an image file size less than 20KB;

[0199] Convert the image file format to jpg format.

[0200] In this embodiment, the Image component of python is used to realize the conversion of image format.

[0201] Video data preprocessing:

[0202] Extract keyframes from the video file.

[0203] In this embodiment, the OpenCV component of Python is used to extract key data from the video, extracting 5 frames per second. The frame extraction frequency can also be set according to the actual situation.

[0204] refer to Figure 3 and 4 In one embodiment of the present invention, knowledge extraction of the power field operation data, including key information extraction from text data, includes the following steps:

[0205] Retrieve power field operation data that is of text data type;

[0206] Define the entity and relation types in power field operation data;

[0207] Extract a portion of the power field operation data that is text data and label it according to the definition to obtain the training set;

[0208] Construct a segmented convolutional neural network model based on an attention mechanism;

[0209] The attention-based segmented convolutional neural network model is trained using the training set.

[0210] The power field operation data to be extracted, the target entities, and the relationship types are input into the segmented convolutional neural network model based on the attention mechanism for entity and relationship extraction. In one specific embodiment, the segmented convolutional neural network model based on the attention mechanism represents the word sequence of the input power field operation data into vector patterns for each word using word2vec and Skip-gram models, and then concatenates them with the position vectors corresponding to each word.

[0211] Feature vectors for each of the aforementioned words are extracted using convolutional layers;

[0212] The pooling layer divides the power field operation data into three segments and pools them separately.

[0213] The pooled feature vectors are passed through a softmax layer and then output.

[0214] The structure of the attention-based segmented convolutional neural network model is as follows: Figure 4 As shown, where b1~b qrepresents the vector of each sentence obtained after the input T (text data) is coded by the PCNN (segmented convolutional neural network model) network; Align Weights represents the weight adjustment of each sentence vector by the PCNN; and Bag features represents the feature vector obtained after the sentence vector is reduced in dimension.

[0215] In an embodiment of the present application, the knowledge extraction of the power field operation data includes semantic label extraction of image data, including the following steps:

[0216] A list of common targets contained in the power field operation image is constructed.

[0217] A list of common violations of the power field operation is constructed through the list of common targets.

[0218] The power field operation data of which part of the data type is image data is marked for violation;

[0219] Each violation in the list of common violations is trained through a deep learning algorithm combined with the marked power field operation data to obtain a recognition model of different violations.

[0220] The semantic label extraction of the power field operation data of which the data type is image data is performed through the recognition model, and the output of the recognition model is the semantic label.

[0221] In an embodiment, the list of common targets is as shown in Table 3.

[0222] Table 3: List of common targets contained in the power field operation image

[0223] Serial number Label Serial number Label Serial number Label 1 Safety helmet 10 People 19 Safety sign 2 Safety helmet chin strap 11 Hook 20 Tower guy 3 Insulating gloves 12 Safety belt 21 Foundation bolt 4 Insulating boots 13 Safety belt waistband 22 Waistcoat 5 Work clothes 14 Safety rope 23 Large vehicle 6 Safety fence 15 Porcelain bottle 24 Anchor pile 7 Metal ladder 16 Escalator 25 Deep foundation pit 8 Crane 17 Carabiner 26 Ventilation duct 9 Boom 18 Cement retaining wall

[0224] The list of common violations is as shown in Table 4.

[0225] Table 4: List of common violation recognition models of power field operation

[0226]

[0227] In some embodiments, the semantic label (target, event) of the picture is extracted using the violation recognition model combined with artificial correction.

[0228] Reference Figure 5 In an embodiment of the present application, the semantic label extraction of the power field operation data of which the data type is image data through the recognition model includes the following steps:

[0229] The power field operation data of the image data type is input into an encoder, and features x={x1, x2, x3,..., xn} of n positions are extracted by a convolutional neural network of the decoder n}, x n ∈R D , wherein x n is a D-dimensional vector, D is the number of channels, and n is the size of the feature map, in specific embodiments of the present application, n is the length * width of the image;

[0230] The features x are input into a decoder;

[0231] The features x are assigned different weights by an Attention mechanism of the decoder;

[0232] When generating the tth word of the semantic label, the context vector z t input into the recurrent neural network of the decoder is g t-1 , the hidden layer state of the previous stage of the recurrent neural network; the context vector z t is a weighted average of the features x, and is obtained by the following formula:

[0233]

[0234] , wherein a t,m is a weight measuring the image feature of the mth position when generating the tth word; the weight is a function of the hidden layer state g t-1 of the previous stage and the image feature x m of the mth position; the word is a character of the semantic label of the image data;

[0235] The context vector is input into a long short-term memory network, and a hidden variable of the long short-term memory network is generated to output a model result y t ;

[0236] The model result y t is taken as a semantic label, and the model result y t is a k-dimensional probability, representing the probability of the tth word appearing in the power field operation data, and k is the size of a text dictionary used by the recognition model.

[0237] Figure 5 , wherein p is the length of the number of sequence words of the final output, and the size is uncertain.

[0238] In some embodiments, in order to solve the problem of inaccurate extraction of part of the labels by the model, artificial intervention is adopted to calibrate part of the data, thereby improving the accuracy of the semantic label, and finally obtaining an image semantic label b.

[0239] Reference Figure 6In an embodiment of the present application, the knowledge extraction of the power field operation data includes semantic label extraction of video data, comprising the following steps:

[0240] Obtaining power field operation data of which data type is video data;

[0241] Extracting key frames in the video;

[0242] Extracting semantic labels of the key frames by the method of semantic label extraction of image data;

[0243] Associating the key frames with corresponding videos, and taking the semantic labels of the key frames as attribute values of the corresponding videos.

[0244] Reference Figure 7 The basic idea of the fusion is to extract semantic labels of text, picture and video data respectively, extract relevant entities, their relationships and attributes in the text data by a segmented convolutional neural network model based on attention mechanism, and store them in a table as text semantic labels; extract relevant picture semantic label set by a self-developed deep learning model (violation identification model); extract key frames of the video data, extract relevant label information by the picture data semantic extraction method, and then filter and merge them into a video label set. In an embodiment of the present application, the labels of different modal power field operation data are fused, comprising the following steps:

[0245] Linking the label A of the power field operation data of which data type is text data with the label B of the power field operation data of which data type is image data with the highest similarity by calculating the similarity of the label A and all labels B of the power field operation data of which data type is image data by the longest common subsequence algorithm;

[0246] Linking the label A of the power field operation data of which data type is text data with the label C of the power field operation data of which data type is video data with the highest similarity by calculating the similarity of the label A and all labels C of the power field operation data of which data type is video data by the longest common subsequence algorithm;

[0247] Linking the label B of the power field operation data of which data type is image data with the label C of the power field operation data of which data type is video data with the highest similarity by calculating the similarity of the label B and all labels C of the power field operation data of which data type is video data by the longest common subsequence algorithm.

[0248] The following is an example of specific algorithm.

[0249]

[0250] The purpose of reasoning is to infer potential relationships between entity pairs using machine learning methods, so as to realize the discovery of implicit knowledge and further improve the knowledge graph. In the power field operation scene, implicit risks in the operation process can be discovered. For example: the multi-modal power field operation risk cognition graph contains the relationship (Zhang San, not wearing insulating gloves, cutting high trees with electricity), (cutting high trees with electricity, exists, risk of electric shock), and then the potential relationship (Zhang San, exists, risk of electric shock) can be mined. In an embodiment of the present application, the multi-modal power field operation risk cognition graph is also subjected to reasoning to infer potential relationships between different triple entity, and the multi-modal power field operation risk cognition graph is improved, comprising the following steps:

[0251] The entities and relationships in the multi-modal power field operation risk cognition graph are mapped into a low-dimensional vector space by the TransD model, and the entities and relationships mapped into the low-dimensional vector space are reasoned by the random walk algorithm, and the reasoning process is as follows:

[0252] The eigenvalue function s of each path is calculated h，Q(t) , so as to obtain a plurality of paths; the eigenvalue function s h，Q(t) is the probability of reaching the tail entity vector t from the head entity vector h along the path Q; the path Q is composed of a plurality of relationship vectors:

[0253]

[0254] Wherein, T z-1 is the scope of the relationship vector r z ;

[0255] When the path Q reaches any intermediate entity vector e, the updating method of the eigenvalue function s h，Q(t) is:

[0256]

[0257] In the initial stage of random walk, if e'∈s h,Q(e) , then the eigenvalue function s h,Q(t) =1, otherwise the eigenvalue function s h,Q(t) =0;

[0258] I(r l (e', e)) is an indicator function, if r l (e', e) exists, then I(r l (e', e))=1, otherwise I(r l (e', e))=0;

[0259] Wherein, r l(e', e) is the relationship r between any entity e' and entity e on the random walk path l , ran(Q) is the entity set on the walk path Q, s h,Q(e，) (e') is the eigenvalue vector of entity e' on the walk path;

[0260] I(r l (e', e)) is a normalization function defined by the present application, because the r l (e', e) function output value has diversity, therefore, through the self-defined function, the r l (e', e) function result is normalized.

[0261] Through the random walk algorithm, a plurality of path features Q1,..., Qn are obtained, and a score function f k (h k , t Q ) is established for each head entity and tail entity combination (h k , t k ) under the relationship vector:

[0262]

[0263] Where θ n is the score of the training sample under the relationship vector on the nth walk path feature, and the value is a floating point number of [0-1];

[0264] The probability that the head entity and tail entity combination (h k , t k ) has a relationship is P k :

[0265]

[0266] Where r k is the relationship of the kth training sample;

[0267] The loss function is:

[0268] minw k [y k lnP k +(1-y k )ln(1-P k )];

[0269] Where y k is a label of whether the head entity and tail entity combination (h k , t k ) has the relationship vector r, if the triple vector (h k , r, t k ) exists, then y k= 1, otherwise y k = 0, w k is an error size of a model predicting a relationship of a kth head entity and tail entity combination and a real combination relationship; the loss function is used for evaluating a degree of inconsistency between a model predicting a head-tail entity combination relationship and a real head-tail entity combination relationship, and guiding a model training process to continuously adjust parameters.

[0270] By constructing Schema, knowledge in the knowledge graph can be refined to form a fixed mode, and the mode can better serve the standard knowledge and subsequent knowledge application. The Schema construction in the knowledge graph is equivalent to the establishment process of the ontology (Ontology), and the concept and its superior and inferior, the attribute and its value, the relationship and the relationship domain (Domain) concept set constitute the most basic ontology. In an embodiment of the present application, the Schema for constructing the multi-modal knowledge graph comprises:

[0271] a list of core entities and attributes for constructing the power field operation risk cognition graph;

[0272] a list of core relationships for constructing the power field operation risk cognition graph.

[0273] In an embodiment, the list of core entities and attributes and the list of core relationships are shown in the following table.

[0274] Table 5: List of core entities and attributes of the power field operation risk cognition graph

[0275]

[0276] Table 6: List of core relationships of the power field operation risk cognition graph

[0277]

[0278]

[0279] In an embodiment of the present application, it further comprises a definition of a power field operation modal data structure.

[0280] Since power production operations involve marketing, dispatching, infrastructure, and other professions, operation data is stored in PMS, E infrastructure, safety risk management and control platform, and other business systems, and the same modal data structure is not the same, which is not convenient for unified processing of data. Therefore, it is necessary to define a standard data structure for operation text, image, video and other data, so as to facilitate the integration of data from various channels. The data structure is shown in the following table:

[0281] Table 7: Text data structure

[0282] Field name schema value type remark Identification id string Name title string Content content string When the file belongs to the document format, it can be empty Generation time date string Type type string Such as standards, specifications, work tickets, reports, etc. Suffix suffix string Storage address url string When the file is a text segment, it can be empty Size size int Unit byte

[0283] Table 8 image data structure

[0284]

[0285]

[0286] Table 9 video data structure

[0287] Field name schema value type remark Identification id string Name title string Content content string Generation time date string Type type string Such as face, qualification certificate, on-site operation Suffix suffix string Such as jpg, png, etc. Storage address url string Scene scen string The picture comes from which operation scene Associated work ticket ticket string Size size int Unit byte

[0288] Based on the above defined data structure, the data such as work order, rules and regulations, illegal picture, work monitoring video involved in power field operation can be obtained from the business systems such as PMS, E infrastructure, safety risk management and control platform through interface calling mode.

[0289] In an embodiment of the application, a multi-modal knowledge graph storage is further included, specifically, the multi-modal knowledge graph entities, relationships and attributes and other information extracted from the power field operation text, image, video and other data are stored using a Neo4j graph database to realize storage of the multi-modal knowledge graph of power field operation.

[0290] Reference Figure 8 and 9 In an embodiment of the application, the auxiliary decision making through the multi-modal power field operation risk cognition graph includes four modules of data layer, knowledge graph layer, knowledge reasoning layer and man-machine interaction layer.

[0291] Among them, the data layer mainly stores the data such as work order, rules and regulations, personnel / equipment, work monitoring video, illegal picture and other data collected from the systems such as PMS, unified video platform, safety risk management and control platform;

[0292] The knowledge graph layer mainly carries out cleaning and pretreatment of the data of the data layer, and constructs the multi-modal power field operation risk cognition knowledge graph;

[0293] The knowledge reasoning layer mainly stores the multi-modal knowledge graph and graph management using a Neo4j graph database, and carries out vectorization and reasoning analysis of the entities and relationships based on the multi-modal knowledge graph;

[0294] The man-machine interaction layer mainly realizes display of the multi-modal knowledge graph, and question answering and retrieval functions based on the knowledge graph.

[0295] Relying on the multi-modal power field operation risk cognition graph auxiliary decision making system, the work content and field operation picture can be input at the front end of the system, and the system can automatically identify whether there is work illegal and risk based on the reasoning function of the multi-modal knowledge graph, if there is, the corresponding risk disposal measures, risk level and other information will be recommended, and the risk control decision is made.

[0296] Second aspect

[0297] A job safety risk identification device based on a multi-modal knowledge graph, comprising a data acquisition unit, a data processing unit and an output unit;

[0298] The data acquisition unit is used to acquire power field operation data of multiple modalities;

[0299] The data processing unit constructs the Schema of the multi-modal knowledge graph;

[0300] Knowledge extraction is performed on power field operation data of different modalities respectively to obtain labels of the power field operation data;

[0301] The labels of the power field operation data of different modalities are fused;

[0302] Knowledge storage is performed through the fused labels and the Schema to obtain a multi-modal power field operation risk cognitive graph;

[0303] Auxiliary decision is made through the multi-modal power field operation risk cognitive graph to identify power field operation risks;

[0304] The output unit outputs power field operation risks.

[0305] In an embodiment of the present application, the data types of the power field operation data include text data, image data and video data.

[0306] In an embodiment of the present application, the text data includes at least one of a job work ticket, a regulation system, personnel information, equipment information, a work report and a violation case; and further includes preprocessing of the text data, specifically:

[0307] Obtaining text data A;

[0308] Identifying the file type of the text data A; the file type includes a text file and a non-text file;

[0309] If the file type of the text data A is a text file, then output the pure text data of the text data A through a Tika component, otherwise, extract the pure text data of the text data A through an optical character recognition technology and output.

[0310] In an embodiment of the present application, knowledge extraction on the power field operation data includes key information extraction of the text data, comprising the following steps:

[0311] Obtaining power field operation data of which the data type is text data;

[0312] Defining entities and relationship types in the power field operation data;

[0313] Extracting power field operation data of a part of data type as text data, and obtaining a training set according to defined row labels;

[0314] Constructing a segmented convolutional neural network model based on an attention mechanism;

[0315] Training the segmented convolutional neural network model based on the attention mechanism through the training set;

[0316] Inputting power field operation data to be extracted, target entities and relation types into the segmented convolutional neural network model based on the attention mechanism for entity relation extraction.

[0317] In an embodiment of the present application, knowledge extraction of the power field operation data includes semantic label extraction of image data, including the following steps:

[0318] Constructing a common target list contained in power field operation images;

[0319] Constructing a common violation behavior list of power field operation through the common target list;

[0320] Performing violation behavior labeling on power field operation data of a part of data type as image data;

[0321] Training each violation behavior in the common violation behavior list through a deep learning algorithm combined with the labeled power field operation data to obtain an identification model of different violation behaviors;

[0322] Performing semantic label extraction on power field operation data of data type as image data through the identification model, and the output of the identification model is the semantic label.

[0323] In an embodiment of the present application, the semantic label extraction of the power field operation data of data type as image data through the identification model includes the following steps:

[0324] Inputting power field operation data of data type as image data into an encoder, and extracting features x={x1,x2,x3,...,xn} of n positions through a convolutional neural network of a decoder n},x n ∈R D , wherein x n is a D-dimensional vector, D is a channel number, and n is a size of a feature map;

[0325] Inputting the features x into the decoder;

[0326] Allocating different weights to the features x through an Attention mechanism of the decoder;

[0327] In the generation of the tth word of the semantic label, the context vector of the recurrent neural network of the input decoder is z t , the hidden layer state of the previous stage of the recurrent neural network is g t-1 ; the context vector z t is a weighted average of the features x, obtained by the following formula:

[0328]

[0329] Where a t,m is a weight measuring the image feature of the mth position when generating the tth word; the weight is a function of the hidden layer state g t-1 of the previous stage and the mth position image feature x m ; the word is a character of the semantic label of the image data;

[0330] The context vector is taken as the input of the long short-term memory network, and the model result y t is generated and output by the hidden variable of the long short-term memory network;

[0331] The model result y t is taken as the semantic label, and the model result y t is a k-dimensional probability, representing the probability of the tth word appearing in the power field operation data, and k is the size of the text dictionary used by the recognition model.

[0332] In an embodiment of the present application, the knowledge extraction of the power field operation data includes the semantic label extraction of the video data, including the following steps:

[0333] Obtain the power field operation data of which the data type is video data;

[0334] Extract the key frames in the video;

[0335] Extract the semantic labels of the key frames by the method of semantic label extraction of image data;

[0336] Associate the key frames with the corresponding videos, and take the semantic labels of the key frames as the attribute values of the corresponding videos.

[0337] In an embodiment of the present application, the labels of the power field operation data of different modalities are fused, including the following steps:

[0338] For the label A of the power field operation data of which the data type is text data, calculate the similarity of the label A and all labels B of the power field operation data of which the data type is image data by the longest common subsequence algorithm, and link the label A with the label B with the highest similarity.

[0339] For the label A of the power field operation data of the text data type, the similarity of the label A and all labels C of the power field operation data of the video data type is calculated by using the longest common subsequence algorithm, and the label A is linked with the label C with the highest similarity;

[0340] For the label B of the power field operation data of the image data type, the similarity of the label B and all labels C of the power field operation data of the video data type is calculated by using the longest common subsequence algorithm, and the label B is linked with the label C with the highest similarity.

[0341] In an embodiment of the present application, the multi-modal power field operation risk cognition graph is further inferred, and the potential relationship between different triple entities is inferred to perfect the multi-modal power field operation risk cognition graph, comprising the following steps:

[0342] The entities and relationships in the multi-modal power field operation risk cognition graph are mapped into a low-dimensional vector space by using a TransD model, and the entities and relationships mapped into the low-dimensional vector space are inferred by using a random walk algorithm, and the inference process is as follows:

[0343] The eigenvalue function s of each path is calculated h,Q(t) , so as to obtain a plurality of paths; the eigenvalue function s h,Q(t) is the probability of reaching the tail entity vector t from the head entity vector h along the path Q; the path Q is composed of a plurality of relationship vectors:

[0344]

[0345] Wherein, T z-1 is the scope of the relationship vector r z ;

[0346] When the path Q reaches any intermediate entity vector e, the updating method of the eigenvalue function s h,Q(t) is as follows:

[0347]

[0348] In the initial stage of the random walk, if e' is in s h,Q(e) , then the eigenvalue function s h,Q(t) =1, otherwise the eigenvalue function s h,Q(t) =0;

[0349] I(r l (e', e)) is an indicator function, if r l (e', e) exists, then I(r l (e', e))=1, otherwise I(rl (e', e)) = 0;

[0350] Where, r l (e', e) represents the relationship r between any entity e' and entity e on the random walk path. l ran(Q) is the set of entities on the walk path Q, s h，Q(e，) (e') is the feature vector of entity e' on the walking path;

[0351] Several path features Q1, ..., Q2 are obtained through a random walk algorithm. z For each head entity and tail entity combination (h) under the relation vector k , t k Establish a scoring function f Q (h k , t k ):

[0352]

[0353] Where, θ n The score of the training sample under the relation vector on the nth wandering path feature is a floating-point number in the range of [0-1].

[0354] Head and tail entity combination (h) k , t k The probability that a relationship exists is P. k :

[0355]

[0356] Where, r k The relation of the k-th training sample;

[0357] The loss function is:

[0358] minw k [y k lnP k +(1-y k )ln(1-P k )];

[0359] Among them, y k The head entity and the tail entity are combined (h) k , t k Does it have a label for the relation vector r? If the triple vector (h) k ,r,t k If y exists, then y k =1, otherwise y k =0, w kThe loss function is used to evaluate the degree of inconsistency between the relationship of the kth head entity and tail entity combination predicted by the model and the true relationship of the head entity and tail entity combination, and guide the model training process to continuously adjust the parameters.

[0360] In an embodiment of the present application, the schema for constructing the multi-modal knowledge graph comprises:

[0361] A list of core entities and attributes for constructing the power field operation risk cognition graph;

[0362] A list of core relationships for constructing the power field operation risk cognition graph.

[0363] The third aspect

[0364] An electronic device, which, when in operation, implements the method for identifying operation safety risks based on a multi-modal knowledge graph as described in the first aspect, to identify power field operation risks.

[0365] The fourth aspect

[0366] A storage medium, in which a computer program stored therein, when executed by a computer, implements the method for identifying operation safety risks based on a multi-modal knowledge graph as described in the first aspect, to identify power field operation risks.

[0367] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code. The solutions in the embodiments of the present application can be implemented in various computer languages, such as the object-oriented programming language Java and the interpreted scripting language JavaScript.

[0368] The present application is described with reference to flowcharts and / or block diagrams according to the methods, devices (systems), and computer program products of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to produce a machine, so that the instructions executed by the computer or other programmable data processing devices produce a device that implements the functions described in the flowcharts and / or block diagrams. Figure 1 one flow or multiple flows and / or blocksFigure 1 means for performing the function specified by the block or blocks.

[0369] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flow Figure 1 flow or flows and / or blocks Figure 1 means for performing the function specified by the block or blocks.

[0370] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 flow or flows and / or blocks Figure 1 steps of means for performing the function specified by the block or blocks.

[0371] Although preferred embodiments of the application have been described herein, changes and modifications can be suggested to one skilled in the art and are intended to be encompassed within the scope of the application as set forth in the appended claims. It is the intent, therefore, to be limited only as

[0372] Obviously, numerous modifications and variations of the present application are possible in light of the above teachings. It is therefore to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A method for identifying safety risks of a work based on a multi-modal knowledge graph, characterized in that, The method comprises the following steps: acquiring power field operation data of multiple modalities; constructing a Schema of a multi-modal knowledge graph; performing knowledge extraction on power field operation data of different modalities respectively to obtain labels of the power field operation data; fusing the labels of the power field operation data of different modalities; performing knowledge storage through the fused labels and the Schema to obtain a multi-modal power field operation risk cognition graph; performing auxiliary decision-making through the multi-modal power field operation risk cognition graph to identify power field operation risks; the data types of the power field operation data include text data, image data, and video data; the knowledge extraction on the power field operation data includes semantic label extraction of the image data, and comprises the following steps: constructing a list of common targets contained in power field operation images; constructing a list of common violations of regulations in power field operation through the list of common targets; annotating power field operation data of which the data type is image data in terms of violations of regulations; training each violation of regulations in the list of common violations of regulations through a deep learning algorithm in combination with the annotated power field operation data to obtain an identification model of different violations of regulations; performing semantic label extraction on power field operation data of which the data type is image data through the identification model, and the output of the identification model is the semantic label.

2. The method of claim 1, wherein, the text data includes at least one of a work order, a regulation, personnel information, equipment information, a work report, and a violation case; the method further comprises preprocessing of the text data, specifically: acquiring text data A; identifying the file type of the text data A; the file type includes a text file and a non-text file; if the file type of the text data A is a text file, outputting pure text data of the text data A through a Tika component, otherwise, extracting pure text data of the text data A through an optical character recognition technology and outputting. 3.The method of claim 1, wherein, the knowledge extraction on the power field operation data includes key information extraction of the text data, and comprises the following steps: acquiring power field operation data of which the data type is text data; defining entities and relationship types in the power field operation data; extracting part of the power field operation data of which the data type is text data and annotating according to the definition to obtain a training set; constructing a segmented convolutional neural network model based on an attention mechanism; training the segmented convolutional neural network model based on the attention mechanism through the training set; inputting power field operation data to be extracted, target entities, and relationship types into the segmented convolutional neural network model based on the attention mechanism for entity relationship extraction. 4.The method of claim 1, wherein, the semantic label extraction of the power field operation data of which the data type is image data comprises the following steps: The power field operation data of image data type is input into an encoder, and features x={x1, x2, x3,...,xn} of n positions are extracted through a convolutional neural network of the decoder n} and x n ∈R D , wherein x n is a D-dimensional vector, D is a channel number, and n is a size of a feature map. inputting a feature x into a decoder; allocating different weights to the feature x through an Attention mechanism of the decoder; At the time of generating the t-th word of the semantic label, the context vector of the recurrent neural network of the input decoder is z t , the hidden layer state of the previous stage of the recurrent neural network is g t-1 ; the context vector z t is a weighted average of the features x, obtained by the following formula: wherein a t,m is a weight of an image feature of an mth position when a tth word is generated; the weight is a function of a hidden layer state g t-1 of a previous stage and an mth position image feature x m ; the word is a character of a semantic label of the image data; inputting the context vector into a long short-term memory network, generating and outputting a model result y through a hidden variable of the long short-term memory network t ; The model result y t As a semantic label, the model result y t is a k-dimensional probability, representing the probability of the tth word appearing in the power field operation data, and k is the size of the text dictionary used by the recognition model.

5. The method of claim 1, wherein, the knowledge extraction on the power field operation data includes semantic label extraction of video data, and comprises the following steps: acquiring power field operation data of which the data type is video data; Extracting key frames in a video; Extracting semantic labels of the key frames by a method of semantic label extraction of image data; Associating the key frames with corresponding videos and taking the semantic labels of the key frames as attribute values of the corresponding videos. 6.The method of claim 1, wherein, Fusing labels of power field operation data of different modalities, including the following steps: For the label A of the power field operation data of the data type of text data, the similarity between the label A and all labels B of the power field operation data of the data type of image data is calculated by the longest common subsequence algorithm, and the label A is linked with the label B with the highest similarity; For the label A of the power field operation data of the data type of text data, the similarity between the label A and all labels C of the power field operation data of the data type of video data is calculated by the longest common subsequence algorithm, and the label A is linked with the label C with the highest similarity; For the label B of the power field operation data of the data type of image data, the similarity between the label B and all labels C of the power field operation data of the data type of video data is calculated by the longest common subsequence algorithm, and the label B is linked with the label C with the highest similarity.

7. The method of claim 1, wherein the method further comprises: Further comprising inferring the multi-modal power field operation risk cognition graph to infer the potential relationship between different triple entities, and perfecting the multi-modal power field operation risk cognition graph, including the following steps: Mapping the entities and relationships in the multi-modal power field operation risk cognition graph into a low-dimensional vector space by a TransD model, and inferring the entities and relationships mapped into the low-dimensional vector space by a random walk algorithm, and the inference process is as follows: calculating an eigenvalue function s for each path h,Q(t) , thereby obtaining a number of paths; the eigenvalue function s h,Q(t) is the probability of being able to reach the tail entity vector t from the head entity vector h along the path Q; the path Q consists of a number of relation vectors: wherein T z-1 is the scope of the relation vector r z . When the path Q reaches an arbitrary intermediate entity vector e, the eigenvalue function s h,Q(t) is updated as follows: At the initial stage of the random walk, if e' ∈ s h,Q(e’) then the characteristic function s h,Q(t) = 1, otherwise the characteristic function s h,Q(t) = 0; I(r l (e',e)) is an indicator function, which is 1 if r l (e',e) exists, and 0 otherwise. l (e',e)) = 1, otherwise I(r l (e',e)) = 0. where r l (e', e) is the relationship r between any entity e' on the random walk path and entity e l , ran(Q) is the set of entities on the walk path Q, s h,Q(e’) (e') is the eigenvector of entity e' on the walk path By random walk algorithm get several path features Q1,...,Q z For each head and tail entity combination (h k ,t k ) under the relationship vector, establish a score function f Q (h k ,t k ): where θ n is the score of the training sample under the relation vector on the nth walk path feature, taking the value of a floating point number [0-1]; The head entity and the tail entity combination (h k ,t k ) has a probability P k : wherein r k is the relationship for the kth training sample; The loss function is: minw k [y k lnP k +(1-y k )ln(1-P k )] Among them, y k The head entity and the tail entity are combined (h) k ,t k Does it have a label for the relation vector r? If the triple vector (h) k ,r,t k If y exists, then y k =1, otherwise y k =0, w k The loss function is used to measure the error between the model's prediction of the relationship between the k-th head entity and the tail entity combination and the actual relationship; the loss function is used to evaluate the degree of inconsistency between the model's prediction of the head and tail entity combination relationship and the actual head and tail entity combination relationship, and to guide the continuous adjustment of parameters during the model training process. 8.The method of claim 1, wherein, The Schema for constructing the multi-modal knowledge graph includes: A list of core entities and attributes of the power field operation risk cognition graph; A list of core relationships of the power field operation risk cognition graph.

Images