Power cross-modal bidirectional knowledge transfer method and apparatus, and device and storage medium

Abstract

The present invention relates to the technical field of knowledge transfer, and specifically to a power cross-modal bidirectional knowledge transfer method and apparatus based on intermediary space construction, and a device and a storage medium. The method comprises: using graph modeling and a graph neural network to extract a first feature of first modal data, and using a large language model to extract a second feature of second modal data; on the basis of the similarity between the first feature and the second feature and a corresponding loss function, constructing an intermediary space; using a loss function in the intermediary space to iteratively optimize parameters in the graph neural network and the large language model; and on the basis of the graph modeling, the graph neural network with parameters iteratively optimized, and the feature extracted by the large language model, performing knowledge transfer in the intermediary space. By means of the method, relationship features extracted in different modes are aligned in an intermediary space, thereby realizing bidirectional knowledge transfer between different modal data.

Description

Power cross-modal bidirectional knowledge transfer method, device, equipment and storage medium

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is based on the Chinese patent application with application number 202410232948.3, application date March 1, 2024, and invention name “A method for cross-modal bidirectional knowledge transfer of electric power based on intermediary space construction”, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby introduced into this application as a reference. Technical Field

[0003] The present invention relates to the field of knowledge migration technology, and in particular to a method and apparatus, device and storage medium for cross-modal bidirectional knowledge migration of electric power based on an intermediary space. Background Art

[0004] Knowledge transfer refers to the process of applying knowledge learned from one domain or task to another, such as applying knowledge learned in the visual domain to the text domain. It helps us leverage existing knowledge and experience to solve new problems or tasks, thereby accelerating learning and improving performance. The goal of knowledge transfer is to improve learning efficiency and performance for new tasks by leveraging existing knowledge and experience. In the power sector, knowledge transfer from visual to text domains is used in many business scenarios.

[0005] In the power sector, many business scenarios require knowledge transfer from the visual domain to the textual domain. However, due to the modality differences between the visual domain and the textual domain, direct knowledge transfer is not possible.

[0006] Summary of the Invention

[0007] In view of this, the present invention provides a method and apparatus, device and storage medium for cross-modal bidirectional knowledge migration of electric power based on an intermediary space to solve the problem of how to migrate knowledge between cross-modal data.

[0008] In a first aspect, the present invention provides a method for bidirectional cross-modal knowledge transfer of electric power based on an intermediary space, the method comprising:

[0009] Graph modeling and graph neural networks are used to extract the first features of the first modality data, and a large language model is used to extract the second features of the second modality data;

[0010] Constructing an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function;

[0011] Use loss functions in the intermediate space to iteratively optimize parameters in graph neural networks and large language models;

[0012] Knowledge transfer is performed in the intermediary space based on graph modeling, graph neural networks after iterative parameter optimization, and features extracted from large language models.

[0013] The embodiment of the present invention provides a method for bidirectional cross-modal knowledge transfer in electric power based on an intermediary space. The method uses graph modeling and a graph neural network to extract the first feature of the first modal data, and a large language model to extract the second feature of the second modal data. The method also constructs an intermediary space based on the similarity between the first and second features and the corresponding loss function. The method uses the loss function in the intermediary space to iteratively optimize the parameters in the graph neural network and the large language model. Finally, the method transfers knowledge in the intermediary space based on the features extracted from the graph neural network and the large language model after graph modeling and iterative parameter optimization. Thus, the method aligns the relational features extracted from different modalities in the intermediary space, achieving bidirectional knowledge transfer between different modal data.

[0014] In an optional embodiment, the first modal data is image data, and using graph modeling and graph neural network to extract the first feature of the first modal data includes: using graph modeling to extract the object node set, connection node set and node labels of the first modal data; using graph neural network to respectively extract the first node features of the object node set and the first connection features of the connection node set; determining the node label semantic features and the connection label semantic features based on the product of the weight matrix of the graph neural network and the node label; and determining the first feature of the first modal data based on the joint optimization of the first node features and node label semantic features and the first connection features and connection label semantic features.

[0015] In an optional embodiment, the first feature of the first modal data is determined based on the joint optimization of the first node feature and the node label semantic feature and the first connection feature and the connection label semantic feature, including: jointly processing the first node feature and the node label semantic feature based on the activation function and the weight matrix of the graph neural network to obtain the initial joint node feature; jointly processing the first connection feature and the connection label semantic feature based on the activation function and the weight matrix of the graph neural network to obtain the initial joint connection feature; optimizing the joint node feature based on the initial joint node feature, optimizing the joint connection feature based on the initial joint connection feature and the initial joint node feature of the corresponding connection, to obtain the optimized joint node feature and joint connection feature, and the joint node feature and the joint connection feature constitute the first feature.

[0016] In this embodiment, the first node features and node label semantic features as well as the first connection features and connection label semantic features are extracted through a graph neural network, and feature optimization and update are performed using a joint optimization method, which can effectively propagate and integrate information, thereby obtaining a global context on the entire image data.

[0017] In an optional embodiment, the second modal data is text data, and a large language model is used to extract a second feature of the second modal data, including: obtaining a word order path of the second modal data as an object node and a semantic feature of the second modal data as a connecting node; using a large language model to extract the word order feature of the object node and the semantic feature of the connecting node, the word order feature and the semantic feature constitute the second feature.

[0018] In this embodiment, text feature extraction is performed based on word order-semantic context feature representation, and specifically a large language model is used for feature extraction, which can effectively capture the contextual relationship contained in the word order and semantics of the text.

[0019] In an optional embodiment, an intermediate space is constructed based on the similarity between the first feature and the second feature and the corresponding loss function, including: calculating the similarity between the first feature and the second feature; constructing a loss function based on maximizing the similarity, and the similarity and the loss function constitute the intermediate space.

[0020] In this embodiment, by calculating similarity and constructing a loss function by maximizing similarity, it is possible to accurately realize bidirectional knowledge transfer between different modalities in an intermediary space constructed based on similarity and loss function.

[0021] In an optional embodiment, the loss function is expressed using the following formula:

[0022] In the formula, S() represents the similarity, α1 and α2 represent hyperparameters, and x i Represents the anchor point, x j represents a positive sample, x k Represents the first negative sample, x l represents the second negative sample, M represents the modality, M=1 represents the first modality, and M=2 represents the second modality.

[0023] In an optional embodiment, the first feature includes a joint node feature and a joint connection feature, and the second feature includes a word order feature and a semantic feature; calculating the similarity between the first feature and the second feature includes: using an inner product to calculate a first similarity between the joint node feature and the word order feature; using an inner product to calculate a second similarity between the joint connection feature and the semantic feature; and adding the first similarity and the second similarity to obtain the similarity between the first feature and the second feature.

[0024] In this embodiment, since the first feature and the second feature each include two features, the similarities of the corresponding features are calculated respectively, and then the two similarities are added, so that accurate calculation of the similarities can be achieved.

[0025] In a second aspect, the present invention provides a power cross-modal bidirectional knowledge transfer device constructed based on an intermediary space, the device comprising:

[0026] a feature extraction portion configured to extract a first feature of the first modal data using graph modeling and a graph neural network, and to extract a second feature of the second modal data using a large language model;

[0027] The space construction part is configured to construct an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function;

[0028] The optimization part is configured to iteratively optimize the parameters in the graph neural network and the large language model using the loss function in the intermediate space;

[0029] The knowledge transfer part is configured to perform knowledge transfer in the intermediary space based on graph modeling, graph neural networks after iterative parameter optimization, and features extracted from large language models.

[0030] In a third aspect, the present invention provides a computer device comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the power cross-modal bidirectional knowledge transfer method based on an intermediary space constructed in the above-mentioned first aspect or any corresponding embodiment thereof.

[0031] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the power cross-modal bidirectional knowledge transfer method constructed based on an intermediary space according to the above-mentioned first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] FIG1 is a flow chart of a method for bidirectional cross-modal knowledge transfer in electric power based on an intermediary space according to an embodiment of the present invention;

[0034] FIG2 is a flow chart of another method for bidirectional cross-modal knowledge transfer in electric power based on an intermediary space according to an embodiment of the present invention;

[0035] 3 is a structural block diagram of a power cross-modal bidirectional knowledge transfer device constructed based on an intermediary space according to an embodiment of the present invention;

[0036] FIG4 is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative efforts shall fall within the scope of protection of the present invention.

[0038] According to an embodiment of the present invention, an embodiment of a method for cross-modal bidirectional knowledge transfer of electric power based on an intermediary space is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] In this embodiment, a method for bidirectional cross-modal knowledge transfer of electric power based on an intermediary space is provided, which can be used in electronic devices such as computers, mobile phones, and tablet computers. FIG1 is a flow chart of the method for bidirectional cross-modal knowledge transfer of electric power based on an intermediary space according to an embodiment of the present invention. As shown in FIG1 , the process includes the following steps:

[0040] S101: Use graph modeling and graph neural network to extract the first feature of the first modal data, and use a large language model to extract the second feature of the second modal data.

[0041] Specifically, the first modal data and the second modal data belong to data of different modalities. For example, the first modal data is visual domain data, and the second modal data is text domain data, that is, the first modal data can be image data, and the second modal data can be text data. When extracting the first feature of the first modal data, graph modeling and graph neural network algorithms are used, wherein the graph modeling algorithm can adopt the relational modeling algorithm in the relevant technology, so as to realize the relationship mining in the first modal data. For the mined relationship, the graph neural network algorithm can be used to extract the features therein, so as to realize the accurate extraction of the first feature in the first modal data. When extracting features from the second modal data, any large language model in the relevant technology can be used. The large language model refers to a deep learning model trained with a large amount of text data, which can generate natural language text or understand the meaning of language text.

[0042] S102: constructing an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function.

[0043] Specifically, the intermediary space is used to achieve alignment between features of data from different modalities, that is, mapping the first feature of the first modal data and the second feature of the second modal data into the intermediary space, thereby achieving knowledge transfer between the first modal data and the second modal data. In this embodiment, the construction of the intermediary space is achieved through the calculation of similarity and the construction of a loss function, wherein the loss function is constructed based on the calculated similarity.

[0044] S103, using the loss function in the intermediate space to iteratively optimize the parameters in the graph neural network and the large language model.

[0045] Specifically, since the loss function is determined by calculating the similarity between the first feature of the first modal data and the second feature of the second modal data, the process of tuning the model parameters using the loss function is to make the first feature and the second feature more similar, that is, to align the features extracted by the tuned model parameters, thereby achieving knowledge transfer between different modal data. It should be noted that the process of tuning the model parameters based on the loss function can be implemented with reference to related technologies and will not be repeated here.

[0046] S104, knowledge transfer is performed in the intermediary space based on graph modeling, graph neural network after iterative parameter optimization, and features extracted from the large language model.

[0047] Specifically, after using the loss function to tune the model parameters, the graph neural network after graph modeling and parameter iterative optimization is used to extract the first feature of the first modal data, and the large language model after parameter iterative optimization is used to extract the second feature of the second modal data. At this time, the extracted first feature and second features are the most similar, so knowledge transfer can be performed in the intermediate space.

[0048] The embodiment of the present invention provides a method for bidirectional cross-modal knowledge transfer in electric power based on an intermediary space. The method uses graph modeling and a graph neural network to extract the first feature of the first modal data, and a large language model to extract the second feature of the second modal data. The method also constructs an intermediary space based on the similarity between the first and second features and the corresponding loss function. The method uses the loss function in the intermediary space to iteratively optimize the parameters in the graph neural network and the large language model. Finally, the method transfers knowledge in the intermediary space based on the features extracted from the graph neural network and the large language model after graph modeling and iterative parameter optimization. Thus, the method aligns the relational features extracted from different modalities in the intermediary space, achieving bidirectional knowledge transfer between different modal data.

[0049] In this embodiment, a method for bidirectional cross-modal knowledge transfer in electric power based on an intermediary space is provided. The process includes the following steps:

[0050] S201: Use graph modeling and graph neural network to extract the first feature of the first modal data, and use a large language model to extract the second feature of the second modal data.

[0051] Specifically, using graph modeling and graph neural network to extract the first feature of the first modal data includes:

[0052] S2011, using graph modeling to extract the object node set, connection node set and node labels of the first modal data; wherein, in this embodiment, the relational modeling method NeuralMotifs is used to extract the object node set, connection node set and node labels of the first modal data. The specific extraction method can refer to the technology of the relevant technology and will not be repeated here. For example, if the first modal data is a picture, the content extracted by using the relational modeling method can be formalized as G = {O, C, E}, where O = {o i |i=1,2,...,N o} represents the object node set, C={c ij |i=1,2,...,N c ,j=1,2,...,N c} represents the connected node set, c ij Connect i and o j The two object nodes, E represents the edge set, are virtual, not physical, and are not involved in this embodiment, so it is formalized as G={O,C}.

[0053] It should be noted that the object node set may refer to a set of objects in the picture. For example, if a picture includes two people, glasses, and food, then the set formed by these is the object node set, and the connection node set may refer to the relationship between the objects in the picture. For example, if a person in the picture is wearing glasses and another person is eating food, then wearing and eating represent the relationship between the objects, that is, wearing and eating in the picture constitute a connection node set. As for node labels, the node labels extracted by the relational modeling method NeuralMotifs are generally represented by one-hot encoding. In the picture, the node labels are specifically used to mark the specific positions of the object nodes and the connection nodes in the picture, or the node labels are used to mark the object nodes and the connection nodes in the picture. In this embodiment, one-hot encoding Lo i and Lc ij To represent node o i and c ij .

[0054] S2012, using a graph neural network to extract the first node feature of the object node set and the first connection feature of the connection node set; in this embodiment, when performing feature extraction, the YOLO-V8 algorithm is used as the graph neural network to perform feature extraction.i |i=1,2,...,N o}, extract salient block features As the first feature. For the connected node set C = {c ij |i=1,2,...,N c ,j=1,2,...,N c}, extract the intersection features between salient blocks As the first connection feature. In this way, node feature learning of object nodes and connection nodes is achieved. It should be noted that the feature extraction of object node sets and connection node sets using the YOLO-V8 algorithm can be achieved by referring to the feature extraction method of the algorithm in the relevant technology, which will not be repeated here.

[0055] S2013, based on the product of the weight matrix of the graph neural network and the node label, the node label semantic features and the connection label semantic features are determined; wherein the node label semantic features can be expressed as The semantic features of connection labels can represent Among them, W o and W c Represents the weight matrix of the neural network.

[0056] S2014: Determine a first feature of the first modal data based on joint optimization of the first node feature and the node label semantic feature, and the first connection feature and the connection label semantic feature. The joint optimization specifically includes the following steps: jointly processing the first node feature and the node label semantic feature based on the activation function and the weight matrix of the graph neural network to obtain an initial joint node feature; jointly processing the first connection feature and the connection label semantic feature based on the activation function and the weight matrix of the graph neural network to obtain an initial joint connection feature; optimizing the joint node feature based on the initial joint node feature, and optimizing the joint connection feature based on the initial joint connection feature and the initial joint node feature of the corresponding connection, to obtain an optimized joint node feature and a joint connection feature, wherein the joint node feature and the joint connection feature constitute the first feature.

[0057] Specifically, the initial joint node feature can be expressed as The initial joint connection feature can be expressed as Where W u Represents the weight matrix of the neural network, and relu represents the activation function. When performing joint optimization of features, graph convolution is used to update and Among them, the object node updates itself, and the connection node is updated by the aggregation of neighboring object nodes. Specifically, when updating, first initialize Let the joint node feature Joint connection features Where g represents a fully connected layer, followed by a ReLU activation function. The joint node features and joint connection features obtained through this update optimization process constitute the first feature.

[0058] Specifically, extracting the second feature of the second modal data using the large language model includes the following steps:

[0059] S2015, obtaining the word order path of the second modal data as the object node and the semantic features of the second modal data as the connection node; specifically, in this embodiment, the text data is used as the second modal data, and the word order features of the text data are used as the object node O'={o' i |i=1,2,...,N o'}, the semantic features of the text data are used as the connection node C'={c i '|i=1,2,...,N c' For example, a text data contains No ' words, then this N o' The word order of words in this text data is the word order path. At this time, the first node feature corresponding to the text data is a length of N. o' The semantic path is obtained by extracting keywords such as subject, predicate, and object from the word order path and removing modifiers from the word order path. When the subject, predicate, and object in the word order path are extracted as the semantic path, the length of the semantic path is 3. In this embodiment, a semantic set consisting of r' semantic paths is used as the second connection feature.

[0060] S2016, using a large language model to extract the word order features of the object node and the semantic features of the connection node, the word order features and the semantic features constitute the second feature. In this embodiment, the open source pre-trained model GPT2 is used as the large language model for feature extraction, that is, the object node O'={o' i |i=1,2,...,N o' Word order features in Extract the connection node C'={c i '|i=1,2,...,N c' Semantic features in As the second feature, the specific method of extracting features from a large language model can refer to the method of extracting features in related technologies, which will not be described in detail here.

[0061] S202: Construct an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function.

[0062] Specifically, the above S202 includes:

[0063] S2021, calculating the similarity between the first feature and the second feature; wherein the similarity calculation process specifically includes using the inner product to calculate the first similarity of the joint node feature and the word order feature; using the inner product to calculate the second similarity of the joint connection feature and the semantic feature; adding the first similarity and the second similarity to obtain the similarity between the first feature and the second feature.

[0064] That is, the first similarity of the joint node feature and the word order feature can be calculated using formula (1): S o = <N o' ·N o > (1);

[0065] Among them, N o' Indicates word order features, N o represents the joint node feature, N o and N o' They are all vectors of dimension d, and <> represents the inner product operation.

[0066] The second similarity of the joint connection feature and the semantic feature can be calculated using formula (2): S c = <N c' ·N c > (2);

[0067] Among them, N c' Represents semantic features, N c Represents the joint connection feature. The similarity between the first feature and the second feature can be calculated using formula (3): S = S o +S c (3);

[0068] S2022, construct a loss function based on the maximization of similarity. The similarity and loss function constitute an intermediary space. The loss function is expressed using the following formula (4):

[0069] In the formula, S() represents the similarity, α1 and α2 represent hyperparameters, and x i Represents the anchor point, x j represents a positive sample, x k Represents the first negative sample, x l represents the second negative sample, M represents the modality, M=1 represents the first modality, and M=2 represents the second modality.

[0070] Corresponding to this embodiment, x can be understood as a feature extracted from the first modal data or the second modal data, that is, when the subscript of x is i or j, x can be understood as the first feature extracted from the first modal data (since the first feature includes two features, it is represented by i or j respectively, and the second feature is the same), when the subscript of x is k or l, x can be understood as the second feature extracted from the second modal data, or when the subscript of x is i or j, x can be understood as the second feature extracted from the second modal data, when the subscript of x is k or l, x can be understood as the first feature extracted from the first modal data. In order to better distinguish, a superscript M is set for x. When the superscript M is 1, the corresponding x is a feature in the first modal data. When the superscript M is 2, the corresponding x is a feature in the second modal data. The subscripts i, j, k, and l of M are to correspond to the subscripts of x, that is, they have the same meaning as the subscripts of x.

[0071] S203, using the loss function in the intermediate space to iteratively optimize the parameters in the graph neural network and the large language model; please refer to S103 of the embodiment shown in Figure 1 for details, which will not be repeated here.

[0072] S204: Perform knowledge transfer in the intermediary space based on graph modeling, the graph neural network after iterative parameter optimization, and the features extracted by the large language model. Please refer to S104 of the embodiment shown in Figure 1 for details, and will not be repeated here.

[0073] As a specific application example of an embodiment of the present invention, as shown in FIG2 , taking the visual domain and the text domain as examples, the electric power cross-modal bidirectional knowledge transfer method constructed based on the intermediary space is explained: first, relationship modeling is performed in the two domains respectively, that is, the relational modeling method NeuralMotifs is used in the visual domain to extract the object node set, connection node set and node label of the first modal data, and the word order path of the second modal data is obtained as the object node and the semantic feature of the second modal data is obtained as the connection node in the text domain. Then, a graph neural network based on heterogeneous features (YOLO-V8 algorithm) and a context feature representation based on word order-semantics (large language model) are used to extract relational features from vision and text respectively. Next, an intermediary space is constructed, the extracted relational features are mapped to the intermediary space, and similarity is measured, thereby realizing bidirectional knowledge transfer from the visual domain to the text domain and from the text domain to the visual domain.

[0074] In this embodiment, a power cross-modal bidirectional knowledge transfer device based on an intermediary space is also provided. The device is used to implement the above-mentioned embodiments and preferred implementations, and the details that have been described will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware, is also possible and contemplated.

[0075] This embodiment provides a power cross-modal bidirectional knowledge transfer device based on an intermediary space, as shown in FIG3 , including:

[0076] a feature extraction portion 31 configured to extract first features of the first modal data using graph modeling and a graph neural network, and to extract second features of the second modal data using a large language model;

[0077] The space building module part 32 is configured to build an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function;

[0078] The optimization part 33 is configured to iteratively optimize the parameters in the graph neural network and the large language model using a loss function in an intermediate space;

[0079] The knowledge transfer part 34 is configured to perform knowledge transfer in the intermediate space based on graph modeling, graph neural network after parameter iterative optimization, and features extracted from the large language model.

[0080] In an optional embodiment, the first modality data is image data, and the feature extraction part 31 includes:

[0081] The first extraction part 311 is configured to extract the object node set, the connection node set and the node labels of the first modality data by using graph modeling;

[0082] The second extraction part 312 is configured to respectively extract the first node features of the object node set and the first connection features of the connection node set using a graph neural network;

[0083] The label semantic determination part 313 is configured to determine the node label semantic feature and the connection label semantic feature based on the product of the weight matrix of the graph neural network and the node label;

[0084] The extraction optimization part 314 is configured to determine the first feature of the first modality data based on the joint optimization of the first node feature and the node label semantic feature and the first connection feature and the connection label semantic feature.

[0085] In an optional embodiment, the extraction optimization part 314 is specifically configured to obtain an initial joint node feature by jointly processing the first node feature and the node label semantic feature based on the activation function and the weight matrix of the graph neural network; obtain an initial joint connection feature by jointly processing the first connection feature and the connection label semantic feature based on the activation function and the weight matrix of the graph neural network; optimize the joint node feature based on the initial joint node feature, and optimize the joint connection feature based on the initial joint connection feature and the initial joint node feature of the corresponding connection to obtain the optimized joint node feature and joint connection feature, and the joint node feature and the joint connection feature constitute the first feature.

[0086] In an optional embodiment, the second modal data is text data, and the feature extraction part 31 further includes:

[0087] The path acquisition part 315 is configured to acquire the word sequence path of the second modal data as the object node and the semantic features of the second modal data as the connection node;

[0088] The third extraction part 316 is configured to extract word order features of the object node and semantic features of the connection node using the large language model, and the word order features and semantic features constitute the second features.

[0089] In an optional embodiment, the space construction part 32 includes:

[0090] A similarity calculation part 321 is configured to calculate the similarity between the first feature and the second feature;

[0091] The construction sub-section 322 is configured to construct a loss function based on maximization of similarity, and the similarity and the loss function constitute an intermediary space.

[0092] In an optional embodiment, the loss function is expressed as follows:

[0093] In the formula, S() represents the similarity, α1 and α2 represent hyperparameters, and x i Represents the anchor point, x j represents a positive sample, x k Represents the first negative sample, x l represents the second negative sample, M represents the modality, M=1 represents the first modality, and M=2 represents the second modality.

[0094] In an optional embodiment, the first feature includes a joint node feature and a joint connection feature, and the second feature includes a word order feature and a semantic feature; the similarity calculation part 321 is specifically configured to use the inner product to calculate the first similarity of the joint node feature and the word order feature; use the inner product to calculate the second similarity of the joint connection feature and the semantic feature; and add the first similarity and the second similarity to obtain the similarity of the first feature and the second feature.

[0095] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0096] An embodiment of the present invention further provides a computer device having the power cross-modal bidirectional knowledge transfer device constructed based on the intermediary space as shown in FIG3 .

[0097] Please refer to Figure 4, which is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention. As shown in Figure 4, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. The various components are connected to each other using different buses and can be installed on a common motherboard or installed in other ways as needed. The processor can process instructions executed within the computer device, including instructions stored in or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, with each device providing some of the necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 4 takes a processor 10 as an example.

[0098] The processor 10 may be a central processing unit, a network processor, or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be an application-specific integrated circuit, a programmable logic device, or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic, or any combination thereof.

[0099] The memory 20 stores instructions that can be executed by at least one processor 10, so as to enable at least one processor 10 to execute the method shown in the above embodiment.

[0100] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created based on the use of a computer device for displaying a small program landing page, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely located relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0101] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid-state drive; the memory 20 may also include a combination of the above types of memory.

[0102] The computer device further includes a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0103] The embodiment of the present invention also provides a computer-readable storage medium. The above-mentioned method according to the embodiment of the present invention can be implemented in hardware, firmware, or implemented as a computer code that can be recorded in a storage medium, or implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and downloaded through a network and will be stored in a local storage medium, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state drive, etc.; further, the storage medium can also include a combination of the above-mentioned types of memory. It can be understood that a computer, a processor, a microprocessor controller or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor or hardware, the method shown in the above embodiment is implemented.

[0104] Although the embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention. Such modifications and variations are all within the scope defined by the appended claims. Industrial Applicability

[0105] The present invention provides a method, apparatus, device, and storage medium for cross-modal bidirectional knowledge transfer in the electric power industry based on an intermediary space. The method employs graph modeling and a graph neural network to extract a first feature of data from a first modality, and a large language model to extract a second feature of data from a second modality. An intermediary space is constructed based on the similarity between the first and second features and a corresponding loss function. The loss function in the intermediary space is used to iteratively optimize the parameters in the graph neural network and the large language model. Knowledge is then transferred in the intermediary space based on the features extracted from the graph neural network and the large language model after graph modeling and iterative parameter optimization. Thus, this method aligns relational features extracted from different modalities in the intermediary space, enabling bidirectional knowledge transfer between data from different modalities.

Claims

1. A method for cross-modal bidirectional knowledge transfer in electric power based on an intermediary space, the method comprising: Graph modeling and graph neural networks are used to extract the first features of the first modality data, and a large language model is used to extract the second features of the second modality data; Constructing an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function; Iteratively optimizing parameters in the graph neural network and the large language model using a loss function in an intermediate space; Knowledge transfer is performed in the intermediary space based on graph modeling, graph neural networks after iterative parameter optimization, and features extracted from large language models.

2. The method according to claim 1, wherein The first modal data is image data, and extracting a first feature of the first modal data using graph modeling and graph neural network includes: Extracting an object node set, a connection node set, and node labels of the first modal data using graph modeling; A graph neural network is used to extract the first node features of the object node set and the first connection features of the connection node set respectively; Determine the semantic features of node labels and connection labels based on the product of the weight matrix of the graph neural network and the node label; A first feature of the first modality data is determined based on joint optimization of the first node feature and the node label semantic feature and the first connection feature and the connection label semantic feature.

3. The method according to claim 2, wherein: Determining a first feature of the first modal data based on joint optimization of the first node feature and the node label semantic feature and the first connection feature and the connection label semantic feature includes: The initial joint node feature is obtained by jointly processing the first node feature and the node label semantic feature based on the activation function and the weight matrix of the graph neural network; The initial joint connection feature is obtained by jointly processing the first connection feature and the connection label semantic feature based on the activation function and the weight matrix of the graph neural network; The joint node feature is optimized based on the initial joint node feature, and the joint connection feature is optimized based on the initial joint connection feature and the initial joint node feature of the corresponding connection to obtain the optimized joint node feature and joint connection feature, which constitute the first feature.

4. The method according to claim 1, wherein The second modal data is text data, and the large language model is used to extract the second feature of the second modal data, including: Obtaining a word order path of the second modal data as an object node and a semantic feature of the second modal data as a connection node; A large language model is used to extract word order features of object nodes and semantic features of connection nodes, wherein the word order features and the semantic features constitute second features.

5. The method according to claim 1, wherein The intermediate space is constructed based on the similarity between the first feature and the second feature and the corresponding loss function, including: Calculate the similarity between the first feature and the second feature; A loss function is constructed based on maximizing the similarity, and the similarity and the loss function constitute an intermediate space.

6. The method according to claim 5, wherein: The loss function is expressed as follows: In the formula, S() represents the similarity, α1 and α2 represent hyperparameters, and x i Represents the anchor point, x j represents a positive sample, x k Represents the first negative sample, x l represents the second negative sample, M represents the modality, M=1 represents the first modality, and M=2 represents the second modality.

7. The method according to claim 5, wherein: The first feature includes joint node feature and joint connection feature, and the second feature includes word order feature and semantic feature; Calculate the similarity between the first feature and the second feature, including: The inner product is used to calculate the first similarity of the joint node feature and the word order feature; The inner product is used to calculate the second similarity of the joint connection feature and the semantic feature; The first similarity and the second similarity are added to obtain the similarity between the first feature and the second feature.

8. A power cross-modal bidirectional knowledge transfer device constructed based on an intermediary space, the device comprising: a feature extraction portion configured to extract a first feature of the first modal data using graph modeling and a graph neural network, and to extract a second feature of the second modal data using a large language model; The space construction part is configured to construct an intermediary space based on the similarity between the first feature and the second feature and the corresponding loss function; An optimization part, configured to iteratively optimize parameters in the graph neural network and the large language model using a loss function in an intermediate space; The knowledge transfer part is configured to perform knowledge transfer in the intermediary space based on graph modeling, graph neural networks after iterative parameter optimization, and features extracted from large language models.

9. The device according to claim 8, wherein The first modality data is image data, and the feature extraction part includes: A first extraction part is configured to extract an object node set, a connection node set, and node labels of the first modal data using graph modeling; The second extraction part is configured to respectively extract the first node feature of the object node set and the first connection feature of the connection node set using a graph neural network; The label semantic determination part is configured to determine the node label semantic features and the connection label semantic features based on the product of the weight matrix of the graph neural network and the node label; The extraction optimization part is configured to determine the first feature of the first modality data based on the joint optimization of the first node feature and the node label semantic feature and the first connection feature and the connection label semantic feature.

10. The device according to claim 9, wherein The extraction optimization part is specifically configured to obtain an initial joint node feature by jointly processing the first node feature and the node label semantic feature based on the activation function and the weight matrix of the graph neural network; Based on the activation function and the weight matrix of the graph neural network, the first connection feature and the connection label semantic feature are jointly processed to obtain the initial joint connection feature; based on the initial joint node feature, the joint node feature is Optimization: Optimizing the joint connection feature based on the initial joint connection feature and the initial joint node feature of the corresponding connection to obtain an optimized joint node feature and a joint connection feature, wherein the joint node feature and the joint connection feature constitute the first feature.

11. The device according to claim 8, wherein The second modal data is text data, and the feature extraction part further includes: a path acquisition part configured to acquire a word sequence path of the second modal data as an object node and a semantic feature of the second modal data as a connection node; The third extraction part is configured to use a large language model to extract word order features of the object node and semantic features of the connection node, wherein the word order features and the semantic features constitute the second features.

12. The device according to claim 8, wherein The space construction part includes: a similarity calculation part, configured to calculate the similarity between the first feature and the second feature; The construction sub-part is configured to construct a loss function based on the maximization of the similarity, and the similarity and the loss function constitute an intermediate space.

13. The device according to claim 12, wherein The loss function is expressed as follows: In the formula, S() represents the similarity, α1 and α2 represent hyperparameters, and x i Represents the anchor point, x j represents a positive sample, x k Represents the first negative sample, x l represents the second negative sample, M represents the modality, M=1 represents the first modality, and M=2 represents the second modality.

14. The device according to claim 12, wherein The first feature includes joint node feature and joint connection feature, and the second feature includes word order feature and semantic feature; The similarity calculation part is specifically configured to use the inner product to calculate the first similarity of the joint node feature and the word order feature; use the inner product to calculate the second similarity of the joint connection feature and the semantic feature; and add the first similarity and the second similarity to obtain the similarity of the first feature and the second feature.

15. A computer device comprising: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the power cross-modal bidirectional knowledge transfer method based on the intermediary space constructed as described in any one of claims 1 to 7 by executing the computer instructions.

16. A computer-readable storage medium having computer instructions stored thereon, wherein the computer instructions are used to enable a computer to execute the power cross-modal bidirectional knowledge transfer method based on an intermediary space as claimed in any one of claims 1 to 7.

Images