A multi-modal data fusion method, device and medium for a power market

Abstract

The application discloses a power market-oriented multi-modal data fusion method and device and medium, comprising: constructing a knowledge graph for the power market; the entity of the knowledge graph represents various indexes in the power market; the edge of the knowledge graph represents the relationship between the indexes; the entity features in the knowledge graph are preprocessed to convert various data into unified multi-dimensional vectors; anchor point entities are identified from the knowledge graph, and anchor point entity information is fused; and the knowledge graph is converted into an anchor point graph composed of fused anchor points. In this way, multi-modal fusion in the power market is realized by using the knowledge graph, various indexes are represented in the graph structure, and are characterized as unified multi-dimensional vectors, so that multi-modal multi-field data is effectively integrated, a large knowledge graph is converted into a smaller anchor point graph composed of anchor points, the number of entities is reduced, efficiency is improved, and fast, cross-platform heterogeneous data retrieval and integrated management can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of data processing, in particular to a multi-modal data fusion method for power market, device and medium. BACKGROUND

[0002] At present, the normal operation of the power system depends on the data transmission and mutual cooperation between various business systems, and these business systems are constructed in different years and on different platforms, and the databases, operation platforms and specific data structures used by them are different, which leads to the emergence of a large number of heterogeneous structured and unstructured data in the automation system, such as power grid topology and operation data in different formats, power equipment information, geographic environment data, weather data, audio and video, and a large amount of text data in different formats.

[0003] In order to realize the communication and information integration between these heterogeneous data, the power system needs to add a large number of data conversion interfaces and intermediate links between different platforms. However, these data platforms are relatively independent, and there is a lack of connection between data, making it difficult to realize fast, cross-platform data retrieval and integrated management.

[0004] Therefore, how to establish a multi-modal data fusion method for the power market is a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] Therefore, the purpose of the present application is to provide a multi-modal data fusion method for the power market, device and medium, which can effectively integrate multi-modal and multi-field data, and realize fast, cross-platform heterogeneous data retrieval and integrated management. The specific scheme is as follows:

[0006] A multi-modal data fusion method for the power market, comprising:

[0007] Constructing a knowledge graph for the power market; the entities of the knowledge graph represent a plurality of indexes in the power market; the edges of the knowledge graph represent the relationships between the indexes;

[0008] Preprocessing the entity features in the knowledge graph to convert various data into unified multi-dimensional vectors;

[0009] Identifying anchor entities from the knowledge graph and performing fusion processing on the anchor entity information;

[0010] Converting the knowledge graph into an anchor graph composed of fused anchor entities.

[0011] Preferably, in the above multi-modal data fusion method provided by the embodiment of the present application, the construction of the knowledge graph for the power market comprises:

[0012] collecting key data under a power market; a format of the key data includes structured data, semi-structured data and unstructured data;

[0013] extracting information including power market index names, power market index relationships and power market index attributes from the key data;

[0014] constructing a knowledge graph for the power market according to the extracted information.

[0015] Preferably, in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, the preprocessing of the entity features in the knowledge graph comprises:

[0016] a pre-trained Faster-RCNN is used to convert the picture into a feature vector;

[0017] a pre-trained BERT model is used to convert the text into a feature vector.

[0018] Preferably, in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, the identification of the anchor entity from the knowledge graph comprises:

[0019] key entities in the picture and the question text converted into the feature vectors are extracted as the anchor entity.

[0020] Preferably, in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, the fusion processing of the anchor entity information comprises:

[0021] the importance of the neighbors and relationships corresponding to the anchor entity is obtained;

[0022] the information of the multi-order neighbors is fused in a graph neural network manner with the importance as a weight, as the fused anchor entity.

[0023] Preferably, in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, while converting the knowledge graph into an anchor graph composed of the fused anchor entity, it further comprises:

[0024] when the edge in the knowledge graph has both endpoints on the anchor entity, the edge is retained in the anchor graph;

[0025] when the edge in the knowledge graph has neither endpoint on the anchor entity, the edge is deleted;

[0026] When the edge in the knowledge graph has one end point on the anchor point entity and the other end point on a non-anchor point entity, find another anchor point entity closest to the non-anchor point entity on the knowledge graph, and calculate the distance between the non-anchor point entity and the other anchor point entity; if the distance is less than a set value, add an edge from the anchor point entity to the other anchor point entity in the anchor point graph; if the distance is not less than the set value, delete the edge.

[0027] Preferably, in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, after the knowledge graph facing the power market is constructed, the method further comprises:

[0028] The knowledge graph is co-reference resolved and entity disambiguated to enable knowledge fusion.

[0029] Preferably, in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, after the knowledge graph is converted into the anchor point graph composed of the fused anchor point entities, the method further comprises:

[0030] The confidence of the anchor point entity in the anchor point graph is quantified, and the anchor point entity corresponding to the confidence less than a preset threshold is discarded.

[0031] The embodiment of the present application also provides a multi-modal data fusion device facing the power market, comprising a processor and a memory, wherein the processor realizes the above-mentioned multi-modal data fusion method provided by the embodiment of the present application when executing the computer program stored in the memory.

[0032] The embodiment of the present application also provides a computer readable storage medium for storing a computer program, wherein the computer program is executed by a processor to realize the above-mentioned multi-modal data fusion method provided by the embodiment of the present application.

[0033] As can be seen from the above technical solutions, the multi-modal data fusion method facing the power market provided by the present application comprises: constructing a knowledge graph facing the power market; the entity of the knowledge graph represents various indicators in the power market; the edge of the knowledge graph represents the relationship between the indicators; the entity features in the knowledge graph are preprocessed to convert various data into unified multi-dimensional vectors; the anchor point entities are identified from the knowledge graph, and the anchor point entity information is fused; the knowledge graph is converted into an anchor point graph composed of the fused anchor point entities.

[0034] The present application realizes multi-modal fusion in the power market by using the knowledge graph, represents various indicators into the graph structure, and represents as unified multi-dimensional vectors, thereby facilitating effective integration of multi-modal multi-field data, converting a large knowledge graph into a smaller anchor point graph composed of anchor points, reducing the number of entities, improving efficiency, and realizing fast, cross-platform heterogeneous data retrieval and integrated management.

[0035] In addition, the application also provides a corresponding device and a computer readable storage medium for the multi-modal data fusion method, so that the method is more practical, and the device and the computer readable storage medium have corresponding advantages. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the application or the related art, the following will briefly introduce the drawings needed to be used in the embodiments or the related art description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can be obtained without creative labor on the basis of the provided drawings.

[0037] Figure 1 A flowchart of the multi-modal data fusion method for the power market provided by the embodiment of the application;

[0038] Figure 2 A schematic diagram of the mutual relationship of the power market index system provided by the embodiment of the application;

[0039] Figure 3 A schematic diagram of the power market knowledge extraction provided by the embodiment of the application;

[0040] Figure 4 A schematic diagram of the body structure of the power market data index provided by the embodiment of the application;

[0041] Figure 5 A schematic diagram of the anchor point frame corresponding to the index fusion relationship of the power market provided by the embodiment of the application. DETAILED DESCRIPTION

[0042] The technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are only some of the embodiments of the application, not all the embodiments. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the protection scope of the application.

[0043] The application provides a multi-modal data fusion method for the power market, as shown in Figure 1 The method comprises the following steps:

[0044] S101, constructing a knowledge graph for the power market; the entity of the knowledge graph represents a plurality of indexes in the power market; and the edge of the knowledge graph represents the relationship between the indexes;

[0045] Specifically, various indicators in the electricity market are represented as entities e in the graph, and the relationship between the indicators is represented as an edge in the graph, so an edge can be represented as a triple (e h , r, e t ) in the graph, indicating that the head entity e h and the tail entity e t have a certain relationship r knowledge graph construction process is from raw data, using a series of automatic or semi-automatic technical means, extracting knowledge elements (i.e. facts) from raw data, and storing them in the knowledge base. It is an iterative updating process.

[0046] In practical applications, the common indicator system of each electricity market and the unique indicator system of each electricity market are extracted, and the different levels of indicator items of each electricity market indicator system are used as reference, as shown in Figure 2 , the indicator system covering the whole process of the electricity market is summarized and refined, including energy economy indicators, power system operation indicators, power market transaction indicators, and power market management indicators. Each indicator system has a strong correlation. For example, the power market transaction indicators, changes in the economic situation will affect the market expectations of power users, and then affect the transaction size of their participation in the power market transaction, the power transaction size is constrained by the power system operation, the energy price will affect the cost of the power generation enterprise, and then affect its offer in the power transaction, the fluctuation of the power transaction price will be affected by the market force of the market main body, and the market needs to be managed to ensure the stable operation of the power market.

[0047] S102, pre-process the entity features in the knowledge graph to convert various data into unified multi-dimensional vectors;

[0048] It should be noted that due to the multi-modal characteristics of the electricity market features, some features are pictures and some features are text, so the present application adopts a pre-processing method to represent different modal features as feature vectors for easy processing.

[0049] S103, identifying anchor entities from the knowledge graph and fusing anchor entity information;

[0050] S104, converting the knowledge graph into an anchor graph composed of fused anchor entities.

[0051] In practical applications, after obtaining the anchor entity features, the next step is to delete the non-anchor entities in the graph. Specifically, the key anchor entities identified in the original knowledge graph are extracted as the points of the final anchor graph, so as to convert the original large knowledge graph into a smaller anchor knowledge graph composed of anchor points, thereby greatly improving the performance.

[0052] In the multi-modal data fusion method provided in the embodiment of the present application, the knowledge graph is used to realize multi-modal fusion in the electricity market, various indexes are represented in the graph structure, and are represented as unified multi-dimensional vectors, so that multi-modal multi-field data is effectively integrated, a large knowledge graph is converted into an anchor point graph composed of anchor points, the number of entities is reduced, efficiency is improved, and fast, cross-platform heterogeneous data retrieval and integrated management can be realized.

[0053] In a specific implementation, in the multi-modal data fusion method provided in the embodiment of the present application, the step S101 of constructing the knowledge graph for the electricity market can include: collecting key data under the electricity market; the format of the key data can include structured data, semi-structured data and unstructured data; extracting information including electricity market index name, electricity market index relationship and electricity market index attribute from the key data; and constructing the knowledge graph for the electricity market according to the extracted information.

[0054] Specifically, the key data related to the market participants and the electricity market is sorted out, and the main data sources include government departments such as the National Bureau of Statistics, the National Development and Reform Commission (Energy Bureau), the Beijing Electricity Trading Center, the State Grid Corporation, provincial power grid companies, various power generation enterprises inside and outside the province, power users in the province, various power selling companies, etc. The data types of various market participants are diverse, including energy data of the National Energy Bureau, economic operation data of the National Bureau of Statistics, operation data of the power grid, operation data of the power generation enterprise, transaction data and power consumption data of the power user and the power selling company, market management data of the trading center, and analysis data of the third-party professional consulting agency, etc.

[0055] Due to the involvement of various market participants, the data volume is huge, and the data caliber is not consistent. The data formats involved include various types, including structured data, semi-structured data and unstructured data. The structured data is derived from the new generation of electricity trading platform, financial ERP system, dispatching system and financial system; the semi-structured data is derived from XML files and JSON files; and the unstructured data is mainly various Word files, Excel files, PDF files, picture files, video files and audio files. The data formats of various types of data involved need to be processed. For structured data and semi-structured data, tools such as D2R (DRF format converter) and wrapper (format analysis tool) can be used for processing, while text and other unstructured data need to be extracted through certain technical means, such as natural language processing technology. All data finally needs to be converted into standard data through processing.

[0056] Then, the information such as the electricity market index name, the electricity market index relationship and the electricity market index attribute is extracted.

[0057] Among them, the extraction of the power market index name can take the power market operation index as the parent index, and decompose each index layer by layer according to the index definition to form the child index at each level. All parent and child indexes constitute the entity set E = {index 1, index 2, …, index n} of the knowledge graph. For example, the unit type E1 = {gas unit, coal-fired unit, nuclear power unit, wind power unit, photovoltaic unit}, and the power transaction index system E2 = {medium and long-term market transaction, spot market transaction, generation right market}. As shown in Figure 3 , through index extraction, it can be known that unit 1 is a gas unit, unit 2 is a coal-fired unit, and unit 3 is a nuclear power unit.

[0058] The power market index relationship extraction is carried out from two directions. One is to extract the subordinate relationship between the parent index and the child index according to the index definition; the other is to extract the association relationship between the parent indexes according to the same child indexes under different parent indexes. As shown in Figure 3 , through relationship extraction, (unit 1, A1, B1) is obtained, which indicates that the unit 1 is a gas unit and participates in the spot market transaction.

[0059] The power market index attribute extraction is mainly to extract the index level according to the index definition. The power market operation index is a first-level index, and each child index corresponds to a corresponding sub-level.

[0060] In the specific implementation, after the knowledge graph facing the power market is constructed in the above-mentioned multi-modal data fusion method provided by the embodiment of the present application, the knowledge graph can be further subjected to co-reference resolution and entity disambiguation to enable knowledge fusion.

[0061] Specifically, the co-reference resolution is mainly used to solve the problem that multiple referring terms correspond to the same entity object, such as the entities indicated by “Zheneng” “Zheneng Changxing” and “Changxing Power Plant” in the index entity set being the same, which can be uniformly named as “Changxing Power Plant”; the entity disambiguation is to process the problem that the entity reference of part of the indexes is not clear, such as the entities indicated by “power generation enterprise” “power plant” and “power generation company” in the index entity set being the same, which can be uniformly named as “power generation enterprise”.

[0062] It should be understood that the ontology construction of the knowledge graph refers to the representation structure of the three groups of indexes, as shown in Figure 4 , taking the ontology construction of the “day-ahead market data index” as an example, the entity set of the index is {day-ahead market data index, unit parameter index, generation side declaration index, out-of-clearing data index, unit start-up cost, generation unit declaration price, generation unit declaration output, day-ahead market each transaction period electric energy node price}, the relationship set is {contains}, and the attribute set is {first level, second level, third level}. Figure 4 The middle line segment displays the subordinate relationship between the parent and child indexes; the size and color of the circle distinguish the index level attribute.

[0063] In the implementation, in the multi-modal data fusion method provided by the embodiment of the application, the step S102 of preprocessing the entity features in the knowledge graph can include: converting the picture into a feature vector by using a pre-trained Faster-RCNN; and converting the text into a feature vector by using a pre-trained BERT model.

[0064] Specifically, for the picture, the pre-trained Faster-RCNN is used to convert the picture content into a vector representation; and for the text, the pre-trained BERT model is used to represent the document as a vector. This processing method converts various types of data into unified mathematical symbols, which is beneficial to subsequent knowledge fusion.

[0065] In the implementation, in the multi-modal data fusion method provided by the embodiment of the application, the step S103 of identifying the anchor entity from the knowledge graph can include: extracting key entities as anchor entities from the picture and the question text which have been converted into feature vectors.

[0066] Specifically, for the question text, the extraction of key entities is actually a keyword extraction task, and some existing tools such as Stanford NLP Dependency Parser and Stanford Named Entity Recognizer are used to extract nouns, verbs and named entities as key entities. For the picture, the target detection technology is used to extract the entities in the picture, and the pre-trained Faster-RCNN is used to detect the objects in the picture, and the names of the objects can be obtained, and the names are aligned to the entities in the knowledge graph. The key entities extracted from the picture and the text are used as anchor entities.

[0067] In the implementation, in the multi-modal data fusion method provided by the embodiment of the application, the step S103 of fusing the anchor entity information can include: obtaining the importance degrees of the neighbors and the relationships corresponding to the anchor entity; and using the importance degrees as weights to fuse the information of the multi-order neighbors by using a graph neural network, as the fused anchor entity.

[0068] Specifically, in order to reduce the number of graph entities and increase the calculation efficiency, the anchor entities in the graph are aggregated by using a weighted summation method. For an anchor entity e, the related entities and relationships are denoted as (e i ,r i ), and two rules are given: the first rule is that the more similar the question text vector is, the higher the importance degree is; and the second rule is that if the neighbor is also an anchor entity, the importance degree is high.

[0069] Based on this, for anchor entity e, all the important degrees s(r i ,e i ) of the aggregated neighbor entities and relations are written as a formula:

[0070]

[0071] wherein E0 represents a set of anchor entities, is a learnable vector representing the (e,e i ) relation, h q is the last hidden layer feature extracted using LSTM, represents a feature vector of the entire question text, I = 1, 2, …, n, represents the Ith anchor entity. After modeling the importance degree, the importance degree can be used for aggregation. A relatively shallow graph neural network is adopted to aggregate the information of multiple orders of neighbors, map multiple types of information to a unified representation space, thereby realizing knowledge aggregation and facilitating the operation of downstream tasks, as follows:

[0072] First, the neighbor information of entities and relations in t-1 rounds is aggregated to obtain the information from neighbors in t rounds

[0073]

[0074] Then, the entity information and neighbor features are integrated as the new entity features in t rounds:

[0075]

[0076] First, the weights of the self and the neighbors are calculated Then, the entity features in t rounds are obtained by weighted summation. At this point, the algorithm has aggregated the anchor entities of the knowledge graph, encoded the information of the entity domain in the knowledge graph into the representation vector of each entity, thereby reducing the number of entities, improving the efficiency, and realizing knowledge fusion.

[0077] In the specific implementation, in the above multi-modal data fusion method provided by the embodiment of the application, when the knowledge graph is converted into an anchor point graph composed of fused anchor point entities in step S104, the following operations can also be included: when an edge in the knowledge graph has both endpoints on the anchor point entities, the edge is retained in the anchor point graph; when an edge in the knowledge graph has both endpoints not on the anchor point entities, the edge is deleted; when an edge in the knowledge graph has one endpoint on an anchor point entity u and the other endpoint on a non-anchor point entity v, another anchor point entity g closest to the non-anchor point entity v is found on the knowledge graph, and the distance l between the non-anchor point entity v and the other anchor point entity g is calculated; if the distance l is less than a set value h, an edge from the anchor point entity u to the other anchor point entity g is added to the anchor point graph; if the distance is not less than the set value, the edge is deleted.

[0078] In a specific implementation, in the multi-modal data fusion method provided by the embodiments of the present application, after the step S104 of converting the knowledge graph into an anchor point graph composed of fused anchor point entities, the following step can also be included: quantifying the confidence of the anchor point entities in the anchor point graph, and discarding the anchor point entities corresponding to the confidence less than a preset threshold. In this way, the quality of the anchor point graph is guaranteed by discarding the knowledge with low confidence.

[0079] Further, after obtaining the anchor point graph, the coverage of the indicators, the depth of the indicator hierarchy, and the correlation strength between parent indicators can also be evaluated to ensure the quality of the graph, which is mainly completed based on the judgment of relevant business experts. Taking the "day-ahead market data indicators" as an example, according to the actual business, its sub-indicators "power generation side declaration data" can be further supplemented as "power generation unit declaration minimum price", "power generation unit declaration maximum price", "power generation unit declaration median price", "power generation unit declaration average price", "power generation unit declaration minimum output", "power generation unit declaration maximum output", "total output of different types of units declared", and "total output declared in the same price segment", which further enriches the content of the three-level indicators.

[0080] It can be understood that after the two steps of indicator extraction and knowledge fusion, a complete set of indicator entities and attribute sets can be established, and knowledge reasoning can also be performed on them, only the indicator relationship set is perfected, and the implicit knowledge needs to be further mined based on the indicator definition, so as to enrich and expand the knowledge base. Indicator relationship reasoning mines valuable information from two types of original information: indicator business logic and indicator historical data.

[0081] Based on the indicator business logic, the real relationship between businesses is mined by understanding the business corresponding to each level of indicators in the entity set, thereby enriching the correlation between indicators. For example, the income of a power generation enterprise is the electricity sales income, so there is a correlation between "unit power generation" and "transaction power" in the business logic, and then the parent indicator "power supply side indicator" and the parent indicator "electricity transaction indicator" have a correlation.

[0082] Based on the indicator historical data, mathematical statistical methods such as clustering analysis, correlation analysis, and association rules are used to mine potential relationships based on the historical data of each level of indicators. For example, based on the historical data of the two indicators "supply-demand ratio" and "medium and long-term monthly price", the correlation coefficient is calculated, and it is found that the two are highly positively correlated, that is, there is a certain positive correlation between "supply-demand ratio" and "medium and long-term monthly price".

[0083] Figure 5The anchor point framework corresponding to the power market index fusion relationship is shown. The square box in the figure represents a specific index, that is, an entity of a knowledge graph. The index level attribute is distinguished by an arrow and color, and the relationship between indexes is represented by a dashed line.

[0084] Correspondingly, the embodiment of the present application also discloses a multi-modal data fusion device for a power market, comprising a processor and a memory; wherein the processor realizes the multi-modal data fusion method disclosed in the foregoing embodiment when executing the computer program stored in the memory.

[0085] The more specific process of the foregoing method can refer to the corresponding content disclosed in the foregoing embodiment, which will not be described here.

[0086] Further, the present application also discloses a computer readable storage medium for storing a computer program; the computer program is executed by the processor to realize the foregoing disclosed multi-modal data fusion method.

[0087] The more specific process of the foregoing method can refer to the corresponding content disclosed in the foregoing embodiment, which will not be described here.

[0088] Each embodiment in the specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device and storage medium disclosed by the embodiment, since it corresponds to the method disclosed by the embodiment, the description is relatively simple, and the related parts can refer to the method part.

[0089] The professional person can further realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware, computer software or a combination of the two. In order to clearly show the interchangeability of hardware and software, the composition and steps of each example have been described in the above description. Whether the functions are executed in hardware or software depends on the specific application and design constraints of the technical solution. The professional person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0090] The steps of the method or algorithm described in combination with the embodiments disclosed herein can be directly implemented by hardware, a software module executed by a processor, or a combination of the two. The software module can be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0091] In summary, the multi-modal data fusion method for the electricity market provided by the embodiment of the present application comprises: constructing a knowledge graph for the electricity market; the entities of the knowledge graph represent various indicators in the electricity market; the edges of the knowledge graph represent the relationships between the indicators; the entity features in the knowledge graph are preprocessed to convert various data into unified multi-dimensional vectors; anchor entities are identified from the knowledge graph, and the anchor entity information is fused; and the knowledge graph is converted into an anchor graph composed of fused anchor entities. The present application realizes multi-modal fusion in the electricity market by using a knowledge graph, represents various indicators in a graph structure, and represents them as unified multi-dimensional vectors, thereby facilitating effective integration of multi-modal and multi-field data, converting a large knowledge graph into a smaller anchor graph composed of anchors, reducing the number of entities, improving efficiency, and realizing fast, cross-platform heterogeneous data retrieval and integrated management. In addition, the present application also provides corresponding equipment and computer readable storage media for the multi-modal data fusion method, further making the above method more practical, and the equipment and computer readable storage media have corresponding advantages.

[0092] Finally, it should also be noted that in this document, relational terms such as first and second and the like can only be used to distinguish one entity or action from another entity or action, without necessarily requiring or implying that there is any such actual relationship or order between these entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof are intended to cover non-exclusive inclusions, so that a process, method, article, or apparatus including a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article, or apparatus. Without more limitations, the element defined by the statement "comprises a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus including the element.

[0093] The multi-modal data fusion method for the electricity market provided by the present application, the equipment and the medium are introduced in detail above, and the principles and implementation modes of the present application are described in this document by applying specific examples. The above description of the embodiments is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed; in summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A multi-modal data fusion method for electricity market, characterized in that, The method comprises the following steps: constructing a knowledge graph for the power market; entities in the knowledge graph represent various indicators in the power market; the edges of the knowledge graph represent the relationships between the indicators; preprocessing the entity features in the knowledge graph to convert various types of data into uniform multi-dimensional vectors; identifying anchor entities from the knowledge graph and fusing anchor entity information; the anchor entities include keywords in the text and object entities in the pictures; convert the knowledge graph into an anchor graph composed of fused anchor entities; wherein, while converting the knowledge graph into an anchor graph composed of fused anchor entities, it also includes: when there is an edge in the knowledge graph whose two endpoints are both anchor entities, keep this edge in the anchor graph; when there is an edge in the knowledge graph whose two endpoints are not anchor entities, delete this edge; when there is an edge in the knowledge graph whose one endpoint is an anchor entity and the other endpoint is a non-anchor entity, find another anchor entity closest to the non-anchor entity in the knowledge graph, and calculate the distance between the non-anchor entity and the other anchor entity; if the distance is less than a set value, add an edge between the anchor entity and the other anchor entity in the anchor graph; if the distance is not less than the set value, delete this edge.

2. The multi-modal data fusion method of claim 1, wherein, The method of constructing a knowledge graph for the power market comprises the following steps: collecting key data under the power market; the format of the key data includes structured data, semi-structured data and unstructured data; extracting information including power market indicator names, power market indicator relationships and power market indicator attributes from the key data; constructing a knowledge graph for the power market according to the extracted information.

3. The multi-modal data fusion method of claim 2, wherein, The preprocessing of the entity features in the knowledge graph comprises the following steps: using a pre-trained Faster-RCNN to convert pictures into feature vectors; using a pre-trained BERT model to convert text into feature vectors.

4. The multi-modal data fusion method of claim 3, wherein, The method of identifying anchor entities from the knowledge graph comprises the following steps: extracting key entities from the pictures and question texts converted into feature vectors as anchor entities.

5. The multi-modal data fusion method of claim 4, wherein, The fusion processing of the anchor entity information comprises the following steps: obtaining the importance of the neighbors and relationships corresponding to the anchor entities; using a graph neural network to fuse the information of multi-order neighbors with the importance as weights as the fused anchor entities.

6. The multi-modal data fusion method of claim 5, wherein, After constructing the knowledge graph for the power market, the method further comprises the following steps: performing co-reference resolution and entity disambiguation on the knowledge graph to enable knowledge fusion.

7. The multi-modal data fusion method of claim 6, wherein, After converting the knowledge graph into an anchor graph composed of fused anchor entities, the method further comprises the following steps: quantifying the confidence of the anchor entities in the anchor graph and discarding the anchor entities corresponding to the confidence less than a preset threshold.

8. A multi-modal data fusion device for power market, characterized by, A computer program product comprising a processor and a memory, wherein the processor executes the computer program stored in the memory to implement the multi-modal data fusion method of any one of claims 1 to 7.

9. A computer-readable storage medium, characterized in that, A computer program product for storing a computer program, wherein the computer program is executed by a processor to implement the multi-modal data fusion method of any one of claims 1 to 7.

Images