Multimodal data classification method, device, and storage medium

Abstract

The application discloses a multi-modal data classification method and device and a storage medium, and belongs to the technical field of multi-modal data processing. The application extracts feature vectors of multi-modal energy data, fuses the feature vectors into comprehensive feature vectors, and determines the association relationship between the multi-modal energy data and existing energy field knowledge; according to the association relationship and the data characteristics of the multi-modal energy data, the classification rules of the existing energy field knowledge are updated; and according to the comprehensive feature vectors and the updated classification rules, the multi-modal energy data is classified to obtain a classification result, that is, the multi-modal energy data is generated into comprehensive feature vectors, and the association relationship between the multi-modal energy data and existing knowledge is determined, so that the classification rules are updated through the association relationship and the data characteristics of the multi-modal energy data, to meet the effect of classifying the multi-modal energy data.

Description

Technical Field

[0001] This application relates to the field of multimodal data processing technology, and in particular to a multimodal data classification method, device and storage medium. Background Technology

[0002] In recent years, with the acceleration of digitalization in the energy industry, energy data has experienced explosive growth, covering a wide range of data types, including traditional structured numerical data, semi-structured text, unstructured images, and videos.

[0003] However, the multimodal nature of energy data makes the integration of different types of data a challenge. Structured data, with its clear format and definition, is easy to perform numerical calculations and analysis, but semi-structured text data and unstructured image data have high irregularity and semantic complexity, making it difficult to fully explore the potential connections between multimodal energy data. Consequently, existing data classification methods cannot accurately classify based on the rich information contained in multimodal data, thus affecting the accuracy of multimodal energy data classification results. Summary of the Invention

[0004] The main objective of this application is to provide a multimodal data classification method, device, and storage medium, aiming to solve the technical problem that existing data classification methods cannot accurately classify multimodal data based on the rich information contained in the data, thus affecting the accuracy of the classification results of multimodal energy data.

[0005] To achieve the above objectives, this application provides a multimodal data classification method, which includes the following steps: The feature vectors of multimodal energy data are extracted and fused into a comprehensive feature vector, and the correlation between the multimodal energy data and existing knowledge in the energy field is determined. Based on the aforementioned correlations and the data characteristics of the multimodal energy data, the classification rules for the existing energy domain knowledge are updated; Based on the comprehensive feature vector and the updated classification rules, the multimodal energy data is classified to obtain the classification results.

[0006] In one embodiment, the step of determining the association between the multimodal energy data and existing energy domain knowledge includes: Based on the knowledge graph corresponding to existing energy domain knowledge, determine the position and relationship of the words in the semantic space corresponding to the knowledge graph of the multimodal energy data, and determine the semantic similarity between the words and the words in the existing energy domain knowledge based on the position and the relationship. Identify the semantic information of the statements in the multimodal energy data, and calculate the semantic similarity score between the statements in the multimodal energy data and the statements in the knowledge graph based on the semantic information; The topological structure relationship of the multimodal energy data in the knowledge graph is determined, and the structural similarity score between the multimodal energy data and the knowledge graph is calculated based on the topological structure relationship. The knowledge similarity score is calculated based on the preset weights, the word sense similarity, the semantic similarity score, and the structural similarity score. Based on the knowledge similarity score, the association between the multimodal energy data and the existing energy field knowledge is determined.

[0007] In one embodiment, the step of updating the classification rules of existing energy domain knowledge based on the correlation and the data characteristics of the multimodal energy data includes: The influence coefficient is calculated based on the data characteristics of the multimodal energy data and the characteristic weights that are dynamically adjusted based on the requirements of the preset application scenarios. The data characteristics include at least importance, frequency of change, decision impact, source credibility and timeliness. Based on the influence coefficient and the correlation, the classification rules of the existing energy field knowledge are updated for the first time; Based on a preset smoothing factor, the classification rules after the initial update are smoothed multiple times until a preset stopping condition is met, at which point the smoothing process is terminated, resulting in the updated classification rules.

[0008] In one embodiment, the step of initially updating the classification rules of existing energy field knowledge based on the influence coefficient and the correlation includes: Based on the aforementioned relationships, extract a knowledge subgraph from the existing energy domain knowledge that is related to the multimodal energy data; Based on the influence coefficient, the knowledge subgraph is updated according to the tendency of different preset application scenario requirements, and a higher-level subgraph and a lower-level subgraph are generated based on the multiple knowledge subgraphs obtained from the update. The higher-level subgraph is a subgraph that covers the common knowledge of the multiple knowledge subgraphs, and the lower-level subgraph is a subgraph obtained by fusing the knowledge subgraphs in the multiple knowledge subgraphs whose relevance is higher than a preset relevance. Based on the knowledge classification tendencies corresponding to the multiple knowledge subgraphs obtained from the update, the upper-level subgraph, and the lower-level subgraph, the classification rules for the existing energy field knowledge are updated for the first time.

[0009] In one embodiment, the step of extracting feature vectors from multimodal energy data includes: The feature vectors include structured data feature vectors, semi-structured data feature vectors, and unstructured data feature vectors; When the multimodal energy data is structured data, the key attribute columns in the structured data are determined, the key attribute columns are normalized, and the structured data feature vector is generated based on the processing results. When the multimodal energy data is semi-structured data, regular expressions are used to identify key information of the semi-structured data, and a feature vector of the semi-structured data is generated based on the first word frequency feature of each word in the key information. When the multimodal energy data is unstructured data, if the unstructured data is text data, a text feature vector is generated based on the second word frequency feature of each word in the text data; if the unstructured data is image data, a geometric feature vector is generated based on the geometric features in the image data, and the unstructured data feature vector is generated based on the text feature vector and the geometric feature vector.

[0010] In one embodiment, the step of fusing the feature vectors into a comprehensive feature vector includes: The information entropy of the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector with the corresponding target variable is calculated respectively, wherein the target variable is the data type label corresponding to each type of data in the multimodal energy data; Based on the information entropy, the correlation coefficients between the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector and their corresponding target variables are calculated respectively. Based on the correlation coefficient, the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector are fused to obtain a comprehensive feature vector.

[0011] In one embodiment, the step of classifying the multimodal energy data according to the integrated feature vector and the updated classification rule to obtain the classification result includes: According to the updated classification rules, the parameters of the preset perceptron model are adjusted. The preset perceptron model is a data classification model constructed based on the existing knowledge in the energy field, as well as the corresponding knowledge graph and data samples. Based on the adjusted perceptron model, the comprehensive feature vector is classified and predicted; If the predicted value corresponding to the prediction result is less than or equal to the preset dynamic classification decision threshold, then the multimodal energy data corresponding to the comprehensive feature vector is classified according to the predicted value to obtain the classification result.

[0012] In one embodiment, after the step of classifying the multimodal energy data according to the integrated feature vector and the updated classification rule to obtain the classification result, the method further includes: The classification results are evaluated; If either the accuracy of the classification result or the consistency index of energy categories fails to meet the standard, the feature importance assessment method based on random forest is used to determine the impact of each feature dimension in the comprehensive feature vector on the classification result. Based on the aforementioned impact, adjust the weights corresponding to each feature in the comprehensive feature vector, and adjust the parameters of the perceptron model. Based on the adjusted perceptron model, the data is classified again, and the step of evaluating the classification results is returned until the results meet the expected criteria.

[0013] Furthermore, to achieve the above objectives, this application also provides a multimodal data classification device, the multimodal data classification device comprising: The data processing module is used to extract feature vectors from multimodal energy data, merge the feature vectors into a comprehensive feature vector, and determine the correlation between the multimodal energy data and existing knowledge in the energy field. The rule update module is used to update the classification rules of the existing energy field knowledge based on the correlation and the data characteristics of the multimodal energy data. The data classification module is used to classify the multimodal energy data according to the comprehensive feature vector and the updated classification rules to obtain the classification result.

[0014] In addition, to achieve the above objectives, this application also provides a multimodal data classification device, which includes: a memory, a processor, and a multimodal data classification program stored in the memory and executable on the processor, wherein the multimodal data classification program is configured to implement the steps of the multimodal data classification method as described above.

[0015] In addition, to achieve the above objectives, this application also provides a computer-readable storage medium storing a multimodal data classification program, which, when executed by a processor, implements the steps of the multimodal data classification method as described above.

[0016] The present application proposes one or more technical solutions, which have at least the following technical effects: By extracting feature vectors from multimodal energy data and fusing them into a comprehensive feature vector, the association between the multimodal energy data and existing energy domain knowledge is determined; based on the association and the data characteristics of the multimodal energy data, the classification rules of the existing energy domain knowledge are updated; based on the comprehensive feature vector and the updated classification rules, the multimodal energy data is classified to obtain a classification result. This allows for the extraction of feature vectors from multimodal energy data, the integration of these feature vectors to generate a comprehensive feature vector, the determination of the association between the multimodal energy data and existing energy domain knowledge, and the optimization and updating of the classification rules based on this association and the data characteristics of the multimodal data. This ensures that the updated classification rules meet the requirements for accurate classification of multimodal energy data. Furthermore, during classification, the fused comprehensive feature vector is directly used for classification, ensuring that rich information about various types of data in the multimodal energy data is preserved, thereby guaranteeing the accuracy of the classification results. Attached Figure Description

[0017] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a flowchart illustrating an embodiment of the multimodal data classification method of this application. Figure 2 This is a flowchart illustrating Embodiment 2 of the multimodal data classification method of this application. Figure 3 This is a flowchart illustrating Embodiment 3 of the multimodal data classification method of this application. Figure 4 This is a schematic diagram of the module structure of the multimodal data classification device according to an embodiment of this application; Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the multimodal data classification method in the embodiments of this application.

[0020] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0021] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0022] It should be noted that in recent years, with the accelerated digitalization of the energy industry, energy data has experienced explosive growth, encompassing data types ranging from traditional structured numerical data to emerging semi-structured text and unstructured images, videos, and other multimodal data. Simultaneously, energy companies urgently hope to improve energy production efficiency, optimize energy allocation, and enhance the stability and sustainability of energy systems through effective management and in-depth analysis of energy data. This application proposes a solution against this backdrop, aiming to establish a comprehensive data classification mechanism and related supporting technologies, adopting differentiated management for different types of data to realize the mining and release of data value.

[0023] However, due to the complexity and variability of energy data, as well as the limitations of existing technologies, energy data classification technology faces many challenges in practical applications: Firstly, traditional data classification methods are mostly based on single-modal data, making it difficult to fully explore the potential connections between multimodal energy data. The multimodal nature of energy data makes the integration of different data types a challenge. Structured data, with its clear format and definition, is easy to perform numerical calculations and analysis, but semi-structured text data and unstructured image data have high irregularity and semantic complexity, making it impossible to fully utilize the rich information contained in multimodal data, thus affecting the completeness and accuracy of classification results. Secondly, existing classification methods often lack dynamism. New energy technologies and market incentives are constantly emerging in the energy industry, causing the connotation and value of energy data to change dynamically. However, traditional methods struggle to update the knowledge system (knowledge not classified and incorporated into the existing energy field's knowledge graph) in real time to adapt to these changes, resulting in outdated classification standards that fail to reflect the latest value and sensitivity of the data. Thirdly, manual data classification is extremely labor-intensive and inefficient. Faced with massive amounts of energy data, relying on manual judgment of the category and level of each data point is not only time-consuming and labor-intensive, but also prone to subjective bias, failing to meet the needs of energy companies for rapid data processing and efficient utilization.

[0024] In summary, this application proposes corresponding solutions to the problems existing in current multimodal energy data classification methods. It should be noted that the implementing entity of this application is a data processing system, which includes at least a data processing module, a rule update module, and a data classification module, respectively implementing functions such as multimodal data processing, classification rule updating, and data classification, as detailed in the following embodiments.

[0025] Reference Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the multimodal data classification method of this application.

[0026] In the first embodiment, the multimodal data classification method includes the following steps: S10, extract feature vectors from multimodal energy data, merge the feature vectors into a comprehensive feature vector, and determine the correlation between the multimodal energy data and existing knowledge in the energy field; Understandably, in the energy sector, data can be categorized into three types based on its level of organization: structured data, semi-structured data, and unstructured data. In other words, the multimodal energy data in this embodiment includes at least these three types of data.

[0027] Structured data refers to data with a fixed format and defined fields that can be directly stored in relational databases or tables. For example, an energy consumption statistics table contains tabular data such as annual electricity consumption (unit: 100 million kWh) and coal / natural gas consumption (unit: 10,000 tons), which includes clear columns (year, energy type, value, etc.).

[0028] Semi-structured data refers to data that does not have a strict table structure but contains tags, labels, or other parsable content. For example, power grid dispatch instructions in XML (eXtensible Markup Language) format contain tagged information such as timestamps, load requirements, and priorities.

[0029] Unstructured data refers to data that has no fixed format and requires specific technologies to extract information. For example, geological exploration images, such as seismic wave images or core scans of oil and gas fields, require OCR (Optical Character Recognition) or computer vision analysis. Another example is handwritten wellhead reports, which are paper documents in the petroleum industry containing handwritten annotations and sketches. These require OCR and NLP (Natural Language Processing) technologies to extract key attributes (such as well depth and coordinates).

[0030] Understandably, multimodal energy data has different formats, and their corresponding feature vectors will have certain differences. However, the same content can be recorded with data in different formats at the same time. Therefore, after extracting feature vectors from data of different structural types using different methods, it is also necessary to establish the correlation between data of different structural types in order to integrate them into a comprehensive feature vector. This comprehensive feature vector should be a comprehensive data containing feature vectors of different structural types and different dimensions. At this time, when classifying the multimodal energy data, it is necessary to use the comprehensive feature vector to consider the correlation between multiple data and the rich information between multiple data.

[0031] It should be noted that, since existing data classification and knowledge graph technologies typically classify data for a single type, after determining the comprehensive feature vector, it is necessary to add new rules based on the existing data classification rules. Specifically, it is necessary to determine the relationship between multimodal energy data and existing energy domain knowledge, and based on this relationship, adjust the knowledge graph of the existing energy knowledge domain, thereby adding rules for multimodal energy data classification applications to the original knowledge graph and classification rules.

[0032] This association mainly refers to the relationship between the data of each structural type in the multimodal energy data and the knowledge classified after each single structural type in the current knowledge graph.

[0033] In this embodiment, the step of determining the correlation between the multimodal energy data and existing energy field knowledge includes: Based on the knowledge graph corresponding to existing energy domain knowledge, determine the position and relationship of the words in the semantic space corresponding to the knowledge graph of the multimodal energy data, and determine the semantic similarity between the words and the words in the existing energy domain knowledge based on the position and the relationship. Identify the semantic information of the statements in the multimodal energy data, and calculate the semantic similarity score between the statements in the multimodal energy data and the statements in the knowledge graph based on the semantic information; The topological structure relationship of the multimodal energy data in the knowledge graph is determined, and the structural similarity score between the multimodal energy data and the knowledge graph is calculated based on the topological structure relationship. The knowledge similarity score is calculated based on the preset weights, the word sense similarity, the semantic similarity score, and the structural similarity score. Based on the knowledge similarity score, the association between the multimodal energy data and the existing energy field knowledge is determined.

[0034] Understandably, when incorporating comprehensive feature vectors into existing energy domain knowledge, it is necessary to first determine the relationship between multimodal energy data and existing knowledge, and then use this relationship to revise classification rules. However, since multimodal energy data involves too much content and the formats and contents of different types of data are not uniform, it is necessary to adopt different methods for different contents in multimodal energy data, and to compare similarity from multiple dimensions according to a multi-level similarity measurement method, so as to confirm the relationship between comprehensive multi-dimensional content.

[0035] Specifically, in this embodiment, a multi-level similarity measurement method is used to extract key information from the new knowledge corresponding to multimodal energy data and perform similarity calculation with existing knowledge in the knowledge graph.

[0036] The multi-level similarity measurement method in this application mainly includes the lexical level, the sentence level, and the knowledge graph structure level.

[0037] Specifically, at the lexical level, this mainly involves combining existing ontological knowledge specific to the energy field to analyze the position and relationship of the words to be classified in the semantic space corresponding to existing energy knowledge, and to determine the degree of semantic similarity between the words of new knowledge and existing knowledge. .

[0038] Among them, the semantic space of existing energy knowledge refers to the space constructed according to the semantic content classification of existing energy knowledge. This space includes the semantic content of various words and the correlation topological relationship between words. It can be formed into triples or spatial vectors according to the correspondence between words to represent the relative position and relationship of each word in this space.

[0039] Therefore, in this embodiment, the semantic similarity between the word and the words in existing energy field knowledge can be determined by referring to the position and relationship of each word in the semantic space of the multimodal energy data.

[0040] Specifically, at the sentence level, a Transformer-based semantic similarity model, finely tuned using energy-related data, is employed to capture specialized semantic information within energy-related text sentences. For sentences containing new knowledge... Sentences in the knowledge graph that correspond to existing knowledge in the energy field This model accurately calculates semantic similarity scores. .

[0041] In addition, during the semantic recognition process, whole sentence semantic analysis, short sentence semantic analysis (breaking the whole sentence into several short sentences) and lexical semantic analysis can be used to identify the whole sentence semantics, short sentence semantics and lexical semantics respectively, and the semantic content of the three can be combined to determine the specific content of the whole sentence semantics. That is, the whole sentence semantics is the main focus, and the short sentences and lexicals are the auxiliary factors. The results of the whole sentence semantic analysis are adjusted based on the results of the semantic analysis of short sentences and lexicals.

[0042] When breaking down a complete sentence into short sentences, the long sentence can be broken down into the smallest unit of subject, verb, and object according to the subject-verb-object structure. The content of the attributive and adverbial clauses can be divided into other short sentence content. Alternatively, the short sentences can be broken down into several short sentences consisting of three to five characters according to the integrity of the vocabulary.

[0043] Specifically, at the knowledge graph structure level, we conduct in-depth analysis of the topological relationships of new knowledge within the existing knowledge graph of the energy field (through the aforementioned lexical and semantic analysis, we can determine the degree of similarity between new and existing knowledge, and preliminarily identify the possible topological relationships between them), including information such as node positions, edge types, and weights, and calculate structural similarity scores. The final knowledge similarity score S is obtained.

[0044] ; Among them, α, β, and γ are weighting coefficients, which are dynamically adjusted according to the specific type of knowledge (such as energy technology knowledge, energy market knowledge, etc.) and the actual application scenario (such as short-term forecasting, long-term planning, etc.).

[0045] Understandably, after the above calculation process to obtain the knowledge similarity score S, it is necessary to use the knowledge similarity score S to determine the relationship between the new knowledge and existing knowledge in the energy field.

[0046] Specifically, a threshold can be preset to judge the degree of similarity and association. The higher the threshold, the more accurate the screening results. After calculating the knowledge similarity score, if the knowledge similarity score is higher than the threshold, it is determined that the new knowledge is related to the existing knowledge. The higher the knowledge similarity score, the higher the degree of association.

[0047] The association degree is set as an attribute value within the range of [0, 1]. When the association degree is zero, it is a state of no relationship, that is, the knowledge similarity score is less than or equal to the threshold. When the association degree is 1, it is the same content. A conversion method is defined based on the proportion of the knowledge similarity score that exceeds the threshold, and an extreme point is set. When the knowledge similarity score exceeds the threshold by twice, the association degree value is automatically set to 1. When the knowledge similarity score exceeds the threshold by one time but less than two times, it is converted to a value in [0, 1] based on the percentage of the excess.

[0048] Understandably, the higher the degree of correlation, if it is close to 1, the more similar the new knowledge and the existing knowledge are. Conversely, the lower the degree of correlation, the more similar the new knowledge and the existing knowledge are.

[0049] S20, based on the correlation and the data characteristics of the multimodal energy data, update the classification rules of the existing energy field knowledge; Understandably, after determining the correlation, the degree of similarity between new knowledge and existing knowledge can be determined. However, there are still some contents in the new knowledge that have low or moderate similarity to existing knowledge. If the new knowledge is directly included into the knowledge graph corresponding to the existing energy field knowledge based solely on the correlation, the accuracy of the data classification results will be poor. Therefore, in this embodiment, after determining the correlation, it is also necessary to update the classification rules of the existing energy field knowledge in combination with the data characteristics of multimodal energy data, so as to ensure that all knowledge in the new knowledge can be classified into the knowledge graph corresponding to the existing energy field knowledge.

[0050] The data characteristics of multimodal energy data mainly include the importance of energy data, frequency of change, impact on decision-making, source reliability and timeliness, etc., which mainly cover the characteristics of energy demand restricted by different application environments. For example, the power supply of equipment that needs to be kept on is of high importance, the timeliness requirements of energy change data required by operation and maintenance personnel are high, and the requirements of high source reliability and accurate change frequency of energy data are required when dispatching grid load.

[0051] In this embodiment, the step of updating the classification rules of existing energy domain knowledge based on the correlation and the data characteristics of the multimodal energy data includes: The influence coefficient is calculated based on the data characteristics of the multimodal energy data and the weighting coefficients that are dynamically adjusted based on the requirements of the preset application scenarios. The data characteristics include at least importance, frequency of change, decision impact, source credibility and timeliness. Based on the aforementioned impact coefficient, the classification rules for existing energy-related knowledge are updated for the first time. Based on a preset smoothing factor, the classification rules after the initial update are smoothed multiple times until a preset stopping condition is met, at which point the smoothing process is terminated, resulting in the updated classification rules.

[0052] Understandably, in this embodiment, it is necessary to update the classification rules of existing energy knowledge by combining the correlation and data characteristics, so that the classification rules of existing energy knowledge can adapt to the characteristics and content of new energy knowledge, and achieve accurate classification of new energy knowledge.

[0053] Specifically, in this embodiment, the influence coefficient A is determined based on the correlation between new energy knowledge and existing energy field knowledge, taking into account various characteristics of energy data, including the importance of the energy data itself. (Importance refers to the criticality of the data in the data application scenario, such as using the data to build models, assess risks, and perform predictive analysis, etc.) Frequency of change (This frequency of change refers to a precisely quantified update cycle, such as real-time, hourly, daily, and monthly, as well as the degree of impact on decision-making.) (The degree of influence on decision-making refers to the assessment indicators used to determine whether the data is a key basis for decision-making.) The credibility of the data source. (The credibility of the data source refers to the rating or score after reviewing the data provider's authority, historical records, transparency of collection methods, etc.) and the timeliness of the data. (This timeliness refers to data latency, that is, the delay from data generation to its availability. For example, for high-frequency trading or real-time power grid dispatching, "second-level" timeliness is a hard requirement; for long-term strategic planning, "quarterly" or "annual" data may also have timeliness.) The formula for calculating this influence coefficient is as follows: ; in, The weighting coefficients for each factor can be dynamically adjusted based on the actual characteristics of energy data (such as the differences in characteristics of different types of energy data) and the needs of application scenarios (such as the different emphases of real-time monitoring and post-event analysis).

[0054] It should also be noted that, since some data in multimodal energy data are used in multiple scenarios (for example, data used in power grid analysis may also account for a high proportion of applications in operation and maintenance monitoring analysis), multiple sets of weights can be configured for certain specific data to calculate multiple sets of influence coefficients. These multiple influence coefficients can then be combined to adjust the existing classification results of energy domain knowledge. In other words, the tendency of multimodal energy data to be used in multiple domains and the correlation between multimodal energy data and multiple domains are included in the step of updating classification rules to ensure accurate classification of the complex information involved in multimodal energy data.

[0055] Understandably, after calculating the influence coefficient, one can then proceed according to... Update the classification rule r.

[0056] in, It is the adjustment amount based on the rules determined by new knowledge, where r is the classification rule corresponding to existing knowledge in the energy field. This refers to the classification rules updated initially using the influence coefficients.

[0057] Furthermore, to ensure the stability and consistency of rule updates, a multi-round smoothing process is also adopted in this embodiment.

[0058] Specifically, in the first round, the first smoothing factor is used. The classification rules for the first round of smoothing The rules are obtained after initial smoothing; In the second round, based on the results of the first round, a second smoothing factor with different parameter values ​​is used. That is, the classification rules for the second round of smoothing. The process is then repeated N times in the manner described above, until the preset number of smoothing rounds is reached. At this point, the smoothing process stops, and a stable update rule is obtained. .

[0059] The first and second smoothing factors mentioned above refer to weight parameters used to balance the new and old classification rules, representing the respective proportions of the new and old classification rules in the classification rule update process, thereby reducing abrupt changes during classification rule updates. The smoothing factor is a value between 0 and 1.

[0060] The first smoothing factor is greater than the second smoothing factor, and during the N rounds of smoothing, the smoothing factor used in the corresponding round should be adaptively reduced as the number of rounds increases.

[0061] The larger the smoothing factor used in the smoothing process, the more trust is placed in the new classification rule during the classification rule optimization process.

[0062] In this embodiment, the step of initially updating the classification rules of existing energy field knowledge based on the influence coefficient and the correlation includes: Based on the aforementioned relationships, extract a knowledge subgraph from the existing energy domain knowledge that is related to the multimodal energy data; Based on the influence coefficient, the knowledge subgraph is updated according to the tendency of different preset application scenario requirements, and a higher-level subgraph and a lower-level subgraph are generated based on the multiple knowledge subgraphs obtained from the update. The higher-level subgraph is a subgraph that covers the common knowledge of the multiple knowledge subgraphs, and the lower-level subgraph is a subgraph obtained by fusing the knowledge subgraphs in the multiple knowledge subgraphs whose relevance is higher than a preset relevance. Based on the knowledge classification tendencies corresponding to the multiple knowledge subgraphs obtained from the update, the upper-level subgraph, and the lower-level subgraph, the classification rules for the existing energy field knowledge are updated for the first time.

[0063] Understandably, based on the association relationship, the correspondence between the current multimodal energy data and the knowledge graph corresponding to existing energy field knowledge can be determined. Based on this association relationship, knowledge branches or knowledge subgraphs that can classify multimodal energy knowledge can be roughly selected from the knowledge graph. However, in this application, when considering the classification of multimodal energy data, the data characteristics of various types of data and the cross-modal association information between multimodal energy data are also considered. Therefore, relying solely on the knowledge graph of existing energy field knowledge cannot fully meet the expected requirements when classifying multimodal energy knowledge.

[0064] In this example, the existing knowledge graph's classification rules for data classification will be comprehensively adjusted based on the correlation and the influence coefficient calculated based on the data characteristics. This will enable dynamic adjustment to adapt to the correlation between multimodal energy data across different modalities and reflect the different tendencies of data characteristics.

[0065] Specifically, in this embodiment, a knowledge subgraph related to multimodal energy data is first extracted from existing energy domain knowledge based on the association relationships. This knowledge subgraph is then modified and updated. Based on the updated knowledge subgraph, more complex upper-level and lower-level subgraphs (used to describe multi-dimensional, multi-data characteristics and data display tendencies in different application scenarios) are generated. Based on the data classification tendencies of the updated knowledge subgraph, upper-level subgraph, and lower-level subgraph, the classification rules of existing energy domain knowledge are initially updated. This achieves the construction of a new knowledge graph that combines existing energy domain knowledge with a knowledge graph that represents more complex data relationships, thereby achieving dynamic classification of multimodal energy data and ensuring that new multimodal energy knowledge can be more accurately classified into the corresponding knowledge graph.

[0066] The influence coefficient is calculated by dynamic weighting of data characteristics and different preset application scenario requirements. The influence coefficient mainly represents the importance of data characteristics in different application scenarios. For example, there are obvious differences in data requirements in situations with high timeliness requirements and situations with high-quality data requirements.

[0067] Therefore, when updating the knowledge subgraph according to the influence coefficient and the tendency of different preset application scenario requirements, the information of data characteristics is also modified in the knowledge subgraph, thereby increasing the comprehensive description and classification ability of the knowledge subgraph for complex multi-source data.

[0068] It should be noted that, since the updated knowledge subgraph covers cross-modal information between complex data, it is difficult to directly integrate the updated knowledge subgraph into the existing knowledge graph of the energy field. It is necessary to further generate upper-level and lower-level subgraphs, and combine the data classification of the three levels to reasonably update the classification rules of the knowledge graph.

[0069] The upper-level subgraph is a subgraph that covers the common knowledge of the multiple knowledge subgraphs, which is equivalent to the backbone of the knowledge graph. It divides the data into major categories through key information, and includes data with different application scenarios in multimodal energy data. At the same time, by generating upper-level subgraphs, complex data information can be summarized so that the upper-level subgraphs and the existing knowledge graphs in the energy field can be linked.

[0070] The lower-level subgraph is a subgraph obtained by fusing knowledge subgraphs with a correlation higher than a preset correlation level from the multiple knowledge subgraphs. That is, merging two or more knowledge subgraphs with a certain correlation level to obtain a new lower-level subgraph, thereby increasing the descriptive effect of the updated knowledge subgraph by covering the content of the two or more updated knowledge subgraphs.

[0071] Specifically, the knowledge subgraph with a certain degree of relevance refers to a knowledge subgraph with a relevance higher than a preset relevance level. The relevance level of this knowledge subgraph is calculated by determining three levels: the knowledge structure of the knowledge subgraph, the data characteristic tendency within the knowledge subgraph, and the content of the corresponding data within the knowledge subgraph. The similarity between multiple knowledge subgraphs is calculated based on the content of these three levels, and it is ensured that when there is a knowledge subgraph with a relevance higher than the preset relevance level, the corresponding knowledge subgraphs are merged to obtain a new knowledge subgraph.

[0072] In addition, in this embodiment, new descriptive lower-level subgraphs are added. By identifying each updated knowledge subgraph, the connection between every two knowledge subgraphs is determined, thereby enhancing the knowledge structure between the updated knowledge subgraphs to be more complex and stable.

[0073] S30, the multimodal energy data is classified according to the comprehensive feature vector and the updated classification rules to obtain the classification result.

[0074] Understandably, after obtaining the updated classification rules, the comprehensive feature vector can be incorporated into the knowledge graph corresponding to the existing energy field knowledge, and the corresponding processing results can be obtained.

[0075] This embodiment extracts feature vectors from multimodal energy data, merges these feature vectors into a comprehensive feature vector, and determines the correlation between the multimodal energy data and existing energy domain knowledge. Based on this correlation and the data characteristics of the multimodal energy data, the classification rules of the existing energy domain knowledge are updated. The multimodal energy data is then classified according to the comprehensive feature vector and the updated classification rules to obtain the classification results. This approach involves first extracting feature vectors from the multimodal energy data, then integrating these feature vectors to generate a comprehensive feature vector. Simultaneously, the correlation between the multimodal energy data and existing energy domain knowledge is determined. Based on this correlation and the data characteristics of the multimodal data, the classification rules of the existing energy domain knowledge are optimized and updated to ensure accurate classification of the multimodal energy data. Furthermore, the fused comprehensive feature vector is directly used for classification to preserve the rich information of various data types within the multimodal energy data, thereby guaranteeing the accuracy of the classification results.

[0076] like Figure 2 As shown, based on the first embodiment, a second embodiment of the multimodal data classification method of this application is proposed. In this embodiment, the method further includes: The feature vectors include structured data feature vectors, semi-structured data feature vectors, and unstructured data feature vectors; Understandably, multimodal energy data includes structured data, semi-structured data, and unstructured data. Therefore, different types of feature vectors can be extracted for different data structures, including structured data feature vectors, semi-structured data feature vectors, and unstructured data feature vectors.

[0077] S110, when the multimodal energy data is structured data, determine the key attribute columns in the structured data, normalize the key attribute columns, and generate the structured data feature vector based on the processed result. Understandably, when multimodal energy data is structured data, due to the highly clear structured characteristics of structured data (taking structured energy data such as financial statements and energy consumption of energy companies as examples), the corresponding key information can be directly extracted from the data tables.

[0078] Therefore, when extracting feature vectors from structured data, a relational database model can be constructed to filter numerical key attribute columns (e.g., energy consumption values, financial expenditure values, etc.) relevant to energy data classification. Let the set of numerical attributes be... For each numeric attribute The range of its values ​​is mapped to the interval [0,1] using a normalization method: ; in, and Each is an attribute The minimum and maximum values ​​of these values ​​are used to form the final structured data feature vector. .

[0079] S120, when the multimodal energy data is semi-structured data, the key information of the semi-structured data is identified by regular expressions, and the feature vector of the semi-structured data is generated according to the first word frequency feature of each word in the key information. Understandably, when multimodal energy data is semi-structured, due to the slightly poor structured characteristics of semi-structured data, it is necessary to use specific methods to decompose and analyze the information in the semi-structured data and extract the meaning and corresponding characteristics of the data contained therein.

[0080] Therefore, when extracting feature vectors from semi-structured data, for semi-structured energy data such as energy equipment maintenance record documents and XML format configuration files, regular expressions are used to identify key information, and this information is divided into words. The first word frequency feature of each word is then calculated to construct word frequency vectors.

[0081] In this context, it is assumed that all the words identified from the semi-structured data are as follows: The first term frequency features corresponding to them are respectively set as The resulting semi-structured data feature vector is: .

[0082] Among them, identifying key information in semi-structured data through regular expressions mainly refers to extracting target fields or content (key information) from text that has certain formatting rules but is not fully structured by defining specific text patterns (regular expressions).

[0083] For example, the regular expression in this embodiment can be set to the following behavior: One method is to locate text segments that conform to specific patterns (such as date formats \d{4}-\d{2}-\d{2}). Secondly, use parentheses () to extract key parts (such as separating timestamps and error messages from logs); Thirdly, it handles different variations of the same field (such as phone numbers that can be matched as (123)456-7890 or 123-456-7890).

[0084] S130, when the multimodal energy data is unstructured data, if the unstructured data is text data, then a text feature vector is generated based on the second word frequency feature of each word in the text data; if the unstructured data is image data, then a geometric feature vector is generated based on the geometric features in the image data, and the unstructured data feature vector is generated based on the text feature vector and the geometric feature vector.

[0085] Understandably, when multimodal energy data is unstructured, the specific content of the unstructured data needs to be considered. It may be paper-based text data or electronic / paper-based image and video data. The extraction methods for these two types of data are different. Therefore, when constructing the feature vector of unstructured data, it is necessary to further determine the data type of the unstructured data. After determining its type, features are extracted using different methods, and corresponding feature vectors are formed. The two feature vectors are then combined into the final unstructured feature vector.

[0086] Specifically, preprocessing is performed on unstructured energy text data such as policy articles and descriptions of energy equipment in the energy sector.

[0087] Specifically, the bag-of-words model was used to statistically analyze the second word frequency features of the words in the text data. Let the distinct words in the text be... The corresponding second word frequency features (descriptions about energy use, energy records, etc. in the text data) are as follows: Thus, the text feature vector can be obtained. .

[0088] This involves segmenting unstructured graphics such as energy equipment, engineering drawings, and equipment operation videos, and extracting geometric features, such as the shape of components (circles, squares, etc., which can be described by geometric parameters) and dimensions. Let the extracted geometric feature set be... Geometric feature vectors can be formed. .

[0089] Furthermore, the text feature vectors and geometric feature vectors are concatenated in a certain order to form unstructured data feature vectors. .

[0090] In this embodiment, the step of fusing the feature vectors into a comprehensive feature vector includes: The information entropy of the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector with the corresponding target variable is calculated respectively, wherein the target variable is the data type label corresponding to each type of data in the multimodal energy data; Based on the information entropy, the correlation coefficients between the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector and their corresponding target variables are calculated respectively. Based on the correlation coefficient, the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector are fused to obtain a comprehensive feature vector.

[0091] It should be noted that the correlation measurement method based on information entropy is introduced. The role of information entropy is to evaluate the information content of variables by calculating the uncertainty of variables, and then analyze the correlation between variables. In this embodiment, the correlation coefficient between the feature vector of multimodal energy data and the target variable is calculated, and the correlation coefficient is used to quantify the correlation between the two. Based on the correlation, the relevant information of various feature vectors can be integrated on the feature vectors corresponding to different structural types of data to generate the corresponding comprehensive feature vector.

[0092] The target variable is the data type label corresponding to each type of data in the multimodal energy data. When calculating the feature vector corresponding to each multimodal energy data, the correlation information of its corresponding data type label can be included. That is, the corresponding information of the data type label is added to the feature vector, increasing the amount of information held by the feature vector, so as to facilitate more accurate classification.

[0093] Specifically, it is necessary to target the feature vectors of structured data. semi-structured data feature vectors unstructured data feature vectors Calculate the information entropy between the above three variables and the target variable Y. , and The correlation was calculated separately. (Correlation between structured data feature vectors and target variables) (Correlation between semi-structured data feature vectors and target variables) and (Correlation between feature vectors of unstructured data and target variables).

[0094] Among them, the method for calculating the information entropy between structured data feature vectors, semi-structured data feature vectors and unstructured data feature vectors and the corresponding target variable Y is a general calculation. Specifically, it calculates the joint distribution of various types of structured data feature vectors and target variables (i.e., the pattern in which they appear together) and their distribution if they are completely independent, and determines the degree of difference between the two. The specific calculation formula will not be repeated here, but can be referred to the existing information entropy calculation scheme.

[0095] ; ; ; based on , and The scores are weighted and fused from each modality to generate a comprehensive feature vector T. .

[0096] The multimodal energy data in this embodiment includes structured data, semi-structured data, and unstructured data. Therefore, different types of feature vectors can be extracted for data of different structure types, specifically including structured data feature vectors, semi-structured data feature vectors, and unstructured data feature vectors. The correlation between each feature vector and the target variable can be calculated based on the above feature vectors, and a comprehensive feature vector can be generated on this basis to ensure that the comprehensive feature vector retains the information on the correlation between the feature vectors of each structure type and the target variable, thereby ensuring the accuracy of subsequent classification of the comprehensive feature vector.

[0097] like Figure 3 As shown, based on the first embodiment, a third embodiment of the multimodal data classification method of this application is proposed. In this embodiment, the method further includes: S31, Adjust the parameters of the preset perceptron model according to the updated classification rules, wherein the preset perceptron model is a data classification model constructed based on the existing energy field knowledge, as well as the corresponding knowledge graph and data samples; S32, Based on the adjusted perceptron model, perform classification prediction on the comprehensive feature vector; S33, if the predicted value corresponding to the prediction result is less than or equal to the preset dynamic classification decision threshold, then the multimodal energy data corresponding to the comprehensive feature vector is classified according to the predicted value to obtain the classification result.

[0098] It should be noted that when classifying knowledge, the common approach is to build a perceptron model to identify different types of knowledge and classify them according to existing rules. However, in this embodiment, the classification process for multimodal energy data (as new knowledge) requires updating the existing classification rules based on the characteristics of the data and the corresponding vectors. Therefore, after the classification rules are updated, the perceptron model built based on the existing classification rules needs to be adjusted, including optimizing parameters, weights, and the number of nodes, so that the perceptron model can better adapt to the updated classification rules.

[0099] Specifically, the structure is adjusted to suit the characteristics of energy data. The number of neurons in the hidden layer and the connection weights are dynamically optimized using a genetic algorithm-based strategy. The fused feature vector is then input into the optimized multilayer perceptron model.

[0100] Specifically, let the fused comprehensive feature vector of the input layer be... , where d is the feature dimension.

[0101] After processing through multiple hidden layers, the original output y of the model is obtained.

[0102] Furthermore, after the perceptron model is adjusted, the comprehensive feature vector can be input and classified through the perceptron model. However, in order to ensure the accuracy of classification, a dynamic classification decision threshold is introduced in this embodiment. A classification threshold is set by the decision threshold to predict the classification of the comprehensive feature vector. If the predicted value corresponding to the prediction result is less than or equal to the dynamic classification threshold, the comprehensive feature vector is classified according to the predicted value.

[0103] Specifically, the predicted value refers to the variance of the model's predictions during classification decisions, taking into account the uncertainty of the model's predictions. This is achieved by performing M forward propagations on the input sample during the inference phase of the MLP (Multi-Layer Perceptron) model, randomly applying Dropout during each forward propagation, resulting in M ​​predicted outcomes. The predicted value is the variance of these predicted outcomes. Predicted outcomes with a variance greater than a preset dynamic classification decision threshold are marked as samples awaiting manual review, where energy experts, using an updated knowledge system, further assess and classify them. Predicted outcomes with a variance less than or equal to the threshold are classified based on a comparison between the predicted value and the dynamic classification decision threshold.

[0104] Specifically, let the misclassification cost of category c be... Category importance weight is Then the decision threshold for category c is: .

[0105] Among them, if ( If the predicted probability is that sample i belongs to category c, then sample i is classified as category c.

[0106] In addition, it should be noted that the classification in this embodiment includes data type classification and data level classification. Specifically, taking power grid energy data as an example, the overall macroscopic operation data of the entire power grid is regarded as first-level data, the operation data of any area in the power grid is regarded as second-level data, and the data of any electrical equipment in any area is regarded as third-level data, etc.

[0107] It should be noted that the classification results are mainly based on a comprehensive classification that takes into account the correlation information between various types of data in multimodal energy data, the similarity between multimodal energy data and existing knowledge, and the correlation information between various types of data and their corresponding data types, thereby ensuring the accuracy of the classification.

[0108] In this embodiment, after the step of classifying the multimodal energy data according to the integrated feature vector and the updated classification rule to obtain the classification result, the method further includes: The classification results are evaluated; If either the accuracy of the classification result or the consistency index of energy categories fails to meet the standard, the feature importance assessment method based on random forest is used to determine the impact of each feature dimension in the comprehensive feature vector on the classification result. Based on the aforementioned impact, adjust the weights corresponding to each feature in the comprehensive feature vector, and adjust the parameters of the perceptron model. Based on the adjusted perceptron model, the data is classified again, and the step of evaluating the classification results is returned until the results meet the expected criteria.

[0109] Understandably, when classifying data using the adjusted perceptron model, there may still be some classification errors or omissions. Therefore, after using the model for classification, it is necessary to supplement the evaluation and verification scheme, evaluate the classification results of multimodal energy data accordingly, and make corresponding adjustments based on the evaluation results.

[0110] Specifically, in the evaluation of classification results, the scheme of accuracy and energy category consistency index is mainly used to comprehensively evaluate the classification results.

[0111] ; ; in, The proportion of category c in the classification results. This represents the proportion of category c in the actual energy category distribution, measuring the consistency between the classification results and the actual energy category distribution.

[0112] Furthermore, if the classification results do not meet the corresponding standards (either accuracy or energy category consistency index fails to meet the standard), a feedback adjustment mechanism is initiated, and a feature dimension that has a significant impact on the classification results is determined using a feature importance assessment method based on random forest.

[0113] Specifically, assuming there are T decision trees in the random forest model, for feature j, calculate the reduction in its impurity in each decision tree. The importance score (impact) of feature j.

[0114] ; Based on importance score The fused features are adjusted, and features j with low importance but high noise (when...) are adjusted. , The new feature values ​​are weighted by a preset threshold. ( At the same time, features that are of high importance but lack complete information are supplemented or enhanced.

[0115] In addition, the parameters of the MLP model need to be readjusted, and the classification should be performed again after the parameters are adjusted until the results meet the expected criteria.

[0116] This embodiment adjusts the parameters of a preset perceptron model according to updated classification rules. The preset perceptron model is a data classification model constructed based on existing energy domain knowledge, corresponding knowledge graphs, and data samples. Based on the adjusted perceptron model, the comprehensive feature vector is classified and predicted. If the predicted value is less than or equal to a preset dynamic classification decision threshold, the multimodal energy data corresponding to the comprehensive feature vector is classified according to the predicted value, resulting in a classification result. Thus, by adjusting the perceptron model constructed based on existing classification rules, the adjusted perceptron model can classify multimodal energy data. By introducing a dynamic classification decision threshold, the classification result of the adjusted model is predicted, and it is determined whether the classification result meets the classification accuracy requirements, ensuring the accuracy of data classification.

[0117] In summary, the embodiments of this application mainly include: for structured numerical data, selecting key attributes and standardizing them to convert them into feature vectors in a unified format; for semi-structured data, generating quantifiable text feature vectors by parsing key information and counting word frequencies; for unstructured data, performing word frequency statistics on text and extracting features such as geometric shape and size from graphics, then sequentially concatenating the features of different parts to form a complete unstructured feature vector. Simultaneously, an information entropy-based method is used to measure the correlation between each modal data feature and the target variable, and a dynamic weight adjustment strategy is executed based on the calculated correlation score. The weights are adjusted in real time according to the contribution of each modal feature to the classification result, and then the modal features are weighted and fused to generate a comprehensive feature vector. This fully considers the multimodal dynamic characteristics of energy data, improves the quality of fused features and the adaptability of classification to data changes. Furthermore, knowledge graph technology is also used to analyze the energy field... The new knowledge is analyzed in depth, first being transformed into a knowledge graph. At the vocabulary, sentence, and knowledge graph structure levels, optimization algorithms and professional models are used to calculate its similarity to existing knowledge. Dynamically adjusted weight coefficients are used to determine the association between new and existing knowledge, laying a solid foundation for rule updates. Based on this association, an influence coefficient is calculated considering factors such as the importance of energy data, frequency of change, decision impact, source credibility, and timeliness. This coefficient is used to update classification rules, and multiple rounds of smoothing ensure the stability and consistency of rule updates, enabling rules to adapt to the dynamic changes in energy data and maintain classification effectiveness. Furthermore, a multilayer perceptron model with a genetic algorithm-optimized structure is used to automatically classify fused features. Multiple evaluation indicators are used to comprehensively verify the classification results. Feature importance analysis is used to adjust features and rules, and reclassification is performed to meet the actual needs of energy data processing.

[0118] Furthermore, embodiments of this application also propose a multimodal data classification device, referring to... Figure 4 The multimodal data classification device includes: The data processing module 10 is used to extract feature vectors from multimodal energy data, merge the feature vectors into a comprehensive feature vector, and determine the correlation between the multimodal energy data and existing knowledge in the energy field. The rule update module 20 is used to update the classification rules of the existing energy field knowledge according to the correlation and the data characteristics of the multimodal energy data; The data classification module 30 is used to classify the multimodal energy data according to the comprehensive feature vector and the updated classification rules to obtain the classification result.

[0119] This embodiment extracts feature vectors from multimodal energy data, merges these feature vectors into a comprehensive feature vector, and determines the correlation between the multimodal energy data and existing energy domain knowledge. Based on this correlation and the data characteristics of the multimodal energy data, the classification rules of the existing energy domain knowledge are updated. The multimodal energy data is then classified according to the comprehensive feature vector and the updated classification rules to obtain the classification results. This approach involves first extracting feature vectors from the multimodal energy data, then integrating these feature vectors to generate a comprehensive feature vector. Simultaneously, the correlation between the multimodal energy data and existing energy domain knowledge is determined. Based on this correlation and the data characteristics of the multimodal data, the classification rules of the existing energy domain knowledge are optimized and updated to ensure accurate classification of the multimodal energy data. Furthermore, the fused comprehensive feature vector is directly used for classification to preserve the rich information of various data types within the multimodal energy data, thereby guaranteeing the accuracy of the classification results.

[0120] It should be noted that each module in the above-mentioned device can be used to implement each step in the above-mentioned method and achieve the corresponding technical effect. This embodiment will not elaborate further here.

[0121] Reference Figure 5 , Figure 5 This is a schematic diagram of the hardware operating environment of the device involved in the embodiments of this application.

[0122] like Figure 5As shown, the device may include: a processor 1001, such as a CPU; a communication bus 1002; a user interface 1003; a network interface 1004; and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0123] Those skilled in the art will understand that Figure 5 The structure shown does not constitute a limitation on the device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0124] like Figure 5 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a multimodal data classification program.

[0125] exist Figure 5 In the device shown, the network interface 1004 is mainly used for data communication with an external network; the user interface 1003 is mainly used for receiving user input commands; the device calls the multimodal data classification program stored in the memory 1005 through the processor 1001 and performs the following operations: The feature vectors of multimodal energy data are extracted and fused into a comprehensive feature vector, and the correlation between the multimodal energy data and existing knowledge in the energy field is determined. Based on the aforementioned correlations and the data characteristics of the multimodal energy data, the classification rules for the existing energy domain knowledge are updated; Based on the comprehensive feature vector and the updated classification rules, the multimodal energy data is classified to obtain the classification results.

[0126] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: Based on the knowledge graph corresponding to existing energy domain knowledge, determine the position and relationship of the words in the semantic space corresponding to the knowledge graph of the multimodal energy data, and determine the semantic similarity between the words and the words in the existing energy domain knowledge based on the position and the relationship. Identify the semantic information of the statements in the multimodal energy data, and calculate the semantic similarity score between the statements in the multimodal energy data and the statements in the knowledge graph based on the semantic information; The topological structure relationship of the multimodal energy data in the knowledge graph is determined, and the structural similarity score between the multimodal energy data and the knowledge graph is calculated based on the topological structure relationship. The knowledge similarity score is calculated based on the preset weights, the word sense similarity, the semantic similarity score, and the structural similarity score. Based on the knowledge similarity score, the association between the multimodal energy data and the existing energy field knowledge is determined.

[0127] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: The influence coefficient is calculated based on the data characteristics of the multimodal energy data and the characteristic weights that are dynamically adjusted based on the requirements of the preset application scenarios. The data characteristics include at least importance, frequency of change, decision impact, source credibility and timeliness. Based on the influence coefficient and the correlation, the classification rules of the existing energy field knowledge are updated for the first time; Based on a preset smoothing factor, the classification rules after the initial update are smoothed multiple times until a preset stopping condition is met, at which point the smoothing process is terminated, resulting in the updated classification rules.

[0128] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: Based on the aforementioned relationships, extract a knowledge subgraph from the existing energy domain knowledge that is related to the multimodal energy data; Based on the influence coefficient, the knowledge subgraph is updated according to the tendency of different preset application scenario requirements, and a higher-level subgraph and a lower-level subgraph are generated based on the multiple knowledge subgraphs obtained from the update. The higher-level subgraph is a subgraph that covers the common knowledge of the multiple knowledge subgraphs, and the lower-level subgraph is a subgraph obtained by fusing the knowledge subgraphs in the multiple knowledge subgraphs whose relevance is higher than a preset relevance. Based on the knowledge classification tendencies corresponding to the multiple knowledge subgraphs obtained from the update, the upper-level subgraph, and the lower-level subgraph, the classification rules for the existing energy field knowledge are updated for the first time.

[0129] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: The feature vectors include structured data feature vectors, semi-structured data feature vectors, and unstructured data feature vectors; When the multimodal energy data is structured data, the key attribute columns in the structured data are determined, the key attribute columns are normalized, and the structured data feature vector is generated based on the processing results. When the multimodal energy data is semi-structured data, regular expressions are used to identify key information of the semi-structured data, and a feature vector of the semi-structured data is generated based on the first word frequency feature of each word in the key information. When the multimodal energy data is unstructured data, if the unstructured data is text data, a text feature vector is generated based on the second word frequency feature of each word in the text data; if the unstructured data is image data, a geometric feature vector is generated based on the geometric features in the image data, and the unstructured data feature vector is generated based on the text feature vector and the geometric feature vector.

[0130] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: The information entropy of the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector with the corresponding target variable is calculated respectively, wherein the target variable is the data type label corresponding to each type of data in the multimodal energy data; Based on the information entropy, the correlation coefficients between the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector and their corresponding target variables are calculated respectively. Based on the correlation coefficient, the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector are fused to obtain a comprehensive feature vector.

[0131] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: According to the updated classification rules, the parameters of the preset perceptron model are adjusted. The preset perceptron model is a data classification model constructed based on the existing knowledge in the energy field, as well as the corresponding knowledge graph and data samples. Based on the adjusted perceptron model, the comprehensive feature vector is classified and predicted; If the predicted value corresponding to the prediction result is less than or equal to the preset dynamic classification decision threshold, then the multimodal energy data corresponding to the comprehensive feature vector is classified according to the predicted value to obtain the classification result.

[0132] Furthermore, the processor 1001 can call the multimodal data classification program stored in the memory 1005 and also perform the following operations: The classification results are evaluated; If either the accuracy of the classification result or the consistency index of energy categories fails to meet the standard, the feature importance assessment method based on random forest is used to determine the impact of each feature dimension in the comprehensive feature vector on the classification result. Based on the aforementioned impact, adjust the weights corresponding to each feature in the comprehensive feature vector, and adjust the parameters of the perceptron model. Based on the adjusted perceptron model, the data is classified again, and the step of evaluating the classification results is returned until the results meet the expected criteria.

[0133] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0134] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0135] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the multimodal data classification method described in the above embodiments.

[0136] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0137] The aforementioned computer-readable storage medium may be included in the multimodal data classification device; or it may exist independently and not assembled into the multimodal data classification device.

[0138] The aforementioned computer-readable storage medium carries one or more programs that, when executed by the multimodal data classification device, cause the multimodal data classification device to: The feature vectors of multimodal energy data are extracted and fused into a comprehensive feature vector, and the correlation between the multimodal energy data and existing knowledge in the energy field is determined. Based on the aforementioned correlations and the data characteristics of the multimodal energy data, the classification rules for the existing energy domain knowledge are updated; Based on the comprehensive feature vector and the updated classification rules, the multimodal energy data is classified to obtain the classification results.

[0139] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0140] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0141] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0142] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described multimodal data classification method, and is capable of solving the technical problem of multimodal data classification. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the multimodal data classification method provided in the above embodiments, and will not be repeated here.

[0143] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

[0144] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0145] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0146] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0147] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A multimodal data classification method, characterized in that, The multimodal data classification method includes the following steps: The feature vectors of multimodal energy data are extracted and fused into a comprehensive feature vector, and the correlation between the multimodal energy data and existing knowledge in the energy field is determined. The classification rules for existing energy domain knowledge are updated based on the aforementioned correlation and the data characteristics of the multimodal energy data. This includes: calculating an influence coefficient based on the data characteristics of the multimodal energy data and dynamically adjusted characteristic weights based on preset application scenario requirements, wherein the data characteristics include at least importance, frequency of change, decision impact, source credibility, and timeliness; performing an initial update to the classification rules for existing energy domain knowledge based on the influence coefficient and the correlation; and performing multiple rounds of smoothing processing on the initially updated classification rules based on a preset smoothing factor until a preset stopping condition is met, resulting in updated classification rules. The initial update of the classification rules includes: extracting knowledge subgraphs from the existing energy domain knowledge that are related to the multimodal energy data based on the association relationship; updating the knowledge subgraphs according to the influence coefficient and the tendency corresponding to different preset application scenario requirements, and generating a higher-level subgraph and a lower-level subgraph based on the updated knowledge subgraphs, wherein the higher-level subgraph is a subgraph that covers the common knowledge of the multiple knowledge subgraphs, and the lower-level subgraph is a subgraph obtained by fusing the knowledge subgraphs with a correlation degree higher than a preset correlation degree; and updating the classification rules of the existing energy domain knowledge for the first time based on the knowledge classification tendency corresponding to the updated knowledge subgraphs, the higher-level subgraph, and the lower-level subgraph. Based on the comprehensive feature vector and the updated classification rules, the multimodal energy data is classified to obtain the classification results.

2. The method as described in claim 1, characterized in that, The step of determining the correlation between the multimodal energy data and existing energy field knowledge includes: Based on the knowledge graph corresponding to existing energy domain knowledge, determine the position and relationship of the words in the semantic space corresponding to the knowledge graph of the multimodal energy data, and determine the semantic similarity between the words and the words in the existing energy domain knowledge based on the position and the relationship. Identify the semantic information of the statements in the multimodal energy data, and calculate the semantic similarity score between the statements in the multimodal energy data and the statements in the knowledge graph based on the semantic information; The topological structure relationship of the multimodal energy data in the knowledge graph is determined, and the structural similarity score between the multimodal energy data and the knowledge graph is calculated based on the topological structure relationship. The knowledge similarity score is calculated based on the preset weights, the word sense similarity, the semantic similarity score, and the structural similarity score. Based on the knowledge similarity score, the association between the multimodal energy data and the existing energy field knowledge is determined.

3. The method as described in claim 1, characterized in that, The step of extracting feature vectors from multimodal energy data includes: The feature vectors include structured data feature vectors, semi-structured data feature vectors, and unstructured data feature vectors; When the multimodal energy data is structured data, the key attribute columns in the structured data are determined, the key attribute columns are normalized, and the structured data feature vector is generated based on the processing results. When the multimodal energy data is semi-structured data, regular expressions are used to identify key information of the semi-structured data, and a feature vector of the semi-structured data is generated based on the first word frequency feature of each word in the key information. When the multimodal energy data is unstructured data, if the unstructured data is text data, a text feature vector is generated based on the second word frequency feature of each word in the text data; if the unstructured data is image data, a geometric feature vector is generated based on the geometric features in the image data, and the unstructured data feature vector is generated based on the text feature vector and the geometric feature vector.

4. The method as described in claim 3, characterized in that, The step of fusing the feature vectors into a comprehensive feature vector includes: The information entropy of the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector with the corresponding target variable is calculated respectively, wherein the target variable is the data type label corresponding to each type of data in the multimodal energy data; Based on the information entropy, the correlation coefficients between the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector and their corresponding target variables are calculated respectively. Based on the correlation coefficient, the structured data feature vector, the semi-structured data feature vector, and the unstructured data feature vector are fused to obtain a comprehensive feature vector.

5. The method as described in claim 1, characterized in that, The step of classifying the multimodal energy data according to the integrated feature vector and the updated classification rules to obtain the classification result includes: According to the updated classification rules, the parameters of the preset perceptron model are adjusted. The preset perceptron model is a data classification model constructed based on the existing knowledge in the energy field, as well as the corresponding knowledge graph and data samples. Based on the adjusted perceptron model, the comprehensive feature vector is classified and predicted; If the predicted value corresponding to the prediction result is less than or equal to the preset dynamic classification decision threshold, then the multimodal energy data corresponding to the comprehensive feature vector is classified according to the predicted value to obtain the classification result.

6. The method as described in claim 5, characterized in that, After the step of classifying the multimodal energy data according to the integrated feature vector and the updated classification rule to obtain the classification result, the method further includes: The classification results are evaluated; If either the accuracy of the classification result or the consistency index of energy categories fails to meet the standard, the feature importance assessment method based on random forest is used to determine the impact of each feature dimension in the comprehensive feature vector on the classification result. Based on the aforementioned impact, adjust the weights corresponding to each feature in the comprehensive feature vector, and adjust the parameters of the perceptron model. Based on the adjusted perceptron model, the data is classified again, and the step of evaluating the classification results is returned until the results meet the expected criteria.

7. A multimodal data classification device, characterized in that, The multimodal data classification device includes: a memory, a processor, and a multimodal data classification program stored in the memory and executable on the processor, the multimodal data classification program being configured to implement the steps of the multimodal data classification method as described in any one of claims 1 to 6.

8. A storage medium, characterized in that, The storage medium stores a program that implements a multimodal data classification method, and the program that implements the multimodal data classification method is executed by a processor to implement the steps of the multimodal data classification method as described in any one of claims 1 to 6.

Images