Power market information extraction and pushing method and system based on multi-modal semantic fusion

Abstract

The invention relates to an electricity market information extracting and pushing method and system based on multi-modal semantic fusion. The method comprises the steps that an electricity market field pre-training corpus is constructed; adopting a staged pre-training strategy; performing multi-modal coding on the input power market information data; a text confidence score and an image confidence score are generated, a cross-modal attention incidence matrix is established, and image-text fine-grained interaction features are captured and fused to generate unified semantic representation; a user model containing long-term stable preferences and short-term dynamic interests is established based on unified semantic representation generated through fusion, a sequence decreasing exploration strategy is adopted, high-correlation intelligence is pushed through known preferences, new domain content is randomly inserted to mine potential interests, and the weight of pushed content is adjusted in real time according to user feedback. The problems of heterogeneous sources, diversified formats, different standards, asymmetric information and the like of power market information data can be effectively solved, and accurate and real-time power market information pushing service is provided for users.

Description

Technical Field

[0001] The present invention relates to the field of power market information technology, and in particular to a method and system for extracting and pushing power market information based on multimodal semantic fusion. Background Art

[0002] With the rapid development of smart grids and energy internet, massive multimodal data resources have been generated in the power market, including professional literature, equipment operation logs, grid topology maps, equipment monitoring images, etc.

[0003] In existing technologies, information extraction systems based on natural language processing mainly use pre-trained models in general fields for knowledge mining, but they face three significant drawbacks: First, models trained on general corpus lack the support of specialized vocabulary specific to the power market. For example, there are deviations in the semantic understanding of specialized terms such as "converter valve lockout" and "transient overvoltage," resulting in a decrease in the accuracy of extracting key information. Second, traditional multimodal processing methods for analyzing the association of graphic and text data remain at the shallow feature matching stage, failing to effectively capture the deep semantic associations between device schematics and operating parameters, and the granularity of cross-modal feature fusion is insufficient. Third, existing push systems mostly use static user portrait models, and the rigid push strategies lead to insufficient exploration of technical intelligence in new fields, and lack a dynamic adjustment mechanism based on real-time user feedback; especially when faced with application scenarios where long-term stable technical requirements unique to the power market (such as relay protection parameter optimization) coexist with short-term dynamic research hotspots (such as new energy grid connection technology), existing systems find it difficult to achieve accurate personalized push. Summary of the Invention

[0004] To this end, the technical problem to be solved by the present invention is to overcome the problem that the information extraction system based on natural language processing in the existing technology is not applicable to the power market field, and to provide a power market information extraction and push method and system based on multimodal semantic fusion, which can effectively solve the problems of heterogeneous information data sources, diversified formats, different standards, and information asymmetry in the power market, and provide power market users with accurate and real-time power market information push services.

[0005] To solve the above technical problems, the present invention provides a method for extracting and pushing power market information based on multimodal semantic fusion, comprising: Build a pre-trained corpus in the power market field, including basic terminology documents, complex technical documents and multimodal data; Use common corpus to pre-train the basic model and learn common semantic representations; introduce basic terminology documents from the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; load complex technical literature and multimodal data, and use contrastive learning tasks to optimize the basic model's understanding of cross-modal relationships; The trained basic model is applied to actual power market information processing, performing multimodal encoding on the input power market information data, including text encoding and image encoding. The term matching and semantic consistency scores of text features are calculated to generate a text confidence score. The key area recognition rate and context relevance score of image features are calculated to generate an image confidence score. Based on the text confidence score and image confidence score, a cross-modal attention association matrix is established to capture the fine-grained interactive features of text and images and fuse them to generate a unified semantic representation. Based on the fusion of unified semantic representation, a user model with long-term stable preferences and short-term dynamic interests is established. A descending order exploration strategy is adopted to push highly relevant intelligence based on known preferences and randomly insert new domain content to explore potential interests. The weight of pushed content is adjusted in real time based on user feedback.

[0006] In this paper, the complexity of data in the pre-trained corpus of the power market field is annotated, and a complexity score is calculated for each data, including term density, sentence length, and logical nesting level, where: term density is defined as the proportion of professional vocabulary in the document, sentence length is determined in bytes, and logical nesting level is quantified by the depth of the dependency syntactic parse tree; The training phase is divided according to the calculated complexity score, and the training batches are dynamically adjusted from low to high data complexity. Low-complexity data is used for initial fine-tuning, and medium- and high-complexity data is used for cross-modal correlation optimization.

[0007] In the present invention, the basic terminology document of the power market is introduced, and the process of identifying the power market proprietary vocabulary includes: Dynamically mask proprietary vocabulary in the power market, with the masking ratio increasing with the training stage. The masking ratio is ≤10% in the initial stage and increases to 30% in the final stage. The model predicts masked terms based on the context, and the prediction results are checked for consistency with the domain knowledge graph; When the prediction result fully matches the domain knowledge graph, the masking ratio of the next training batch is automatically increased by 10% to 15% of the current ratio; When the prediction result partially matches the domain knowledge graph, maintain the current masking ratio and increase the masking frequency of the same type of terms until the prediction result fully matches the domain knowledge graph; When the prediction result does not match the domain knowledge graph at all, the masking ratio is reduced to the initial value and the prediction result is regenerated until the prediction result completely matches the domain knowledge graph or there is a partial match.

[0008] In the present invention, the process of calculating the term matching degree and semantic consistency score of the text features and generating the text confidence score includes: By comparing the terminology database, the coverage of the power market professional terms in the text is calculated to obtain the term matching degree; Extract local semantics from the text, analyze the global semantics of the text, compare the degree of correlation between local semantics and global semantics, measure the semantic coherence of the text context, and determine the semantic consistency score; The comprehensive text confidence score is the weighted sum of term matching and semantic consistency, and the weight is dynamically adjusted based on historical push feedback.

[0009] In the present invention, the process of calculating the key area recognition rate and context relevance score of the image features and generating the image confidence score includes: The visual saliency detection model is used to calculate the proportion of key device areas in the image and obtain the key area recognition rate. First, the global feature vector of the image is extracted, and then the deep feature vector of the text description corresponding to the image is obtained. The cosine similarity between the global feature vector of the image and the deep feature vector of the text description is calculated. The auxiliary information in the image background is detected and its logical relationship with the core components is analyzed. The relationship between regions is modeled through a graph neural network, and a context association graph is generated to obtain a context association score. The comprehensive image confidence score is the weighted sum of the key area recognition rate and the context relevance score, and the weight is dynamically adjusted based on historical push feedback.

[0010] In the present invention, the construction of the cross-modal attention association matrix includes: Text-to-image attention: Calculates the similarity between each text word and the image region and generates an association weight; Image-to-text attention: Generate visual heatmaps by predicting region importance and reversely associate text descriptions; Fuse bidirectional attention weights to generate fine-grained interaction features.

[0011] In the present invention, the long-term preference is obtained by analyzing historical interaction data; the short-term interest is obtained by analyzing the user's current session behavior characteristics.

[0012] In the present invention, the descending order exploration strategy includes: ; Where: ε(t) is the exploration probability function value that decreases with the number of pushes, ε0 is the initial exploration rate, λ is the decay rate coefficient of the exploration probability, and t represents the number of pushes; When the random number is less than ε(t), new domain content is randomly selected for push; when the random number is greater than or equal to ε(t), content is pushed according to user preferences.

[0013] In the present invention, a dynamic exclusion factor R is introduced into the descending order exploration strategy to calculate the semantic similarity between the content to be pushed and the historical pushed content. If the similarity exceeds the threshold, the dynamic exclusion factor R is used to reduce its priority. If the similarity does not exceed the threshold, the dynamic exclusion factor R is not introduced, and the content to be pushed is determined according to the original priority ranking. The dynamic exclusion factor R is dynamically enhanced with the number of negative feedbacks from users. The formula is: ; Where: N neg is the number of negative feedback, N total is the total number of feedbacks, α, β, and γ are adjustable parameters, α represents the basic inhibition strength, β represents the negative feedback sensitivity, γ represents the decay rate coefficient of the dynamic repulsion factor, and t represents the number of pushes.

[0014] To solve the above technical problems, the present invention also provides a power market information extraction and push system based on multimodal semantic fusion, comprising: A data preprocessing module is used to build a pre-trained corpus in the power market domain, which contains basic terminology documents, complex technical documents and multimodal data; The model training module includes: a general semantic understanding unit, which is used to pre-train the basic model using general corpus; a professional terminology recognition unit, which is used to introduce basic terminology documents of the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; and a cross-modal association unit, which is used to load complex technical documents and multimodal data and optimize the basic model's understanding of cross-modal associations through comparative learning tasks. The multimodal processing module includes: an encoding unit for performing text encoding and image encoding on the input power market information data; a confidence calculation unit for calculating the term matching degree and semantic consistency score of text features to generate a text confidence score, and calculating the key area recognition rate and context relevance score of image features to generate an image confidence score; a feature fusion unit for establishing a cross-modal attention association matrix based on the text confidence and image confidence scores, capturing the fine-grained interaction features between text and images, and fusing them to generate a unified semantic representation; The user modeling and push module includes: an interest modeling unit, which is used to establish a user model that includes long-term stable preferences and short-term dynamic interests based on the unified semantic representation generated by fusion; a strategy execution unit, which is used to adopt a descending order exploration strategy, combine known preferences to push highly relevant intelligence and randomly insert new domain content to explore potential interests; and a feedback adjustment unit, which is used to adjust the push content weight in real time according to user feedback.

[0015] The above technical solution of the present invention has the following advantages over the prior art: The method for extracting and pushing power market information based on multimodal semantic fusion, described in the present invention, first constructs a specialized corpus for the power market domain containing basic terminology, complex documents, and multimodal data. A phased pre-training strategy is then employed: Based on general semantic understanding, domain adaptability is enhanced through a power market-specific vocabulary recognition module. Furthermore, a contrastive learning task is used to optimize the model's understanding of cross-modal associations between equipment drawings and technical documents. Secondly, an innovative dual confidence evaluation mechanism was designed at the feature processing layer, which performs quantitative evaluation from four dimensions: term matching, semantic consistency, key area recognition rate, and context relevance, and realizes fine-grained interactive fusion of image and text features through a cross-modal attention association matrix.

[0016] Finally, a user model that integrates long-term stable preferences and short-term dynamic interests was constructed at the push strategy layer. A descending order exploration strategy was adopted to balance the relationship between the push of known preferences and the exploration of new areas, and the push weight was dynamically adjusted through a real-time feedback mechanism.

[0017] Combined with the above technical features, the beneficial effects brought about by this application are mainly reflected in the following three aspects: (1) In terms of domain adaptability, the power market-specific corpus and phased training mechanism improve the accuracy of information extraction, especially when processing technical documents such as equipment failure reports, and reduce the error rate of professional terminology recognition; (2) In terms of cross-modal processing, fine-grained feature fusion based on the attention mechanism improves the accuracy of image-text association analysis. In equipment maintenance cases, it can accurately associate abnormal areas in wiring diagrams with corresponding current waveform features. (3) In terms of intelligent push, the dynamic user model combined with the exploration strategy increases the reach of high-value information, reduces the amount of users actively searching for technical intelligence in new fields, and significantly improves the knowledge acquisition efficiency of power market technicians. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to make the content of the present invention more clearly understood, the present invention is further described in detail below based on specific embodiments of the present invention in conjunction with the accompanying drawings, wherein: Figure 1This is a flowchart of the steps of the method for extracting and pushing power market information based on multimodal semantic fusion of the present invention; Figure 2 It is a flowchart of the steps of quantitatively labeling data complexity and dynamically optimizing the training process of the present invention; Figure 3 This is a flowchart of the steps of introducing the basic terminology document of the power market to identify the proprietary vocabulary of the power market; Figure 4 is a flow chart of the steps of generating text confidence scores according to the present invention; Figure 5 is a flow chart of the steps of generating an image confidence score according to the present invention; Figure 6 is a flowchart of the steps of constructing a cross-modal attention correlation matrix of the present invention; Figure 7 This is a structural framework diagram of the power market information extraction and push system based on multimodal semantic fusion of the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described below with reference to the accompanying drawings and specific embodiments so that those skilled in the art can better understand the present invention and implement it. However, the embodiments are not intended to limit the present invention.

[0020] Example 1:

[0021] Reference Figure 1 As shown, the present invention discloses a method for extracting and pushing power market information based on multimodal semantic fusion. Aiming at the problem that the semantic understanding deviation of the general model in the power market field leads to inaccurate push, the technical improvement is achieved through the following steps: S10. Build a pre-trained corpus in the power market field, including basic terminology documents, complex technical literature and multimodal data.

[0022] In this embodiment, a pre-trained corpus dedicated to the power market field is first constructed, covering basic terminology documents (such as IEC standard glossary), complex technical documents (such as relay protection device research reports) and multimodal data (including equipment wiring diagrams, infrared thermal imaging images and supporting technical documents); the corpus adopts a hierarchical storage architecture, with basic terminology as the underlying knowledge anchor, complex documents providing context, and multimodal data establishing image-text association samples.

[0023] S20. Use general corpus to pre-train the basic model and learn general semantic representations; introduce basic terminology documents from the electricity market for domain adaptation training to enhance the basic model's ability to recognize electricity-specific vocabulary; load complex technical literature and multimodal data, and use contrastive learning tasks to optimize the basic model's understanding of cross-modal associations.

[0024] In this embodiment, a three-stage model training is implemented: General semantic foundation: Use general corpus to pre-train the basic model and acquire general language understanding capabilities; Professional vocabulary enhancement: Introducing basic terminology documents of the power market (such as the IEC standard glossary) for domain adaptation training and constructing domain word vector space; Cross-modal association optimization: Load complex technical literature (such as a research report on relay protection devices) and supporting drawings. Using a contrastive learning framework, the schematic description text and structural diagram of the same device are combined into positive sample pairs, and the pictures and texts of different devices are combined into negative sample pairs. The model is trained to capture the semantic association between the pictures and texts.

[0025] Among them: the basic model is a general language model in the existing technology, which is a deep language model pre-trained based on large-scale general text data (such as Wikipedia, news, books, web pages, etc.). This type of model usually belongs to a neural network with a Transformer architecture, which captures the contextual dependencies of the text through the self-attention mechanism. It can adopt an encoder model (suitable for text understanding tasks) or a decoder model (suitable for text generation tasks). It can learn general language rules, semantic representations and world knowledge from massive texts. This basic model is used as the starting point of the initial pre-training stage, and it already has general language understanding capabilities (such as grammar and common semantics).

[0026] Domain knowledge injection was then used to address the problem of general model misunderstanding of professional vocabulary. The pre-trained model was then applied to the task of disambiguating terminology in the power market, and its accuracy was improved through cross-modal retrieval. For example, when processing the description of "circuit breaker tripping coil failure," the model was able to accurately associate the electromagnet components in the coil structure diagram with the current parameter anomaly described in the text.

[0027] S30. Apply the trained basic model to actual power market information processing, and perform multimodal encoding on the input power market information data, including text encoding and image encoding; calculate the term matching degree and semantic consistency score of text features to generate a text confidence score; calculate the key area recognition rate and context relevance score of image features to generate an image confidence score; based on the text confidence score and image confidence score, establish a cross-modal attention association matrix to capture the fine-grained interactive features of text and images and fuse them to generate a unified semantic representation.

[0028] The model trained in step S20 is deployed to an actual power market information processing scenario, and the following operations are performed on the input data: parallel encoding processing and dual-channel confidence evaluation; Wherein: Parallel encoding processing includes: Text encoding: Using a dynamic word segmentation strategy, it prioritizes matching against the power market terminology dictionary to generate semantic vectors containing part-of-speech tags (e.g., the verb "excitation" and the noun "arrester"). Image coding: Use region proposal networks to locate key equipment components (such as transformer bushing contacts), extract visual features, and record spatial coordinate information.

[0029] The two-channel confidence assessment includes: Text confidence score: Calculates term dictionary matching (such as the exact match score of "neutral point offset") and contextual semantic consistency (using the attention mechanism to detect reasonable collocation of professional terms); Image confidence score: Based on the recognition rate of key areas (such as the completeness of the switchgear panel logo) and context relevance (such as the rationality of the spatial relationship between the insulator image and adjacent hardware).

[0030] Cross-modal feature interaction: The model fuses text and image confidence scores to construct an attention association matrix, capturing fine-grained interactive features between text and images and generating a unified semantic representation. For example, when "acetylene content exceeds the standard" appears in a transformer oil chromatographic analysis report, the model automatically enhances the association weight with the arc path area in the transformer's internal discharge schematic, achieving aligned fusion of fault features.

[0031] In step S30, noise data is filtered through a quantitative evaluation mechanism, which can improve the efficiency of filtering invalid information in the substation inspection report processing, and the feature fusion module improves the accuracy of matching graphic and text evidence in fault diagnosis.

[0032] S40. Generate a unified semantic representation based on fusion to establish a user model that includes long-term stable preferences and short-term dynamic interests. Adopt a descending order exploration strategy, use known preferences to push highly relevant intelligence and randomly insert new domain content to explore potential interests. Adjust the push content weight in real time based on user feedback.

[0033] Based on the unified semantic representation after fusion, a user model is constructed, user preferences are modeled, and user portraits are established, including: long-term stable preferences, capturing high-frequency search topics and short-term dynamic interests that have lasted for more than 6 months, and tracking emerging hotspots in the past two weeks; a hybrid push mechanism with descending order is designed, with an initial setting of 30% of the push volume for new field content, and the exploration ratio is reduced exponentially as the number of pushes increases, to ensure that users have both stable knowledge intake and no information overload; an online learning module is deployed, and when a user collects / ignores a piece of pushed information, the system adjusts the weight coefficient of the relevant technical topic accordingly and triggers an incremental update of the corpus.

[0034] The method for extracting and pushing electricity market information disclosed in the present invention adopts domain knowledge embedding, multimodal fine processing, and dynamic push strategy, which effectively solves the professional barriers, modal gaps and demand drift problems in electricity market information processing, and provides reliable technical support for the construction of smart grids.

[0035] Furthermore, combined with practical applications, the pre-training corpus construction and phased training strategy are further optimized. In order to achieve refined control of the training process through the dynamic scheduling mechanism of data complexity, the data complexity is quantitatively labeled and the training process is dynamically optimized based on the complexity score. Figure 2 As shown, the specific steps include: Multidimensional feature analysis is performed on each piece of data in the pre-trained corpus mentioned above (including text, images, and image-text pairs): a dictionary of power market terminology is used for precise matching, and term density is defined as the percentage of professional vocabulary to the total number of words; bytes are used as the unit of measurement rather than characters to avoid statistical bias in mixed Chinese and English scenarios; long and difficult sentences in technical documents (such as equipment operating specifications containing multiple conditional clauses) are segmented, and the single sentence length threshold is set to 512 bytes to accommodate model input limitations; a semantic tree is constructed based on a dependency syntax analysis tool, and the tree depth value is used to represent logical complexity. For example, the syntactic tree depth of "When the bus voltage exceeds 10% of the rated value and lasts for more than 2 seconds, the overvoltage protection device should be activated" is 5 levels, while the simple descriptive sentence "The circuit breaker is tripped" has a depth of only 2 levels.

[0036] Based on the above complexity scores, dynamic optimization of the training process is implemented. Low-complexity data (term density <15%, sentence length <256 bytes, nesting level ≤3) is used in the initial fine-tuning stage, such as simple contextual data such as equipment nameplate information and standardized operating procedures; medium-complexity data (term density 15%-30%, sentence length 256-384 bytes, nesting level 4-5) is used for transition training, such as causal descriptions in equipment maintenance records; high-complexity data (term density >30%, sentence length >384 bytes, nesting level ≥6) is used for cross-modal association optimization, such as academic papers containing multiple hypothesis inferences.

[0037] In the initial stage, low, medium and high complexity data are mixed in a ratio of 7:2:1 to ensure a smooth transition of the model; after training every 10 groups of data, the ratio is automatically adjusted based on the performance of the validation set, and the maximum proportion of high complexity data can be increased to 40%.

[0038] Specifically, refer to Figure 3 As shown in the figure, when introducing the basic terminology document of the electricity market, the process of identifying the electricity market specific vocabulary includes: Dynamic masking is performed on proprietary terms in the power market, with the masking ratio increasing with each training stage. In the initial stage, the masking ratio is ≤10%, mainly targeting high-frequency basic terms (such as "insulation resistance" and "phase-to-phase short circuit") to ensure that the model initially establishes a weak association between terms and context. The masking ratio is then linearly increased, with a focus on complex terms (such as "zero-sequence current protection" and "transient voltage stability"). A partial masking strategy (such as masking "zero sequence" and retaining "current protection") is adopted to force the model to understand the internal structure of the terms. In the final stage, the masking ratio is increased to 30%, and random interval masking (such as masking "excitation__second harmonic") is implemented for low-frequency professional terms (such as "excitation inrush second harmonic") to enhance the model's ability to recover from fragmented terms.

[0039] The model predicts masked terms based on the context, and performs consistency checks on the prediction results with the domain knowledge graph. Different verification logics can be used, such as full match verification, partial match verification, and conflict detection. Finally, different strategies are constructed based on the verification results: When the prediction result fully matches the domain knowledge graph, the masking ratio of the next training batch is automatically increased by 10% to 15% of the current ratio; When the prediction result partially matches the domain knowledge graph, the current masking ratio is maintained, but focused training is performed on this type of term, increasing the masking frequency of the same type of term, and repeating the masking 3-5 times in the same context until the prediction result fully matches the domain knowledge graph; When the prediction result does not match the domain knowledge graph at all, the masking ratio is reduced to the initial value to regenerate the prediction result, and the relationship network of the term is extracted from the knowledge graph to regenerate the enhanced training samples. The term definition sentences are learned first, and then gradually transitioned to complex application scenarios until the prediction result completely matches or partially matches the domain knowledge graph.

[0040] Specifically, in order to achieve cross-modal processing, it is necessary to quantify the generation of text and images so that they can be input into the cross-modal attention matrix. Therefore, it is necessary to calculate the term matching and semantic consistency scores of text features to generate text confidence scores, and calculate the key area recognition rate and context relevance scores of image features to generate image confidence scores.

[0041] Further, refer to Figure 4 As shown, in the text encoding stage, this embodiment uses a three-level evaluation system to generate text confidence scores: Terminology matching calculation: This method uses a terminology knowledge base in the power market (including 82,000 standard terms) to perform a word-by-word scan. This not only counts the frequency of term occurrences but also records the hierarchical relationships of terms (e.g., "differential protection" belongs to the subcategory of "relay protection") and distinguishes between core and auxiliary terms.

[0042] Semantic consistency analysis: Local semantic extraction uses a sliding window mechanism, using a fixed-length window to extract words of a fixed number of characters and generate a semantic vector for each window. Global semantics are modeled using a full-text attention mechanism, focusing on capturing the logical flow guided by transition words (such as "however" and "therefore"). When calculating relevance, the semantic evolution trajectory is tracked, and when inconsistent statements are detected, the semantic consistency score automatically decreases.

[0043] Dynamic weight adjustment: Initially, the weighting for term matching is set at 60% and for semantic consistency at 40%. This is adjusted in real time based on user feedback (e.g., annotation behavior on pushed content). If users frequently annotate terminology incorrectly, the weighting for term matching can be increased to as high as 75%. A feedback-weight transfer function is established to ensure that a single weight adjustment does not exceed 5% to prevent system oscillation.

[0044] Furthermore, in the image coding stage, referring to Figure 5 As shown, this embodiment uses dual feature evaluation to construct an image quality index and generate an image confidence score: Key area identification: Perform pixel-level segmentation on the image to identify the core components of the device. Correct misdetected areas through prior knowledge constraints and introduce spatial continuity assessment to avoid misidentifying noise points as key components.

[0045] Context association modeling: first extract the global feature vector of the image, then obtain the deep feature vector of the text description corresponding to the image, calculate the cosine similarity of the global feature vector of the image and the deep feature vector of the text description, detect the auxiliary information in the image background, analyze its logical relationship with the core components, and model the inter-region association through the graph neural network, construct a regional association graph, and obtain the context association degree. The graph neural network is a deep learning model that specializes in processing graph structure data, which can automatically learn the mutual influence between entities in complex relationships. In this embodiment, the application of the graph neural network can meet the needs of image confidence score evaluation.

[0046] Adaptive weighting: The initial weights are set to 70% for recognition rate and 30% for relevance. When the user interaction rate for image content decreases, the relevance weight increases by 2% each time.

[0047] In this embodiment, the multimodal processing framework is further improved through the text confidence score and the image confidence score. When the text confidence score and the image confidence score are less than the set value, the set value is the minimum score representing the correlation between text and image, and the data re-collection process is automatically triggered to intercept low-quality input. For modalities with high text confidence scores and image confidence scores, they can be further amplified in the attention matrix, and the user's interaction data on the pushed content is returned to the confidence model in real time, forming a continuous optimization cycle.

[0048] Specifically, in this embodiment, in order to generate a unified semantic representation of text and image, the text confidence score and image confidence score are used, referring to Figure 6 As shown in Figure 2, a dynamic interactive cross-modal attention correlation matrix is constructed, including: Text-to-image attention mapping: This method establishes a text-driven visual attention mechanism, performing similarity calculations between each text word vector and image region features. This method implements enhanced calculations for specialized power market terminology. When core power market terms are detected, the similarity score is doubled. A multi-head attention mechanism is used to parallelize the associations between different semantic subspaces. For example, one attention head specifically processes the spatial position of devices (e.g., matching "left casing" with the left quadrant of the image).

[0049] Image-to-text attention feedback: Establish a correction mechanism for visual information to text, obtain a visual heat map of regional importance, identify key decision-making areas in the image, introduce prior knowledge of power market equipment, set a basic attention threshold for standard components, and prevent the model from ignoring key areas with unclear visual features; calculate cross-attention between the feature vectors of high-heat image areas and all text words to find the most relevant text description.

[0050] Bidirectional attention weights are integrated to generate fine-grained interactive features: Hadamard product operations are performed on the text-to-image weight matrix and the image-to-text weight matrix to highlight high-confidence regions of bidirectional confirmation; unidirectional high-weight regions (only text or image attention) are downgraded, and the weight coefficient is multiplied by a reliability discount of 0.3-0.5; when the sum of the bidirectional weights is greater than 1.5, the feature enhancement channel is activated to amplify the key feature dimensions by 2-3 times, establish cross-modal memory units, store high-frequency co-occurrence patterns (such as the association pattern between "low oil level" and the oil level gauge image), and accelerate the processing of similar scenarios in the future.

[0051] In this embodiment, a user knowledge graph is constructed through diachronic behavior analysis, including long-term preferences and short-term interests.

[0052] The long-term preferences are obtained through historical interaction data analysis. The time window is set to 6-24 months of historical interaction data. This covers all-dimensional behaviors such as literature review (such as IEEE paper download records), equipment attention (such as transformer model collection), and expert consultation (such as questions about relay protection issues). In particular, the in-depth interaction behavior of users in pushing content is recorded, including detailed indicators such as average reading time and annotation density. Capture the evolution trend of interests, identify stable points of interest, build a knowledge graph specialized in the power market, map user behavior to the technical system, define a preference intensity index, in which recent behavior has a higher weight but does not exceed 50% of the total, to avoid the "recency effect" from distorting long-term judgments, identify 3-5 core technical directions, and form a preference portrait base.

[0053] The short-term interest is obtained by analyzing the user's current session behavior characteristics, monitoring the high-frequency terms and document jump paths in the current work session in real time, capturing the interaction intensity characteristics, including page dwell time and mouse track hotspots, and setting the half-life of short-term interest to 72 hours. The interest intensity automatically decays by a certain percentage every 24 hours to ensure that the system switches the tracking focus in time. When an interest shift is detected, a new interest tracking cycle is started.

[0054] In the process of pushing power market information, it is not desirable to push only the content that users are interested in. Therefore, the present invention also proposes a descending order exploration strategy to construct a dynamic attenuation exploration-utilization balance mechanism. The descending order exploration strategy is: ; Where: ε(t) is the exploration probability function value that decreases with the number of pushes, ε0 is the initial exploration rate, λ is the decay rate coefficient of the exploration probability, and t represents the number of pushes.

[0055] The initial exploration rate ε0 is set to 30%-50%, with a low value for technically conservative users and a high value for innovation-oriented users. The decay coefficient λ is set adaptively. When short-term interest activity is detected, the λ value is reduced to slow down the decay rate. The exploration probability is updated according to the number of pushes (not natural time), and ε(t) is recalculated after each push. When the user refuses to explore the content 10 times in a row, ε(t) is temporarily increased to (ε0+ε(t)) / 2 to avoid falling into the local optimum.

[0056] When the random number is less than ε(t), new domain content is randomly selected for push; when the random number is greater than or equal to ε(t), content is pushed according to user preferences.

[0057] The random number is an independent random variable uniformly distributed in the interval [0,1). It uses a cryptographically secure pseudo-random number generator and initializes the seed with a hardware entropy source to ensure unpredictability and statistical uniformity. Before each push, a new random number is generated and compared with ε(t). When the random number is less than ε(t), exploration is triggered, and content is randomly selected from the candidate pool of new domains. When the random number is greater than or equal to ε(t), content that closely matches the user's known preferences is prioritized.

[0058] Furthermore, the actual interaction depth of users with the explored content (such as reading completion rate and collection behavior) is recorded. When the success rate is >60% for five consecutive times, ε0 is increased by 5%. Through short-term interest analysis, the relevance of the explored content is dynamically verified. If it is detected that the user actively retrieves and pushes new domain terms, the λ value recalibration is triggered.

[0059] For sub-fields with rapid technological updates (such as the Internet of Things in the power market), a piecewise decay function is adopted: slow decay in the first 20 pushes and accelerated decay in the later stages.

[0060] Through the descending exploration strategy, the stability and innovation of the power market professional knowledge service can be balanced. While maintaining the stable push of core content, the push of new field information can be improved.

[0061] In actual applications, the content pushed using the above method is often repeated. Moreover, after accurately determining the user's preferences through multiple iterative optimizations, when pushing electricity market information in the same technical field, the repetition rate of text content is even higher. Although new field information is continuously pushed through the descending order exploration strategy, this information is not what the customers like, and no new push strategy can be formed, causing trouble for users.

[0062] To address this issue, we further optimized the descending order exploration strategy by introducing a dynamic exclusion factor, R, to calculate the semantic similarity between the content to be pushed and the content pushed previously. We then compared the similarity between the candidate push content and the user's most recent 30 pushes. For text, we compared the term overlap ratio, and for images, we compared the device structure features. We also set a basic similarity threshold, which can be dynamically adjusted based on the user's interest model: lowering it for specialized, in-depth users and raising it for knowledge-rich users. If the similarity exceeds the threshold, the priority is lowered by the dynamic exclusion factor R. If the similarity does not exceed the threshold, the dynamic exclusion factor R is not introduced, and the content to be pushed is determined according to the original priority order. The dynamic exclusion factor R is dynamically enhanced with the number of negative feedback from users. The formula is: ; Where: N neg is the number of negative feedback, N total is the total number of feedbacks, α, β, γ are adjustable parameters, and t represents the number of pushes.

[0063] Specifically, explicit negative feedback (such as clicking "not interested") and implicit negative feedback (such as content retention time <15 seconds) are recorded, and the number of negative feedback is updated in real time. α represents the basic inhibition strength, β represents the negative feedback sensitivity, and γ represents the attenuation rate coefficient of the dynamic rejection factor. It is dynamically adjusted through online learning. For example: when it is detected that a user has rejected similar content for more than three consecutive times, the β value is automatically triggered to increase, forming a rapid response mechanism.

[0064] In this embodiment, a dynamic exclusion factor is introduced and an intelligent exclusion mechanism enhanced by negative feedback is established, so that the power market professional knowledge push system has "anti-harassment" capabilities. While ensuring the completeness of technical content coverage, it reduces user fatigue and prevents the appearance of repeated content.

[0065] Example 2:

[0066] Based on the above embodiment 1, refer to Figure 7 As shown, the present invention also discloses a power market information extraction and push system based on multimodal semantic fusion, which is used to implement the push method described in Example 1, including: A data preprocessing module is used to build a pre-trained corpus in the power market domain, which contains basic terminology documents, complex technical documents and multimodal data; The model training module includes: a general semantic understanding unit, which is used to pre-train the basic model using general corpus; a professional terminology recognition unit, which is used to introduce basic terminology documents of the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; and a cross-modal association unit, which is used to load complex technical documents and multimodal data and optimize the basic model's understanding of cross-modal associations through comparative learning tasks. The multimodal processing module includes: an encoding unit for performing text encoding and image encoding on the input power market information data; a confidence calculation unit for calculating the term matching degree and semantic consistency score of text features to generate text confidence, and calculating the key area recognition rate and context relevance score of image features to generate image confidence; a feature fusion unit for establishing a cross-modal attention association matrix based on the confidence scores of text and image, capturing the fine-grained interaction features between text and image, and fusing them to generate a unified semantic representation; The user modeling and push module includes: an interest modeling unit, which is used to establish a user model that includes long-term stable preferences and short-term dynamic interests based on the unified semantic representation generated by fusion; a strategy execution unit, which is used to adopt a descending order exploration strategy, combine known preferences to push highly relevant intelligence and randomly insert new domain content to explore potential interests; and a feedback adjustment unit, which is used to adjust the push content weight in real time according to user feedback.

[0067] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A method for extracting and pushing electricity market information based on multimodal semantic fusion, characterized by: include: Build a pre-trained corpus in the power market field, including basic terminology documents, complex technical documents and multimodal data; Use common corpus to pre-train the basic model and learn common semantic representations; introduce basic terminology documents from the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; load complex technical literature and multimodal data, and use contrastive learning tasks to optimize the basic model's understanding of cross-modal relationships; The trained basic model is applied to actual power market information processing, and multimodal encoding is performed on the input power market information data, including text encoding and image encoding; Calculate the term matching and semantic consistency scores of text features to generate text confidence scores; Calculate the key area recognition rate and context relevance score of image features to generate an image confidence score; Based on the text confidence score and image confidence score, a cross-modal attention correlation matrix is established to capture the fine-grained interaction features of images and texts and fuse them to generate a unified semantic representation; Based on the fusion of unified semantic representation, a user model with long-term stable preferences and short-term dynamic interests is established. A descending order exploration strategy is adopted to push highly relevant intelligence based on known preferences and randomly insert new domain content to explore potential interests. The weight of pushed content is adjusted in real time based on user feedback.

2. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized by: We annotated the data complexity of a pre-trained corpus from the power market domain and calculated a complexity score for each data entry, including term density, sentence length, and logical nesting level. Term density is defined as the percentage of specialized vocabulary in a document, sentence length is determined in bytes, and logical nesting level is quantified by the depth of the dependency parse tree. The training phase is divided according to the calculated complexity score, and the training batches are dynamically adjusted from low to high data complexity. Low-complexity data is used for initial fine-tuning, and medium- and high-complexity data is used for cross-modal correlation optimization.

3. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The process of introducing the basic terminology document of the power market and identifying the proprietary terms of the power market includes: Dynamically mask proprietary vocabulary in the power market, with the masking ratio increasing with the training stage. The masking ratio is ≤10% in the initial stage and increases to 30% in the final stage. The model predicts masked terms based on the context, and the prediction results are checked for consistency with the domain knowledge graph; When the prediction result fully matches the domain knowledge graph, the masking ratio of the next training batch is automatically increased by 10% to 15% of the current ratio; When the prediction result partially matches the domain knowledge graph, maintain the current masking ratio and increase the masking frequency of the same type of terms until the prediction result fully matches the domain knowledge graph; When the prediction result does not match the domain knowledge graph at all, the masking ratio is reduced to the initial value and the prediction result is regenerated until the prediction result completely matches the domain knowledge graph or there is a partial match.

4. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The process of calculating the term matching and semantic consistency scores of text features and generating text confidence scores includes: By comparing the terminology database, the coverage of the power market professional terms in the text is calculated to obtain the term matching degree; Extract local semantics from the text, analyze the global semantics of the text, compare the degree of correlation between local semantics and global semantics, measure the semantic coherence of the text context, and determine the semantic consistency score; The comprehensive text confidence score is the weighted sum of term matching and semantic consistency, and the weight is dynamically adjusted based on historical push feedback.

5. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The process of calculating the key area recognition rate and context relevance score of image features and generating the image confidence score includes: The visual saliency detection model is used to calculate the proportion of key device areas in the image and obtain the key area recognition rate. First, the global feature vector of the image is extracted, and then the deep feature vector of the text description corresponding to the image is obtained. The cosine similarity between the global feature vector of the image and the deep feature vector of the text description is calculated. The auxiliary information in the image background is detected and its logical relationship with the core components is analyzed. The relationship between regions is modeled through a graph neural network, and a context association graph is generated to obtain a context association score. The comprehensive image confidence score is the weighted sum of the key area recognition rate and the context relevance score, and the weight is dynamically adjusted based on historical push feedback.

6. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The construction of the cross-modal attention association matrix includes: Text-to-image attention: Calculates the similarity between each text word and the image region and generates an association weight; Image-to-text attention: Generate visual heatmaps by predicting region importance and reversely associate text descriptions; Fuse bidirectional attention weights to generate fine-grained interaction features.

7. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized by: The long-term preference is obtained through analysis of historical interaction data; The short-term interests are obtained by analyzing the user's current session behavior characteristics.

8. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized by: Exploration strategies in descending order include: ； Where: ε(t) is the exploration probability function value that decreases with the number of pushes, ε0 is the initial exploration rate, λ is the decay rate coefficient of the exploration probability, and t represents the number of pushes; When the random number is less than ε(t), new domain content is randomly selected for push; when the random number is greater than or equal to ε(t), content is pushed according to user preferences.

9. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 8 is characterized by: A dynamic exclusion factor R is introduced into the descending order exploration strategy to calculate the semantic similarity between the content to be pushed and the historical pushed content. If the similarity exceeds the threshold, the dynamic exclusion factor R is used to reduce its priority. If the similarity does not exceed the threshold, the dynamic exclusion factor R is not introduced, and the content to be pushed is determined according to the original priority ranking. The dynamic exclusion factor R is dynamically enhanced with the number of negative feedbacks from users. The formula is: ； Where: N neg is the number of negative feedback, N total is the total number of feedbacks, α, β, and γ are adjustable parameters, α represents the basic inhibition strength, β represents the negative feedback sensitivity, γ represents the decay rate coefficient of the dynamic repulsion factor, and t represents the number of pushes.

10. A power market information extraction and push system based on multimodal semantic fusion, used to implement the push method according to any one of claims 1 to 9, characterized in that: include: A data preprocessing module is used to build a pre-trained corpus in the power market domain, which contains basic terminology documents, complex technical documents and multimodal data; The model training module includes: a general semantic understanding unit, which is used to pre-train the basic model using general corpus; a professional terminology recognition unit, which is used to introduce basic terminology documents of the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; and a cross-modal association unit, which is used to load complex technical documents and multimodal data and optimize the basic model's understanding of cross-modal associations through comparative learning tasks. The multimodal processing module includes: an encoding unit for performing text encoding and image encoding on the input power market information data; a confidence calculation unit for calculating the term matching degree and semantic consistency score of text features to generate a text confidence score, and calculating the key area recognition rate and context relevance score of image features to generate an image confidence score; a feature fusion unit for establishing a cross-modal attention association matrix based on the text confidence and image confidence scores, capturing the fine-grained interaction features between text and images, and fusing them to generate a unified semantic representation; The user modeling and push module includes: an interest modeling unit, which is used to establish a user model that includes long-term stable preferences and short-term dynamic interests based on the unified semantic representation generated by fusion; a strategy execution unit, which is used to adopt a descending order exploration strategy, combine known preferences to push highly relevant intelligence and randomly insert new domain content to explore potential interests; and a feedback adjustment unit, which is used to adjust the push content weight in real time according to user feedback.

Images