Power market information extraction and pushing method and system based on multi-modal semantic fusion

Abstract

The application relates to a power market information extraction and pushing method and system based on multi-modal semantic fusion, which comprises the following steps: constructing a power market field pre-training corpus; adopting a phased pre-training strategy; performing multi-modal coding on input power market information data; generating text confidence score and image confidence score, establishing a cross-modal attention correlation matrix, capturing fine-grained interactive feature fusion of text and image, and generating unified semantic representation; based on the generated unified semantic representation, a user model containing long-term stable preferences and short-term dynamic interests is established, a decreasing order exploration strategy is adopted, known preferences are used to push highly relevant information, new field content is randomly inserted to mine potential interests, and the content weight of the push is adjusted in real time according to user feedback. The application can effectively solve the problems of heterogeneous data sources, diversified formats, inconsistent standards and information asymmetry of power market information data, and provide accurate and real-time power market information pushing services for users.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power market information, in particular to a power market information extraction and pushing method and system based on multi-modal semantic fusion. BACKGROUND

[0002] With the rapid development of smart grid and energy internet, a large amount of multi-modal data resources are generated in the power market, including professional literature, equipment operation log, power grid topology map, and equipment monitoring image.

[0003] In the prior art, the information extraction system based on natural language processing mainly uses a pre-training model in a general field for knowledge mining, but faces three significant defects:

[0004] Firstly, the model trained by general corpus lacks the support of a professional vocabulary library specific to the power market field, for example, the semantic understanding of professional terms such as "current valve lock" and "transient overvoltage" is biased, resulting in a decrease in the accuracy of key information extraction;

[0005] Secondly, the traditional multi-modal processing method stays in the shallow feature matching stage for the correlation analysis of graphic and text data, and fails to effectively capture the deep semantic association between the equipment schematic diagram and the operating parameters, and the granularity of cross-modal feature fusion is insufficient;

[0006] Thirdly, the existing pushing system mainly uses a static user portrait model, and the fixed pushing strategy leads to insufficient exploratory ability for new field technical intelligence, and lacks a dynamic adjustment mechanism based on real-time user feedback; In particular, when facing the application scenarios where long-term stable technical needs (such as relay protection parameter optimization) and short-term dynamic research hotspots (such as new energy grid connection technology) coexist in the power market field, the existing system is difficult to achieve precise personalized pushing. SUMMARY

[0007] Therefore, the present application aims to solve the problem that the information extraction system based on natural language processing in the prior art is not applicable in the power market field, and provides a power market information extraction and pushing method and system based on multi-modal semantic fusion, which can effectively solve the problems of heterogeneous information data sources, diversified formats, inconsistent standards, and information asymmetry in the power market, and provide precise and real-time power market information pushing services for power market users.

[0008] To solve the above technical problems, the present application provides a power market information extraction and pushing method based on multi-modal semantic fusion, comprising:

[0009] Constructing a power market field pre-training corpus, including basic term documents, complex technical literature, and multi-modal data;

[0010] The general corpus is used to pre-train the base model to learn general semantic representation; the base term document of the power market is introduced for domain adaptation training to enhance the recognition ability of the base model to the power specific vocabulary; the complex technical literature and multi-modal data are loaded, and the contrast learning task is used to optimize the understanding of the base model to the cross-modal correlation;

[0011] The trained base model is applied to the actual power market information processing, and the input power market information data is multi-modal coded, including text coding and image coding; the term matching degree and semantic consistency score of the text feature are calculated to generate the text confidence score; the key area recognition rate and context correlation degree score of the image feature are calculated to generate the image confidence score; based on the text confidence score and the image confidence score, a cross-modal attention correlation matrix is established to capture the fine-grained interaction feature fusion of the text and image to generate a unified semantic representation;

[0012] Based on the unified semantic representation generated by fusion, a user model containing long-term stable preferences and short-term dynamic interests is established, an order-decreasing exploration strategy is used, known preferences are used to push high-relevance information and random insertion of new field content is used to mine potential interests, and the content weight of the push is adjusted in real time according to user feedback.

[0013] In the present application, the data complexity in the power market field pre-training corpus is labeled, the complexity score of each data is calculated, including term density, sentence length and logical nesting level, wherein: the term density is defined as the proportion of professional vocabulary in the document, the sentence length is determined according to the byte, and the logical nesting level is quantified by the depth of the dependency syntax analysis tree;

[0014] According to the calculated complexity score, the training stage is divided, and the training batch is dynamically adjusted according to the data complexity from low to high, the low complexity data is used for initial fine-tuning, and the medium and high complexity data is used for cross-modal correlation optimization.

[0015] In the present application, the base term document of the power market is introduced, and the process of identifying the power market specific vocabulary includes:

[0016] The power market specific vocabulary is dynamically masked, and the masking proportion increases with the training stage, the masking proportion is ≤10% in the initial stage, and the masking proportion is increased to 30% in the final stage;

[0017] The model predicts the masked term according to the context, and the prediction result is consistent with the domain knowledge graph;

[0018] When the prediction result is completely matched with the domain knowledge graph, the masking proportion of the next training batch is automatically increased by 10% to 15% of the current proportion;

[0019] When the prediction result partially matches the domain knowledge graph, keep the current masking proportion and increase the masking frequency of the same type of term until the prediction result completely matches the domain knowledge graph;

[0020] When the prediction result does not match the domain knowledge graph at all, reduce the masking proportion to the initial value to regenerate the prediction result until the prediction result completely matches the domain knowledge graph or there is partial matching.

[0021] In the present application, the process of calculating the term matching degree and the semantic consistency score of the text feature to generate the text confidence score includes:

[0022] By comparing the term library, the coverage rate of the power market professional term in the text is calculated to obtain the term matching degree;

[0023] Extract the local semantics in the text, analyze the global semantics of the text, compare the correlation degree of the local semantics and the global semantics, measure the coherence of the text context semantics, and determine the semantic consistency score;

[0024] The text confidence score is the weighted sum of the term matching degree and the semantic consistency, and the weight is dynamically adjusted according to the historical push feedback.

[0025] In the present application, the process of calculating the key region recognition rate and the context correlation degree score of the image feature to generate the image confidence score includes:

[0026] By a visual saliency detection model, the proportion of the key equipment region in the image is calculated to obtain the key region recognition rate;

[0027] First, the global feature vector of the image is extracted, then the deep feature vector of the text description corresponding to the image is obtained, the cosine similarity of the global feature vector of the image and the deep feature vector of the text description is calculated, the auxiliary information in the image background is detected, the logical relationship between the auxiliary information and the core component is analyzed, the inter-regional correlation is modeled through a graph neural network, a context correlation graph is generated, and the context correlation degree score is obtained;

[0028] The image confidence score is the weighted sum of the key region recognition rate and the context correlation degree score, and the weight is dynamically adjusted according to the historical push feedback.

[0029] In the present application, the construction of the cross-modal attention correlation matrix includes:

[0030] Text-to-image attention: calculate the similarity between each text word and image region to generate correlation weight;

[0031] Image-to-text attention: generate a visual heat map through region importance prediction, and reversely associate the text description;

[0032] The bidirectional attention weight is fused to generate fine-grained interaction features.

[0033] In the application, the long-term preference is obtained by analyzing historical interaction data, and the short-term interest is obtained by analyzing user current session behavior characteristics.

[0034] In the application, the order-decreasing exploration strategy comprises:

[0035] ;

[0036] Wherein: epsilon (t) is the exploration probability function value decreasing with the number of pushes, epsilon 0 is the initial exploration rate, lambda is the decay rate coefficient of the exploration probability, and t represents the number of pushes.

[0037] When the random number is less than epsilon (t), the new field content push is randomly selected, and when the random number is greater than or equal to epsilon (t), the user preference push is performed.

[0038] In the application, the dynamic exclusion factor R is introduced in the order-decreasing exploration strategy, the semantic similarity of the to-be-pushed content and the historical pushed content is calculated, if the similarity exceeds a threshold value, the priority of the to-be-pushed content is reduced through the dynamic exclusion factor R, if the similarity does not exceed the threshold value, the dynamic exclusion factor R is not introduced, the to-be-pushed content is determined according to the original priority, and the dynamic exclusion factor R is dynamically enhanced according to the number of negative feedbacks, and the formula is:

[0039] ;

[0040] Wherein: N neg is the number of negative feedbacks, N total is the total number of feedbacks, alpha, beta and gamma are adjustable parameters, alpha represents the basic inhibition intensity, beta represents the negative feedback sensitivity, gamma represents the decay rate coefficient of the dynamic exclusion factor, and t represents the number of pushes.

[0041] To solve the above technical problems, the application further provides a power market information extraction and push system based on multi-modal semantic fusion, comprising:

[0042] A data preprocessing module is used to construct a power market field pre-training corpus, and the corpus comprises basic term documents, complex technical literature and multi-modal data.

[0043] A model training module comprises: a general semantic understanding unit used for pre-training a basic model using a general corpus; a professional term recognition unit used for introducing a power market basic term document for field adaptation training to enhance the recognition ability of the basic model to power-specific words; and a cross-modal association unit used for loading complex technical literature and multi-modal data, and optimizing the understanding of the basic model to the cross-modal association relationship through a contrast learning task.

[0044] The multi-modal processing module comprises: an encoding unit configured to perform text encoding and image encoding on input power market information data; a confidence score calculation unit configured to calculate a term matching degree and a semantic consistency score of a text feature to generate a text confidence score, and calculate a key region recognition rate and a context correlation score of an image feature to generate an image confidence score; and a feature fusion unit configured to establish a cross-modal attention correlation matrix based on the text confidence score and the image confidence score, capture fine-grained interaction features of the text and the image, and fuse to generate a unified semantic representation.

[0045] The user modeling and pushing module comprises: an interest modeling unit configured to establish a user model comprising long-term stable preferences and short-term dynamic interests based on the unified semantic representation generated by fusion; a strategy execution unit configured to adopt a decreasing order exploration strategy, combine known preferences to push highly relevant information, and randomly insert new field content to mine potential interests; and a feedback adjustment unit configured to adjust the weight of the pushed content in real time according to user feedback.

[0046] The above technical solutions of the present application have the following advantages compared with the prior art:

[0047] The power market information extraction and pushing method based on multi-modal semantic fusion disclosed in the present application firstly constructs a power market field corpus comprising basic terms, complex documents and multi-modal data, adopts a phased pre-training strategy: on the basis of general semantic understanding, the field adaptability is strengthened through a power market specific vocabulary recognition module, and the cross-modal correlation understanding of equipment drawings and technical documents is optimized by using a contrast learning task optimization model.

[0048] Secondly, a double confidence evaluation mechanism is innovatively designed at the feature processing layer, which quantitatively evaluates from four dimensions of term matching degree, semantic consistency, key region recognition rate and context correlation degree, and realizes fine-grained interaction fusion of text and image features through a cross-modal attention correlation matrix.

[0049] Finally, a user model fusing long-term stable preferences and short-term dynamic interests is constructed at the pushing strategy layer, a decreasing order exploration strategy is adopted to balance the relationship between known preference pushing and new field exploration, and the pushing weight is dynamically adjusted through a real-time feedback mechanism.

[0050] In combination with the above technical features, the beneficial effects brought by the present application mainly lie in the following three aspects:

[0051] (1) In terms of field adaptability, the information extraction accuracy is improved through the power market special corpus and the phased training mechanism, and especially when processing technical documents such as equipment failure reports, the error rate of professional term recognition is reduced.

[0052] (2) Cross-modal processing aspect, fine-grained feature fusion based on attention mechanism improves the accuracy of image-text association analysis, and can accurately associate the abnormal area of the wiring diagram with the corresponding current waveform feature in the equipment maintenance case;

[0053] (3) Push intelligent aspect, dynamic user model combined with exploration strategy improves the reach rate of high-value information, reduces the active search volume of users for new field technical information, and significantly improves the knowledge acquisition efficiency of power market technical personnel. BRIEF DESCRIPTION OF DRAWINGS

[0054] In order to make the content of the present application more easily understood, the present application will be further described in detail below according to specific embodiments of the present application and in conjunction with the accompanying drawings, wherein:

[0055] Figure 1 is a step flow chart of the power market information extraction and push method based on multi-modal semantic fusion of the present application;

[0056] Figure 2 is a step flow chart of the data complexity quantification labeling and dynamic optimization of the training process of the present application;

[0057] Figure 3 is a step flow chart of the introduction of the power market basic terminology document for power market specific vocabulary recognition of the present application;

[0058] Figure 4 is a step flow chart of the generation of the text confidence score of the present application;

[0059] Figure 5 is a step flow chart of the generation of the image confidence score of the present application;

[0060] Figure 6 is a step flow chart of the construction of the cross-modal attention association matrix of the present application;

[0061] Figure 7 is a structural framework diagram of the power market information extraction and push system based on multi-modal semantic fusion of the present application. DETAILED DESCRIPTION

[0062] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present application and implement it, but the embodiments are not limiting to the present application.

[0063] Embodiment one:

[0064] Reference Figure 1As shown, the application discloses a power market information extraction and pushing method based on multi-modal semantic fusion, aiming at the problem of inaccurate pushing caused by semantic understanding deviation of general models in the power market field, and realizes technical improvement through the following steps:

[0065] S10, a power market field pre-training corpus is constructed, including basic term documents, complex technical documents and multi-modal data.

[0066] In this embodiment, a pre-training corpus dedicated to the power market field is first constructed, covering basic term documents (such as IEC standard term table), complex technical documents (such as relay protection device research report) and multi-modal data (including equipment wiring diagram, infrared thermal imaging diagram and supporting technical documents); the corpus adopts a hierarchical storage architecture, the basic terms serve as bottom knowledge anchors, the complex documents provide context, and the multi-modal data establish graphic-text correlation samples.

[0067] S20, a general corpus is used to pre-train a basic model to learn general semantic representation; power market basic term documents are introduced for field adaptation training to enhance the recognition ability of the basic model for power-specific vocabulary; complex technical documents and multi-modal data are loaded, and a contrast learning task is used to optimize the understanding of the basic model for cross-modal correlation.

[0068] In this embodiment, three-stage model training is implemented:

[0069] General semantic foundation: a general corpus is used to pre-train a basic model to obtain general language understanding ability;

[0070] Professional vocabulary reinforcement: power market basic term documents (such as IEC standard term table) are introduced for field adaptation training to construct a field word vector space;

[0071] Cross-modal correlation optimization: complex technical documents (such as relay protection device research report) and supporting drawings are loaded, a contrast learning framework is used, the principle diagram description text and the structure schematic diagram of the same device are formed into a positive sample pair, different device graphic-text is formed into a negative sample pair, and the model is trained to capture graphic-text semantic correlation.

[0072] Among them: the basic model is a language general model of the prior art, which is a deep language model pre-trained based on large-scale general text data (such as Wikipedia, news, books, web pages, etc.), such model usually belongs to a neural network of Transformer architecture, captures the context dependency of text through a self-attention mechanism, can be an encoder type model (suitable for text understanding task) or a decoder type model (suitable for text generation task), and can learn general language rules, semantic representation and world knowledge from massive texts; the basic model is used as the starting point of the initial pre-training stage and has general language understanding ability (such as grammar, common semantics).

[0073] Again, the problem of professional vocabulary understanding deviation of general model is solved by domain knowledge injection. The pre-trained model is applied in power market terminology disambiguation task, and the accuracy can be improved through cross-modal retrieval. For example, when processing the description of "circuit breaker tripping coil fault", the model can accurately associate the electromagnet component in the coil structure diagram with the current parameter abnormality in the text description.

[0074] S30, apply the trained base model to actual power market information processing, and perform multi-modal encoding on the input power market information data, including text encoding and image encoding; calculate the term matching degree and semantic consistency score of the text features to generate a text confidence score; calculate the key region recognition rate and context correlation degree score of the image features to generate an image confidence score; based on the text confidence score and the image confidence score, establish a cross-modal attention association matrix to capture the fine-grained interaction feature fusion of text and image to generate a unified semantic representation.

[0075] The model trained through step S20 is deployed to the actual power market information processing scene to perform parallel encoding processing and dual-channel confidence evaluation on the input data.

[0076] The parallel encoding processing includes:

[0077] Text encoding: adopt dynamic segmentation strategy, preferentially match power market terminology dictionary, and generate semantic vector containing part-of-speech tags (such as verb "excitation" and noun "lightning arrester");

[0078] Image encoding: locate the key components of the equipment (such as transformer bushing joints) through the region proposal network, and extract the visual features while recording the spatial coordinate information.

[0079] The dual-channel confidence evaluation includes:

[0080] Text confidence score: calculate the term dictionary matching degree (such as the exact matching score of "neutral point offset") and the context semantic consistency (detect the reasonable collocation of professional terms through attention mechanism);

[0081] Image confidence score: based on the key region recognition rate (such as the completeness of identifying the switch cabinet panel identifier) and the context correlation degree (such as the rationality of the spatial relationship between the insulator image and the adjacent hardware).

[0082] Cross-modal feature interaction: fuse the text and image confidence to construct an attention association matrix to capture the fine-grained interaction feature fusion of text and image to generate a unified semantic representation; for example, when "acetylene content exceeds the standard" appears in the transformer oil chromatographic analysis report, the model automatically enhances the association weight of the arc path area in the transformer internal discharge schematic diagram, realizing the alignment and fusion of fault features.

[0083] In step S30, the noise data is filtered by the quantitative evaluation mechanism, the filtering efficiency of invalid information in the substation inspection report processing can be improved, and the feature fusion module improves the accuracy of the graphic and text evidence matching of fault diagnosis.

[0084] S40, based on the unified semantic representation generated by fusion, a user model containing long-term stable preference and short-term dynamic interest is established, an order decreasing exploration strategy is adopted, high correlation information is pushed by using known preference, new field content is randomly inserted to mine potential interest, and the weight of the pushed content is adjusted in real time according to user feedback.

[0085] Based on the unified semantic representation after fusion, a user model is constructed, and user preference modeling is established to establish a user portrait, including: long-term stable preference, capturing high-frequency search topics for more than 6 months and short-term dynamic interest, tracking new hot spots in the past two weeks; a mixed push mechanism with decreasing order is designed, 30% of the push amount is set as new field content in the initial stage, and the exploration proportion is reduced according to an exponential curve with the increase of the push times, so that the user can have stable knowledge input and avoid information overload; an online learning module is deployed, when the user performs a collection / ignore operation on the pushed information, the system adjusts the weight coefficient of the related technical theme, and triggers the incremental update of the corpus.

[0086] The power market information extraction and push method disclosed by the application adopts field knowledge embedding, multi-modal fine processing and dynamic push strategy, effectively solves the problems of professional barriers, modal gap and demand drift in power market information processing, and provides reliable technical support for smart grid construction.

[0087] Further, in combination with actual application, the pre-training corpus construction and phased training strategy are further optimized, and it is intended to realize fine control of the training process through a data complexity dynamic scheduling mechanism, for this purpose, the data complexity is quantitatively labeled, and the dynamic optimization of the training process is implemented based on the complexity score, as shown in Figure 2 The specific steps include the following steps:

[0088] Multidimensional feature analysis is performed on each piece of data in the pre-trained corpus mentioned above (including text, images, and image-text pairs): a dictionary of power market terminology is used for precise matching, and term density is defined as the percentage of professional vocabulary to the total number of words; bytes are used as the unit of measurement rather than characters to avoid statistical bias in mixed Chinese and English scenarios; long and difficult sentences in technical documents (such as equipment operating specifications containing multiple conditional clauses) are segmented, and the single sentence length threshold is set to 512 bytes to accommodate model input limitations; a semantic tree is constructed based on a dependency syntax analysis tool, and the tree depth value is used to represent logical complexity. For example, the syntactic tree depth of "When the bus voltage exceeds 10% of the rated value and lasts for more than 2 seconds, the overvoltage protection device should be activated" is 5 levels, while the simple descriptive sentence "The circuit breaker is tripped" has a depth of only 2 levels.

[0089] Based on the above complexity scores, dynamic optimization of the training process is implemented. Low-complexity data (term density <15%, sentence length <256 bytes, nesting level ≤3) is used in the initial fine-tuning stage, such as simple contextual data such as equipment nameplate information and standardized operating procedures; medium-complexity data (term density 15%-30%, sentence length 256-384 bytes, nesting level 4-5) is used for transition training, such as causal descriptions in equipment maintenance records; high-complexity data (term density >30%, sentence length >384 bytes, nesting level ≥6) is used for cross-modal association optimization, such as academic papers containing multiple hypothesis inferences.

[0090] In the initial stage, low, medium and high complexity data are mixed in a ratio of 7:2:1 to ensure a smooth transition of the model; after training every 10 groups of data, the ratio is automatically adjusted based on the performance of the validation set, and the maximum proportion of high complexity data can be increased to 40%.

[0091] Specifically, refer to Figure 3 As shown in the figure, when introducing the basic terminology document of the electricity market, the process of identifying the electricity market specific vocabulary includes:

[0092] Dynamic masking is performed on proprietary terms in the power market, with the masking ratio increasing with each training stage. In the initial stage, the masking ratio is ≤10%, mainly targeting high-frequency basic terms (such as "insulation resistance" and "phase-to-phase short circuit") to ensure that the model initially establishes a weak association between terms and context. The masking ratio is then linearly increased, with a focus on complex terms (such as "zero-sequence current protection" and "transient voltage stability"). A partial masking strategy (such as masking "zero sequence" and retaining "current protection") is adopted to force the model to understand the internal structure of the terms. In the final stage, the masking ratio is increased to 30%, and random interval masking (such as masking "excitation__second harmonic") is implemented for low-frequency professional terms (such as "excitation inrush second harmonic") to enhance the model's ability to recover from fragmented terms.

[0093] The model predicts the masked terms according to the context, and the prediction result is consistent with the domain knowledge graph. Different verification logics can be used, such as complete matching verification, partial matching verification, and conflict detection. Finally, different strategies are constructed according to the verification result:

[0094] When the prediction result completely matches the domain knowledge graph, the masking proportion of the next training batch is automatically increased by 10% to 15% of the current proportion;

[0095] When the prediction result partially matches the domain knowledge graph, the current masking proportion is maintained, but focused training needs to be implemented for this type of term, the masking frequency of the same type of term is increased, and the masking is repeated 3-5 times in the same context until the prediction result completely matches the domain knowledge graph;

[0096] When the prediction result does not match the domain knowledge graph at all, the masking proportion is reduced to the initial value to generate a prediction result again, the relationship network of the term is extracted from the knowledge graph to generate a reinforced training sample, the term definition sentence is first learned, and then gradually transitioned to a complex application scenario until the prediction result completely matches the domain knowledge graph or there is partial matching.

[0097] Specifically, in order to realize cross-modal processing, it is necessary to quantize the generated text and image so that the text and image can be input in the cross-modal attention matrix. Therefore, it is necessary to calculate the term matching degree and semantic consistency score of the text feature, generate a text confidence score, and calculate the key area recognition rate and context correlation degree score of the image feature, and generate an image confidence score.

[0098] Further, as shown in Figure 4 In the text encoding stage, the embodiment adopts a three-level evaluation system to generate a text confidence score:

[0099] Term matching degree calculation: The power market domain term knowledge base (containing 82,000 standard terms) is used for word-by-word scanning. Not only the term frequency is counted, but also the hierarchical relationship of the term (such as "differential protection" belonging to "relay protection" subclass) is recorded to distinguish core terms from auxiliary terms.

[0100] Semantic consistency analysis: Local semantic extraction uses a sliding window mechanism to extract a fixed number of words using a fixed length window to generate a semantic vector for each window. Global semantics are modeled through a full-text attention mechanism, focusing on capturing the logical flow guided by transition words (such as "however" and "therefore"). When calculating the correlation degree, the semantic evolution track is tracked, and when contradictory expressions are detected, the semantic consistency score automatically decreases.

[0101] Dynamic weight adjustment: In the initial stage, the term matching degree weight is set to 60%, and the semantic consistency is set to 40%. According to user feedback (such as labeling behavior on the pushed content), the weight is adjusted in real time. When the user frequently labels the term error, the term matching degree weight can be increased to 75% at most. A feedback-weight transfer function is established to ensure that the single weight adjustment amplitude does not exceed 5%, avoiding system shock.

[0102] Further, in the image encoding stage, referring to Figure 5 as shown, the embodiment adopts double feature evaluation to construct an image quality index and generate an image confidence score:

[0103] Key area identification: The image is pixel-level segmented to identify the core components of the device. The false detection area is corrected by prior knowledge constraint. Spatial continuity evaluation is introduced to avoid misjudgment of noise points as key components.

[0104] Context association modeling: First, the global feature vector of the image is extracted, and then the deep feature vector of the text description corresponding to the image is obtained. The cosine similarity of the global feature vector of the image and the deep feature vector of the text description is calculated. The auxiliary information in the image background is detected, and the logical relationship between the auxiliary information and the core components is analyzed. The inter-regional association is modeled by a graph neural network, a deep learning model specially processing graph structure data, which can automatically learn the mutual influence between entities in complex relationships. In the embodiment, the graph neural network can meet the demand of image confidence score evaluation.

[0105] Adaptive weighting: The initial weight is set to 70% for recognition rate and 30% for association degree. When the user interaction rate on image type content decreases, the association degree weight increases by 2% each time.

[0106] In the embodiment, the multi-modal processing framework is further improved by text confidence score and image confidence score. When the text confidence score and the image confidence score are less than a set value, which is the minimum score representing the association degree between text and image, the data reacquisition process is automatically triggered, and low-quality input is intercepted. For modalities with high text confidence score and image confidence score, further amplification can be obtained in the attention matrix. Moreover, the user interaction data on the pushed content is real-time fed back to the confidence model, forming a continuous optimization cycle.

[0107] Specifically, in the embodiment, in order to generate a unified semantic representation of text and image, the text confidence score and the image confidence score are used to construct a dynamic interactive cross-modal attention association matrix, as shown in Figure 6

[0108] ​Text-to-image attention mapping: Establish a text-driven visual attention mechanism, perform similarity operations between each text word vector and image region features, implement reinforcement calculation on power market professional terms, and double the similarity score when detecting core terms in the power market field. Use a multi-head attention mechanism to calculate the correlation of different semantic subspaces in parallel, such as: one attention head is dedicated to handling device spatial position relationships (e.g., "left sleeve" matching the left quadrant region of the image).

[0109] Image-to-text attention feedback: Establish a visual information correction mechanism for text, obtain a region importance visual heat map, identify key decision-making areas in the image, introduce power market device prior knowledge, set a basic attention threshold for standard components, and avoid the model ignoring key but visually unobvious areas; calculate cross-attention between the feature vector of high-heat image regions and all text words to find the most relevant text description.

[0110] Fusion of bidirectional attention weights to generate fine-grained interaction features: Perform Hadamard product operations on the text-to-image weight matrix and the image-to-text weight matrix to highlight high-confidence areas confirmed by both directions; Implement weight reduction for one-way high-weight regions (only text or image unilateral attention), with a reliability discount factor of 0.3-0.5; When the bidirectional weight sum is >1.5, activate the feature enhancement channel to amplify the key feature dimension by 2-3 times, establish a cross-modal memory unit to store high-frequency co-occurrence patterns (such as the association pattern of "low oil level" and the oil level gauge image), and accelerate the processing of similar scenarios in the future.

[0111] In this embodiment, a user knowledge graph is constructed through diachronic behavior analysis, including long-term preferences and short-term interests.

[0112] The long-term preferences are obtained through historical interaction data analysis, with a time window set to 6-24 months of historical interaction data, covering full-dimensional behaviors such as literature review (e.g., IEEE paper download records), device attention (e.g., transformer model collection), expert consultation (e.g., relay protection question asking), and recording in-depth interaction behaviors of user push content, including average reading time, labeling density, and other fine-grained indicators;

[0113] Capture interest evolution trends, identify stable interest points, and construct a power market field-specific knowledge graph, map user behavior to the technical system, and define a preference intensity index with higher recent behavior weight but not exceeding 50% of the total, avoiding "recent cause effect" distortion of long-term judgment, identifying 3-5 core technical directions, and forming a preference portrait base.

[0114] The short-term interest is obtained by analyzing the current session behavior characteristics of the user, real-time monitoring of high-frequency terms in the current work session, document jump path, capturing interaction intensity characteristics, including page dwell time, mouse trajectory hotspots, setting the half-life of the short-term interest to 72 hours, automatically attenuating the interest intensity by a certain percentage every 24 hours, ensuring that the system timely switches the tracking focus, when detecting interest shift, starting a new interest tracking cycle.

[0115] In the process of power market information pushing, it is not desirable to only push the content of interest to the user, therefore, the application also proposes a decreasing order exploration strategy, a dynamic decay exploration-exploitation balance mechanism is constructed, the decreasing order exploration strategy is:

[0116] ;

[0117] Wherein: epsilon (t) is the exploration probability function value decreasing with the number of pushes, epsilon0 is the initial exploration rate, lambda is the decay rate coefficient of exploration probability, t represents the number of pushes.

[0118] The initial exploration rate epsilon0 is set to 30%-50%, the low value is taken for the technology conservative user, and the high value is taken for the innovation-oriented user, the decay coefficient lambda is adaptively set, when detecting the active short-term interest, the value of lambda is reduced to slow down the decay speed;

[0119] The exploration probability is updated according to the number of pushes (non-natural time), epsilon (t) is recalculated after each push, when the user continuously rejects the exploration content for 10 times, epsilon (t) is temporarily increased to (epsilon0+epsilon (t)) / 2, avoiding falling into local optimum.

[0120] When the random number < epsilon (t), randomly select new field content pushing, when the random number >= epsilon (t), push according to user preference.

[0121] Wherein: the random number is an independent random variable obeying uniform distribution in the interval [0,1), a cryptographically secure pseudo-random number generator is used, the seed is initialized by a hardware entropy source, ensuring unpredictability and statistical uniformity; a new random number is generated before each push and compared with epsilon (t), when the random number < epsilon (t), the exploration behavior is triggered, the content is randomly selected from the new field candidate pool, when the random number >= epsilon (t), the content highly matched with the known preference of the user is preferentially selected.

[0122] Further, record the actual interaction depth (such as reading completion rate, collection behavior) of the user to the exploration content, when the success rate is continuously greater than 60% for 5 times, epsilon0 is increased by 5%; through short-term interest analysis, the relevance of the exploration content is dynamically verified, such as detecting that the user actively retrieves the new field term pushed, then triggering the recalibration of the value of lambda.

[0123] For sub-fields with rapid technological updates (such as the power market Internet of Things), a segmented decay function is used: the first 20 times of push are slowly decayed, and later times are accelerated.

[0124] Through the order-decreasing exploration strategy, the stability and innovation of power market professional knowledge services can be balanced. Under the premise of maintaining the stability of core content push, the push of new field information can be improved.

[0125] In practical applications, the content pushed by the above method often appears repeatedly. After multiple iterations and optimization to accurately determine the user's preferences, the repetition rate of text content is higher when pushing power market information in the same technical field. Although new field information is constantly pushed through the order-decreasing exploration strategy, these information is not what the customer likes, and it cannot form a new push strategy, causing user confusion.

[0126] To solve this problem, the order-decreasing exploration strategy is further optimized by introducing a dynamic exclusion factor R. The semantic similarity between the content to be pushed and the historical push content is calculated, and the similarity between the candidate push content and the user's last 30 historical push content is compared. The term overlap rate is compared for the text part, and the device structure features are compared for the image part. A basic similarity threshold is set, which can be dynamically adjusted according to the user interest model: it is lowered for professional depth users and raised for knowledge breadth users.

[0127] If the similarity exceeds the threshold, the priority is reduced through the dynamic exclusion factor R. If the similarity does not exceed the threshold, the dynamic exclusion factor R is not introduced, and the content to be pushed is determined according to the original priority. The dynamic exclusion factor R is dynamically enhanced with the number of negative feedbacks, and the formula is:

[0128] ;

[0129] Where: N neg is the number of negative feedbacks, N total is the total number of feedbacks, α, β, γ are adjustable parameters, and t represents the number of pushes.

[0130] Specifically, explicit negative feedback (such as clicking "not interested") and implicit negative feedback (such as content dwell time < 15 seconds) are recorded, and the number of negative feedbacks is updated in real time. α represents the basic suppression intensity, β represents the negative feedback sensitivity, and γ represents the decay rate coefficient of the dynamic exclusion factor. Through online learning, the values are dynamically adjusted, for example: when it is detected that the user has rejected the same type of content for 3 times in a row, the value of β is automatically increased, forming a rapid response mechanism.

[0131] In the embodiment, a dynamic exclusion factor is introduced to establish a smart exclusion mechanism with negative feedback enhancement, so that the power market professional knowledge pushing system has the ability to prevent harassment, reduces user fatigue, and prevents repeated content from appearing while ensuring complete coverage of technical content.

[0132] Embodiment two:

[0133] Based on the above embodiment one, referring to Figure 7 The application also discloses a power market information extraction and pushing system based on multi-modal semantic fusion, which is used for implementing the pushing method in embodiment one and comprises:

[0134] A data preprocessing module is used for constructing a power market field pre-training corpus, and the corpus comprises basic term documents, complex technical literature and multi-modal data.

[0135] A model training module comprises: a general semantic understanding unit used for pre-training a basic model by using general corpus; a professional term recognition unit used for introducing power market basic term documents for field adaptation training to enhance the recognition ability of the basic model to power-specific words; and a cross-modal association unit used for loading complex technical literature and multi-modal data and optimizing the understanding of the basic model to cross-modal association by a contrast learning task.

[0136] A multi-modal processing module comprises: an encoding unit used for text encoding and image encoding of input power market information data; a confidence calculation unit used for calculating a term matching degree and a semantic consistency score of text features to generate a text confidence, and calculating a key region recognition rate and a context association degree score of image features to generate an image confidence; and a feature fusion unit used for establishing a cross-modal attention association matrix based on the confidence scores of the text and the image, capturing fine-grained interactive features of the text and the image and fusing to generate a unified semantic representation.

[0137] A user modeling and pushing module comprises: an interest modeling unit used for establishing a user model containing long-term stable preferences and short-term dynamic interests based on the unified semantic representation generated by fusion; a strategy execution unit used for pushing high-relevance intelligence combined with known preferences by adopting a decreasing order exploration strategy and randomly inserting new field content to mine potential interests; and a feedback adjustment unit used for adjusting the weight of the pushing content in real time according to user feedback.

[0138] Those skilled in the art will appreciate that embodiments of the present application can be devised for a method, a system, or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer readable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0139] Obviously, the above-described embodiments are only examples for clearly illustrating the present application, and are not intended to limit the present application. Based on the above description, other different forms of changes or variations can be made by those skilled in the art. Here, it is not necessary and impossible to enumerate all the embodiments. The obvious changes or variations derived therefrom are still within the protection scope of the present application.

Claims

1. A method for extracting and pushing electricity market information based on multimodal semantic fusion, characterized by: include: Build a pre-trained corpus in the power market field, including basic terminology documents, complex technical documents and multimodal data; Use common corpus to pre-train the basic model and learn common semantic representations; introduce basic terminology documents from the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; load complex technical literature and multimodal data, and use contrastive learning tasks to optimize the basic model's understanding of cross-modal relationships; The trained basic model is applied to actual power market information processing, and multimodal encoding is performed on the input power market information data, including text encoding and image encoding; Calculate the term matching and semantic consistency scores of text features to generate text confidence scores; Calculate the key area recognition rate and context relevance score of image features to generate an image confidence score; Based on the text confidence score and image confidence score, a cross-modal attention correlation matrix is ​​established to capture the fine-grained interaction features of images and texts and fuse them to generate a unified semantic representation; The construction of the cross-modal attention association matrix includes: text-to-image attention: calculating the similarity between each text word and the image region and generating association weights; image-to-text attention: generating a visual heat map through region importance prediction and reversely associating text descriptions; fusing bidirectional attention weights to generate fine-grained interaction features; Based on the fusion of unified semantic representation, a user model is established that includes long-term stable preferences for high-frequency search topics and short-term dynamic interests in emerging hot spots. A descending order exploration strategy is adopted to push highly relevant intelligence based on known preferences and randomly insert new field content to explore potential interests. The weight of pushed content is adjusted in real time based on user feedback.

2. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized by: We annotated the data complexity of a pre-trained corpus from the power market domain and calculated a complexity score for each data entry, including term density, sentence length, and logical nesting level. Term density is defined as the percentage of specialized vocabulary in a document, sentence length is determined in bytes, and logical nesting level is quantified by the depth of the dependency parse tree. The training phase is divided according to the calculated complexity score, and the training batches are dynamically adjusted from low to high data complexity. Low-complexity data is used for initial fine-tuning, and medium- and high-complexity data is used for cross-modal correlation optimization.

3. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The process of introducing the basic terminology document of the power market and identifying the proprietary terms of the power market includes: Dynamically mask proprietary vocabulary in the power market, with the masking ratio increasing with the training stage. The masking ratio is ≤10% in the initial stage and increases to 30% in the final stage. The model predicts masked terms based on the context, and the prediction results are checked for consistency with the domain knowledge graph; When the prediction result fully matches the domain knowledge graph, the masking ratio of the next training batch is automatically increased by 10% to 15% of the current ratio; When the prediction result partially matches the domain knowledge graph, maintain the current masking ratio and increase the masking frequency of the same type of terms until the prediction result fully matches the domain knowledge graph; When the prediction result does not match the domain knowledge graph at all, the masking ratio is reduced to the initial value and the prediction result is regenerated until the prediction result completely matches the domain knowledge graph or there is a partial match.

4. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The process of calculating the term matching and semantic consistency scores of text features and generating text confidence scores includes: By comparing the terminology database, the coverage of the power market professional terms in the text is calculated to obtain the term matching degree; Extract local semantics from the text, analyze the global semantics of the text, compare the degree of correlation between local semantics and global semantics, measure the semantic coherence of the text context, and determine the semantic consistency score; The comprehensive text confidence score is the weighted sum of term matching and semantic consistency, and the weight is dynamically adjusted based on historical push feedback.

5. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The process of calculating the key area recognition rate and context relevance score of image features and generating the image confidence score includes: The visual saliency detection model is used to calculate the proportion of key device areas in the image and obtain the key area recognition rate. First, the global feature vector of the image is extracted, and then the deep feature vector of the text description corresponding to the image is obtained. The cosine similarity between the global feature vector of the image and the deep feature vector of the text description is calculated. The auxiliary information in the image background is detected and its logical relationship with the core components is analyzed. The relationship between regions is modeled through a graph neural network, and a context association graph is generated to obtain a context association score. The comprehensive image confidence score is the weighted sum of the key area recognition rate and the context relevance score, and the weight is dynamically adjusted based on historical push feedback.

6. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized in that: The long-term preference is obtained through analysis of historical interaction data; The short-term dynamic interest is obtained by analyzing the user's current session behavior characteristics.

7. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 1 is characterized by: Exploration strategies in descending order include: ； Where: ε(t) is the exploration probability function value that decreases with the number of pushes, ε0 is the initial exploration rate, λ is the attenuation rate coefficient of the exploration probability, and t represents the number of pushes; when the random number is less than ε(t), new domain content is randomly selected for push; when the random number is greater than or equal to ε(t), push is performed according to user preferences.

8. The method for extracting and pushing electricity market information based on multimodal semantic fusion according to claim 7 is characterized in that: A dynamic exclusion factor R is introduced into the descending order exploration strategy to calculate the semantic similarity between the content to be pushed and the historical pushed content. If the similarity exceeds the threshold, the dynamic exclusion factor R is used to reduce its priority. If the similarity does not exceed the threshold, the dynamic exclusion factor R is not introduced, and the content to be pushed is determined according to the original priority ranking. The dynamic exclusion factor R is dynamically enhanced with the number of negative feedbacks from users. The formula is: ； Where: N neg is the number of negative feedback, N total is the total number of feedbacks, α, β, and γ are adjustable parameters, α represents the basic inhibition strength, β represents the negative feedback sensitivity, γ represents the decay rate coefficient of the dynamic repulsion factor, and t represents the number of pushes.

9. A power market information extraction and push system based on multimodal semantic fusion, used to implement the push method according to any one of claims 1 to 8, characterized in that: include: A data preprocessing module is used to build a pre-trained corpus in the power market domain, which contains basic terminology documents, complex technical documents and multimodal data; The model training module includes: a general semantic understanding unit, which is used to pre-train the basic model using general corpus; a professional terminology recognition unit, which is used to introduce basic terminology documents of the power market for domain adaptation training to enhance the basic model's ability to recognize power-specific vocabulary; and a cross-modal association unit, which is used to load complex technical documents and multimodal data and optimize the basic model's understanding of cross-modal associations through comparative learning tasks. The multimodal processing module includes: an encoding unit for performing text encoding and image encoding on the input power market information data; a confidence calculation unit for calculating the term matching degree and semantic consistency score of text features to generate a text confidence score, and calculating the key area recognition rate and context relevance score of image features to generate an image confidence score; a feature fusion unit for establishing a cross-modal attention association matrix based on the text confidence and image confidence scores, capturing the fine-grained interaction features between text and images, and fusing them to generate a unified semantic representation; The user modeling and push module includes: an interest modeling unit, which is used to establish a user model that includes long-term stable preferences and short-term dynamic interests based on the unified semantic representation generated by fusion; a strategy execution unit, which is used to adopt a descending order exploration strategy, combine known preferences to push highly relevant intelligence and randomly insert new domain content to explore potential interests; and a feedback adjustment unit, which is used to adjust the push content weight in real time according to user feedback.

Images