Image-text fusion cross-modal attribution electric power marketing operation verification method and device

Abstract

The application discloses a cross-modal attribution electric power marketing operation verification method and device based on image-text fusion, relates to the technical field of intelligent electric power marketing, and improves the intelligent level of on-site operation verification of electric power marketing. The method comprises the following steps: in response to an electric power marketing operation verification request, acquiring on-site image data and associated text data of electric power marketing operation; performing feature extraction on the on-site image data based on a preset image processing algorithm, obtaining image features, and performing image preliminary verification to obtain a first verification result; performing feature extraction on the associated text data based on a preset text processing algorithm, obtaining text features, and performing text preliminary verification to obtain a second verification result; inputting the image features and the text features into a pre-trained cross-modal fusion model to obtain fusion features, performing cross-modal attribution analysis based on the fusion features, and obtaining a third verification result; and performing risk level evaluation according to the third verification result to obtain a verification report.

Description

Technical Field

[0001] This application relates to the field of intelligent power marketing technology, and in particular to a cross-modal attribution power marketing operation verification method and device based on image and text fusion. Background Technology

[0002] With the continued advancement of the "dual-carbon" strategy and the deepening of power market reform, electricity demand from users is growing rapidly. Consequently, the workload of on-site operations at the marketing level, including electricity inspections, verification of metering device installation standards, and on-site hazard investigations, has increased significantly. Traditional on-site inspections rely heavily on manual experience, which is not only inefficient but also susceptible to subjective judgment in hazard identification. Effective testing methods for metering device installation standards are lacking, and errors frequently occur during meter information entry. Furthermore, the image data collected on-site is fragmented with textual information such as work orders, meter nameplates, and asset numbers, making efficient correlation analysis difficult and resulting in poor interpretability of verification conclusions and delayed risk assessment.

[0003] In related technologies, to address the aforementioned issues, some solutions employ single-modal image recognition techniques for detecting electrical hazards or use OCR for recognizing electricity meter readings. Specifically, these solutions require using computer vision models to extract and classify features from images of metering devices captured on-site, or using optical character recognition (OCR) technology to extract numerical and textual information from meter nameplates and independently output the detection results. Additionally, some improved solutions attempt to perform simple matching between the recognized text and preset work order templates to determine information consistency.

[0004] In the process of developing the relevant technology, the applicant recognized that the relevant technology has at least the following technical problems: On the one hand, general computer vision models have low accuracy and serious hallucination problems in complex lighting, occlusion, or tilted shooting scenarios, and on-site interference factors often lead to frequent false detections and missed detections. On the other hand, they cannot achieve cross-modal attribution analysis, nor can they form a closed-loop risk assessment and automatic report generation function, making it difficult to support rapid and reliable decision-making by grassroots inspectors. The level of intelligence in grassroots on-site operation inspections of power grid marketing is not high, and the quality of power services cannot be guaranteed. Summary of the Invention

[0005] In view of this, this application provides a cross-modal attribution method and device for verifying power marketing operations through image and text fusion. The main purpose is to solve the problem that the current level of intelligence in on-site operation inspections at the grassroots level of power grid marketing is not high and the quality of power services cannot be guaranteed.

[0006] According to a first aspect of this application, a cross-modal attribution method for verifying electricity marketing operations using image-text fusion is provided, the method comprising: In response to a request for verification of electricity marketing operations, obtain on-site image data and related text data of the electricity marketing operations; Based on a preset image processing algorithm, feature extraction is performed on the on-site image data to obtain image features, and a preliminary image verification is performed based on the image features to obtain a first verification result; Based on a preset text processing algorithm, feature extraction is performed on the associated text data to obtain text features, and a preliminary text check is performed based on the text features to obtain a second check result; The image features and text features are input into a pre-trained cross-modal fusion model to obtain fused features. Cross-modal attribution analysis is performed based on the fused features to obtain a third verification result. Risk level assessment is then performed based on the third verification result to obtain a verification report.

[0007] According to a second aspect of this application, a cross-modal attribution electricity marketing operation verification device based on image-text fusion is provided, the device comprising: The acquisition module is used to acquire on-site image data and associated text data of power marketing operations in response to power marketing operation verification requests; The image verification module is used to extract features from the on-site image data based on a preset image processing algorithm to obtain image features, and to perform preliminary image verification based on the image features to obtain a first verification result; The text verification module is used to extract features from the associated text data based on a preset text processing algorithm to obtain text features, and to perform preliminary text verification based on the text features to obtain a second verification result. The fusion verification module is used to input the image features and the text features into a pre-trained cross-modal fusion model to obtain fused features, and to perform cross-modal attribution analysis based on the fused features to obtain a third verification result, so as to conduct risk level assessment based on the third verification result and obtain a verification report.

[0008] According to a third aspect of this application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method described in any of the first aspects above.

[0009] According to a fourth aspect of this application, a computer-readable storage medium is provided, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the method described in any one of the first aspects above.

[0010] By employing the aforementioned technical solutions, this application provides a cross-modal attribution method and apparatus for verifying power marketing operations through image-text fusion. This application simultaneously acquires on-site images and associated text data, and performs preliminary verification separately. This effectively mitigates the risk of false positives and false negatives under interference from lighting, occlusion, etc., in single-modal scenarios. Furthermore, by inputting image and text features into the cross-modal fusion model, visual information such as meter installation status and nameplate numbers can deeply interact with textual information such as work orders and asset numbers. This allows for the use of visual evidence to locate abnormal areas and the verification of consistency through textual semantics in attribution analysis, significantly improving the interpretability of verification conclusions. Based on risk level assessment of fused features and automatically generated verification reports, this provides closed-loop decision support for grassroots operations, from feature extraction to attribution to risk assessment. This significantly improves the intelligence and reliability of on-site power marketing operation verification, ensuring the quality of power services.

[0011] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0012] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This illustration shows a flowchart of a cross-modal attribution method for verifying electricity marketing operations using image-text fusion, provided in an embodiment of this application. Figure 2 This illustration shows a structural schematic diagram of a cross-modal attribution power marketing operation verification device for image-text fusion provided in an embodiment of this application; Figure 3 A schematic diagram of the device structure of a computer device provided in an embodiment of this application is shown. Detailed Implementation

[0013] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0014] This application provides a cross-modal attribution method for verifying electricity marketing operations using image-text fusion, such as... Figure 1 As shown, the method includes: S10: In response to a request for verification of electricity marketing operations, obtain on-site image data and associated text data of the electricity marketing operations.

[0015] In this embodiment, grassroots inspectors initiate on-site verification tasks via mobile apps or smart tools, enabling the verification system to detect the power marketing operation verification request. In response to the power marketing operation verification request, the verification system acquires two types of data. One type is on-site image data, which consists of multi-angle photos taken in real-time using a mobile phone or dedicated camera, showing the overall view of the meter box, the front of the meter, the seal location, wiring details, and meter box markings. The other type is associated text data, which is structured and unstructured text automatically retrieved from the marketing business system or on-site work orders, including asset numbers, meter models, user names, historical meter codes, installation specifications, and past hazard records. Both types of data are bound together by the same work order number to ensure a consistent business context for subsequent analysis.

[0016] The above process enables the simultaneous acquisition of image and text information, fundamentally resolving the issue of fragmented image data compared to work orders, nameplates, and other information in traditional inspections. This provides an aligned data foundation for subsequent cross-modal fusion. For example, when inspectors conduct a routine check of a community's meter box, the inspection system acquires two photos: one of the meter box's exterior and the other of the meter's front. Simultaneously, it retrieves the meter's asset number [GX-×××-2023] and text information from the work order, such as "Last inspection, seal intact," thus completing data binding.

[0017] S20: Based on the preset image processing algorithm, feature extraction is performed on the on-site image data to obtain image features, and preliminary image verification is performed based on the image features to obtain the first verification result.

[0018] In this embodiment, the verification system employs a target detection and instance segmentation model fine-tuned for power scenarios to extract features from on-site image data. Specifically, it first extracts low-level image features such as significant edges, contour integrity, and regional connectivity using classic algorithms like edge detection and connected component labeling. Then, the model locates key components such as meters, seals, terminal boxes, and warning signs, outputting the bounding box, mask, and preliminary category for each component. Subsequently, based on these image features, the verification system performs a first preliminary image verification. If it detects missing seals, open meter box doors, detached warning signs, or exposed wiring, it generates a first hazard identification result, thus obtaining a first verification result. If it detects meter tilt angles exceeding a threshold or missing meter box fixing screws, it generates a first metering inspection result, along with a preliminary confidence level, thus obtaining a first verification result.

[0019] In this way, visual methods can quickly identify obvious anomalies, effectively utilize the spatial structure and appearance information in images, provide reliable visual evidence for subsequent fusion, and greatly improve efficiency compared to manual visual inspection. For example, in the above photo of the meter box, the image processing algorithm detected that the seal in the lower left corner of the meter box was missing and the warning sign was detached, thus generating the first hazard identification result [missing seal, detached warning sign], and giving an initial confidence level of 0.85.

[0020] S30: Based on a preset text processing algorithm, feature extraction is performed on the associated text data to obtain text features, and a preliminary text check is performed based on the text features to obtain a second check result.

[0021] In this embodiment, the verification system uses a pre-trained language model fine-tuned for the power marketing field, combined with regular rules, to extract features from associated text data, perform named entity recognition and keyword extraction, and obtain key entity features such as meter model, rated voltage and current, asset number, and most recent calibration date. Then, through a preset rule base and semantic similarity calculation, it determines whether the current work order description is consistent with the specification requirements, thereby generating a second verification result.

[0022] In this way, by providing independent judgments on potential risks and measurement issues from the textual dimension through the above process, we can supplement business semantics such as historical records and ledger information that are difficult to obtain directly from images. This can significantly improve the detection capability of information-related defects such as asset mismatch and overdue rotation, and provide textual confidence basis for subsequent cross-modal fusion. For example, based on the retrieved textual data, the verification system identifies the meter in the asset ledger [last verification date was 2019, exceeding the rotation cycle by 1 year], and at the same time, the work order notes [user reported measurement abnormality], thus generating a second verification result [meter overdue and not rotated], and calculating the textual semantic similarity value.

[0023] S40: Input image features and text features into a pre-trained cross-modal fusion model to obtain fused features, and perform cross-modal attribution analysis based on the fused features to obtain the third verification result. Based on the third verification result, conduct risk level assessment to obtain a verification report.

[0024] In this embodiment, the verification system employs a cross-modal fusion model with a dual-tower interactive attention architecture. Specifically, image features are encoded using ViT to obtain an image token sequence, and text features are encoded using a domain-fine-tuned TextEncoder to obtain a text token sequence. These two sequences are concatenated and fed into a multi-layer Cross-Attention module to achieve deep alignment and information complementarity, outputting a unified multi-modal fusion feature. Based on this, the cross-modal attribution analysis module uses a cross-modal version of Grad-CAM to calculate attention weights for image regions and text entities, clarifying which image region and text segment contribute most to the final judgment, thereby generating an interpretable attribution heatmap and text highlighting. Finally, the verification system integrates the preliminary single-modal results with the attribution weights, outputting a third verification result, and based on this, performs a risk level assessment, automatically generating a standardized verification report containing a problem list, attribution explanation, risk level, and rectification suggestions.

[0025] In this way, by deeply integrating image and text features, visual evidence and business constraints can corroborate and explain each other, significantly improving the interpretability and anti-interference ability of verification conclusions. Simultaneously, it achieves closed-loop decision support from feature extraction to attribution to risk assessment, greatly enhancing the intelligence level of on-site verification in power marketing. For example, the fusion model found that the missing seal area in the image and the text "overdue for replacement" both pointed to a risk of electricity theft at the meter. Attribution analysis showed that the image contributed 65% and the text contributed 35%, ultimately outputting the third verification result as "[Seal missing and overdue for replacement, comprehensive risk level A]", and automatically generating a verification report including rectification suggestions "[Replace the seal and meter within 7 days]".

[0026] In this embodiment, optionally, feature extraction is performed on the on-site image data based on a preset image processing algorithm to obtain image features, and a preliminary image verification is performed based on the image features to obtain a first verification result. This includes: using a preset image processing algorithm to extract image features from the on-site image data to obtain the underlying features of the on-site image data, wherein the image processing algorithm includes the Canny edge detection algorithm, the Sobel gradient calculation algorithm, and the improved connected component labeling algorithm, and the underlying features include significant edge features, contour integrity features, and regional connectivity features; using a pre-tuned instance segmentation model for power marketing scenarios to locate key components in the on-site image data, determining multiple key part regions, and outputting the region identification result for each key component region, wherein the region identification result for each key component region includes the bounding box, instance mask, and preliminary category of the corresponding key part region, and the key component regions include at least the meter region, the seal region, the terminal box region, and the warning sign region; and generating a first verification result based on the underlying features and the region identification result for each key component region, wherein the first verification result includes a first hidden danger identification result and / or a first metering inspection result.

[0027] In this embodiment, the verification system employs the Canny edge detection algorithm to calculate the changes in pixel grayscale gradients in the image, extracting significant edge features such as the meter box outline, meter edges, and wiring routing. These features reflect the integrity of the geometric boundaries of the equipment. Simultaneously, the Sobel gradient calculation algorithm is used to obtain the gradient magnitude and direction in the horizontal and vertical directions of the image, thereby obtaining outline integrity features to determine whether key components such as meters and seals are broken, missing, or abnormally bent. Furthermore, an improved connected component labeling algorithm is used to cluster and label regions with similar pixel values ​​in the image, obtaining region connectivity features. These features can identify the connection relationships between components when the meter box door is closed and whether there are any unexpected disconnected areas. These three types of low-level features describe the spatial structure and geometric attributes of the image from different dimensions, providing objective quantitative evidence for subsequent component location and anomaly detection, avoiding the uncertainty caused by directly relying on the black-box output of deep learning models.

[0028] After acquiring the underlying features, the verification system calls a pre-tuned instance segmentation model for the power marketing scenario. For example, it uses a YOLOv8 instance segmentation model trained on numerous on-site photos of metering devices to accurately locate key components in the same on-site image data. This model outputs the bounding box, instance mask, and preliminary category for each key component region. The bounding box identifies the component's location range in the image, the instance mask provides pixel-level segmentation contours, and the preliminary category can be something like "meter," "seal," "terminal box," or "warning sign." Taking the meter region as an example, the instance segmentation model not only provides the rectangular bounding box of the meter but also generates pixel-level masks for specific parts such as the meter dial, casing, and terminals. For the seal region, the instance segmentation model outputs the precise contour of the seal body and its lead seal line. These region identification results are corroborated with the previously extracted underlying features. For instance, if the connected component markers of a seal region show abnormal separation from the meter box door, and the edge detection shows a sudden drop in intensity at that location, it can be preliminarily suspected that the seal is missing or damaged. Ultimately, the verification system will combine the underlying features with the identification results of each key component area to generate the first verification result.

[0029] In this way, by combining classic vision algorithms with domain-specific fine-tuning models, we can ensure the interpretability and computational efficiency of feature extraction, while leveraging deep models to improve the robustness of key component identification in complex scenarios. This can effectively alleviate the false positive and false negative problems of single general-purpose models under conditions such as uneven lighting and occlusion, and provide a reliable visual evidence foundation for subsequent cross-modal fusion.

[0030] In this embodiment of the application, optionally, a first verification result is generated based on the underlying features and the area identification results of each key component area, including: when it is determined that a first type of hidden danger event has occurred based on the underlying features and the area identification results of each key component area, a first hidden danger identification result is generated, and a first verification result is generated based on the first hidden danger identification result, wherein the first type of hidden danger event is missing seal, meter box door not closed, warning label detached or wiring exposed; when it is determined that a second type of hidden danger event has occurred based on the underlying features and the area identification results of each key component area, a first metering inspection result is generated, and a preliminary confidence level of the first metering inspection result is calculated, and the first metering inspection result and the preliminary confidence level are used as the first verification result, wherein the second type of hidden danger event is that the meter tilt angle exceeds a preset angle threshold or the meter box fixing screw is missing.

[0031] In this embodiment, during the process of generating the first verification result based on the underlying features and the identification results of each key component area, the verification system processes the two types of potential incidents separately according to preset anomaly judgment rules. Specifically, for the first type of potential incident, namely, missing seals, open meter box doors, detached warning signs, or exposed wiring, the verification system will make a joint judgment by fusing the region identification results output by the underlying features and the instance segmentation model. For example, if the region connectivity analysis in the underlying features shows an unexpected break in the connection area between the seal body and the meter box shell, and the seal mask area output by the instance segmentation model at this location is significantly smaller than the standard template, such as less than 30% of the normal area, and the edge detection shows a high-intensity gradient change at the break boundary, then it can be determined that the seal is missing. For the open meter box door, the verification system will use contour integrity features to detect whether the edge of the meter box door is connected to the contour of the main body of the box. The system checks for gaps exceeding a preset distance, supplemented by a cross-union ratio (CUI) between the door mask and the enclosure mask output by the instance segmentation model being below a threshold. For detached warning labels, the system locates the area where the label should appear using significant edge features. If no edge group matching the label template is detected within this area, and the connected component markers cannot form the closed area unique to the label, a detachment judgment is generated. For exposed wiring, the system uses Sobel gradient calculation to extract texture abrupt changes between the wire and the insulation layer. Combined with the terminal area mask output by the instance segmentation model, it identifies situations where the edge continuity of the insulation layer-covered area is interrupted and the reflection intensity of the exposed conductor area is significantly higher. Once the system determines that any of the above-mentioned first-type hazard events has occurred, it generates a first hazard identification result. This result is directly used as a component of the first verification result without additional confidence calculation, as these events usually involve clear security or electricity theft risks and require immediate alerts.

[0032] For the second type of potential incident, namely, the meter tilt angle exceeding the preset angle threshold or the meter box fixing screws being missing, the verification system will calculate the preliminary confidence level of the inspection result while generating the first measurement inspection result. Taking meter tilt as an example, the bounding box of the meter area output by the instance segmentation model contains the coordinates of four vertices. The verification system obtains the actual tilt angle by calculating the angle between the horizontal and vertical directions of the bounding box, or by using the tilt angle of the minimum bounding rectangle of the meter area instance mask. When the angle is greater than 15°, the verification system generates the first measurement inspection result of "meter tilt exceeds the standard". The calculation of the preliminary confidence level takes into account both image imaging quality and component visibility integrity. Specifically, on the one hand, it extracts the normalized image edge intensity value within the meter area. The higher the intensity, the clearer the meter outline and the more reliable the angle measurement. On the other hand, it calculates the area coverage of the meter area mask relative to the standard complete template. The higher the coverage, the more reliable the measurement result is. For example, the confidence value between 0 and 1 is obtained by weighting the normalized edge strength and the area coverage. For missing fixing screws of the meter box, the verification system will locate the screw hole area through the instance segmentation model, and then detect whether there is a circular or hexagonal connected region in the area that matches the screw template. If there is no match and the edge of the hole shows a non-smooth gap, it is determined to be missing. The confidence is quantified based on the edge clarity and contrast of the hole area, and the first measurement inspection result and its preliminary confidence are output together as the first verification result.

[0033] For example, during an on-site inspection, the system found that a meter's bounding box tilt angle was 22°, but the lower part of the meter was obscured by tree branches, resulting in an area coverage of only 0.6 and an edge strength of 0.9. After weighted calculation, the confidence level was 0.75, thus generating the first measurement inspection result: "Meter tilt 22°, confidence level 0.75". Simultaneously, the model detected that the screw hole in the lower left corner of the meter box had no corresponding connected component, and the edge strength was 0.85, generating another measurement inspection result: "Screw missing, confidence level 0.85". This classification mechanism allows for direct output of alarms for clearly identified high-risk hazard events without confidence level modification; while quantified measurement anomalies are assigned an objective confidence level, facilitating dynamic weight adjustment based on image quality during subsequent cross-modal fusion. This effectively avoids misjudgments caused by image blur or occlusion, improving the practicality and robustness of the initial inspection results.

[0034] Optionally, generating a first verification result based on the first hazard identification result includes: performing edge detection on the local area where the hazard is located corresponding to the first hazard identification result, calculating the average gradient magnitude of edge pixels in the local area, and normalizing the average gradient magnitude by dividing it by the highest gradient magnitude of all images in the current verification task to obtain a normalized image edge intensity value, and using the first hazard identification result and the normalized image edge intensity value as the first verification result; and / or, obtaining the target instance mask of the hazard target corresponding to the first hazard identification result through a segmentation model, performing pixel-level comparison between the target instance mask and the standardized template mask of the category to which the hazard target belongs, counting the number of overlapping pixels, and calculating the ratio between the number of overlapping pixels and the total number of pixels in the standardized template mask to obtain the image area coverage, and using the first hazard identification result and the image area coverage as the first verification result.

[0035] In this embodiment, the verification system further adds objective imaging quality quantification indicators to the identification result to accurately evaluate the credibility of the image branch in subsequent cross-modal fusion. Specifically, for each first-class hidden danger event identified as missing seal, open meter box door, detached warning sign, or exposed wiring, the verification system first locates the local area of ​​the hidden danger in the image, such as the lower left corner of the meter box corresponding to missing seal, or the original installation position of the sign corresponding to detached warning sign; then, edge detection is performed on the local area, usually using the Canny operator to calculate the gradient magnitude of each pixel in the area, and then calculating the average gradient magnitude of all edge pixels. The average gradient magnitude reflects the clarity of the image of the hidden danger area. The higher the magnitude, the stronger the edge contrast and the clearer the outline; the lower the magnitude, the more likely the image is out of focus, insufficient lighting, or motion blur. In addition, to eliminate the influence of dimensional differences caused by variations in overall brightness and contrast between different images, the verification system divides the average gradient amplitude of the current potential hazard area by the highest gradient amplitude counted from all images taken under the same work order in this verification task, and then normalizes the result to obtain a normalized image edge intensity value ranging from 0 to 1. The closer the normalized image edge intensity value is to 1, the better the imaging quality of the potential hazard area, and the more reliable the subsequent judgment based on that image; conversely, a value closer to 1 indicates poor image quality and requires caution. For example, in one verification, if the system detects a missing seal, extracts the local area, and calculates an average gradient amplitude of 128, while the highest gradient amplitude of a clear meter photo in this task is 160, then the normalized edge intensity value is 0.8; if another photo has an average gradient amplitude of only 32 due to backlighting, then the normalized intensity value is 0.2. The verification system will output the first hazard identification result along with this normalized image edge intensity value as the first verification result.

[0036] Furthermore, in addition to edge strength, the verification system also quantifies the visual integrity of potential hazards by using image region coverage, which is particularly suitable for scenarios where the target is partially occluded. Specifically, it first uses an instance segmentation model to obtain the instance mask of the hazard target corresponding to the first hazard identification result, that is, the pixel-level precise outline occupied by the target in the image. For example, for the hazard of "warning sign detachment", the verification system first obtains a standardized template mask of the area where the warning sign should originally appear. This template is pre-established by the system based on a large number of normal samples and includes the standard shape, size and pixel distribution of the sign. Subsequently, the target instance mask output by the segmentation model in the actual image is compared pixel by pixel with the standardized template mask. The number of overlapping pixels is counted and then divided by the total number of pixels in the standardized template mask to obtain the image region coverage. The image region coverage is between 0 and 1. If the target is completely exposed without any occlusion, the coverage is close to 1; if half of the target is covered by leaves or soil, the coverage is about 0.5; if the target is completely invisible, the coverage is 0. Image region coverage is independent of edge strength and can specifically reflect whether a potential hazard is partially obscured by foreign objects, viewing angle, or environmental factors. For example, in a field inspection, the verification system determines that there is "exposed wiring," but the wiring area is covered by a small piece of mud covering approximately 40% of the area. The overlap between the wiring mask output by the segmentation model and the standardized template mask is 600 pixels, and the total number of pixels in the template is 1000. Therefore, the region coverage is 0.6, and the verification system records this result along with the normalized edge strength value.

[0037] In this way, by simultaneously providing normalized image edge intensity values ​​and image region coverage, the verification system objectively describes image quality from two orthogonal dimensions: clarity and completeness. This enables the subsequent cross-modal fusion to adaptively adjust the weights of image features according to specific imaging conditions. Specifically, when the image is blurred or the target is occluded, the contribution of image branches is automatically reduced, and more reliance is placed on text information. This effectively suppresses misjudgments and omissions caused by interference from the on-site environment, significantly improving the robustness and reliability of power marketing operation verification in complex real-world scenarios.

[0038] In this embodiment of the application, optionally, feature extraction is performed on the associated text data based on a preset text processing algorithm to obtain text features, and a preliminary text check is performed based on the text features to obtain a second check result, including: using a pre-trained language model finely tuned for the power marketing field, combined with regular rules, performing named entity recognition and keyword extraction operations on the work order text, meter nameplate OCR recognition results, and asset ledger in the associated text data to obtain key entities, the key entities including at least the meter model, rated voltage and current, asset number, and most recent calibration date; using a text processing algorithm, semantic similarity calculation is performed on the key entities and the keywords included in the preset rule base to obtain similarity calculation results; based on the similarity calculation results, a second check result is generated, wherein the second check result includes a second hidden danger identification result and / or a second metering inspection result.

[0039] In this embodiment, the process of extracting features from associated text data based on a preset text processing algorithm to obtain a second verification result aims to independently uncover potential risks and measurement anomalies from the text dimension, complementing the image branch. The verification system first uses a pre-trained language model finely tuned for the power marketing field, combined with regularization rules, to perform named entity recognition and keyword extraction operations on the associated text data. The pre-trained language model, fine-tuned for the power marketing field, can be a BERT model further trained on a large amount of text such as power work orders, equipment ledgers, and operating procedures. Named entity recognition is used to automatically extract entities with business meaning from unstructured text, such as [meter model: DDZY102] recorded in the work order text, [asset number: GX123456] in the meter nameplate OCR recognition result, [rated voltage: 220V] and [last inspection date: 2022-03-15] in the asset ledger, etc. Keyword extraction utilizes preset regular expression rules, such as matching words directly related to potential hazards, such as [seal], [detachment], [tilt], [overdue], [not rotated], etc. Through this joint extraction method, the verification system can transform text information scattered from different sources into a structured set of key entities, providing clear input for subsequent consistency comparison and semantic judgment.

[0040] After obtaining the key entities, the verification system further employs text processing algorithms to calculate the semantic similarity between these entities and standard content in a pre-defined rule base. This rule base, pre-constructed by power marketing experts, includes standard model specifications for various equipment, statutory verification cycles, asset numbering rules, and common hidden danger keywords and their synonyms. The semantic similarity calculation uses a domain-adjusted sentence vector model to convert the entities extracted on-site and the standard text in the rule base into high-dimensional vectors, then calculates the cosine similarity between them, resulting in a similarity score between 0 and 1. For example, for meter models, the verification system compares the OCR-recognized "DDZY102-Z" with the standard model "DDZY102" in the rule base using vectorization; if the semantic similarity is high (e.g., 0.95 or higher), a match is considered. For hidden danger keywords, the system semantically matches "[the lead seal is missing]" in the work order with "[the seal is missing]" in the rule base; if the similarity exceeds a pre-defined threshold (e.g., 0.7), a corresponding hidden danger judgment is triggered.

[0041] Based on the similarity calculation results, the verification system will generate a second verification result. Specifically, if it matches potential hazard keywords such as "missing seal," "meter tilt," "exposed wiring," and "overdue for replacement," and the physical comparison reveals that the meter has exceeded its verification validity period, and if the current date minus the most recent verification date is greater than the replacement cycle, the verification system will generate a second hazard identification result as the second verification result. If it finds discrepancies in metering-related information such as inconsistent meter models, asset numbers not matching the ledger, or abnormal rated voltage and current, it will generate a second metering inspection result as the second verification result. For example, during an inspection, the system extracts the message "[User reports that the meter is running fast]" from the work order and identifies "[Last inspection: 2019-05-10]" from the asset ledger. The rule base specifies an inspection cycle of 4 years, currently set for 2024. After calculating semantic similarity, it determines "[Overdue for inspection]". At the same time, the OCR system identifies the model number "[DDZY102]" from the nameplate, which has a semantic similarity of 0.96 with "[DDZY102-Z]" in the ledger. Therefore, it considers the match successful and generates a second potential hazard identification result: "[Meter overdue for inspection]".

[0042] Thus, by combining the domain-fine-tuned language model with regular rules in the above process, we can not only understand the synonymous expressions of power industry terms, but also use rules to accurately match strong pattern information. This can effectively overcome the rigidity problem of pure rule methods and the insufficient interpretability of pure deep learning models, providing preliminary verification conclusions with semantic basis from the text side for subsequent cross-modal fusion, and significantly improving the usability of text information in complex marketing scenarios.

[0043] Optionally, based on the similarity calculation result, a second verification result is generated, including: when the similarity calculation result indicates that the key entity information includes the target keyword or the asset ledger shows that it has expired and has not been rotated, a second hidden danger identification result is generated, and the second hidden danger identification result is used as the second verification result, wherein the target keyword is a keyword used to characterize meter tilting or seal detachment; when the similarity calculation result indicates that the nameplate identification model in the key entity is inconsistent with the model registered in the system, a second measurement inspection result is generated, and the key entity is matched with the standard value in the marketing business master data or asset ledger by precise string matching or fuzzy matching, and the ratio of the number of successfully matched entities to the total number of entities that should be matched included in the key entity is counted to obtain the text entity matching rate, and the second measurement inspection result and the text entity matching rate are used as the second verification result, or the text entity matching rate is converted into text confidence through a monotonically increasing mapping function and the second measurement inspection result and the text confidence are used as the second verification result.

[0044] In this embodiment, when generating the second verification result based on the similarity calculation result, the verification system adopts differentiated output strategies for different judgment scenarios to balance intuitive alarms and refined confidence assessment. Specifically, when the semantic similarity calculation and keyword matching result indicates that the key entity information contains preset target keywords, such as words that directly reflect physical hazards, such as "meter tilt," "seal detachment," and "exposed wiring," or when the "last inspection date" in the asset ledger is found to have exceeded the prescribed rotation cycle, for example, the ledger shows "last inspection date: 2019-03-10," and the rule base stipulates that the inspection cycle is 4 years. If the current date exceeds "2023-03-10," it is determined to be "expired and not rotated." The verification system then directly generates the second hazard identification result to obtain the second verification result. This type of result does not require additional confidence values ​​because the keyword hit or the clear expiration of the ledger itself constitutes highly credible textual evidence and can be directly output as the second verification result to warn grassroots inspectors of the existence of clear hazard risks. For example, the verification system extracts the message "[Customer reported that the seal appears to have been tampered with]" from the work order notes. After semantic similarity matching, the similarity with "[Seal detached]" reaches 0.82, exceeding the threshold of 0.7. At the same time, the asset ledger shows that the meter's last inspection was 6 months overdue, thus generating a second hidden danger identification result: "[Seal suspected of detachment and not replaced overdue]".

[0045] When the similarity calculation result indicates that the nameplate identification model of a key entity is inconsistent with the model registered in the system, the verification system generates a second metrological inspection result and further calculates an objective text confidence score for this result to obtain a second verification result. Specifically, the verification system compares each key entity obtained on-site through OCR recognition or work order reporting, such as asset number, meter model, user number, rated current, voltage level, etc., with the standard values ​​stored in the marketing business master data or asset ledger. The comparison methods include exact string matching that requires complete character consistency and fuzzy matching that considers a match if the edit distance is less than a threshold or some characters are the same, to accommodate individual character recognition errors that may occur during on-site OCR. Subsequently, the verification system counts the number of successfully matched entities and divides it by the total number of entities that should be matched in this verification to obtain the text entity matching rate, which is between 0 and 1. For example, in a verification process, the OCR identifies the asset number as [GX12345A], while the ledger standard is [GX12345B], indicating a mismatch through fuzzy matching; the meter model is identified as [DDZY102], and the ledger also shows [DDZY102], resulting in an exact match; the rated voltage is identified as [220V], and the ledger also shows [220V], resulting in a successful match. Therefore, the total number of entities to be matched is 3, with 2 successful matches, resulting in a text entity matching rate of 0.667. The verification system will directly use this text entity matching rate as the text confidence score, or it will use a monotonically increasing mapping function, such as a sigmoid transform or linear scaling, to smoothly map it to a more reasonable confidence range. Finally, the second measurement inspection result, along with this text confidence score, will be output as the second verification result, for example, outputting "[Asset number mismatch, text confidence score 0.67]".

[0046] Through the above process, the metrological inspection results can be accompanied by an objective quantitative indicator that reflects the overall consistency of the text information. When subsequent cross-modal fusion is performed, the verification system can automatically decide whether to trust the text branch more or rely more on the image branch based on this confidence level. This effectively addresses common on-site issues such as nameplate damage, OCR misidentification, and non-standard work order filling, and significantly improves the robustness and interpretability of metrological inspections in scenarios with incomplete information or noise.

[0047] Optionally, in this embodiment, cross-modal attribution analysis based on fusion features is performed to obtain a third verification result, including: determining a first weight and a second weight; performing weighted fusion of the first hazard identification result in the first verification result with the fusion features according to the first weight and the second weight to obtain a third hazard identification result; and using the third hazard identification result as the third verification result; and / or, if the second verification result includes text confidence, obtaining the image contribution output by the cross-modal fusion model; using a complementary normalization strategy to determine a third weight proportional to the text confidence; determining a fourth weight proportional to the image contribution; and the sum of the third weight and the fourth weight is 1; performing weighted fusion of the second measurement inspection result in the second verification result with the fusion features according to the third weight and the fourth weight to obtain a third measurement inspection result; and using the third measurement inspection result as the third verification result.

[0048] In this embodiment, during the process of obtaining the third verification result through cross-modal attribution analysis based on fusion features, the core task of the verification system is to adaptively weight and fuse the preliminary results of the image branch and the text branch, and to achieve dynamic balance using the modal contribution output by the cross-modal model. Specifically, for the hazard identification scenario, the verification system first needs to determine the first weight and the second weight. The first weight is proportional to the image region coverage carried in the first verification result, and the second weight is proportional to the text contribution directly output by the attention layer of the cross-modal fusion model. The text contribution reflects the relative importance of the associated text to the current hazard judgment. The verification system adopts a complementary normalization strategy, that is, setting the first weight to the ratio of the image region coverage to the sum of the image region coverage and the text contribution, and the second weight to the ratio of the text contribution to the same sum, ensuring that the sum of the two is 1. In practical applications, it's important to note that when the target hazard is severely obscured by leaves, soil, or the camera angle, resulting in low coverage, the first weight automatically decreases, and the verification system will place more trust in the historical records or work order descriptions from the text branch. Conversely, when the target is fully exposed and the coverage is close to 1, the first weight automatically increases, and visual evidence from the image takes precedence. Subsequently, the verification system uses the calculated first and second weights to perform a weighted fusion of the first hazard identification result from the first verification result with the multimodal hazard score after feature decoding, obtaining the final third hazard identification result. For example, in a site image, the sealed area is 40% obscured by tree branches, resulting in an image area coverage of 0.6. The text branch extracts "[seal suspected to be detached]" from the work order, and historical records show that the meter was previously involved in electricity theft, giving the text contribution of 0.8. After complementary normalization, the first weight is assumed to be 0.429, and the second weight is approximately 0.571. After fusion, the system outputs the third hazard identification result, simultaneously indicating the attribution source in the report.

[0049] In metrological inspection scenarios, when the second verification result includes text confidence, such as a quantitative indicator derived from text entity matching rate or semantic similarity, the verification system further integrates image contribution to generate a third metrological inspection result. In this case, the verification system obtains image contribution from the attention mechanism of the cross-modal fusion model. Image contribution reflects the relative importance of image information of visual features such as meter tilt angle and terminal box integrity to the current metrological judgment. The verification system employs a complementary normalization strategy similar to the aforementioned one. Specifically, the third weight is proportional to the text confidence, and the fourth weight is proportional to the image contribution, with the sum of the two being 1. When the text confidence is high, such as a perfect match between the nameplate OCR and the ledger, and an entity matching rate of 0.95, the third weight automatically takes precedence, and the verification system places greater trust in the business consistency judgment on the text side. When the image is clear and key components are fully visible, such as an unobstructed front of the meter and high edge strength, resulting in a higher image contribution, the fourth weight increases accordingly, and the visual measurement result dominates. Based on this, the verification system will perform weighted fusion of the second measurement inspection result in the second verification result with the multimodal measurement score after feature decoding, according to the third and fourth weights, to obtain the third measurement inspection result. For example, in a verification, the nameplate is damaged, causing OCR recognition errors, with a text entity matching rate of only 0.3 and a text confidence of 0.3. However, the cross-modal model identifies the meter model from the image and matches it with the standard template, with an image contribution of 0.9. In this case, the calculated third weight is 0.25 and the fourth weight is 0.75. After fusion, the third measurement inspection result is output, effectively avoiding misjudgments caused by OCR misidentification.

[0050] In this way, through the aforementioned weighted fusion mechanism, the verification system can dynamically adjust the weights of images and text based on their respective credibility in complex on-site environments such as obstruction, damage, and uneven lighting. This achieves intelligent complementarity between image and text branches, which not only significantly improves the accuracy and robustness of the third-party verification results, but also enhances the interpretability of the conclusions by clearly displaying the source of the weights. This allows grassroots inspectors to intuitively understand whether the final judgment mainly relies on visual evidence or business text, thereby increasing their trust in the verification conclusions output by the system.

[0051] In this embodiment of the application, optionally, determining the first weight and the second weight includes: when the first verification result includes the normalized image edge intensity value corresponding to the first hazard identification result and the image region coverage, calculating the image confidence level using the following formula. :

[0052] in, It is an adjustable balance coefficient with a value ranging from 0 to 1. Represents the normalized image edge intensity value. The method represents the image region coverage; obtains the text contribution of the cross-modal fusion model output, and adopts a complementary normalization strategy to make the first weight proportional to the image confidence and the second weight proportional to the text contribution, and the sum of the first weight and the second weight is 1, so as to determine the first weight and the second weight; accordingly, the method also includes: collecting on-site environmental context information of the power marketing operation site, including at least one of light intensity, rain and fog weather indicators, shooting time period, equipment shaking degree and image histogram contrast; determining the environmental correction coefficient matching the on-site environmental context information according to the preset environmental correction rules or lookup table, re-assigning the first weight by multiplying the first weight and the environmental correction coefficient, and adjusting the second weight accordingly, so that the sum of the re-assigned first weight and the adjusted second weight is 1, and using the re-assigned first weight and the adjusted second weight to perform weighted fusion of the first hidden danger identification result and the fusion feature in the first verification result.

[0053] In this embodiment, when determining the first weight and the second weight for weighted fusion, if the first verification result simultaneously includes the normalized image edge intensity value and image region coverage corresponding to the first hazard identification result, the verification system will first use a weighted average formula to calculate the image confidence level. The specific calculation formula is as follows: Formula 1: Formula 1:

[0054] in, This is an adjustable balance coefficient, with a value ranging from 0 to 1; This represents the normalized value obtained by performing Canny edge detection on the target area of ​​the potential hazard, calculating the average gradient magnitude of the effective edge pixels, and dividing it by the global maximum gradient magnitude of the current task, reflecting the edge sharpness. This represents the proportion of the visible region calculated using the pixel-level intersection-union ratio (IUU) of the instance segmentation mask and the normalized template mask, reflecting the completeness of the target. In this embodiment, The default value is 0.5, indicating that edge strength and coverage are equally important; when the scene is dominated by factors such as rain, fog, reflection, and motion blur that cause a decrease in edge strength, it can be adjusted. Increase the confidence level to 0.7-0.9 to make the edge strength more important; when the main cause of reduced coverage at the site is obstruction by leaves, soil, or foreign objects, the confidence level can be adjusted accordingly. The confidence level is lowered to 0.1-0.3, making the coverage rate more important for the confidence score. Using Formula 1, the verification system can obtain an objective image confidence score that is completely independent of the subjective output of the depth model. Its value range is also between 0 and 1.

[0055] Subsequently, the verification system obtains the text contribution directly output by the Cross-Attention layer in the cross-modal fusion model. The text contribution reflects the relative importance of the text information to the current hazard assessment. The verification system uses a complementary normalization strategy to set the first and second weights, making the first weight proportional to the image confidence score and the second weight proportional to the text contribution score. Additionally, when the image is both clear and complete, the image confidence score... When the image confidence score approaches 1, the first weight will automatically take precedence; when the image edges are blurred and the target is severely occluded, the image confidence score will be lowered. If the weighting decreases significantly, the primary weighting will decrease, and the verification system will rely more on the text branches.

[0056] In practical applications, to further adapt to complex and ever-changing on-site environments, the verification system automatically collects on-site environmental context information before weighted fusion, including light intensity, rain / fog weather indicators, shooting time period, equipment shake level, and overall image histogram contrast. Light intensity can be obtained through the mobile device's light sensor or image EXIF ​​data; rain / fog weather indicators can be detected by calling a weather API or by analyzing the degree of haze in the image; the shooting time period can be daytime or nighttime; and equipment shake level can be determined through mobile phone gyroscope data or image motion blur analysis. The verification system can preset a set of environmental correction rules and lookup tables. For example, when the light intensity is below 50 lux, the image quality is considered limited, and the environmental correction coefficient is set to 0.6; when rain / fog weather is detected, the environmental correction coefficient is set to 0.5; when nighttime supplemental lighting is used but the histogram is overexposed, the environmental correction coefficient is set to 0.7; and when the environment is clear, the lighting is uniform, and there is no shake, the environmental correction coefficient is set to 1.2 or remains at 1. In this embodiment, the environmental correction coefficient ranges from 0 to 2. The verification system can match the most suitable environmental correction coefficient based on the currently collected environmental information, and then dynamically adjust the first weight, ensuring that the sum of the two adjusted weights is still 1. It should be noted that if the recalculated first weight exceeds 1, it is truncated to 1, and the second weight is set to 0; if the recalculated first weight is lower than 0, it is set to 0, and the second weight is set to 1. After adjustment, the verification system uses the recalculated first weight and the adjusted second weight to perform a weighted fusion of the first hazard identification result score and the multimodal hazard score after feature decoding in the first verification result, obtaining the final third hazard identification result.

[0057] For example, during an on-site inspection in heavy rain, the inspection system detected a missing seal. Due to the blurred edges caused by rain, the normalized image edge intensity value was calculated to be 0.32; due to foliage obstruction, the calculated image area coverage was 0.45. If the value of is 0.6 (emphasizing edge strength), the calculated image confidence score is 0.372. Assuming that the historical record shows that the meter has repeatedly warned of seal abnormalities, resulting in a text contribution of 0.8 in the cross-modal model output, the original first weight after complementary normalization is approximately 0.317, and the second weight is 0.683. Further, the verification system begins to acquire environmental information. Assuming that the rainy / foggy weather label is confirmed as true, the light intensity is 45 lux, the gyroscope jitter is moderate, and the lookup table gives an environmental correction coefficient of 0.5, the adjusted first weight is 0.1585, and the second weight is 0.8415. During the final fusion, the image branch weight drops to approximately 16%, the text branch weight rises to 84%, and the verification system outputs the third hazard identification result, mainly relying on the work order history.

[0058] In this way, through the above process, in adverse environments such as heavy rain, dense fog, and insufficient nighttime lighting, the verification system can automatically shift trust towards reliable text information, avoiding missed or incorrect judgments due to decreased image quality; while in clear and unobstructed conditions, it restores high trust in the image, achieving all-weather adaptive and stable verification.

[0059] Optionally, a risk level assessment is performed based on the third verification results to obtain a verification report, including: maintaining a power marketing risk assessment rule table, which defines the severity levels corresponding to each type of hidden danger and metering problem; mapping the third hidden danger identification results and third metering inspection results included in the third verification results to the power marketing risk assessment rule table item by item, and determining the initial risk level according to the highest priority principle; using the user number, meter asset number, or meter box geographical location as an index, retrieving historical verification records within a preset time range, and retrieving the number of times the third hidden danger identification results and third metering inspection results appear in the historical verification records and the time interval of the most recent appearance; calculating the aggravation coefficient based on the number of times and the time interval, and adjusting the initial risk level with reference to the aggravation coefficient to obtain the overall risk level; calling a preset report template, combining it with the overall risk level, and generating a verification report, wherein the verification report includes basic work order information, captured images and hidden danger annotation heatmaps, third verification results, cross-modal attribution explanations for each problem, overall risk level and corresponding color codes, preset rectification measures and rectification deadlines, and electronic signatures and timestamps of the inspectors.

[0060] In this embodiment, after obtaining the third verification result, the verification system will enter the risk level assessment and verification report generation stage to achieve a complete process from intelligent identification to closed-loop management. Specifically, firstly, the verification system will maintain a power marketing risk assessment rule table jointly formulated by the power marketing business department and the safety supervision agency. This power marketing risk assessment rule table clearly defines the severity levels of various types of hidden dangers and metering problems. For example, hidden dangers that directly involve electricity theft or personal safety risks, such as missing seals, open meter box doors, and exposed wiring, are classified as the highest level, A; meter tilt exceeding 15°, warning signs falling off, and severe corrosion of the meter box are classified as B; slight deformation of the meter box and blurred but identifiable signs are classified as C; and problems with only slight inconsistencies in asset information that do not affect metering accuracy are classified as the lowest level, D. The verification system will map the third hidden danger identification results and the third measurement inspection results output item by item in the third verification results to the rule table, match the corresponding severity level for each problem, and then determine the initial risk level according to the principle of highest priority. That is to say, as long as there is a level A problem in the current work order, the initial overall risk is set to level A; if there is no level A but there is a level B problem, it is set to level B, and so on.

[0061] However, determining the risk level solely based on a single on-site inspection may overlook the cumulative effect of recurring historical hazards. Therefore, in this embodiment, the verification system further introduces a dynamic correction mechanism based on historical verification records. Specifically, the verification system uses the user ID, meter asset number, or meter box geographical location as a unique index to retrieve all historical verification records of the object within the past 24 months or a preset time period from the marketing business system and the historical verification database, forming a list of historical hazards. Then, for each issue in the current third verification result, the system searches whether the issue has appeared in the historical record, and counts its recurrence count and the time interval between the most recent occurrence and the current occurrence. Based on these two dimensions, the verification system calculates an aggravation coefficient. For example, if the same hazard has occurred 3 or more times in the past 12 months, the aggravation coefficient is 1.5; if the most recent occurrence is less than 3 months ago, an additional 0.3 is added, making the aggravation coefficient 1.8; if it has never occurred in the past, the aggravation coefficient remains unchanged at 1.0. The verification system multiplies the original severity score of each issue by a weighting factor to obtain a weighted severity score. The weighted score can be, for example, 100 points for level A, 70 points for level B, 40 points for level C, and 10 points for level D. The system then remaps the scores to levels A through D, or adjusts the initial risk level based on the rule that if any issue, after weighting, reaches level A, the overall risk level is upgraded. For example, suppose a meter tilt issue initially falls under level B, but historical records show four tilting instances within the past 12 months, with the most recent one occurring only two months ago. The weighting factor would be 1.5 + 0.3 = 1.8, making the weighted severity equivalent to level A, ultimately raising the overall risk level from B to A. This mechanism allows for the automatic identification and increased penalties for long-standing or recurring hazards, achieving intelligent management.

[0062] After determining the overall risk level, the verification system automatically calls the preset standardized report template to generate a complete verification report with one click. The report content includes at least the work order's basic information, the original images taken and a heatmap of hazard markings, a list of all third-party hazard identification results and third-party measurement inspection results, the overall risk level and its corresponding color coding, preset rectification measures and deadlines for each issue, and the inspector's electronic signature and timestamp. Specifically, the work order's basic information can include the user name, meter asset number, and verification time; the hazard marking heatmap is used to visually display the abnormal locations; the list can include cross-modal attribution explanations, such as "[Seal missing: Image area contributes 63%, text history contributes 37%]"; the overall risk level and its corresponding color coding can be A (red), B (orange), C (yellow), and D (green); the rectification measures and deadlines can be in the form of "[Seal missing: Reseal and re-photograph within 7 days, otherwise automatically pushed to the municipal company for supervision]".

[0063] Furthermore, in this embodiment, the report supports exporting to PDF format or direct integration with the marketing business system for archiving. Inspectors can upload the report with one click after on-site confirmation, without having to manually fill out any forms. For example, in a high-risk inspection, the inspection system outputs an overall risk level of red, level A. The report automatically generates items such as "[Seal missing (historically repeated 3 times, aggravation coefficient 1.8, recommended rectification period of 3 days)]" and "[Exposed wiring (first time found, recommended rectification period of 7 days)]", along with an attribution heatmap and textual evidence.

[0064] In this way, the closed-loop process described above completely liberates grassroots inspectors from tedious manual grading and reporting, thereby achieving full-process intelligentization of "identification upon shooting, grading upon identification, and reporting upon grading," significantly improving the efficiency and traceability of closed-loop management of hidden dangers, and providing strong technical support for the management of on-site operations in power marketing.

[0065] The method provided in this application, by simultaneously acquiring on-site images and associated text data and performing preliminary verification separately, can effectively mitigate the risk of false positives and false negatives under interference from lighting and occlusion in a single modality. Furthermore, by inputting image features and text features into a cross-modal fusion model, visual information such as meter installation status and nameplate numbers can interact deeply with text information such as work orders and asset numbers. This allows for the use of visual evidence to locate abnormal areas and the verification of consistency through text semantics in attribution analysis, significantly improving the interpretability of verification conclusions. Based on the risk level assessment of fused features and the automatically generated verification report, it provides closed-loop decision support for grassroots operations, from feature extraction to attribution to risk assessment, greatly improving the intelligence level and reliability of on-site verification in power marketing and ensuring the quality of power services.

[0066] Furthermore, as Figure 1 In a specific implementation of the method, this application provides an image-text fusion cross-modal attribution power marketing operation verification device, such as... Figure 2 As shown, the device includes: an acquisition module 201, an image verification module 202, a text verification module 203, and a fusion verification module 204.

[0067] The acquisition module 201 is used to acquire on-site image data and associated text data of power marketing operations in response to a power marketing operation verification request. The image verification module 202 is used to extract features from the on-site image data based on a preset image processing algorithm to obtain image features, and to perform preliminary image verification based on the image features to obtain a first verification result; The text verification module 203 is used to extract features from the associated text data based on a preset text processing algorithm to obtain text features, and to perform preliminary text verification based on the text features to obtain a second verification result; The fusion verification module 204 is used to input the image features and the text features into a pre-trained cross-modal fusion model to obtain fusion features, and to perform cross-modal attribution analysis based on the fusion features to obtain a third verification result, so as to conduct risk level assessment based on the third verification result and obtain a verification report.

[0068] In a specific application scenario, the image verification module 202 is used to extract image features from the on-site image data using a preset image processing algorithm to obtain the underlying features of the on-site image data. The image processing algorithm includes the Canny edge detection algorithm, the Sobel gradient calculation algorithm, and an improved connected component labeling algorithm. The underlying features include significant edge features, contour integrity features, and region connectivity features. A pre-tuned instance segmentation model, fine-tuned for the power marketing scenario, is used to locate key components in the on-site image data, identifying multiple key component regions and outputting the region identification result for each key component region. The region identification result for each key component region includes the bounding box, instance mask, and preliminary category of the corresponding key component region. The key component regions include at least metering areas, sealing areas, terminal box areas, and warning sign areas. Based on the underlying features and the region identification results for each key component region, the first verification result is generated. The first verification result includes a first hazard identification result and / or a first metering inspection result.

[0069] In a specific application scenario, the image verification module 202 is used to generate a first hidden danger identification result when a first type of hidden danger event is determined to have occurred based on the underlying features and the area identification results of each of the key component areas, and to generate a first verification result based on the first hidden danger identification result, wherein the first type of hidden danger event is missing seal, meter box door not closed, warning label detached or wiring exposed; when a second type of hidden danger event is determined to have occurred based on the underlying features and the area identification results of each of the key component areas, a first metering inspection result is generated, and a preliminary confidence level of the first metering inspection result is calculated, and the first metering inspection result and the preliminary confidence level are used as the first verification result, wherein the second type of hidden danger event is meter tilt angle exceeding a preset angle threshold or meter box fixing screw missing.

[0070] In specific application scenarios, the image verification module 202 is used to perform edge detection on the local area where the hidden danger is located corresponding to the first hidden danger identification result, count the average gradient magnitude of the edge pixels in the local area, divide the average gradient magnitude by the highest gradient magnitude of all images in the current verification task to normalize it, obtain a normalized image edge intensity value, and use the first hidden danger identification result and the normalized image edge intensity value as the first verification result; and / or, obtain the target instance mask of the hidden danger target corresponding to the first hidden danger identification result through the segmentation model, compare the target instance mask with the standardized template mask of the category to which the hidden danger target belongs at the pixel level, count the number of overlapping pixels, and calculate the ratio between the number of overlapping pixels and the total number of pixels of the standardized template mask to obtain the image area coverage, and use the first hidden danger identification result and the image area coverage as the first verification result.

[0071] In specific application scenarios, the text verification module 203 is used to perform named entity recognition and keyword extraction operations on the work order text, meter nameplate OCR recognition results, and asset ledger in the associated text data using a pre-trained language model fine-tuned for the power marketing field, combined with regular rules, to obtain key entities. The key entities include at least the meter model, rated voltage and current, asset number, and most recent calibration date. The text processing algorithm is used to calculate the semantic similarity between the key entities and the keywords included in the preset rule base to obtain the similarity calculation result. Based on the similarity calculation result, the second verification result is generated, wherein the second verification result includes the second hidden danger identification result and / or the second metering inspection result.

[0072] In specific application scenarios, the text verification module 203 is used to generate a second hidden danger identification result when the similarity calculation result indicates that the key entity information includes target keywords or the asset ledger shows that it has expired and has not been rotated, and to use the second hidden danger identification result as the second verification result. The target keywords are keywords used to characterize meter tilting and seal detachment. When the similarity calculation result indicates that the nameplate identification model in the key entity is inconsistent with the model registered in the system, a second measurement inspection result is generated. The key entity is matched with the standard value in the marketing business master data or asset ledger using precise string matching or fuzzy matching. The ratio of the number of successfully matched entities to the total number of entities that should be matched in the key entity is counted to obtain the text entity matching rate. The second measurement inspection result and the text entity matching rate are used as the second verification result. Alternatively, the text entity matching rate can be converted into a text confidence score using a monotonically increasing mapping function, and the second measurement inspection result and the text confidence score are used as the second verification result.

[0073] In specific application scenarios, the fusion verification module 204 is used to determine a first weight and a second weight, and to perform weighted fusion of the first hidden danger identification result in the first verification result with the fusion feature based on the first weight and the second weight to obtain a third hidden danger identification result, and to use the third hidden danger identification result as the third verification result; and / or, when the second verification result includes text confidence, to obtain the image contribution output by the cross-modal fusion model, to use a complementary normalization strategy to determine a third weight proportional to the text confidence, to determine a fourth weight proportional to the image contribution, and the sum of the third weight and the fourth weight is 1, and to perform weighted fusion of the second measurement inspection result in the second verification result with the fusion feature based on the third weight and the fourth weight to obtain a third measurement inspection result, and to use the third measurement inspection result as the third verification result.

[0074] In specific application scenarios, the fusion verification module 204 is used to calculate the image confidence level using the following formula when the first verification result includes the normalized image edge intensity value and image region coverage corresponding to the first hidden danger identification result. :

[0075] in, It is an adjustable balance coefficient with a value ranging from 0 to 1. This represents the normalized image edge intensity value. The image region coverage is represented; the text contribution output by the cross-modal fusion model is obtained, and a complementary normalization strategy is adopted to make the first weight proportional to the image confidence and the second weight proportional to the text contribution, and the sum of the first weight and the second weight is 1, so as to determine the first weight and the second weight. Accordingly, the fusion verification module 204 is also used to collect on-site environmental context information of the power marketing operation site. The on-site environmental context information includes at least one of light intensity, rain and fog weather indicators, shooting time period, equipment vibration degree, and image histogram contrast. According to the preset environmental correction rules or lookup table, an environmental correction coefficient matching the on-site environmental context information is determined. The first weight is re-assigned using the product of the first weight and the environmental correction coefficient, and the second weight is adjusted accordingly so that the sum of the re-assigned first weight and the adjusted second weight is 1. The first hidden danger identification result in the first verification result and the fusion feature are weighted and fused using the re-assigned first weight and the adjusted second weight.

[0076] In a specific application scenario, the fusion verification module 204 is used to maintain a power marketing risk assessment rule table, which defines the severity levels of each type of hidden danger and metering problem. The module maps the third hidden danger identification results and third metering inspection results included in the third verification results to the power marketing risk assessment rule table, determining the initial risk level according to the highest priority principle. Using user ID, meter asset number, or meter box geographical location as an index, it retrieves historical verification records within a preset time range, searching for the frequency of occurrence of the third hidden danger identification results and the third metering inspection results in the historical verification records and the time interval between their most recent occurrences. Based on the frequency and time interval, it calculates an aggravation coefficient, adjusts the initial risk level with reference to the aggravation coefficient, and obtains the overall risk level. It then calls a preset report template, combines it with the overall risk level, and generates the verification report, which includes basic work order information, captured images and a hidden danger annotation heatmap, the third verification results, cross-modal attribution explanations for each problem, the overall risk level and corresponding color coding, preset rectification measures and rectification deadlines, and the electronic signature and timestamp of the inspectors.

[0077] The device provided in this application, by simultaneously acquiring on-site images and associated text data and performing preliminary checks separately, can effectively mitigate the risk of false positives and false negatives under interference from lighting and occlusion in a single modality. Furthermore, by inputting image features and text features into a cross-modal fusion model, visual information such as meter installation status and nameplate numbers can interact deeply with text information such as work orders and asset numbers. This allows for the use of visual evidence to locate abnormal areas and the verification of consistency through text semantics in attribution analysis, significantly improving the interpretability of the verification conclusions. Based on the risk level assessment of the fused features and the automatically generated verification report, it provides closed-loop decision support for grassroots operations, from feature extraction to attribution to risk assessment, greatly improving the intelligence level and reliability of on-site verification in power marketing and ensuring the quality of power services.

[0078] It should be noted that other corresponding descriptions of the functional units involved in the cross-modal attribution power marketing operation verification device for image-text fusion provided in this application embodiment can be found in the following references. Figure 1 The corresponding description in [the document] will not be repeated here.

[0079] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0080] The above embodiments and the technical features in the embodiments can be combined arbitrarily. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0081] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

[0082] In an exemplary embodiment, see Figure 3 The invention also provides a computer device including a bus, a processor, a memory, and a communication interface. It may also include an input / output interface and a display device, wherein the various functional units can communicate with each other via the bus. The memory stores a computer program, and the processor executes the program stored in the memory to perform the image-text fusion cross-modal attribution power marketing operation verification method described in the above embodiments.

[0083] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image-text fusion cross-modal attribution power marketing operation verification method.

[0084] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented in hardware or by using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) and includes several instructions to cause a computer device (such as a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0085] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of a preferred embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing this application.

[0086] Those skilled in the art will understand that the modules in the apparatus of the implementation scenario can be distributed within the apparatus of the implementation scenario as described, or they can be located in one or more apparatuses different from this implementation scenario, with corresponding changes. The modules of the above-described implementation scenario can be combined into one module, or they can be further divided into multiple sub-modules.

[0087] The serial numbers in this application are for descriptive purposes only and do not represent the superiority or inferiority of the implementation scenario.

[0088] The above disclosures are only a few specific implementation scenarios of this application. However, this application is not limited to these. Any variations that can be conceived by those skilled in the art should fall within the protection scope of this application.

Claims

1. A cross-modal attribution method for verifying electricity marketing operations using image-text fusion, characterized in that, include: In response to a request for verification of electricity marketing operations, obtain on-site image data and related text data of the electricity marketing operations; Based on a preset image processing algorithm, feature extraction is performed on the on-site image data to obtain image features, and a preliminary image verification is performed based on the image features to obtain a first verification result; Based on a preset text processing algorithm, feature extraction is performed on the associated text data to obtain text features, and a preliminary text check is performed based on the text features to obtain a second check result; The image features and text features are input into a pre-trained cross-modal fusion model to obtain fused features. Cross-modal attribution analysis is performed based on the fused features to obtain a third verification result. Risk level assessment is then performed based on the third verification result to obtain a verification report.

2. The method according to claim 1, characterized in that, The on-site image data is feature extracted based on a preset image processing algorithm to obtain image features, and a preliminary image verification is performed based on the image features to obtain a first verification result, including: The image processing algorithm is used to extract image features from the scene image data to obtain the low-level features of the scene image data. The image processing algorithm includes the Canny edge detection algorithm, the Sobel gradient calculation algorithm and the improved connected component labeling algorithm. The low-level features include significant edge features, contour integrity features and region connectivity features. Using an instance segmentation model pre-tuned for the power marketing scenario, key components in the field image data are located, multiple key part regions are identified, and the region recognition result of each key component region is output. The region recognition result of each key component region includes the bounding box, instance mask, and preliminary category of the corresponding key part region. The key component regions include at least the meter region, the seal region, the terminal box region, and the warning sign region. Based on the underlying features and the region identification results of each of the key component areas, the first verification result is generated, wherein the first verification result includes the first hidden danger identification result and / or the first measurement inspection result.

3. The method according to claim 2, characterized in that, The generation of the first verification result based on the underlying features and the region identification results of each key component region includes: When a first type of hidden danger event is determined to have occurred based on the underlying features and the area identification results of each of the key component areas, a first hidden danger identification result is generated, and a first verification result is generated based on the first hidden danger identification result. The first type of hidden danger event is a missing seal, an open meter box door, a detached warning label, or exposed wiring. When a second type of hidden danger event is determined to have occurred based on the underlying features and the area identification results of each of the key component areas, a first metering inspection result is generated, and a preliminary confidence level of the first metering inspection result is calculated. The first metering inspection result and the preliminary confidence level are used as the first verification result. The second type of hidden danger event is when the meter tilt angle exceeds a preset angle threshold or when the meter box fixing screws are missing.

4. The method according to claim 3, characterized in that, The step of generating the first verification result based on the first hazard identification result includes: Edge detection is performed on the local area where the hazard is located, corresponding to the first hazard identification result. The average gradient magnitude of the edge pixels within the local area is calculated, and the average gradient magnitude is divided by the highest gradient magnitude of all images in the current verification task for normalization, resulting in a normalized image edge intensity value. The first hazard identification result and the normalized image edge intensity value are then used as the first verification result; and / or, The target instance mask of the hidden danger target corresponding to the first hidden danger identification result is obtained through the segmentation model. The target instance mask is compared with the standardized template mask of the category to which the hidden danger target belongs at the pixel level. The number of overlapping pixels is counted, and the ratio between the number of overlapping pixels and the total number of pixels of the standardized template mask is calculated to obtain the image region coverage. The first hidden danger identification result and the image region coverage are used as the first verification result.

5. The method according to claim 1, characterized in that, The method uses a preset text processing algorithm to extract features from the associated text data, obtains text features, and performs a preliminary text check based on the text features to obtain a second check result, including: Using a pre-trained language model finely tuned for the power marketing field, combined with regular rules, named entity recognition and keyword extraction operations are performed on the work order text, meter nameplate OCR recognition results and asset ledger in the associated text data to obtain key entities. The key entities include at least the meter model, rated voltage and current, asset number and most recent calibration date. Using the aforementioned text processing algorithm, semantic similarity is calculated between the key entity and the keywords included in the preset rule base to obtain the similarity calculation result; Based on the similarity calculation results, the second verification result is generated, wherein the second verification result includes the second hazard identification result and / or the second measurement inspection result.

6. The method according to claim 5, characterized in that, The step of generating the second verification result based on the similarity calculation result includes: When the similarity calculation result indicates that the key entity information includes the target keyword or the asset ledger shows that it has expired and has not been rotated, a second hidden danger identification result is generated, and the second hidden danger identification result is used as the second verification result. The target keyword is a keyword used to characterize meter tilting and seal detachment. When the similarity calculation result indicates that the nameplate identification model in the key entity is inconsistent with the model registered in the system, a second measurement check result is generated. The key entity is matched with the standard value in the marketing business master data or asset ledger using precise string matching or fuzzy matching. The ratio of the number of successfully matched entities to the total number of entities that should be matched in the key entity is counted to obtain the text entity matching rate. The second measurement check result and the text entity matching rate are used as the second verification result, or the text entity matching rate is converted into a text confidence score using a monotonically increasing mapping function, and the second measurement check result and the text confidence score are used as the second verification result.

7. The method according to claim 1, characterized in that, The cross-modal attribution analysis based on the fusion features yields a third verification result, including: Determine a first weight and a second weight; weight and fuse the first hazard identification result in the first verification result with the fusion feature according to the first weight and the second weight to obtain a third hazard identification result; and use the third hazard identification result as the third verification result; and / or, If the second verification result includes text confidence, the image contribution output by the cross-modal fusion model is obtained. A complementary normalization strategy is adopted to determine a third weight that is proportional to the text confidence and a fourth weight that is proportional to the image contribution. The sum of the third weight and the fourth weight is 1. The second measurement check result in the second verification result and the fusion feature are weighted and fused according to the third weight and the fourth weight to obtain a third measurement check result. The third measurement check result is used as the third verification result.

8. The method according to claim 7, characterized in that, The determination of the first weight and the second weight includes: When the first verification result includes the normalized image edge intensity value and image region coverage corresponding to the first hazard identification result, the image confidence level is calculated using the following formula. : in, It is an adjustable balance coefficient with a value ranging from 0 to 1. This represents the normalized image edge intensity value. This indicates the coverage of the image region; The text contribution of the cross-modal fusion model is obtained, and a complementary normalization strategy is adopted to make the first weight proportional to the image confidence and the second weight proportional to the text contribution, and the sum of the first weight and the second weight is 1, so as to determine the first weight and the second weight. Accordingly, the method further includes: Collect on-site environmental context information of power marketing operations, including at least one of light intensity, rain and fog weather indicators, shooting time period, equipment vibration level, and image histogram contrast. According to the preset environmental correction rules or lookup table, an environmental correction coefficient matching the on-site environmental context information is determined. The first weight is reassigned using the product of the first weight and the environmental correction coefficient, and the second weight is adjusted accordingly so that the sum of the reassigned first weight and the adjusted second weight is 1. The reassigned first weight and the adjusted second weight are then used to perform weighted fusion of the first hidden danger identification result in the first verification result and the fusion feature.

9. The method according to claim 1, characterized in that, The risk level assessment based on the third verification result, resulting in a verification report, includes: Maintain the electricity marketing risk assessment rule table, which defines the severity level of each type of hidden danger and the corresponding metering problem. The third hidden danger identification results and the third metering inspection results included in the third verification results are mapped to the electricity marketing risk assessment rule table one by one, and the initial risk level is determined according to the principle of highest priority. Using user ID, meter asset number, or meter box geographical location as an index, retrieve historical verification records within a preset time range, and search for the number of times the third hidden danger identification result and the third metering inspection result appear in the historical verification records and the time interval between the most recent occurrences. Aggravation coefficient is calculated based on the number of times and the time interval, and the initial risk level is adjusted with reference to the aggravation coefficient to obtain the overall risk level; The system calls a preset report template and generates the verification report based on the overall risk level. The verification report includes basic work order information, captured images and a heat map of potential hazards, the third verification results, cross-modal attribution explanations for each issue, the overall risk level and its corresponding color code, preset rectification measures and rectification deadlines, and the electronic signature and timestamp of the inspectors.

10. A cross-modal attribution power marketing operation verification device based on image and text fusion, characterized in that, include: The acquisition module is used to acquire on-site image data and associated text data of power marketing operations in response to power marketing operation verification requests; The image verification module is used to extract features from the on-site image data based on a preset image processing algorithm to obtain image features, and to perform preliminary image verification based on the image features to obtain a first verification result; The text verification module is used to extract features from the associated text data based on a preset text processing algorithm to obtain text features, and to perform preliminary text verification based on the text features to obtain a second verification result. The fusion verification module is used to input the image features and the text features into a pre-trained cross-modal fusion model to obtain fused features, and to perform cross-modal attribution analysis based on the fused features to obtain a third verification result, so as to conduct risk level assessment based on the third verification result and obtain a verification report.

Images