Method and device for identifying defects in power marketing field operation based on multi-modal collaboration, storage medium and equipment

Abstract

The application discloses a multi-modal cooperation-based power marketing field operation defect identification method and device, a storage medium and equipment, relates to the technical field of electric power data processing, and mainly aims to solve the problem of poor identification accuracy of existing power marketing field operation defects. It comprises the following steps: acquiring image data, text data and structured data of power marketing field operation; performing feature extraction on the multi-modal data to obtain image features, text features and structured features, and fusing the image features, text features and structured features to obtain a cross-modal fusion feature vector; and identifying the cross-modal fusion feature vector based on a defect identification model that has completed model training to obtain a defect identification result.

Description

Technical Field

[0001] This application relates to the field of power data processing technology, and in particular to a method, apparatus, storage medium and equipment for identifying defects in power marketing field operations based on multimodal collaboration. Background Technology

[0002] The power grid is shifting from traditional unidirectional power transmission to a new model integrating power generation, grid, load, and storage, with high penetration of distributed energy. Massive amounts of distributed photovoltaic power, residential energy storage, industrial and commercial energy storage, and new energy charging piles are being scattered and connected to the user side. Power flow at the distribution network end is bidirectional. Marketing operations are moving from monthly meter readings and post-event audits to real-time measurement and on-site source-load collaborative management. This has led to an explosive growth in on-site surveys, metering acceptance, electricity consumption audits, and energy efficiency surveys. In particular, the level of intelligence in on-site power marketing operations, such as electricity consumption inspections and metering installation acceptance, directly impacts grid safety and service quality. Therefore, on-site operations typically require monitoring to ensure safety and data validity.

[0003] Currently, the identification of defects in existing power marketing field operations is typically based on manual inspection or model recognition using a single image modality. However, traditional operations rely on manual visual inspection and recording, resulting in low defect identification efficiency, high subjectivity, and a high risk of missed or incorrect defect identification, as well as difficulties in data traceability. Furthermore, single-modal recognition schemes fail to integrate multi-source data such as text and structured data, making them unsuitable for complex operational scenarios and limiting their recognition accuracy. Therefore, there is an urgent need for a multimodal collaborative method for identifying defects in power marketing field operations to address these technical challenges. Summary of the Invention

[0004] In view of this, this application provides a method, device, storage medium and equipment for identifying defects in field operations of power marketing based on multimodal collaboration, with the main purpose of solving the problem of poor accuracy in identifying defects in existing field operations of power marketing.

[0005] According to one aspect of this application, a method for identifying defects in field operations of power marketing based on multimodal collaboration is provided, comprising: Acquire multimodal data of on-site operations in power marketing, including image data, text data, and structured data; Feature extraction is performed on the multimodal data to obtain image features, text features, and structured features. The image features, text features, and structured features are then fused to obtain a cross-modal fused feature vector. The defect identification model, which has already completed model training, is used to identify the cross-modal fusion feature vector to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing domain, and the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model.

[0006] Furthermore, before the defect identification model based on the completed model training identifies the cross-modal fusion feature vector to obtain the defect identification result, the method further includes: Obtain defect identification samples, which include cross-modal fusion feature samples labeled with defect type, defect severity, identification confidence, image spatial location information, physical installation location information, associated equipment information, and defect root cause analysis results; A power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model, and knowledge information from the power marketing field is integrated into the multimodal large model based on knowledge integration technology. The defect identification model is obtained by training the power marketing expert model and the multimodal large model based on the defect identification samples. The multimodal large model is used to provide training prompts based on the knowledge information in the power marketing field to guide the power marketing expert model in defect identification training.

[0007] Furthermore, the prompting information includes image prompting information, text prompting information, and structured prompting information. The step of training the power marketing expert model and the multimodal large model based on the defect identification samples to obtain the defect identification model includes: The multimodal large model is fine-tuned and trained based on the defect identification samples to obtain the image prompt information, the text prompt information, and the structured prompt information. The defect identification samples are then fused based on the image prompt information, the text prompt information, and the structured prompt information. The power marketing expert model is trained based on the fused defect identification samples.

[0008] Furthermore, the method also includes: Obtain user feedback data on the defect identification results and update modal data; The defect identification samples are updated based on the user feedback data and the updated modal data, and the defect identification model is updated and trained based on the incremental training strategy and the updated defect identification samples.

[0009] Further, the step of extracting features from the multimodal data to obtain image features, text features, and structured features, and fusing the image features, text features, and structured features to obtain a cross-modal fused feature vector, includes: The image data is used to extract features based on a visual feature extraction algorithm to obtain the image features; The text data is used to extract features based on a semantic feature extraction algorithm to obtain the text features; The structured data is obtained by extracting features from the structured data based on feature transformation technology; The image features, text features, and structured features are aligned and fused using an attention mechanism, and the fused multimodal features are then processed for feature co-association based on cross-modal attribution analysis. Based on scene graph analysis, spatial and logical relationships are constructed for the multimodal features after association processing, and cross-modal fusion feature vectors are generated.

[0010] Furthermore, before performing feature extraction on the multimodal data to obtain image features, text features, and structured features, the method further includes: The image data is enhanced, denoised, and corrected to obtain standardized image data. The text data is deduplicated, segmented, and semantically analyzed to obtain standardized text data. The structured data is processed to remove missing values ​​and outliers to obtain standardized structured data. Feature extraction is then performed based on multimodal data containing the standardized image data, the standardized text data, and the standardized structured data.

[0011] Furthermore, after the defect identification model based on the completed model training identifies the cross-modal fusion feature vector to obtain the defect identification result, the method further includes: Based on scene image analysis, the spatial location of the defect identification result is identified, and the defect location is determined. Based on the structured data, the correlation between the power grid physical location and the defect location is constructed, and the correlation factors of the defect identification result are determined based on cross-modal attribution analysis. The defect location result is obtained based on the physical location correlation of the power grid and the correlation factors.

[0012] According to another aspect of this application, a device for identifying defects in field operations of power marketing based on multimodal collaboration is provided, comprising: The acquisition module is used to acquire multimodal data of power marketing field operations, including image data, text data, and structured data. The fusion module is used to extract features from the multimodal data to obtain image features, text features, and structured features, and to fuse the image features, text features, and structured features to obtain a cross-modal fusion feature vector. The identification module is used to identify the cross-modal fusion feature vector based on the defect identification model that has been trained, and to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing field, and the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model.

[0013] Furthermore, the device also includes: a construction module and a training module. The acquisition module is also used to acquire defect identification samples, which include cross-modal fusion feature samples labeled with defect type, defect severity, identification confidence, image spatial location information, physical installation location information, associated equipment information, and defect root cause analysis results. The construction module is used to build an expert model for power marketing based on a defect classification expert model and an equipment feature extraction expert model, and to integrate knowledge information in the field of power marketing into the multimodal large model based on knowledge integration technology. The training module is used to train the power marketing expert model and the multimodal large model based on the defect identification samples to obtain a defect identification model. The multimodal large model is used to provide training prompts based on the knowledge information in the power marketing domain to guide the power marketing expert model in defect identification training.

[0014] Furthermore, the training module is specifically used to fine-tune the multimodal large model based on the defect identification samples to obtain the image prompt information, the text prompt information, and the structured prompt information, and to fuse the defect identification samples based on the image prompt information, the text prompt information, and the structured prompt information; and to train the power marketing expert model based on the fused defect identification samples.

[0015] Furthermore, The acquisition module is also used to acquire user feedback data on the defect identification results and update modal data; The update module is further configured to update the defect identification sample based on the user feedback data and the updated modal data, and update and train the defect identification model based on the incremental training strategy and the updated defect identification sample.

[0016] Furthermore, The fusion module is specifically used to extract features from the image data based on a visual feature extraction algorithm to obtain the image features; extract features from the text data based on a semantic feature extraction algorithm to obtain the text features; extract features from the structured data based on feature transformation technology to obtain the structured data; align and fuse the image features, text features, and structured features through an attention mechanism; perform feature collaborative association processing on the fused multimodal features based on cross-modal attribution analysis; and construct spatial and logical relationships on the associated multimodal features based on scene graph analysis to generate a cross-modal fused feature vector.

[0017] Furthermore, the device also includes: The processing module is used to enhance, denoise, and correct the image data to obtain standardized image data; to perform deduplication, word segmentation, and semantic analysis on the text data to obtain standardized text data; and to remove missing and outlier values ​​from the structured data to obtain standardized structured data, so as to perform feature extraction based on multimodal data containing the standardized image data, the standardized text data, and the standardized structured data.

[0018] Furthermore, the device also includes: The determination module is used to identify the spatial location of the defect identification result based on scene image analysis, and determine the defect location; construct the power grid physical location association relationship corresponding to the defect location based on the structured data, and determine the association factors of the defect identification result based on cross-modal attribution analysis; and obtain the defect location result based on the power grid physical location association relationship and the association factors.

[0019] According to another aspect of this application, a storage medium is provided, wherein at least one executable instruction is stored therein, the executable instruction causing a processor to perform an operation corresponding to the above-described method for identifying defects in field operations of power marketing based on multimodal collaboration.

[0020] According to another aspect of this application, a terminal is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other through the communication bus; The memory is used to store at least one executable instruction, which causes the processor to perform the operation corresponding to the above-mentioned method for identifying defects in field operations of power marketing based on multimodal collaboration.

[0021] By employing the above technical solutions, the technical solutions provided in the embodiments of this application have at least the following advantages: This application provides a method, apparatus, storage medium, and device for identifying defects in power marketing field operations based on multimodal collaboration. Compared with existing technologies, the embodiments of this application acquire multimodal data from power marketing field operations, including image data, text data, and structured data; extract features from the multimodal data to obtain image features, text features, and structured features; fuse the image features, text features, and structured features to obtain a cross-modal fused feature vector; and identify the defect based on a defect identification model that has completed model training. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing field, while the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model. This achieves automation and accuracy in defect identification, improves defect identification efficiency, and avoids the high rate of missed and incorrect identifications in single-modal identification. As a result, it achieves a significant improvement in defect identification accuracy and a substantial optimization of processing efficiency. Furthermore, it enables full-dimensional tracing of defective equipment location, physical location, power grid connection, and root cause factors, thereby improving the efficiency of closed-loop management of on-site operations.

[0022] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0023] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a method for identifying defects in field operations of power marketing based on multimodal collaboration, as provided in an embodiment of this application, is shown. Figure 2 This illustration shows a schematic diagram of a model training process provided in an embodiment of this application; Figure 3 This paper illustrates a block diagram of a device for identifying defects in field operations of power marketing based on multimodal collaboration, as provided in an embodiment of this application. Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of this application is shown. Detailed Implementation

[0024] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0027] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0028] Based on this, in one embodiment, the present invention provides a method for identifying defects in on-site operations of power marketing based on multimodal collaboration. Taking the application of this method to computer equipment such as servers as an example, the server can be an independent server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0029] This application provides a method for identifying defects in field operations of power marketing based on multimodal collaboration, such as... Figure 1 As shown, the method includes: 101. Obtain multimodal data of power marketing field operations.

[0030] In this embodiment, the current execution terminal acts as the main body for identifying defects in power marketing field operations. It can associate data with the power marketing field operation system, load cached multimodal data, and also perform real-time input based on front-end devices. This embodiment does not impose specific limitations. The multimodal data includes image data, text data, and structured data. Image data includes, but is not limited to, images of metering devices, electrical equipment, and line layouts taken during field operations. Text data includes, but is not limited to, field operation records, equipment ledger text, and descriptions of historical defects. Structured data includes, but is not limited to, equipment asset numbers, parameter specifications, installation location information, power grid topology association data, historical defect statistics, and power grid GIS system association data.

[0031] In some examples, the metering devices can be electricity meters, meter boxes, or power lines. The corresponding image data can be high-definition images and short videos of the meter and meter box installation scene, covering various indoor and outdoor environments such as strong light and backlighting, to obtain image data of the equipment in normal or defective condition. Operators can collect multimodal data on-site via a mobile app, including capturing high-definition images or short videos of the metering device, power lines, and installation scene, and uploading them to the backend system; the current execution terminal can input or upload text data such as on-site work records and equipment ledger numbers; the current execution terminal automatically associates the structured data corresponding to the work scene, including equipment parameters, installation location, and power grid GIS association information, achieving one-stop collection of multi-source data.

[0032] It should be noted that the on-site operation scenarios of power grid electricity marketing include, but are not limited to, electricity inspection, metering device installation and acceptance, equipment operation and maintenance inspection, etc. For the identification and location of defects in metering devices, such as lost seals, equipment damage, etc., potential electricity hazards, such as aging lines, poor grounding, etc., and defects in metering installation specifications, such as wiring errors, incorrect installation locations, etc., intelligent solutions with multimodal collaboration and domain knowledge enhancement are implemented.

[0033] 102. Extract features from the multimodal data to obtain image features, text features, and structured features, and fuse the image features, text features, and structured features to obtain a cross-modal fused feature vector.

[0034] In this embodiment of the application, for feature extraction of multimodal data, multimodal collaborative feature fusion technology can be used to extract and fuse features of multimodal data. Specifically, corresponding feature extraction methods are used to extract image data, text data, and structured data to obtain corresponding image features, text features, and structured features. Furthermore, cross-modal fusion can be performed through attention mechanisms, cross-modal attribution analysis, scene graph analysis, and other technologies to obtain cross-modal fused feature vectors.

[0035] In another embodiment of this application, for further definition and explanation, the step of extracting features from the multimodal data to obtain image features, text features, and structured features, and fusing the image features, text features, and structured features to obtain a cross-modal fused feature vector, includes: The image data is used to extract features based on a visual feature extraction algorithm to obtain the image features; The text data is used to extract features based on a semantic feature extraction algorithm to obtain the text features; The structured data is obtained by extracting features from the structured data based on feature transformation technology; The image features, text features, and structured features are aligned and fused using an attention mechanism, and the fused multimodal features are then processed for feature co-association based on cross-modal attribution analysis. Based on scene graph analysis, spatial and logical relationships are constructed for the multimodal features after association processing, generating cross-modal fusion feature vectors.

[0036] To achieve feature extraction and fusion of multimodal data, and thus meet the goal of defect identification using fused features, specifically, during feature extraction, for image data, visual feature extraction algorithms are used to extract features from the image data to obtain the image features. Visual feature extraction algorithms include, but are not limited to, classic convolutional neural network feature extractors and Transformer models. For text data, semantic feature extraction algorithms are used to extract features from the text data to obtain the text features. Semantic feature extraction algorithms may include, but are not limited to, Word2Vec and BERT models. For structured data, feature extraction is performed based on feature transformation techniques to obtain the structured data. Feature transformation techniques include, but are not limited to, categorical feature (discrete feature) transformation and numerical feature (continuous feature) transformation.

[0037] It should be noted that after obtaining image features, text features, and structured features, attention mechanisms can be used to align and fuse these features. Then, cross-modal attribution analysis can be used to perform feature collaborative association processing on the fused multimodal features. At this point, cross-modal attribution analysis can employ algorithms such as Multi-SHAP and cross-attention weight tracing to associate the aligned and fused multimodal features. Subsequently, based on scene graph analysis, spatial and logical relationships are constructed on the associated multimodal features to generate cross-modal fused feature vectors.

[0038] In one specific embodiment, visual features such as defect areas and equipment appearance in image data are extracted using a visual feature extraction algorithm; semantic features such as defect descriptions and equipment information in text data are extracted using a semantic feature extraction algorithm; and features such as equipment parameters and location in structured data are extracted using feature transformation technology. After unifying the feature dimensions, an attention mechanism is used to align and fuse the three types of features. Cross-modal attribution analysis is then used to mine the correlations between features, such as associating meter box damage image features with meter box damage records in historical defect descriptions. The cross-modal attribution analysis for mining correlations between features employs a three-level technical implementation: sampling attribution weight calculation, correlation ranking, and feature selection. Specifically, this includes: 1. Attribution Weight Calculation. Based on the gradient-based attribution method, preferably Grad-CAM++, the attribution weights of visual features, semantic features, and structured features on the defect identification results are calculated separately. The higher the weight value, the stronger the correlation between the feature and the defect. Simultaneously, mutual information (MI) calculation is introduced to quantify the mutual information values ​​between different modal features. When the mutual information value is ≥0.6, it is determined to be a strongly correlated feature pair, such as features of a missing seal image or features describing the loss of a seal in text.

[0039] 2. Relevance Ranking. Construct a cross-modal feature correlation matrix with feature type as the row and feature dimension as the column. Fill in the product of attribution weight and mutual information value as the correlation score between features. Use a ranking algorithm, such as bubble sort, to select the top 30% of feature pairs with the highest correlation and remove unrelated or weakly related features.

[0040] 3. Feature Association Validation. Based on rules from the domain knowledge dictionary, the selected feature pairs are validated. For example, the association between the installation height feature in equipment parameters and the installation specification description feature in the text is verified to ensure compliance with business specifications. False association features that do not conform to the specifications are eliminated. Finally, through the fusion of associated feature pairs, the inherent relationships between features of different modalities are uncovered, providing support for feature alignment and fusion.

[0041] In one specific embodiment, spatial and logical relationships between features are constructed based on scene graph analysis, such as the topological relationship between defect locations and equipment installation locations, to generate cross-modal fusion feature vectors. This involves acquiring the correlation relationships, spatial and logical relationships, and generating the cross-modal fusion feature vectors.

[0042] In an example of relationship acquisition, based on the strongly correlated feature pairs selected by cross-modal attribution analysis in the previous step, a multimodal feature scene graph is constructed using scene graph analysis technology. The nodes in the graph represent various features, such as defect features, equipment features, and location features, and the edges represent the relationships between features. The scene graph is trained using a graph neural network (GNN) to learn the association weights between feature nodes. When the weight value is ≥0.7, it is determined to be a stable association. At the same time, domain knowledge is combined to supplement the logical associations between features, such as the subordinate association between defect features and equipment features, and the mapping association between location features and topological features, ultimately forming a complete feature association network.

[0043] In an example of spatial and logical relationship acquisition, spatial relationship acquisition can be based on the defect pixel coordinates in image data and the equipment component segmentation results. It calculates the spatial distance and orientation relationship between defect features and equipment component features. For example, if a defect is located 5mm above the meter terminal, the spatial relationship is quantified into a spatial feature vector using a spatial attention mechanism. Simultaneously, by combining the equipment installation location coordinates from structured data, a spatial coordinate system is constructed, mapping image spatial features to physical spatial features to form a unified spatial relationship description. Logical relationship acquisition can be based on rules in the domain knowledge dictionary and semantic descriptions in text data. It employs logical reasoning algorithms, such as predicate logic reasoning, to mine the logical relationships between features, including causal relationships (e.g., wiring errors and metering deviations), constraint relationships (e.g., installation height <1.5 meters and installation specification defects), and subordinate relationships (e.g., missing seals and metering device defects). These logical relationships are then converted into logical feature vectors and fused with the spatial feature vectors.

[0044] In an example of cross-modal fusion feature vector generation, a process of feature alignment, feature fusion, and dimensionality unification is employed. Feature alignment utilizes an attention mechanism to unify the dimensions of visual, semantic, and structured features to 512 dimensions. Simultaneously, based on a feature association network, features from different modalities are aligned to ensure that features related to the same defect are in adjacent positions in the vector space. Feature fusion employs a weighted fusion strategy, fusing spatial feature vectors, logical feature vectors, and the three basic feature vectors. The fusion weights are dynamically adjusted based on attribution weights; for example, features with higher correlation receive greater weights. Vector optimization involves standardizing the fused feature vectors using a batch normalization (BN) layer to remove redundant dimensions. Principal component analysis (PCA) is then used to compress the feature vectors to 256 dimensions, ultimately generating a cross-modal fusion feature vector that contains the core information of multimodal features and possesses spatial and logical association attributes, meeting the input requirements of the defect recognition model.

[0045] In another embodiment of this application, for further definition and explanation, before the step of extracting features from the multimodal data to obtain image features, text features, and structured features, the method further includes: The image data is enhanced, denoised, and corrected to obtain standardized image data. The text data is deduplicated, segmented, and semantically analyzed to obtain standardized text data. The structured data is processed to remove missing values ​​and outliers to obtain standardized structured data. Feature extraction is then performed based on multimodal data containing the standardized image data, the standardized text data, and the standardized structured data.

[0046] To ensure the effectiveness of images, text, and structured data, thereby improving the accuracy of feature extraction from multimodal data, the current execution end preprocesses the multimodal data. In one specific embodiment, the image data undergoes enhancement, denoising, and correction processing to obtain standardized image data, conforming to a unified format and standard. Enhancement processing may include, but is not limited to, geometric space transformation and pixel color enhancement; denoising processing may include, but is not limited to, mean filtering, median filtering, and frequency domain denoising; and correction processing may include, but is not limited to, geometric distortion correction and color correction. This application embodiment does not impose specific limitations. In one specific embodiment, the text data undergoes deduplication, word segmentation, and semantic analysis processing to obtain standardized text data. The text data may also include speech-based text, i.e., transcribed into standardized text using speech-to-text technology. Deduplication processing may include, but is not limited to, precise deduplication and fuzzy deduplication; word segmentation processing may include, but is not limited to, dictionary-based word segmentation and statistical machine learning-based word segmentation; and semantic analysis processing may include, but is not limited to, sentiment analysis and keyword or entity extraction. This application embodiment does not impose specific limitations. In one specific embodiment, the structured data is processed to remove missing values ​​and outliers to obtain standardized structured data. Missing value processing includes, but is not limited to, deletion, filling or imputation methods, while outlier removal processing includes, but is not limited to, statistical discrimination removal or correction, and model algorithm identification of anomalies, in order to construct a standardized data table. This embodiment of the application does not impose specific limitations.

[0047] 103. Based on the defect recognition model that has completed model training, the cross-modal fusion feature vector is identified to obtain the defect recognition result.

[0048] In this embodiment, after obtaining the cross-modal fusion feature vector, the current execution end identifies the cross-modal fusion feature vector based on the defect identification model that has completed model training, and obtains the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing domain, and it refers to an open-source general-purpose large model with the ability to understand and fuse image, text, and structured data across modalities. For example, it can be constructed based on a general-purpose model type MLLM (Multimodal Large Language Model). This embodiment does not impose specific limitations on this model. Furthermore, the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model. The defect classification expert model can be constructed using a teacher-student distillation architecture, and correspondingly, the equipment feature extraction expert model can be constructed based on an improved LayoutLMv3 model of the Transformer architecture. This embodiment does not impose specific limitations on this model. At this point, a general multimodal large model can be used to perform cross-modal semantic understanding of the fused features, and to make a preliminary judgment on whether the defect exists and its approximate type. Then, based on the defect classification expert model, the defect area in the image is accurately located and its type is subdivided, such as distinguishing between missing seals and damaged boxes. Finally, the equipment feature extraction expert model extracts features such as equipment parameters and installation specifications to help verify whether the defect meets the judgment criteria. The model outputs the defect identification results, including the defect type, severity, and identification confidence.

[0049] In another embodiment of this application, for further definition and explanation, before the step of identifying the cross-modal fusion feature vector based on the defect identification model that has completed model training and obtaining the defect identification result, the method further includes: Obtain defect identification samples; A power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model, and knowledge information from the power marketing field is integrated into the multimodal large model based on knowledge integration technology. Based on the defect identification samples, the power marketing expert model and the multimodal large model are trained to obtain the defect identification model.

[0050] To achieve intelligent defect identification in power marketing field operations, the current execution end pre-trains a defect identification model. Specifically, it first acquires defect identification samples. These samples include cross-modal fusion feature samples labeled with defect type, defect severity, identification confidence, image spatial location information, physical installation location information, associated equipment information, and defect root cause analysis results. These samples can be pre-labeled by machine and then pushed to management personnel for manual verification to update the sample set. This application embodiment does not impose specific limitations.

[0051] Furthermore, an expert model for power marketing and a multimodal large model are constructed. At this point, the expert model for power marketing can be built based on the defect classification expert model and the equipment feature extraction expert model. Simultaneously, based on knowledge integration technology, knowledge information from the power marketing domain is integrated into the multimodal large model to provide training prompts according to this knowledge information, thereby guiding the power marketing expert model in defect identification training. Figure 2 The diagram shows a complete model training process.

[0052] In one specific embodiment, a multimodal large model is used as the base model. This model is an open-source, general-purpose multimodal large model, initially fine-tuned using labeled defect identification samples, and serves as the foundation for the entire defect identification model, providing cross-modal semantic understanding capabilities. The defect classification expert model, one of the domain expert models, is built based on an object detection algorithm. After knowledge distillation and joint training, it is specifically responsible for defect region localization and defect type subdivision. The equipment feature extraction expert model, corresponding to the aforementioned domain expert models in the training process, is built based on a layout analysis algorithm. After knowledge distillation and joint training, it is specifically responsible for extracting key equipment features and verifying defect judgment criteria. The final defect identification model is a collaborative architecture. Here, the individual models are not trained independently. Specifically, the general-purpose multimodal large model as the base model, along with the defect classification expert model, the domain expert model, and the equipment feature extraction expert model, are jointly trained and enhanced with knowledge to collaboratively achieve intelligent defect identification.

[0053] It should be noted that the relationships between the models in this application embodiment can be represented by hierarchical relationships, collaborative relationships, and dependency relationships. Specifically, regarding hierarchical relationships, the general multimodal large model is at the core level, providing cross-modal feature support and basic semantic understanding capabilities for the two domain expert models. The two domain expert models are at the auxiliary enhancement level, focusing on the two sub-tasks of defect classification and equipment feature extraction based on the output of the basic model, thereby enhancing the professional recognition accuracy of the models and compensating for the deficiencies of the basic model in domain knowledge. Specifically, regarding collaborative relationships, the three adopt a collaborative mode of parallel reasoning and interactive feedback. In the defect recognition process, the general multimodal large model first performs preliminary recognition of cross-modal fusion features, outputting the existence and approximate type of defects. Based on the preliminary recognition results of the basic model, the defect classification expert model accurately locates the defect region in the image and subdivides the defect type. The equipment feature extraction expert model extracts features such as equipment parameters and installation specifications, verifies whether the defects meet the domain judgment criteria, and feeds back the verification results to the basic model and the defect classification expert model to correct recognition biases. Ultimately, the three components work together to output accurate defect identification results, achieving a complete identification process from basic understanding to precise segmentation to standardized verification. Specifically, regarding dependencies, the performance of the two domain expert models depends on the cross-modal understanding capability of the base model. Through knowledge distillation technology, the capabilities of the base model are transferred to the domain expert models, ensuring the lightweight and high accuracy of the expert models. The identification accuracy of the base model depends on feedback correction from the domain expert models, and also on the domain knowledge support provided by the device feature extraction expert model. The three are interdependent and complementary, forming a complete collaborative architecture for defect identification.

[0054] In another embodiment of this application, for further definition and explanation, the step of training the power marketing expert model and the multimodal large model based on the defect identification samples to obtain the defect identification model includes: The multimodal large model is fine-tuned and trained based on the defect identification samples to obtain the image prompt information, the text prompt information, and the structured prompt information. The defect identification samples are then fused based on the image prompt information, the text prompt information, and the structured prompt information. The power marketing expert model is trained based on the fused defect identification samples.

[0055] To improve model training accuracy and thus defect identification accuracy, the current execution end first fine-tunes the multimodal large model based on defect identification samples during model training, obtaining image-based, text-based, and structured prompts. Then, it fuses these prompts to train the power marketing expert model. This expert model can include a defect classification expert model based on object detection algorithms and an equipment feature extraction expert model based on layout analysis algorithms, respectively optimized for defect region identification and key equipment feature extraction.

[0056] In one specific embodiment, the equipment feature extraction expert model's specific model structure includes a three-tier architecture of layout analysis, feature extraction, and feature matching. The bottom layer is an improved LayoutLMv3 model based on the Transformer architecture, responsible for layout segmentation and semantic understanding of equipment-related images (such as meter dials, equipment nameplates, and text) and text (such as equipment ledgers). The middle layer is a feature extraction module, composed of a Convolutional Neural Network (CNN) and a Bidirectional Long Short-Term Memory (BiLSTM) network. The CNN is used to extract visual features of the equipment from the images, such as nameplate text and equipment outlines, while the BiLSTM is used to extract semantic features of the equipment from the text, such as equipment model and specifications. The top layer is a feature fusion and matching module, which uses an attention mechanism to fuse visual and semantic features, outputting standardized equipment feature vectors. It also integrates a domain knowledge dictionary to achieve accurate matching between features and standards. Furthermore, the model input is multimodal equipment data, namely images, text, and structured fragments, and the output is key equipment features, such as equipment model, specifications, installation specifications, and component locations.

[0057] In one specific embodiment, to achieve training optimization, the optimization of defect region identification includes embedding a defect region attention mechanism into the model. By labeling defect regions in the data, the trained model focuses on defect-prone areas of the equipment, such as wiring terminals and seal locations, increasing the feature extraction weight of defect regions. Simultaneously, the Non-Maximum Suppression (NMS) algorithm can be introduced to eliminate redundant defect region prediction boxes, improving defect region localization accuracy. Optimization of key equipment feature extraction includes employing a transfer learning strategy to fine-tune the pre-trained LayoutLMv3 model on a power marketing equipment dataset, optimizing the model's text recognition and layout segmentation capabilities. For key parameters in equipment features, such as meter range and line diameter, a specific feature loss function is constructed by combining L1 loss and cosine loss to enhance the model's accuracy in extracting key parameters. Domain knowledge constraints can also be introduced, incorporating content from the equipment feature standard dictionary as regularization terms into model training to avoid excessive deviation between extracted features and the standard deviation. Through a hard example mining strategy, the model's feature extraction capabilities for blurred or occluded equipment, such as nameplates covered in dirt or meters photographed against the light, are trained to improve the model's scene adaptability.

[0058] In one specific embodiment, defect identification samples are fused based on image prompts, text prompts, and structured prompts. The fused defect identification samples are then used to train an electricity marketing expert model. Here, the prompts include image prompts, text prompts, and structured prompts. Each prompt is designed with three types of prompt templates based on multimodal data types, corresponding to image, text, and structured data respectively. All templates use a structure of instructions, knowledge, and examples. The image prompt template is: "Instruction: Identify the defect type of the metering device in the current image; Knowledge: The rule for determining a missing seal is that the metering device seal is missing or damaged, requiring confirmation by associating it with the meter box exterior image; Example: Input image (missing seal) - Output defect type: missing seal." The text prompt template is: "Instruction: Determine if a defect exists and its type based on the on-site operation record text; Knowledge: The business specification requirement for wiring errors is that the three-phase meter wiring must correspond to the phase identifier, with no incorrect or missing connections; Example: Input text 'Three-phase meter A-phase wiring reversed' - Output defect type: wiring error." The structured prompt template is: "Instruction: Based on the equipment parameter data, verify whether the equipment has any installation specification defects; Knowledge: The installation height of the metering device is not less than 1.5 meters; Example: Input parameter "e.g., installation height 1.2 meters" - Output defect type: installation location mismatch"; All prompt templates are associated with specific clauses in the domain knowledge dictionary to ensure the accuracy and relevance of the knowledge.

[0059] In one example of integrating training data, a template embedding and hybrid training approach is used to fuse the prompt templates with the original training samples. First, for each training sample (containing images, text, and structured defect recognition samples), a corresponding prompt template is matched, and the template content is concatenated with the training data. Image data is associated with the template through text labels, while text and structured data are directly concatenated with the template text. Second, the fused training data is divided into training, validation, and test sets in an 8:1:1 ratio. The training set includes both the original training data and the template-fused data, while the validation and test sets use only the original training data, avoiding the template's influence on the model's generalization ability. Finally, during model training, the template-fused data and the original data are alternately input into the model. Gradient descent is used to optimize the model parameters, guiding the model to learn the domain knowledge in the prompt templates while retaining the feature learning ability of the original data. This ensures the model can both master professional rules and adapt to real-world data. Here, the model that guides the learning of defect judgment rules and business specifications through knowledge injection refers to a combination of a general multimodal large model and a device feature extraction expert model. The general multimodal large model receives multimodal training data integrated with prompt templates, achieving cross-modal semantic understanding and initially learning the general logic of defect judgment rules and business specifications. Meanwhile, the device feature extraction expert model focuses on knowledge related to device features, combining it with device feature standards from the prompt templates to strengthen the learning of knowledge such as device parameters and installation specifications. The two work together to ensure that the model can not only grasp the overall defect judgment logic but also accurately extract device-related professional knowledge, achieving comprehensive integration of domain knowledge.

[0060] In a specific embodiment, when using knowledge distillation to transfer the cross-modal understanding capability of the base model to a lightweight expert model, a teacher-student distillation architecture can be adopted. Specifically, the general multimodal large model serves as the teacher model, while the defect classification expert model and the equipment feature extraction expert model serve as student models. The transfer process consists of three steps: 1. Teacher model pre-training: First, the general multimodal large model is fully trained on the labeled dataset to enable it to have mature cross-modal understanding capabilities, such as semantic association between images and text, comprehensive judgment of defect features, and save the trained model parameters and output logs. 2. Distillation Loss Function Settings: Construct a hybrid loss function consisting of hard loss and soft loss. The hard loss is the cross-entropy loss between the student model's prediction and the true label, ensuring the accuracy of the student model's predictions. The soft loss is the KL divergence loss between the student model's output and the teacher model's output, guiding the student model to learn the cross-modal understanding logic of the teacher model. The weight of the KL divergence loss is set to 0.3-0.5 to balance the influence of hard loss and soft loss. 3. Collaborative distillation training: The teacher model is fixed, i.e., its parameters are not updated, while the student model is trained on the labeled dataset. During the training process, the original multimodal data and the output features of the teacher model are input simultaneously. The parameters of the student model are optimized through backpropagation using a hybrid loss function, so that the student model is lightweight while inheriting the cross-modal understanding ability of the teacher model. After training, the student model is pruned and optimized to further reduce the number of model parameters and ensure that it meets the inference speed requirements of the field terminal device.

[0061] In a model training example, a collaborative training approach can be used, which involves jointly training a base model and a domain expert model. The model parameters are iteratively optimized using labeled training samples, enhancing the model's ability to identify specialized scenarios such as defects in metering devices and potential electrical hazards. An incremental training strategy is employed during training to retain core samples and prevent knowledge loss. Specifically, the collaborative training is implemented using a parallel training architecture with interactive feedback, and the specific steps are as follows: 1. Initialization training: The general multimodal large model, the defect classification expert model, and the equipment feature extraction expert model are initialized independently. The basic model is initialized with pre-trained parameters, and the domain expert model is initialized with random parameters. 2. Parallel forward propagation: Training samples are simultaneously input into three models. The base model outputs cross-modal fusion features and preliminary defect identification results, the defect classification expert model outputs defect region localization and type subdivision results, and the equipment feature extraction expert model outputs key equipment features and specification verification results. 3. Interactive Feedback and Loss Calculation: An interactive feedback mechanism is constructed to input the fused features of the base model into two domain expert models, assisting the expert models in optimizing feature extraction accuracy. Simultaneously, the outputs of the two expert models are fed back to the base model to correct cross-modal understanding biases. The joint loss function of the three models is calculated as follows: Joint Loss = Base Model Loss (Cross-entropy Loss) + 0.4 × Defect Classification Expert Model Loss (Focal Loss) + 0.4 × Equipment Feature Extraction Expert Model Loss (Mixed Loss) + 0.2 × Interactive Feedback Loss (Mean Squared Error Loss). The interactive feedback loss measures the consistency of outputs between the models. 4. Parameter Co-update: The joint loss is synchronously transmitted to the three models through the backpropagation algorithm, and the parameters of the three models are updated at the same time. The learning rate of the base model is set to 1e-5, and the learning rate of the domain expert model is set to 5e-5 to ensure that the expert model can quickly adapt to the domain data. 5. Iterative optimization: Repeat steps 1-4, performing validation every 10 epochs. Adjust the weight coefficients of the joint loss based on the recognition accuracy of the validation set, remove redundant samples, and add difficult samples until the model converges. Model convergence means that the validation set accuracy has not significantly improved for 5 consecutive epochs, and joint training is complete.

[0062] In another embodiment of this application, for further definition and explanation, the steps also include: Obtain user feedback data on the defect identification results and update modal data; The defect identification samples are updated based on the user feedback data and the updated modal data, and the defect identification model is updated and trained based on the incremental training strategy and the updated defect identification samples.

[0063] To ensure the iterative learning capability of the defect identification model and thus improve the accuracy of repeated defect identification, the defect identification results, including defect type, severity, confidence level, equipment location, physical location, and root cause analysis, are presented in a combination of text and graphics. Furthermore, the defect identification results are stored in the backend database in both structured tables and a common data exchange format, containing core information such as work order number, equipment asset number, defect type, severity, identification confidence level, equipment location, physical location, grid connection node, root cause analysis, data collection time, and identification time. The defect identification results can also be automatically synchronized to the power marketing field operation management system and the business expansion work order archiving and verification system, triggering the creation of defect rectification work orders and achieving a closed-loop business process of identification, location, rectification, and archiving. This application's embodiments do not impose specific limitations.

[0064] It should be noted that the current execution end outputs the defect identification results to the user after receiving them, in order to obtain user feedback data, namely, the identification errors corrected by the operators, newly added multimodal data of on-site operations, and updates the modal data, that is, updating the labeled dataset every quarter to supplement newly added defect type samples and error correction samples. Then, an incremental training strategy is used to fine-tune the defect identification model, update the judgment rules in the domain knowledge dictionary, continuously optimize the model's adaptability to new scenarios and defect types, and finally, retain the optimal model version and automatically deploy updates through the backend system to ensure continuous improvement in model performance.

[0065] In another embodiment of this application, for further definition and explanation, after the step of identifying the cross-modal fusion feature vector based on the defect identification model that has completed model training and obtaining the defect identification result, the method further includes: Based on scene image analysis, the spatial location of the defect identification result is identified, and the defect location is determined. Based on the structured data, the correlation between the power grid physical location and the defect location is constructed, and the correlation factors of the defect identification result are determined based on cross-modal attribution analysis. The defect location result is obtained based on the physical location correlation of the power grid and the correlation factors.

[0066] To accurately locate defects from the defect identification results and meet the localization requirements, the current execution end first identifies the spatial location of the defect based on scene image analysis. This involves identifying the spatial coordinates of the defect in the image using scene image analysis technology, clarifying the specific location of the defect relative to the equipment body, such as the meter terminal block or the meter box lock location (this embodiment does not specify a particular location). Simultaneously, a power grid physical location association relationship corresponding to the defect location can be constructed based on the structured data. Furthermore, cross-modal attribution analysis is used to determine the association factors of the defect identification results. This involves automatically associating the equipment asset number and installation location information in the structured data, combined with power grid GIS data, to determine the physical address of the defect, such as "meter box in room xxx, building x, XX community," and the association relationship between the power grid topology nodes.

[0067] In a specific embodiment, when determining the correlation factors of the defect identification results based on cross-modal attribution analysis, the root cause of the defect is traced by combining work records in text data and equipment parameters in structured data. For example, wiring errors may originate from failure to follow phase identification during installation, or meter box damage may be due to external impact. The method for tracing the root cause of defects through cross-modal attribution analysis includes a three-step approach: multimodal feature tracing, attribution weight ranking, and root cause verification, as detailed below: Multimodal feature tracing: Based on cross-modal fusion feature vectors, the original features of each modality are traced back, such as defect features in images, job record features in text, and equipment parameter features in structured data. The contribution of each original feature to the defect identification result is calculated through gradient attribution algorithm. The higher the contribution, the stronger the correlation between the feature and the root cause of the defect. Attribution weight ranking: Construct a defect root cause attribution matrix with defect type as the row and original feature as the column, fill in the contribution of each feature, i.e., the attribution weight, and use a ranking algorithm to select the top 50% core features as candidate factors for defect root causes. For example, candidate factors for wiring errors include the wiring operation description in the work record, the phase identifier in the equipment parameters, and the wiring position in the image. Root cause verification and determination: Combining defect judgment rules and business specifications in the domain knowledge dictionary, candidate factors are logically verified. Bayesian inference algorithms are used to calculate the probability of each candidate factor causing the defect. The factor with the highest probability and that conforms to the business logic is the root cause of the defect. At the same time, the root causes of similar defects are compared with the historical defect case library to verify the rationality of the current root cause. Finally, the root cause of the defect and the corresponding related evidence, such as screenshots of work records, equipment parameter data, and image features, are output to achieve accurate tracing of the root cause of the defect and finally output the defect location result.

[0068] In the domain knowledge dictionary construction example of this application embodiment, the construction content includes defect judgment rules, business specification requirements, and equipment characteristic standards. The defect judgment rules may include, but are not limited to, missing seals (referring to missing or damaged seals on metering devices, requiring confirmation by comparing the meter box appearance image with the equipment ledger), business specification requirements (including, but not limited to, the metering device installation location being away from humid and high-temperature environments, and a height of not less than 1.5 meters above the ground), and equipment characteristic standards (including, but not limited to, three-phase meter wiring corresponding to phase identifiers, with no incorrect or missing connections). The knowledge dictionary is converted into prompt templates for model knowledge injection. Specifically, the methods for constructing the defect judgment rules, business specification requirements, and equipment characteristic standards are as follows: 1. Defect Judgment Rule Construction: A three-dimensional approach of literature review, on-site investigation, and expert review was adopted. First, industry standards and existing literature, such as the "Electricity Marketing On-site Operation Specifications" and the "Metering Device Defect Judgment Standards," were reviewed to initially screen defect types and corresponding judgment conditions. Second, on-site operation experts and metering technology experts were organized to analyze defect cases collected from various sites one by one, clarifying the core judgment indicators for each type of defect. For example, the core indicators for lost seals are missing / damaged seal images and mismatched seal numbers in equipment ledgers, forming initial judgment rules. Finally, the initial rules were verified and revised through expert review meetings, eliminating redundant and ambiguous clauses and supplementing judgment details for special scenarios, such as adjusting defect judgment thresholds under severe weather conditions. Ultimately, a structured defect judgment rule library was formed, with each rule containing a defect code, defect name, core judgment indicators, and associated data types, including image, text, structured data, and judgment threshold fields.

[0069] 2. Construction of Business Standard Requirements: Based on relevant standards for power grid marketing field operations, such as the "Installation Specifications for Metering Devices" and the "Electricity Inspection Management Measures," the core standard requirements in the business process are broken down and a standard system is constructed according to the operational stages, namely installation, inspection, and acceptance, as well as the hierarchy of equipment type, standard clauses, and assessment indicators. For high-frequency scenarios in field operations, practical standard details are supplemented based on feedback from frontline operators, such as specific spacing requirements for installation locations, inspection frequency and focus, etc. At the same time, standard indicators are quantified, such as installation height ≥ 1.5 meters and wiring error ≤ ±5%, to ensure that the standard requirements are quantifiable and implementable, ultimately forming a structured business standard dictionary.

[0070] 3. Equipment Feature Standard Construction: Collect factory technical parameters and industry standards for various power marketing equipment, such as meters, meter boxes, and lines. Extract core equipment features, such as meter phase identification, meter box dimensions, and line wire diameter parameters, to construct an equipment feature library. For different equipment types, define the standard range of features; for example, for three-phase meters, phase identification is A / B / C, and wire diameter is 10mm² / 16mm². Simultaneously, link the feature fields from equipment ledger data to ensure consistency between feature standards and actual field equipment parameters. Through review by equipment manufacturer technicians, correct any deviations in the feature standards, ultimately forming a structured equipment feature standard dictionary, achieving standardized and regulated descriptions of equipment features.

[0071] This application provides a method for identifying defects in power marketing field operations based on multimodal collaboration. Compared with existing technologies, this application acquires multimodal data from power marketing field operations, including image data, text data, and structured data. Feature extraction is performed on the multimodal data to obtain image features, text features, and structured features. These features are then fused to obtain a cross-modal fused feature vector. Finally, a defect identification model that has undergone model training is used to identify the cross-modal fused feature vector to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing field, while the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model. This achieves automation and accuracy in defect identification, improves defect identification efficiency, and avoids the high rate of missed and incorrect identification in single-modal identification. As a result, it achieves a significant improvement in defect identification accuracy and a substantial optimization of processing efficiency. Furthermore, it enables full-dimensional tracing of defective equipment location, physical location, power grid connection, and root cause factors, thereby improving the efficiency of closed-loop management of on-site operations.

[0072] Furthermore, as a response to the above Figure 1The implementation of the method shown in this application provides a device for identifying defects in on-site operations of power marketing based on multimodal collaboration, such as... Figure 3 As shown, the device includes: The acquisition module 21 is used to acquire multimodal data of power marketing field operations, including image data, text data and structured data; The fusion module 22 is used to extract features from the multimodal data to obtain image features, text features and structured features, and to fuse the image features, text features and structured features to obtain a cross-modal fusion feature vector. The identification module 23 is used to identify the cross-modal fusion feature vector based on the defect identification model that has been trained, and to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing field, and the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model.

[0073] Furthermore, the device also includes: a construction module and a training module. The acquisition module is also used to acquire defect identification samples, which include cross-modal fusion feature samples labeled with defect type, defect severity, identification confidence, image spatial location information, physical installation location information, associated equipment information, and defect root cause analysis results. The construction module is used to build an expert model for power marketing based on a defect classification expert model and an equipment feature extraction expert model, and to integrate knowledge information in the field of power marketing into the multimodal large model based on knowledge integration technology. The training module is used to train the power marketing expert model and the multimodal large model based on the defect identification samples to obtain a defect identification model. The multimodal large model is used to provide training prompts based on the knowledge information in the power marketing domain to guide the power marketing expert model in defect identification training.

[0074] Furthermore, the training module is specifically used to fine-tune the multimodal large model based on the defect identification samples to obtain the image prompt information, the text prompt information, and the structured prompt information, and to fuse the defect identification samples based on the image prompt information, the text prompt information, and the structured prompt information; and to train the power marketing expert model based on the fused defect identification samples.

[0075] Furthermore, The acquisition module is also used to acquire user feedback data on the defect identification results and update modal data; The update module is further configured to update the defect identification sample based on the user feedback data and the updated modal data, and update and train the defect identification model based on the incremental training strategy and the updated defect identification sample.

[0076] Furthermore, The fusion module is specifically used to extract features from the image data based on a visual feature extraction algorithm to obtain the image features; extract features from the text data based on a semantic feature extraction algorithm to obtain the text features; extract features from the structured data based on feature transformation technology to obtain the structured data; align and fuse the image features, text features, and structured features through an attention mechanism; perform feature collaborative association processing on the fused multimodal features based on cross-modal attribution analysis; and construct spatial and logical relationships on the associated multimodal features based on scene graph analysis to generate a cross-modal fused feature vector.

[0077] Furthermore, the device also includes: The processing module is used to enhance, denoise, and correct the image data to obtain standardized image data; to perform deduplication, word segmentation, and semantic analysis on the text data to obtain standardized text data; and to remove missing and outlier values ​​from the structured data to obtain standardized structured data, so as to perform feature extraction based on multimodal data containing the standardized image data, the standardized text data, and the standardized structured data.

[0078] Furthermore, the device also includes: The determination module is used to identify the spatial location of the defect identification result based on scene image analysis, and determine the defect location; construct the power grid physical location association relationship corresponding to the defect location based on the structured data, and determine the association factors of the defect identification result based on cross-modal attribution analysis; and obtain the defect location result based on the power grid physical location association relationship and the association factors.

[0079] This application provides a device for identifying defects in power marketing field operations based on multimodal collaboration. Compared with existing technologies, this application acquires multimodal data from power marketing field operations, including image data, text data, and structured data; extracts features from the multimodal data to obtain image features, text features, and structured features; fuses the image features, text features, and structured features to obtain a cross-modal fused feature vector; and identifies the cross-modal fused feature vector based on a defect identification model that has completed model training to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing field, while the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model. This achieves automation and accuracy in defect identification, improves defect identification efficiency, and avoids the high rate of missed and incorrect identification in single-modal identification. As a result, it achieves a significant improvement in defect identification accuracy and a substantial optimization of processing efficiency. Furthermore, it enables full-dimensional tracing of defective equipment location, physical location, power grid connection, and root cause factors, thereby improving the efficiency of closed-loop management of on-site operations.

[0080] According to one embodiment of this application, a storage medium is provided, the storage medium storing at least one executable instruction, the computer-executable instruction being able to execute the method for identifying defects in field operations of power marketing based on multimodal collaboration in any of the above method embodiments.

[0081] Figure 4 The diagram shows a structural schematic of a terminal according to one embodiment of the present application. The specific embodiments of the present application do not limit the specific implementation of the terminal.

[0082] like Figure 4 As shown, the terminal may include: a processor 302, a communications interface 304, a memory 306, and a communications bus 308.

[0083] The processor 302, communication interface 304, and memory 306 communicate with each other via communication bus 308.

[0084] Communication interface 304 is used to communicate with other network elements such as clients or other servers.

[0085] The processor 302 is used to execute program 310, which can specifically execute the relevant steps in the above embodiment of the method for identifying defects in field operations of power marketing based on multimodal collaboration.

[0086] Specifically, program 310 may include program code that includes computer operation instructions.

[0087] Processor 302 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The terminal includes one or more processors, which may be processors of the same type, such as one or more CPUs; or they may be processors of different types, such as one or more CPUs and one or more ASICs.

[0088] Memory 306 is used to store program 310. Memory 306 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0089] Specifically, program 310 can be used to cause processor 302 to perform the following operations: Acquire multimodal data of on-site operations in power marketing, including image data, text data, and structured data; Feature extraction is performed on the multimodal data to obtain image features, text features, and structured features. The image features, text features, and structured features are then fused to obtain a cross-modal fused feature vector. The defect identification model, which has already completed model training, is used to identify the cross-modal fusion feature vector to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing domain, and the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model.

[0090] Obviously, those skilled in the art should understand that the modules or steps of this application described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented here, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, this application is not limited to any particular combination of hardware and software.

[0091] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for identifying defects in field operations of power marketing based on multimodal collaboration, characterized in that, include: Acquire multimodal data of on-site operations in power marketing, including image data, text data, and structured data; Feature extraction is performed on the multimodal data to obtain image features, text features, and structured features. The image features, text features, and structured features are then fused to obtain a cross-modal fused feature vector. The defect identification model, which has already completed model training, is used to identify the cross-modal fusion feature vector to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing domain, and the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model.

2. The method according to claim 1, characterized in that, Before the defect identification model, which has already undergone model training, identifies the cross-modal fusion feature vector to obtain the defect identification result, the method further includes: Obtain defect identification samples, which include cross-modal fusion feature samples labeled with defect type, defect severity, identification confidence, image spatial location information, physical installation location information, associated equipment information, and defect root cause analysis results; A power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model, and knowledge information from the power marketing field is integrated into the multimodal large model based on knowledge integration technology. The defect identification model is obtained by training the power marketing expert model and the multimodal large model based on the defect identification samples. The multimodal large model is used to provide training prompts based on the knowledge information in the power marketing field to guide the power marketing expert model in defect identification training.

3. The method according to claim 2, characterized in that, The prompt information includes image prompt information, text prompt information, and structured prompt information. The step of training the power marketing expert model and the multimodal large model based on the defect identification samples to obtain the defect identification model includes: The multimodal large model is fine-tuned and trained based on the defect identification samples to obtain the image prompt information, the text prompt information, and the structured prompt information. The defect identification samples are then fused based on the image prompt information, the text prompt information, and the structured prompt information. The power marketing expert model is trained based on the fused defect identification samples.

4. The method according to claim 2, characterized in that, The method further includes: Obtain user feedback data on the defect identification results and update modal data; The defect identification samples are updated based on the user feedback data and the updated modal data, and the defect identification model is updated and trained based on the incremental training strategy and the updated defect identification samples.

5. The method according to claim 1, characterized in that, The step of extracting features from the multimodal data to obtain image features, text features, and structured features, and then fusing the image features, text features, and structured features to obtain a cross-modal fused feature vector, includes: The image data is used to extract features based on a visual feature extraction algorithm to obtain the image features; The text data is used to extract features based on a semantic feature extraction algorithm to obtain the text features; The structured data is obtained by extracting features from the structured data based on feature transformation technology; The image features, text features, and structured features are aligned and fused using an attention mechanism, and the fused multimodal features are then processed for feature co-association based on cross-modal attribution analysis. Based on scene graph analysis, spatial and logical relationships are constructed for the multimodal features after association processing, generating cross-modal fusion feature vectors.

6. The method according to claim 1, characterized in that, Before performing feature extraction on the multimodal data to obtain image features, text features, and structured features, the method further includes: The image data is enhanced, denoised, and corrected to obtain standardized image data. The text data is deduplicated, segmented, and semantically analyzed to obtain standardized text data. The structured data is processed to remove missing values ​​and outliers to obtain standardized structured data. Feature extraction is then performed based on multimodal data containing the standardized image data, the standardized text data, and the standardized structured data.

7. The method according to claim 1, characterized in that, After the defect identification model based on the completed model training identifies the cross-modal fusion feature vector and obtains the defect identification result, the method further includes: Based on scene image analysis, the spatial location of the defect identification result is identified, and the defect location is determined. Based on the structured data, the correlation between the power grid physical location and the defect location is constructed, and the correlation factors of the defect identification result are determined based on cross-modal attribution analysis. The defect location result is obtained based on the physical location correlation of the power grid and the correlation factors.

8. A device for identifying defects in on-site operations of power marketing based on multimodal collaboration, characterized in that, include: The acquisition module is used to acquire multimodal data of power marketing field operations, including image data, text data, and structured data. The fusion module is used to extract features from the multimodal data to obtain image features, text features, and structured features, and to fuse the image features, text features, and structured features to obtain a cross-modal fusion feature vector. The identification module is used to identify the cross-modal fusion feature vector based on the defect identification model that has been trained, and to obtain the defect identification result. The defect identification model is constructed by combining a multimodal large model and a power marketing expert model. The multimodal large model incorporates knowledge information from the power marketing field, and the power marketing expert model is constructed based on a defect classification expert model and an equipment feature extraction expert model.

9. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method of claim 1.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method of claim 1.

Images