Electric power image-text cross-modal alignment method and system, storage medium and equipment
By using probabilistic feature alignment and fusion-based distillation, the CLIP model in the power field was retrained, which solved the modal gap problem, improved the accuracy of power equipment identification and fault diagnosis, and enhanced its reliability and generalization ability in complex environments.
Patent Information
- Application Number
- CN202610125655.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-29
- Publication Date
- 2026-05-19
AI Technical Summary
The CLIP model has poor performance in the power sector due to modal gaps, particularly in areas such as fine-grained defect identification, terminology alignment, and reliability in complex environments.
We employ a probabilistic feature alignment and fusion-based distillation method to retrain the CLIP model in the power field. By sharing prior distributions and random reference vectors, we optimize the feature distribution and generate hybrid soft labels by combining real text and images with the output of the pre-trained model. We then construct an objective function to update the model.
It significantly improves the accuracy of intelligent inspection of power equipment, the accuracy of fault report archiving, and the real-time performance and reliability of operation safety monitoring, while maintaining the model's zero-sample generalization capability.
Smart Images
Figure CN122067040A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of cross-modal learning technology in computer vision and natural language processing, specifically to a method, system, storage medium, and device for cross-modal alignment of power graphs and text. Background Technology
[0002] Contrastive Language Image Pretraining (CLIP) is a crucial component in building modern visual language foundational models. While CLIP demonstrates superior zero-shot performance on downstream tasks, its multimodal feature space suffers from a modality gap problem—the discrepancy between image and text feature clusters—which limits its performance on downstream tasks. Existing work attempts to address this modality gap by modifying the pretraining or fine-tuning process; however, these methods often face challenges such as high training costs on large datasets or degradation in zero-shot performance.
[0003] In intelligent operation and maintenance and safety management in the power sector, accurate cross-modal alignment of visual and linguistic data has crucial application value. However, the inherent modal gap problem of the CLIP model directly restricts its practical application effectiveness. Specifically:
[0004] 1. In intelligent inspection scenarios for power equipment, the system needs to locate corresponding defects in inspection images based on text descriptions (such as "damaged insulator" or "foreign object hanging on the conductor"). However, due to modal gaps causing insufficient alignment between image and text features, the model lacks the ability to distinguish fine-grained features of power equipment components. For example, it is difficult to accurately distinguish the subtle visual differences between "damaged insulator" and "dirty insulator," or between "normal clamp" and "overheated clamp," resulting in missed or false detections and affecting inspection efficiency and accuracy.
[0005] 2. In scenarios involving the automatic generation and archiving of fault diagnosis reports, it is necessary to associate fault images taken on-site with professional text descriptions. Modal gaps cause inaccurate alignment between power industry terminology and visual features in the model. For example, it may be unable to correctly associate the text description of "surge arrester counter abnormality" with the specific visual pattern of the counter pointer position, resulting in mismatch between fault images and report text, affecting the speed of fault handling and the quality of knowledge base construction.
[0006] 3. In on-site safety monitoring scenarios, violations need to be identified in real time using natural language commands (such as "not wearing a safety helmet" or "crossing the warning line"). Modal gaps reduce the reliability of text-image association in complex environments. For example, in environments with variable lighting and cluttered backgrounds, the model may not be able to reliably match the textual concept of "working at height without a safety belt" with the status of personnel and equipment in the image, increasing safety risks.
[0007] To address the modal gap problem in CLIP, existing literature primarily focuses on optimizing the CLIP feature space during the pre-training or fine-tuning stages. In the pre-training stage, methods include multi-task learning using contrastive targets and auxiliary losses, or modifying the encoder architecture to share weights. In the fine-tuning stage, "cue word tuning" improves task-specific performance by optimizing trainable labels to match the feature space.
[0008] While these methods improve cross-modal alignment capabilities to some extent, they face challenges in terms of computational cost and zero-shot transfer performance, particularly in adapting to the application characteristics of the power sector. On the one hand, pre-training methods require massive datasets and enormous computing power, resulting in high costs, and publicly available pre-trained models (such as OpenAI CLIP) are not optimized for the power sector. On the other hand, while fine-tuning methods have lower computational costs, they compromise the model's general zero-shot capability. However, the scale of available domain image and text data for training in the power sector is limited, and the tasks are diverse (equipment identification, defect classification, safety compliance detection, etc.). Traditional fine-tuning is prone to overfitting and decreased generalization ability, making it difficult to adapt to unseen power equipment types or newly emerging defect patterns. Summary of the Invention
[0009] The technical problem to be solved by this invention is that the CLIP model has poor performance in the power field in terms of fine-grained defect identification, technical terminology alignment and reliability in complex environments due to modal gaps.
[0010] To solve the above-mentioned technical problems, the present invention provides the following technical solution: A method for cross-modal alignment of power graphs includes: Acquire image data of power equipment and annotate it with text prompts to construct an image-text pairing dataset for the power industry; An image-text alignment method for the power industry based on probabilistic feature alignment and fusion contrast distillation is used to retrain and update the pre-trained CLIP model in the power industry using an image-text pairing dataset in the power field. The updated CLIP model was quantitatively evaluated using modal gap, alignment, and uniformity indices, and the CLIP model parameters were adjusted and optimized.
[0011] This invention proposes a power image-text alignment method based on probabilistic feature alignment and fusion-based comparative distillation. This is a lightweight domain retraining method for CLIP models applied between the pre-training and fine-tuning stages. This method requires only a single-cycle training on a limited-scale power domain image-text dataset to effectively align the image and text feature spaces, reduce modal gaps, and maintain the model's original strong zero-shot generalization ability. This significantly improves the accuracy of intelligent power equipment inspection, the accuracy of fault report archiving, and the real-time performance and reliability of operational safety monitoring.
[0012] Retraining and updating pre-trained CLIP models in the power sector includes: Set up a prior distribution shared by image and text modalities. For each image-text pair in the power field image-text pairing dataset, randomly select a reference vector from the prior distribution. Construct a probabilistic feature alignment loss function by minimizing the distance between the image features and text features of the image-text pair and the reference vector. The pre-trained CLIP model is used as the "teacher model" and the retrained CLIP model is used as the "student model". Based on minimizing the KL divergence between the outputs of the "teacher model" and the "student model", a fusion contrastive distillation loss function is constructed. Based on the probabilistic feature alignment loss function and the fusion comparison distillation loss function, an objective function is constructed to retrain and update the pre-trained CLIP model in the power field.
[0013] In this Based on probabilistic feature alignment loss function and fusion comparative distillation loss function, an objective function is constructed to retrain and update the pre-trained CLIP model in the power field, expressed by the following formula: ; In the formula, Let be the objective function. , For parameters of the image encoder and text encoder, The loss function is the probabilistic feature alignment function. To integrate and compare distillation loss functions, These are the weighting coefficients.
[0014] Construct a probabilistic feature alignment loss function, represented by the following formula: ; ; ; In the formula, The loss function is the probabilistic feature alignment function. , These are two feature vectors of an image-text pair. For the first Image feature vectors in an image-text pair For the first Text feature vectors in an image-text pair , These are the image encoder and the text encoder, respectively, obtained through pre-training using the CLIP model. , These are the parameters for the image encoder and the text encoder, respectively. , The first Image data and text data of image-text pairs. The squared Euclidean distance is the squared Euclidean distance. For the first Reference vectors for each image-text pair.
[0015] This invention does not directly minimize the gap between features, but rather optimizes the feature distributions of images and text to follow a shared prior distribution. To achieve this, on the one hand, this invention proposes a probabilistic feature alignment method, which prompts both image and text features to follow the shared prior distribution by minimizing the distance between them and random reference vectors sampled from the shared prior distribution. Specifically, probabilistic feature alignment uses a penalty term to make image and text features explicitly follow the same distribution. By matching the image and text feature distributions, it avoids feature over-concentration and maintains uniformity among samples, thus achieving a good balance between multimodal alignment and feature space uniformity.
[0016] In this The prior distribution for shared image and text modalities uses a standard Gaussian distribution.
[0017] In this The fusion-comparison distillation loss function is characterized by the following formula: ; ; ; In the formula, To integrate and compare distillation loss functions, for, for, The number of image-text pairs for a single batch of training. , for The Middle The, the An image-text pair, In the current batch of samples, the CLIP model is based on the first... The feature vector of the image in the image-text pair, the first The predicted text in an image-text pair is the conditional probability of its corresponding positive sample, used to measure the likelihood of a match between an image and text in a batch of data. In the current batch of samples, the CLIP model is based on the first... The feature vector of the Chinese text in the first image-text pair, the first... The predicted image in an image-text pair is the conditional probability of its corresponding positive sample, used to measure the likelihood of a match between text and image in a batch of data. , All are hybrid distributions that combine hard matching information and soft matching.
[0018] technology On the other hand, this invention proposes a fusion-contrast distillation method. This method updates the model by generating fused soft labels by combining real-world image-text pair labels with the output of a pre-trained CLIP model. Specifically, fusion-contrast distillation applies knowledge distillation, where the "teacher" is a pre-trained model and the "student" is a domain-retrained model. The process of fusion-contrast distillation minimizes the KL divergence between the outputs of the teacher and student models. By combining probabilistic feature alignment with fusion-contrast distillation, the contrastive language-image model can reduce modal gaps in the power field and maintain feature uniformity without forgetting past general knowledge. This method enables the model to deeply understand the professional visual features and textual descriptions in the power field while retaining the basic ability to handle unknown power scenarios or new equipment.
[0019] In this The updated CLIP model was quantitatively evaluated using modal gap, alignment, and uniformity indices. CLIP model parameters were then adjusted and optimized, including: Evaluate the clustering gap between image and text features using the modal gap metric:
[0020] Modal clearance index, , These are the average feature vectors of the image modality and the text modality across all test samples, respectively. The squared Euclidean distance; The alignment index is used to evaluate the degree of alignment of positive sample pairs in the feature space:
[0021] For alignment indicators, The number of samples in the test set. Indicates the use of an image encoder For the Image data in image-text pairs The feature vector obtained through feature extraction. Indicates the use of an image encoder For the Image-text pairs of Chinese text data Feature vectors are obtained through feature extraction. The squared Euclidean distance; The uniformity index is used to evaluate the degree of uniformity of the eigenvector distribution on the hypersphere: ; ; As a uniformity index, , From the test sample set The image features and text features corresponding to any selected sample. The union of sets; The performance of the updated CLIP model is comprehensively evaluated based on the calculated modal gap index, alignment index, and uniformity index. The hyperparameters are then optimized and the model is retrained to obtain the CLIP model with the best overall index.
[0022] application The system of the power graph cross-modal alignment method.
[0023] The image-text pairing module is used to acquire image data of power equipment and annotate it with text prompts to build an image-text pairing dataset in the power field. The update module is used for the power image-text alignment method based on probabilistic feature alignment and fusion contrast distillation. It uses the power field image-text pairing dataset to retrain and update the pre-trained CLIP model in the power field. The evaluation and optimization module is used to quantitatively evaluate the updated CLIP model using modal gap, alignment, and uniformity indices, and to adjust and optimize CLIP model parameters.
[0024] A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform... The aforementioned method for cross-modal alignment of power graphs.
[0025] Book An electronic device includes a processor and a memory; wherein the memory stores a computer program adapted for loading and execution by the processor. The aforementioned method for cross-modal alignment of power graphs.
[0026] Compared with the prior art, the beneficial effects of the present invention are: The probabilistic feature alignment scheme proposed in this invention has the advantage of effectively avoiding feature space degradation or collapse by introducing a shared prior distribution and random reference vectors, ensuring that the learned features have good uniformity and discriminative power. This improves the model's fine-grained ability to distinguish between various types of power equipment and different defect modes (such as contamination, damage, and overheating), and enhances the model's robustness and generalization ability in power field environments with limited samples and varied scenarios.
[0027] The advantage of the fusion-comparative distillation scheme proposed in this invention lies in its innovative integration of the general knowledge of the pre-trained model as a "soft label" with domain data labels. This allows the model to focus on learning the alignment relationships specific to the power industry without losing its original, broad visual language conceptual knowledge. While significantly improving the cross-modal matching accuracy for known power equipment and defects, it fully preserves the model's zero-sample reasoning ability to handle unknown equipment types, novel defects, or rare operating conditions, ensuring the long-term effectiveness and adaptability of the intelligent system in the iterative development of power business. Attached Figure Description
[0028] Figure 1 Embodiments of the present invention Cross-modal alignment method for power graphics
[0029] Figure 2 This is a schematic diagram of probability feature alignment according to an embodiment of the present invention.
[0030] Figure 3 This is a schematic diagram of fusion-comparative distillation according to an embodiment of the present invention. Detailed Implementation
[0031] To facilitate understanding of the technical solution of the present invention by those skilled in the art, the technical solution of the present invention will now be further described in conjunction with the accompanying drawings.
[0032] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.
[0033] Please see Figure 1 As shown, the present invention provides a method for cross-modal alignment of power graphs, comprising: S10: Acquire image data of power equipment and annotate it with text prompts to construct an image-text pairing dataset for the power industry.
[0034] In one embodiment of the present invention, an image-text pairing dataset in the power field is constructed, including: S11: Collect images of power equipment in different scenarios and name them according to transmission, transformation and distribution and equipment type.
[0035] S12: Manually screen the power equipment images collected in the previous step, removing a large number of duplicate, blurry, obstructed, poorly angled, poorly lit, overexposed, low-resolution, or images where key components cannot be clearly seen.
[0036] S13: For each image after filtering, manually add a prompt. For example, for an image of an insulator containing cracks, the prompt could be "a insulator with cracks".
[0037] S14: The power image-text pairing dataset labeled with prompt words is randomly divided into training and validation sets and test sets according to a 9:1 ratio. Then, the training and validation sets are randomly divided into training and validation sets according to a 9:1 ratio to obtain the power equipment image-text pairing dataset.
[0038] S20, an image-text alignment method for the power industry based on probabilistic feature alignment and fusion contrast distillation, uses an image-text pairing dataset from the power industry to retrain and update a pre-trained CLIP model in the power industry.
[0039] please In one embodiment of the present invention, retraining and updating a pre-trained CLIP model in the power sector includes: S21. Set a prior distribution shared by image and text modalities. For each image-text pair in the power field image-text pairing dataset, randomly select a reference vector from the prior distribution. Construct a probabilistic feature alignment loss function by minimizing the distance between the image features and text features of the image-text pair and the reference vector.
[0040] S22, the pre-trained CLIP model is used as the "teacher model" and the retrained CLIP model is used as the "student model". Based on minimizing the KL divergence between the outputs of the "teacher model" and the "student model", a fusion contrastive distillation loss function is constructed.
[0041] S23. Based on the probabilistic feature alignment loss function and the fusion comparison distillation loss function, an objective function is constructed to retrain and update the pre-trained CLIP model in the power field.
[0042] In this embodiment, let and Let be the image encoder and be the text encoder pre-trained using the CLIP model, with parameters respectively. for Image encoder It can convert images into digital vectors, a text encoder. Text can be converted into numerical vectors. While the CLIP model has achieved excellent performance across a wide range of domains and tasks, its image-text alignment remains imperfect. In the highly specialized power sector, equipment structures are complex, defect patterns are diverse, and technical terminology is specific, resulting in significant differences between the correlation between visual features and textual descriptions and those in the general domain. CLIP's general feature space struggles to accurately capture these domain-specificities, leading to insufficient image-text matching accuracy and robustness in specific scenarios such as power inspection and fault diagnosis, failing to meet the demands of high-reliability operations. Therefore, targeted domain retraining of the pre-trained CLIP model is necessary to enhance its cross-modal understanding capabilities in power scenarios.
[0043] In this embodiment, the present invention proposes a power image-text alignment method based on probabilistic feature alignment and fusion contrast distillation, which is applicable to image-text pairing datasets in the power field. Retraining and optimization in the next domain and The parameters are used to obtain new parameters. The optimization is mainly achieved through the following objective function. and : ; In the formula, Let be the objective function. , For parameters of the image encoder and text encoder, The loss function is the probabilistic feature alignment function. To integrate and compare distillation loss functions, These are the weighting coefficients. And, For the first Image data consisting of image-text pairs. For the first Text data of image-text pairs, For dataset Number of image-text pairs.
[0044] In this embodiment, let and It is an image-text pair Let have two eigenvectors. For the prior distribution of image and text modal sharing, this invention uses a standard Gaussian distribution. As ,in, Represented as random variables. The working principle of probabilistic feature alignment is to adjust the dataset... Each image-text pair in the text, from A reference vector is randomly selected from the data. To make both image and text features as close as possible to this reference point, the loss function is formulated as follows: and Minimize the problem between: ; In the formula, The loss function is the probabilistic feature alignment function. , These are two feature vectors of an image-text pair. For the first Image feature vectors in an image-text pair For the first Text feature vectors in an image-text pair The distance is the squared Euclidean distance. For the first Reference vectors for each image-text pair.
[0045] From the loss function above, it can be seen that the gap between the image modality and the text modality is achieved through a random reference vector shared by each image-text pair. Indirect minimization, while the feature vectors gradually follow the modality-shared prior distribution. . The randomness of the random reference vector helps avoid model overfitting to the dataset and reshapes the model's ability. Probabilistic feature alignment can improve the uniformity of the baseline. Therefore, the learned representation can be well transferred to downstream tasks such as cross-modal retrieval in the power field.
[0046] In this embodiment, although using probabilistic feature alignment for the power domain can maintain feature space uniformity and improve generalization ability, the above optimization is only performed on the power domain image-text pairing dataset D, which may lead to the discarding of knowledge from the original pre-trained CLIP model. To address this, this invention proposes a fusion-contrast distillation method, which combines the real image-text pair labels from the power domain image-text pairing dataset D with the output of the pre-trained CLIP model to generate hybrid soft labels to update the model. This method both preserves the knowledge of the original pre-trained CLIP model and enhances the contrastive learning effect.
[0047] Fusion-contrast distillation aims to learn new pairing relationships on the image-text pairing dataset D in the power field while retaining previously learned knowledge. Therefore, by fusing the "standard answer" and the "teacher's experience," it learns new knowledge while preserving old knowledge. Its loss function is shown below: ; ; ; ; ; ; ; In the formula, To integrate and compare distillation loss functions, for, for, The number of image-text pairs for a single batch of training. , , for The Middle , , Image-text pairs; This indicates that in the current batch of samples, the CLIP model is based on the first... The feature vector of the image in the image-text pair, the first The predicted text in an image-text pair is the conditional probability of its corresponding positive sample, which is used to measure the likelihood of a match between an image and text in a batch of data. Indicates the first Feature vectors of image samples in an image-text pair Indicates the first Feature vectors of text in an image-text pair; This indicates that in the current batch of samples, the CLIP model is based on the first... The feature vector of the Chinese text in the first image-text pair, the first... The predicted image in an image-text pair is the conditional probability of its corresponding positive sample, used to measure the likelihood of a match between text and image in a batch of data. Indicates the first Feature vectors of text samples in an image-text pair Indicates the first Feature vectors of images in an image-text pair; To incorporate hard matching information (if and only if) This contribution is A mixed distribution of ) and soft matching (similarity between images and text predicted by the teacher model), This indicates that in the current batch of samples, the teacher model is based on the first... The feature vector of the image in the image-text pair, the first The predicted text in an image-text pair is the conditional probability of its corresponding positive sample; To incorporate hard matching information (if and only if) This contribution is A mixed distribution of ) and soft matching (similarity between images and text predicted by the teacher model), This indicates that in the current batch of samples, the teacher model is based on the first... The feature vector of the Chinese text in the first image-text pair, the first... Image-text pair prediction image It is the conditional probability of its corresponding positive sample; Indicator functions, in The value is 1 if yes, and 0 otherwise. It is a weighting parameter that determines the balance between hard-matching information and model prediction; These are feature vectors generated by the image-text encoder of the pre-trained (teacher) model. Feature vectors generated for retraining the (student) model in the domain.
[0048] S30 uses modal gap, alignment, and uniformity indices to quantitatively evaluate the updated CLIP model and adjust and optimize CLIP model parameters.
[0049] In this embodiment, S31 The modal gap metric is used to evaluate the clustering gap between image and text features; a lower value is better. It is calculated using the following formula: ; Modal clearance index, , These are the average feature vectors of the image modality and the text modality across all test samples, respectively. The squared Euclidean distance; S32 Using alignment metrics to evaluate the degree of alignment of positive sample pairs in the feature space The lower the value, the better. It is calculated using the following formula:
[0050] For alignment indicators, The number of samples in the test set. Indicates the use of an image encoder For the Image data in image-text pairs The feature vector obtained through feature extraction. Indicates the use of an image encoder For the Image-text pairs of Chinese text data Feature vectors are obtained through feature extraction. The squared Euclidean distance; S33 The uniformity index is used to evaluate the degree of uniformity of the eigenvector distribution on the hypersphere. The lower the value, the better. It is calculated using the following formula: ; ; As a uniformity index, , From the test sample set The image features and text features corresponding to any selected sample. The union of sets; S34 The performance of the updated CLIP model is comprehensively evaluated based on the calculated modal gap index, alignment index, and uniformity index. The hyperparameters are then optimized and the model is retrained to obtain the CLIP model with the best overall index.
[0051] To quantitatively verify the effectiveness of the method of this invention, comparative experiments were conducted on the power equipment defect dataset constructed in this invention. The experiments compared the original CLIP model, the CLIP model using traditional full-parameter fine-tuning, and the CLIP model using the method of this invention for domain retraining (Ours). Evaluation metrics included modal gap (GAP), alignment (ALG), uniformity (UNF), and zero-shot classification accuracy. The results are as follows: As shown: surface Comparative experiments for quantitative verification of the method of the present invention
[0052]
[0053] In this context, "↓" indicates that the lower the value, the better, and "↑" indicates that the higher the value, the better.
[0054] Experimental results show that both the traditional fine-tuning method and the method of this invention significantly outperform the original CLIP model in reducing modal gaps and optimizing alignment, proving that both domain adaptation methods can effectively improve the alignment of text and image features in the power domain. The traditional fine-tuning method is slightly better in terms of "alignment," which is a typical manifestation of its excessive focus on the pairing relationships in the training set.
[0055] Regarding maintaining feature space uniformity: Traditional fine-tuning severely compromises feature uniformity, indicating that the feature distribution becomes concentrated and discriminative power decreases. However, the method of this invention, by introducing probabilistic feature alignment, further improves the uniformity index compared to the original model, resulting in a more uniform and reasonable feature distribution, which is directly related to the model's better generalization ability.
[0056] Regarding maintaining zero-shot generalization ability: Traditional fine-tuning leads to a significant drop in zero-shot classification accuracy, demonstrating its serious "catastrophic forgetting". However, the method of this invention significantly reduces the modality gap while not only maintaining zero-shot accuracy but also improving it. This is entirely due to the effective preservation and utilization of pre-trained general knowledge by fusion-contrast distillation.
[0057] In summary, the method proposed in this invention successfully achieved its design goals: on limited power sector data, it achieved a balance between reducing modal gaps, improving feature alignment, and enhancing distribution uniformity, and most importantly, maintained the model's zero-sample transfer capability. This verifies that the proposed method can specifically enhance cross-modal understanding and matching performance in power sector scenarios without sacrificing model generality.
[0058] An application The system of the power graph cross-modal alignment method.
[0059] The image-text pairing module is used to acquire image data of power equipment and annotate it with text prompts, constructing an image-text pairing dataset for the power industry.
[0060] The update module is used for a power image-text alignment method based on probabilistic feature alignment and fusion contrast distillation. It retrains and updates a pre-trained CLIP model in the power domain using an image-text pairing dataset.
[0061] The evaluation and optimization module is used to quantitatively evaluate the updated CLIP model using modal gap, alignment, and uniformity indices, and to adjust and optimize CLIP model parameters.
[0062] Book A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform... The aforementioned method for cross-modal alignment of power graphs.
[0063] Computer-readable storage media can be electronic media, magnetic media, optical media, electromagnetic media, infrared media, or semiconductor systems or propagation media. Computer-readable storage media can also include semiconductor or solid-state memory, magnetic tape, removable computer disks, random access memory (RAM), read-only memory (ROM), hard disks, and optical discs. Optical discs can include optical disc-read-only memory (CD-ROM), optical disc-read / write (CD-RW), and DVDs.
[0064] Book An electronic device includes a processor and a memory; wherein the memory stores a computer program adapted for loading and execution by the processor. The aforementioned method for cross-modal alignment of power graphs.
[0065] The processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.
[0066] The memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. The memory can also be internal memory of the random access memory (RAM) type. The processor and memory can be integrated into one or more independent circuits or hardware, such as application-specific integrated circuits (ASICs). It should be noted that when a computer program in the memory is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention.
[0067] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.
[0068] The above-described embodiments are merely examples of implementation methods of the invention. The scope of protection of the present invention is not limited to the above-described embodiments. For those skilled in the art, several modifications and improvements can be made without departing from the concept of the present invention, and these all fall within the scope of protection of the present invention.
Claims
1. A method for cross-modal alignment of power graphs, characterized in that, include: Acquire image data of power equipment and annotate it with text prompts to construct an image-text pairing dataset for the power industry; An image-text alignment method for the power industry based on probabilistic feature alignment and fusion contrast distillation is used to retrain and update the pre-trained CLIP model in the power industry using an image-text pairing dataset in the power field. The updated CLIP model was quantitatively evaluated using modal gap, alignment, and uniformity indices, and the CLIP model parameters were adjusted and optimized.
2. The cross-modal alignment method for power graphs according to claim 1, characterized in that, Retraining and updating pre-trained CLIP models in the power sector includes: Set up a prior distribution shared by image and text modalities. For each image-text pair in the power field image-text pairing dataset, randomly select a reference vector from the prior distribution. Construct a probabilistic feature alignment loss function by minimizing the distance between the image features and text features of the image-text pair and the reference vector. The pre-trained CLIP model is used as the "teacher model" and the retrained CLIP model is used as the "student model". Based on minimizing the KL divergence between the outputs of the "teacher model" and the "student model", a fusion contrastive distillation loss function is constructed. Based on the probabilistic feature alignment loss function and the fusion comparison distillation loss function, an objective function is constructed to retrain and update the pre-trained CLIP model in the power field.
3. The cross-modal alignment method for power graphs according to claim 2, characterized in that, Based on probabilistic feature alignment loss function and fusion comparative distillation loss function, an objective function is constructed to retrain and update the pre-trained CLIP model in the power field, expressed by the following formula: ； In the formula, Let be the objective function. , For parameters of the image encoder and text encoder, The loss function is the probabilistic feature alignment function. To integrate and compare distillation loss functions, These are the weighting coefficients.
4. The cross-modal alignment method for power graphs according to claim 2, characterized in that, Construct a probabilistic feature alignment loss function, represented by the following formula: ； ； ； In the formula, The loss function is the probabilistic feature alignment function. , These are two feature vectors of an image-text pair. For the first Image feature vectors in an image-text pair For the first Text feature vectors in an image-text pair , These are the image encoder and the text encoder, respectively, obtained through pre-training using the CLIP model. , These are the parameters for the image encoder and the text encoder, respectively. , The first Image data and text data of image-text pairs. The distance is the squared Euclidean distance. For the first A reference vector for each image-text pair.
5. The cross-modal alignment method for power graphs according to claim 2, characterized in that, The prior distribution for image and text modal sharing uses a standard Gaussian distribution.
6. The cross-modal alignment method for power graphs according to claim 2, characterized in that, The fusion-comparison distillation loss function is characterized by the following formula: ； ； ； In the formula, To integrate and compare distillation loss functions, for, for, The number of image-text pairs for a single batch of training. In the current batch of samples, the CLIP model is based on the first... The feature vector of the image in the image-text pair, the first The predicted text in an image-text pair is the conditional probability of its corresponding positive sample, used to measure the likelihood of a match between an image and text in a batch of data. In the current batch of samples, the CLIP model is based on the first... The feature vector of the Chinese text in the first image-text pair, the first... The predicted image in an image-text pair is the conditional probability of its corresponding positive sample, used to measure the likelihood of a match between text and image in a batch of data. , All are hybrid distributions that combine hard matching information and soft matching.
7. The cross-modal alignment method for power graphs according to claim 1, characterized in that, The updated CLIP model was quantitatively evaluated using modal gap, alignment, and uniformity indices. CLIP model parameters were then adjusted and optimized, including: Evaluate the clustering gap between image and text features using the modal gap metric: ； Modal clearance index, , These are the average feature vectors of the image modality and the text modality across all test samples, respectively. The squared Euclidean distance; The alignment index is used to evaluate the degree of alignment of positive sample pairs in the feature space: For alignment indicators, The number of samples in the test set. Indicates the use of an image encoder For the Image data in image-text pairs The feature vector obtained through feature extraction. Indicates the use of an image encoder For the Image-text pairs of Chinese text data Feature vectors are obtained through feature extraction. The uniformity index is used to evaluate the degree of uniformity of the eigenvector distribution on the hypersphere: ； ； As a uniformity index, , From the test sample set The image features and text features corresponding to any selected sample. The union of sets; The performance of the updated CLIP model is comprehensively evaluated based on the calculated modal gap index, alignment index, and uniformity index. The hyperparameters are then optimized and the model is retrained to obtain the CLIP model with the best overall index.
8. A system applying the cross-modal alignment method for power graphs according to any one of claims 1-7, characterized in that, include: The image-text pairing module is used to acquire image data of power equipment and annotate it with text prompts to build an image-text pairing dataset in the power field. The update module is used for the power image-text alignment method based on probabilistic feature alignment and fusion contrast distillation. It uses the power field image-text pairing dataset to retrain and update the pre-trained CLIP model in the power field. The evaluation and optimization module is used to quantitatively evaluate the updated CLIP model using modal gap, alignment, and uniformity indices, and to adjust and optimize CLIP model parameters.
9. A computer-readable storage medium, characterized in that, It stores executable instructions, which, when executed by a processor, enable the processor to perform the cross-modal alignment method for power graphs as described in any one of claims 1-7.
10. An electronic device, characterized in that, include: A processor and a memory; wherein the memory stores a computer program adapted for the processor to load and execute the cross-modal alignment method for power graphics according to any one of claims 1-7.