Abnormality identification method and device, electronic equipment and storage medium

Abstract

This application provides an anomaly identification method, apparatus, electronic device, and storage medium. It determines the anomaly type by matching image features with descriptive text vectors in a pre-established mapping relationship. Even when faced with anomaly types not present in the training set, accurate identification is possible as long as their descriptive text vectors exist in the mapping relationship. This effectively overcomes the difficulty in identifying new anomaly types due to scarce labeled data, significantly improving the comprehensiveness of anomaly identification. Because the fused image integrates multimodal information, it enhances the cross-modal capability and generalization ability of the anomaly identification method, enabling more accurate differentiation between normal changes and genuine anomalies, reducing the probability of false positives and false negatives, and improving the reliability and accuracy of the anomaly identification method under different environmental and temporal conditions.

Description

Technical Field

[0001] This application belongs to the field of photovoltaic system technology, and in particular relates to an anomaly identification method and device. Background Technology

[0002] In the current booming photovoltaic industry, anomaly identification of photovoltaic modules is crucial. Related technologies use labeled anomaly samples to train models for anomaly identification. However, obtaining labeled data is extremely difficult. Because photovoltaic modules exhibit diverse anomalies, such as cracks and hot spots, many anomalies occur infrequently in actual data, resulting in very few or no labeled samples in the training set. Collecting and labeling these samples is costly and labor-intensive. Furthermore, if certain anomaly types are absent from the training set, the model cannot identify these new anomaly types. In addition, existing anomaly identification methods lack generalization ability. Because photovoltaic power station images are rich in modalities, including visible light and infrared, the anomaly characteristics differ significantly across modalities. Existing models are mostly trained on a single modality, exhibiting poor cross-modal capabilities and prone to false positives and false negatives. Summary of the Invention

[0003] In view of this, embodiments of this application provide an anomaly identification method, apparatus, electronic device, and storage medium, which can achieve accurate identification, effectively overcome the problem of difficulty in identifying new anomaly types due to the scarcity of labeled data, and greatly improve the comprehensiveness of anomaly identification.

[0004] In a first aspect, embodiments of this application provide an anomaly identification method, including: A fused image of a photovoltaic module is obtained, wherein the fused image is obtained based on a multimodal image; Based on a pre-established image encoder, features are extracted from the fused image to obtain image features; The image features are matched with the descriptive text vectors in the pre-established mapping relationship to obtain the matching result, wherein the mapping relationship includes the correspondence between the anomaly type and the descriptive text vector of the anomaly type; The anomaly type of the photovoltaic module is obtained based on the matching results.

[0005] In some embodiments, the multimodal image includes: a visible light image and an infrared image, and acquiring the fused image of the photovoltaic module includes: The visible light image and the infrared image are registered; The registered visible light image and the infrared image are fused to obtain an initial fused image; The initial fused image is standardized to obtain the fused image.

[0006] In some embodiments, the method further includes: If the matching result is that no descriptive text vector is matched, the distance between the image feature and each descriptive text vector is determined; If the distance is greater than the distance threshold, it is determined that a new anomaly type exists; The new exception type is output so that the user can make a determination. Upon user confirmation, the mapping relationship is updated based on the new exception type.

[0007] In some embodiments, the method further includes: Anomalies are identified based on the fused image; Output a language description result based on the anomaly type and the anomaly location.

[0008] In some embodiments, the method further includes: Obtain environmental data and historical power generation data of the photovoltaic modules; The environmental data, the historical power generation data, and the anomaly type are input into a pre-established prediction model to obtain power generation prediction data within a first preset time period.

[0009] In some embodiments, the method further includes: Determine the severity of the anomaly based on the anomaly type; A predicted degradation score is obtained based on the theoretical power generation data of the photovoltaic module within the first preset time period and the predicted power generation data. Based on the theoretical power generation data and the predicted power generation data, the economic loss of power generation within the first preset time period is obtained. A benefit-to-cost ratio score for economics and maintenance is obtained based on the maintenance costs corresponding to the power generation economic losses and anomaly types. The maintenance priority is calculated based on the severity of the anomaly, the predicted degradation score, and the benefit ratio score. Maintenance recommendations are generated based on the aforementioned maintenance priorities.

[0010] In some embodiments, the method further includes: Acquire sample data, wherein the sample data includes: positive sample pairs and negative sample pairs, the positive sample pairs include: sample fusion image pairs of the same photovoltaic module within a second preset time period, and the negative sample pairs include: sample fusion image pairs of the same photovoltaic module whose shooting time is greater than a time threshold, whose state has changed, and / or different photovoltaic modules; The initial image encoder is trained based on the sample data to obtain a trained image encoder. The loss function of the image encoder includes maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The image encoder includes a window self-attention mechanism module and a shift window attention mechanism module. The window self-attention mechanism module is used to divide the sample fusion image into multiple non-overlapping windows and calculate self-attention for each window. The shift window attention mechanism module is used to perform shift operations between adjacent windows of the sample fusion image so that adjacent windows can interact with each other.

[0011] Secondly, embodiments of this application provide an anomaly detection device, comprising: An acquisition module is used to acquire a fused image of a photovoltaic module, wherein the fused image is obtained based on a multimodal image; The extraction module is used to extract features from the fused image based on a pre-established image encoder to obtain image features; The matching module is used to match the image features with the descriptive text vectors in the pre-established mapping relationship to obtain the matching result, wherein the mapping relationship includes the correspondence between the anomaly type and the descriptive text vector of the anomaly type; The module is used to obtain the anomaly type of the photovoltaic module based on the matching results.

[0012] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the methods described above.

[0013] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements any of the methods described above.

[0014] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes an electronic device to execute any of the methods described above.

[0015] The beneficial effects of the embodiments in this application compared with the prior art are: This application provides an anomaly identification method that determines the anomaly type by matching image features with descriptive text vectors in a pre-established mapping relationship. Even when faced with anomaly types not present in the training set, accurate identification can be achieved as long as their descriptive text vectors exist in the mapping relationship. This effectively overcomes the difficulty in identifying new anomaly types due to the scarcity of labeled data, greatly improving the comprehensiveness of anomaly identification. Since the fused image integrates multimodal information, it improves the cross-modal capability and generalization ability of the anomaly identification method. To a certain extent, it can reduce the impact of differences in the appearance of the same component caused by the passage of time, changes in environmental factors, equipment aging, dust accumulation, etc., on anomaly identification. This makes the features extracted by the image encoder more stable, enabling more accurate differentiation between normal changes and true anomalies, reducing the probability of false detection and missed detection, and improving the reliability and accuracy of the anomaly identification method under different environmental and time conditions. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 A schematic diagram illustrating the implementation process of an anomaly identification method provided for the purposes of this application; Figure 2 A schematic diagram illustrating the implementation process of a power generation forecasting data prediction method provided in this application embodiment; Figure 3 A schematic diagram of the structure of an image encoder provided in this application; Figure 4 This is a schematic diagram of the structure of an anomaly identification device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0018] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0019] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0020] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once detected," or "in response to detection."

[0022] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0023] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0024] Based on the problems in related technologies, this application provides an anomaly identification method that can be applied to electronic devices, including: mobile phones, tablets, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptops, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. Figure 1 A schematic diagram illustrating the implementation flow of an anomaly identification method provided for the purposes of this application is shown below. Figure 1 As shown, the anomaly detection methods include: Step S101: Obtain a fused image of the photovoltaic module, wherein the fused image is obtained based on a multimodal image.

[0025] In this embodiment, a photovoltaic module, also known as a solar cell module, is a device that directly converts solar energy into electrical energy by encapsulating multiple solar cells together in series and parallel. It is the core component of a solar power generation system. The fused image is an image obtained based on multimodal images. Multimodal images typically contain image information collected by different types of sensors, such as visible light images and infrared images. The fused image integrates the characteristics and advantages of these different modal images. A single modal image may not fully reflect the state of the photovoltaic module; the fused image can provide richer and more accurate information, helping to more accurately identify anomalies in the photovoltaic module. A multimodal image is a collection of images collected by various types of sensors. Different modal images have different characteristics and information expression methods. For example, visible light images reflect the appearance and shape of an object, while infrared images reflect the temperature distribution of an object. Different modal images provide information about the photovoltaic module from different angles; comprehensively utilizing this information can improve the accuracy and reliability of anomaly identification.

[0026] In this embodiment, different types of sensors (such as visible light cameras and infrared thermal imagers) can be used to acquire visible light and infrared images of the photovoltaic module, respectively. Then, the acquired multimodal images are registered to align the different modal images in space, ensuring accurate fusion. Next, an image fusion algorithm (such as weighted average, pyramid fusion, wavelet transform fusion, etc.) is used to fuse the registered visible light and infrared images to obtain an initial fused image. Finally, the initial fused image is standardized, such as by adjusting its brightness, contrast, and size, to meet the requirements of subsequent processing, resulting in the final fused image. The sensors can be mounted on a drone.

[0027] Step S102: Extract features from the fused image based on a pre-established image encoder to obtain image features.

[0028] In this embodiment, the image encoder is a neural network model or other algorithmic structure used to convert image data into feature vectors. It can automatically learn key features in an image and represent the image as a low-dimensional feature vector. By converting the fused image into image features, it facilitates subsequent matching with descriptive text vectors and anomaly type determination. Image features are information extracted from an image that represents its essential attributes; they are typically in low-dimensional vector form and contain information about the image's shape, texture, color, and other aspects.

[0029] In this embodiment, the image encoder can learn robust cross-modal feature representations through tasks such as modal comparison and temporal comparison, making the features sensitive to the normal mode of components and capable of amplifying subtle abnormal deviations. The fused image can be input into the image encoder, which automatically learns the features in the fused image through multiple feature extraction layers, progressively extracting low-level features (such as edges and textures) and high-level features (such as object shape and semantic information). Finally, the image encoder represents the fused image as a low-dimensional image feature vector, which contains key information about the image for subsequent matching and anomaly type determination.

[0030] Step S103: Match the image features with the descriptive text vectors in the pre-established mapping relationship to obtain the matching result. The mapping relationship includes the correspondence between the anomaly type and the descriptive text vector of the anomaly type.

[0031] In this embodiment, the mapping relationship is a pre-established correspondence between anomaly types and their descriptive text vectors. The descriptive text vector is a vector obtained by encoding the textual description of the anomaly type. By matching image features with the descriptive text vector, the anomaly type of the photovoltaic module can be quickly and accurately determined. The descriptive text vector is a vector obtained by encoding the textual description of the anomaly type, typically using natural language processing techniques (such as word embedding models) to convert text into vector form. For example, natural language descriptive words (such as "crack," "hot spot," "dust obstruction," "corrosion," etc.) can be used.

[0032] In this embodiment, a mapping relationship between anomaly types and descriptive text vectors can be pre-established. For each anomaly type, relevant text descriptions are collected, and natural language processing techniques are used to convert these text descriptions into descriptive text vectors. Similarity calculation methods (such as cosine similarity, Euclidean distance, etc.) are used to calculate the similarity between the image feature vectors and each descriptive text vector. Matching results are obtained based on the similarity calculation results; typically, the anomaly type corresponding to the descriptive text vector with the highest similarity is selected as the initial matching result.

[0033] Step S104: Obtain the anomaly type of the photovoltaic module based on the matching result.

[0034] In this embodiment of the application, the matching result is the result obtained by matching the image features with the descriptive text vectors, which usually represents the similarity or matching degree between the image features and each descriptive text vector.

[0035] In this embodiment, if a descriptive text vector with a similarity higher than a preset threshold exists in the matching results, the anomaly type corresponding to that descriptive text vector is determined as an anomaly type of the photovoltaic module. If the similarity of all descriptive text vectors in the matching results is lower than the preset threshold, it may indicate that a new anomaly type exists in the photovoltaic module, requiring further analysis and processing, such as manual judgment or the use of other anomaly detection methods. In some embodiments, if no descriptive text vector is matched in the matching results, it is determined that there is no anomaly.

[0036] The method provided in this application determines the anomaly type by matching image features with descriptive text vectors in a pre-established mapping relationship. Even when faced with anomaly types that have not appeared in the training set, as long as their descriptive text vectors exist in the mapping relationship, accurate identification can be achieved. This effectively overcomes the problem of difficulty in identifying new anomaly types due to the scarcity of labeled data, greatly improving the comprehensiveness of anomaly identification. Since the fused image integrates multimodal information, it improves the cross-modal capability and generalization ability of the anomaly identification method. To a certain extent, it can reduce the impact of differences in the appearance of the same component caused by the passage of time, changes in environmental factors, equipment aging, dust accumulation, etc. on anomaly identification, making the features extracted by the image encoder more stable, more accurately distinguishing between normal changes and real anomalies, reducing the probability of false detection and missed detection, and improving the reliability and accuracy of the anomaly identification method under different environmental and time conditions.

[0037] The method provided in this application embodiment can be applied to scenarios requiring inspection and monitoring, such as large-scale ground-mounted photovoltaic power plants (distributed or centralized), with a large number of components and a large area. It can be applied to hybrid monitoring systems combining drone inspection and fixed ground cameras. It can also be applied to photovoltaic power plant operation and maintenance (O&M) systems / monitoring systems for early fault detection, preventative maintenance, efficiency optimization, and maintenance cost control. In some embodiments, step S101 can be implemented through the following steps: Step S1011: Register the visible light image and the infrared image.

[0038] In this embodiment, the visible light image is acquired by a visible light camera. Visible light images clearly show the surface structure and appearance defects (such as scratches, stains, and damage) of the photovoltaic module, providing an intuitive basis for identifying appearance anomalies. The infrared image is acquired by an infrared thermal imager. It is based on the infrared radiation emitted by an object to create an image, reflecting the object's temperature distribution. Objects at different temperatures will appear in different shades or colors in the infrared image (depending on the thermal imager's display settings). During operation, if there are internal faults in the photovoltaic module (such as damaged cells or poor connections), it will cause abnormal increases or decreases in local temperature. Infrared images can capture these temperature changes, thereby helping to identify internal faults in the photovoltaic module. Registration involves aligning the images acquired by different sensors (here, visible light and infrared images) in spatial position, so that the same object or feature point in the two images has the same correspondence in spatial coordinates. Because the shooting angles and positions of the visible light camera and the infrared thermal imager may differ, the acquired images may have spatial misalignment. Registration can eliminate this misalignment, ensuring that the information from different modalities can accurately correspond during subsequent fusion, thereby improving the quality of the fused image and the accuracy of anomaly detection.

[0039] In this embodiment, feature points are extracted from both the visible light image and the infrared image. These extracted feature points are then matched to find corresponding feature point pairs in the two images. Based on the matched feature point pairs, a transformation matrix is ​​estimated to transform the infrared image to the visible light image coordinate system. Commonly used transformation models include rigid body transformations (including translation and rotation) and affine transformations (including translation, rotation, scaling, and tilting). The estimated transformation matrix is ​​then used to transform the infrared image, aligning it spatially with the visible light image, thus completing the registration.

[0040] Step S1012: The registered visible light image and the infrared image are fused to obtain an initial fused image.

[0041] In this embodiment of the application, the initial fused image is the image obtained by fusing the registered visible light image and infrared image according to a certain fusion algorithm, which initially integrates the information of the two images.

[0042] In this embodiment, a weighted average method can be used for fusion, or the visible light image and the infrared image can be decomposed into pyramid structures of different scales, and then fused at each scale. Finally, the fused pyramid image is reconstructed to obtain the initial fused image.

[0043] Step S1013: Standardize the initial fused image to obtain the fused image.

[0044] In this embodiment, the standardization process involves performing a series of adjustments on the initial fused image to give it uniform parameters such as brightness, contrast, and size, so as to meet the requirements of subsequent processing (such as feature extraction and matching).

[0045] In this embodiment, the brightness distribution of the initial fused image can be adjusted using methods such as histogram equalization to enhance image contrast and make image details clearer. Histogram equalization stretches the grayscale histogram of the image to make it evenly distributed across the entire grayscale range. Methods such as linear contrast stretching and nonlinear contrast adjustment can be used to further enhance image contrast and highlight important information in the image. The size of the initial fused image can be adjusted to a uniform size to meet the input requirements of the subsequent image encoder. Methods such as bilinear interpolation and bicubic interpolation can be used to adjust the image size.

[0046] The method provided in this application uses visible light images and infrared images to reflect information about photovoltaic modules from different angles. Through registration and fusion, the information from the two images is organically combined, fully utilizing the advantages of multimodal images. This allows for a more comprehensive acquisition of the photovoltaic module's state information and improves the ability to identify photovoltaic module anomalies. The registration process eliminates spatial misalignment between different modal images, making the fused image more accurate and clear. Standardization processing further adjusts parameters such as image brightness, contrast, and size, improving image quality and providing a better foundation for subsequent feature extraction and anomaly identification. Visible light images and infrared images have different adaptability to ambient light and photovoltaic module operating conditions. The fused image combines the advantages of both, enabling more stable identification of photovoltaic module anomalies under different lighting conditions (such as strong light and weak light) and operating conditions (such as different loads and different times), exhibiting strong robustness.

[0047] In some embodiments, after step S104, the method further includes: Step S105: If the matching result is that no description text vector is matched, determine the distance between the image feature and each description text vector.

[0048] In this embodiment, distance is used as a quantitative indicator to measure the degree of difference between two vectors in vector space. Common distance measurement methods include Euclidean distance and cosine similarity (which can be converted into distance form). The magnitude of the distance reflects the similarity between image features and the anomaly type represented by the descriptive text vector. The smaller the distance, the higher the similarity; the larger the distance, the lower the similarity. By setting a distance threshold, it can be determined whether the image features correspond to a known anomaly type.

[0049] In this embodiment of the application, a selected distance metric method is used to sequentially calculate the distance between the image feature vector and each descriptive text vector in the database, thereby obtaining a set of distance values.

[0050] Step S106: If the distance is greater than the distance threshold, determine that a new anomaly type exists.

[0051] In this embodiment, the distance threshold is a pre-set value used as a standard to determine whether the distance between the image feature and the descriptive text vector is too large. When the distance is greater than the threshold, the anomaly represented by the image feature is considered not to belong to a known anomaly type, and a new anomaly may exist. A new anomaly type refers to an anomaly that was not previously recorded or identified, which is determined during the photovoltaic module anomaly identification process by matching the image feature with known descriptive text vectors and finding that the distance between the image feature and all known descriptive text vectors is greater than the distance threshold.

[0052] In this embodiment, a suitable distance threshold is determined through experimentation or experience based on actual application scenarios and historical data. T The distances between the calculated image feature vectors and each descriptive text vector are compared with the distance thresholds. T Compare them. If all distances are greater than... T If so, it is determined that a new anomaly type exists.

[0053] Step S107: Output the new exception type so that the user can make a determination.

[0054] In this embodiment, information confirming the existence of a new anomaly type can be presented to the user in an intuitive way, such as by popping up a prompt window on the user interface displaying information like "Possible new anomaly type detected, please confirm." Simultaneously, image feature information and distance values ​​related to the new anomaly type can be displayed to the user, allowing for a more comprehensive understanding of the situation and enabling accurate judgment.

[0055] Step S108: If the user confirms, update the mapping relationship based on the new exception type.

[0056] In this embodiment, a confirmation button is provided on the user interface. When the user confirms the existence of a new anomaly type, they click the confirmation button. After receiving the user's confirmation, the image features of the new anomaly type are associated with the corresponding descriptive text (which may be a detailed description entered by the user or a preliminary description generated by the system), and this new mapping relationship is stored in the database or model parameters, thus completing the update of the mapping relationship.

[0057] The method provided in this application, by detecting cases where no known descriptive text vector is matched and the distance is greater than a threshold, can promptly identify new anomaly types in photovoltaic modules. This avoids the limitations of only identifying known anomaly types, making the anomaly identification system more comprehensive and complete, and providing a more reliable guarantee for the stable operation of the photovoltaic system. Allowing users to confirm new anomaly types and update mapping relationships enables the system to continuously adjust and optimize according to actual conditions, adapting to various anomalies that may occur in photovoltaic modules under different environments and operating stages, thus improving the system's adaptability and flexibility. The process of discovering and confirming new anomaly types helps accumulate knowledge and experience about photovoltaic module anomalies, providing data support for subsequent research and model optimization. By continuously updating mapping relationships, the anomaly identification model can become more accurate and intelligent, improving its ability to identify various anomalies.

[0058] In some embodiments, after step S103, the method further includes: Step S109: Identify abnormal locations based on the fused image.

[0059] In this embodiment, the abnormal location refers to the specific area in the photovoltaic module image where an anomaly occurs. It can be described using coordinates, pixel range, or relative position (such as within a specific section of the module).

[0060] In this embodiment, the fused image can be preprocessed, including denoising and contrast enhancement, to improve image quality and make anomalous features more apparent. For example, median filtering can be used to remove noise from the image, and histogram equalization can be used to enhance image contrast. Image processing and machine learning algorithms are then used to extract features from the fused image. Edge detection algorithms (such as Canny edge detection) can be used to extract edge information from the image, or deep learning models (such as convolutional neural networks, CNNs) can be used to automatically learn feature representations in the image. Based on the extracted features, anomaly detection algorithms (such as statistical methods, clustering methods, etc.) or deep learning models (such as object detection models) are used to determine the location of anomalous regions. For example, a threshold-based anomaly detection method can be used, setting an appropriate threshold to mark regions with pixel values ​​exceeding the threshold as anomalous regions; or object detection models such as YOLO (You Only Look Once) can be used to directly detect the location and boundaries of anomalous regions.

[0061] Step S110: Output a language description result based on the anomaly type and the anomaly location.

[0062] In this embodiment, the language description result is the output content describing the abnormal conditions of the photovoltaic module in natural language. It includes key information such as the abnormality type and location, expressed in easy-to-understand language, making it convenient for non-experts to understand.

[0063] In this embodiment, the identified anomaly types and location information are integrated. The anomaly location can be associated with the anomaly type using specific coordinates or regional descriptions, forming a complete information set. Natural language generation technology is used to convert the integrated information into a natural language description. A predefined template can be used, where the anomaly type and location information are filled into the template to generate the language description result. For example, if the template is "[anomaly type] anomaly was found in the [X coordinate, Y coordinate] region of the photovoltaic module," and the anomaly type is a hot spot with anomaly location coordinates of (100, 200), the generated language description result would be "A hot spot anomaly was found in the (100, 200) region of the photovoltaic module." Alternatively, a more advanced natural language processing model, such as a Transformer-based model, can be used to automatically generate a more natural and fluent language description based on the input anomaly type and location information.

[0064] The method provided in this application identifies abnormal locations by fusing multiple image information sources. It comprehensively utilizes the advantages of different images, enabling more accurate capture of abnormal features, reducing the possibility of false positives and false negatives, and improving the accuracy and reliability of anomaly detection. The output description is presented in natural language, allowing even non-professionals to easily understand the abnormal conditions of photovoltaic modules, including the type and location of the anomaly, providing intuitive and easy-to-understand information support for operation and maintenance decisions.

[0065] In some embodiments, the method further includes: Step S111: Obtain the environmental data and historical power generation data of the photovoltaic module.

[0066] In this embodiment, environmental data refers to various data related to the environment in which the photovoltaic module is located, typically including light intensity, temperature, humidity, wind speed, and wind direction. These data reflect the external working environmental conditions of the photovoltaic module. Historical power generation data is the actual electricity generated by the photovoltaic module over a past period, usually recorded in time series format, and can be statistically analyzed at different time granularities such as hourly, daily, and monthly. Historical power generation data includes the power generation performance of the photovoltaic module under different environmental conditions and operating states, providing important reference for prediction models. By analyzing historical data, the model can learn the patterns and trends of photovoltaic module power generation, thereby improving the accuracy of predictions.

[0067] In this embodiment, various environmental sensors, such as light sensors, temperature sensors, humidity sensors, and wind speed and direction sensors, can be installed around the photovoltaic modules. These sensors can collect relevant data about the environment in which the photovoltaic modules are located in real time. The actual power generation data of the photovoltaic modules at different time points can be read by a meter installed at the output end of the photovoltaic modules. The meter can be a smart meter, capable of recording and transmitting power generation data in real time.

[0068] Step S112: Input the environmental data, the historical power generation data, and the anomaly type into the pre-established prediction model to obtain power generation prediction data within a first preset time period.

[0069] In this embodiment, the prediction model is a model based on mathematical algorithms and statistical methods, used to predict the power generation of photovoltaic modules within a first preset time period based on input environmental data, historical power generation data, and anomaly types. Common prediction models include linear regression models, neural network models, and support vector machine models. The first preset time period is a pre-set time length used to specify the time range for the power generation prediction data output by the prediction model; it can be several hours, a day, a week, etc., determined according to actual needs.

[0070] In this embodiment, a sufficient amount of environmental data, historical power generation data, and corresponding anomaly type data can be collected as a training set. The data is preprocessed, including normalization and standardization, to ensure the data has a uniform scale, facilitating model training. A suitable prediction model, such as a neural network model, is selected based on the data characteristics and prediction requirements. The model is trained using the training set, and the prediction error is minimized by adjusting the model's parameters (such as the weights and biases of the neural network). Methods such as cross-validation can be used to evaluate the model's performance and prevent overfitting. The trained model is validated using an independent test set to evaluate its prediction accuracy and generalization ability. Based on the validation results, the model is optimized, such as by adjusting the model structure and increasing the amount of training data, until the model achieves satisfactory performance. The prediction model can be a time series prediction model based on Transformer, LSTM, or Temporal CNN.

[0071] In the embodiments of this application, Figure 2 This is a schematic diagram illustrating the implementation process of a power generation forecasting data prediction method provided in an embodiment of this application, as shown below. Figure 2 As shown, environmental data, historical power generation data, and anomaly types can be input into the prediction model to output power generation prediction data. The prediction result can be a specific power generation value or a probability distribution of power generation, etc.

[0072] The method provided in this application comprehensively considers multiple factors such as environmental data, historical power generation data, and anomaly types, enabling it to more fully reflect the factors influencing photovoltaic module power generation. Environmental data reflects external operating conditions, historical power generation data provides past power generation patterns, and anomaly types consider the impact of abnormal module conditions on power generation, thus enabling the prediction model to more accurately predict future power generation.

[0073] In some embodiments, after step S112, the method further includes: Step S113: Determine the severity of the anomaly based on the anomaly type.

[0074] In this embodiment, the severity of anomalies is used as an indicator to measure the impact of abnormal conditions on the performance and power generation capacity of photovoltaic modules. Different types of anomalies, such as minor hot spots or severe microcracks in the cells, have vastly different impacts on the modules, and this impact can be quantified through the severity of the anomaly.

[0075] In this embodiment, a mapping table between anomaly types and their severity can be pre-established based on actual operating experience and professional knowledge of photovoltaic modules. For example, minor hot spots are defined as low severity, while severe microcracks in the cells are defined as high severity. Once the anomaly type of the photovoltaic module is identified, the corresponding severity can be directly determined by querying this mapping table.

[0076] Step S114: Obtain a predicted degradation score based on the theoretical power generation data of the photovoltaic module within the first preset time period and the predicted power generation data.

[0077] In this embodiment, the theoretical power generation data refers to the amount of electricity that should be generated within a first preset time period under ideal conditions, i.e., when the photovoltaic module is without any abnormalities and is in a standard environment (such as standard light intensity, standard temperature, etc.). It is typically calculated based on the rated power of the photovoltaic module, the first preset time period, and standard environmental parameters. The predicted degradation score is used to measure the degree of power generation degradation relative to the theoretical power generation caused by photovoltaic module abnormalities. It is a quantitative indicator calculated using both theoretical power generation data and predicted power generation data.

[0078] In this embodiment, the power generation attenuation within a first preset time period can be obtained by subtracting the predicted power generation data from the theoretical power generation data. The ratio of the power generation attenuation to the theoretical power generation data is used to predict the attenuation score. The formula is: ,in, To predict the decay score, For theoretical data on power generation, (This refers to power generation forecast data).

[0079] Step S115: Based on the power generation theoretical data and the power generation prediction data, obtain the power generation economic loss within the first preset time period.

[0080] In this embodiment, the economic loss of power generation refers to the economic loss caused by the difference between the actual power generation and the theoretical power generation due to photovoltaic module malfunctions within a first preset time period. The economic loss of power generation can be calculated as power loss x electricity price x first preset time period.

[0081] Step S116: Based on the economic losses from power generation and the maintenance costs corresponding to the anomaly types, obtain a benefit ratio score for economics and maintenance.

[0082] In this embodiment, the benefit ratio score is a rating index calculated by comprehensively considering the economic loss from power generation and the maintenance cost corresponding to the anomaly type. It reflects the proportional relationship between the economic benefits and maintenance costs brought about by the maintenance of photovoltaic module anomalies.

[0083] In this embodiment, a pre-established mapping table between anomaly types and maintenance costs can be queried based on the anomaly type to determine the corresponding maintenance cost. The benefit ratio score is obtained by dividing the power generation economic loss by the maintenance cost. This can be calculated using the following formula. ,in, Rate the return on investment. Economic losses due to power generation For maintenance costs. and These are the corresponding weights.

[0084] Step S117: Calculate the maintenance priority based on the severity of the anomaly, the predicted attenuation score, and the benefit ratio score.

[0085] In this embodiment, the maintenance priority is a priority ranking index calculated based on a combination of indicators such as the severity of the anomaly, the predicted degradation score, and the yield ratio score. This index is used to determine the order in which photovoltaic modules undergo anomaly repairs.

[0086] The formula for calculating maintenance priority can be expressed as: ; in, As a maintenance priority, To indicate an abnormally severe level, , and For weights.

[0087] In this embodiment, based on actual needs and expert experience, a weighting coefficient is determined for the severity of the anomaly, the predicted attenuation score, and the benefit ratio score, for example, weights of 0.4, 0.3, and 0.3 respectively. The severity of the anomaly, the predicted attenuation score, and the benefit ratio score are multiplied by their respective weighting coefficients and then summed to obtain the maintenance priority score. The formula is: Maintenance Priority = Severity of Anomaly × Weight 1 + Predicted Attenuation Score × Weight 2 + Benefit Ratio Score × Weight 3. The maintenance priorities are then ranked according to their scores; the higher the score, the higher the maintenance priority.

[0088] Step S118: Output maintenance recommendations based on the maintenance priority.

[0089] In this embodiment, the maintenance suggestion is a specific suggestion for the maintenance of photovoltaic modules output according to the maintenance priority, including maintenance time, maintenance method, whether parts need to be replaced, etc.

[0090] In this embodiment, different maintenance priority thresholds are set according to the actual situation, dividing maintenance priorities into three levels: high, medium, and low. Based on the maintenance priority level, corresponding maintenance suggestions are generated. For example, for high-priority anomalies, it is recommended to immediately arrange for maintenance personnel to inspect and repair them; for medium-priority anomalies, it is recommended to arrange maintenance in the near future; and for low-priority anomalies, maintenance can be scheduled within the equipment maintenance cycle. Furthermore, the maintenance suggestions may also include whether parts need to be replaced, the specific steps for maintenance, etc.

[0091] The method provided in this application calculates maintenance priority by comprehensively considering multiple factors such as the severity of the anomaly, predicted degradation score, and benefit ratio score. This allows for a comprehensive assessment of the impact of photovoltaic module anomalies from both technical performance and economic benefit dimensions, making the determination of the maintenance sequence more scientific and reasonable. It prioritizes anomalies with significant system impact and good economic benefits, thereby improving maintenance efficiency. Timely maintenance of high-priority anomalies can reduce their continuous impact on the photovoltaic module's power generation performance, improve the overall power generation efficiency of the photovoltaic system, increase power generation, and enhance energy utilization efficiency.

[0092] In some embodiments, prior to step S102, the method further includes: Step S1: Obtain sample data, wherein the sample data includes positive sample pairs and negative sample pairs. The positive sample pairs include sample fusion image pairs of the same photovoltaic module within a second preset time period. The negative sample pairs include sample fusion image pairs of the same photovoltaic module whose shooting time is greater than a time threshold, whose state has changed, and / or different photovoltaic modules.

[0093] In this embodiment, the sample data is a dataset used to train the initial image encoder, containing both positive and negative sample pairs. This data provides the foundation for the image encoder to learn image features, enabling it to distinguish image features under different conditions. Positive sample pairs consist of fused image pairs of samples from the same photovoltaic module within a second preset time period. This means that the two fused images come from the same photovoltaic module, and the shooting time interval is within the second preset time period; they have high similarity in image features. Negative sample pairs include fused image pairs of samples from modules where the shooting time is greater than a time threshold, the state has changed, and / or different photovoltaic modules. That is, image pairs with large shooting time intervals, state changes, or from different modules exhibit significant differences in features.

[0094] In this embodiment, an image acquisition device (such as a camera) is used to photograph photovoltaic modules, acquiring images at different times and under different states. For the same photovoltaic module, multiple images are captured within a second preset time period to form candidate images for positive sample pairs; images captured after a time threshold, with changes in state, or from different photovoltaic modules are used to form candidate images for negative sample pairs. The collected images are fused according to specific requirements. Based on the definitions of positive and negative sample pairs, corresponding sample pairs are constructed from the fused images. Sample fusion images of the same photovoltaic module within the second preset time period are paired to form positive sample pairs; sample fusion images captured after a time threshold, with changes in state, or from different photovoltaic modules are paired to form negative sample pairs.

[0095] In this embodiment, within the temporal contrastive learning framework (for image sequences of the same component captured at different times), the construction of positive and negative sample pairs is based on the principle of "temporal continuity": A positive sample pair (Temporal PositivePair) refers to images of the same component captured within temporally adjacent or short time windows (e.g., t and t+Δt). Although there may be changes in illumination, angle jitter, or slight background differences, semantic consistency should be maintained; therefore, the model is encouraged to bring their embeddings closer together. A negative sample pair (Temporal Negative Pair) refers to images with a larger time span, potentially changing states (e.g., increased dust, slight hot spots), or different components. To prevent the model from ignoring potential anomalous changes, the system pushes these sample embeddings further apart, making the model sensitive to temporal evolution features (trends, early signs of anomalies). In this way, the encoder can automatically learn "normal features that remain stable over time" and "anomalous early sign features that gradually deviate over time," providing a more discriminative dynamic representation for subsequent trend prediction and early anomaly identification.

[0096] Step S2: Train the initial image encoder based on the sample data to obtain a trained image encoder. The loss function of the image encoder includes maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The image encoder includes a window self-attention mechanism module and a shift window attention mechanism module. The window self-attention mechanism module is used to divide the sample fusion image into multiple non-overlapping windows and calculate self-attention for each window. The shift window attention mechanism module is used to perform shift operations between adjacent windows of the sample fusion image so that adjacent windows can interact with each other.

[0097] In this embodiment, the initial image encoder is an untrained image feature extraction model. Its function is to convert the input sample fusion image into a representative feature vector for subsequent similarity calculations and other operations. The loss function measures the difference between the image encoder's output and the expected target. The goal of the loss function is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The window self-attention mechanism module divides the sample fusion image into multiple non-overlapping windows and calculates self-attention within each window. The self-attention mechanism allows the model to focus on the correlation between different positions within a window, thereby capturing local features within the window. This helps the image encoder extract local feature information of the image, and by focusing on the interrelationships between positions within the window, it better understands the local structure and details of the image. The shifted window attention mechanism module performs a shift operation between adjacent windows of the sample fusion image, breaking the independence between windows and enabling information interaction between adjacent windows. This expands the model's perception range of image features, capturing global information across windows. This compensates for the limitation of the window self-attention mechanism, which can only focus on local information, allowing the image encoder to learn both local and global features of the image simultaneously, improving the comprehensiveness and accuracy of feature extraction.

[0098] In this implementation, random initial values ​​can be assigned to various parameters of the initial image encoder (such as weights, biases, etc.) to provide a starting point for the training process. Sample fused image pairs from the sample data are input into the initial image encoder. Feature extraction is performed on the images using a window self-attention mechanism module and a shifted window attention mechanism module to obtain the feature vector of each sample fused image. For positive and negative sample pairs, the similarity between their feature vectors is calculated. Common similarity calculation methods include cosine similarity and Euclidean distance. Based on the requirements of the loss function—maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs—the loss function value under the current model parameters is calculated. Based on the calculated loss function value, the gradient is calculated using the backpropagation algorithm, and the parameters of the image encoder are updated along the opposite direction of the gradient to reduce the loss function value. The process of forward propagation, similarity calculation, loss function calculation, and parameter update is repeated until the loss function value converges or reaches the preset number of training rounds. When the loss function value converges or reaches the preset number of training rounds, the training process stops, and the image encoder at this point is the trained image encoder. It has learned effective image feature representations, which can meet the needs of subsequent tasks.

[0099] Figure 3 A schematic diagram of the structure of an image encoder provided in this application is shown below. Figure 3As shown, the window self-attention mechanism is called window self-attention, and the shift window attention mechanism module is called shift window self-attention. The image encoder as a whole takes bottom-up feature encoding as its core, combined with top-down feature upsampling fusion, and supplemented by cross-layer feature transfer and local feature enhancement to form a deep fusion system of multi-scale features. The input fused image is embedded by patch and normalized by layers, and then feature is encoded in four layers in sequence. Each layer extracts features through the window and shift window attention mechanism modules and fuses them to obtain C2-C5 level features. Before encoding each layer, the output of the previous layer is downsampled to provide input, realizing the downsampling and deepening of features layer by layer. In each layer encoding stage, multiple rounds of SConvBilu module are embedded to perform local feature enhancement. At the same time, the concat module is used to build bidirectional cross-layer fusion. Bidirectional cross-layer fusion includes: receiving the encoded features of the previous layer (such as C2 input to C3 layer, C3 input to C4 layer) and receiving the decoded features of the next layer after upsampling (ConvTrans) (such as C5 upsampled input to C4 layer, C4... Upsampling input to layer C3); from layer 1 to layer 4, a progressive link of “encoding-downsampling” is formed, and from layer 4 to layer 1, a reverse fusion link of “decoding-upsampling” is formed. The two links are intertwined through the concat module, so that each layer of features retains its own encoded details and deep information, and also integrates complementary features from other layers, and finally outputs deep features C5 after multi-layer enhancement and fusion.

[0100] In this embodiment of the image encoder, the input fused image is embedded by Patch and then downsampled and encoded in four layers. Each layer extracts features through window and shift window self-attention and generates C2-C5 level features. Each layer embeds the SConvBilu module to achieve local feature enhancement. At the same time, it relies on the Concat module to fuse the upper layer encoded features and the lower layer Conv upsampled decoded features to construct a bidirectional cross-layer feature fusion link from bottom to top encoding and from top to bottom decoding, so as to realize the deep mining and complementary fusion of multi-scale features, and finally output the fused deep feature C5.

[0101] In this embodiment, the window self-attention module divides the image into segments of varying sizes. The attention mechanism is performed within a window, which differs from performing the attention mechanism directly on the image. The following are the formulas for the self-attention mechanism in each window: ; Where Q is the query vector, K is the key vector, V is the value vector, and d is the scaling factor. B is a learnable relative position offset that preserves the spatial information of the window.

[0102] In this embodiment, Window self-attention, compared to attention mechanisms applied directly to the entire feature map, can significantly reduce computational costs, accurately capture local structured features, and retain the ability to perceive spatial location.

[0103] Shift-window self-attention is a variation of window self-attention, shifting the starting position of the window by M / 2. It moves non-adjacent MxM parts to the right (leftmost parts to right, topmost parts to bottom), allowing even non-adjacent parts to form an MxM window. However, since these positions are not spatially adjacent, a mask is needed when calculating attention scores to avoid including the attention scores for pixels in non-adjacent areas. The purpose of shift-window self-attention is to solve the problem of lost cross-window feature associations caused by the "window isolation" of basic window self-attention, while fully retaining the advantage of "low computational complexity." This allows the model to accurately capture local defect details and recognize global defect structures across windows.

[0104] In this structure, by leveraging the properties of the FPN architecture, a feature pyramid is constructed by upsampling and downsampling and feature fusion of feature maps at different scales. This pyramid consists of "high-resolution shallow details + high semantic deep features," thus solving the problem of false detection / false detection of multi-scale targets (especially small targets).

[0105] In this embodiment, the loss function is used to maximize the similarity of positive sample pairs while minimizing the similarity of negative sample pairs. The loss function can be expressed as: ; in, is the projected embedding vector of the positive sample pair (the i-th sample and its positive pair j in the batch). Similarity function, usually cosine similarity. Or dot product, This represents the temperature parameter. 2N represents the batch size (including both positive and negative sample views). To exclude itself (avoid self-similarity). Total loss. .

[0106] The method provided in this application, through training with positive and negative sample pairs and employing a window self-attention mechanism module and a shifted window attention mechanism module, enables the image encoder to simultaneously learn local and global features of an image, more accurately capturing key information in the image and improving the quality of feature extraction. The introduction of negative sample pairs allows the model to learn differences in image features under different conditions, avoiding overfitting to specific image patterns. This allows the trained image encoder to better adapt when processing new and unseen photovoltaic module images, improving the model's generalization ability.

[0107] The method provided in this application, through self-supervised and contrastive learning, can learn effective image representations even in environments with sparse or no annotations, thereby identifying anomalies. This solves the problem of difficulty in obtaining anomaly samples in existing technologies. Through modal and temporal comparison mechanisms, the system enhances its robustness to changes in component appearance under visible light and infrared, multiple angles, multiple times, and different environments, thereby reducing false alarms and missed alarms. This enables the system to identify local anomalies such as fine cracks, localized hot spots, and dusty edges earlier and accurately output their locations, improving maintenance response efficiency. By fusing anomaly types with environmental and historical data, the system can provide early warnings of efficiency degradation or power loss, rather than only detecting anomalies after output degradation, thus improving the foresight of maintenance. It can realize drone inspections, fixed camera monitoring, automatic analysis, and maintenance scheduling suggestions, reducing labor costs and improving operation and maintenance efficiency.

[0108] Based on the foregoing embodiments, this application further provides an anomaly identification method, which is applied to an anomaly identification system. The anomaly identification system includes: an image acquisition module, a preprocessing and fusion module, a representation learning module, a few-sample target detection module, a trend power prediction module, an active learning module, and an alarm and maintenance decision module.

[0109] The image acquisition module includes a visible light camera, an infrared / thermal imaging camera, and a drone inspection platform. It also includes environmental sensors (light, temperature, humidity) and electrical data acquisition equipment (current, voltage, output power). The preprocessing and modal fusion module is responsible for spatial registration, modal fusion, and standardization of visible light and infrared images. The representation learning module uses the fused images to perform self-supervised temporal contrastive learning to train the image encoder. The few-shot target detection module, based on the Grounding DINO architecture, uses a small number of fine-calibrated samples plus a text prompt dictionary for few-shot or zero-shot fine-tuning to locate abnormal components and defects. The trend power prediction module inputs the embedding sequence output by the image encoder, anomaly detection results, environmental and power data into a time series model (such as Transformer, LSTM, Temporal CNN, or TimeXer) for power prediction. The category expansion and active learning module monitors the distribution of new samples in the embedding space, identifies suspected new anomaly categories, and updates the detection module and prompt dictionary after manual confirmation. The alarm and maintenance suggestion module combines anomaly location, prediction results, location and maintenance cost to generate priority maintenance suggestions.

[0110] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0111] According to the foregoing embodiments, this application provides an anomaly identification device. The various modules and units included in the device can be implemented by a processor in a computer device; of course, they can also be implemented by specific logic circuits. In the implementation process, the processor can be a central processing unit (CPU), a microprocessor unit (MPU), a digital signal processor (DSP), or a field programmable gate array (FPGA), etc.

[0112] This application provides an anomaly detection device. Figure 4 This is a schematic diagram of the structure of an anomaly detection device provided in an embodiment of this application, as shown below. Figure 4 As shown, the anomaly detection device 400 includes: The acquisition module 401 is used to acquire a fused image of the photovoltaic module, wherein the fused image is obtained based on a multimodal image; Extraction module 402 is used to extract features from the fused image based on a pre-established image encoder to obtain image features; The matching module 403 is used to match the image features with the descriptive text vectors in the pre-established mapping relationship to obtain the matching result, wherein the mapping relationship includes: the correspondence between the anomaly type and the descriptive text vector of the anomaly type; The module 404 is used to obtain the anomaly type of the photovoltaic module based on the matching result.

[0113] In some embodiments, the multimodal image includes: a visible light image and an infrared image, and the acquisition module 401 includes: A registration unit is used to register the visible light image and the infrared image; The fusion unit is used to fuse the registered visible light image and the infrared image to obtain an initial fused image; A standardization unit is used to standardize the initial fused image to obtain the fused image.

[0114] In some embodiments, the anomaly detection device 400 further includes: The distance determination module is used to determine the distance between the image feature and each descriptive text vector when the matching result is that no descriptive text vector is matched; An anomaly determination module is used to determine the existence of a new anomaly type when the distance is greater than a distance threshold; The output module is used to output the new exception type so that the user can make a determination; An update module is used to update the mapping relationship based on the new exception type upon user confirmation.

[0115] In some embodiments, the anomaly detection device 400 further includes: A location recognition module is used to identify abnormal locations based on the fused image; The result output module is used to output a language description result based on the exception type and the exception location.

[0116] In some embodiments, the anomaly detection device 400 further includes: The acquisition module is used to acquire environmental data and historical power generation data of the photovoltaic module; The input module is used to input the environmental data, the historical power generation data, and the anomaly type into a pre-established prediction model to obtain power generation prediction data within a first preset time period.

[0117] In some embodiments, the anomaly detection device 400 further includes: A severity determination module is used to determine the severity of an anomaly based on the anomaly type. The attenuation score determination module is used to obtain a predicted attenuation score based on the theoretical power generation data of the photovoltaic module within the first preset time period and the predicted power generation data. The power generation economic loss determination module is used to determine the power generation economic loss within the first preset time period based on the power generation theoretical data and the power generation prediction data. The benefit ratio scoring module is used to obtain a benefit ratio score for economics and maintenance based on the power generation economic loss and the maintenance cost corresponding to the anomaly type. The priority determination module is used to calculate the maintenance priority based on the severity of the anomaly, the predicted attenuation score, and the benefit ratio score. The maintenance suggestion output module is used to output maintenance suggestions based on the maintenance priority.

[0118] In some embodiments, the anomaly detection device 400 further includes: The sample data acquisition module is used to acquire sample data, wherein the sample data includes positive sample pairs and negative sample pairs. The positive sample pairs include sample fusion image pairs of the same photovoltaic module within a second preset time period. The negative sample pairs include sample fusion image pairs of the same photovoltaic module whose shooting time is greater than a time threshold, whose state has changed, and / or different photovoltaic modules. The training module is used to train the initial image encoder based on the sample data to obtain a trained image encoder. The loss function of the image encoder includes maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The image encoder includes a window self-attention mechanism module and a shift window attention mechanism module. The window self-attention mechanism module is used to divide the sample fusion image into multiple non-overlapping windows and calculate self-attention for each window. The shift window attention mechanism module is used to perform shift operations between adjacent windows of the sample fusion image so that adjacent windows can interact with each other.

[0119] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0120] In addition, the anomaly recognition device and the digital human video interaction device mentioned above can be software units, hardware units, or a combination of software and hardware units. They can also be integrated into electronic devices as independent accessories, or exist as independent terminal devices.

[0121] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0122] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 5 As shown, the electronic device of this embodiment may include: at least one processor 30 ( Figure 5 Only one processor 30, memory 31, and computer program 32 stored in memory 31 and executable on at least one processor 30 are shown. When the processor 30 executes the computer program 32, it implements the steps in any of the above method embodiments, or the processor 30 executes the computer program 32 to implement the functions of each module / unit in the above device or system embodiments.

[0123] For example, computer program 32 may be divided into one or more modules / units, one or more of which are stored in memory 31 and executed by processor 30 to complete this application. One or more modules / units may be a series of computer program 32 instruction segments capable of performing a specific function, which describe the execution process of computer program 32 in an electronic device.

[0124] This application also provides a computer-readable storage medium storing a computer program 32, which, when executed by a processor 30, implements the steps described in the above-described method embodiments.

[0125] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0126] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program 32 instructing related hardware. The computer program 32 can be stored in a computer-readable storage medium, and when executed by the processor 30, it can implement the steps of the various method embodiments described above. The computer program 32 includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a terminal, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0127] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0128] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0129] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0130] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0131] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. An anomaly identification method, characterized in that, include: A fused image of a photovoltaic module is obtained, wherein the fused image is obtained based on a multimodal image; Based on a pre-established image encoder, features are extracted from the fused image to obtain image features; The image features are matched with the descriptive text vectors in the pre-established mapping relationship to obtain the matching result, wherein the mapping relationship includes the correspondence between the anomaly type and the descriptive text vector of the anomaly type; The anomaly type of the photovoltaic module is obtained based on the matching results.

2. The method according to claim 1, characterized in that, The multimodal images include: visible light images and infrared images; acquiring the fused image of the photovoltaic module includes: The visible light image and the infrared image are registered; The registered visible light image and the infrared image are fused to obtain an initial fused image; The initial fused image is standardized to obtain the fused image.

3. The method according to claim 1, characterized in that, The method further includes: If the matching result is that no descriptive text vector is matched, the distance between the image feature and each descriptive text vector is determined; If the distance is greater than the distance threshold, it is determined that a new anomaly type exists; The new exception type is output so that the user can make a determination. Upon user confirmation, the mapping relationship is updated based on the new exception type.

4. The method according to claim 1, characterized in that, The method further includes: Anomalies are identified based on the fused image; Output a language description result based on the anomaly type and the anomaly location.

5. The method according to claim 1, characterized in that, The method further includes: Obtain environmental data and historical power generation data of the photovoltaic modules; The environmental data, the historical power generation data, and the anomaly type are input into a pre-established prediction model to obtain power generation prediction data within a first preset time period.

6. The method according to claim 5, characterized in that, The method further includes: Determine the severity of the anomaly based on the anomaly type; A predicted degradation score is obtained based on the theoretical power generation data of the photovoltaic module within the first preset time period and the predicted power generation data. Based on the theoretical power generation data and the predicted power generation data, the economic loss of power generation within the first preset time period is obtained. A benefit-to-cost ratio score for economics and maintenance is obtained based on the maintenance costs corresponding to the power generation economic losses and anomaly types. The maintenance priority is calculated based on the severity of the anomaly, the predicted degradation score, and the benefit ratio score. Maintenance recommendations are generated based on the aforementioned maintenance priorities.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: Acquire sample data, wherein the sample data includes: positive sample pairs and negative sample pairs, the positive sample pairs include: sample fusion image pairs of the same photovoltaic module within a second preset time period, and the negative sample pairs include: sample fusion image pairs of the same photovoltaic module whose shooting time is greater than a time threshold, whose state has changed, and / or different photovoltaic modules; The initial image encoder is trained based on the sample data to obtain a trained image encoder. The loss function of the image encoder includes maximizing the similarity of positive sample pairs and minimizing the similarity of negative sample pairs. The image encoder includes a window self-attention mechanism module and a shift window attention mechanism module. The window self-attention mechanism module is used to divide the sample fusion image into multiple non-overlapping windows and calculate self-attention for each window. The shift window attention mechanism module is used to perform shift operations between adjacent windows of the sample fusion image so that adjacent windows can interact with each other.

8. An anomaly detection device, characterized in that, include: An acquisition module is used to acquire a fused image of a photovoltaic module, wherein the fused image is obtained based on a multimodal image; The extraction module is used to extract features from the fused image based on a pre-established image encoder to obtain image features; The matching module is used to match the image features with the descriptive text vectors in the pre-established mapping relationship to obtain the matching result, wherein the mapping relationship includes the correspondence between the anomaly type and the descriptive text vector of the anomaly type; The module is used to obtain the anomaly type of the photovoltaic module based on the matching results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images