Fault information determination method and device, storage medium and electronic device

By splicing visible and infrared light images and combining convolutional neural network structures of multiple scales, the regional feature correlation in photovoltaic fault images is solved, and the problem of low recognition accuracy in photovoltaic fault image diagnosis is achieved, achieving more efficient faults Identify.

CN118918439APending Publication Date: 2024-11-08HUANENG JILIN POWER GENERATION JIUTAI ELECTRIC FACTORY +2
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411044417.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

The recognition accuracy rate in existing photovoltaic fault image diagnosis tasks is low, especially because the performance area of ​​photovoltaic faults on the image is very small and the local performance of infrared light images is similar, resulting in the limitations of commonly used image recognition technologies in photovoltaic fault diagnosis.

Method used

By obtaining the stitching of visible light images and infrared light images, the feature matrix of the stitching image is determined, and inputting it into multiple convolutional neural networks to obtain the regional feature matrix, calculate the correlation relationship matrix between the regional feature matrix, and combine it with multiple scales Parallel convolutional neural network structure to mine the correlation of different regional features to achieve better photovoltaic fault recognition.

Benefits of technology

The recognition accuracy of photovoltaic fault images is improved, and the problem of low recognition accuracy in photovoltaic fault diagnosis is overcome by commonly used image recognition technology, which achieves more effective fault recognition.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118918439A_ABST
    Figure CN118918439A_ABST
Patent Text Reader

Abstract

The invention discloses a fault information determination method and device, a storage medium and an electronic device, and the method comprises the steps: obtaining a visible light image and an infrared light image corresponding to a target object, and splicing the visible light image and the infrared light image to obtain a spliced image; determining a first feature matrix of the spliced image, and inputting the first feature matrix into a plurality of convolutional neural networks to obtain a plurality of regional feature matrixes corresponding to the spliced image, with different convolutional neural networks having different convolution kernels; and determining an association relationship matrix between every two regional feature matrixes, and determining the fault information of the target object according to the plurality of association relationship matrices, through the embodiment of the application, the problem of low identification accuracy in a photovoltaic fault image diagnosis task in related technologies is solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of communications, and in particular, to a method and device for determining fault information, a storage medium, and an electronic device. Background Art

[0002] Traditional photovoltaic fault detection mainly relies on manual inspection, which is inefficient and has high labor costs. Therefore, it is very important to rely on unmanned inspection technology to achieve photovoltaic fault detection. In unmanned inspection technology, one of the most important tasks is to realize automatic identification of photovoltaic faults based on image input and artificial intelligence technology.

[0003] There are two main types of images used for automatic photovoltaic fault identification: visible light images taken directly and infrared images taken by infrared cameras. Since visible light images have more detailed texture information, and infrared images can capture internal damage and faults that are invisible to the outside through thermal radiation, in recent studies, researchers have focused on using both types of images as input at the same time, and using deep learning methods of general computer vision to combine the advantages of these two types of images to achieve better photovoltaic fault image recognition.

[0004] However, since some photovoltaic faults appear in very small areas on the image and some fault types have very similar local manifestations in infrared images, these challenges make commonly used image recognition technologies, such as ResNet, face limitations in photovoltaic fault image diagnosis tasks.

[0005] Therefore, in the related art, there is a problem of low recognition accuracy in photovoltaic fault image diagnosis tasks.

[0006] Therefore, it is necessary to improve the related technology to overcome the above-mentioned defects in the related technology. Summary of the invention

[0007] The embodiments of the present application provide a method and device for determining fault information, a storage medium, and an electronic device to at least solve the problem of low recognition accuracy in photovoltaic fault image diagnosis tasks in the related art.

[0008] According to one aspect of an embodiment of the present application, a method for determining fault information is provided, including: obtaining a visible light image and an infrared light image corresponding to a target object, and splicing the visible light image and the infrared light image to obtain a spliced ​​image; determining a first feature matrix of the spliced ​​image, and inputting the first feature matrix into multiple convolutional neural networks respectively to obtain multiple regional feature matrices corresponding to the spliced ​​image, wherein different convolutional neural networks have different convolution kernels; determining an association relationship matrix between each pair of regional feature matrices, and determining the fault information of the target object based on the multiple association relationship matrices.

[0009] In an exemplary embodiment, determining the fault information of the target object based on the multiple association relationship matrices includes: adding the multiple regional feature matrices to obtain a second feature matrix; multiplying the second feature matrix by each association relationship matrix to obtain multiple third feature matrices; and determining the fault information of the target object based on the multiple third feature matrices.

[0010] In an exemplary embodiment, determining the fault information of the target object according to the multiple third characteristic matrices includes: concatenating the multiple third characteristic matrices to obtain a fourth characteristic matrix; and determining the fault information of the target object according to the fourth characteristic matrix.

[0011] In an exemplary embodiment, determining the association relationship matrix between the pairwise regional feature matrices includes: calculating the cosine similarity of the pairwise regional feature matrices; and determining the association relationship matrix between the pairwise regional feature matrices according to the cosine similarity.

[0012] In an exemplary embodiment, the first feature matrix is ​​respectively input into multiple convolutional neural networks to obtain multiple regional feature matrices corresponding to the stitched image, including: inputting the first feature matrix into a partial network structure in each convolutional neural network, so that the partial network structure outputs a fifth feature matrix according to the first feature matrix, wherein the partial network structure includes: a convolution layer, an activation layer, and a pooling layer; inputting the fifth feature matrix into an adaptive pooling layer in each convolutional neural network, so that the adaptive pooling layer performs pooling and scaling on the fifth feature matrix to obtain a sixth feature matrix; inputting the sixth feature matrix into a fully connected layer in each convolutional neural network, so that the fully connected layer outputs the regional feature matrix according to the sixth feature matrix.

[0013] In an exemplary embodiment, determining a first feature matrix of the stitched image includes: performing feature extraction on the stitched image to determine a plurality of feature vectors of the stitched image; transforming each feature vector to obtain a plurality of sub-feature matrices; and merging the plurality of sub-feature matrices to obtain the first feature matrix.

[0014] According to another aspect of an embodiment of the present application, a device for determining fault information is also provided, including: a stitching module, used to acquire a visible light image and an infrared light image corresponding to a target object, and stitch the visible light image and the infrared light image to obtain a stitched image; a first determination module, used to determine a first feature matrix of the stitched image, and input the first feature matrix into multiple convolutional neural networks respectively to obtain multiple regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels; a second determination module, used to determine a correlation relationship matrix between each pair of regional feature matrices, and determine the fault information of the target object based on the multiple correlation relationship matrices.

[0015] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining fault information when running.

[0016] According to another aspect of an embodiment of the present application, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the method for determining fault information through the computer program.

[0017] According to another aspect of the embodiments of the present application, a computer program product is provided, including a computer program, wherein the computer program is executed by a processor to perform the above-mentioned method for determining fault information.

[0018] Through the embodiment of the present application, the visible light image and infrared light image corresponding to the target object are obtained, and the visible light image and the infrared light image are spliced ​​to obtain a spliced ​​image; the first feature matrix of the spliced ​​image is determined, and the first feature matrix is ​​respectively input into multiple convolutional neural networks to obtain multiple regional feature matrices corresponding to the spliced ​​image, wherein different convolutional neural networks have different convolution kernels; the correlation relationship matrix between the two regional feature matrices is determined, and the fault information of the target object is determined based on the multiple correlation relationship matrices. That is, in the embodiment of the present application, the feature information of different sizes of regions on the image is determined through parallel convolutional neural network structures of multiple scales, and the correlation between different regional feature matrices is mined, thereby achieving better photovoltaic image fault recognition and solving the problem of low recognition accuracy in photovoltaic fault image diagnosis tasks in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The exemplary embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0020] Figure 1 It is a hardware structure block diagram of a computer device of a method for determining fault information in an embodiment of the present application;

[0021] Figure 2 is a flow chart of a method for determining fault information according to an embodiment of the present application;

[0022] Figure 3 is a schematic diagram of a method for determining fault information according to an embodiment of the present application;

[0023] Figure 4 It is a structural block diagram of a device for determining fault information according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms and "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] The method embodiments provided in the embodiments of the present application can be executed in a computer device or a similar computing device. Taking running on a computer device as an example, Figure 1 FIG. 1 is a hardware structure block diagram of a computer device for determining fault information in an embodiment of the present application. Figure 1 As shown, the computer device may include one or more ( Figure 1Only one is shown in the figure) processor 102 (processor 102 may include but is not limited to a microprocessor (Microprocessor Unit, referred to as MPU) or a programmable logic device (Programmable logic device, referred to as PLD)) and a memory 104 for storing data. In an exemplary embodiment, the above-mentioned computer device may also include a transmission device 106 and an input and output device 108 for communication functions. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer device. Figure 1 More or fewer components as shown, or with Figure 1 Equivalent functions or comparisons shown Figure 1 A different configuration with more features is shown.

[0027] The memory 104 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the method for determining fault information in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, which is equivalent to implementing the above method. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0028] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device. In one embodiment, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0029] In this embodiment, a method for determining fault information is provided. Figure 2 is a flowchart of a method for determining fault information according to an embodiment of the present application, which is run in the above-mentioned computer device, and the process includes the following steps:

[0030] Step S202, obtaining a visible light image and an infrared light image corresponding to the target object, and splicing the visible light image and the infrared light image to obtain a spliced ​​image;

[0031] Optionally, the stitched image can be obtained in the following manner: establish a first channel matrix for the visible light image, and establish a second channel matrix for the infrared light image; stitch the first channel matrix and the second channel matrix in the channel dimension to obtain a stitched channel matrix; and determine the stitched image based on the stitched channel matrix.

[0032] Step S204, determining a first feature matrix of the stitched image, and inputting the first feature matrix into a plurality of convolutional neural networks respectively, to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels;

[0033] It should be noted that the convolution kernels of the above-mentioned multiple convolutional neural networks are different. For example, the convolution kernel of the convolutional neural network can be The above examples are only for a better understanding of the embodiments of the present application. The embodiments of the present application do not limit the size of the convolution kernel.

[0034] Step S206, determining a correlation matrix between two regional feature matrices, and determining the fault information of the target object according to a plurality of the correlation matrices.

[0035] Through the above steps, the visible light image and infrared light image corresponding to the target object are obtained, and the visible light image and the infrared light image are spliced ​​to obtain a spliced ​​image; the first feature matrix of the spliced ​​image is determined, and the first feature matrix is ​​respectively input into multiple convolutional neural networks to obtain multiple regional feature matrices corresponding to the spliced ​​image, wherein different convolutional neural networks have different convolution kernels; the correlation relationship matrix between the two regional feature matrices is determined, and the fault information of the target object is determined based on the multiple correlation relationship matrices. That is, in the embodiment of the present application, the feature information of different sizes of regions on the image is determined by parallel convolutional neural network structures of multiple scales, and the correlation between different regional feature matrices is mined, thereby achieving better photovoltaic image fault recognition and solving the problem of low recognition accuracy in photovoltaic fault image diagnosis tasks in related technologies.

[0036] Optionally, in order to better understand the above step S206, the above step S206 can be implemented in the following way: add the multiple regional feature matrices to obtain a second feature matrix; multiply the second feature matrix with each association relationship matrix respectively to obtain multiple third feature matrices; determine the fault information of the target object according to the multiple third feature matrices.

[0037] Specifically, determining the fault information of the target object according to the multiple third characteristic matrices includes: concatenating the multiple third characteristic matrices to obtain a fourth characteristic matrix; and determining the fault information of the target object according to the fourth characteristic matrix.

[0038] That is to say, the number of feature matrices in multiple regions is 4, and the convolution kernel of the convolutional neural network can be In the case of 2×2, the second characteristic matrix DF is determined by the following formula all : in, DF 2×2 is the feature matrix of the above multiple regions.

[0039] The second characteristic matrix is ​​multiplied by each association relationship matrix to obtain a plurality of third characteristic matrices SDF i,j ,include: Among them, W i,j and b i,j are training parameters.

[0040] The plurality of third characteristic matrices are concatenated to obtain a fourth characteristic matrix F, comprising:

[0041]

[0042] Determining the fault information of the target object according to the fourth characteristic matrix includes: inputting the fourth characteristic inverse matrix into a linear discriminator, and determining the fault information of the target object through information output by the linear discriminator, for example: the identified fault categories are represented by 0-7, which are: 0-no fault, 1-dust accumulation of components, 2-obstruction by weeds, 3-partial obstruction, 4-open circuit of group string, 5-damage of components, 6-diode fault, 7-hot spot fault.

[0043] Optionally, determining the association relationship matrix between the pairwise regional feature matrices includes: calculating the cosine similarity of the pairwise regional feature matrices; and determining the association relationship matrix between the pairwise regional feature matrices according to the cosine similarity.

[0044] In the embodiment of the present application, the correlation relationship between every two feature matrices is learned by calculating the cosine similarity between every two feature matrices.

[0045] Optionally, the first feature matrix is ​​respectively input into multiple convolutional neural networks to obtain multiple regional feature matrices corresponding to the stitched image, including: inputting the first feature matrix into a partial network structure in each convolutional neural network, so that the partial network structure outputs a fifth feature matrix according to the first feature matrix, wherein the partial network structure includes: a convolution layer, an activation layer, and a pooling layer; inputting the fifth feature matrix into an adaptive pooling layer in each convolutional neural network, so that the adaptive pooling layer performs pooling and scaling on the fifth feature matrix to obtain a sixth feature matrix; inputting the sixth feature matrix into a fully connected layer in each convolutional neural network, so that the fully connected layer outputs the regional feature matrix according to the sixth feature matrix.

[0046] In addition, before outputting the fifth characteristic matrix, the network depth can be deepened through a loop. For example, when the number of loops is 2, that is, the partial network structure is used twice, the fifth characteristic matrix is ​​obtained after the partial network structure loop is completed.

[0047] The above-mentioned adaptive pooling layer can be understood as an AdaptivePool layer. The AdaptivePool layer is a variant of a pooling layer in deep learning, which can automatically adjust the size of the pooling operation according to the size of the input data, thereby adapting to input data of different sizes. This adaptability makes the model more flexible and applicable to input data of different sizes, and then the fifth feature matrix is ​​pooled and scaled through the above-mentioned adaptive pooling layer to obtain the sixth feature matrix.

[0048] In an exemplary embodiment, determining a first feature matrix of the stitched image includes: performing feature extraction on the stitched image to determine a plurality of feature vectors of the stitched image; transforming each feature vector to obtain a plurality of sub-feature matrices; and merging the plurality of sub-feature matrices to obtain the first feature matrix.

[0049] Optionally, select a suitable feature extraction method, such as color histogram, edge detection, texture features, etc.; convert the extracted features into a matrix form, for example, convert the feature vector into a matrix through the numpy library; merge the feature matrices of all images into a first feature matrix, wherein each row of the first feature matrix represents a feature of an image.

[0050] In order to better understand the process of the above-mentioned method for determining fault information, the process of the above-mentioned method for determining fault information is described below in combination with an optional embodiment, but it is not used to limit the technical solution of the embodiment of the present application.

[0051] In an embodiment of the present application, a method for determining fault information is provided. For example, a visible light image and a corresponding infrared light image are used as input, and the output is a fault category. The identified fault category is represented by 0-7, which are: 0-no fault, 1-dust accumulation of components, 2-weed shading, 3-partial shading, 4-string open circuit, 5-component damage, 6-diode fault, 7-hot spot fault, Figure 3 is a schematic diagram of a method for determining fault information according to an embodiment of the present application, such as Figure 3 As shown, the details are as follows:

[0052] First, the visible light image and the infrared light image are concatenated in the channel dimension, and then the overall feature matrix FeatureMap of the visible light image and the infrared light image is learned through the deep convolutional neural network model ResNet-50. The size of FeatureMap can be expressed as C×H×W, that is, FeatureMap∈R C×H×W .

[0053] Then, the FeatureMap is input into different convolution kernel sizes (the convolution kernel sizes are In the convolutional neural network of 2×2 and 2×2, the features of image regions of different sizes are learned by changing the size of the convolution kernel.

[0054] It should be noted that the structure of each convolutional neural network with different convolution kernel sizes is the same. Take the convolutional neural network as an example:

[0055] 1) Adoption The basic three-layer network structure of convolution layer-activation layer-2×2 pooling layer is cycled to deepen the network depth. The number of cycles is k. For example, when k=2, the network structure is Convolutional layer - Activation layer - 2×2 pooling layer - Convolution layer-activation layer-2×2 pooling layer, that is, the three-layer network structure is used twice. After the basic network structure cycle is completed, the feature matrix is ​​obtained

[0056]

[0057] 2) Through the AdaptivePool layer, By pooling, we can scale

[0058]

[0059] 3) Finally, the dimension length is unified to L through the fully connected layer, that is:

[0060]

[0061] Among them, W1 and b1 are training parameters.

[0062] Similarly, we can get DF 2×2 ∈R L ,in, DF 2×2 ∈R L It can represent regional features of different scales.

[0063] Next, DF 2×2 The four matrices are multiplied two by two, and the association between each two feature matrices is learned by calculating the cosine similarity between each two feature matrices:

[0064]

[0065] It should be noted that by mining the correlation between regional features of different scales through the wide-area correlation structure, fault types can be better identified and distinguished.

[0066] Further, DF 2×2 The four matrices are added together to obtain all the information about features of different scales:

[0067]

[0068] DF all Multiply the six association matrices obtained by dot product and pass through a fully connected layer to assign the association back to the feature information itself:

[0069]

[0070] Among them, W i,j and b i,j are training parameters.

[0071] Finally, these matrices containing correlation information and features of different scales are concatenated and passed through a linear discriminator to output y, which represents the fault type to which the image belongs:

[0072]

[0073] y=Softmax(W2 F+b2)∈R 8 ;

[0074] In the embodiment of the present application, a parallel convolutional neural network structure of multiple scales is used to learn the feature information of different area sizes on the image. The correlation between feature matrices of different scales is mined through the designed wide-area correlation structure, thereby achieving better photovoltaic image fault recognition.

[0075] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROX / RAX, a disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0076] In this embodiment, a device for determining fault information is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0077] Figure 4 is a structural block diagram of a device for determining fault information according to an embodiment of the present application, the device comprising:

[0078] A stitching module 42, used to obtain a visible light image and an infrared light image corresponding to the target object, and stitch the visible light image and the infrared light image to obtain a stitched image;

[0079] A first determination module 44 is used to determine a first feature matrix of the stitched image, and input the first feature matrix into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels;

[0080] The second determination module 46 is used to determine a correlation matrix between two regional feature matrices, and determine the fault information of the target object according to a plurality of the correlation matrices.

[0081] Through the above-mentioned device, the visible light image and infrared light image corresponding to the target object are obtained, and the visible light image and the infrared light image are spliced ​​to obtain a spliced ​​image; the first feature matrix of the spliced ​​image is determined, and the first feature matrix is ​​respectively input into multiple convolutional neural networks to obtain multiple regional feature matrices corresponding to the spliced ​​image, wherein different convolutional neural networks have different convolution kernels; the correlation relationship matrix between the two regional feature matrices is determined, and the fault information of the target object is determined according to the multiple correlation relationship matrices. That is, in the embodiment of the present application, the feature information of different sizes of regions on the image is determined by parallel convolutional neural network structures of multiple scales, and the correlation between different regional feature matrices is mined, so as to achieve better photovoltaic image fault recognition and solve the problem of low recognition accuracy in photovoltaic fault image diagnosis tasks in related technologies.

[0082] In an exemplary embodiment, the second determination module 46 is used to add the multiple regional feature matrices to obtain a second feature matrix; multiply the second feature matrix with each association relationship matrix to obtain multiple third feature matrices; and determine the fault information of the target object based on the multiple third feature matrices.

[0083] In an exemplary embodiment, the second determination module 46 is configured to concatenate the plurality of third characteristic matrices to obtain a fourth characteristic matrix; and determine the fault information of the target object according to the fourth characteristic matrix.

[0084] In an exemplary embodiment, the second determination module 46 is configured to calculate the cosine similarity of the pairwise regional feature matrices; and determine the correlation relationship matrix between the pairwise regional feature matrices according to the cosine similarity.

[0085] In an exemplary embodiment, the first determination module 44 is used to input the first feature matrix into a partial network structure in each convolutional neural network so that the partial network structure outputs a fifth feature matrix based on the first feature matrix, wherein the partial network structure includes: a convolution layer, an activation layer, and a pooling layer; input the fifth feature matrix into the adaptive pooling layer in each convolutional neural network so that the adaptive pooling layer performs pooling and scaling on the fifth feature matrix to obtain a sixth feature matrix; input the sixth feature matrix into the fully connected layer in each convolutional neural network so that the fully connected layer outputs the regional feature matrix based on the sixth feature matrix.

[0086] In an exemplary embodiment, the first determination module 44 is used to perform feature extraction on the stitched image to determine multiple feature vectors of the stitched image; transform each feature vector to obtain multiple sub-feature matrices; and merge the multiple sub-feature matrices to obtain the first feature matrix.

[0087] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned method for determining fault information when running.

[0088] Optionally, in this embodiment, the storage medium may be configured to store a computer program for performing the following steps:

[0089] S1, acquiring a visible light image and an infrared light image corresponding to a target object, and splicing the visible light image and the infrared light image to obtain a spliced ​​image;

[0090] S2, determining a first feature matrix of the stitched image, and inputting the first feature matrix into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels;

[0091] S3, determining a correlation matrix between two regional feature matrices, and determining the fault information of the target object according to a plurality of the correlation matrices.

[0092] Optionally, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0093] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0094] An embodiment of the present application further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0095] Optionally, in this embodiment, the processor may be configured to perform the following steps through a computer program:

[0096] S1, acquiring a visible light image and an infrared light image corresponding to a target object, and splicing the visible light image and the infrared light image to obtain a spliced ​​image;

[0097] S2, determining a first feature matrix of the stitched image, and inputting the first feature matrix into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels;

[0098] S3, determining a correlation matrix between two regional feature matrices, and determining the fault information of the target object according to a plurality of the correlation matrices.

[0099] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0100] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any one of the above method embodiments are implemented.

[0101] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above method embodiments are implemented.

[0102] Optionally, in this embodiment, the computer program product may be configured to perform the following steps through a computer program:

[0103] S1, acquiring a visible light image and an infrared light image corresponding to a target object, and splicing the visible light image and the infrared light image to obtain a spliced ​​image;

[0104] S2, determining a first feature matrix of the stitched image, and inputting the first feature matrix into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels;

[0105] S3, determining a correlation matrix between two regional feature matrices, and determining the fault information of the target object according to a plurality of the correlation matrices.

[0106] An embodiment of the present application also provides a computer program, which includes computer instructions, which are stored in a computer-readable storage medium; a processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps in any one of the above method embodiments.

[0107] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0108] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that herein, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present application is not limited to any specific combination of hardware and software.

[0109] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principles of the present application shall be included in the protection scope of the present application.

Claims

1. A method for determining fault information, characterized in that: include: Acquire a visible light image and an infrared light image corresponding to the target object, and stitch the visible light image and the infrared light image to obtain a stitched image; Determine a first feature matrix of the stitched image, and input the first feature matrix into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels; Determine a correlation matrix between the feature matrices of each region, and determine the fault information of the target object according to a plurality of the correlation matrices.

2. The method for determining fault information according to claim 1, characterized in that: Determining the fault information of the target object according to the plurality of association relationship matrices includes: Adding the plurality of regional feature matrices to obtain a second feature matrix; Multiplying the second characteristic matrix by each association relationship matrix respectively to obtain a plurality of third characteristic matrices; Fault information of the target object is determined according to the plurality of third feature matrices.

3. The method for determining fault information according to claim 2, characterized in that: Determining the fault information of the target object according to the plurality of third characteristic matrices includes: splicing the plurality of third characteristic matrices to obtain a fourth characteristic matrix; Fault information of the target object is determined according to the fourth characteristic matrix.

4. The method for determining fault information according to claim 1, characterized in that: Determine the correlation matrix between the feature matrices of each region, including: Calculating the cosine similarity of the pairwise regional feature matrices; The correlation relationship matrix between the pairwise regional feature matrices is determined according to the cosine similarity.

5. The method for determining fault information according to claim 1, characterized in that: The first feature matrix is ​​input into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the spliced ​​image, including: Inputting the first feature matrix into a partial network structure in each convolutional neural network, so that the partial network structure outputs a fifth feature matrix according to the first feature matrix, wherein the partial network structure includes: a convolution layer, an activation layer, and a pooling layer; Inputting the fifth feature matrix into the adaptive pooling layer in each convolutional neural network, so that the adaptive pooling layer performs pooling scaling on the fifth feature matrix to obtain a sixth feature matrix; The sixth feature matrix is ​​input into a fully connected layer in each convolutional neural network, so that the fully connected layer outputs the regional feature matrix according to the sixth feature matrix.

6. The method for determining fault information according to claim 1, characterized in that: Determining a first characteristic matrix of the stitched image includes: Performing feature extraction on the stitched image to determine a plurality of feature vectors of the stitched image; Transform each eigenvector to obtain multiple sub-eigenmatrices; The multiple sub-feature matrices are combined to obtain the first feature matrix.

7. A device for determining fault information, characterized in that: include: A stitching module, used to obtain a visible light image and an infrared light image corresponding to a target object, and stitch the visible light image and the infrared light image to obtain a stitched image; A first determination module is used to determine a first feature matrix of the stitched image, and input the first feature matrix into a plurality of convolutional neural networks respectively to obtain a plurality of regional feature matrices corresponding to the stitched image, wherein different convolutional neural networks have different convolution kernels; The second determination module is used to determine the correlation relationship matrix between the feature matrices of each region, and determine the fault information of the target object according to a plurality of the correlation relationship matrices.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method described in any one of claims 1 to 6 when executed.

9. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 6 through the computer program.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Solar cell panel defect detection method based on multi-scale joint convolutional neural network

    CN110689011A

  • Dual-light image target detection method, system, equipment and medium

    CN117422858A

  • Pavement crack detection method, medium, and system

    WO2024092968A1