Snow accumulation determination device, snow accumulation determination method, and snow accumulation determination program

Abstract

To provide a snow accumulation determination device, a snow accumulation determination method, and a snow accumulation determination program which highly accurately determine a snow accumulation state on a solar panel for images of a camera view point not included in learning data.SOLUTION: According to one embodiment, a snow accumulation determination device includes an acquisition unit that acquires an image including a solar panel at a first time. Additionally, the snow accumulation determination device includes a learning model that uses the image including the solar panel at the first time as input and uses a result obtained by identifying a region where snow accumulation exists on the image including the solar panel at the first time as output. Furthermore, the snow accumulation determination device includes a learning unit that learns the learning model by using pseudo-snow accumulation images in which pixels of the image including the solar panel at a second time are converted into pixels corresponding to snow accumulation and pseudo-class label images applied with corresponding object labels. The snow accumulation determination device also includes a determination unit that determines the state of snow accumulation on the solar panel on the basis of the result obtained by identifying the region.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] An embodiment of the present invention relates to a snow accumulation determination device, a snow accumulation determination method, and a snow accumulation determination program. [Background technology]

[0002] Photovoltaic power generation requires a guarantee of the amount of power generated during actual operation of the power plant, compared to the power generation forecast made during the operation plan. In areas with snowfall, the amount of power generated becomes unstable due to snow accumulation on the solar panels, so it is desirable to exclude snow accumulation from the output guarantee.

[0003] Snow gauges are sometimes used to determine snowfall, but because their installation costs are high, meteorological data is used to make the determination. However, snow remains on solar panels even after snowfall, and even on sunny days, the amount of power generated becomes unstable due to the remaining snow, so determining the amount of snow based on meteorological data lacks reliability.

[0004] Currently, there are two known methods for determining snow accumulation on solar panels: a system that compares the amount of power generated by solar panels to determine snow accumulation, and a system that uses camera images to determine the snow accumulation on roofs and road surfaces. The latter system determines the snow accumulation on target roofs and road surfaces from brightness information on the images. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2019-154321 A Summary of the Invention [Problem to be solved by the invention]

[0006] In addition, technology is being developed that uses machine learning to determine the state of snow on solar panels, and there is a demand for technology that can make more accurate determinations.

[0007] Therefore, an embodiment of the present invention provides a snow accumulation determination device, a snow accumulation determination method, and a snow accumulation determination program that determine the snow accumulation condition on a solar panel with high accuracy for images from a camera viewpoint that are not included in the learning data. [Means for solving the problem]

[0008] According to one embodiment, the snow accumulation determination device includes an acquisition unit that acquires an image including a solar panel at a first time. The snow accumulation determination device further includes a learning model that receives an image including a solar panel at the first time as an input and outputs a result of identifying an area where snow is present on the image including the solar panel at the first time. The snow accumulation determination device further includes a learning unit that trains the learning model using a pseudo-snow accumulation image in which pixels of an image including the solar panel at a second time that is earlier than the first time are converted into pixels corresponding to snow accumulation, and a pseudo-class label image in which the pixels constituting the image are given labels of corresponding subjects. The snow accumulation determination device further includes a determination unit that determines the state of snow accumulation on the solar panel based on the result of identifying the area. [Brief description of the drawings]

[0009] [Figure 1] 1 is a schematic configuration diagram of a snow accumulation determination device according to a first embodiment. [Diagram 2] 3 is another schematic configuration diagram of the snow accumulation determination device in the first embodiment. FIG. [Diagram 3] 1 is a functional block diagram of a snow accumulation determination device according to a first embodiment. FIG. [Figure 4] 4 is a block diagram showing details of a pseudo-snow-covered image generating unit in the first embodiment. FIG. [Diagram 5] 5 is an example of a flowchart for determining snow accumulation in the first embodiment. [Figure 6] 5 is an example of a flowchart for generating a pseudo snow-covered image in the first embodiment. [Figure 7]10 is a comparative example between a past image and a class label image corresponding to the past image in the first embodiment. [Figure 8] 10 is a comparative example between a pseudo-snow-covered image in the first embodiment and a pseudo class label image corresponding to the pseudo-snow-covered image. [Figure 9] 13 is a block diagram showing details of a pseudo-snow-covered image generating section in a modified example of the first embodiment. FIG. [Figure 10] FIG. 11 is a block diagram showing details of a pseudo-snow-covered image generating unit in the second embodiment. [Figure 11] 13 is an example of a flowchart for generating a second pseudo snow-covered image in the second embodiment. [Figure 12] FIG. 11 is a hardware configuration diagram of a snow accumulation determination device according to a third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. The present embodiment does not limit the present invention. The drawings are schematic or conceptual, and the ratio of each part is not necessarily the same as the actual one. In the specification and drawings, elements similar to those described above with respect to the previous drawings are given the same reference numerals, and detailed descriptions are omitted as appropriate.

[0011] (First embodiment) FIG. 1 is a schematic diagram of a snow accumulation determination device according to the first embodiment.

[0012] The snow accumulation determination device 2 in this embodiment is installed, for example, in the building 400 of the solar power plant 1, and performs snow accumulation determination using machine learning for the snow accumulation condition on the solar panels 100 in the solar power plant 1. The snow accumulation determination device 2 can perform snow accumulation determination using images even in a case where the learning data does not include an image including snow accumulation on the solar panels 100 captured by the imaging device 6, such as before the solar power plant 1 starts operation.

[0013] In FIG. 1, a string configuration 500 of solar panels 100 includes a plurality of solar panels 100 connected in series. Furthermore, in FIG. 1, an array configuration 700 of solar panels 100 includes a plurality of string configurations 500 connected in parallel. The array configuration 700 shown in FIG. 1 includes a plurality of string configurations 500 extending in the horizontal direction and adjacent to each other in the depth direction, and each string configuration 500 includes a plurality of solar panels 100 arranged in one or more rows in the horizontal direction. In the present embodiment, snow accumulation determination is performed, for example, using an image of a portion of the array configuration 700 captured by the imaging device 6. In the present embodiment, snow accumulation analysis can be applied not only to an image of a portion of the array configuration 700 captured by the imaging device 6, but also to an image of a portion of the solar panel 100 or the string configuration 500 captured by the imaging device 6.

[0014] The imaging device 6 is, for example, at least one surveillance camera installed on the roof of a house or in the solar power plant 1. The imaging device 6 is also installed on a moving object such as a robot or a drone. In this case, the imaging device 6 and the snow accumulation determination device 2 may be electrically connected by wireless connection. The imaging device 6 is, for example, a monocular camera or a fisheye camera equipped with a lens and an image sensor. The imaging device 6 is, for example, a visible light camera that captures reflected light using visible light, an infrared camera, a camera capable of acquiring a depth map, a distance sensor, or the like.

[0015] The image captured by the imaging device 6 may be either a moving image or a still image. When capturing a moving image, subsequent processing may be performed for each frame, or subsequent processing may be performed based on at least one of the multiple frames. Subsequent processing may also be performed based on an image obtained by performing arithmetic processing such as averaging processing on multiple frames, or on an image obtained by performing arithmetic processing such as panoramic synthesis on multiple images acquired from multiple surveillance cameras. The image captured by the imaging device 6 may be a three-channel color image composed of an R image, a G image, and a B image, or may have a color system different from the RGB color system by a color system conversion process. The different color system may be, for example, the HLS color system. Alternatively, it may be a monochrome image obtained by multiplying each channel information of a color image by a predetermined coefficient to convert it into a single channel, and the pixel Y constituting the monochrome image may be calculated by, for example, formula (1). Y=0.2126×R+0.7512×G+0.0722×B (1)

[0016] Furthermore, the snow accumulation determination device 2 may share the imaging device 6 used in the monitoring system 3. In this case, the snow accumulation determination device 2 may acquire images captured by the imaging device 6 via the network device 20, or may acquire images of the imaging device 6 stored in the memory unit 4 of the monitoring system 3. By sharing the imaging device 6 and other equipment, the snow accumulation determination device 2 can reduce the introduction costs of the equipment.

[0017] In this example, the result of snow accumulation determination made by the snow accumulation determination device 2 is displayed on the display device 5 of the monitoring system 3, but for example, a display device may be connected to the snow accumulation determination device 2 for display.

[0018] FIG. 2 is another schematic configuration diagram of the snow accumulation determination device in the first embodiment.

[0019] The snow accumulation determination device 2 and the monitoring system 3 may be installed, for example, in a management station 600 located away from the photovoltaic power plant 1. In this case, the video or images captured by the imaging device 6 are transmitted via the network 30.

[0020] The network 30 is constructed, for example, by a transmission medium such as a wired line, such as an optical cable or a metal cable, or a wireless line, such as a satellite line or a micro-radio, in addition to the network device 20. The network device 20 is also constructed, for example, by equipment required for transmitting videos and images, such as a router, a switch, a media converter, etc., depending on the type of transmission medium. The following description will be given using the configuration in FIG. 2.

[0021] FIG. 3 is a functional block diagram of the snow accumulation determination device in the first embodiment.

[0022] The snow accumulation determination device 2 includes an acquisition unit 11, a memory unit 12, a learning model 13, a detection unit 14, a judgment unit 15, a pseudo-snow accumulation image generation unit 17, and a learning unit 18. The snow accumulation determination device 2 can be realized, for example, by installing a program for the snow accumulation determination device 2 in a PC (Personal Computer). A CPU (Central Processing Unit) in the snow accumulation determination device 2 executes the program for the snow accumulation determination device 2 to realize the functions of the acquisition unit 11, the memory unit 12, the learning model 13, the detection unit 14, the judgment unit 15, the pseudo-snow accumulation image generation unit 17, and the learning unit 18. For example, the memory unit 12 is constructed as a database on an auxiliary storage device such as an HDD (Hard Disk Drive).

[0023] In addition, in this embodiment, an example will be described in which the snow accumulation determination device 2 determines whether snow accumulation exists on the solar panel 100 when images including snow accumulation on the solar panel 100 at the imaging target location are not stored in the memory unit 12, for example, before the solar power plant 1 starts operation or when the imaging position of the imaging device 6 is changed.

[0024] The acquisition unit 11 acquires, from the imaging device 6, one image including the solar panel 100 at the imaging target location at a first time. The image acquired here is an image that is used to determine whether or not there is snow on the solar panel 100. The acquisition unit 11 also acquires the image via the network 30. Note that the illustration of the network device 20 is omitted in this figure. The first time is a reference time.

[0025] The pseudo-snow-covered image generating unit 17 generates a pseudo-snow-covered image in which pixels of an image including one or more solar panels 100 are converted into pixels corresponding to snow, and a corresponding pseudo class label image, based on an image including the solar panels 100 at the imaging target location at a second time that is earlier than the first time (hereinafter also simply referred to as a past image) and a class label image corresponding to this image. In addition, the pseudo-snow-covered image generating unit 17 stores the generated pseudo-snow-covered image and pseudo class label image in the memory unit 12. A method of generating a pseudo-snow-covered image will be described later.

[0026] A class label is data related to a subject included in an image, and is used as learning data for performing region division (e.g., semantic segmentation). The subject refers to something that exists regardless of the shooting conditions of the imaging device 6 when the image is acquired, such as "solar panel 100," "mounting," and "background such as sky or ground." The subject also refers to information that occurs in an image depending on the environment at the time of imaging, such as "external disturbances such as blown-out highlights, ghosts, lens flare, halation, or shadows." These subjects may be defined by the user as classes independent of each other, and the latter may be included in the former. In this case, classes such as "external disturbances such as blown-out highlights, ghosts, lens flare, halation, or shadows" are not defined.

[0027] A class label image is an image in which the objects contained in an image are classified into classes, and the pixels that make up the image are assigned with the labels of the corresponding classes. The class label image may express each class with color information using RGB, or may use indexed information, such as managing each class by ID. When expressing each class with color information, the class label image may express one class with information such as R=255, G=0, B=0. When expressing each class with indexed information, the class label image may express one class with color information such as R=1, G=1, B=1, for example, for one class assigned ID=1. Hereinafter, a pseudo class label image also refers to an image in which the objects contained in a pseudo snowfall image are classified into classes, and the pixels that make up the image are assigned with the labels of the corresponding classes.

[0028] The class label image may contain pixels that do not belong to any class. For example, when an image captured by a surveillance camera contains pixels that indicate the time, it is difficult to classify the pixels as classes, and the pixels may be classified as void pixels that do not belong to any class in the area that does not depend on the detection of the area where snow is present on the solar panel 100.

[0029] In addition, although the present embodiment will be described with an example of a three-channel color image, the image used may be, for example, a one-channel image. For example, for a grayscale image, one class may be expressed by information on gradation such as G=255, or for an index image, one class may be expressed by information on a palette index such as G=1.

[0030] The learning unit 18 trains the learning model 13 using the pseudo snow-covered image and the pseudo class label image read from the storage unit 12. This process is performed for each pair of the pseudo snow-covered image and the pseudo class label image. The learning unit 18 may also train the learning model 13 using a past image and a class label image.

[0031] The learning model 13 learns using a pseudo-snowfall image and teaching data corresponding to this image, the teaching target being snowfall and the like existing on the solar panel 100. In this example, the teaching data, the teaching target being snowfall existing on the solar panel 100, uses a pseudo-class label image. The learning model 13 performs a convolution operation on the pseudo-snowfall image, and expresses the probability of which class each pixel on the pseudo-snowfall image belongs to based on a feature map that is the result of the convolution operation. The learning model 13 updates parameters such as the kernel and bias used during the convolution operation based on the error that occurs between the class output as the most probable class and the class indicated by the teaching data. The feature map of the pseudo-snowfall image may be obtained by arbitrarily changing the calculation formula, such as the size and number of kernels and the presence or absence of bias. The learning model 13 may also perform padding during the convolution operation, and may also perform pooling operations such as maximum pooling and average pooling. In the case of a three-channel color image, the learning model 13 may perform a convolution operation on each of the R, G, and B images and combine the results of each operation, for example, by taking the sum of each element. Areas on the solar panel 100 where snow is present can be detected based on the feature map, for example, by performing a transposed convolution operation. These areas can be identified on a pixel-by-pixel basis.

[0032] In the present embodiment, the learning model 13 learns by performing a convolution operation on a pseudo snowfall image or the like. However, learning may be performed by a method based on vector correlation analysis such as Vision Transformer.

[0033] Based on the above learning results, the learning model 13 receives as input a single image including the solar panel 100 at a first time, and identifies and outputs an area in this image where snow is present on the solar panel 100. For example, the learning model 13 performs area division on the input image, and outputs an image showing the area in the image where snow is present. Furthermore, the learning model 13 may output the number of pixels showing the area of ​​snow present on the solar panel 100 as an output result.

[0034] The detection unit 14 outputs one image including the solar panel 100 at the first time acquired by the acquisition unit 11 to the learning model 13. The detection unit 14 also receives the result of region segmentation output from the learning model 13 as an input.

[0035] The determination unit 15 receives the result of the area division from the detection unit 14 as an input and determines the state of snow accumulation on the solar panel 100. For example, the state of snow accumulation on the solar panel 100 is determined based on the snow accumulation rate. The snow accumulation rate on the solar panel 100 is the ratio of the area of ​​pixels corresponding to snow existing on the solar panel 100 to the area of ​​pixels corresponding to the solar panel 100. The area of ​​pixels corresponding to snow existing on the solar panel 100 is calculated by performing area division on the acquired image. On the other hand, in the fixed imaging device 6, the area of ​​pixels corresponding to the solar panel 100 can be calculated by having the snow accumulation determination device 2 recognize the area in advance using, for example, a teaching operation by a person or an affine transformation, and storing it in the storage unit 12. The determination unit 15 can calculate the snow accumulation rate using these two values. For example, the determination unit 15 can compare the calculated snow accumulation rate with a threshold value, and determine that there is snow accumulation on the solar panel 100 when the snow accumulation rate exceeds the threshold value.

[0036] Furthermore, the determination unit 15 may be configured to output a signal based on the determination of the state of snow accumulation when the snow accumulation determination device 2 determines that there is snow accumulation on the solar panel 100. An example of a signal based on the occurrence of snow accumulation is an alarm signal. The output alarm signal may be displayed on the display device 5. Furthermore, instead of displaying the output alarm signal on the display device 5, it may be output to an alarm device (not shown) other than one that displays on a screen, such as an indicator or a warning light.

[0037] FIG. 4 is a block diagram showing details of the pseudo snow-covered image generating unit in the first embodiment.

[0038] The pseudo-snow-covered image generating unit 17 includes a source image acquiring unit 21, a conversion determining unit 22, a brightness setting unit 23, a generating unit 24, and a storage unit 25. Based on at least one or more past images and class label images corresponding to these images, the pseudo-snow-covered image generating unit 17 generates a pseudo-snow-covered image in which pixels of an image including one or more solar panels 100 are converted into pixels equivalent to snow, and a corresponding pseudo class label image.

[0039] The pseudo-snowfall image generator 17 comprehensively generates pseudo-snowfall images based on, for example, criteria preset by the user. The learning model 13 learns using these data, making it possible to detect snowfall without using learning data related to actual snowfall.

[0040] The source image acquisition unit 21 acquires at least one past image and a class label image corresponding to this image. For example, these data may be images captured by the imaging device 6 about one day before the start of operation of the solar power plant 1. The past image may be an image captured before the start of operation of the solar power plant 1, and the class label image may be created by providing instructions to this image by user input, for example.

[0041] For example, the user may upload these images in a file format such as jpg or bmp via a user interface displayed on the display device 5, and the source image acquisition unit 21 may acquire these image files.

[0042] The conversion determination unit 22 determines whether or not to convert each pixel in the past image that corresponds to a pixel to which a specific label has been assigned in the class label image into a pixel corresponding to snow accumulation, based on a determination criterion preset by the user. For example, the conversion determination unit 22 may accept an input of the determination criterion settings by the user from a user interface displayed on the display device 5. For example, the user may input these settings when the snow accumulation determination device is initially set up or when the settings are changed after the device has started operating.

[0043] In addition, the user may define between class labels or within a class label as a criterion for determining whether or not to convert to a pixel corresponding to snowfall. Definition between class labels means that pixels to which a predetermined label is assigned are converted to pixels corresponding to snowfall, and pixels to which labels other than the predetermined label are assigned are not converted. For example, the conversion determination unit 22 determines that for a pixel to which a label "solar panel 100" is assigned, the corresponding pixel in the past image is converted to a pixel corresponding to snowfall, and that for a pixel to which a label "sky" is assigned, conversion is not performed. In the following, an example of converting a pixel to which a label "solar panel 100" is assigned is described, but other than this example, for example, for a pixel to which a label "ground" is assigned, the corresponding pixel in the past image is converted to snowfall. This allows the snowfall determination device 2 to increase the variety of pseudo-snowfall images to be generated, and to increase the amount of learning data for the learning model 13.

[0044] The definition within the class label means that, among the pixels to which a certain label is assigned, pixels that satisfy a certain condition are converted to pixels corresponding to snow accumulation, and pixels that do not satisfy the certain condition are not converted. For example, the conversion determination unit 22 determines that, among the pixels to which the label "solar panel 100" is assigned, pixels having a luminance value within a certain range are converted, and that pixels not having a luminance value within the certain range are not converted.

[0045] In addition, in the above-mentioned intra-class label definition, the conversion determination unit 22 may allow the user to set how to convert the pixels to be converted as a more detailed condition. For example, the conversion determination unit 22 may allow the user to set more detailed criteria, such as what percentage of all the corresponding pixels should be converted as snow, whether all the corresponding pixels should be converted uniformly, or whether only the pixels present at the bottom of the solar panel 100 should be converted. The way snow accumulates on the solar panel 100 varies depending on the location of the solar panel 100, as well as the way the snow falls and the wind direction. Therefore, the conversion determination unit 22 allows the user to input these detailed conditions, thereby generating a pseudo-snowfall image that reflects the environment and weather conditions of the location where the solar panel 100 is installed.

[0046] Furthermore, the conversion determination unit 22 may determine to generate multiple pseudo-snow-covered images for pixels to be converted based on multiple criteria. For example, the conversion determination unit 22 may determine to convert 10 percent of all applicable pixels to uniformly cover snow, and further determine to convert 20 percent of all applicable pixels to uniformly cover snow. The setting of the determination criteria is a system design item. In this way, the conversion determination unit 22 determines to generate multiple pseudo-snow-covered images based on multiple criteria, so that the pseudo-snow-covered image generating unit 17 can comprehensively generate pseudo-snow-covered images.

[0047] When converting to pixels corresponding to snowfall, the luminance setting unit 23 sets the luminance value of the converted pixel. The luminance value of the pixel to be converted as snowfall may be obtained, for example, from an area in which snowfall is shown using one or more past images, and the luminance setting unit 23 may obtain it from the average of the luminance values ​​of pixels labeled as snowfall. Furthermore, if there is no past image containing snowfall, the luminance value may be determined by a user's input. Furthermore, the luminance setting unit 23 may obtain the luminance value of the pixel to be converted as snowfall within a predetermined range. For example, the luminance setting unit 23 may obtain the variance of the luminance values ​​of the pixels labeled as snowfall and set the luminance range from the range of the variance.

[0048] The luminance value or luminance range set by the luminance setting unit 23 may be stored in the memory unit 12. This allows the pseudo-snow-covered image generating unit 17 to utilize the luminance value or luminance range once set when generating a pseudo-snow-covered image for the second or subsequent times.

[0049] The generation unit 24 compares the input past image with the class label image, and converts pixels in the past image that correspond to pixels in the class label image that have been assigned a specific label into pixels equivalent to snow, thereby generating a pseudo-snow-covered image.

[0050] Furthermore, the generating unit 24 generates a pseudo class label image in which a predetermined label in the class label image is converted to a label of snowfall. For example, when a pixel in a past image corresponding to a pixel labeled with "solar panel 100" in the class label image is converted, the generating unit 24 converts the label name to "panel on solar panel 100". Furthermore, since converting the label name changes the class, the generating unit 24 may convert the pixel value of the label in addition to converting the label name.

[0051] The storage unit 25 stores the generated pseudo snow-covered images and pseudo class label images in the memory unit 12. The pseudo snow-covered images and pseudo class label images are stored in the memory unit 12 in one-to-one correspondence.

[0052] FIG. 5 is an example of a flowchart for determining snow accumulation in the first embodiment.

[0053] In this flowchart, the flow of snow accumulation determination will be mainly described. In this flowchart, the learning model 13 will be described as a trained model that has been trained using past images and class label images.

[0054] In step S1, the acquisition unit 11 acquires one image including the solar panel 100 at the image capture location at a first time via the network 30. The acquisition unit 11 may acquire images repeatedly at a predetermined timing. In this case, the snow accumulation determination device 2 performs snow accumulation determination every time the acquisition unit 11 acquires an image. In step S2, the detection unit 14 inputs the image acquired by the acquisition unit 11 to the learning model 13.

[0055] In step S3, the learning model outputs the result of the region division, and the detection unit 14 accepts this result as an input. In step S4, the determination unit 15 calculates the snow accumulation rate from the result of the region division and performs snow accumulation determination. For example, the determination unit 15 calculates the area corresponding to snow accumulation on the solar panel 100 as the number of pixels from the result of the region division. The determination unit 15 also reads out the value stored in the storage unit 12 as the number of pixels of the area corresponding to the solar panel 100. The determination unit 15 calculates the snow accumulation rate using these two values. The determination unit 15 also calculates the ratio of the area of ​​pixels corresponding to snow accumulation on the solar panel 100 to the area of ​​pixels corresponding to the solar panel 100, and calculates the snow accumulation rate. The determination unit 15 also compares the snow accumulation rate with a threshold value, and outputs an alarm signal to the display device 5 if the snow accumulation rate is equal to or greater than the threshold value. Necessary data such as the number of pixels of the area of ​​pixels corresponding to the solar panel 100 and the threshold value may be loaded from the storage unit 12 to the memory when the program is executed.

[0056] In step S5, the determination unit 15 outputs the calculated snow accumulation rate as a result of the snow accumulation determination to the display device 5. In addition, when the calculated snow accumulation rate is equal to or greater than a threshold value, the determination unit 15 may output a warning to the display device 5.

[0057] FIG. 6 is an example of a flowchart for generating a pseudo snow-covered image in the first embodiment.

[0058] In this flowchart, the flow of generating a pseudo-snowfall image is mainly described. In this flowchart, an example is assumed in which the snowfall determination device 2 generates a pseudo-snowfall image using multiple past images that do not include snowfall and were captured by the imaging device 6 before the start of operation of the solar power plant 1. In addition, the class label images corresponding to the past images are described as having been created in advance by instructions from the user.

[0059] In addition, the following description is based on the assumption that a user has input a judgment criterion in advance when the conversion judgment unit 22 judges whether to convert a past image into a pseudo-snow-covered image. This judgment criterion is that if a past image contains pixels corresponding to "solar panel 100", the image is converted into "snow covering the solar panel 100". When performing conversion, the conversion judgment unit 22 judges that 20 percent of the pixels corresponding to "solar panel 100" are to be uniformly converted into pixels corresponding to snow. In addition, the following description is based on the assumption that the memory unit 12 stores past images taken in various time periods over the course of a day or so.

[0060] In step S21, the source image acquisition unit 21 acquires a past image and a class label image. For example, the user presses an image upload button as a user interface displayed on the display device 5, and the source image acquisition unit 21 acquires the past image and the class label image. In step S22, the conversion determination unit 22 compares the class label image with the past image and determines whether to convert the past image into a pseudo-snowfall image. If the class label image includes the label "solar panel 100", the source image acquisition unit 21 determines that the pixels to which this label is assigned should be converted as "snowfall existing on the solar panel 100". In converting "solar panel 100" to "snowfall existing on the solar panel 100", the source image acquisition unit 21 converts 20 percent of all pixels to which the corresponding label is assigned as pixels corresponding to snowfall, and determines that all the corresponding pixels should be converted uniformly.

[0061] If, in step S22, the conversion source image acquisition unit 21 determines that the past image is to be converted into a pseudo-snow-covered image, in step S23 the luminance setting unit 23 sets the luminance value of the converted pixel. In this example, the luminance setting unit 23 calculates the average luminance value of the pixel to which the class label "snow-covered" is assigned, using other past images stored in the memory unit 12. If, in step S22, the conversion determination unit 22 determines that the past image is not to be converted into a pseudo-snow-covered image, the snow accumulation determination device 2 ends the process without generating a pseudo-snow-covered image.

[0062] In step S24, the generating unit 24 generates a pseudo-snow-covered image from the past image based on the condition determined by the conversion determining unit 22 and the brightness value set by the brightness setting unit 23. Specifically, the generating unit 24 converts pixels in the past image corresponding to pixels labeled with "solar panel 100" in the class label image into pixels corresponding to snow. The generating unit 24 also converts 20% of these pixels into pixels corresponding to snow. The generating unit 24 also converts the label "solar panel 100" into "snow on solar panel 100" for the class label image corresponding to the converted past image, generating a pseudo class label image. In step S25, the saving unit 25 stores the pseudo-snow-covered image and the pseudo class label image in the memory unit 12 in one-to-one correspondence.

[0063] FIG. 7 is a comparative example between a past image and a class label image corresponding to the past image in the first embodiment.

[0064] In FIG. 7, for the sake of simplicity, an image captured of one solar panel 100 will be described, rather than an array configuration 700 of the solar panels 100.

[0065] FIG. 7A is an example of a past image acquired by the source image acquisition unit 21. In the past image, the solar panel 100, the ground 130, the sky 140, and the mount 150 are captured. FIG. 7B is an example of a class label image corresponding to the past image. In this example, in the class label image, the solar panel 100 is colored green, the ground 130 is colored pink, the sky 140 is colored blue, and the mount 150 is colored black. For the sake of explanation, the green colored parts are represented by a single gray color, the pink colored parts are represented by thin diagonal hatching, the blue colored parts are represented by thick diagonal hatching, and the black colored parts are represented by dot hatching.

[0066] The source image acquisition unit 21 acquires a past image and a class label image as shown in Figures 7A and 7B. The generation unit 24 generates a pseudo snow-covered image and a pseudo class label image based on the acquired past image and class label image.

[0067] FIG. 8 is a comparative example between the pseudo snow-covered image in the first embodiment and a pseudo class label image corresponding to the pseudo snow-covered image.

[0068] FIG. 8A is an example of a pseudo-snowfall image generated by the generating unit 24. In this example, FIG. 8A shows an example in which a pseudo-snowfall image is generated from the past image of FIG. 7A, and all pixels corresponding to the solar panel 100 are converted into pixels corresponding to snowfall. FIG. 7B is an example of a pseudo-class label image corresponding to the pseudo-snowfall image. In this example, the snowfall 110 on the solar panel is colored light blue, and for the sake of explanation, this light blue colored part is represented in a single color of white. In this example, compared to FIG. 7B, the pixel labeled as the solar panel 100, that is, the pixel colored green, has been changed to the label of the snowfall 110 on the solar panel, that is, the pixel colored light blue.

[0069] In this example, the generation unit 24 converts all pixels corresponding to the solar panel 100 into pixels corresponding to snow accumulation, but it may also convert some of the pixels corresponding to the solar panel 100 into pixels corresponding to snow accumulation. Furthermore, the generation unit 24 may convert, other than the solar panel 100, for example, some or all of the pixels corresponding to the ground 130 into pixels corresponding to snow accumulation. This allows the generation unit 24 to comprehensively generate pseudo snow accumulation images.

[0070] FIG. 9 is a block diagram showing details of a pseudo-snow-covered image generating section in a modified example of the first embodiment.

[0071] In this modification, the snow accumulation determination device 2 generates pseudo snow accumulation images and pseudo class label images by the pseudo camera image generation unit 29. The pseudo camera image generation unit 29 generates a plurality of pseudo snow accumulation images and pseudo class label images using, for example, a GAN (Generative Adversarial Network). Since the pseudo camera image generation unit 29 generates pseudo snow accumulation images and pseudo class label images using a GAN, it is possible to generate a large number of images. Moreover, the configuration other than the pseudo snow accumulation image generation unit 17 is the same as that of the first embodiment, and therefore description thereof will be omitted.

[0072] For example, the pseudo camera image generating unit 29 includes a generation model 27 that receives latent variables as input and outputs a pseudo snow-covered image and a pseudo class label image. The pseudo camera image generating unit 29 also includes a discrimination model 28 that receives a past image and a class label image, a pseudo snow-covered image and a pseudo class label image as input, and outputs a discrimination result as to whether the input image is a past image or a pseudo snow-covered image. If the input image pair is a pseudo snow-covered image and a pseudo class label image generated by the generation model 27, the discrimination model 28 outputs "0" as a pseudo image, and if the input image pair is a past image and a class label image, the discrimination model 28 outputs "1" as a real image. The generation model 27 learns so that the discrimination model 28 outputs a generated image pair as a real image. On the other hand, the discrimination model 28 is trained to output "0" if the input image pair is a pseudo-snowfall image and a pseudo class label image generated by the generative model 27, indicating that the images are pseudo, and to output "1" if the input image pair is a past image and a class label image, indicating that the images are real.

[0073] The generative model 27 generates a pseudo-snow covered image and a pseudo class label image, for example, by performing a transposed convolution operation on the input latent variables. The discriminative model 28 generates a feature map by performing a convolution operation on the input pseudo-snow covered image or past image, and compares the feature map with the pseudo class label image or class label image to determine whether the image is pseudo or real.

[0074] The storage unit 25 stores all the generated pseudo snow-covered images and pseudo class label images in the memory unit 12 in one-to-one correspondence with each other.

[0075] According to this embodiment, the snow accumulation determination device 2 acquires an image captured by the imaging device 6 at a first time, and calculates the snow accumulation rate on the solar panel 100 in the image. This allows the user to quantitatively determine the state of snow accumulation on the solar panel 100. The user can also use the determination result as a basis for excluding snow from the power generation amount guarantee when snow is present. Similarly, the user can also use the determination result as a basis for determining snow removal work.

[0076] Furthermore, according to this embodiment, the snow accumulation determination device 2 uses existing equipment such as the imaging device 6 to acquire images, thereby reducing the introduction cost of the snow accumulation determination device 2. Furthermore, the snow accumulation determination device 2 uses existing equipment, thereby reducing the need for an increase in the amount of equipment, thereby reducing maintenance costs.

[0077] Furthermore, according to this embodiment, the generator 24 generates pseudo snow-covered images and pseudo class label images based on past images. The learning model 13 learns the pseudo snow-covered images using the pseudo class label images as teaching data. This allows the learning model 13 to learn to detect snow-covered areas using the pseudo snow-covered images, etc., even if the learning data does not include actual snow-covered images, and allows for highly accurate snowfall determination.

[0078] Furthermore, according to this embodiment, the conversion determination unit 22 determines whether or not to convert a past image into a pseudo-snow-covered image based on a determination criterion set by the user. The pseudo-snow-covered image generation unit 17 can generate a pseudo-snow-covered image that reflects the environment and weather conditions of the location where the solar panel 100 is installed by reflecting the user's settings when converting into a pseudo-snow-covered image. Furthermore, the generation unit 24 can comprehensively generate pseudo-snow-covered images based on the determination criterion set by the user.

[0079] Furthermore, according to the modified example of this embodiment, the generative model 27 can generate a large number of pseudo snow images and pseudo class label images from non-photographed images by adversarial learning. This allows the snow accumulation determination device 2 to prevent the learning unit 18 from becoming insufficient in learning, and to perform highly accurate snow accumulation determination.

[0080] Second embodiment FIG. 10 is a block diagram showing details of the pseudo snow-covered image generating unit in the second embodiment.

[0081] In this embodiment, the configuration of the block diagram other than the pseudo snow-covered image generating unit 17 is the same as in the first embodiment, so a description thereof will be omitted.

[0082] The pseudo-snow-covered image generating unit 17 of this embodiment includes a lighting unit 26 that performs lighting by illuminating a previous image acquired by the source image acquiring unit 21 or a pseudo-snow-covered image generated by the generating unit 24 with pseudo light.

[0083] The lighting unit 26 artificially irradiates light onto the past image or the pseudo-snow-covered image (lighting process) to generate a second pseudo-snow-covered image. The lighting process can be performed by computer graphics. The lighting unit 26 may generate a plurality of second pseudo-snow-covered images from one past image or pseudo-snow-covered image by changing the position, intensity, or number of light sources. The lighting unit 26 may also generate the second pseudo-snow-covered image from both the past image and the pseudo-snow-covered image.

[0084] In addition, the user of the snowfall determination device 2 may prepare data instructing the depth of each pixel in the past image or the pseudo snowfall image in order to change the way light is irradiated onto an area in the past image or the pseudo snowfall image according to the position, number, or strength of the light source. The data instructing the depth is, for example, a depth map. The depth stored in the depth map may be an absolute scale having a unit, or a relative scale with a certain depth set to 1. For example, the depth map can be acquired using a sensor capable of measuring depth, such as a 3DLiDAR. Alternatively, the depth map may be generated by applying image recognition technology to the past image without using a sensor, estimating the depth, and using the estimated result as the depth map. Alternatively, the depth map may be generated by, for example, receiving an input from the conversion source image acquisition unit 21 together with the past image from the user.

[0085] In reality, light with different intensities and angles is irradiated onto an object depending on the weather and time of day. For this reason, the lighting unit 26 may allow the user to set parameters such as the weather and time of day, and irradiate the past image or the pseudo snow-covered image with different lights depending on the parameters. For example, the lighting unit 26 may change the intensity and angle of light depending on the weather and time of day. If the weather is sunny and it is noon, the sun is at its zenith in the sky, so the lighting unit 26 may irradiate the image from directly above with a strong light source appropriate for sunny weather. In addition, the lighting unit 26 may display a user interface for parameter setting on the display device 5 and receive input from the user.

[0086] Furthermore, since the second pseudo-snow-covered image is an image in which lighting processing has been applied to the pseudo-snow-covered image, the same class label image as the pseudo-snow-covered image can be used.

[0087] FIG. 11 is an example of a flowchart for generating the second pseudo snow-covered image in the second embodiment.

[0088] In this example, a flow will be described in which the lighting unit 26 generates a plurality of second pseudo snow-covered images from one pseudo snow-covered image. Also, steps S31 to S34 are similar to steps S21 to S24 in the flow, and therefore a description thereof will be omitted.

[0089] In step S35, the lighting unit 26 accepts the pseudo-snow-covered image as an input, performs a lighting process on this image, and generates a second pseudo-snow-covered image. The lighting unit 26 also generates a plurality of second pseudo-snow-covered images by changing the position, intensity, or number of light sources. In step S36, the storage unit 25 stores the pseudo-snow-covered image and the pseudo class label image in the memory unit 12 in association with each other. The storage unit 25 also stores the second pseudo-snow-covered image and the pseudo class label image in the memory unit 12 in association with each other. In this example, the pseudo-snow-covered image and the pseudo class label image are stored as the same record in a database table constructed on the memory unit 12. The second pseudo-snow-covered image and the pseudo class label image are also stored as the same record.

[0090] According to this embodiment, the lighting unit 26 performs lighting processing on the past image or the pseudo-snow-covered image to generate a second pseudo-snow-covered image. This allows the snow accumulation determination device 2 to increase the amount of learning data for the learning model 13. Furthermore, as the amount of learning data increases, the output accuracy of the learning model 13 can be improved, allowing the snow accumulation determination device 2 to perform more accurate snow accumulation determination.

[0091] Third embodiment FIG. 12 is a hardware configuration diagram of a snow accumulation determining device according to the third embodiment.

[0092] 12 includes a processor 52 such as a CPU, a main memory device 53 such as a RAM, an auxiliary memory device 54 such as a HDD, a network interface 55 such as a LAN (Local Area Network) board, a device interface 56 such as a memory slot or memory port, and a bus 57 that connects these devices together. The snow accumulation determination device 2 is, for example, a computer such as a PC, and includes input devices such as a keyboard and a mouse, and an output device such as an LCD (Liquid Crystal Display) monitor.

[0093] In this embodiment, a program for causing a computer to execute the information processing of the snow accumulation determination device 2 in any one of the first and second embodiments is installed in the auxiliary storage device 54. The snow accumulation determination device 2 loads this program in the main storage device 53 and executes it by the processor 52. As a result, the functions of the blocks shown in Figures 3, 4, etc. are realized in the snow accumulation determination device 2, making it possible to perform the snow accumulation determination described in the first and second embodiments.

[0094] This program can be installed, for example, by attaching an external device 58 on which this program is recorded to the device interface 56 and storing this program from the external device 58 in the auxiliary storage device 54. Examples of the external device 58 are a computer-readable recording medium and a recording device having such a recording medium built in. Examples of the recording medium are a CD-ROM (Compact Disk Read Only Memory), a CD-R (Compact Disk Recordable), a flexible disk, a DVD-ROM (Digital Versatile Disk Read Only Memory), and a DVD-R (Digital Versatile Disk Recordable), and an example of the recording device is a HDD. In addition, this program can be installed, for example, by downloading it via the network interface 55.

[0095] According to this embodiment, the functions of the snow accumulation determining device 2 in any one of the first and second embodiments can be realized by software.

[0096] Although several embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel snow accumulation determination device 2 described in this specification can be embodied in various other forms. In addition, various omissions, substitutions, and modifications can be made to the form of the snow accumulation determination device 2 described in this specification without departing from the gist of the invention. The appended claims and their equivalents are intended to include such forms and modifications that fall within the scope and gist of the invention. [Explanation of symbols]

[0097] 1: Solar power plant, 2: Snow accumulation determination device, 3: Monitoring system, 4: Memory unit, 5: display device, 6: imaging device, 11: acquisition unit, 12: storage unit, 13: Learning model, 14: Detection unit, 15: Judgment unit, 17: Artificial snowfall image generation unit, 18: learning unit, 20: network device, 21: source image acquisition unit, 22: conversion determination unit, 23: brightness setting unit, 24: generation unit, 25: storage unit, 26: Lighting section, 27: Generative model, 28: Discrimination model, 29: Pseudo camera image generating unit, 30: Network, 52: Processor, 53: main memory device, 54: auxiliary memory device, 55: network interface, 56: device interface, 57: bus, 58: external device, 100: Solar panel, 110: Snow on the solar panel, 130: Ground, 140: Sky, 150: Base, 400: Building, 500: String configuration, 700: Array configuration

Claims

1. an acquisition unit that acquires an image including the solar panel at a first time; a learning model that receives an image including a solar panel at the first time as an input and outputs a result of identifying an area where snow is present on the image including the solar panel at the first time; a learning unit that trains the learning model using a pseudo-snow-covered image in which pixels of an image including the solar panel at a second time that is a time before the first time are converted into pixels corresponding to snow, and a pseudo-class label image in which the pixels constituting the pseudo-snow-covered image are assigned labels of corresponding objects; and a determination unit that determines a state of snow accumulation on the solar panel based on a result of identifying the area. Snowfall determination device.

2. The snow accumulation determination device of claim 1, wherein the determination unit determines the state of snow accumulation on the solar panel based on a snow accumulation rate, which is the ratio of the area of ​​pixels corresponding to snow existing on the solar panel to the area of ​​pixels corresponding to the solar panel.

3. The snow accumulation determination device according to claim 2 , wherein the determination unit determines the state of snow accumulation on the solar panel by comparing the snow accumulation rate with a threshold value.

4. a source image acquisition unit that acquires an image including the solar panel at the second time and a class label image in which a label of a corresponding object is assigned to each pixel constituting the image; a generation unit that converts pixels in the image including the solar panel at the second time that correspond to pixels in the class label image to which a predetermined label has been assigned into pixels corresponding to snow, and generates the pseudo-snow-covered image. The snow accumulation determination device according to claim 1 .

5. The snow accumulation determination device according to claim 4 , wherein the generation unit converts predetermined labels in the class label image into labels related to snow accumulation, and further generates a pseudo class label image.

6. The snow accumulation determination device according to claim 4 , wherein the predetermined label includes a label given to the solar panel or a label given to the ground.

7. The snow accumulation determination device according to claim 4 , further comprising a lighting unit that artificially irradiates light onto the image including the solar panel or the pseudo-snow accumulation image at the second time to generate a second pseudo-snow accumulation image.

8. The snow accumulation determination device according to claim 7 , wherein the lighting unit emits different lights depending on parameters including time or weather.

9. a conversion determination unit that determines whether or not to convert each pixel in the image including the solar panel at the second time, the pixel corresponding to a pixel in the class label image to which a predetermined label has been assigned, into a pixel corresponding to snow accumulation; and a luminance setting unit that sets the luminance of the pixel after conversion. The snow accumulation determination device according to claim 4.

10. The snow accumulation determination device of claim 1 , further comprising a pseudo camera image generation unit that generates the pseudo snow accumulation image and the pseudo class label image, the pseudo camera image generation unit generating the pseudo snow accumulation image and the pseudo class label image using a Generative Adversarial Network (GAN).

11. The pseudo camera image generating unit A generative model that receives latent variables as input and outputs the pseudo snowfall image and the pseudo class label image; a classification model that receives as input the pseudo-snow-covered image and the pseudo class label image, an image including the solar panel at the second time and a class label image to which a label of a subject corresponding to the image is assigned, and outputs a classification result that identifies whether the input image is an image including the solar panel at the second time or a pseudo-snow-covered image. The snow accumulation determination device according to claim 10.

12. The acquisition unit acquires an image including the solar panel at a first time; a learning model inputting an image including a solar panel at the first time, and outputting a result of identifying an area where snow is present on the image including the solar panel at the first time; a learning unit causes the learning model to learn using a pseudo-snow-covered image in which pixels of an image including the solar panel at a second time that is a time before the first time are converted into pixels corresponding to snow, and a pseudo-class label image in which a label of a corresponding subject is assigned to pixels constituting the pseudo-snow-covered image; a determination unit that determines a state of snow accumulation on the solar panel based on a result of identifying the area; A method for determining snow depth, comprising:

13. The acquisition unit acquires an image including the solar panel at a first time; a learning model inputting an image including a solar panel at the first time, and outputting a result of identifying an area where snow is present on the image including the solar panel at the first time; a learning unit causes the learning model to learn using a pseudo-snow-covered image in which pixels of an image including the solar panel at a second time that is a time before the first time are converted into pixels corresponding to snow, and a pseudo-class label image in which a label of a corresponding subject is assigned to pixels constituting the pseudo-snow-covered image; a determination unit that determines a state of snow accumulation on the solar panel based on a result of identifying the area; A snow accumulation determination program that causes a computer to execute a snow accumulation determination method including the steps of:

Images