Machine learning device, system for determining necessity of weeding work, and method for determining necessity of weeding work

Abstract

The purpose of the present invention is to appropriately determine the necessity of weeding work. This machine learning device comprises: a storage unit that stores first original data including a plurality of items of first image data and second original data including a plurality of items of second image data, each of the plurality of items of first image data including a vegetation area that overlaps at least one of the plurality of items of second image data, and each of the plurality of items of second image data being labeled with necessity information pertaining to weeding work; and a learning unit that, on the basis of learning data based on the first original data and the second original data, generates a trained model for estimating the necessity of the weeding work on the basis of estimation target image data having a resolution closer to that of the first image data than that of the second image data. The resolution of the second image data is higher than the resolution of the first image data.

Description

Machine learning device, system for determining whether weeding work is necessary, and method for determining whether weeding work is necessary

[0001] The present disclosure relates to a technique for determining whether or not weeding work is necessary.

[0002] Patent Literature 1 discloses a detector that analyzes inputs of a set of sample image data in which an object to be learned exists as a correct image and a set of sample image data that is completely different from the object, and describes rules, features, etc. that are useful for extracting a specific object from the image data. Patent Literature 1 also discloses that an example of an object to be detected by the detector is a tree.

[0003] Japanese Patent Application Laid-Open No. 2017-224935

[0004] However, labeling images for training as correct or incorrect can be difficult. If the labeling is inappropriate, it may not be possible to properly detect whether an image is a tree or not.

[0005] Therefore, an object of the present disclosure is to make it possible to properly determine whether or not weeding work is necessary.

[0006] The machine learning device includes a memory unit that stores first original data including a plurality of first image data and second original data including a plurality of second image data, each of the plurality of first image data including a vegetation area that overlaps with at least one of the plurality of second image data, and each of the plurality of second image data being labeled with information on the necessity of weeding work; and a learning unit that generates a trained model for estimating the necessity of weeding work based on estimation target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first original data and the second original data, and wherein the resolution of the second image data is higher than the resolution of the first image data.

[0007] In addition, the system for determining whether weeding work is necessary is equipped with an input unit to which image data of the estimation target is input, and a trained model that has undergone machine learning to estimate whether weeding work is necessary based on learning data, a processing unit that estimates whether weeding work is necessary by inputting the image data of the estimation target into the trained model, and a result output unit that outputs a determination result of whether weeding work is necessary based on the estimation result of whether weeding work is necessary, wherein the learning data is data based on a plurality of first image data and information on whether weeding work is necessary based on second image data that has a higher resolution than the resolution of the first image data, and the resolution of the image data of the estimation target is closer to the resolution of the first image data than the resolution of the second image data.This is a system for determining whether weeding work is necessary that is equipped with a weeding work necessity determination device.

[0008] Further, a method for determining whether weeding work is necessary includes preparing first original data including a plurality of first image data and second original data including a plurality of second image data, wherein the resolution of the second image data is higher than that of the first image data, each of the plurality of first image data includes a vegetation area that overlaps with at least one of the plurality of second image data, and each of the plurality of second image data is labeled with information on whether weeding work is necessary, generating a trained model for estimating whether weeding work is necessary based on estimation target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first original data and the second original data, and estimating whether weeding work is necessary by inputting the estimation target image data into the trained model.

[0009] The above-described machine learning device can generate a trained model that can appropriately determine whether or not weeding work is necessary.

[0010] The above-described weeding necessity determination system can properly determine whether weeding is necessary.

[0011] The above-described method for determining whether weeding is necessary can generate a trained model that can properly determine whether weeding is necessary, thereby making it possible to properly determine whether weeding is necessary.

[0012] FIG. 1 is a block diagram showing the overall configuration of a railway system including a weeding necessity determination system. FIG. 2 is a diagram showing an example of a captured image of a track. FIG. 3 is an explanatory diagram showing an example of the position of the track and the captured image in a satellite image. FIG. 4 is a functional block diagram of a machine learning device. FIG. 5 is a flowchart showing an example of processing by the machine learning device. FIG. 6 is a functional block diagram of a necessity determination device. FIG. 7 is a flowchart showing an example of processing by the necessity determination device. FIG. 8 is a diagram showing an example of a display of an estimation result of the necessity of weeding. FIG. 9 is an example of a display showing an estimation result of the necessity of weeding and the degree of necessity. FIG. 10 is a diagram showing an example of a weeding route. FIG. 11 is an explanatory diagram showing an example of machine learning processing according to a modified example.

[0013] Hereinafter, a machine learning device, a system for determining whether weeding is necessary, and a method for determining whether weeding is necessary will be described according to the embodiments.

[0014] FIG. 1 is a block diagram showing the overall configuration of a railway system 30 including a weeding necessity determination system 32.

[0015] The railway system 30 is a system for operating the railway vehicle 20. The weeding necessity determination system 32 is a system for determining whether weeding is necessary for vegetation that grows along the track 10 on which the railway vehicle 20 runs.

[0016] An example of the railway system 30 will be described.

[0017] The track 10 is a path that guides a railway vehicle 20 along a predetermined route. Here, the track 10 includes two rails 12, 12. The two rails 12, 12 are fixed onto ballast 16 via sleepers 13.

[0018] The ballast 16 is a roadbed that supports the rails 12. The ballast 16 includes a plurality of blocks laid on the roadbed. The blocks are, for example, crushed rocks or gravel. The roadbed is, for example, the ground.

[0019] The railway vehicle 20 includes a car body 22 and a bogie 24. The bogie 24 includes a plurality of rotatably supported wheels 25W. The plurality of wheels 25W are guided by two rails 12, 12 and run on the rails 12, 12. The bogie 24 supports the car body 22 from below. As the bogie 24 runs on the track 10, the railway vehicle 20, including the car body 22, runs along the track 10.

[0020] The railway vehicle 20 may be any vehicle that travels on the track 10, including an electric train, a freight train locomotive, a freight car, a passenger train locomotive, or a passenger car. The freight car or passenger car may be a trailer pulled by a locomotive, or may be a powered vehicle that has its own power. The locomotive may be an electric locomotive or an internal combustion engine such as a diesel locomotive. The railway vehicle 20 may be a commercial vehicle for transporting people or cargo, or may be a business vehicle for monitoring track conditions. The railway vehicle 20 may also be a road-rail vehicle that can travel on both tracks and roads.

[0021] Over time, the condition of the track 10 may change, requiring maintenance of the track 10. Maintenance may be performed, for example, by monitoring the condition of the rails 12, 12, ballast 16, sleepers 13, or the fastening devices fastening the rails 12 to the sleepers.

[0022] In some areas, it is desirable to manage the condition of vegetation 18 growing along the track 10. For example, when vegetation 18 grows on the track 10, a dedicated vehicle for weeding is dispatched to the location where the vegetation 18 has grown. The dedicated vehicle then performs weeding work. Weeding work may be performed by, for example, spraying hot water or water containing herbicides. Weeding work may be performed manually or mechanically by workers dispatched to the location where the vegetation 18 has grown.

[0023] Here, the location of the vegetation 18 may spread over a wide area depending on the area where the track 10 is laid. Therefore, it is desirable to be able to appropriately determine whether or not weeding work is required for the location of the vegetation 18 that has spread over a wide area.

[0024] The present disclosure relates to a technology that enables determining the need for weeding work for locations where widespread vegetation 18 occurs by determining the need for weeding work based on image data with relatively low resolution, such as satellite images. The present disclosure also relates to a technology that enables generating an appropriately trained trained model for determining the need for weeding work, for example, by using training data labeled based on image data with relatively high resolution.

[0025] The railway system 30 includes a terminal device 40, a weeding necessity determination system 32, and data servers 90 and 92. The necessity determination system 32 includes a processing device 50. In addition to the processing device 50, the necessity determination system 32 may include the terminal device 40, the data server 90, or the data server 92.

[0026] The terminal device 40, the necessity determination system 32, and the data servers 90 and 92 are connected via a communication network 38 so as to be able to communicate with each other.

[0027] The track condition information is transmitted from the terminal device 40 to the data server 90. As a result, track condition collection data 91a is stored in the data server 90. The necessity determination system 32 can receive the condition of each position on the track 10 from the data server 90.

[0028] The data server 92 stores satellite image data 93a. The satellite image data 93a is data detected by a sensor 95 mounted on an artificial satellite 94.

[0029] The necessity determination system 32 determines whether weeding work is necessary based on the satellite image data. The necessity determination system 32 also generates a trained model for determining whether weeding work is necessary based on the track condition information and the satellite image data.

[0030] The communication network 38 may be a wired or wireless network, or may be a combination of these. The communication network 38 may also be a public communication network, a communication network using dedicated lines, or a communication network combining a public communication network and a dedicated line.

[0031] The functions of the data server 90 or the data server 92 may be incorporated into the necessity determination system 32. Furthermore, the track condition information may be transmitted directly from the terminal device 40 to the necessity determination system 32. In this case, the data server 90 may be omitted.

[0032] The configuration of each part of the railway system 30 will be described.

[0033] The terminal device 40 is a device mounted on the railway vehicle 20. The terminal device 40 is a device for detecting the state of the track 10 when the railway vehicle 20 travels on the track 10 and providing the detection result to the necessity determination system 32.

[0034] For example, the terminal device 40 includes an image sensor 42 and a communication device 46 as a transmitter. The image sensor 42 is a sensor that captures images of the track 10 and is supported by the railway vehicle 20. The communication device 46 is a communication device that includes a transmission circuit.

[0035] In this embodiment, the terminal device 40 includes a running position detection unit 44. The running position detection unit 44 detects the running position of the railcar 20 on the track 10. The running position is the position of the railcar 20 in the longitudinal direction of the track 10. The running position of the railcar 20 may be a position (e.g., kilometers) based on a fixed position in the longitudinal direction of the track 10 (e.g., the starting point of the track or any station), or may be a position based on an arbitrary position in the longitudinal direction of the track 10. For example, the running position detection unit 44 may include a rotation speed detection sensor that detects the number of rotations of the wheels and output a running distance from a certain position based on the detection result of the rotation speed detection sensor. A sensor that detects the vehicle speed based on the number of rotations of the wheels of the railcar 20 is sometimes called a tachometer generator. Because the running distance can be determined by integrating the speed, the running position detection unit 44, which includes a rotation speed detection sensor, may output the speed at regular intervals.

[0036] Furthermore, for example, the running position detection unit 44 may include a GPS (Global Positioning System) receiving unit in a GNSS (Global Navigation Satellite System), and output latitude and longitude information determined by a signal received by the GPS receiving unit, or a position in the longitudinal direction of the track 10 based on the latitude and longitude information.

[0037] The image sensor 42 may be an image sensor supported so as to capture images of the area below the railway vehicle 20. The imaging range of the image sensor 42 may be a range that includes both the rails 12, 12, or a range that includes one of the rails 12, 12. The image sensor 42 may be a line sensor, a two-dimensional sensor, or a three-dimensional sensor. If the image sensor 42 is a line sensor, data for a predetermined distance may be treated as one captured image.

[0038] The image sensor 42 may be a visible light image sensor or a non-visible light image sensor. The image sensor 42 may be, for example, an infrared light sensor or a multispectral sensor. In order to determine whether or not weeding is necessary based on the captured image 56a captured by the image sensor 42, it is preferable that a visualized image be generated based on the output of the image sensor, regardless of whether the image sensor 42 is a visible light image sensor or a non-visible light image sensor.

[0039] The terminal device 40 may also include a track condition information data generating device 48. The track condition information data generating device 48 is, for example, a computer including a processor configured with electric circuits. The track condition information data generating device 48 generates data that associates the track condition detected by the image sensor 42 with the running position detected by the running position detection unit 44 and the image capture time. The data is transmitted to the outside of the railway vehicle 20 via the communication device 46.

[0040] The data server 90 is a computer equipped with a storage device 91. The data server 90 may be a cloud server. Information associating a track state with a position is transmitted from the terminal device 40 to the data server 90. The storage device 91 of the data server 90 stores track state collection data 91a in which the track state is associated with the position of the track state on the track 10 and the imaging time.

[0041] The data server 90 may store data transmitted from a plurality of railway vehicles 20. By the data server 90 collecting the data transmitted from a plurality of railway vehicles 20, it is possible to comprehensively collect information on the state of the track 10.

[0042] The data server 92 is a computer equipped with a storage device 93. The data server 92 may be a cloud server. Satellite image data is transmitted from an artificial satellite 94 to the data server 92. The storage device 93 of the data server 92 stores satellite image data 93a.

[0043] Here, the artificial satellite 94 is equipped with a sensor 95. The sensor 95 is, for example, an optical sensor capable of detecting a state corresponding to the state of vegetation on the ground. For example, plants may tend to easily reflect green wavelength light and easily absorb blue and red wavelength light. Plants may also tend to easily reflect near-infrared wavelength light. Therefore, the sensor 95 may be, for example, an optical sensor capable of detecting one or more of green wavelength light, blue and red wavelength light, and near-infrared wavelength light. The sensor 95 may also be a multispectral sensor that detects multiple types of wavelength light.

[0044] The artificial satellite 94 captures images of the earth's surface using a sensor 95. The captured satellite image data may include the captured image of the earth's surface as well as the image capture location and image capture time. The image capture location is specified, for example, by latitude and longitude. The image capture time is specified, for example, by year, month, day, and date. The satellite image data is wirelessly transmitted from the artificial satellite 94 to a ground station 96. The satellite image data is transmitted from the ground station 96 to the data server 92. As a result, satellite image data 93a is stored in the storage device 93 of the data server 92.

[0045] The weeding necessity determination system 32 includes a processing device 50. The processing device 50 is configured by a computer including a processor 51 such as a CPU, a storage device 54, a communication device 52, and the like.

[0046] The processing device 50 is communicatively connected to the data servers 90, 92 via the communication device 52. Track condition information and satellite image data are input to the communication device 52. The satellite image data input includes both satellite image data for learning and satellite image data that is used to estimate the need for weeding work.

[0047] The communication device 52 is an example of an input unit including an input circuit. The communication device 52 is an example of an input unit to which track condition information and maintenance work condition information are input. The input unit to which track condition information is input and the input unit to which satellite image data is input may be configured with separate input circuits.

[0048] The processor 51 includes an arithmetic circuit. The processor 51 is an example of a processing unit that performs machine learning to generate a trained estimation model 60. The processor 51 is also an example of a processing unit that applies the estimation target image data 61 to the trained estimation model 60 to estimate the necessity of weeding. That is, in this embodiment, the necessity determination system 32 is an example of a machine learning device and also an example of an estimation device that determines the necessity of weeding. The processor 51 may perform processing to determine the priority of weeding based on the estimation result of the necessity of weeding. The processor 51 may also determine a weeding route based on the estimation result of the necessity of weeding.

[0049] The storage device 54 is configured by a non-volatile storage device such as a hard disk drive (HDD), a solid-state drive (SSD), etc. The storage device 54 stores a program 55, orbit status information 56, label information 57, satellite image data 58, additional information 59, an estimation model 60, estimation target image data 61, proposed maintenance location information 62, maintenance work condition information 63, and a maintenance work schedule 64. The storage device 54 is a storage unit.

[0050] The program 55 describes the processing that causes the processor 51 to function as a processing unit. Therefore, by the processor 51 executing the processing described in the program 55 stored in the storage device 54 or the like, the processing as a processing unit is performed, such as generating a trained estimation model 60 and estimating the need for weeding work. The number of processors 51 may be one or more. The multiple processors 51 may be incorporated into one computer. The multiple processors 51 may be incorporated into multiple computers, and the multiple computers may perform the processing as processing units in a distributed manner.

[0051] The track condition information 56 includes captured images of the track 10 captured by the image sensor 42 of the terminal device 40 mounted on the railway vehicle 20. The memory device 54 stores the track condition information 56 at multiple locations on the track 10 in order to generate learning data for machine learning.

[0052] The satellite image data 58 includes satellite images captured by a sensor 95 mounted on an artificial satellite 94. The storage device 54 stores the satellite image data 58 of a plurality of locations on the Earth's surface in order to generate learning data for machine learning.

[0053] 2 , it is assumed that the captured image 56a of the track 10 is an image captured by an image sensor 42 supported by the railway vehicle 20 so as to face the track 10. Therefore, it is assumed that the imaging range of the captured image 56a is a range that includes the surrounding area of ​​one rail 12 or two rails 12, 12, for example, a range of several tens of centimeters to several meters. In this way, the image captured by the image sensor 42 supported by the railway vehicle 20 is an example of second image data. Furthermore, the storage device 54 stores a plurality of pieces of track condition information 56 including the captured image 56a, and the plurality of pieces of track condition information is an example of second original data that includes a plurality of second image data.

[0054] As shown in Fig. 3, the satellite image 58a is assumed to be an image captured by a sensor 95 mounted on an artificial satellite 94 located far from the Earth's surface. Therefore, the range of the satellite image 58a is assumed to be wider than the range of the captured image 56a, for example, a range of several tens to several hundreds of kilometers. The storage device 54 may store multiple satellite image data 58 including the satellite image 58a, or may store only one. The multiple satellite images 58a may include non-overlapping areas of the Earth's surface and may be images set to cover a wider area.

[0055] When each of the plurality of satellite images 58a is divided into a plurality of areas, image data of an area among the plurality of areas that includes the position corresponding to the captured image 56a is an example of the first image data. Furthermore, the plurality of satellite images 58a that include a plurality of the first image data are an example of first original data that includes a plurality of first image data.

[0056] When one satellite image 58a is divided into a plurality of areas, image data of an area among the plurality of areas that includes a position corresponding to the captured image 56a is an example of first image data. Also, one satellite image 58a that includes a plurality of the first image data is an example of first original data that includes a plurality of first image data.

[0057] When the plurality of satellite images 58a are not divided into a plurality of areas, the satellite image 58a among the plurality of satellite images 58a that includes a position corresponding to the captured image 56a is an example of the first image data. Furthermore, data including the plurality of satellite images 58a as the first image data is an example of the first original data.

[0058] In any case, the resolution of the captured image 56a is higher than the resolution of the satellite image 58a, and therefore the resolution of the captured image 56a as second image data is higher than the resolution of the first image data corresponding to the resolution of the satellite image 58a.

[0059] The difference in resolution may be caused by, for example, a difference in the size of the imaging range. Note that resolution is the imaging field of view per pixel of the captured image, and is sometimes called ground resolution for the satellite image 58a. High resolution means a high ability to resolve, that is, a small imaging field of view per pixel.

[0060] The resolution of the satellite image 58a is assumed to be, for example, several hundred meters, and the resolution of the captured image 56a is assumed to be, for example, about 1 mm. It is assumed that the resolution of the captured image 56a is higher than the resolution of the satellite image 58a as the first image.

[0061] As shown in FIG. 3 , the track 10, the vegetation condition of which is to be estimated, is likely to be laid in an area within the imaging range of at least one satellite image 58a among the plurality of satellite images 58a. Captured images 56a capturing locations P1, P2, P3, and so on along the track 10 within the at least one satellite image 58a are likely to be stored. If one pixel Px of the satellite image 58a is first image data, the area of ​​the pixel Px serving as the first pixel may include captured images 56a of multiple locations P1, P2, P3, and so on. The vegetation area PL, the vegetation of which is to be estimated, is an area along the rails 12, 12. Therefore, each of the multiple captured images 56a serving as the plurality of second images is likely to include a vegetation area that overlaps with at least one of the multiple first image data included in one or more satellite images 58a.

[0062] The multiple pieces of first image data included in one or more satellite images 58a may be an area of ​​one pixel or an area of ​​multiple pixels. In other words, the first image data may be data that two-dimensionally represents an object image, or may be data for one pixel that represents a partial extraction of data that represents the image.

[0063] The satellite image 58a is an image of a relatively large area, and is therefore suitable for grasping the vegetation situation over a wide area. On the other hand, the satellite image 58a has low resolution, and therefore it may be difficult to appropriately judge the vegetation situation.

[0064] In contrast, the captured image 56a captured from the railway vehicle 20 is an image of a relatively small area and may therefore be inappropriate for grasping the vegetation situation over a wide area. On the other hand, the captured image 56a has high resolution and may therefore be suitable for appropriately determining the vegetation situation.

[0065] Therefore, in this embodiment, training data is generated based on the satellite image 58 a and the captured image 56 a, thereby generating appropriately labeled training data. Machine learning is performed based on the training data, thereby generating an appropriately trained estimation model 60.

[0066] Furthermore, when estimating the vegetation condition, the estimation target image data 61 is applied to the estimation model 60, thereby making it possible to estimate the vegetation condition over a wide range.

[0067] The label information 57 stored in the storage device 54 is information labeled with each of the multiple captured images 56a. For example, the label information 57 is information expressing the necessity of weeding work in consideration of the vegetation condition in the imaging range captured in the captured image 56a. The label information 57 may be information input by a person who observes the vegetation condition in the captured image 56a, or may be information obtained by determining the vegetation condition by applying image processing such as image recognition processing to the captured image 56a. The labeled information may be information expressing the degree of necessity of weeding work using three or more values, for example, a value expressed in the range from 0 to 1. The labeled information may also be information judging the necessity of weeding work using two values, for example, information judging the necessity of weeding work using 0 and 1.

[0068] The additional information 59 is information that may affect the determination of the vegetation condition or the need for weeding. The additional information 59 may include, for example, topographical information 59a, seasonal weather information 59b, growth factor information 59c, and other additional information 59d. The additional information 59 is stored in the storage device 54 as known information before the generation of the estimation model 60 or before the application of the estimation model 60.

[0069] The estimation model 60 is a model for estimating the necessity of weeding work by applying the estimation target image data 61. The estimation model 60 before learning is generated as the estimation model 60 after learning by machine learning processing.

[0070] The estimated target image data 61 is satellite image data corresponding to the ground surface area including the orbit 10 for which it is desired to determine whether or not weeding work is necessary. Like the learning satellite image data 58, the estimated target image data 61 may also be an image captured by the image sensor 42 of the artificial satellite 94. Note that the estimated target image data 61 does not need to be captured by the same artificial satellite as the learning satellite image data 58.

[0071] The maintenance proposal location information 62 is information obtained by applying the estimation target image data 61 to the estimation model 60 .

[0072] The maintenance work condition information 63 is information stored in the storage device 54 as known information when formulating the maintenance work schedule 64. The maintenance work condition information 63 is, for example, information about the route of the track 10, diagram information, and vehicle locations for weeding work.

[0073] The maintenance work schedule 64 is a schedule created based on the proposed maintenance location information 62 and the maintenance work condition information 63 .

[0074] A display device 68 and an input unit 69 may also be connected to the processing device 50. The display device 68 may be a liquid crystal display device, an organic EL (Electro-Luminescence) display device, or the like. A display device provided on a smartphone, a tablet terminal, or the like may also be used as the display device 68. The input unit 69 accepts various instructions from the user for the processing device 50. The input unit 69 may be a keyboard including a plurality of switches, a mouse, a touch panel, or the like.

[0075] The weeding necessity determination system 32 will be described, focusing on the function of the machine learning device 70. FIG. 4 is a functional block diagram showing the machine learning device 70.

[0076] The machine learning device 70 includes a data generation unit 72 and a learning unit 76. The data generation unit 72 and the learning unit 76 are realized as processing functions of the processor 51.

[0077] The data generator 72 generates learning data 74 based on a plurality of satellite image data 58 as the first original data and a plurality of pieces of orbital state information 56 as the second original data.

[0078] The learning data 74 is data in which each of the multiple first image data included in the satellite image 58a is labeled with a label indicating whether or not weeding work is required.

[0079] That is, as described above, each of the satellite image data 58 includes the satellite image 58a. The satellite image data 58 may also include position information 58b relating to the image capture position of the satellite image 58a on the Earth's surface and time information 58c relating to the image capture time.

[0080] The location information 58b may include, for example, the latitude and longitude coordinates of at least one vertex or center of the surface area corresponding to the satellite image 58a. The location information 28b may include the latitude and longitude coordinates of four vertices of the surface area corresponding to the satellite image 58a. If the satellite image 58a is divided into multiple areas, the location information of each area may also be identified based on the location information 58b. The area in which the captured image 56a is located may be identified based on the location information.

[0081] The time information 58c is, for example, information relating to the date and time when the image was captured.

[0082] The position information 58b and the time information 58c are identified by the artificial satellite 94 and are associated with the satellite image 58a as accompanying information.

[0083] The track condition information 56 also includes a captured image 56a of the track 10 captured from the railway vehicle 20. The track condition information 56 may also include position information 56b relating to the position at which the captured image 56a was captured and time information 56c relating to the time at which the image was captured.

[0084] The position information 56b is information identified based on the output of the traveling position detection unit 44 of the terminal device 40. The time information 56c is information identified in the terminal device 40 as the time when an image was captured by the image sensor 42. The time information 56c is, for example, information related to the date and time when the image was captured. The position information 56b and the time information 56c are information identified in the terminal device 40 and associated with the captured image 56a as information accompanying the captured image 56a.

[0085] The storage device 54 stores label information 57 in which the captured image 56a is labeled with information on the necessity of weeding work. The data generating unit 72 generates learning data 74 by referring to the label information 57.

[0086] That is, the label information 57 is information in which the captured image 56a is labeled with information on the necessity of weeding work. As described above, each of the multiple captured images 56a is associated with one of the vegetation areas in the multiple satellite images 58a. In this embodiment, the vegetation area is the area on the track 10 next to the rails 12, 12 and the sleepers 13. Therefore, the information on the necessity of weeding work for the captured image 56a is associated with information on the necessity of weeding work for one of the vegetation areas in the multiple satellite images 58a. Therefore, it is possible to generate learning data 74 in which each of the satellite images 58a is labeled with information on the necessity of weeding work.

[0087] The labeling of the satellite image 58a as to whether or not weeding is required may be performed by labeling divided areas in each satellite image 58a, or by labeling the entire satellite image 58a. In this embodiment, an example is described in which each satellite image 58a is divided into a plurality of areas, and each of the divided areas associated with the captured image 56a is labeled. The divided area associated with the captured image 56a is the area corresponding to the first image data. The divided area may include a plurality of pixels in the satellite image 58a, or may be an area including a single pixel.

[0088] The satellite image 58a may be image data showing the distribution of a vegetation index. A vegetation index is a value that indicates the presence or absence, amount, or activity of vegetation. Such a vegetation index may be, for example, a normalized difference vegetation index (NDVI). The NDVI is an index value calculated by the following formula:

[0089] NDVI=(IR-R) / (IR+R) where R is the reflectance of red light, and IR is the reflectance of the near-infrared region.

[0090] The vegetation index need not be NDVI as long as it indicates the presence or absence, amount, or activity of vegetation.

[0091] Based on the location information 58b of the satellite image 58a and the location information 56b of the captured image 56a, the captured image 56a corresponding to the entire satellite image 58a or a subdivided area is identified. As a result, learning data 74 is generated in which the weeding necessity information labeled on the identified captured image 56a is also labeled on the entire satellite image 58a or a subdivided area. For example, the presence or absence of a corresponding captured image 56a is determined for each subdivided NDVI image 74a in the satellite image 58a, and learning data 74 is generated in which the degree of weeding necessity is labeled in the range from 0 to 1 for the area where the corresponding captured image 56a exists. The NDVI image 74a is an area including one or more pixels. In other words, in this embodiment, an example is described in which the first image data are multiple NDVI images including one or more pixels labeled with the degree of weeding necessity.

[0092] The data generating unit 72 may add additional information 74 b to each of the multiple NDVI images 74 a based on the additional information 59 .

[0093] The additional information 74b may include topographical information 74b1 based on the topographical information 59a in the additional information 59. The topographical information 59a is information indicating the state of the earth's surface, such as the elevation of each location on the earth's surface, the proximity or presence of a water source, etc. The satellite image 58a includes location information 58b, and the location of the NDVI image 74a included in the satellite image 58a within the satellite image 58a can be identified. Therefore, the location of the NDVI image 74a on the earth's surface can be identified. By referencing the topographical information 59a based on the identified location, the elevation of the NDVI image 74a can be identified. Furthermore, by referencing the topographical information 59a based on the identified location, the distance between the NDVI image 74a and a water source, or the presence or absence of a water source within a predetermined distance, can be identified.

[0094] The data generating unit 72 can add additional information 74b including topographical information 74b1 such as altitude, proximity or presence of a water source, etc., based on the positional information of the NDVI image 74a and the topographical information 59a.

[0095] The additional information 74b may include data acquisition time information 74b2 of the NDVI image 74a. The data acquisition time information 74b2 is information about the time when the satellite image 58a including the NDVI image 74a was captured.

[0096] The information on the time of image capture may be the time of image capture. The time of image capture may be identified based on time information 58c included in satellite image data 58. The date of image capture may be added as information categorized by season, such as spring, summer, autumn, winter, rainy season, or dry season. The time of image capture may be added as information categorized by time period, such as night, day, or evening.

[0097] The information at the time of image capture may be weather information at the time of image capture. If the seasonal weather information 59b includes past weather information for each region, such as whether it was sunny, cloudy, or rainy, the weather information at the time of image capture, such as whether it was cloudy, can be determined based on the data acquisition time of the NDVI image 74a and the seasonal weather information 59b.

[0098] The data generating unit 72 can add additional information 74b including data acquisition time information 74b2 based on the position information and time information of the NDVI image 74a.

[0099] The additional information 74b may include growth factor information 74b3. The growth factor information 74b3 is information about factors that may affect the growth of vegetation in the area corresponding to the NDVI image 74a.

[0100] The growth factor information 74b3 may be the amount of precipitation or the hours of sunshine. The amount of precipitation or the hours of sunshine may be the amount of precipitation or the hours of sunshine for a predetermined period. The predetermined period may be, for example, a year or a month.

[0101] The growth factor information 74b3 may be an average temperature or a minimum and maximum temperature. The average temperature or the minimum and maximum temperature may be an average temperature or a minimum and maximum temperature for a predetermined period. The predetermined period may be, for example, a year or a month.

[0102] The topographical information and seasonal information may affect the growth of vegetation, and may therefore be understood as an example of growth factor information.

[0103] The additional information 74b may further include other information.

[0104] The additional information 74b may include, for example, identification information of the artificial satellite 94. For example, the satellite image data 58 may include information for identifying the artificial satellite 94, and the identification information of the artificial satellite 94 may be added based on that information.

[0105] The additional information 74b may include, for example, information on the date on which the previous weeding work was performed in the area corresponding to the NDVI image 74a. For example, the storage device 54 records the history of weeding work on the track 10, and by referring to the work record, the information on the date on which the previous weeding work was performed can be added.

[0106] The additional information 74b may include, for example, information about the type of plants that make up the vegetation in the area corresponding to the NDVI image 74a. The type of plants may be identified by manually recognizing the captured image 56a associated with the NDVI image 74a or by image processing.

[0107] It is optional whether or not the training data 74 includes the additional information 74b. When the training data 74 includes the additional information 74b, it is not necessary to include all of the information listed above, and any one or more pieces of information may be added. Furthermore, other information may also be added.

[0108] The learning unit 76 includes a model generation unit 76a that applies the learning data 74 to a learning model 76b to estimate the need for weeding work based on the estimation target image data 61. For example, the model generation unit 76a uses the NDVI image 74a and additional information 74b as input data and the labeled information as correct answer data to execute a process of determining parameters for the learning unit 76. As a result, the learning model 76b is trained, and a trained model 84a is generated.

[0109] The estimated target image data 61 is data with a resolution closer to that of the NDVI image 74a based on the satellite image 58a than to that of the captured image 56a. For example, the estimated target image data 61 is data captured by an artificial satellite. Therefore, a trained model 84a can be generated that can determine whether weeding is necessary based on the estimated target image data 61 captured over a relatively wide area.

[0110] FIG. 5 is a flowchart showing an example of processing by the machine learning device 70.

[0111] In step S1, a satellite image 58a is acquired. The satellite image 58a acquired here is an image captured in a wavelength range in which a vegetation index can be calculated, but it is assumed that the image is not an image that indicates a vegetation index.

[0112] In the next step S2, a vegetation index, for example, NDVI, is calculated for each pixel based on the wavelength components of light indicated by the satellite image 58a. This generates an NDVI image as a satellite image used for learning. Note that the resolution may be changed when converting to an NDVI image.

[0113] In the next step S3, the position of the orbit 10 in the NDVI image is identified. The position of the orbit 10 in the NDVI image may be identified, for example, based on latitude and longitude information of each area divided in the NDVI image and latitude and longitude information of the orbit 10. The position of the orbit 10 in the NDVI image may be identified, for example, by extracting the position of the orbit 10 by image processing or the like based on a satellite image captured in an area overlapping the range of the NDVI image, and then overlaying the extracted position of the orbit 10 on the NDVI image.

[0114] In the next step S4, location information is associated with the area of ​​the NDVI image identified as containing the orbit 10. The location information may be, for example, latitude and longitude. For example, the latitude and longitude of the area included in the NDVI image may be identified based on the latitude and longitude information contained in the satellite image 58a. The location information may be, for example, the distance in kilometers of the orbit 10. Once the path of the orbit 10 within the range of the NDVI image is identified as described above, the distance in kilometers of the orbit 10 located in that area may also be identified based on any position in the NDVI image.

[0115] As a result of the above, the generated data may be, for example, a group of information in which NDVI values ​​are associated with latitude and longitude information, or a group of information in which NDVI values ​​are associated with kilometerage information. The NDVI value may be the value of a single pixel or the average value of multiple pixels. The NDVI data used for learning may be vectorized data of a distribution image of NDVI values ​​in the target area.

[0116] The above is the processing performed on the satellite image 58a.

[0117] As for the captured image 56a, first, the captured image 56a is acquired in step S7.

[0118] In the next step S8, the captured image 56a is associated with location information. The location information associated with the captured image 56a is information acquired by the terminal device 40, and may be, for example, latitude and longitude information or kilometers.

[0119] In the next step S9, labeling is performed. As mentioned above, labeling may be performed by a human or by computer processing.

[0120] The label may be a label that classifies the necessity of weeding using a binary value. Instead of or in addition to the label indicating the necessity of weeding, the label may be a label that classifies the type of plant that appears in the captured image 56a. Note that the type of plant may affect the determination of the necessity of weeding, so this is an example of a label related to the necessity of weeding.

[0121] As a result of the above, the generated data is, for example, information in which information on the necessity of weeding work is associated with latitude and longitude information or distance in kilometers.

[0122] After steps S4 and S9, the process proceeds to step S5. In step S5, each of the plurality of NDVI data as satellite images is associated with at least one of the plurality of captured images 56a. Then, the weeding necessity information of the NDVI data is labeled based on the weeding necessity information labeled for the associated captured image 56a.

[0123] The correspondence between the area included in the NDVI image and the label of the captured image 56a may be established based on, for example, location information possessed by both. For example, based on latitude and longitude information or kilometer distance information, the labeled captured image 56a may be associated with the closest area among multiple areas having NDVI data.

[0124] It should be noted that the area included in the NDVI image is larger than the range of the captured image 56a, so that the area included in the NDVI image may include a plurality of captured images 56a.

[0125] In this case, the weeding necessity information associated with the NDVI data may be determined taking into consideration a plurality of pieces of weeding necessity information for a plurality of captured images 56a. For example, the weeding necessity information associated with the NDVI data may be an average value of a plurality of pieces of weeding necessity information labeled on a plurality of captured images 56a, or may be most frequent data. The additional information 59 may be added to the captured images 56a as desired.

[0126] In the next step S6, a trained model 84a is generated using the training data 74.

[0127] The weeding necessity determination system 32 will be described, focusing on its function as a necessity determination device 80. FIG.

[0128] The necessity determination device 80 includes an input unit 81, a data generation unit 82, and an inference processing unit 84. The input unit 81 is a circuit that inputs data from the outside or from the storage device 54 to the processing function. The data generation unit 82 and the inference processing unit 84 are realized as processing functions of the processor 51.

[0129] Estimated target image data 61 is input to the input unit 81. The estimated target image data 61 includes, for example, a satellite image 61a for which it is desired to determine whether weeding is necessary, location information 61b, and time information 61c. As described above, the estimated target image data 61 only needs to have a resolution closer to that of the satellite image 58a serving as the first image than to that of the captured image 56a serving as the second image, and the images do not need to be captured by the same satellite, or do not need to be captured by a different satellite.

[0130] The data generation unit 82 generates estimation target data 83 suitable for estimation processing by the trained model 84a based on the estimation target image data 61. For example, the data generation unit 82 may convert the satellite image 61a into an NDVI image 83a. The data generation unit 82 may divide the satellite image 61a into areas having a size suitable for input to the trained model 84a and estimate the need for weeding work for each divided area. In accordance with the NDVI image 74a in the training data 74, the divided areas may be sized to include one pixel or multiple pixels.

[0131] The data generation unit 82 may refer to the additional information 74b stored in the storage device 54 and add the additional information 83b to the image to be estimated, for example, the NDVI image 83a. The additional information 59 referred to at this time is the additional information 59 stored in the storage device 54. The additional information 59 may be new data that has been updated from the data at the time of generating the trained model 84a.

[0132] The inference processing unit 84 includes a trained model 84a that has undergone machine learning to estimate the need for weeding work. The estimation target data 83 is input to the trained model 84a, and the need for weeding work is estimated.

[0133] Here, the additional information 83b included in the trained model 84a may be the same as the additional information 74b added as the training data 74.

[0134] For example, if both the learning data 74 and the estimation target image data include topographical information 74b1, the need for weeding work can be estimated taking into account the topographical information.

[0135] For example, the elevation or the presence or absence of a water source and its proximity can affect the state of vegetation and can affect the determination of whether weeding is necessary. Therefore, by estimating whether weeding is necessary based on the learning data 74 learned based on the learning data 74 including the elevation or the presence or absence of a water source and its proximity, the necessity of weeding can be estimated taking into account the topographical information.

[0136] Furthermore, for example, if both the learning data 74 and the estimation target image data include seasonal information, the seasonal information can be taken into account when estimating whether or not weeding work is necessary.

[0137] For example, seasons such as spring, summer, autumn, winter, rainy season, or dry season may affect the state of vegetation and may affect the determination of whether or not weeding is necessary. Therefore, by estimating whether or not weeding is necessary based on the training data 74 that has been trained based on the training data 74 that includes seasonal information, it is possible to estimate whether or not weeding is necessary while taking seasonality into consideration.

[0138] Furthermore, for example, if both the learning data 74 and the estimation target image data include the shooting time, time zone, or weather information, the need for weeding work can be estimated by taking into account the shooting time, time zone, or weather information.

[0139] For example, the time of shooting, the time period, or weather information may affect the way the sun shines, the presence or absence of clouds, etc., which may affect the satellite image 58a and may affect the determination of whether or not weeding is necessary. Therefore, by estimating whether or not weeding is necessary based on the learning data 74 learned based on the time of shooting, the time period, or weather information, the estimation of whether or not weeding is necessary can be made taking into account the time of shooting, the time period, or weather information.

[0140] Furthermore, for example, if both the learning data 74 and the estimation target image data include growth factor information, the need for weeding work can be estimated taking into account the growth factor information.

[0141] For example, the growth factor information 74b3 may affect the state of vegetation and thus affect the determination of whether weeding is necessary. Therefore, by estimating whether weeding is necessary based on the learning data 74 learned based on the growth factor information, the necessity of weeding can be estimated taking the growth factor information into consideration.

[0142] Furthermore, for example, both the learning data 74 and the estimation target image data may further include other information.

[0143] For example, if the added information is the identification information of an artificial satellite, the need for weeding work can be estimated by taking into account the characteristics of the image captured by the artificial satellite.

[0144] Furthermore, if the added information includes information on the date of the previous weeding work, the need for weeding work can be determined taking into account the date of the previous weeding work.

[0145] Furthermore, if the added information or label information includes information about the type of plants that make up the vegetation, it can be estimated whether or not weeding work is necessary, taking into account the type of plants.

[0146] The result of the determination of the necessity of weeding work based on the estimation result of the necessity of weeding work is output by the result output unit 85. The result output unit 85 may be, for example, the display device 68. The result output unit 85 is a circuit that outputs the determination result as a signal, and the determination result may be stored in the storage device 54.

[0147] FIG. 7 is a flowchart showing an example of processing by the necessity determining device 80.

[0148] In step S11, estimation target image data 61 is acquired. The estimation target image data 61 is processed as necessary into data suitable for estimation. For example, as described above, it may be converted into an NDVI image or converted into a size suitable for estimation. Here, estimation target image data 61 is acquired for multiple locations along the long track 10 so that it can be estimated whether or not weeding work is required for the locations along the track 10.

[0149] In the next step S12, the estimation target data 83 is input to the trained model 84a via the input unit 81 and the data generation unit 82. As a result, the necessity of weeding work is estimated.

[0150] In the next step S13, the areas where weeding is required are prioritized. Prioritization is performed, for example, when the degree of weeding necessity is estimated in three or more stages. For example, it is assumed that the degree of weeding necessity is labeled in three or more stages, and the trained model 84a has also been trained to be able to classify and estimate the degree of weeding necessity in three or more stages. Furthermore, for example, even if the trained model 84a has been trained to output estimation results in two stages, "necessary" and "unnecessary," if a score regarding the compatibility reliability of "necessary" or "unnecessary" is output, the degree of weeding necessity may be prioritized in three or more stages according to the score.

[0151] The estimation result of the necessity of weeding work may be displayed as a determination result, for example, as shown in a display image 100 in Fig. 8. The display image 100 is, for example, an image in which the degree of necessity of weeding work is associated with the position of the railway track 10.

[0152] In FIG. 8 , a degree of necessity for weeding work is associated with each section of the railway track 10. The image includes a track image 102 that represents the actual railway track 10. The track image 102 includes a degree image 103 that indicates the degree of necessity for weeding work. The degree image 103 may be distinguished by color, shade, pattern, or the like. For example, the degree may be distinguished so that the degree of maintenance increases as the degree transitions from green to yellow to red. By viewing this image, the degree of necessity for weeding work at any position on the railway track 10 can be easily grasped. In other words, the necessity determination device 80 controls the display device 68 to change the degree image 103 associated with the track image 102 according to the estimation result of the necessity for weeding work.

[0153] Separately from the track image 102, a detailed image 104 showing the degree of necessity of weeding work may be displayed in an enlarged area of ​​a portion of the track image 102. The detailed image 104 is a graph with the horizontal axis representing the longitudinal position (e.g., kilometers) on the railway track 10 and the horizontal axis representing the degree of necessity of weeding work. The detailed image 104 may be displayed, for example, by selecting a portion of the track image 102 by clicking, touching, or the like. This detailed image 104 allows the condition of a portion of the railway track 10 to be grasped in more detail.

[0154] The image indicating the degree of necessity of weeding may be an image that displays the areas where weeding is required in a table format according to the degree of necessity, as shown in Fig. 9. The image indicating the degree of necessity of weeding may be an image that includes a message specifying the areas where weeding is required and the degree of necessity.

[0155] In the example shown in Figure 9, areas requiring weeding are displayed in a list format including an identification code (ID), route, kilometer distance, and the degree of necessity of work. The degree of necessity of work is expressed as A or B, with degree A indicating a higher necessity of work than degree B, for example.

[0156] As shown in the next step S14, an optimization process is executed to determine a weeding route for weeding at a plurality of locations where weeding is required.

[0157] That is, each of the individual data to be estimated is data for a partial range located along the track 10, which is the work route. The processor 51, which serves as a processing unit, determines the weeding route based on the estimation result of the need for weeding and the position information of each of the individual data on the track 10, which is the work route.

[0158] That is, the route information of the track 10 is stored as the maintenance work condition information 63. Each location where weeding work is requested can be identified as a location in the route information of the track 10. Therefore, a route for working at multiple weeding work requested locations along the track 10 can be found by applying an algorithm for solving a combinatorial optimization problem. For example, by applying the traveling salesman problem, a work route can be found by determining the route order that minimizes travel costs when starting from a weeding base station and visiting multiple weeding work locations in accordance with constraints imposed by the route of the track 10. If priorities for weeding work exist, constraints according to the priorities can be added.

[0159] FIG. 10 is a diagram showing an example of a display of a weeding work route. For example, in the example shown in FIG. 10, weeding work locations 001, 002, 003, and 004 are displayed on a route diagram showing routes R1, R2, and R3. Weeding work locations 001, 002, 003, and 004 are each assigned a work level of A or B. Arrows 78a indicating the work route, i.e., the direction of movement between weeding work locations, may be added to the route diagram. By looking at the arrows 78a, the worker can recognize the direction of movement when moving between weeding work locations. This allows the worker to recognize the direction of movement when moving between weeding work locations and move efficiently.

[0160] The machine learning device 70 and the method for determining the necessity of weeding configured as described above can more appropriately label the captured images 56a, which have a relatively high resolution, as to whether weeding is necessary, than label the NDVI images 74a included in the satellite image 58a, which has a relatively low resolution. Then, based on the appropriate labeling of the captured images 56a, training data 74 is generated in which each of the multiple NDVI images 74a is labeled with whether weeding is necessary, thereby generating appropriately labeled training data 74. Based on the training data 74, a trained model 84a for estimating the necessity of weeding based on the estimation target image data 61 is generated, thereby making it possible to appropriately determine the necessity of weeding.

[0161] Furthermore, since the necessity of weeding work is estimated using images with a relatively low resolution, it is possible to properly estimate the necessity of weeding work on the tracks 10 located over a wide area based on wide-area image data, for example, the satellite image 61a.

[0162] In addition, since data showing the distribution of vegetation indices such as NDVI images is used as the first image data for learning, which has relatively low resolution, it is possible to generate a trained model 84a that can appropriately estimate the need for weeding work, which is affected by the state of vegetation.

[0163] In addition, since the learning data 74 includes satellite images 58a detected by the sensor 95 of the artificial satellite 94, a learned model 84a can be generated for estimating the need for weeding work based on data observed over a wide area.

[0164] Furthermore, since the training data 74 is data to which topographical information has been added, a trained model 84a is generated that takes into account the vegetation conditions that are affected by the topographical information.

[0165] Furthermore, information at the time of image data acquisition may affect the value of the first image data, such as the satellite image 58a, or may affect seasonality. Therefore, by learning based on the training data 74 to which the information at the time of data acquisition has been added, it is possible to generate a trained model 84a that can appropriately determine whether or not weeding work is necessary, taking into account the situation at the time the image data was acquired.

[0166] Furthermore, if the learning data 74 is data to which growth factor information has been added, it is possible to properly estimate the need for weeding work taking into account future growth predictions.

[0167] In addition, by understanding the vegetation condition using the captured image 56a detected by the sensor 42 mounted on the railway vehicle 20 and labeling whether or not weeding work is necessary, it is easier to accurately estimate whether or not weeding work is necessary.

[0168] In addition, the data generation unit 72 associates each of the NDVI images 74a with at least one of the captured images 56a, and labels the information on whether or not weeding is required for each of the NDVI images 74a based on the information on whether or not weeding is required that is labeled for the associated at least one captured image 56a, thereby making it possible to appropriately label the information on whether or not weeding is required for the NDVI images 74a.

[0169] Furthermore, by inputting the estimation target image data 61 into the trained model 84a trained as described above, the need for weeding work can be estimated, thereby appropriately estimating the need for weeding work.

[0170] Furthermore, since the priority of weeding work is determined based on the estimation result of the necessity of weeding work, it is easy to carry out weeding work appropriately according to the priority.

[0171] Furthermore, the weeding route is determined based on the estimation result of whether or not weeding is necessary and information on the position where weeding is required on the track 10, which is the route to be worked on, thereby enabling efficient weeding work.

[0172] Furthermore, the above-mentioned judgment result is displayed on the display device 68, so that the judgment result as to whether or not weeding work is necessary can be recognized by visually checking the display device 68.

[0173] By constructing a weeding work determination system 32 that includes the above-mentioned machine learning device 70 and necessity determination device 80, a system can be constructed that can estimate and determine whether weeding work is necessary from the generation of the trained model 84a.

[0174] In the present embodiment, an example has been described in which the system for determining whether or not weeding is necessary has the function of a machine learning device. The machine learning device may be configured as a device physically separated from the system for determining whether or not weeding is necessary.

[0175] In this case, the system for determining whether or not weeding is necessary does not have the function of a machine learning device, and can be used solely as a device for determining whether or not weeding is necessary.

[0176] In the present embodiment, an example of estimating the need for weeding work on the track 10 has been described, but the need for weeding work in other locations may also be estimated. For example, the need for weeding work on roads, farmland, parks, golf courses, etc. may be estimated.

[0177] In this embodiment, the labeling of each of the plurality of first image data as to whether or not weeding work is required may be performed using any algorithm.

[0178] For example, labeling of the necessity of weeding for each of the plurality of first image data can be performed by associating the plurality of first images with the plurality of second images that share a vegetation area. Therefore, the association between the plurality of first images and the plurality of second images can be performed based on the similarity of the image features. The similarity of the image features can be determined using a rule-based algorithm or by applying machine learning. For example, matching based on the similarity of the image features can be performed using algorithms such as SIFT (Scale-invariant feature transform), ORB (Oriented FAST and Rotated BRIEF), SuperPoint, or D2-Net.

[0179] Furthermore, it is not necessary to label each of the multiple first image data as to whether or not weeding is necessary during the pre-processing stage of learning. For example, as shown in FIG. 11 , satellite image data 58 including a satellite image (image data) 58a and location information 58b, and railway vehicle data 200 including a captured image 56a, location information 56b, and label information 57 may be input to the machine learning device 210 as learning data. In this way, the trained machine learning device 210 may estimate the necessity of weeding 224 corresponding to the label information 57 from the estimation target image data 220 corresponding to the satellite image (image data) 58a.

[0180] For example, the machine learning device 210 performs self-supervised learning using, as learning data, satellite image data 58 including a satellite image 58a and location information 58b, and railway vehicle data 200 including a captured image 56a, location information 56b, and label information 57. Then, estimation target image data 220 corresponding to the satellite image (image data) 58a is input to the trained model. The trained model 222 can then output the masked label information, i.e., an estimation result of whether or not weeding is required 224.

[0181] Furthermore, the above-mentioned satellite image data 58 and the railway vehicle side data 200 are input into the machine learning device 210 as learning data, and as a result, an association between the satellite image data 58 and the railway vehicle side data 200 is learned from the commonality of images or commonality of location information between the satellite image data 58 and the railway vehicle side data 200, and as a result, the necessity of weeding work according to the label information 57 may be estimated from the estimation target image data 220 corresponding to the satellite image data 58.

[0182] The present disclosure discloses the following aspects.

[0183] A first aspect is a machine learning device that includes a memory unit that stores first original data including a plurality of first image data and second original data including a plurality of second image data, each of the plurality of first image data including a vegetation area that overlaps with at least one of the plurality of second image data, and each of the plurality of second image data being labeled with information on the necessity of weeding work; and a learning unit that generates, based on learning data based on the first original data and the second original data, a trained model for estimating the necessity of weeding work based on estimation target image data that has a resolution closer to the first image data than the second image data, and wherein the resolution of the second image data is higher than the resolution of the first image data.

[0184] This machine learning device can more appropriately label the second image data, which has a relatively high resolution, as to whether or not weeding is necessary than the first image data, which has a relatively low resolution. Appropriate labeling of the second image data generates appropriate training data. Using the training data, a trained model for estimating the need for weeding based on the estimation target image data can be generated, thereby accurately determining whether or not weeding is necessary.

[0185] A second aspect is the machine learning device according to the first aspect, wherein the first image data is data indicating a distribution of vegetation indices.

[0186] In this case, by generating a trained model using training data that is labeled with whether or not weeding work is necessary, based on first image data that shows the distribution of vegetation indices that indicate the vegetation condition, a trained model that can properly estimate whether or not weeding work is necessary can be generated.

[0187] A third aspect is the machine learning device according to the first or second aspect, wherein the first image data is data detected by a sensor mounted on an artificial satellite.

[0188] This makes it possible to generate a trained model for estimating the need for weeding work based on data observed over a wide area.

[0189] A fourth aspect is a machine learning device according to any one of the first to third aspects, wherein the learning data is data in which topographical information is added to each of the plurality of first image data.

[0190] This generates a trained model that takes into account topographical information.

[0191] A fifth aspect is a machine learning device according to any one of the first to fourth aspects, wherein the learning data is data in which data acquisition time information is added to each of the plurality of first image data.

[0192] Information at the time the first image data was acquired may affect the value of the first image data or seasonality. Therefore, by learning based on training data including the first image data to which information at the time of data acquisition has been added, it is possible to generate a trained model that can appropriately determine whether weeding is necessary, taking into account the situation at the time the first image data was acquired.

[0193] A sixth aspect is a machine learning device according to any one of the first to fifth aspects, wherein the learning data is data in which growth factor information is added to each of the plurality of first image data.

[0194] The growth factor information may affect the future growth state of the plant. By learning based on the learning data including the first image data to which the growth factor information has been added, it is possible to appropriately estimate the need for weeding work taking into account future growth predictions.

[0195] A seventh aspect is a machine learning device according to any one of the first to sixth aspects, wherein the second image data is data detected by a sensor mounted on a railway vehicle.

[0196] This makes it easy to grasp the vegetation conditions along the tracks using the second image data detected by the sensor mounted on the railway vehicle, making it easier to properly estimate whether weeding work is necessary.

[0197] An eighth aspect is a machine learning device according to any one of the first to seventh aspects, further comprising a data generation unit that generates learning data based on the first original data and the second original data, in which each of the plurality of first image data is labeled with the necessity of weeding work.

[0198] As a result, learning is performed based on the learning data in which each of the multiple first image data is labeled with the necessity of weeding work.

[0199] A ninth aspect is a machine learning device according to the eighth aspect, wherein the data generation unit associates each of the plurality of first image data with at least one of the plurality of second image data, and labels the weeding necessity information for each of the plurality of first image data based on the weeding necessity information labeled for the associated at least one second image data.

[0200] This allows the weeding necessity information to be labeled for the second image data based on the weeding necessity information labeled for the second image data.

[0201] A tenth aspect is a system for determining the necessity of weeding work, comprising: an input unit to which image data of an estimation target is input; a processing unit that includes a trained model that has undergone machine learning to estimate the necessity of weeding work based on learning data, and that estimates the necessity of weeding work by inputting the image data of the estimation target into the trained model; and a result output unit that outputs a determination result of the necessity of weeding work based on the estimation result of the necessity of weeding work, wherein the learning data is data based on a plurality of first image data and information on the necessity of weeding work that is based on second image data having a higher resolution than the resolution of the first image data, and the resolution of the estimation target image data is closer to the resolution of the first image data than to the resolution of the second image data.

[0202] According to a tenth aspect, appropriate training data is generated by labeling each of a plurality of first image data with the necessity of weeding based on appropriate labeling of the second image data having a relatively high resolution. Based on the training data, a trained model for estimating the necessity of weeding based on the estimation target image data is appropriately generated. The necessity of weeding is appropriately estimated based on the trained model.

[0203] An eleventh aspect is the system for determining the necessity of weeding work according to the tenth aspect, wherein the processing unit determines the priority of the weeding work based on the estimation result of the necessity of the weeding work.

[0204] This allows you to determine the priority of weeding work.

[0205] A twelfth aspect is a system for determining whether or not weeding work is necessary relating to the tenth or eleventh aspect, wherein each of the plurality of first image data is data for a partial range located along the work route, and the processing unit determines the weeding work route based on the estimation result of whether or not weeding work is necessary and the positional information of the plurality of first image data on the work route.

[0206] This allows the weeding route to be determined taking into account the route of the work target, enabling efficient weeding work.

[0207] A thirteenth aspect is the system for determining the necessity of weeding work according to any one of the tenth to twelfth aspects, wherein the result output unit is a display device that displays the determination result.

[0208] This allows the user to visually check the display device and recognize the result of the determination as to whether or not weeding work is necessary.

[0209] A fourteenth aspect is a system for determining the necessity of weeding work according to any one of the tenth to thirteenth aspects, further comprising a machine learning device according to any one of the first to eighth aspects.

[0210] This makes it possible to construct a system that can estimate and determine whether weeding work is necessary by generating a trained model using training data based on first original data including multiple first image data and second original data including multiple second image data, and applying the image data to be estimated to the trained model.

[0211] A method for determining whether weeding work is necessary in a fifteenth aspect includes preparing first original data including a plurality of first image data and second original data including a plurality of second image data, wherein the resolution of the second image data is higher than that of the first image data, each of the plurality of first image data includes a vegetation area that overlaps with at least one of the plurality of second image data, and each of the plurality of second image data is labeled with information on whether weeding work is necessary, and generating a trained model for estimating whether weeding work is necessary based on estimation target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first original data and the second original data, and estimating whether weeding work is necessary by inputting the estimation target image data into the trained model.

[0212] This allows a trained model to be generated using learning data based on first original data including multiple first image data and second original data including multiple second image data, and the image data to be estimated can be applied to the trained model to estimate whether or not weeding work is necessary.

[0213] The configurations described in the above embodiments and modifications can be combined as appropriate as long as they are not mutually contradictory.

[0214] It should be noted that the functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, application-specific integrated circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0215] The above description is illustrative in all respects and is not intended to limit the scope of the present invention. It is understood that numerous variations not illustrated can be envisaged without departing from the scope of the present invention.

Claims

1. A machine learning device that stores first original data including a plurality of first image data and second original data including a plurality of second image data, each of the plurality of first image data includes a vegetation area overlapping at least one of the plurality of second image data, and necessity information for weeding work is labeled on each of the plurality of second image data, a storage unit; a learning unit that generates a learned model for estimating the necessity for weeding work based on estimated target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first original data and the second original data; and the resolution of the second image data is higher than the resolution of the first image data.

2. The machine learning device according to claim 1, wherein the first image data is data indicating the distribution of vegetation indices.

3. The machine learning device according to claim 1 or 2, wherein the first image data is data detected by a sensor mounted on an artificial satellite.

4. The machine learning device according to claim 1 or 2, wherein the learning data is data obtained by adding terrain information to each of the plurality of first image data.

5. The machine learning device according to claim 1 or 2, wherein the learning data is data obtained by adding data acquisition time information to each of the plurality of first image data.

6. The machine learning device according to claim 1 or 2, wherein the learning data is data obtained by adding growth factor information to each of the plurality of first image data.

7. The machine learning device according to claim 1 or 2, wherein the second image data is data detected by a sensor mounted on a railway vehicle.

8. The machine learning device according to claim 1 or 2, further comprising a data generation unit that generates learning data in which necessity for weeding work is labeled on each of the plurality of first image data, based on the first original data and the second original data.

9. A machine learning device according to claim 8, wherein the data generation unit associates each of the plurality of first image data with at least one of the plurality of second image data, and labels the necessity information of the weeding operation for each of the plurality of first image data based on the necessity information of the weeding operation labeled on the at least one second image data associated therewith.

10. A necessity determination system for a weeding operation, comprising: an input unit to which estimation target image data is input; a processing unit including a learned model in which machine learning for estimating the necessity of a weeding operation is performed based on learning data, and estimating the necessity of a weeding operation by inputting the estimation target image data into the learned model; and a result output unit that outputs a determination result of the necessity of a weeding operation based on the estimation result of the necessity of a weeding operation, wherein the learning data is data based on a plurality of first image data and necessity information of a weeding operation based on second image data having a resolution higher than that of the first image data, and the resolution of the estimation target image data is closer to the resolution of the first image data than the resolution of the second image data.

11. A necessity determination system for a weeding operation according to claim 10, wherein the processing unit determines the priority order of the weeding operation based on the estimation result of the necessity of the weeding operation.

12. A necessity determination system for a weeding operation according to claim 10 or claim 11, wherein each of the plurality of first image data is data of a partial range located along the work target path, and the processing unit obtains a weeding operation path based on the estimation result of the necessity of the weeding operation and the position information of the plurality of first image data in the work target path.

13. A necessity determination system for a weeding operation according to claim 10 or claim 11, wherein the result output unit is a display device that displays the determination result.

14. A necessity determination system for a weeding operation according to claim 10 or claim 11, further comprising the machine learning device according to claim 1 or claim 2.

15. Prepare first original data including a plurality of first image data and second original data including a plurality of second image data, where the resolution of the second image data is higher than that of the first image data, each of the plurality of first image data includes a vegetation area overlapping at least one of the plurality of second image data, and necessity information for weeding work is labeled for each of the plurality of second image data. Generate a learned model for estimating the necessity of weeding work based on the learning data based on the first original data and the second original data, based on the estimated target image data having a resolution closer to the first image data than the second image data. A method for determining the necessity of weeding work, which estimates the necessity of weeding work by inputting the estimated target image data into the learned model.

Images