Machine learning device, system for determining necessity of weeding work, and method for determining necessity of weeding work

Abstract

To enable the necessity of weeding work to be appropriately determined.SOLUTION: A machine learning device comprises: a storage unit that stores first original data including a plurality of pieces of first image data and second original data including a plurality of pieces of second image data, each of the plurality of pieces of first image data including a vegetation area that overlaps at least one of the plurality of pieces of second image data, and each of the plurality of pieces of second image data being labeled with necessity information on weeding work; and a learning unit that, on the basis of training data based on the first original data and the second original data, generates a trained model for estimating the necessity of weeding work on the basis of estimation target image data having a resolution closer to that of the first image data than that of the second image data. The resolution of the second image data is higher than the resolution of the first image data.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] This disclosure relates to a technique for determining the necessity of a weeding operation.

Background Art

[0002] Patent Document 1 discloses a detector that analyzes the input of a set of sample image data in which an object to be learned exists as a correct image and a set of sample image data that is completely different from that object, and describes rules, rules, and features useful for extracting a specific object from the image data. Further, Patent Document 1 describes that an example of the detection target by the detector is a tree.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, it may be difficult to label correct and incorrect for images for learning. If the labeling is inappropriate, it may not be possible to properly detect whether it is a tree.

[0005] Therefore, an object of the present disclosure is to be able to appropriately determine the necessity of a weeding operation.

Means for Solving the Problems

[0006] The machine learning device stores first source data including a plurality of first image data and second source data including a plurality of second image data. Each of the plurality of first image data includes a vegetation area that overlaps at least one of the plurality of second image data. Whether weeding is required is labeled for each of the plurality of second image data. The machine learning device further includes a storage unit, and a learning unit that generates a learned model for estimating whether weeding is required based on estimation target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first source data and the second source data. The resolution of the second image data is higher than the resolution of the first image data.

[0007] In addition, a weeding necessity determination system includes an input unit to which estimation target image data is input, a learned model in which machine learning for estimating whether weeding is required is performed based on learning data, a processing unit that estimates whether weeding is required by inputting the estimation target image data to the learned model, and a result output unit that outputs a determination result of whether weeding is required based on the estimation result of whether weeding is required. The learning data is data based on a plurality of first image data and information on whether weeding is required based on second image data having a resolution higher than the resolution of the first image data. The resolution of the estimation target image data is closer to the resolution of the first image data than the resolution of the second image data. The weeding necessity determination system includes a weeding necessity determination device.

[0008] In addition, the method for determining the necessity of weeding work prepares first original data including a plurality of first image data and second original data including a plurality of second image data. Here, the resolution of the second image data is higher than that of the first image data, and each of the plurality of first image data includes a vegetation area overlapping at least one of the plurality of second image data. Necessity information for weeding work is labeled for each of the plurality of second image data. A learned model for estimating the necessity of weeding work is generated based on learning data based on the first original data and the second original data, based on estimation target image data having a resolution closer to the first image data than the second image data. The necessity of weeding work is estimated by inputting the estimation target image data into the learned model. This is a method for determining the necessity of weeding work.

Advantages of the Invention

[0009] The above machine learning device can generate a learned model capable of appropriately determining the necessity of weeding work.

[0010] The above system for determining the necessity of weeding work can appropriately determine the necessity of weeding work.

[0011] The above method for determining the necessity of weeding work can generate a learned model capable of appropriately determining the necessity of weeding work and can appropriately determine the necessity of weeding work.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, a machine learning device, a necessity determination system for weeding work, and a necessity determination method for weeding work according to an embodiment will be described.

[0014] FIG. 1 is a block diagram showing the overall configuration of a railway system 30 including a necessity determination system 32 for weeding work.

[0015] The railway system 30 is a system for the railway vehicle 20 to run. The necessity determination system 32 for weeding work is a system for determining whether weeding work is necessary for the vegetation generated along the track 10 on which the railway vehicle 20 travels.

[0016] An example of the railway system 30 will be described.

[0017] The track 10 is a path that guides the railway vehicle 20 along a predetermined route. Here, the track 10 includes two rails 12, 12. The two rails 12, 12 are fixed on the ballast 16 via the tie plate 13.

[0018] The ballast 16 is a roadbed that supports the rails 12, 12. The ballast 16 includes a plurality of lumps laid on the roadbed. The lumps are, for example, crushed stones or gravel obtained by crushing rocks. The roadbed is, for example, the ground.

[0019] The railway vehicle 20 includes a car body 22 and a bogie 24. The bogie 24 includes a plurality of wheels 25W rotatably supported. The plurality of wheels 25W run on the two rails 12, 12 while being guided by the two rails 12, 12. The bogie 24 supports the car body 22 from below. When the bogie 24 runs on the track 10, the railway vehicle 20 including the car body 22 runs along the track 10.

[0020] The railway vehicle 20 may be any vehicle running on the track 10, such as a tram, a locomotive of a freight train, a freight car, a locomotive of a passenger train, or a passenger car. The freight car or the passenger car may be an attached car towed by a locomotive or a powered car having its own power. The locomotive may be an electric locomotive or an internal combustion locomotive such as a diesel locomotive. The railway vehicle 20 may be a commercial vehicle for transporting people or goods, or a utility vehicle for monitoring the track condition. The railway vehicle 20 may be a rail-road vehicle capable of running on both the track and the road.

[0021] Over time, the condition of the track 10 may vary, and it is required to maintain the track 10. Maintenance is performed, for example, by monitoring the condition of the rails 12, 12, the ballast 16, the tie plate 13, or the fastening device that fastens and fixes the rail 12 to the tie plate.

[0022] In some regions, it is desired to manage the condition of the vegetation 18 growing along the track 10. For example, when vegetation 18 appears on the track 10, a dedicated vehicle for weeding is dispatched to the location where the vegetation 18 appears. Weeding work is performed by the dedicated vehicle. The weeding work may be performed, for example, by spraying hot water or water containing a herbicide. The weeding work may be performed manually or mechanically by workers dispatched to the location where the vegetation 18 appears.

[0023] Here, the location where the vegetation 18 occurs may spread over a wide range according to the area where the track 10 is laid. Therefore, it is desirable to be able to appropriately determine the necessity of the weeding operation for the location where the vegetation 18 spreads over a wide range.

[0024] The present disclosure relates to a technique for determining the necessity of a weeding operation for the location where the vegetation 18 spreads over a wide range by determining the necessity of the weeding operation based on image data with relatively low resolution such as satellite images. Further, the present disclosure relates to a technique for generating a learned model that can be appropriately learned by using learning data labeled based on image data with relatively high resolution when generating, for example, a learned model for determining the necessity of a weeding operation.

[0025] The railway system 30 includes a terminal device 40, a necessity determination system 32 for the weeding operation, and data servers 90 and 92. The necessity determination system 32 includes a processing device 50. The necessity determination system 32 may include the terminal device 40, the data server 90, or the data server 92 in addition to the processing device 50.

[0026] The terminal device 40, the necessity determination system 32, and the data servers 90 and 92 are connected to be mutually communicable via a communication network 38.

[0027] Track state information is transmitted from the terminal device 40 to the data server 90. Thereby, track state collection data 91a is accumulated in the data server 90. The necessity determination system 32 can receive the state of each position of the track 10 from the data server 90.

[0028] Satellite image data 93a is stored in the data server 92. The satellite image data 93a is data detected by a sensor 95 mounted on an artificial satellite 94.

[0029] The necessity determination system 32 determines the necessity of the weeding operation based on satellite image data. Further, the necessity determination system 32 generates a learned model for determining the necessity of the weeding operation based on the orbital state information and the satellite image data.

[0030] The communication network 38 may be wired, wireless, or a combination of them. Also, the communication network 38 may be a public communication network, a communication network using a dedicated line, or a communication network that combines a public communication network and a dedicated line.

[0031] Note that the function of the data server 90 or the data server 92 may be incorporated into the necessity determination system 32. Also, the orbital state information may be directly transmitted from the terminal device 40 to the necessity determination system 32. In this case, the data server 90 may be omitted.

[0032] Each component configuration of the railway system 30 will be described.

[0033] The terminal device 40 is a device mounted on the railway vehicle 20. The terminal device 40 is a device that detects the state of the track 10 when the railway vehicle 20 travels on the track 10 and provides the detection result to the necessity determination system 32.

[0034] For example, the terminal device 40 includes an image sensor 42 and a communication device 46 as a transmission unit. The image sensor 42 is a sensor that images the track 10 and is supported by the railway vehicle 20. The communication device 46 is a communication device including a transmission circuit.

[0035] In this embodiment, the terminal device 40 includes a traveling position detection unit 44. The traveling position detection unit 44 detects the traveling position of the railway vehicle 20 on the track 10. The traveling position is the position of the railway vehicle 20 in the longitudinal direction of the track 10. The traveling position of the railway vehicle 20 may be a position (for example, kilometer level) based on a certain fixed position (for example, the starting point of the line, any station) in the longitudinal direction of the track 10, or may be a position based on an arbitrary position in the longitudinal direction of the track 10. For example, the traveling position detection unit 44 may include a rotation speed detection sensor that detects the rotation speed of the wheels, and output the traveling distance from any position based on the detection result of the rotation speed detection sensor. A sensor that detects the vehicle speed based on the rotation speed of the railway vehicle 20 is sometimes called a tachogenerator. Since the traveling distance can be specified by integrating the speed, the traveling position detection unit 44 including the rotation speed detection sensor may output the speed at regular intervals.

[0036] Also, for example, the traveling position detection unit 44 may include a GPS (Global Positioning System) receiver in the GNSS (Global Navigation Satellite System), and output the latitude and longitude information obtained by the received signal of the GPS receiver or the position in the longitudinal direction of the track 10 based on the latitude and longitude information.

[0037] The image sensor 42 may be an image sensor supported to image the lower region from the railway vehicle 20. The imaging range by the image sensor 42 may be a range including both the rails 12, or a range including one of the rails 12, 12. The image sensor 42 may be a line sensor, a 2D sensor, or a 3D sensor. If the image sensor 42 is a line sensor, it is advisable to handle the data for a predetermined distance as one captured image.

[0038] The image sensor 42 may be a visible light image sensor or a non-visible light image sensor. The image sensor 42 may be, for example, an infrared light sensor or a multi-spectral sensor. In order to determine the necessity of the weeding operation based on the captured image 56a captured by the image sensor 42, regardless of whether the image sensor 42 is a visible light image sensor or a non-visible light image sensor, an image visualized based on the output of the image life may be generated.

[0039] Also, the terminal device 40 may include an orbital state information data generation device 48. The orbital state information data generation device 48 is, for example, a computer including a processor configured by an electric circuit. The orbital state information data generation device 48 generates, for example, data in which the state of the orbit detected by the image sensor 42 is associated with the traveling position detected by the traveling position detection unit 44 and the imaging time. The data is transmitted outside the railway vehicle 20 via the communication device 46.

[0040] The data server 90 is a computer provided with a storage device 91. The data server 90 may be a cloud server. Information associating the position with the orbital state is transmitted from the terminal device 40 to the data server 90. In the storage device 91 of the data server 90, orbital state collection data 91a in which the position and imaging time of the orbital state in the orbit 10 are associated with the orbital state is stored.

[0041] Data transmitted from a plurality of railway vehicles 20 may be stored in the data server 90. By collecting the data transmitted from a plurality of railway vehicles 20, the data server 90 can comprehensively collect the state of the orbit 10.

[0042] The data server 92 is a computer provided with a storage device 93. The data server 92 may be a cloud server. Satellite image data is transmitted from the artificial satellite 94 to the data server 92. In the storage device 93 of the data server 92, satellite image data 93a is stored.

[0043] Here, the artificial satellite 94 is equipped with a sensor 95. The sensor 95 is, for example, an optical sensor capable of detecting a state corresponding to the state of ground vegetation. For example, plants tend to easily reflect green wavelength light and tend to easily absorb blue and red wavelength light. Also, plants may tend to easily reflect near-infrared wavelength light. Therefore, the sensor 95 may be, for example, an optical sensor capable of detecting one or more of green wavelength light, blue and red wavelength light, and near-infrared wavelength light. The sensor 95 may be a multispectral sensor that detects multiple types of wavelength light.

[0044] The artificial satellite 94 images the ground surface with the sensor 95. The captured satellite image data may include, in addition to the captured image of the ground surface, the imaging position and the imaging time. The imaging position is specified, for example, by latitude and longitude. The imaging time is specified, for example, by year, month, day, and time. The satellite image data is wirelessly transmitted from the artificial satellite 94 to the ground station 96. The satellite image data is transmitted from the ground station 96 to the data server 92. Thereby, the satellite image data 93a is stored in the storage device 93 of the data server 92.

[0045] The necessity determination system 32 for weeding work includes a processing device 50. The processing device 50 is configured by a computer including a processor 51 such as a CPU, a storage device 54, a communication device 52, and the like.

[0046] The processing device 50 is communicably connected to the data servers 90 and 92 via the communication device 52. Orbit state information and satellite image data are input to the communication device 52. Note that, as the satellite image data, both the satellite image data for learning and the satellite image data to be the target for estimating the necessity of weeding work are input.

[0047] The communication device 52 is an example of an input unit including an input circuit. The communication device 52 is an example of an input unit to which orbit state information is input and maintenance work condition information is input. The input unit to which orbit state information is input and the input unit to which satellite image data is input may be configured by separate input circuits.

[0048] The processor 51 includes an arithmetic circuit. The processor 51 is an example of a processing unit that performs machine learning to generate a learned estimation model 60. Further, the processor 51 is also an example of a processing unit that applies the estimation target image data 61 to the learned estimation model 60 to estimate the necessity of the weeding work. That is, in the present embodiment, the necessity determination system 32 is an example of a machine learning device and also an example of an estimation device that determines the necessity of the weeding work. The processor 51 may perform a process of determining the priority of the weeding work based on the estimation result of the necessity of the weeding work. The processor 51 may obtain a weeding work route based on the estimation result of the necessity of the weeding work.

[0049] The storage device 54 is composed of a non-volatile storage device such as an HDD (hard disk drive) or an SSD (Solid-state drive). The storage device 54 stores a program 55, track state information 56, label information 57, satellite image data 58, additional information 59, an estimation model 60, estimation target image data 61, maintenance proposal location information 62, maintenance work condition information 63, and a maintenance work schedule 64. The storage device 54 is a storage unit.

[0050] The program 55 describes processes for the processor 51 to realize functions as a processing unit. Therefore, by the processor 51 executing the processes described in the program 55 stored in the storage device 54 or the like, processes as a processing unit for generating a learned estimation model 60 and estimating the necessity of the weeding work are executed. The processor 51 may be one or a plurality. A plurality of processors 51 may be incorporated in one computer. A plurality of processors 51 may be incorporated in a plurality of computers, and the plurality of computers may perform the processes as a processing unit in a distributed manner.

[0051] The track state information 56 includes a captured image of the track 10 captured by the image sensor 42 of the terminal device 40 mounted on the railway vehicle 20. The storage device 54 stores track state information 56 at a plurality of locations on the track 10 in order to generate learning data for machine learning.

[0052] The satellite image data 58 includes satellite images captured by the sensor 95 mounted on the artificial satellite 94. In the storage device 54, satellite image data 58 at a plurality of locations on the ground surface is stored in order to generate learning data for machine learning.

[0053] Here, for example, as shown in FIG. 2, it is assumed that the captured image 56a of the orbit 10 is an image captured by the image sensor 42 supported by the railway vehicle 20 so as to face the orbit 10. For this reason, it is assumed that the imaging range of the captured image 56a is a range including the peripheral area of one rail 12 or two rails 12, 12, for example, a range of several tens of centimeters to several meters. In this way, the image captured by the image sensor 42 supported by the railway vehicle 20 is an example of the second image data. Further, the storage device 54 stores a plurality of pieces of orbit state information 56 including the captured image 56a, and the plurality of pieces of orbit state information are an example of second original data including a plurality of pieces of second image data.

[0054] It is assumed that the satellite image 58a is an image captured by the sensor 95 mounted on the artificial satellite 94 far from the ground surface as shown in FIG. 3. For this reason, it is assumed that the range of the satellite image 58a is wider than the imaging range of the captured image 56a, for example, a range of several tens of kilometers to several hundreds of kilometers. The storage device 54 may store a plurality of pieces of satellite image data 58 including the satellite image 58a, or may store only one piece. The plurality of satellite images 58a may include non-overlapping ground surface areas and may be images set to cover a wider area.

[0055] When each of the plurality of satellite images 58a is subdivided into a plurality of areas, the image data of the area including the position corresponding to the captured image 56a among the plurality of areas is an example of the first image data. Further, the plurality of satellite images 58a including a plurality of pieces of the first image data are an example of first original data including a plurality of pieces of the first image data.

[0056] When one satellite image 58a is divided into a plurality of areas, the image data of the area including the position corresponding to the captured image 56a among the plurality of areas is an example of the first image data. Also, one satellite image 58a including a plurality of the first image data is an example of the first original data including a plurality of the first image data.

[0057] When the plurality of satellite images 58a are not divided into a plurality of areas, the satellite image 58a including the position corresponding to the captured image 56a among the plurality of satellite images 58a is an example of the first image data. Also, the data including a plurality of the satellite images 58a as the first image data is an example of the first original data.

[0058] In any case, the resolution of the captured image 56a is higher than the resolution of the satellite image 58a. Therefore, the resolution of the captured image 56a as the second image data is higher than the resolution of the first image data corresponding to the resolution of the satellite image 58a.

[0059] The difference in the resolution may be caused by, for example, the difference in the size of the imaging range. Note that the resolution is the imaging field of view per pixel of the captured image, and the satellite image 58a may be referred to as the ground resolution. A high resolution means a high ability to resolve, that is, a small imaging field of view per pixel.

[0060] The resolution of the satellite image 58a is, for example, several hundred meters, and the resolution of the captured image 56a is, for example, assumed to be about 1 mm. It is assumed that the resolution of the captured image 56a is higher than the resolution of the satellite image 58a as the first image.

[0061] As shown in FIG. 3, the track 10 for which the vegetation status is to be estimated may be laid in an area within the imaging range of at least one of the plurality of satellite images 58a. It is conceivable that the captured images 56a of the portions P1, P2, P3... along the track 10 within the at least one satellite image 58a are stored. Assuming that one pixel Px of the satellite image 58a is the first image data, the area of one pixel Px as the first pixel may also include the captured images 56a of a plurality of portions P1, P2, P3... The vegetation area PL to be estimated for vegetation is an area along the rails 12, 12. For this reason, it is assumed that each of the plurality of captured images 56a as the plurality of second images includes a vegetation area that overlaps at least one of the plurality of first image data included in one or more satellite images 58a.

[0062] Note that the plurality of first image data included in one or more satellite images 58a may be an area of one pixel or an area of a plurality of pixels. That is, the first image data may be data that two-dimensionally represents the target image, or may be data of one pixel in which the data representing the image is partially extracted.

[0063] Since the satellite image 58a is an image that captures a relatively large area, it is suitable for grasping the vegetation status over a wide range. On the other hand, since the satellite image 58a has a low resolution, it may be difficult to appropriately judge the vegetation status.

[0064] On the contrary, since the captured image 56a captured from the railway vehicle 20 is an image that captures a relatively small area, it may be unsuitable for grasping the vegetation status over a wide range. On the other hand, since the captured image 56a has a high resolution, it may be suitable for appropriately judging the vegetation status.

[0065] Therefore, in the present embodiment, learning data is generated based on the satellite image 58a and the captured image 56a, so as to generate appropriately labeled learning data. By performing machine learning based on the learning data, an appropriately learned estimation model 60 is generated.

[0066] In addition, when estimating the vegetation status, the estimation target image data 61 is applied to the estimation model 60, so that the vegetation status over a wide range can be estimated.

[0067] The label information 57 stored in the storage device 54 is information labeled for each of the plurality of captured images 56a. For example, the label information 57 is information representing the necessity of a weeding operation in view of the vegetation status of the imaging range captured in the captured image 56a. The label information 57 may be information input by a person looking at the vegetation status with respect to the captured image 56a, or may be information obtained by performing image processing such as image recognition processing on the captured image 56a to determine the vegetation status. The information to be labeled may be information representing the degree of necessity of the weeding operation with multiple values of 3 or more, for example, a value represented in the range of 0 to 1. The information to be labeled may be information for determining the necessity of the weeding operation with binary values, for example, information determined with 0 and 1.

[0068] The additional information 59 is information that may affect the determination of the vegetation status or the determination of the necessity of the weeding operation. The additional information 59 may include, for example, terrain information 59a, seasonal weather information 59b, growth factor information 59c, and other additional information 59d. The additional information 59 is stored in the storage device 54 as known information before the generation of the estimation model 60 or before the application of the estimation model 60.

[0069] The estimation model 60 is a model for estimating the necessity of a weeding operation by applying the estimation target image data 61. The estimation model 60 before learning is generated as the estimation model 60 after learning by machine learning processing.

[0070] The image data 61 to be estimated is satellite image data corresponding to the ground area including the trajectory 10 for which it is desired to determine the necessity of the weeding operation. Similar to the satellite image data 58 for learning, the image data 61 to be estimated may also be an image captured by the image sensor 42 of the artificial satellite 94. Note that the image data 61 to be estimated does not necessarily have to be captured by the same artificial satellite as the satellite image data 58 for learning.

[0071] The maintenance proposal location information 62 is information obtained by applying the image data 61 to be estimated to the estimation model 60.

[0072] The maintenance work condition information 63 is information stored in the storage device 54 as known information when formulating the maintenance work schedule 64. The maintenance work condition information 63 is, for example, information regarding the route of the trajectory 10, diagram information, and the location of the vehicle for the weeding operation.

[0073] The maintenance work schedule 64 is a schedule created based on the maintenance proposal location information 62 and the maintenance work condition information 63.

[0074] Further, a display device 68 and an input unit 69 may be connected to the processing device 50. The display device 68 may be a liquid crystal display device, an organic EL (Electro - luminescence) display device, or the like. As the display device 68, a display device provided in a smartphone, a tablet terminal, or the like may also be used. The input unit 69 receives various instructions from the user to the processing device 50. The input unit 69 may be a keyboard including a plurality of switches, a mouse, a touch panel, or the like.

[0075] Regarding the necessity determination system 32 for the weeding operation, the function as the machine learning device 70 will be described. FIG. 4 is a functional block diagram showing the machine learning device 70.

[0076] The machine learning device 70 includes a data generation unit 72 and a learning unit 76. The data generation unit 72 and the learning unit 76 are realized as the processing functions of the processor 51.

[0077] The data generation unit 72 generates learning data 74 based on a plurality of satellite image data 58 as first original data and a plurality of orbital state information 56 as second original data.

[0078] The learning data 74 is data in which the necessity of weeding work is labeled for each of a plurality of first image data included in the satellite image 58a.

[0079] That is, each of the satellite image data 58 includes the satellite image 58a as described above. The satellite image data 58 may include position information 58b regarding the imaging position of the satellite image 58a on the ground surface and time information 58c regarding the imaging time.

[0080] The position information 58b may include, for example, the latitude and longitude coordinates of at least one vertex or the center of the ground surface area corresponding to the satellite image 58a. The position information 28b may include the latitude and longitude coordinates of the four vertices of the ground surface area corresponding to the satellite image 58a. When the satellite image 58a is divided into a plurality of areas, the position information of each area can also be specified based on the above position information 58b. Based on the position information, the area where the captured image 56a is located can be specified.

[0081] The time information 58c is, for example, information regarding the date and time of imaging.

[0082] The position information 58b and the time information 58c are specified in the artificial satellite 94 and are information associated as information accompanying the satellite image 58a.

[0083] Further, the orbital state information 56 includes a captured image 56a obtained by capturing the railway vehicle 20 from the orbit 10. It may include position information 56b regarding the position where the captured image 56a was captured and time information 56c regarding the imaging time.

[0084] The position information 56b is information specified based on the output of the traveling position detection unit 44 of the terminal device 40. The time information 56c is information specified as the imaging time by the image sensor 42 in the terminal device 40. The time information 56c is, for example, information regarding the date and time of imaging. The position information 56b and the time information 56c are information specified in the terminal device 40 and associated as information accompanying the captured image 56a.

[0085] The storage device 54 stores label information 57 in which the necessity information for the weeding work is labeled on the captured image 56a. The data generation unit 72 generates learning data 74 with reference to the label information 57.

[0086] That is, the label information 57 is information in which the necessity information for the weeding work is labeled on the captured image 56a. Also, as described above, each of the plurality of captured images 56a is associated with any one of the vegetation areas of the plurality of satellite images 58a. In the present embodiment, the vegetation area is an area adjacent to the rails 12, 12 and the macraggi 13 on the track 10. Therefore, the necessity information for the weeding work for the captured image 56a is associated as the necessity information for the weeding work for any one of the vegetation areas of the plurality of satellite images 58a. Therefore, learning data 74 in which the necessity for the weeding work is labeled on each of the satellite images 58a can be generated.

[0087] The labeling of the necessity for the weeding work for the satellite image 58a may be a labeling for the areas divided in each satellite image 58a, or may be a labeling for the entire satellite image 58a. In the present embodiment, an example of labeling each of the areas in the satellite image 58a that is associated with the captured image 56a among the divided areas is described. The area in the divided areas that is associated with the captured image 56a is the area corresponding to the first image data. The divided area may include a plurality of pixels or may be an area including one pixel in the satellite image 58a.

[0088] The satellite image 58a may be image data indicating the distribution of vegetation indices. A vegetation index is a value indicating the presence, amount, or activity level of vegetation. Such a vegetation index may be, for example, a Normalized Difference Vegetation Index (NDVI). NDVI is an index value obtained by the following formula.

[0089] NDVI = (IR - R) / (IR + R) However, R is the reflectance of red light, and IR is the reflectance in the near-infrared region.

[0090] The vegetation index may be any index as long as it can indicate the presence, amount, or activity level of vegetation, and it does not have to be NDVI.

[0091] Based on the position information 58b of the satellite image 58a and the position information 56b of the captured image 56a, the captured image 56a corresponding to the whole or a subdivided area of the satellite image 58a is identified. Thereby, learning data 74 labeled with the necessity information of the weeding operation for the identified captured image 56a is generated for the whole or a subdivided area of the satellite image 58a. For example, for each of the subdivided NDVI images 74a in the satellite image 58a, the presence or absence of the corresponding captured image 56a is determined, and learning data 74 labeled with the degree of necessity of the weeding operation in the range from 0 to 1 for the area where the corresponding captured image 56a exists is generated. The NDVI image 74a is an area including one or more pixels. That is, in the present embodiment, an example is described in which a plurality of NDVI images including one or more pixels labeled with the degree of necessity of the weeding operation are a plurality of first image data.

[0092] The data generation unit 72 may add additional information 74b to each of the plurality of NDVI images 74a based on the additional information 59.

[0093] The additional information 74b may include terrain information 74b1 based on the terrain information 59a in the additional information 59. The terrain information 59a is information indicating the state of the ground surface, for example, information expressing the elevation of each position on the ground surface, the proximity or presence / absence of a water source, etc. Position information 58b is specified in the satellite image 58a, and the position within the satellite image 58a can also be specified for the NDVI image 74a included in the satellite image 58a. Therefore, the position of the NDVI image 74a on the ground surface can be specified. Based on the specified position, by referring to the terrain information 59a, the elevation of the NDVI image 74a can be specified. Also, based on the specified position, by referring to the terrain information 59a, the distance between the NDVI image 74a and the water source, or the presence / absence of a water source at a predetermined distance can be specified.

[0094] The data generation unit 72 can add additional information 74b including terrain information 74b1 such as elevation, proximity or presence / absence of a water source, etc., based on the position information of the NDVI image 74a and the terrain information 59a.

[0095] The additional information 74b may include information 74b2 at the time of data acquisition of the NDVI image 74a. The information 74b2 at the time of data acquisition is information at the time of imaging of the satellite image 58a including the NDVI image 74a.

[0096] The information at the time of imaging may be the imaging time. The imaging time can be specified based on the time information 58c included in the satellite image data 58. The date and month of shooting may be added as information classified by season, such as spring, summer, autumn, winter, rainy season or dry season. The time of shooting may be added as information classified by time zone, such as night, day or evening.

[0097] The information at the time of imaging may be the weather information at the time of shooting. If the seasonal weather information 59b includes past weather information such as sunny, cloudy or rainy in each region, based on the data acquisition time of the NDVI image 74a and the seasonal weather information 59b, the weather information at the time of shooting, for example, information on whether it is cloudy, can be specified.

[0098] Based on the position information and time information of the NDVI image 74a, the data generation unit 72 can add additional information 74b including data acquisition time information 74b2.

[0099] The additional information 74b may include growth factor information 74b3. The growth factor information 74b3 is information regarding factors that can affect the growth of vegetation in the area corresponding to the NDVI image 74a.

[0100] The growth factor information 74b3 may be precipitation or sunshine duration. The precipitation or sunshine duration may be the precipitation or sunshine duration within a predetermined period. The predetermined period may be, for example, annual or monthly.

[0101] The growth factor information 74b3 may be average temperature or minimum and maximum temperatures. The average temperature or minimum and maximum temperatures may be the average temperature or minimum and maximum temperatures within a predetermined period. The predetermined period may be, for example, annual or monthly.

[0102] Since the above terrain information and information regarding seasons can affect the growth of vegetation, they may be regarded as an example of growth factor information.

[0103] The additional information 74b may further include other information.

[0104] The additional information 74b may include, for example, identification information of the artificial satellite 94. For example, if the satellite image data 58 includes information for identifying the artificial satellite 94, the identification information of the artificial satellite 94 may be added based on such information.

[0105] The additional information 74b may include, for example, information on the date of the previous weeding operation in the area corresponding to the NDVI image 74a. For example, if the history of the weeding operation for orbit 10 is recorded in the storage device 54, information on the date of the previous weeding operation can be added by referring to the operation record.

[0106] The additional information 74b may include, for example, information regarding the types of plants that make up the vegetation in the area corresponding to the NDVI image 74a. The types of plants may be identified by recognizing the captured image 56a associated with the NDVI image 74a by a person or through image processing.

[0107] Note that whether the training data 74 includes the above additional information 74b is arbitrary. When the training data 74 includes the above additional information 74b, it is not necessary to include all the information listed as above, and any one or more pieces of information may be added. Also, other information may be added.

[0108] The learning unit 76 includes a model generation unit 76a that estimates the necessity of a weeding operation based on the image data 61 to be estimated by applying the training data 74 to the learning model 76b. For example, the model generation unit 76a executes a process of obtaining the parameters of the learning unit 76 using the NDVI image 74a and the additional information 74b as input data and the labeled information as correct answer data. Thereby, the learning model 76b is learned and the learned model 84a is generated.

[0109] The image data 61 to be estimated is data having a resolution closer to the NDVI image 74a based on the satellite image 58a than the captured image 56a. For example, the image data 61 to be estimated is data captured by an artificial satellite. Therefore, a learned model 84a can be generated that determines the necessity of a weeding operation based on the image data 61 to be estimated, which is captured over a relatively wide area.

[0110] FIG. 5 is a flowchart showing an example of processing by the machine learning device 70.

[0111] In step S1, the satellite image 58a is acquired. The satellite image 58a acquired here is an image captured in a wavelength range capable of calculating a vegetation index, but it is assumed that it is not an image showing the vegetation index.

[0112] In the next step S2, vegetation indices, such as NDVI, of each pixel are calculated based on the wavelength components of the light shown in the satellite image 58a. As a result, an NDVI image as a satellite image to be used for learning is generated. Note that the resolution may be changed when converting to the NDVI image.

[0113] In the next step S3, the position of orbit 10 in the NDVI image is specified. The specification of the position of orbit 10 in the NDVI image may be made, for example, based on the latitude and longitude information of each area divided in the NDVI image and the latitude and longitude information of orbit 10. The specification of the position of orbit 10 in the NDVI image may be made, for example, by extracting the position of orbit 10 by image processing or the like based on a satellite image that has photographed a range overlapping the range of the NDVI image, and superimposing the extracted position of orbit 10 on the NDVI image.

[0114] In the next step S4, location information is associated with the area in the NDVI image where orbit 10 is determined to exist. The location information may be, for example, latitude and longitude. For example, the latitude and longitude of the area included in the NDVI image can be specified based on the latitude and longitude information of the satellite image 58a. The location information may be, for example, the kilometer level of orbit 10. As described above, if the path of orbit 10 in the range of the NDVI image is specified, the kilometer level of orbit 10 located in the area can also be specified based on any position in the NDVI image.

[0115] As a result of the above, the generated data may be, for example, a group of information in which NDVI values are associated with latitude and longitude information, or a group of information in which NDVI values are associated with kilometer level information. Note that the NDVI value may be the value of one pixel or the average value of a plurality of pixels. The NDVI data used for learning may be data obtained by vectorizing the distribution image of NDVI values in the target area.

[0116] The above is the processing performed on the satellite image 58a.

[0117] For the captured image 56a, first, the captured image 56a is acquired in step S7.

[0118] In the next step S8, location information is associated with the captured image 56a. The location information associated with the captured image 56a is information acquired by the terminal device 40, and for example, is latitude and longitude information or kilometer scale.

[0119] In the next step S9, labeling is performed. As described above, the label may be made by a person or may be made by computer processing.

[0120] The label may be a label that classifies the necessity of the weeding operation in binary values. The labeling may be, instead of or in addition to, a label that classifies the type of plant captured in the captured image 56a. Note that since the type of plant can affect the determination of the necessity of the weeding operation, it is an example of a label regarding the necessity of the weeding operation.

[0121] As a result of the above, the generated data is, for example, information in which the necessity information of the weeding operation is associated with latitude and longitude information or kilometer scale.

[0122] After steps S4 and S9, the process proceeds to step S5. In step S5, each of a plurality of NDVI data as satellite images is associated with at least one of the plurality of captured images 56a. Then, the necessity information of the weeding operation of the NDVI data is labeled based on the necessity information of the weeding operation labeled on the associated captured image 56a.

[0123] The association between the area included in the NDVI image and the label of the captured image 56a can be made based on, for example, the location information possessed by both. For example, based on the latitude and longitude information or kilometer scale information, the labeled captured image 56a is associated with the closest one among the plurality of areas having the NDVI data.

[0124] Note that since the area included in the NDVI image is wider than the range of the captured image 56a, a plurality of captured images 56a may be included in the area included in the NDVI image.

[0125] In this case, the necessity information of the weeding operation associated with the NDVI data may be determined in consideration of the necessity information of a plurality of weeding operations for a plurality of captured images 56a. For example, the necessity information of the weeding operation associated with the NDVI data may be the average value of the necessity information of a plurality of weeding operations labeled on a plurality of captured images 56a, or may be the most frequent data. The additional information 59 may be optionally added to the captured image 56a.

[0126] In the next step S6, a learned model 84a is generated using the learning data 74.

[0127] The necessity determination system 32 for the weeding operation will be described focusing on the function as the necessity determination device 80. FIG. 6 is a functional block diagram showing the necessity determination device 80.

[0128] The necessity determination device 80 includes an input unit 81, a data generation unit 82, and an inference processing unit 84. The input unit 81 is a circuit that inputs data from the outside or the storage device 54 to the processing function. The data generation unit 82 and the inference processing unit 84 are realized as the processing functions of the processor 51.

[0129] The estimation target image data 61 is input to the input unit 81. The estimation target image data 61 includes, for example, a satellite image 61a, position information 61b, and time information 61c for which it is desired to determine the necessity of the weeding operation. As described above, the estimation target image data 61 only needs to be closer to the resolution of the satellite image 58a as the first image than the resolution of the captured image 56a as the second image, and does not have to be an image captured by the same artificial satellite, nor does it have to be an image captured by an artificial satellite.

[0130] The data generation unit 82 generates estimation target data 83 suitable for the estimation process by the learned model 84a based on the estimation target image data 61. For example, the data generation unit 82 may convert the satellite image 61a into an NDVI image 83a. The data generation unit 82 may divide the satellite image 61a into areas having a size suitable for input to the learned model 84a, and estimate the necessity of weeding work for each divided area. In accordance with the NDVI image 74a in the learning data 74, the divided area may be sized to include one pixel or a plurality of pixels.

[0131] The data generation unit 82 may refer to the additional information 74b stored in the storage device 54, and add the additional information 83b to an image to be estimated, for example, the NDVI image 83a. The additional information 59 referred to at this time is the additional information 59 stored in the storage device 54. The additional information 59 may be new data updated with respect to the data at the time of generation of the learned model 84a.

[0132] The inference processing unit 84 includes a learned model 84a in which machine learning for estimating the necessity of weeding work is performed. The above-mentioned estimation target data 83 is input to the learned model 84a, and the necessity of weeding work is estimated.

[0133] Here, the additional information 83b included in the learned model 84a is preferably the same information as the additional information 74b added as the learning data 74.

[0134] For example, if both the learning data 74 and the estimation target image data include the terrain information 74b1, the necessity of weeding work can be estimated in consideration of the terrain information.

[0135] For example, the altitude, the presence or absence and proximity of a water source affect the vegetation state and can affect the determination of the necessity of weeding work. Therefore, by estimating the necessity of weeding work based on the learning data 74 learned based on the learning data 74 including the altitude, the presence or absence and proximity of a water source, etc., the necessity of weeding work can be estimated in consideration of the terrain information.

[0136] Also, for example, if both the learning data 74 and the image data of the object to be estimated include seasonal information, the necessity of the weeding operation can be estimated taking into account the seasonal information.

[0137] For example, seasons such as spring, summer, autumn, winter, rainy season or dry season can affect the state of vegetation and the determination of the necessity of the weeding operation. Therefore, by estimating the necessity of the weeding operation based on the learning data 74 learned based on the learning data 74 including seasonal information, the necessity of the weeding operation can be estimated taking into account seasonality.

[0138] Also, for example, if both the learning data 74 and the image data of the object to be estimated include shooting time, time zone or weather information, the necessity of the weeding operation can be estimated taking into account the shooting time, time zone or weather information.

[0139] For example, the shooting time, time zone or weather information can affect the way the sun shines, the presence or absence of clouds, etc., can affect the satellite image 58a, and can affect the determination of the necessity of the weeding operation. Therefore, by estimating the necessity of the weeding operation based on the learning data 74 learned based on the shooting time, time zone or weather information, the necessity of the weeding operation can be estimated taking into account the shooting time, time zone or weather information.

[0140] Also, for example, if both the learning data 74 and the image data of the object to be estimated include growth factor information, the necessity of the weeding operation can be estimated taking into account the growth factor information.

[0141] For example, the growth factor information 74b3 can affect the state of the vegetation and the determination of the necessity of the weeding operation. Therefore, by estimating the necessity of the weeding operation based on the learning data 74 learned based on the growth factor information, the necessity of the weeding operation can be estimated taking into account the growth factor information.

[0142] Also, for example, both the learning data 74 and the image data of the object to be estimated may further include other information.

[0143] For example, if the information to be added is the identification information of a satellite, the necessity of the weeding operation can be estimated in consideration of the characteristics of the captured image of the satellite, etc.

[0144] Also, if the information to be added includes the date information of the previous weeding operation, the necessity of the weeding operation can be determined in consideration of the date of the previous weeding operation.

[0145] Also, if the information to be added or the information to be labeled includes information regarding the types of plants constituting the vegetation, the necessity of the weeding operation can be estimated in consideration of the types of the plants.

[0146] The determination result of the necessity of the weeding operation based on the estimation result of the necessity of the weeding operation is output by the result output unit 85. The result output unit 85 may be, for example, the above-described display device 68. The result output unit 85 is a circuit that outputs the determination result as a signal, and the determination result may be stored in the storage device 54.

[0147] FIG. 7 is a flowchart showing an example of the processing by the necessity determination device 80.

[0148] In step S11, the estimation target image data 61 is acquired. The estimation target image data 61 is processed into data suitable for estimation as necessary. For example, as described above, it is converted into an NDVI image or converted into a size suitable for estimation. Here, a plurality of pieces of estimation target image data 61 along the track 10 are acquired so that the necessity of the weeding operation can be estimated for the locations along the long extending track 10.

[0149] In the next step S12, the estimation target data 83 is input to the learned model 84a via the input unit 81 and the data generation unit 82. Thereby, the necessity of the weeding operation is estimated.

[0150] In the next step S13, prioritization is performed for the locations where weeding work is required. The prioritization is performed, for example, when the necessity degree of the weeding work is estimated in three or more levels. For example, it is assumed that the necessity degree of the weeding work is labeled in three or more levels, and the learned model 84a is also learned so that it can classify and estimate the necessity degree of the weeding work in three or more levels. Also, for example, even when the learned model 84a is learned to output the estimation result in two levels of necessary and unnecessary, and when a score regarding the conformity reliability of necessary or unnecessary is output, the necessity degree of the weeding work may be prioritized in three or more levels according to the score.

[0151] The estimation result of the necessity of the weeding work may be displayed as a determination result, for example, like the display image 100 shown in FIG. 8. The display image 100 is, for example, an image in which the necessity degree of the weeding work is associated with the position of the railway track 10.

[0152] In FIG. 8, the necessity degree of the weeding work is associated with each section of the railway track 10. The image includes a track image 102 representing the actual railway track 10. The degree image 103 for displaying the necessity degree of the weeding work is included in the track image 102. The degree image 103 may be identified by color, shading, pattern, etc. For example, it may be distinguished so that the maintenance degree increases as the degree transitions from green to yellow and then to red. By looking at this image, it is easy to grasp the degree of necessity of the weeding work at any position on the railway track 10.

[0153] Separate from the track image 102, a detailed image 104 representing the necessity degree of the weeding work may be displayed in a range where a part of the track image 102 is enlarged. The detailed image 104 is a graph with the longitudinal position (for example, in kilometers) on the railway track 10 on the horizontal axis and the necessity degree of the weeding work on the vertical axis. The detailed image 104 may be displayed, for example, by selecting a part of the track image 102 by clicking, touch operation, etc. With this detailed image 104, the state of a part of the railway track 10 can be grasped in more detail.

[0154] An image indicating the necessity degree of the weeding operation may be an image that displays the locations where the weeding operation is requested in a tabular format according to the degree of necessity, as shown in FIG. 9. The image indicating the necessity degree of the weeding operation may also be an image that includes a message specifying the locations and degrees where the weeding operation is requested.

[0155] In the example shown in FIG. 9, the locations where the weeding operation is necessary are displayed in a list format including an identification code (ID), a route, a kilometer post, and the necessity degree of the operation. The necessity degree of the operation is expressed by A or B. For example, degree A indicates a higher necessity of the operation than degree B.

[0156] As shown in the next step S14, an optimization process is executed to obtain a weeding operation route for performing the weeding operation at a plurality of locations where the weeding operation is requested.

[0157] That is, each of the individual data to be estimated is data of a partial range located along the track 10 which is the operation target route. The processor 51 as the processing unit obtains the weeding operation route based on the estimation result of the necessity of the above weeding operation and the position information of each of the individual data in the track 10 which is the operation target route.

[0158] That is, the route information of the track 10 is stored as the maintenance operation condition information 63. Each position where the weeding operation is requested can be specified as a position in the route information of the track 10. Therefore, the route for working at a plurality of weeding operation request locations along the track 10 can be obtained by applying an algorithm for solving the combinatorial optimization problem. For example, by applying the traveling salesman problem, when starting from the base station of the weeding operation and traveling through a plurality of weeding operation locations according to the constraint conditions by the route of the track 10, the operation route can be obtained by determining the route order with the minimum moving cost. When there is a priority order for the weeding operation, constraint conditions corresponding to the priority order may be added.

[0159] FIG. 10 is a diagram showing an example of a display of a weeding work route. For example, in the example shown in FIG. 10, the weeding work locations 001, 002, 003, and 004 are displayed on a route map showing routes R1, R2, and R3. A work degree A or B is assigned to each of the weeding work locations 001, 002, 003, and 004. An arrow 78a indicating the work route, that is, the moving direction between the weeding work locations, may be added to the route map. By looking at the arrow 78a, an operator can recognize the moving direction when moving between the weeding work locations. Thereby, the operator can recognize the moving direction when moving between the weeding work locations and move efficiently.

[0160] According to the machine learning device 70 and the method for determining the necessity of weeding work configured as described above, rather than labeling the necessity of weeding work for the NDVI image 74a included in the satellite image 58a with relatively low resolution, the labeling of the necessity of weeding work for the captured image 56a with relatively high resolution can be performed more appropriately. Then, based on the appropriate labeling for the captured image 56a, by generating the learning data 74 in which the necessity of weeding work is labeled for each of the plurality of NDVI images 74a, the learning data 74 that is appropriately labeled is generated. Based on the learning data 74, by generating a learned model 84a for estimating the necessity of weeding work based on the estimation target image data 61, the necessity of weeding work can be determined appropriately.

[0161] Also, in order to estimate the necessity of weeding work based on an image with relatively low resolution, the necessity of weeding work for the track 10 existing over a wide range can be estimated appropriately based on wide-range image data, for example, the satellite image 61a.

[0162] Also, as the first image data for learning with relatively low resolution, since data showing the distribution of vegetation indices such as NDVI images is used, a learned model 84a that can appropriately estimate the necessity of weeding work affected by the state of vegetation can be generated.

[0163] In addition, since the learning data 74 includes the satellite image 58a detected by the sensor 95 of the artificial satellite 94, a learned model 84a for estimating the necessity of the weeding operation can be generated based on the data obtained by observing a wide area.

[0164] In addition, since the learning data 74 is data with terrain information added, a learned model 84a that takes into account the vegetation situation affected by the terrain information is generated.

[0165] In addition, the information at the time of image data acquisition may affect the value of the first image data such as the satellite image 58a or may affect the seasonality. Therefore, by learning based on the learning data 74 to which the information at the time of data acquisition is added, a learned model 84a that can appropriately determine the necessity of the weeding operation in consideration of the situation at the time of acquisition of the image data can be generated.

[0166] Furthermore, if the learning data 74 is data with growth factor information added, the necessity of the weeding operation considering future growth prediction can be appropriately estimated.

[0167] In addition, by grasping the vegetation situation based on the captured image 56a detected by the sensor 42 mounted on the railway vehicle 20 and labeling the necessity of the weeding operation, it is easy to appropriately estimate the necessity of the weeding operation.

[0168] In addition, the data generation unit 72 associates each of the NDVI images 74a with at least one of the captured images 56a, and labels the necessity information of the weeding operation for each of the NDVI images 74a based on the necessity information of the weeding operation labeled on the at least one captured image 56a associated therewith. Therefore, the necessity information of the weeding operation for the NDVI image 74a can be appropriately labeled.

[0169] In addition, by inputting the estimation target image data 61 into the learned model 84a learned as described above and estimating the necessity of the weeding operation, the necessity of the weeding operation is appropriately estimated.

[0170] In addition, since the priority order of the weeding operation is determined based on the estimation result of the necessity of the weeding operation, it is easy to appropriately perform the weeding operation according to the priority order.

[0171] In addition, based on the estimation result of the necessity of the weeding operation and the information on the position where the weeding operation is requested on the track 10 which is the work target path, a weeding operation path is obtained, enabling an efficient weeding operation.

[0172] In addition, by displaying the determination result on the display device 68, it is possible to recognize the determination result of the necessity of the weeding operation by visually recognizing the display device 68.

[0173] By constructing the weeding operation determination system 32 including the machine learning device 70 and the necessity determination device 80, it is possible to construct a system capable of estimating and determining the necessity of the weeding operation from the generation of the learned model 84a.

[0174] In this embodiment, an example in which the weeding operation necessity determination system has a function as a machine learning device has been described. The machine learning device may be configured as a device physically separated from the weeding operation necessity determination system.

[0175] In this case, the weeding operation necessity determination system does not have a function as a machine learning device and can be used solely as a device for determining the necessity of the weeding operation.

[0176] In this embodiment, an example of estimating the necessity of the weeding operation on the track 10 has been described, but the necessity of the weeding operation for other locations may also be estimated. For example, the necessity of the weeding operation on roads, farmlands, parks, golf courses, etc. may be estimated.

[0177] In this embodiment, the labeling of the necessity of the weeding operation for each of the plurality of first image data may be performed by any algorithm.

[0178] For example, the labeling of the necessity of the weeding operation for each of a plurality of first image data can be performed by associating a plurality of first images and a plurality of second images that share a vegetation area. Therefore, the association of the plurality of first images and the plurality of second images can be performed based on the similarity of the features of the images. The similarity of the features of the images may be determined by a rule-based algorithm or may be determined by the application of machine learning. For example, the matching based on the similarity of the features of the images may be performed using an algorithm such as SIFT (Scale-invariant feature transform), ORB (Oriented FAST and Rotated BRIEF), Super Point, or D2-Net.

[0179] Also, in the preprocessing stage of learning, it is not essential that the labeling of the necessity of the weeding operation for each of the plurality of first image data be performed. For example, as shown in FIG. 11, satellite image data 58 including image data 58a and position information 58b, etc., and railway vehicle side data 200 including captured image 56a, position information 56b, and label information 57 may be input to a machine learning device 210 as learning data. Thereby, the learned machine learning device 210 may estimate the necessity 224 of the weeding operation corresponding to the label information 57 from the estimated target image data 220 corresponding to the image data 58a.

[0180] For example, the machine learning device 210 performs self-supervised learning using the satellite image data 58 including the image data 58a and the position information 58b, etc., and the railway vehicle side data 200 including the captured image 56a, the position information 56b, and the label information 57 as learning data. Then, the estimated target image data 220 corresponding to the image data 58a is input to the learned model. Then, the learned model 222 can output the masked label information, that is, the estimation result of the necessity 224 of the weeding operation.

[0181] In addition, as a result of the satellite image data 58 and the railway vehicle side data 200 being input into the machine learning device 210 as learning data, an association between the satellite image data 58 and the railway vehicle side data 200 is learned from the commonality of the images or the commonality of the position information between the satellite image data 58 and the railway vehicle side data 200. As a result, a necessity determination result of the weeding operation corresponding to the label information 57 may be estimated from the estimated target image data 220 corresponding to the satellite image data 58.

[0182] The present disclosure discloses the following aspects.

[0183] A first aspect includes a storage unit that stores first source data including a plurality of first image data and second source data including a plurality of second image data. Each of the plurality of first image data includes a vegetation area that overlaps at least one of the plurality of second image data, and necessity information of a weeding operation is labeled for each of the plurality of second image data. A learning unit generates a learned model for estimating the necessity of a weeding operation based on estimated target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first source data and the second source data. The machine learning device has a higher resolution of the second image data than the resolution of the first image data.

[0184] According to this machine learning device, labeling of the necessity of a weeding operation for the first image data with relatively low resolution can be performed more appropriately than labeling of the necessity of a weeding operation for the second image data with relatively high resolution. And by appropriate labeling of the second image data, appropriate learning data is generated. By generating a learned model for estimating the necessity of a weeding operation based on the estimated target image data based on the learning data, the necessity of a weeding operation can be appropriately determined.

[0185] A second aspect is the machine learning device according to the first aspect, wherein the first image data is data indicating the distribution of vegetation indices.

[0186] In this case, a learned model that can appropriately estimate the necessity of the weeding operation can be generated by generating a learned model using the learning data in which the necessity of the weeding operation is labeled on the first image data showing the distribution of the vegetation index indicating the vegetation situation.

[0187] A third aspect is the machine learning device according to the first or second aspect, wherein the first image data is data detected by a sensor mounted on an artificial satellite.

[0188] Thereby, a learned model for estimating the necessity of the weeding operation can be generated based on the data obtained by observing a wide area.

[0189] A fourth aspect is the machine learning device according to any one of the first to third aspects, wherein the learning data is data obtained by adding terrain information to each of the plurality of first image data.

[0190] Thereby, a learned model taking into account the terrain information is generated.

[0191] A fifth aspect is the machine learning device according to any one of the first to fourth aspects, wherein the learning data is data obtained by adding data acquisition time information to each of the plurality of first image data.

[0192] The information at the time of data acquisition of the first image data may affect the value of the first image data or may affect the seasonality. Therefore, by learning based on the learning data including the first image data to which the data acquisition time information is added, a learned model that can appropriately determine the necessity of the weeding operation in consideration of the situation at the time of acquisition of the first image data can be generated.

[0193] A sixth aspect is the machine learning device according to any one of the first to fifth aspects, wherein the learning data is data obtained by adding growth factor information to each of the plurality of first image data.

[0194] Growth factor information can affect the future growth state of plants. By learning based on the training data including the first image data with growth factor information added, it is possible to appropriately estimate the necessity of weed control work considering future growth prediction.

[0195] A seventh aspect is a machine learning device according to any one of the first to sixth aspects, wherein the second image data is data detected by a sensor mounted on a railway vehicle.

[0196] Thereby, the vegetation state along the track can be easily grasped by the second image data detected by the sensor mounted on the railway vehicle, and it is easy to appropriately estimate the necessity of weed control work.

[0197] An eighth aspect is a machine learning device according to any one of the first to seventh aspects, further comprising a data generation unit that generates training data in which the necessity of weed control work is labeled for each of the plurality of first image data based on the first original data and the second original data.

[0198] Thereby, each of the plurality of first image data is learned based on the training data in which the necessity of weed control work is labeled.

[0199] A ninth aspect is a machine learning device according to the eighth aspect, wherein the data generation unit associates each of the plurality of first image data with at least one of the plurality of second image data, and labels the necessity information of weed control work for each of the plurality of first image data based on the necessity information of weed control work labeled on the at least one second image data associated therewith.

[0200] Thereby, based on the necessity information of weed control work labeled on the second image data, it is possible to label the necessity information of weed control work for the second image data.

[0201] The tenth aspect is a necessity determination system for a weeding operation, comprising an input unit to which estimation target image data is input, a learned model in which machine learning for estimating the necessity of a weeding operation is performed based on learning data, a processing unit that estimates the necessity of a weeding operation by inputting the estimation target image data into the learned model, and a result output unit that outputs a determination result of the necessity of a weeding operation based on an estimation result of the necessity of a weeding operation. The learning data is data based on a plurality of first image data and necessity information of a weeding operation based on second image data having a resolution higher than that of the first image data. The resolution of the estimation target image data is closer to the resolution of the first image data than the resolution of the second image data.

[0202] According to the tenth aspect, appropriate learning data is generated by labeling each of the plurality of first image data with the necessity of a weeding operation based on appropriate labeling for the second image data having a relatively high resolution. Based on the learning data, a learned model for estimating the necessity of a weeding operation based on the estimation target image data is appropriately generated. Based on the learned model, the necessity of a weeding operation is appropriately estimated.

[0203] The eleventh aspect is a necessity determination system for a weeding operation according to the tenth aspect, wherein the processing unit determines the priority order of the weeding operation based on the estimation result of the necessity of the weeding operation.

[0204] Thereby, the priority order of the weeding operation can be grasped.

[0205] The twelfth aspect is a necessity determination system for a weeding operation according to the tenth or eleventh aspect, wherein each of the plurality of first image data is data of a partial range located along a work target path, and the processing unit obtains a weeding operation path based on the estimation result of the necessity of the weeding operation and the position information of the plurality of first image data in the work target path.

[0206] As a result, considering the work target path, a weeding work path is obtained, enabling efficient weeding work.

[0207] The 13th aspect is a necessity determination system according to any one of the 10th to 12th aspects, wherein the result output unit is a display device that displays the determination result.

[0208] As a result, by visually recognizing the display device, the determination result of the necessity of weeding work can be recognized.

[0209] The 14th aspect is a necessity determination system according to any one of the 10th to 13th aspects, further comprising a machine learning device according to any one of the 1st to 8th aspects.

[0210] As a result, a learned model can be generated from learning data based on first original data including a plurality of first image data and second original data including a plurality of second image data, and by applying the estimation target image data to the learned model, a system capable of estimating and determining the necessity of weeding work can be constructed.

[0211] The method for determining the necessity of weeding work according to the 15th aspect prepares first original data including a plurality of first image data and second original data including a plurality of second image data, where the resolution of the second image data is higher than that of the first image data, each of the plurality of first image data includes a vegetation area overlapping at least one of the plurality of second image data, and necessity information of weeding work is labeled for each of the plurality of second image data. A learned model for estimating the necessity of weeding work is generated based on learning data based on the first original data and the second original data, based on estimation target image data having a resolution closer to the first image data than the second image data, and the necessity of weeding work is estimated by inputting the estimation target image data into the learned model.

[0212] Thus, a learned model can be generated using learning data based on first source data including a plurality of first image data and second source data including a plurality of second image data, and by applying the image data to be estimated to the learned model, it is possible to estimate the necessity of a weeding operation.

[0213] In addition, each configuration described in each of the above embodiments and each modification can be appropriately combined as long as they do not conflict with each other.

[0214] In addition, the functions of the elements disclosed in this specification can be executed using a circuit or processing circuit including a general-purpose processor, a dedicated processor, an integrated circuit, an ASIC (Application Specific Integrated Circuits), a conventional circuit, and / or a combination thereof configured or programmed to execute the disclosed functions. Since a processor includes transistors and other circuits, it is regarded as a processing circuit or a circuit. In the present disclosure, a circuit, unit, or means is hardware that executes the listed functions or hardware programmed to execute the listed functions. The hardware may be the hardware disclosed in this specification or other known hardware programmed or configured to execute the listed functions. When the hardware is a processor considered to be a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used for the configuration of the hardware and / or the processor.

[0215] The above description is illustrative in all aspects and the present invention is not limited thereto. It is understood that countless modifications not illustrated can be assumed without departing from the scope of the present invention.

Description of Reference Numerals

[0216] 10 Railway track 20 Railway vehicle 32 System for determining the necessity of a weeding operation 42 Image sensor 50 Processing device 51 Processor 54 Storage device 55 Program 56 Orbital state information 56a Captured image 57 Label information 58 Satellite image data 58a Satellite image 60 Estimation model 61 Estimation target image data 61a Satellite image 62 Maintenance proposal location information 64 Maintenance work schedule 68 Display device 70 Machine learning device 72 Data generation unit 74 Learning data 74a NDVI image 74b Additional information 74b1 Terrain information 74b2 Information at data acquisition time 74b3 Growth factor information 76 Learning unit 76a Model generation unit 76b Learning model 78a Arrow (movement path) 80 Weeding work necessity determination device 83 Estimation target data 83a NDVI image 83b Additional information 84 Inference processing unit 84a Trained model 85 Result output unit 91a Orbital state collection data 94 Artificial satellite 95 Sensor 100 Display image

Claims

1. A storage unit that stores first source data including a plurality of first image data and second source data including a plurality of second image data, wherein each of the plurality of first image data includes a vegetation area overlapping with at least one of the plurality of second image data, and whether weeding is required is labeled for each of the plurality of second image data; A learning unit that generates a learned model for estimating whether weeding is required based on estimated target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first source data and the second source data; Comprising: A machine learning device in which the resolution of the second image data is higher than the resolution of the first image data.

2. The machine learning device according to claim 1, wherein the first image data is data indicating the distribution of vegetation indices.

3. The machine learning device according to claim 1 or claim 2, wherein the first image data is data detected by a sensor mounted on an artificial satellite.

4. The machine learning device according to claim 1 or claim 2, wherein the learning data is data obtained by adding terrain information to each of the plurality of first image data.

5. The machine learning device according to claim 1 or claim 2, wherein the learning data is data obtained by adding data acquisition time information to each of the plurality of first image data.

6. The machine learning device according to claim 1 or claim 2, wherein the learning data is data obtained by adding growth factor information to each of the plurality of first image data.

7. The machine learning device according to claim 1 or claim 2, wherein the second image data is data detected by a sensor mounted on a railway vehicle.

8. The machine learning device according to claim 1 or claim 2, further comprising a data generation unit that generates learning data in which whether weeding is required is labeled for each of the plurality of first image data based on the first source data and the second source data.

9. The machine learning device according to claim 8, The data generation unit associates each of the plurality of first image data with at least one of the plurality of second image data, and labels the necessity information of the weeding operation for each of the plurality of first image data based on the necessity information of the weeding operation labeled in the at least one second image data associated therewith, a machine learning device.

10. An input unit to which estimation target image data is input; A processing unit including a learned model in which machine learning for estimating the necessity of a weeding operation is performed based on learning data, and estimating the necessity of a weeding operation by inputting the estimation target image data into the learned model; A result output unit that outputs a determination result of the necessity of a weeding operation based on the estimation result of the necessity of a weeding operation; Comprising: The learning data is data based on a plurality of first image data and necessity information of a weeding operation based on second image data having a resolution higher than that of the first image data; A system for determining the necessity of a weeding operation, comprising a device for determining the necessity of a weeding operation, wherein the resolution of the estimation target image data is closer to the resolution of the first image data than the resolution of the second image data.

11. The system for determining the necessity of a weeding operation according to claim 10, wherein the processing unit determines the priority order of the weeding operation based on the estimation result of the necessity of the weeding operation, a system for determining the necessity of a weeding operation.

12. The system for determining the necessity of a weeding operation according to claim 10 or claim 11, wherein each of the plurality of first image data is data of a partial range located along the work target path, and the processing unit obtains a weeding operation path based on the estimation result of the necessity of the weeding operation and the position information of the plurality of first image data in the work target path, a system for determining the necessity of a weeding operation.

13. The system for determining the necessity of a weeding operation according to claim 10 or claim 11, wherein the result output unit is a display device that displays the determination result, a system for determining the necessity of a weeding operation.

14. The system for determining the necessity of a weeding operation according to claim 10 or claim 11, further comprising the machine learning device according to claim 1 or claim 2, a system for determining the necessity of a weeding operation.

15. Prepare first original data including a plurality of first image data and second original data including a plurality of second image data, where the resolution of the second image data is higher than that of the first image data, each of the plurality of first image data includes a vegetation area overlapping at least one of the plurality of second image data, and necessity information for weeding work is labeled for each of the plurality of second image data. Generate a learned model for estimating the necessity of weeding work based on estimated target image data having a resolution closer to the first image data than the second image data, based on learning data based on the first original data and the second original data. A method for determining the necessity of weeding work, which estimates the necessity of weeding work by inputting the estimated target image data into the learned model.

Images