Harvesting vehicle

Abstract

To solve a problem of a harvesting work vehicle, in which it is difficult to detect a lodging state of a harvesting crop in a field at an early stage and harvest crops by distinguishing crops with preferable quality.SOLUTION: A harvesting vehicle is provided with an imaging device to perform image processing, analyze a state of a field in a stage of peripheral clipping by analysis using artificial intelligence by comparing with registration data, or the like, change a work route, and harvest crops by performing segmentation of crops.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to field crop analysis and field mapping using a harvesting vehicle equipped with an imaging device and a satellite positioning system, which allows for autonomous driving. [Background technology]

[0002] There are harvesting vehicles that are equipped with an imaging device and have the function of detecting whether crops have fallen over. (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-2475 Summary of the Invention [Problem to be solved by the invention]

[0004] In the prior art, there is a technique for detecting the state of lodging of a crop based on the width of an area that spreads at the same height as the height of the crop.

[0005] However, when an imaging device captures an area exceeding 20 m from a work vehicle, it is possible to detect the height of crops over a wide range, but due to the imaging angle, it may not be able to detect partially fallen areas, or it may be misdetected due to noise from weeds, etc.

[0006] This invention improves the accuracy of lodging detection by analyzing color shades from image data and using artificial intelligence to compare the color sequence with reference data. It aims to estimate the degree of lodging during perimeter harvesting at the beginning of work and determine the work process, estimate areas that cannot be photographed, and measure the condition of the field more accurately by incorporating additional data as work progresses. [Means for solving the problem]

[0007] The first aspect of the present invention is achieved by the following technical means.

[0008] When harvesting the periphery of the field, the weight and moisture of the crop are measured, the turning position of the work vehicle is set to the corner of the field, and the estimated weight and moisture of the crop in the unharvested area and the unharvested area are added to the registered field map data. 、 an imaging device mounted on a work vehicle photographs the field at predetermined intervals, converts the arrangement data of green and brown areas in the photographed images into a line image to determine the state of lodging of the crops, and adds the areas that will fall from the lodging state to the field map data; adding a route from the lodging area to farm field map data; The farm field map data is used as the data standard for automatic driving. .

[0009]

[0010]

[0011]

[0012]

[0013]

[0014]

[0015]

[0016] [Effects of the Invention]

[0017] According to the present invention By estimating the degree of lodging during the initial picking around the crops and determining the work schedule, it becomes easier to create work schedules and routes for separating and harvesting high-quality crops.

[0018] [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 1 is an external view of a harvesting vehicle according to an embodiment of the present invention. [Figure 2] A field map and information screen diagram of the harvesting vehicle of the present invention [Figure 3] Work flow diagram of the harvesting machine of the present invention using a field map [Figure 4]A block diagram of the image data state change of the imaging device of the present invention. [Figure 5] A block diagram showing a state change due to the combination of image data of the imaging device of the present invention. [Figure 6] Image of the lodged crop of the present invention [Figure 7] Image of lodged crops analyzed using the RGB color model of the present invention and converted into a dot image [Figure 8] Image of the dot image of the fallen crop obtained by averaging and linear processing [Figure 9] Image of the present invention's unlodged crop [Figure 10] Image of unlodged crops analyzed using the RGB color model of the present invention and converted into a dot image [Figure 11] Image of unlodged crops averaged and linearly processed using the dot images of the present invention [Figure 12] A diagram showing how the state of data from an imaging device is changed based on the judgment of artificial intelligence using a neural network according to the present invention. [Figure 13] Schematic diagram of how the unimaged region of the present invention is created from the captured image. [Figure 14] Schematic diagram of how the unimaged area of ​​the present invention is created from past image data [Figure 15] The field map of the present invention shows the state of lodging of crops in polygonal form. DETAILED DESCRIPTION OF THE INVENTION

[0020] The present invention will be described below with reference to the embodiments shown in the drawings.

[0021] The harvesting vehicle shown in Figs. 1 to 15 shows an example of this embodiment.

[0022] The harvesting vehicle of the present invention will now be described.

[0023] In FIG. 1, a combine harvester 1 as an example of a harvesting vehicle according to an embodiment of the present invention has a vehicle body 2. A pair of left and right traveling devices 11 are provided below the vehicle body 2. The traveling devices 11 according to the embodiment are, for example, configured as endless tracks, so-called crawlers. A riding section 12 on which an operator can ride is provided on the front right side of the vehicle body. A harvesting device 13 that harvests crops in the field is disposed at the front of the vehicle body. A transporting device 14 that transports harvested grain is disposed behind the harvesting device 13. A thresher 16 that threshers the grain transported by the transporting device 14 is disposed behind the transporting device 14. A grain tank 17 that stores grain processed by the thresher 16 is disposed to the right of the thresher 16. A discharge device 18 that discharges grain from the grain tank 17 into a container on a truck outside the field is connected to the rear of the grain tank 17. A straw discharge device 19 that discharges straw is disposed at the rear of the vehicle body.

[0024] The combine harvester 1 of the embodiment is equipped with a satellite positioning system as an example of a current position measurement unit. A receiver 41 of the satellite positioning system is installed on the top surface of the riding section 12. In Japan, the Global Navigation Satellite System version of the satellite positioning system is called GNSS, and therefore will be referred to as GNSS hereinafter.

[0025] The receiver 41 can receive signals from GNSS satellites and measure the current position of the combine harvester 1. Therefore, the combine harvester 1 of the embodiment can run autonomously (automatic running, unmanned running) using GNSS, or can run according to the operation of an operator who rides on the riding section 12 (manual running, manned running). Note that during autonomous running, the operator who operates a terminal capable of wireless communication with the combine harvester 1 can be outside the combine harvester 1 (outside or inside the field), or can ride on the riding section 12 while carrying the terminal.

[0026] The combine harvester 1 of this embodiment is equipped with an inertial measurement unit (IMU). This IMU 22 uses a three-axis gyro and a three-directional accelerometer to calculate three-dimensional angular velocity and acceleration, making it possible to derive the vehicle's heading and tilt. By using this information to correct the satellite positioning system described above, the vehicle's position can be measured more accurately.

[0027] In the combine harvester 1 of the embodiment, obstacle sensors 31, 32, and 33 as an example of crop detection means are installed at the front and on both the left and right sides of the vehicle body 2. The obstacle sensors 31 to 33 are capable of detecting crops and obstacles in front of and on both the left and right sides of the vehicle body 2. Note that although the obstacle sensors have been exemplified as an example of crop detection means, the present invention is not limited to this, and an imaging device such as a camera can also be used.

[0028] The combine harvester 1 of this embodiment is equipped with an imaging device 21 capable of capturing 360-degree omnidirectional images. The imaging device 21 is equipped with LIDAR. LIDAR is a type of optical remote sensing technology that measures scattered light from pulsed laser radiation to analyze the shape and properties of an object. It also enables measurement of the distance to an object, from long distances to short distances, and the object's moving speed. Therefore, it is possible to distinguish between people, other animals, and fixed objects. The imaging device 21 can also use millimeter-wave radar. This feature allows it to detect objects even during early morning work in conditions with morning fog, sudden weather changes, backlight, or when the combine harvester is in the shade. Furthermore, a device such as a CCD camera that converts light from a subject into an electrical signal using an optical system such as a lens may also be used. The imaging device 21 is equipped with any of these sensors, or a combination of these sensors, that can measure the shape, color, movement, and distance of an object.

[0029] The background of the present invention will be explained.

[0030] When harvesting vehicles are operated by humans, the degree of lodging of crops in the field is estimated based on human visual inspection and experience, and the work process may need to be changed accordingly.

[0031] For example, if there is a large amount of lodging in the center of a field, the lodged area can be determined from the overall field area, the amount that can be harvested can be estimated, and the field can be divided based on the lodged area. This can involve determining how best to plan the harvesting process, such as whether to proceed with the usual perimeter mowing, or to make a middle cut near the lodged area and mow the lodged area all at once.

[0032] However, unmanned robotic harvesting vehicles cannot make this judgment and operate in fields without distinguishing between the same type of crop, which can lead to the mixing of crops with poor quality crops due to lodging.

[0033] In this invention, by calculating the lodging state of crops in the field early on during edge cutting, it is possible to plan the harvesting process taking into account the subsequent transport vehicles, dryer classification at the drying and processing facility, etc. Furthermore, by gradually improving accuracy as the work progresses, it becomes possible to classify and harvest crops of higher quality, and even though the harvesting work is unmanned, it is possible to perform the harvesting work taking into account human judgment.

[0034] The details of the first invention of the present invention, in which unharvested areas and estimated crop weight and moisture data for the unharvested areas are added to farm field map data, will be described.

[0035] A map of the corresponding field is pre-registered on the harvesting vehicle. This field map data may be stored on the vehicle's terminal or in the cloud. The field map data also includes the field's perimeter registered in latitude and longitude, and the location of the work vehicle on the map can be displayed based on GNSS and IMU data.

[0036] A typical harvesting vehicle begins by mowing the perimeter of the field. The work vehicle of the present invention is a robotic work vehicle capable of unmanned operation, but it begins with a similar process in its work process. To enable automatic operation, this work is performed by remotely or manned operation to perform the mowing work on the perimeter of a field where many obstacles exist, and by determining a safe basic route and storing and registering turning positions and turning areas, it is used to determine the route for unmanned operation. Another effect is that the mowing process on the perimeter clearly indicates the unharvested area. This area is determined by enclosing it on the perimeter. Normally, the inside of the enclosure is considered the unharvested area, but it is also possible to switch between inside and outside.

[0037] The harvesting machine's grain tank is equipped with a weight scale and moisture meter, which can detect the weight of the grain when it is harvested in the outer periphery. The work vehicle's traveling speed and distance are measured, and it is possible to calculate the harvested area from the work vehicle's harvest width. This makes it possible to convert the weight per unit area of ​​the harvested grain in the outer periphery. In this way, it is possible to calculate the weight of the crop in the outer periphery, the weight of the unit area, the moisture and average moisture at each point in the outer periphery from the harvested grain, and it is also possible to estimate the estimated crop weight in the unharvested area, as well as the moisture and moisture deviation at each point in the unharvested area.

[0038] By measuring the weight and moisture content of the crop when the outer areas are harvested, and registering the weight and moisture content of the unharvested areas as field map data when the outer areas are harvested, the data can also be uploaded to the cloud, and at drying and processing facilities, the weight and moisture content of the crop in the unharvested areas can be used to select the dryer to load the crop into after harvesting in advance.For example, if the weight of the unharvested area is estimated to be 3 tons, and there are areas with a moisture content of 23% or more and 20% or less, at an estimated ratio of 2 tons and 1 ton, rather than loading the crop all at once into a 30-koku dryer, if there is one vacant 20-koku and one 10-koku dryer, then harvesting can be done so that the crop is distributed in this ratio, reducing fuel costs for the dryers and allowing quality to be differentiated by classification.

[0039] Furthermore, by registering the direction change position of the work vehicle as a corner, it becomes the reference point for turning during automatic driving, and by turning on the inside of that point, it can be used to prevent damage to the ridge by the driving part, to avoid obstacles at the edge of the ridge, and to determine the turning direction and moisture measurement points for the entire field. In particular, if you want to register the outer periphery for center cutting from the start, registering the direction change position as a corner is an effective setting.

[0040] Figure 2 shows a field map and information screen of the harvesting vehicle. Select the field to be worked on. 42 shows a diagram of operating a mobile terminal, but similar operations can be performed using the display screen equipped on the work vehicle. Since this invention is based on an unmanned vehicle, the explanation will be given using a mobile terminal.

[0041] Reference numeral 42 is an image of the field when the outer perimeter mowing is complete. The image acquisition by the imaging device and the calculation of images and information data of the unharvested areas will be explained later, but the system has all the image data and information data of the field, including estimation calculations, at the end of the outer perimeter mowing.

[0042] Image 42, as shown in Figure 15, displays the fallen areas of the entire field. These fallen areas are displayed as polygonal dots, with the color changing depending on the degree of lodging. For example, when a user touches the location of a fallen area with their finger, the information corresponding to that area is read out. In other words, when the outer periphery is harvested, various pieces of information are registered in conjunction with the location information. Image 43 is an image of the fallen area when it is actually photographed. By looking at this image, the user can check the condition of the field. Because work is done at high speed, some areas may not be fully visible, so the image data is useful information. Work vehicles use their imaging devices to sequentially photograph the area and analyze the traveling position and image data to create an aerial map of the field.

[0043] 44 is the content calculated from field information. It is possible to calculate the degree of lodging, and for example, calculate and display the lodging rate for the entire field using three levels: weak, medium, and strong. This information is linked to the corresponding area, and selecting the area number will display detailed data for the corresponding area.

[0044] The 45 displays information about the overall field. It displays the weight per unit area based on information from when the outer perimeter was harvested. It also calculates and displays the average moisture content. It estimates the unharvested area and calculates the estimated weight of the unharvested portion based on information from the outer perimeter harvest. Because this information is available at the time the outer perimeter was harvested, it is useful for later transporting the crop and preparing post-harvest equipment.

[0045] Reference numeral 46 is the setting screen for the work that will follow. Since the present invention is about dealing with fallen areas using an imaging device, the settings related to this will be explained, but there are many other settings that accompany it. It is possible to set how to deal with fallen areas, change the reaping process, perform intermediate reaping, and create images of unharvested areas.

[0046] Figure 3 explains the series of steps for adding the estimated crop weight and moisture data for the unharvested areas and the unharvested areas to the field map data.

[0047] This content explains a system of the present invention that determines the lodging state of crops, adds information to field map data, and maps the lodging state within unharvested areas.

[0048] For harvesting vehicles, it is desirable to start harvesting crops in a field by performing edge cutting (S3-1). By addressing the corners of the field as described above, the field map is divided into areas, making it possible to identify unharvested areas. This can be set by the user, but since the center of the field is often registered on the field map, the process of S3-2 can be determined automatically. If S3-3 is determined, this is a difficult state for autonomous driving, and a user decision is required. Conversely, comparing the center position of the field map with registered data makes it possible to determine where the unharvested area is located relative to the current position of the implement, allowing for immediate autonomous driving.

[0049] Reading the field corners (S3-4) enables calculation of the unharvested area (S3-7). The weight of the swath (S3-5), the moisture content of the swath (S3-6), and the harvest yield per unit area (S3-8) are calculated from the combine's running performance as described above. This allows calculation of the weight of the unharvested area (S3-9), enabling post-harvest support based on the estimated weight.

[0050] The unharvested area contains some fallen crops, and if this part is selected for harvesting in S3-10, the appropriate storage tank classification calculation S3-11 is performed as a corresponding operation. This operation involves harvesting the unharvested part separately, but this becomes the basic data for calculating the travel route S3-11 based on the estimated volume of the fallen part and the size of the storage tank.

[0051] Meanwhile, there is a process of creating field map data from this unharvested area. A field map basically contains the latitude and longitude of the outer perimeter and the center of the field. Field map data is constructed by adding data to this.

[0052] When the imaging of the perimeter mowing begins in S3-17, the imaging data is registered and associated with the field map according to the location where the image was taken. The data is compared with the satellite positioning system and registered as polygon data in the field data map as the first mapping.

[0053] Figure 15 shows an example of a screen that displays areas detected as lodged in the unharvested area. If you want to know the condition of the field, just touch the location on the screen you want to know about, and the data registered and associated with each area on the image will be read out. At the time the data was acquired, you can read out images of the crops, moisture, air temperature, soil temperature, weather, the degree of lodging as judged by the image, and the state of harvest of the rice ears. In addition, an overall image is displayed showing the percentage of lodged areas in the entire field, the moisture distribution of the entire field, and moisture deviation.

[0054] The degree of lodging in this field can be determined using image processing, as described below. A simple classification divides the degree of lodging into three stages: light lodging, moderate lodging, and severe lodging. This is done by calculating the angle of the lodged stalks and the height of the rice ears from the field.

[0055] Returning to the explanation of Figure 3, the location and area of ​​the fallen area can be estimated by creating data in step S3-18 for detecting the fallen area. Although the area inside the plant may not be clear if only the outer perimeter is mowed, this can be estimated using the image processing method of the present invention, which will be described later, and the data is added to the field map data as the first map and growth registration S3-19.

[0056] Furthermore, the storage tank appropriate classification calculation S3-11 adds data to the second map, which shows the route that would be taken if harvested using normal procedures, and the yield registration S3-12. Data such as the estimated weight of the entire field, the degree of lodging, and the area and ratio of lodging are added to the field map.

[0057] Furthermore, the planned driving route for actual harvesting is registered as a third map in driving route registration S3-14 after calculating the driving route S3-13. The field map is further expanded with the information data from the first, second, and third maps to create a fourth map in classification registration S3-20, and this field map data is sent to the cloud 226 and provided to the user.

[0058] The user checks this field map data and confirms that there are no problems with the driving route or harvesting divisions, and then confirms the registered details in S3-15. Once each map has been corrected in S3-16, a field map is created and registered in S3-21, and the edge harvesting work is complete. From this point on, automated driving will begin based on the registered field map data. The registered field map overlays the harvested crop quality and field area, and remains as image data even after harvesting is complete, and can be used as basic data for the next season's cultivation. By doing this for each field, a user database will be built.

[0059] FIG. 4 is a block diagram showing the use of images from an imaging device.

[0060] The imaging device performs crop color analysis 50, shape analysis 60, and height (distance) analysis 70. However, because images contain a wide variety of information, much of the data is unnecessary. Masking is inevitably required, and each analysis calculation determines the amount of masking to remove non-standard data. In this invention, this corresponds to crop distance analysis 90. A color standard is also set for greens, and a distinction is made between crops and weeds based on saturation and density. Weeds are masked as noise. The same judgment is made for browns as for greens, determining damaged rice or rice malt, which are then subject to masking. Masking is based on length, as shown in 91 and 92.

[0061] The black line at position 93 is used to mask the image of the field. By masking the lower black line from the image, assuming that the image represents the field, it becomes possible to analyze the crop image more accurately. When determining whether the crop has fallen, there is also a method of removing noise by using magnification and a wide angle. In the present invention, the masking process is a method of not incorporating unknown information into the calculation, which will be explained in a later section.

[0062] Image data is difficult to handle, so image processing is performed to convert it into regions. These regions can be represented as a collection of small rectangles, and the whole is displayed as a polygon. The outer periphery is not a smooth line type, but the resolution is within a range that is acceptable for field analysis.

[0063] In the color analysis 50, green polygon areas are determined to identify stalks, and brown polygon areas are determined to identify ears of rice. The imaging data is an imaging system that can display green polygon areas in dots, and it is also possible to display linear color analysis in gradually different colors. The dot display of brown polygon areas has a similar function.

[0064] Even if the image is taken from the same position, the imaging device compares both the enlarged and wide-angle images to remove noise caused by lighting conditions and clearly capture the arrangement of greens and browns. This corresponds to the controls at 52 and 53 in Figure 4. A basic map of the fallen state is registered, and whether or not the plant has fallen is determined based on the degree of agreement with this map data.

[0065] Shape analysis 60 analyzes the shape of the crop. Color analysis 50 primarily analyzes the arrangement of brown and green tones, which allows for a general assessment of the lodging state of the crop, but also analyzes the relative position of the color in the field. Because the imaging device can measure distance, it is possible to determine the position of the rice ear and stalk. Figure 4 shows two imaging devices, 21 and 23 in Figure 1. The difference in these positions is used to determine the three-dimensional shape of the object. Rice Ear Position Confirmation 1 and Rice Ear Position Confirmation 2 confirm the shape of the rice ear and analyze its position from the field. Simultaneously, plant height 1 and plant height 2 determine the shape of the stalk and the shape of weeds, allowing the height of the stalk from the field to be measured. It is also possible to determine the direction of the crop from the position of the stalk and rice ear. Installing multiple imaging devices in this way allows the depth of the object to be determined from the difference in imaging angle, and also prevents noise due to lighting conditions, making the above measurements easier.

[0066] The field data 80 includes crop moisture at the time of harvesting, the air temperature around the field, and the soil temperature of the field, and by comprehensively analyzing the data from the color analysis 50 and shape analysis 60, it is possible to estimate and calculate the degree of lodging 95, lodging area 96, and grain quality 97.

[0067] The method for changing the state of the image for the collapse analysis in the captured image is explained in Figures 6 to 11, and the analysis method using artificial intelligence with a neural network, which is a method for changing the state of the input and output of each sensor to change the state, is explained in Figure 12.

[0068] We will now explain a system of the present invention that compares data on the arrangement of green and brown areas, determines the lodging state of crops, adds information to field map data, and maps the lodging state in unharvested areas.

[0069] When harvesting the outer periphery of a field, an image of the field is taken at predetermined intervals using an image capture device mounted on a work vehicle, and the image data is displayed as color-coded dots to determine the condition of the imaged items.

[0070] Figure 6 shows the state of fallen rice plants captured by the imaging device 21. Rice stalks 101, 102, and 103 are tilted horizontally relative to the ground, while rice ears 111, 112, and 113 are oriented vertically relative to the ground. Similarly, weeds 104 and 105 in the surrounding area often fall horizontally as they are pulled along by the fallen rice plants.

[0071] This image data was analyzed using the RGB color model and converted into a dot image, as shown in Figure 7. Since the drawing cannot be displayed in color, we will explain it in words: rice stalks 101A, 102A, and 103A are green dots and are tilted horizontally relative to the ground, while rice ears 111A, 112A, and 113A are brown dots and are oriented vertically relative to the ground. Weeds 104A and 105A are green dots and are tilted horizontally relative to the ground.

[0072] Furthermore, the wavelengths of light from the greens that can be identified as stems or weeds and the browns that can be identified as rice ears are quite different, making them easy to distinguish. They are also different from the black that is the ground in the field, and by analyzing several images of data, misrecognition and inability to analyze are greatly reduced.

[0073] Figure 8 shows the dot image data after averaging and linear processing. This processing makes the relationship between the stalks and the ears of rice clear. Rice stalks 101B, 102B, and 103B are green lines that are tilted horizontally relative to the ground, while ears of rice 111B, 112B, and 113B are brown lines that are oriented vertically relative to the ground. Weeds 104B and 105B are green lines that are tilted horizontally relative to the ground. From this line image, the intersection angles α1 and α2 of the green and brown lines are detected, and these are used to determine the lodging state of the crop.

[0074] 9 shows rice plants in an unlodged state captured by imaging device 21. Rice stalks 101C and 103C extend vertically relative to the ground, and rice ears 111C and 113C also extend vertically relative to the ground. Similarly, weeds 104C in the surrounding area also extend vertically.

[0075] This image data was analyzed using the RGB color model and converted into a dot image, as shown in Figure 10. Rice stalks 101D and 103D are green dots that extend vertically relative to the ground, while rice ears 111D and 113D are brown dots that also extend vertically relative to the ground. Weeds 104D are green dots that also extend vertically relative to the ground.

[0076] Figure 11 shows the dot image data averaged and linearly processed. This processing makes the relationship between the stalks and the ears of rice clear. Rice stalks 101E and 103E are green lines that extend vertically relative to the ground, while rice ears 111E and 113E are brown lines that extend vertically relative to the ground. Weeds 104E are green lines that extend vertically relative to the ground. From this linear image, the intersection angles α3 and α4 of the green and brown lines are detected, and these are used to determine the lodging state of the crop.

[0077] As explained in the overview of image state modification, it is possible to convert the arrangement data of green and brown regions into a line image, detect the intersection angle of the green and brown lines, and calculate the degree of lodging based on the intersection angle. In this way, by decomposing and analyzing the image data into individual elements and comparing the relationships between the analyzed information with existing data, it is possible to perform calculations to establish a different information judgment, thereby making it possible to make judgments that are closer to human experience. As an example of how the present invention determines the lodging state of crops, a judgment using artificial intelligence using a neural network, as shown in Figure 12, and a judgment to change the state of the image data from the imaging device is an effective strategy.

[0078] In Figure 12, we will explain the details of the second invention, in which the image captured by the imaging device of the present invention is calculated using a state change system to determine the arrangement of green and brown areas, and then compared with pre-registered data on the arrangement of green and brown areas to determine the lodging state of the crop.

[0079] In this example, three image capture devices are installed. Each image capture device has similar functions, but by utilizing the differences in their installation positions, the accuracy of the three-dimensional shape and color analysis of the image can be further improved, and the main analysis data can be differentiated. The noise removal method has differences in structure and data processing, which can further improve the accuracy of the image analysis.

[0080] The data from the first imaging device 21 and the obstacle sensors 31, 32, and 33 is mainly processed by the crop size detection unit 201. The data for each element from the imaging device and the data for each element from each obstacle sensor are combined and judged in a new way, and control is performed to change the state of the information to one that can determine the size of an object. As a first state change, the information content is changed to the height 203 of the entire crop, the width 204 of the entire crop, the height 205 of the stalk, and the position of the crop part (fruit), which in this example corresponds to the ear of rice, 206.

[0081] The second image capture device 23 mainly comprises a crop color detection unit 202. By combining and judging the elemental data of the image capture device, control is exercised to change the state of the information to determine the color of the object. As a first state change, the information content changes to a green direction 207, a brown direction 208, a masking amount determination 209, and a crop density 210.

[0082] Due to these state changes, the image state of FIG. 6 changes to that of FIG. 8, and that of FIG. 9 changes to that of FIG. 11, and the image state is changed by the decision unit 211.

[0083] As described above, the reference data to be compared is registered in data storage unit 216, and image matching is performed by comparing 211 and 216 and matching similar data, but the degree of similarity and whether it is within the range of image state change is calculated again by first state change units 201 and 202 to arrive at a determination of falling 202, a determination of a fallen area 213, a determination of the degree of falling 214, and an estimate of quality 215. This second state change is a combination of state change, matching, and judgment in which a new combination of element data is matched with the registered reference data, and is based on the so-called artificial intelligence function.

[0084] This function selects an ideal route in the work route change and determination unit 217, and the route is reflected in the automatic driving function by carrying out the work. However, for unmanned operation, the work route must be determined by checking it against various setting data.

[0085] The external data is how to revise the preset route setting 228. The field map data obtained as a result of reaping the outer periphery is also used as the environment map 227. This information is exchanged with the user via the communication unit 229 and the cloud 226. The third imaging device 24 also analyzes the state of the field in front of the work vehicle and the state of the reaping unit, and data from the satellite positioning device 22 is also added to determine the work route change and determination unit 217.

[0086] The artificial intelligence in the embodiment controls the traveling unit 218 and the working unit 221. By determining the work route 217, the traveling speed 219 and traveling direction 220 are determined and the machine starts traveling. In addition, by determining the working height 222, the inclination of the working unit 223, the rotation speed of the working unit 224, and the control of the harvesting unit 225, it becomes possible to perform unmanned operation using the artificial intelligence.

[0087] A third aspect of the invention, which is a method for creating image data of an area that cannot be imaged, beyond the imaging range of the imaging device, will now be described.

[0088] 13 shows the relationship between the harvest vehicle 250, the imaging range 302 of the imaging device, and the field 240. The harvest vehicle proceeds with circumferential mowing in a counterclockwise direction, moving in the following order: 250 ⇒ 251 ⇒ 252 ⇒ 253 ⇒ 254 ⇒ 255 ⇒ 256 ⇒ 257.

[0089] The white arc indicates the range that the imaging device can reliably capture. While it is possible to capture much of the travel route by increasing the number of captures, this invention utilizes a method of stretching and utilizing existing captured data. For example, after capturing the image of area 301, which is the forward area that has been captured, a gradation of the image of the interior of 301 as seen from above is created. Using this arrangement, the data for 301A, 301B, and 301C are expanded at the same ratio to create area 302. As the harvesting vehicle travels from 251 to 252, it can capture actual images of area 302, allowing the actual captured image to be compared with the image of 302. This data comparison calculates the difference. If there is a difference in the ratio of the degree of lodging, the ratio is calculated. The moving average of these ratios is calculated to derive a coefficient for correcting the estimated image, such as 302. The neural network function shown in Figure 12 also handles this correction coefficient calculation. The image state change and determination unit 211 corresponds to the estimated image 302, and the data storage unit 216 corresponds to the actually captured image. Using this second state change calculation, the degree of falling 212, the fallen area 213, the degree of falling 214, and the quality estimation 215 are determined, and at the same time, the image video from above 302 is corrected.

[0090] This captured image is corrected to create an estimated image of the uncaptured area using a technique for creating an estimated image of the uncaptured area. Explained with reference to Fig. 13, a correction coefficient is acquired from the image data of 302, and the image gradation of the sky 303A, 303B, and 303C is used in the captured data of the area 303 to create an image 304. The correction coefficient is used to correct not only the image, but also the data for determining the degree of lodging 212, determining the area of ​​lodging 213, determining the degree of lodging 214, and estimating the quality 215.

[0091] This function creates estimated image data of unimaged areas from image data captured based on this correction coefficient, and accuracy can be improved by sequentially overlaying the estimated data. It also has a learning function that repeatedly calculates, corrects, and registers the contents of changes made one after another, and weights the contents of the image changes and changes to the fallen data, etc., with more weighting applied to these changes.

[0092] By using the first and second state changes shown in Figure 12, the state of crops in the field can be determined by comparing, collating, removing, analyzing, and changing the data stored in the imaging device, obstacle sensor, and data storage unit 216, which was previously determined by human vision and experience.

[0093] Further, as an expanded function, the traveling speed 219 and traveling direction 220 of the traveling unit 218 are controlled. The working unit control 221 controls the working unit height 222, the working unit inclination 223, the working unit rotation speed 224, and the part that is the output of the harvesting unit control 225.

[0094] In conventional technology, for example, the height of the crop is determined from distance information and image data from each imaging device and information from obstacle sensors, and the control directly outputs traveling unit control 218 and working unit control 221. However, in the present invention, this is done by performing a first state change and a second state change before outputting traveling unit control 218 and working unit control 221, which not only improves control accuracy but also makes it possible to change the work path while working by obtaining detailed information about the state of the crop, and the output control is changed based on this change.

[0095] Therefore, the first state change and the second state change can only be analyzed by judgment using artificial intelligence, and the traveling unit control 218 and working unit control 221 are also considered to be unique controls that make use of judgment control using artificial intelligence that utilizes a neural network.

[0096] The fifth invention explains the control leading up to the aforementioned output, in which the size of the crop is detected from the imaging device 21, the color of the crop is detected from the imaging device 23, the RGB element analysis of the image is performed, and based on the arrangement of green and brown in the image, the lodging area of ​​the crop is determined 213, the degree of lodging is determined 214, and the quality is estimated 215, and comparison calculations are made with the data registration unit 216 to control the traveling unit 218 and the working unit 221.

[0097] Regarding the fourth invention, which involves overlaying an image on previous data, weighting the dot data of the overlapping areas to the side with higher image accuracy, and synthesizing them to estimate the degree of lodging, weight per unit area, and moisture content of each unharvested area, in addition to the explanation in the previous section, we will explain the correspondence of areas where image data overlap in Figure 13.

[0098] In Figure 13, this corresponds to the area 314. Although the image was taken from different directions, it is the same area, and both captured images can be referenced. However, the polygonal image viewed from above must be synthesized. In this case, both images are added together and synthesis is performed with weighting applied to the image with higher image accuracy. In Figure 13, the image 314 is an image of the work vehicle 257 in front of the imaging device, and because it is more accurate, synthesis processing is performed with weighting applied to the image in front of the work vehicle 257.

[0099] The following describes how to handle past data when the image capturing range of the imaging device is exceeded.

[0100] The actual imaged area is used to create estimated image data. However, when the area far exceeds the range that can be estimated, reliability becomes low, so the approach shown in Figure 14 is taken. The method used is to reuse past data. Last year's field data is registered in the data storage unit, and past image data is used to provisionally display the unimaged areas. This past image data includes data for estimating the degree of lodging, weight per unit area, and moisture content.

[0101] By touching an unphotographed area on the image, the user can select between estimated image data or past image data. Based on the selected image data, the system has a function to display the degree of lodging, weight per unit area, and moisture content. Based on this estimated data, the user can change the route or adjust the harvest classification.

[0102] If no past image data exists, the data is displayed as blank, or data close to the relevant area is used while displaying that the data accuracy is low.

[0103] The image data from these imaging devices cannot capture the entire field in one image; the image data must be combined. Figure 5 is a block diagram showing a standard image combination. Fields are often rectangular, with four sides and a square. Therefore, by using the data for the square and the center data for the four sides, it is possible to capture a minimum amount of field data. A simple technique for combining these images is to change the color to that seen from above, and by combining the images while performing RGB color analysis between the square and the four sides in order, it is possible to create an aerial image of the field using imaging devices installed on a work machine on the ground.

[0104] Figure 5 outlines the procedure for combining the images and the final process for creating a map of the lodging area and degree of lodging in the field. [Explanation of symbols]

[0105] 1. Combine 21 First imaging device 22 Inertial Measurement Unit (IMU) 23 Second imaging device 24 Third imaging device 31, 32, 33 Obstacle sensors 41 Satellite positioning system receiver (GNSS receiver) S3-12 Second map (yield) S3-14 Third Map (Driving Route) S3-19 First Map (Growth) S3-20 Fourth Map (Division) 201 Crop size detection unit 202 Crop color detection unit 211 Image status change and determination unit

Claims

[Claim 1] The weight and moisture of the crop are measured when the periphery of the field is harvested, the direction change position of the work vehicle is set to the corner of the field, and the estimated weight and moisture of the crop in the unharvested area and the unharvested area are added to the registered field map data; an imaging device mounted on a work vehicle photographs the field at predetermined intervals, converts the arrangement data of green and brown areas in the photographed images into a line image to determine the state of lodging of the crops, and adds the areas that will fall from the lodging state to the field map data; adding a route from the lodging area to farm field map data; A harvesting vehicle that uses the field map data as a data standard for automatic driving.

Images