Harvesting system

Abstract

To solve the problem that it is difficult to detect the lodging state of harvested crops in an early stage and to separately harvest the crops having good quality in the harvesting work of the crops in a field.SOLUTION: The working vehicle is equipped with a system for photographing a field with an imaging device mounted on the working vehicle, calculating the photographed image of the imaging device in the arrangement of a color area showing the state of the crop by a state changing system, comparing it with previously registered arrangement data, judging the lodging state of the crop and mapping the lodging state in an unharvested area.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a harvesting system that is equipped with an imaging device on a work vehicle and that enables analysis of crops in a field and creation of a field map. [Background technology]

[0002] There are harvesting vehicles that are equipped with an imaging device and have the function of detecting whether crops have fallen over. (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-2475 Summary of the Invention [Problem to be solved by the invention]

[0004] In the prior art, there is a technique for detecting the state of lodging of a crop based on the width of an area that spreads at the same height as the height of the crop.

[0005] However, when an imaging device captures an area exceeding 20 m from a work vehicle, it is possible to detect the height of crops over a wide range, but due to the imaging angle, it may not be able to detect partially fallen areas, or it may be misdetected due to noise from weeds, etc. This invention improves the accuracy of lodging detection by analyzing color shades from image data and using artificial intelligence to compare the color sequence with reference data. It aims to estimate the degree of lodging during the early stage of work, when cutting around the crop, and determine the work process, as well as to estimate and determine areas that cannot be photographed, and to measure the condition of the field more accurately by incorporating additional data as work progresses. [Means for solving the problem]

[0006] The present invention is solved by the following technical means. The system is equipped with a camera mounted on a work vehicle to photograph the field, and a status change system calculates the image captured by the camera as a sequence of color areas indicating the state of the crops, compares it with pre-registered sequence data, determines the lodging state of the crops, and maps the lodging state within the unharvested area. In addition, the weight and moisture content of the harvested crops are measured when the work vehicle is harvesting, and the position where the work vehicle changes direction is determined to be a corner of the field.The estimated crop weight and moisture data, which are consistent with the lodging state of the crops in the unharvested area, are added to the pre-registered field map data and used as data for automatic driving. In addition, an imaging device equipped on a work vehicle was used to create an RGB image gradation from an image of an area captured when the work vehicle was harvesting a portion of the crops in the field. This RGB image gradation is used to create an estimated image of the unimaged area at the same ratio, and actual captured images are created as the work vehicle progresses, and the estimated image and the actually captured image are compared to calculate a correction coefficient for correcting the difference between the RGB images. The system had the function of creating estimated image data for unimaged areas from image data captured by the imaging device after the next harvesting operation, using this correction coefficient as a reference. In addition, as the work vehicle progresses, the actual image data and estimated image data are added by the imaging device, and within the status change system, the images are overlaid on the field map data from the previous time, and the dot data of the overlaid area is weighted and combined with the actual image data to estimate the degree of lodging in the unharvested areas, the weight per unit area, and the moisture value. The problem is also solved by the following technical means. An imaging device installed on the work vehicle performs RGB image analysis of the image, and the arrangement of green and brown crops in the image determines the area of ​​lodging, degree of lodging, or quality of the crops. This is then compared with the field map data to control the traveling or working parts of the work vehicle during automatic driving. [Effects of the Invention]

[0007] According to the present invention, by determining the work process by estimating the degree of lodging when cutting around the crops at the beginning of the work, it becomes easier to create work processes and work routes for separating and harvesting high-quality crops. In addition, by estimating areas that cannot be photographed and responding to additional data as work progresses, the condition of the field can be measured more accurately and quickly, making it possible to correct the harvesting route and extract high-quality crops. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is an external view of a harvesting vehicle according to an embodiment of the present invention. [Figure 2] A field map and information screen diagram of the harvesting vehicle of the present invention [Figure 3] Work flow diagram of the harvesting machine of the present invention using a field map [Figure 4] A block diagram of the image data state change of the imaging device of the present invention. [Figure 5] A block diagram showing a state change due to the combination of image data of the imaging device of the present invention. [Figure 6] Image of the lodged crop of the present invention [Figure 7] Image of lodged crops analyzed using the RGB color model of the present invention and converted into a dot image [Figure 8] Image of the dot image of the fallen crop obtained by averaging and linear processing [Figure 9] Image of the present invention's unlodged crop [Figure 10] Image of unlodged crops analyzed using the RGB color model of the present invention and converted into a dot image [Figure 11] Image of unlodged crops averaged and linearly processed using the dot images of the present invention [Figure 12] A diagram showing how the state of data from an imaging device is changed based on the judgment of artificial intelligence using a neural network according to the present invention. [Figure 13] Schematic diagram of how the unimaged region of the present invention is created from the captured image. [Figure 14]Schematic diagram of how the unimaged area of ​​the present invention is created from past image data [Figure 15] The field map of the present invention shows the state of lodging of crops in polygonal form. DETAILED DESCRIPTION OF THE INVENTION

[0009] The present invention will be described below with reference to the embodiments shown in the drawings.

[0010] As shown in Figures 1 to 15 harvest The work vehicle is an example of this embodiment.

[0011] The present invention harvest The work vehicle will now be described.

[0012] In FIG. 1, a combine harvester 1 as an example of a harvesting vehicle according to an embodiment of the present invention has a vehicle body 2. A pair of left and right traveling devices 11 are provided below the vehicle body 2. The traveling devices 11 according to the embodiment are, for example, configured as endless tracks, so-called crawlers. A riding section 12 on which an operator can ride is provided on the front right side of the vehicle body. A harvesting device 13 that harvests crops in the field is disposed at the front of the vehicle body. A transporting device 14 that transports harvested grain is disposed behind the harvesting device 13. A thresher 16 that threshers the grain transported by the transporting device 14 is disposed behind the transporting device 14. A grain tank 17 that stores grain processed by the thresher 16 is disposed to the right of the thresher 16. A discharge device 18 that discharges grain from the grain tank 17 into a container on a truck outside the field is connected to the rear of the grain tank 17. A straw discharge device 19 that discharges straw is disposed at the rear of the vehicle body.

[0013] The combine harvester 1 of the embodiment is equipped with a satellite positioning system as an example of a current position measurement unit. A receiver 41 of the satellite positioning system is installed on the top surface of the riding section 12. In Japan, the Global Navigation Satellite System version of the satellite positioning system is called GNSS, and therefore will be referred to as GNSS hereinafter.

[0014] The receiver 41 can receive signals from GNSS satellites and measure the current position of the combine harvester 1. Therefore, the combine harvester 1 of the embodiment can run autonomously (automatic running, unmanned running) using GNSS, or can run according to the operation of an operator who rides on the riding section 12 (manual running, manned running). Note that during autonomous running, the operator who operates a terminal capable of wireless communication with the combine harvester 1 can be outside the combine harvester 1 (outside or inside the field), or can ride on the riding section 12 while carrying the terminal.

[0015] The combine harvester 1 of this embodiment is equipped with an inertial measurement unit (IMU). This IMU 22 uses a three-axis gyro and a three-directional accelerometer to calculate three-dimensional angular velocity and acceleration, making it possible to derive the vehicle's heading and tilt. By using this information to correct the satellite positioning system described above, the vehicle's position can be measured more accurately.

[0016] In the combine harvester 1 of the embodiment, obstacle sensors 31, 32, and 33 as an example of crop detection means are installed at the front and on both the left and right sides of the vehicle body 2. The obstacle sensors 31 to 33 are capable of detecting crops and obstacles in front of and on both the left and right sides of the vehicle body 2. Note that although the obstacle sensors have been exemplified as an example of crop detection means, the present invention is not limited to this, and an imaging device such as a camera can also be used.

[0017] The combine harvester 1 of this embodiment is equipped with an imaging device 21 capable of capturing 360-degree omnidirectional images. The imaging device 21 is equipped with LIDAR. LIDAR is a type of optical remote sensing technology that measures scattered light from pulsed laser radiation to analyze the shape and properties of an object. It also enables measurement of the distance to an object, from long distances to short distances, and the object's moving speed. Therefore, it is possible to distinguish between people, other animals, and fixed objects. The imaging device 21 can also use millimeter-wave radar. This feature allows it to detect objects even during early morning work in conditions with morning fog, sudden weather changes, backlight, or when the combine harvester is in the shade. Furthermore, a device such as a CCD camera that converts light from a subject into an electrical signal using an optical system such as a lens may also be used. The imaging device 21 is equipped with any of these sensors, or a combination of these sensors, that can measure the shape, color, movement, and distance of an object.

[0018] The background of the present invention will be explained.

[0019] harvest When a work vehicle is manned, the degree of lodging of crops in the field is estimated based on human visual inspection and experience, and the work schedule may need to be changed accordingly.

[0020] For example, if there is a large amount of lodging in the center of a field, the lodged area can be determined from the overall field area, the amount that can be harvested can be estimated, and the field can be divided based on the lodged area. This can involve determining how best to plan the harvesting process, such as whether to proceed with the usual perimeter mowing, or to make a middle cut near the lodged area and mow the lodged area all at once.

[0021] However, autonomous driving robots harvest Vehicles cannot make this judgment, so they work in fields without distinguishing between the same type of crop, which results in poor quality crops being mixed in due to lodging.

[0022] In this invention, by calculating the lodging state of crops in the field early on during edge cutting, it is possible to plan the harvesting process taking into account the subsequent transport vehicles, dryer classification at the drying and processing facility, etc. Furthermore, by gradually improving accuracy as the work progresses, it becomes possible to classify and harvest crops of higher quality, and even though the harvesting work is unmanned, it is possible to perform the harvesting work taking into account human judgment.

[0023] The details of the first invention of the present invention, in which unharvested areas and estimated crop weight and moisture data for the unharvested areas are added to farm field map data, will be described.

[0024] A map of the corresponding field is pre-registered on the harvesting vehicle. This field map data may be stored on the vehicle's terminal or in the cloud. The field map data also includes the field's perimeter registered in latitude and longitude, and the location of the work vehicle on the map can be displayed based on GNSS and IMU data.

[0025] In a typical harvesting vehicle, the process of harvesting the perimeter of the field is carried out first. The work vehicle of the present invention is a robotic work vehicle that allows for unmanned operation, and it first carries out a similar process in the work process. In order to enable automatic operation, this work is carried out by remote operation or manned operation when harvesting the perimeter of a field where there are many obstacles, and by determining a safe basic route and storing and registering turning positions and turning areas, it is used to determine the route in the case of unmanned operation. Other effects include the following: harvest The area is clearly indicated by the cutting process on the perimeter. The area is determined by enclosing it on the perimeter. Usually, the inside of the enclosure is left uncovered. harvest It is possible to switch between inside and outside.

[0026] harvestThe work machine's grain tank is equipped with a weight scale and moisture meter, which can detect the weight of the grain when it is harvested on the periphery. The working vehicle's travel speed and distance are measured, and it is possible to calculate the harvested area from the harvest width of the working vehicle. Therefore, it is possible to convert the weight per unit area of ​​the harvested grain on the periphery. In this way, it is possible to calculate the weight of the crop on the periphery, the weight of the unit area, the moisture content and average moisture content at each point on the periphery from the harvested grain on the periphery, and the weight per unit area can be calculated. harvest Estimated crop weight in the area and harvest It is also possible to estimate the moisture content and moisture deviation at each point in the region.

[0027] By measuring the weight and moisture content of the crop when harvesting the outer periphery, the weight and moisture content of the unharvested part can be registered as field map data at the stage of harvesting the outer periphery. This data can also be uploaded to the cloud, and at the drying and processing facility, the weight and moisture content of the unharvested part can be registered as field map data. harvest It is possible to select the dryer to be used after harvesting based on the weight and moisture content of the crops in the area. harvest If the weight of the area is estimated to be 3 tons, and there are areas with moisture contents of 23% or more and 20% or less, with 2 tons and 1 ton respectively, rather than loading the rice all into a 30-kilometer dryer and drying it all at once, if there is one 20-kilometer and one 10-kilometer dryer available, then harvesting can be done so that the rice is distributed in this ratio, which will reduce fuel costs for the dryers and also enable quality to be managed by dividing the rice into categories.

[0028] Furthermore, by registering the direction change position of the work vehicle as a corner, it becomes the reference point for turning during automatic driving, and by turning on the inside of that point, it can be used to prevent damage to the ridge by the driving part, to avoid obstacles at the edge of the ridge, and to determine the turning direction and moisture measurement points for the entire field. In particular, if you want to register the outer periphery for center cutting from the start, registering the direction change position as a corner is an effective setting.

[0029] Figure 2 shows a field map and information screen of the harvesting vehicle. Select the field to be worked on. 42 shows a diagram of operating a mobile terminal, but similar operations can be performed using the display screen equipped on the work vehicle. Since this invention is based on an unmanned vehicle, the explanation will be given using a mobile terminal.

[0030] 42 is an image of the field when the harvesting of the outer periphery was completed. harvest The calculation of the images and information data of the field will be explained later, but by the time the edge cutting is completed, the system has all the image data and information data of the field, including the estimated calculations.

[0031] Image 42, as shown in Figure 15, displays the fallen areas of the entire field. These fallen areas are displayed as polygonal dots, with the color changing depending on the degree of lodging. For example, when a user touches the location of a fallen area with their finger, the information corresponding to that area is read out. In other words, when the outer periphery is harvested, various pieces of information are registered in conjunction with the location information. Image 43 is an image of the fallen area when it is actually photographed. By looking at this image, the user can check the condition of the field. Because work is done at high speed, some areas may not be fully visible, so the image data is useful information. Work vehicles use their imaging devices to sequentially photograph the area and analyze the traveling position and image data to create an aerial map of the field.

[0032] 44 is the content calculated from field information. It is possible to calculate the degree of lodging, and for example, calculate and display the lodging rate for the entire field using three levels: weak, medium, and strong. This information is linked to the corresponding area, and selecting the area number will display detailed data for the corresponding area.

[0033] 45 displays the overall information of the field. It displays the weight per unit area based on the information when the outer periphery is harvested. It also calculates and displays the average moisture value. harvest The area of ​​the area is estimated and the remaining area is determined based on the information on the surrounding area. harvestThis information is available at the time of harvesting the outer periphery, and is useful for transporting the crop and preparing post-harvest equipment.

[0034] 46 is the setting screen for the work that will be carried out from now on. Since the present invention is about dealing with fallen areas using an imaging device, the explanation will be made on the settings related to this, but there are many other settings that accompany it. harvest Allows you to set how to handle image creation for a region.

[0035] Figure 3 explains the series of steps for adding the estimated crop weight and moisture data for the unharvested areas and the unharvested areas to the field map data.

[0036] This content explains a system of the present invention that determines the lodging state of crops, adds information to field map data, and maps the lodging state within unharvested areas.

[0037] harvest It is desirable for the work vehicle to start harvesting crops in the field by carrying out the edge harvesting S3-1. harvest The system is now in a state where it can distinguish between different areas. This can be set by the user, but since the center of the field is often registered on the field map, the S3-2 route can be determined automatically. If the system receives a determination of S3-3, it is difficult for autonomous driving to proceed, and a decision must be made by the user. Conversely, by comparing the center position of the field map with registered data, it is possible to determine where the unworked area is in relation to the current position of the implement, and to immediately begin autonomous driving.

[0038] In the field corner reading S3-4, harvest The area calculation S3-7 is possible. The weight of the periphery cut S3-5, the moisture content of the periphery cut, and the harvest yield per unit area are calculated from the running performance of the combine as described above S3-8. harvestThe weight of the area can be calculated (S3-9), and post-harvest measures can be taken based on the estimated weight.

[0039] Not yet harvest There are some areas where the crops have fallen, and if this area is selected for harvesting in S3-10, the appropriate storage tank classification calculation S3-11 is performed as a corresponding operation. This operation involves harvesting the unharvested areas separately, but this becomes the basic data for calculating the travel route S3-11 based on the estimated volume of the fallen areas and the size of the storage tank.

[0040] On the other hand, this harvest There is a process to create field map data from the area. The field map basically contains the latitude and longitude of the outer perimeter and the center of the field. Field map data is constructed by adding data to this.

[0041] When the imaging of the perimeter mowing begins in S3-17, the imaging data is registered and associated with the field map according to the location where the image was taken. The data is compared with the satellite positioning system and registered as polygon data in the field data map as the first mapping.

[0042] Figure 15 shows harvest An example of a screen displaying the areas detected as lodging is shown in the figure. If you want to know the condition of the field, just touch the area you want to know about on the screen, and the data registered and associated with each area on the image will be read out. At the time the data was acquired, you can read out images of the crops, moisture, air temperature, soil temperature, weather, the degree of lodging as judged by the image, and the state of harvest of the rice ears. Also displayed as an overall image is the percentage of lodging in the entire field, the moisture distribution of the entire field, and moisture deviation.

[0043] The degree of lodging in this field can be determined using image processing, as described below. A simple classification divides the degree of lodging into three stages: light lodging, moderate lodging, and severe lodging. This is done by calculating the angle of the lodged stalks and the height of the rice ears from the field.

[0044] Returning to the explanation of Figure 3, the location and area of ​​the fallen area can be estimated by creating data in step S3-18 for detecting the fallen area. Although the area inside the plant may not be clear if only the outer perimeter is mowed, this can be estimated using the image processing method of the present invention, which will be described later, and the data is added to the field map data as the first map and growth registration S3-19.

[0045] Furthermore, the storage tank appropriate classification calculation S3-11 adds data to the second map, which shows the route that would be taken if harvested using normal procedures, and the yield registration S3-12. Data such as the estimated weight of the entire field, the degree of lodging, and the area and ratio of lodging are added to the field map.

[0046] Furthermore, the planned driving route for actual harvesting is registered as a third map in driving route registration S3-14 after calculating the driving route S3-13. The field map is further expanded with the information data from the first, second, and third maps to create a fourth map in classification registration S3-20, and this field map data is sent to the cloud 226 and provided to the user.

[0047] The user checks this field map data and confirms that there are no problems with the driving route or harvesting divisions, and then confirms the registered details in S3-15. Once each map has been corrected in S3-16, a field map is created and registered in S3-21, and the edge harvesting work is complete. From this point on, automated driving will begin based on the registered field map data. The registered field map overlays the harvested crop quality and field area, and remains as image data even after harvesting is complete, and can be used as basic data for the next season's cultivation. By doing this for each field, a user database will be built.

[0048] FIG. 4 is a block diagram showing the use of images from an imaging device.

[0049] The imaging device performs crop color analysis 50, shape analysis 60, and height (distance) analysis 70. However, because images contain a wide variety of information, much of the data is unnecessary. Masking is inevitably required, and each analysis calculation determines the amount of masking to remove non-standard data. In this invention, this corresponds to crop distance analysis 90. A color standard is also set for greens, and a distinction is made between crops and weeds based on saturation and density. Weeds are masked as noise. The same judgment is made for browns as for greens, determining damaged rice or rice malt, which are then subject to masking. Masking is based on length, as shown in 91 and 92.

[0050] The black line at position 93 is used to mask the image of the field. By masking the lower black line from the image, assuming that the image represents the field, it becomes possible to analyze the crop image more accurately. When determining whether the crop has fallen, there is also a method of removing noise by using magnification and a wide angle. In the present invention, the masking process is a method of not incorporating unknown information into the calculation, which will be explained in a later section.

[0051] Image data is difficult to handle, so image processing is performed to convert it into regions. These regions can be represented as a collection of small rectangles, and the whole is displayed as a polygon. The outer periphery is not a smooth line type, but the resolution is within a range that is acceptable for field analysis.

[0052] In the color analysis 50, green polygon areas are determined to identify stalks, and brown polygon areas are determined to identify ears of rice. The imaging data is an imaging system that can display green polygon areas in dots, and it is also possible to display linear color analysis in gradually different colors. The dot display of brown polygon areas has a similar function.

[0053] Even if the image is taken from the same position, the imaging device compares both the enlarged and wide-angle images to remove noise caused by lighting conditions and clearly capture the arrangement of greens and browns. This corresponds to the controls at 52 and 53 in Figure 4. A basic map of the fallen state is registered, and whether or not the plant has fallen is determined based on the degree of agreement with this map data.

[0054] Shape analysis 60 analyzes the shape of the crop. Color analysis 50 primarily analyzes the arrangement of brown and green tones, which allows for a general assessment of the lodging state of the crop, but also analyzes the relative position of the color in the field. Because the imaging device can measure distance, it is possible to determine the position of the rice ear and stalk. Figure 4 shows two imaging devices, 21 and 23 in Figure 1. The difference in these positions is used to determine the three-dimensional shape of the object. Rice Ear Position Confirmation 1 and Rice Ear Position Confirmation 2 confirm the shape of the rice ear and analyze its position from the field. Simultaneously, plant height 1 and plant height 2 determine the shape of the stalk and the shape of weeds, allowing the height of the stalk from the field to be measured. It is also possible to determine the direction of the crop from the position of the stalk and rice ear. Installing multiple imaging devices in this way allows the depth of the object to be determined from the difference in imaging angle, and also prevents noise due to lighting conditions, making the above measurements easier.

[0055] The field data 80 includes crop moisture at the time of harvesting, the air temperature around the field, and the soil temperature of the field, and by comprehensively analyzing the data from the color analysis 50 and shape analysis 60, it is possible to estimate and calculate the degree of lodging 95, lodging area 96, and grain quality 97.

[0056] The method for changing the state of the image for the collapse analysis in the captured image is explained in Figures 6 to 11, and the analysis method using artificial intelligence with a neural network, which is a method for changing the state of the input and output of each sensor to change the state, is explained in Figure 12.

[0057] We will now explain a system of the present invention that compares data on the arrangement of green and brown areas, determines the lodging state of crops, adds information to field map data, and maps the lodging state in unharvested areas.

[0058] When harvesting the outer periphery of a field, an image of the field is taken at predetermined intervals using an image capture device mounted on a work vehicle, and the image data is displayed as color-coded dots to determine the condition of the imaged items.

[0059] Figure 6 shows the state of fallen rice plants captured by the imaging device 21. Rice stalks 101, 102, and 103 are tilted horizontally relative to the ground, while rice ears 111, 112, and 113 are oriented vertically relative to the ground. Similarly, weeds 104 and 105 in the surrounding area often fall horizontally as they are pulled along by the fallen rice plants.

[0060] This image data was analyzed using the RGB color model and converted into a dot image, as shown in Figure 7. Since the drawing cannot be displayed in color, we will explain it in words: rice stalks 101A, 102A, and 103A are green dots and are tilted horizontally relative to the ground, while rice ears 111A, 112A, and 113A are brown dots and are oriented vertically relative to the ground. Weeds 104A and 105A are green dots and are tilted horizontally relative to the ground.

[0061] Furthermore, the wavelengths of light from the greens that can be identified as stems or weeds and the browns that can be identified as rice ears are quite different, making them easy to distinguish. They are also different from the black that is the ground in the field, and by analyzing several images of data, misrecognition and inability to analyze are greatly reduced.

[0062] Figure 8 shows the dot image data after averaging and linear processing. This processing makes the relationship between the stalks and the ears of rice clear. Rice stalks 101B, 102B, and 103B are green lines that are tilted horizontally relative to the ground, while ears of rice 111B, 112B, and 113B are brown lines that are oriented vertically relative to the ground. Weeds 104B and 105B are green lines that are tilted horizontally relative to the ground. From this line image, the intersection angles α1 and α2 of the green and brown lines are detected, and these are used to determine the lodging state of the crop.

[0063] 9 shows rice plants in an unlodged state captured by imaging device 21. Rice stalks 101C and 103C extend vertically relative to the ground, and rice ears 111C and 113C also extend vertically relative to the ground. Similarly, weeds 104C in the surrounding area also extend vertically.

[0064] This image data was analyzed using the RGB color model and converted into a dot image, as shown in Figure 10. Rice stalks 101D and 103D are green dots that extend vertically relative to the ground, while rice ears 111D and 113D are brown dots that also extend vertically relative to the ground. Weeds 104D are green dots that also extend vertically relative to the ground.

[0065] Figure 11 shows the dot image data averaged and linearly processed. This processing makes the relationship between the stalks and the ears of rice clear. Rice stalks 101E and 103E are green lines that extend vertically relative to the ground, while rice ears 111E and 113E are brown lines that extend vertically relative to the ground. Weeds 104E are green lines that extend vertically relative to the ground. From this linear image, the intersection angles α3 and α4 of the green and brown lines are detected, and these are used to determine the lodging state of the crop.

[0066] As explained in the overview of image state modification, it is possible to convert the arrangement data of green and brown regions into a line image, detect the intersection angle of the green and brown lines, and calculate the degree of lodging based on the intersection angle. In this way, by decomposing and analyzing the image data into individual elements and comparing the relationships between the analyzed information with existing data, it is possible to perform calculations to establish a different information judgment, thereby making it possible to make judgments that are closer to human experience. As an example of how the present invention determines the lodging state of crops, a judgment using artificial intelligence using a neural network, as shown in Figure 12, and a judgment to change the state of the image data from the imaging device is an effective strategy.

[0067] In Figure 12, we will explain the details of the second invention, in which the image captured by the imaging device of the present invention is calculated using a state change system to determine the arrangement of green and brown areas, and then compared with pre-registered data on the arrangement of green and brown areas to determine the lodging state of the crop.

[0068] In this example, three image capture devices are installed. Each image capture device has similar functions, but by utilizing the differences in their installation positions, the accuracy of the three-dimensional shape and color analysis of the image can be further improved, and the main analysis data can be differentiated. The noise removal method has differences in structure and data processing, which can further improve the accuracy of the image analysis.

[0069] The data from the first imaging device 21 and the obstacle sensors 31, 32, and 33 is mainly processed by the crop size detection unit 201. The data for each element from the imaging device and the data for each element from each obstacle sensor are combined and judged in a new way, and control is performed to change the state of the information to one that can determine the size of an object. As a first state change, the information content is changed to the height 203 of the entire crop, the width 204 of the entire crop, the height 205 of the stalk, and the position of the crop part (fruit), which in this example corresponds to the ear of rice, 206.

[0070] The second image capture device 23 mainly comprises a crop color detection unit 202. By combining and judging the elemental data of the image capture device, control is exercised to change the state of the information to determine the color of the object. As a first state change, the information content changes to a green direction 207, a brown direction 208, a masking amount determination 209, and a crop density 210.

[0071] Due to these state changes, the image state of FIG. 6 changes to that of FIG. 8, and that of FIG. 9 changes to that of FIG. 11, and the image state is changed by the decision unit 211.

[0072] As described above, the reference data to be compared is registered in data storage unit 216, and image matching is performed by comparing 211 and 216 and matching similar data, but the degree of similarity and whether it is within the range of image state change is calculated again by first state change units 201 and 202 to arrive at a determination of falling 202, a determination of a fallen area 213, a determination of the degree of falling 214, and an estimate of quality 215. This second state change is a combination of state change, matching, and judgment in which a new combination of element data is matched with the registered reference data, and is based on the so-called artificial intelligence function.

[0073] This function selects an ideal route in the work route change and determination unit 217, and the route is reflected in the automatic driving function by carrying out the work. However, for unmanned operation, the work route must be determined by checking it against various setting data.

[0074] The external data is how to revise the preset route setting 228. The field map data obtained as a result of reaping the outer periphery is also used as the environment map 227. This information is exchanged with the user via the communication unit 229 and the cloud 226. The third imaging device 24 also analyzes the state of the field in front of the work vehicle and the state of the reaping unit, and data from the satellite positioning device 22 is also added to determine the work route change and determination unit 217.

[0075] The artificial intelligence in this embodiment controls the traveling unit 218 and the working unit 221, and by determining the work route 217, the artificial intelligence determines the traveling speed 219 and the traveling direction 220 and starts traveling. In addition, by determining the working height 222, the inclination of the working unit 223, the rotation speed of the working unit 224, and the control of the harvesting unit 225 as a working machine, unmanned operation by the artificial intelligence becomes possible.

[0076] A third aspect of the invention, which is a method for creating image data of an area that cannot be imaged, beyond the imaging range of the imaging device, will now be described.

[0077] 13 shows the relationship between the harvest vehicle 250, the imaging range 302 of the imaging device, and the field 240. The harvest vehicle proceeds with circumferential mowing in a counterclockwise direction, moving in the following order: 250 ⇒ 251 ⇒ 252 ⇒ 253 ⇒ 254 ⇒ 255 ⇒ 256 ⇒ 257.

[0078] The white arc indicates the range that the imaging device can reliably capture. While it is possible to capture much of the travel route by increasing the number of captures, this invention utilizes a method of stretching and utilizing existing captured data. For example, after capturing the image of area 301, which is the forward area that has been captured, a gradation of the image of the interior of 301 as seen from above is created. Using this arrangement, the data for 301A, 301B, and 301C are expanded at the same ratio to create area 302. As the harvesting vehicle travels from 251 to 252, it can capture actual images of area 302, allowing the actual captured image to be compared with the image of 302. This data comparison calculates the difference. If there is a difference in the ratio of the degree of lodging, the ratio is calculated. The moving average of these ratios is calculated to derive a coefficient for correcting the estimated image, such as 302. The neural network function shown in Figure 12 also handles this correction coefficient calculation. The image state change and determination unit 211 corresponds to the estimated image 302, and the data storage unit 216 corresponds to the actually captured image. Using this second state change calculation, the degree of falling 212, the fallen area 213, the degree of falling 214, and the quality estimation 215 are determined, and at the same time, the image video from above 302 is corrected.

[0079] This image is corrected to create an image of the uncaptured area, and the uncaptured area is then imaged. harvest 13, a correction coefficient is acquired from the image data of 302, and an image 304 is created using the image gradation of the sky 303A, 303B, and 303C in the image data of the area 303. The correction coefficient is used to correct not only the image but also the data for determining the degree of falling 212, determining the area of ​​falling 213, determining the degree of falling 214, and estimating the quality 215.

[0080] This function creates estimated image data of unimaged areas from image data captured based on this correction coefficient, and accuracy can be improved by sequentially overlaying the estimated data. It also has a learning function that repeatedly calculates, corrects, and registers the contents of changes made one after another, and weights the contents of the image changes and changes to the fallen data, etc., with more weighting applied to these changes.

[0081] By using the first and second state changes shown in Figure 12, the state of crops in the field can be determined by comparing, collating, removing, analyzing, and changing the data stored in the imaging device, obstacle sensor, and data storage unit 216, which was previously determined by human vision and experience.

[0082] Further, as an expanded function, the traveling speed 219 and traveling direction 220 of the traveling unit 218 are controlled. The working unit control 221 controls the working unit height 222, the working unit inclination 223, the working unit rotation speed 224, and the part that is the output of the harvesting unit control 225.

[0083] In conventional technology, for example, the height of the crop is determined from distance information and image data from each imaging device and information from obstacle sensors, and the control directly outputs traveling unit control 218 and working unit control 221. However, in the present invention, this is done by performing a first state change and a second state change before outputting traveling unit control 218 and working unit control 221, which not only improves control accuracy but also makes it possible to change the work path while working by obtaining detailed information about the state of the crop, and the output control is changed based on this change.

[0084] Therefore, the first state change and the second state change can only be analyzed by judgment using artificial intelligence, and the traveling unit control 218 and working unit control 221 are also considered to be unique controls that make use of judgment control using artificial intelligence that utilizes a neural network.

[0085] The fifth invention explains the control leading up to the aforementioned output, in which the size of the crop is detected from the imaging device 21, the color of the crop is detected from the imaging device 23, the RGB element analysis of the image is performed, and based on the arrangement of green and brown in the image, the lodging area of ​​the crop is determined 213, the degree of lodging is determined 214, and the quality is estimated 215, and comparison calculations are made with the data registration unit 216 to control the traveling unit 218 and the working unit 221.

[0086] Regarding the fourth invention, which involves overlaying an image on previous data, weighting the dot data of the overlapping areas to the side with higher image accuracy, and synthesizing them to estimate the degree of lodging, weight per unit area, and moisture content of each unharvested area, in addition to the explanation in the previous section, we will explain the correspondence of areas where image data overlap in Figure 13.

[0087] In Figure 13, this corresponds to the area 314. Although the image was taken from different directions, it is the same area, and both captured images can be referenced. However, the polygonal image viewed from above must be synthesized. In this case, both images are added together and synthesis is performed with weighting applied to the image with higher image accuracy. In Figure 13, the image 314 is an image of the work vehicle 257 in front of the imaging device, and because it is more accurate, synthesis processing is performed with weighting applied to the image in front of the work vehicle 257.

[0088] The following describes how to handle past data when the image capturing range of the imaging device is exceeded.

[0089] The actual imaged area is used to create estimated image data. However, when the area far exceeds the range that can be estimated, reliability becomes low, so the approach shown in Figure 14 is taken. The method used is to reuse past data. Last year's field data is registered in the data storage unit, and past image data is used to provisionally display the unimaged areas. This past image data includes data for estimating the degree of lodging, weight per unit area, and moisture content.

[0090] By touching an unphotographed area on the image, the user can select between estimated image data or past image data. Based on the selected image data, the system has a function to display the degree of lodging, weight per unit area, and moisture content. Based on this estimated data, the user can change the route or adjust the harvest classification.

[0091] If no past image data exists, the data is displayed as blank, or data close to the relevant area is used while displaying that the data accuracy is low.

[0092] The image data from these imaging devices cannot capture the entire field in one image; the image data must be combined. Figure 5 is a block diagram showing a standard image combination. Fields are often rectangular, with four sides and a square. Therefore, by using the data for the square and the center data for the four sides, it is possible to capture a minimum amount of field data. A simple technique for combining these images is to change the color to that seen from above, and by combining the images while performing RGB color analysis between the square and the four sides in order, it is possible to create an aerial image of the field using imaging devices installed on a work machine on the ground.

[0093] Figure 5 outlines the procedure for combining the images and the final process for creating a map of the lodging area and degree of lodging in the field. [Explanation of symbols]

[0094] 1. Combine 21 First imaging device 22 Inertial Measurement Unit (IMU) 23 Second imaging device 24 Third imaging device 31, 32, 33 Obstacle sensors 41 Satellite positioning system receiver (GNSS receiver) S3-12 Second map (yield) S3-14 Third Map (Driving Route) S3-19 First Map (Growth) S3-20 4th Map (Division) 201 Crop size detection unit 202 Crop color detection unit 211 Image status change and determination unit

Claims

1. A harvesting system equipped with a system that photographs a field with an imaging device equipped on a work vehicle, calculates a sequence of color areas indicating the state of the crop using a status change system, compares the image captured by the imaging device with pre-registered sequence data, determines the lodging state of the crop, and maps the lodging state within unharvested areas.

2. The harvesting system of claim 1 measures the weight and moisture of the harvested crops when the work vehicle harvests, determines the position where the work vehicle changes direction to be a corner of the field, and adds estimated crop weight and moisture data, which is consistent with the lodging state of the crops in the unharvested area, to the field map data registered in advance, and uses this data for automatic driving.

3. An RGB image gradation is created from an image of an area captured by an imaging device equipped on a work vehicle when harvesting a portion of a crop in a field by the work vehicle; This RGB image gradation is used to create an estimated image of the unimaged area at the same ratio, and actual captured images are created as the work vehicle progresses, and the estimated image and the actually captured image are compared to calculate a correction coefficient for correcting the difference between the RGB images.

3. A harvesting system according to claim 1 or claim 2, further comprising a function for generating estimated image data of unimaged areas from image data captured by an imaging device after the next harvesting operation, based on the correction coefficient.

4. A harvesting system as described in claim 3, in which, using actual image data and estimated image data added by an imaging device as the work vehicle progresses, the image is overlaid on the field map data from the previous time within the status change system, and the dot data of the overlaid area is weighted and combined with the actual image data to estimate the degree of lodging, weight per unit area, and moisture value of the unharvested areas.

5. A harvesting system according to claim 1 or claim 2, in which an imaging device equipped on a work vehicle performs RGB image analysis of the image, and determines the area of ​​lodging, degree of lodging or quality of the crop based on the arrangement of green and brown colors of the crop in the image, and controls the traveling or working parts of the work vehicle when it is automatically driven by comparing and calculating this with the field map data.

Images