Harvester and its automatic operation method
The harvester's pan-head imaging device with anti-shake technology and image segmentation algorithms addresses the issue of inaccurate path setting and high costs by stabilizing images and recognizing farmland areas, enabling efficient autonomous operation with general satellite positioning.
Patent Information
- Application Number
- JP2021538493
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Priority Date
- 2018-12-29
- Filing Date
- 2019-09-24
- Publication Date
- 2024-11-20
- Estimated Expiration
- 2039-09-24
AI Technical Summary
Existing harvesters face issues with inaccurate path setting due to vibration and uneven land, leading to blurry images from fixed cameras, which hinder autonomous operation and increase machine breakdowns, and require high-precision satellite positioning, increasing costs.
A harvester equipped with a pan-head imaging device that stabilizes images through anti-shake technology, allowing it to capture images from multiple angles and directions, combined with image segmentation and deep learning algorithms to recognize farmland areas and crop information, enabling autonomous operation without high-precision satellite positioning.
The solution enhances image stability and accuracy, allowing for precise route planning and parameter adjustment, reducing machine breakdowns and costs by utilizing general satellite positioning, thus improving operational efficiency and reducing maintenance.
Smart Images

Figure 0007587503000001 
Figure 0007587503000002 
Figure 0007587503000003
Abstract
Description
[Technical field]
[0001] The present invention relates to the field of automatic operation of agricultural machines, and in particular to a harvester and an automatic operation method thereof. [Background technology]
[0002] Harvesters are crop harvesting machines that harvest the berries and straw of crops such as rice and wheat, as well as machines and equipment for harvesting other crops such as vegetables and fruits. Grain harvesters are integrated machines that can cut and thresh crops in one operation, collect the grains in a storage room, and then transport them to a carrier vehicle via a conveyor. Fruit and vegetable harvesting machines can cut and harvest vegetables and fruits in a field, separate the fruit from the stems, and then sort them in one operation.
[0003] When performing harvesting work, the harvester needs to constantly observe the crops in the field and the field work area so as to adjust the work parameters of the harvester according to the height, maturity, fullness, etc. of the crops. When the agricultural machinery works in the field, it needs to judge the work status of the field and the growth status of the crops in the field in real time, operate the operation of the agricultural machinery, and adjust the complex operation status of the work system. Currently, only machine operators with specialized skills can do this. When the agricultural machinery works in the field, it needs to take into account many factors such as the worked area, the unworked area, and the boundary range of the field, and adjust the operation of the vehicle in real time according to the status of the crops during work and adjust the work parameters. Since it is necessary to take into account the complex work environment during operation, the agricultural machinery of the prior art needs the operator to adjust the operation of the agricultural machinery according to real-time information on the crops on the field. When the work of the agricultural machinery is controlled by manual operation, there is a high probability of judgment errors occurring and a high probability of the machinery breaking down during work.
[0004] The harvester of the prior art has at least one of the following defects. First, during operation, the harvester body shakes up and down due to the vibration of the harvester body itself and the unevenness of the farmland, so that the camera device installed on the harvester body cannot capture an image in a stable position. Therefore, the image captured by the imaging device is blurred and cannot provide information support for intelligent work and automatic driving. In addition, the imaging device of the prior art is fixedly attached to the harvester body, and can only capture images in a single direction, for example, an image in front of the harvester, and cannot adjust the shooting direction and position of the imaging device according to the situation. In addition, the mobile imaging device or fixed imaging device of the prior art, for example, an imaging device of a drone or an imaging device fixed to farmland, captures images around the harvester and then transmits them to the harvester body, and the harvester body reads the images captured by the imaging device. Although the problem of poor image capture has been resolved to some extent, the image is captured based on the position of the image capture device itself or the drone, and the image cannot be captured from the perspective of the harvester itself, so the captured image is not well recognized.
[0005] In the agricultural machinery of the prior art, the inaccuracy of the set working path usually causes errors in the work, which leads to machine failure. In addition, the satellite positioning using PTK has high performance requirements for the agricultural machinery, and the manufacturing and maintenance costs are relatively high, so such a conventional automatic driving positioning method is not applicable to the automatic driving mode of the conventional agricultural machinery. <Prior art documents> <Patent Documents> <Patent Document 1> JP 2018-101860 A <Patent Document 2> JP 2014-171457 A Summary of the Invention [Problem to be solved by the invention]
[0006] One main advantage of the present invention is to provide a harvester and an automated driving method thereof, the harvester recognizing farmland areas in at least one captured visual image based on said image.
[0007] Another advantage of the present invention is to provide a harvester and an automated method thereof, wherein the harvester recognizes crop information, such as type, height, maturity, etc., in at least one captured visual image based on the captured visual image.
[0008] Another advantage of the present invention is to provide a harvester and an automatic driving method thereof, wherein the harvester recognizes unworked areas, worked areas, and farmland boundary areas in the visual image based on the visual image, and controls the driving path of the harvester based on the recognized areas.
[0009] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, wherein the harvester recognizes crop information in the image based on the visual image, and adjusts operation parameters of the harvester based on the recognized information in the image, thereby improving the operation quality and efficiency of the harvester.
[0010] Another advantage of the present invention is to provide a harvester and an automatic driving method thereof, in which an image capture device of the harvester is a pan-head imaging device, and the pan-head imaging device has an anti-shake shooting function, which improves the accuracy and stability of the harvester in capturing visual images.
[0011] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, wherein the image capture device is provided on a harvester body of the harvester, and the harvester captures images of the surroundings of the harvester body using the image capture device.
[0012] Another advantage of the present invention is to provide a harvester and an automatic driving method thereof, wherein the image capture device is provided on a harvester body of the harvester, and the image capture device is provided on the harvester body and captures at least one visual image or visual video based on a field of view position of the harvester body, thereby recognizing information about the surroundings of the harvester body based on the captured video information.
[0013] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, wherein the image capture device captures images of the harvester body in different directions by taking images at different angles and in different directions while being adjusted based on the position of the harvester body.
[0014] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, wherein the image capture device is a mechanical or electronic pan head camera, and the image capture device improves the stability of the visual image.
[0015] Another advantage of the present invention is to provide a harvester and an automatic driving method thereof, in which a route planning system of the harvester automatically plans a route based on current vehicle positioning information, information recognized by an image processing system, and information from a navigation system.
[0016] Another advantage of the present invention is to provide a harvester and method for automatic operation thereof, wherein the harvester plans a travel path and a work path for the harvester based on the area recognized by the visual image.
[0017] Another advantage of the present invention is to provide a harvester and an automated driving method thereof, wherein an image capture device of the harvester captures the visual images of surrounding farmland in real time and updates route navigation information of the harvester plan in real time.
[0018] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, in which the harvester captures an image in real time using the image capture device, recognizes the area in the visual image, and updates or adjusts the working path of the harvester in real time according to changes in the area, thereby improving the working quality of the harvester.
[0019] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, wherein the image processing system of the harvester recognizes the unworked area, the worked area, and the farmland boundary area in an image using an image segmentation technique based on the acquired visual image information, and segments the boundary between two adjacent areas.
[0020] Another advantage of the present invention is to provide a harvester and an automatic operation method thereof, wherein the image processing system of the harvester uses image segmentation technology to recognize plant information such as type, height, and fruit fullness of the crop in the image based on the acquired visual image information, and the operation system of the harvester adjusts operation parameters based on the crop information.
[0021] Another advantage of the present invention is to provide a harvester and an automatic driving method thereof, wherein the image processing system of the harvester recognizes area boundaries in the image based on acquired image information, and the route planning system plans a vehicle driving route based on the recognized area boundaries.
[0022] Another advantage of the present invention is that it provides a harvester and an automatic driving method thereof, which does not require high-precision satellite positioning for the harvester, reducing the difficulty of producing and manufacturing the automatic driving machine and reducing the maintenance costs of the machine.
[0023] Another advantage of the present invention is to provide a harvester and an automatic driving method thereof, wherein the harvester realizes automatic driving and automatic driving operations by performing route planning based on area segmentation information output by the image processing system.
[0024] Other advantages and features of the invention will be set forth fully in the following detailed description and be realized by means of the instrumentalities and combinations particularly pointed out in the appended claims. [Means for solving the problem]
[0025] According to one aspect of the present invention, the above objects and other objects and advantages are attained by a harvester of the present invention, comprising: A harvester body; At least one image capture device provided on the harvester body for capturing images of the surroundings of the harvester body; an image processing system that recognizes farmland information in the video based on the image captured by the image capture device; The harvester body automatically controls its operation based on the farmland information recognized by the image processing system.
[0026] According to one embodiment of the present invention, the harvester further includes a route planning system that plans at least one driving plan route based on the farmland information recognized by the image processing system, and the harvester body controls operation according to the driving plan route planned by the route planning system.
[0027] According to one embodiment of the present invention, the image processing system recognizes farmland information in the image using an image segmentation recognition technique, and plans the farmland area in the image based on the recognized information.
[0028] According to one embodiment of the present invention, the image processing system uses image segmentation recognition technology to recognize crop information in the image, and the harvester body automatically adjusts operation parameters based on the recognized information.
[0029] According to one embodiment of the present invention, the image capture device is a shake-proof pan-head imaging device, which is mounted on the harvester body and takes pictures of the surroundings of the harvester body based on the position of the harvester body.
[0030] According to one embodiment of the present invention, the image capture device is a mechanical anti-shake head device, comprising a head and at least one video camera, the video camera being attached to the harvester body by the head, and the video camera being mounted on the head and supported via the head so as to maintain a balanced state.
[0031] According to one embodiment of the present invention, the image capture device is an electronic camera head device, which controls the viewpoint and zoom of the lens, thereby preventing the lens of the image capture device from shaking during shooting.
[0032] According to one embodiment of the present invention, the image capture device is provided at the front of the harvester body, at the top of the harvester body, on the left side, the right side or at the rear of the harvester body.
[0033] According to one embodiment of the present invention, the image processing system comprises: an image segmentation module for segmenting the image into a plurality of pixel regions, each pixel region including at least one pixel unit; a characterization module for extracting features corresponding to each of the pixel regions based on the pixel units of the pixel regions; and a region partitioning module for recognizing and partitioning regions of the image based on characteristics of the pixel regions.
[0034] According to one embodiment of the present invention, the harvester further includes a positioning device and a navigation system, the positioning device and the navigation system are provided in the harvester body, the positioning device acquires position information of the harvester body, and the navigation system provides navigation information to the grain processing body.
[0035] According to one embodiment of the present invention, the route planning system comprises: a working area setting module for setting a boundary area of the farmland to obtain a working area and the working boundary of the farmland; and a driving route planning module, wherein the image processing system recognizes area planning information of the image and navigation information of the navigation system based on positioning information of the harvester body, to obtain at least one driving planned route.
[0036] According to one embodiment of the present invention, the harvester body includes a vehicle body, at least one work system provided on the vehicle body, and an operation control system, the vehicle body drives and operates the work system, and the operation control system controls the operation of the vehicle body and the work parameters of the work system.
[0037] According to one embodiment of the present invention, the driving control system acquires information about images recognized by the image processing system and captured by the image acquisition device, automatically controls the travel path of the vehicle body, and controls the work parameters of the work system, thereby realizing unmanned automatic driving and mowing work.
[0038] According to another aspect of the present invention, there is further provided an automatic operation method for a harvester, the automatic operation method comprising: (a) acquiring at least one image and recognizing farmland areas and farmland boundaries in said image; (b) planning at least one travel route based on the recognition information; and (c) controlling the harvester body to automatically travel along the travel planned route.
[0039] According to one embodiment of the present invention, step (a) of the above-mentioned autonomous driving method further includes a step of recognizing information corresponding to crops in the farmland in the image, and the crop information includes information such as crop type, crop height, and fruit fullness.
[0040] According to one embodiment of the present invention, the autonomous driving method step (b) comprises: (b.1) recognizing and demarcating areas and boundaries of said image corresponding to agricultural land; and (b.2) planning at least one driving plan route based on the recognized area.
[0041] According to one embodiment of the present invention, step (b.1) of the autonomous driving method further includes a step of segmenting the image using an image segmentation technique and recognizing and partitioning regions of the image.
[0042] According to one embodiment of the present invention, in step (b) of the above-mentioned autonomous driving method, the image processing system uses image segmentation technology to segment the image information and recognize and divide the areas of the image into the unworked area, the worked area, and the farmland boundary area.
[0043] According to one embodiment of the present invention, step (b.1) of the autonomous driving method comprises: Dividing the image into a plurality of said pixel regions and normalizing pixel values of said pixel regions into an array; extracting features of the pixel regions corresponding to each of the arrays; and outputting a classification label for the image based on features corresponding to the pixel regions.
[0044] According to one embodiment of the present invention, step (b.2) of the above-mentioned automatic driving method further includes a step of planning the driving plan route based on positioning information of the harvester body, area planning information of the image, and navigation information of the navigation system.
[0045] According to one embodiment of the present invention, the above-mentioned autonomous driving method further includes a step (b.3) of comparing whether the area division and area boundary range recognized by the image processing system match with the previous area boundary range, and if they do not match, adjusting the area division and area boundary range corresponding to the image, and if they match, keeping the area division and boundary range unchanged.
[0046] According to one embodiment of the present invention, the automatic driving method further includes a step (d) of adjusting working parameters of a working system of the harvester body based on the image recognition information.
[0047] Further objects and advantages of the present invention will become more fully apparent through a study of the following description and drawings.
[0048] These and other objects, features and advantages of the present invention will become more fully apparent through the following detailed description, drawings and claims. [Brief description of the drawings]
[0049] [Figure 1] FIG. 1 is a schematic diagram of a harvester system according to a first preferred embodiment of the present invention; [Diagram 2] FIG. 2 is a schematic diagram of the harvester acquiring images according to the preferred embodiment of the present invention; [Figure 3A] FIG. 2 is a schematic diagram of one image acquired by the harvester according to the preferred embodiment of the present invention; [Figure 3B] FIG. 13 is a schematic diagram of another image captured by the harvester according to the preferred embodiment of the present invention. [Figure 3C] FIG. 13 is a schematic diagram of another image captured by the harvester according to the preferred embodiment of the present invention. [Figure 4] FIG. 2 is a schematic diagram showing how the image processing system of the harvester according to the preferred embodiment of the present invention divides and recognizes the image area. [Figure 5A]FIG. 2 is a schematic diagram of the image processing system of the harvester according to the preferred embodiment of the present invention dividing the image region; [Figure 5B] FIG. 2 is a block diagram of the image processing system of the harvester according to the preferred embodiment of the present invention. [Figure 6] FIG. 2 is a schematic diagram of the image processing system of the harvester according to the preferred embodiment of the present invention extracting and recognizing the image region features. [Figure 7] FIG. 4 is a schematic diagram of region segmentation of the image output by the image processing system of the harvester according to the preferred embodiment of the present invention. [Figure 8] FIG. 13 is a schematic diagram of the division change of boundary lines of the region division of the image output by the image processing system of the harvester according to the preferred embodiment of the present invention. [Figure 9] FIG. 2 is a schematic diagram of an automatic operation scene of the harvester according to the preferred embodiment of the present invention. [Figure 10] FIG. 2 is a schematic diagram of the harvester path planning system according to the preferred embodiment of the present invention; [Figure 11A] FIG. 2 is a schematic diagram of a farmland path plan generated by the path planning system of the harvester according to the preferred embodiment of the present invention; [Figure 11B] FIG. 2 is a schematic diagram of the route planning system of the harvester adjusting a travel route according to the preferred embodiment of the present invention; [Figure 12] FIG. 2 is a schematic diagram of the overall structure of a harvester with an image capture device according to a second preferred embodiment of the present invention; [Figure 13] FIG. 2 is a schematic diagram of the image capture device of the harvester according to the preferred embodiment of the present invention taking images. [Figure 14] FIG. 2 is a structural schematic diagram of the image capture device of the harvester according to the preferred embodiment of the present invention, the image capture device being implemented as a mechanical pan head device. [Figure 15] FIG. 2 is a schematic diagram of the mounting position of the image capture device of the harvester according to the preferred embodiment of the present invention. [Figure 16A] FIG. 2 is a schematic diagram of the image processing system of the harvester according to the preferred embodiment of the present invention recognizing farmland areas in images captured by the image capture device; [Figure 16B] FIG. 2 is a schematic diagram of the image processing system of the harvester according to the preferred embodiment of the present invention recognizing agricultural crops in images captured by the image capture device; [Figure 17] FIG. 2 is a schematic diagram of another optional embodiment of the image capture device of the harvester according to the preferred embodiment of the present invention, wherein the image capture device is implemented as an electronic camera head device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0050] The following description is used to disclose the present invention so that those skilled in the art can practice the present invention. The preferred embodiments in the following description are merely examples, and other obvious variations may occur to those skilled in the art. The basic principles of the present invention defined in the following description can be applied to other embodiments, modifications, improvements, equivalents, and other technical solutions that do not deviate from the spirit and scope of the present invention.
[0051] Those skilled in the art will understand that in the description of the present invention, the orientations or positional relationships indicated by terms such as "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inside," "outside," and the like are based on the orientations or positional relationships shown in the drawings, and are merely for the purpose of simplifying the explanation and description of the present invention, and are not intended to indicate or suggest that the devices or elements shown have a specific orientation or are constructed or operated in a specific orientation, and therefore the above terms should not be understood as limitations on the present invention.
[0052] The term "a" should be understood as "at least one" or "one or more." That is, in one embodiment, the number of elements may be one, and in another embodiment, the number of elements may be multiple, and the term "a" should not be understood as a numerical limitation.
[0053] With reference to Figures 1-9 of the accompanying drawings of the specification of the present invention, a harvester and its automatic operation method according to a first preferred embodiment of the present invention are disclosed and described in the following description, and the harvester may be implemented as a crop harvester machine with grain processing function, a vegetable, fruit harvesting machine, a grass cutting machine, and other types of cutting devices, etc. It can be understood that the types of harvesters described in the present invention are merely exemplary and not limiting. The harvester obtains at least one image of the surroundings, visually recognizes and processes farmland area types in the image, and segments the types and boundaries of various areas of the farmland in the image.
[0054] The area types of the farmland divided by the harvester include at least one worked area 100, at least one unworked area 200, and at least one farmland boundary area 300, and based on the area types divided by the harvester, a navigation system plans a driving route for the vehicle to realize unmanned automatic driving and unmanned automatic driving operations.
[0055] In addition, in order for an autonomous vehicle to recognize and acquire a vehicle driving route in autonomous driving mode, it is necessary to acquire accurate vehicle positioning information, which usually requires highly accurate satellite positioning information. In order for the autonomous vehicle to realize an autonomous driving function in a high-speed driving state, it is necessary to constantly update information such as obstacle information on the road, road vehicle information, and road pedestrians. The image acquired by the harvester of the present invention is image data information corresponding to agricultural crop grains in farmland, and the image is an image of the surroundings of the vehicle acquired based on the current position of the vehicle. The harvester does not require highly accurate satellite positioning information, and only requires general satellite positioning with meter-level accuracy (such as GPS positioning or Beidou positioning). Therefore, the image acquired and processed by the harvester is different from that of an autonomous vehicle, and the route plan and driving mode formed by the harvester are also not the same. It can be understood that the function of the harvester of the present invention to visually recognize the area of the farmland and the autonomous driving function are different from the recognition mode of an autonomous vehicle.
[0056] As shown in Figures 1 and 2, the harvester acquires at least one image of the surroundings, and based on the acquired image, recognizes and divides the area type and the boundary between the areas corresponding to the farmland. The harvester acquires the image of the surroundings of the harvester by taking fixed point photos, video shooting, taking photos on the move, etc. It can be understood that the manner in which the harvester acquires the image is merely exemplary and not limiting. The harvester includes a harvester body 10 and at least one image acquisition device 20, and the image acquisition device 20 acquires at least one image of the surroundings of the harvester body 10.
[0057] Preferably, the image acquisition device 20 is provided on the harvester body 10 and acquires the image of the periphery of the harvester body 10 by taking a photograph or a video. More preferably, the image acquisition device 20 is provided in front of the harvester body 10 and can acquire an image in front of the harvester body 10 in real time, and the harvester body 10 sets a travel route based on an area recognized and divided from image information captured by the image acquisition device 20. Note that the image captured by the image acquisition device 20 is an image within the field of view of the harvester body 10. That is, the image acquisition device 20 acquires an image in the field of view of the harvester body 10 and adjusts the travel direction of the harvester body 10 according to the mounting position of the harvester body 10.
[0058] Also, the image capture device 20 captures a view of the top and bottom in the direction of travel of the harvester body 10, and the images may be captured two-dimensional planar images or three-dimensional stereoscopic images. It can be understood that the types of images captured by the image capture device 20 are exemplary only and not limiting.
[0059] In the first preferred embodiment of the present invention, the harvester body 10 is implemented as a grain harvester machine, and while being controlled, travels to an unworked area 200 of farmland, performs harvesting work, and harvests crops such as rice, wheat, and corn in the unworked area 200. The harvester body 10 automatically drives in the field based on an area segmented from an image acquired by the image acquisition device 20, and performs unmanned automatic driving. It can be understood that the type of the harvester body 10 is merely illustrative and not limiting.
[0060] As shown in FIG. 3A to FIG. 3C, the image acquisition device 20 acquires images of the surroundings of the harvester body 10 in real time while the harvester body 10 is traveling. FIG. 3A shows the image captured by the image acquisition device 20 when the harvester body 10 is a grain harvester. Depending on whether the grain has been harvested or not, the area in the farmland is divided into at least one uncut area 100a, at least one cut area 200a, and at least one farmland boundary area 300a, and the cut area 200a is an area where the crops have been cut, and the original crops in the cut area 200a have been cut. The uncut area 100a is an area where the crops are alive, and the crops are growing in the uncut area 100a. The farmland boundary area 300a is the ridges that separate the crops within the farmland, the outer boundary around the farmland, and obstacle areas that exist within the farmland, and no crops are planted in the farmland boundary area 300a.
[0061] 3B shows the image captured by the image capture device 20 when the harvester body 10 is a grass cutting machine. Depending on whether the crop is planted or not, the area in the farmland is divided into at least one uncut area 100b, at least one cut area 200b, and at least one farmland boundary area 300b, where the uncut area 100b represents an area where the crop is not cut, the cut area 200b represents an area where the crop is planted, and the farmland boundary 300b represents the ridges separating the planted crops, the outer boundary around the farmland, and the obstacle area present in the farmland.
[0062] 3C shows the image captured by the image capture device 20 when the harvester body 10 is a rootstock or fruit harvesting machine, such as a vegetable harvester machine. Depending on whether the crops have been sprayed with a chemical solution or not, the area in the farmland is divided into at least one unharvested area 100c, at least one harvested area 200c, and at least one farmland boundary area 300c. The unharvested area 100c represents an area where the crop has not been harvested, the harvested area 200c represents an area where the crop has been harvested, and the farmland boundary 300b is the ridges separating the planted crops, the outer boundary around the farmland, and the obstacle area present in the farmland.
[0063] 1 and 4, the unworked area 100, the worked area 200, and the farmland boundary area 300 are recognized from the image acquired by the image acquisition device 20 using image segmentation recognition technology, and boundary lines between the areas are distinguished. The harvester further includes an image processing system 30, which recognizes the unworked area 100, the worked area 200, and the farmland boundary area 300 from the image based on the image of the farmland acquired by the image acquisition device 20 using image segmentation recognition technology.
[0064] It can be understood that the image processing system 30 uses an image segmentation recognition technique to recognize the area and boundary in the image as the area and boundary of the farmland ahead of the travel of the harvester body 10. The harvester body 10 travels and works in an unworked area in the farmland while being controlled based on the area and boundary recognized by the image processing system 30 using the image segmentation recognition technique. For example, an image of the farmland ahead of the harvester machine is acquired by the image acquisition device 20 provided at the front end of the harvester machine, and the image taken by the image acquisition device 20 is segmented and recognized by the image processing system 30, and the unworked area 100, the worked area 200, and the farmland boundary area 300 in the travel direction of the harvester machine are recognized and segmented. The harvester body 10, i.e., the host computer of the harvester machine, plans a vehicle travel path and a mowing operation based on the area and boundary recognized by the image processing system 30.
[0065] It can be understood that the image processing system 30 uses image segmentation recognition technology to recognize information such as the type of crop, the height of the crop, and the fullness of the fruit of the crop in the image provided by the image acquisition device 20. The image processing system 30 can determine whether the crop has been cut based on the type of crop and the height of the crop recognized in the image, and adjust operation parameters based on the fullness of the fruit of the crop recognized in the image. In other words, the image processing system 30 can recognize the area type and boundary of the farmland based on the image provided by the image acquisition device 20, and can also recognize the type, height, fullness of the fruit, and maturity of the crop in the farmland.
[0066] The image processing system 30 selects one of the segmentation recognition methods, such as a threshold-based segmentation method, an area-based segmentation method, an edge-based segmentation method, and a segmentation method based on a specific theory, to segment and recognize the image acquired by the image acquisition device 20, thereby recognizing the area and boundary in the image. Preferably, the image processing system 30 uses a deep learning algorithm to segment and recognize the image, and performs area segmentation and boundary definition for the image. That is, the image processing system 30 uses a deep learning algorithm to recognize the area and boundary of the corresponding farmland in the image, and the harvester body travels and works according to the recognized and segmented area and boundary. More preferably, the image processing system 30 uses an image segmentation recognition technique of a convolutional neural network algorithm in the deep learning algorithm to recognize the unworked area 100, the worked area 200, and the farmland boundary area 300 in the corresponding farmland from the image.
[0067] The processing algorithm used by the image processing system 30 is merely illustrative and is not limiting. Therefore, the image processing system 30 may further use other algorithms to segment and recognize the acquired image, thereby recognizing the farmland area and boundaries in the image.
[0068] As shown in FIG. 5A and FIG. 6, the image processing system 30 divides the image acquired by the image acquisition device 20 into a plurality of pixel regions 301, and each of the pixel regions 301 includes at least one pixel unit. According to the image corresponding to the area around the harvester body 10, the pixel regions 301 of the image correspond to image information of a particular area of farmland or crops in the photographed farmland. It can be understood that a normalization process is performed on each of the divided pixel regions 301 to normalize the pixel units of the pixel regions 301 to a numerical value or an array corresponding to a pixel value. That is, the image processing system 30 normalizes the divided pixel regions 301 to a corresponding numerical value or an array in order to extract image features and divide the area.
[0069] The image processing system 30 extracts image features corresponding to the pixel regions 301 based on the arrays corresponding to each of the pixel regions 301. The image processing system 30 obtains image features corresponding to the pixel regions 301 based on the arrays corresponding to the pixel regions 301. When the image processing system 30 uses a convolutional neural network algorithm such as a two-dimensional convolutional neural network, the image processing system 30 inputs a two-dimensional array or a three-dimensional array corresponding to the pixel regions 301 to the input layer of the convolutional neural network. The hidden layer of the convolutional neural network performs feature extraction on the array of the input layer, and performs feature selection and information filtering after feature extraction. The convolutional neural network outputs classification labels of the pixel regions 301 corresponding to the unworked area 100, the worked area 200, and the farmland boundary area 300, respectively, based on the features corresponding to the arrays.
[0070] 6 and 7, the image processing system 30 recognizes region features corresponding to the pixel region 301 by extracting features of an array corresponding to the pixel region 301. The features corresponding to the pixel region 301 mainly include crop stalk height features, spacing of crop stalks in a farmland, crop color, farmland land color, crop type features, farmland land features, crop fruit fullness, number of crop fruits, etc. The image processing system 30 outputs a classification label corresponding to the pixel region 301 based on the extracted features, and the classification label marks a region type and a boundary line corresponding to the pixel region 301 based on the feature information.
[0071] As shown in FIG. 5B, the image processing system 30 includes an image segmentation module 31, a characterization module 32, and an area partitioning module 33. The image segmentation module 31 acquires an image captured by the image acquisition module 20 and a plurality of pixel areas 301 formed by segmenting the image, each of which includes at least one pixel unit. The characterization module 32 uses a deep learning algorithm to extract feature types corresponding to the pixel areas 301, select features, and filter information. The area partitioning module 33 partitions the image based on the features corresponding to the pixel areas 301 extracted by the characterization module 32, thereby generating classification labels corresponding to the unworked area 100, the worked area 200, and the farmland boundary area 300.
[0072] Preferably, the image segmentation module 31 segments the image into a plurality of pixel regions 301, each of which has the same size, shape and extent. It can be understood that the image segmentation module 31 may further segment based on the size of the image pixel threshold, that is, the pixel regions 301 segmented by the image segmentation module 31 may have different sizes, shapes and extents. When the characterization module 32 of the image processing system 30 uses a convolutional neural network algorithm, it is further preferred that the pixel regions 301 segmented by the image segmentation module 31 are single pixel units.
[0073] The characterization module 32 includes a pixel processing module 321, a feature extraction module 322, and a feature output module 323. The pixel processing module 321 processes an array corresponding to a pixel unit in the pixel region 301. That is, the pixel processing module 321 normalizes the pixel region 301 into an array that is easy to process. After inputting the array of the pixel region 301 processed by the pixel processing module 321, the feature extraction module 322 extracts a feature type corresponding to the array, selects features, filters information, keeps usable data, and eliminates interference data, so that the feature extraction result is more accurate. The feature output module 323 outputs the features extracted by the feature extraction module 322, and combines the features output by the feature output module 323 through the region partition module 33 to generate the classification label of the corresponding region.
[0074] The region partitioning module 33 partitions each region corresponding to the image and sets region boundaries based on the features corresponding to the pixel regions 301 extracted by the characterization module 32. Therefore, the region partitioning module 33 further includes a region segmentation module 331 and a boundary demarcation module 332, where the region segmentation module 331 partitions different regions based on the features of the pixel regions 301, and the boundary demarcation module 332 identifies the region extents by partitioning the boundary extents corresponding to the regions.
[0075] The image acquisition device 20 acquires images in a field of view in front of the harvester body 10 in real time while the harvester body 10 of the harvester is traveling. The image processing system 30 acquires images captured by the image acquisition device 20 in real time, and recognizes the area division and area boundary range of the image corresponding to the farmland using an image division recognition technique. If the area division and area boundary range recognized by the image processing system 30 do not match the previous area boundary range, the area division and area boundary range corresponding to the image are adjusted.
[0076] 8, problems such as vibration and deviation in traveling direction inevitably occur during traveling and operation of the harvester body 10. When the traveling direction of the harvester body 10 deviates or the vehicle vibrates, causing a change in area division, the image processing system 30 updates the area division and area boundary range corresponding to the image in real time.
[0077] As shown in FIG. 1, the harvester further includes a positioning device 40 and a navigation system 50, and the positioning device 40 is provided in the harvester body 10 to acquire positioning information of the harvester body 10. Preferably, the positioning device 40 acquires the position information of the harvester body 10 using satellite positioning information such as a GPS or a Beidou positioning device. The navigation system 50 is provided in the harvester body 10, and by navigating the travel of the harvester body 10, the harvester body 10 realizes unmanned automatic driving and operation based on the positioning information of the positioning device 40, the area planning information obtained by the image processing system 30, and the navigation information of the navigation system 50.
[0078] It can be understood that the farmland area division and area boundary range information, etc., obtained by the image processing system 30 based on the image are updated in real time to the navigation system 50, and the navigation information of the navigation system 50 is updated. Preferably, the navigation system 50 is implemented as an inertial combined navigation system. It can be understood that the type of the navigation system 50 is merely exemplary and not limiting, and may be implemented as other types of navigation devices.
[0079] In response to the above, the harvester body 10 of the harvester includes a vehicle body 11, a work system 12 provided on the vehicle body 11, and a driving control system 13, and the work system 12 performs grain processing work such as harvesting work while being driven by the vehicle body 11. The driving control system 13 controls the travel of the vehicle body 11 and the work of the work system 12. The driving control system 13 has an unmanned driving mode and an operation driving mode. In the unmanned driving mode of the harvester, the driving control system 13 controls the vehicle body 11 to automatically drive and the work system 12 to work. On the other hand, in the operation driving mode of the harvester, the driving control system allows a driver to manually operate the vehicle body 11 and control the work of the work system.
[0080] In a first preferred embodiment of the present invention, the harvester is a harvester machine, and the working system 12 is implemented as a mowing work machine. The driving control system 13 controls the traveling of the vehicle body 11 and the work of the working system 12. That is, the driving control system 13 controls the adjustment of the work parameters of the working system 12 while the vehicle body 11 is traveling. The driving control system 13 acquires information such as the type of crop, the height of the crop, the fullness of the fruit, and the diameter of the stem of the crop in the image recognized by the image processing system 30, and adjusts the work parameters of the working system 12 based on the acquired information, for example, adjusting the working speed, working width, working height of the working system 12, and adjusting post-processing parameters, etc.
[0081] FIG. 9 of the attached drawings of the specification of the present invention shows an embodiment of the unmanned operation and harvesting work of the harvester on farmland. In the unmanned operation mode, the operation control system 13 of the harvester body 10 acquires the positioning information of the vehicle body 11 provided by the positioning device 40, the navigation information provided by the navigation system 50, and the area recognition information provided by the image processing system 30, and further controls the vehicle body 11 to travel to the unworked area 100 of the farmland and perform the harvesting work. The image acquisition device 20 acquires the image of the area ahead of the vehicle body 11 in real time while the harvester body 10 is traveling and working, and the image processing system 30 recognizes the area range and boundary range from the image using image segmentation recognition technology. If the area division and boundary range obtained by the image processing system 30 are not consistent with the previous area division and boundary range, the image processing system 30 replaces the original area division and boundary range and updates the navigation data of the navigation system 50 so that the driving control system 13 obtains new navigation information and adjusts the driving and working paths.
[0082] As shown in FIG. 10 to FIG. 11B, the harvester generates at least one planned route based on the position information of the harvester body 10 acquired by the positioning device 40, the area planning information of the image recognized by the image processing system 30, and the navigation information of the navigation system 50. The driving control system 13 of the harvester body 10 controls the travel of the vehicle body 11 and the work of the work system 12 according to the generated planned route. Therefore, the harvester further includes a route planning system 60 that plans at least one vehicle travel route for the harvester body 10. The route planning system 60 acquires the positioning information of the positioning device 40, the area planning information of the image recognized by the image processing system 30, and the navigation information of the navigation system 50, and plans a travel route of the vehicle body 11 based on the acquired information.
[0083] As shown in Figures 11A and 11B, the route planning system 60 recognizes or sets at least one corresponding work area 601 and work boundary 602 in the farmland, the work area 601 is the widest work range of the harvester, and the driving control system 13 controls the vehicle body 11 to run within the range of the work boundary 602. It can be understood that, regarding the work area 601 and the work boundary 602, the image processing system 30 may recognize the widest area range and boundary of the work area 601 by recognizing the farmland boundary area 300 in an image. Or, the route planning system 60 may set the work area 601 of the harvester.
[0084] The path planning system 60 plans at least one travel route based on the working boundary 602 at the outermost side of the working area 601. When the width of the working area 601 is wider than the working width of the working system 12, the path planning system 60 plans a "turn"-shaped travel route or an "S"-shaped travel route. It can be understood that the form of the travel route planned by the path planning system 60 is merely exemplary and is not limiting. Therefore, other forms of travel routes are also applicable.
[0085] Preferably, when the vehicle body 11 travels to the far boundary of the working area 601, the route planning system 60 re-plans at least one driving route based on the range of the current unworked area 100. That is, when the vehicle body 11 travels to the far boundary of the working area 601, the route planning system 60 updates the working area 601 and the working boundary 602 for the vehicle body 11, and plans a new driving route based on the updated working area 601.
[0086] It can be understood that the driving control system 13 controls the vehicle body 11 to travel according to the travel path planned by the path planning system 60, and controls the work system 12 to mow the crops at the outermost part of the work area 401. In other words, the driving control system 13 controls the work system 12 to mow the crops in the unworked area 100 according to the work boundary 602.
[0087] 10, the path planning system 60 of the harvester includes a work area setting module 61, a travel path planning module 62, and a path adjustment module 63. The work area setting module 61 obtains a work area 601 and a work boundary 602 of the farmland based on the boundary area of the farmland in the image recognized by the image processing system 30, or sets the work area 601 and the work boundary 602 in which the harvester body 10 works in the farmland. Since the unworked area 100 and the worked area 200 have changed due to the work of the harvester body 10, the work area setting module 61 generates new unworked area 100 and worked area 200 by updating the range of the work area 601 and the boundary line of the work boundary 602 in real time.
[0088] The travel path planning module 62 obtains at least one travel planned route 603 based on the positioning information of the harvester body 10, the area planning information of the image recognized by the image processing system 30, and the navigation information of the navigation system 50, and the driving control system 13 controls the vehicle body 11 to travel according to the travel planned route 603. The route adjustment module 63 adjusts the travel direction of the harvester body 10 based on the information of the agricultural crops in the image recognized by the image processing system 30, and forms a vehicle travel path 604 that is approximately overlapping or parallel to the travel planned route 603. If the image processing system 30 recognizes that the mowing range of the agricultural crops in the image should be adjusted, the vehicle travel path generated by the route adjustment module 63 deviates from the travel planned route 603.
[0089] With reference to Figs. 12-16B of the attached drawings of the specification of the present invention, a harvester with an image capture device according to a second preferred embodiment of the present invention is described in the following. The harvester includes a harvester body 10 and at least one image capture device 20, the image capture device 20 is provided on the harvester body 10, and the harvester body 10 controls a running direction and / or a working parameter based on the image or video information captured by the image capture device 20 by taking an image or a video image of the farmland where the harvester body 10 is located. The image capture device 20 takes farmland information around the farmland where the harvester body 10 is located, based on the position of the harvester body 10. It can be understood that the image capture device 20 takes an image within a visual field range, for example, an image within the visual field of the driver, and adjusts the operating parameters of the harvester body 10 based on the taken image, for example, adjusting a running route, a running speed, a working parameter, etc.
[0090] The image acquisition device 20 is mounted on the harvester body 10, and the image and video information acquired by the image acquisition device 20 is transmitted to the harvester body 10, and the harvester body 10 adjusts the operation parameters based on the information. The image acquisition device 20 is mounted on the harvester body 10, and captures clear images even if the harvester body 10 shakes. In other words, the image acquisition device 20 is an anti-shake imaging device, and can avoid vibration caused by mechanical vibration of the harvester body 10 itself and unevenness of the ground during shooting. The harvester body 10 controls the travel path and work parameters by an operator's operation based on the image information captured by the image acquisition device 20, or automatically controls the travel path and work parameters, thereby realizing the operation of the harvester. In other words, the harvester body 10 adjusts the operation and work parameters based on the image information captured by the image acquisition device 20, and realizes accurate work and / or unmanned automatic driving work.
[0091] Preferably, in a second preferred embodiment of the present invention, the image capture device 20 is implemented as a pan head imaging device that captures stable images or videos even when vibrated or shaken.
[0092] In a second preferred embodiment of the present invention, the image capture device 20 is a mechanical head device, which is mounted to be mechanically connected to the harvester body 10, and realizes image shake-proof shooting by mechanical shake prevention. It can be understood that the type of the image capture device 20 is only exemplary and not limiting. Therefore, other types of structures and mounting configurations are also applicable.
[0093] As shown in Figs. 12 to 14, the image acquisition device 20 includes a camera platform 21 and at least one video camera 22, and the video camera 22 is attached to the harvester body 10 by the camera platform 21, and the attachment position of the video camera 22 is fixed by the camera platform 21. The bottom end of the camera platform 21 is mounted on the harvester body 10, and the camera platform 21 is fixed via the harvester body 10, and the top end of the camera platform 21 is connected to the video camera 22. The video camera 22 maintains a balanced state by being supported by the camera platform 21, and stably captures or films images. The video camera 22 captures images or films around the harvester body 10 under the supporting action of the camera platform 21, and captures images within the field of view of the harvester body 10 based on the attachment position of the camera platform 21. It can be understood that the video camera 22 of the image capture device 20 captures at least one visual image by taking a photograph based on the position of the harvester body 10. That is, the video camera 22 of the image capture device 20 captures the image within the field of view of the harvester body 10, thereby avoiding the problem of inaccurate image data due to the change in the position of the imaging device 20 and the position of the harvester body 10.
[0094] In addition, in order for an autonomous vehicle to recognize and acquire a vehicle driving route in autonomous driving mode, accurate vehicle positioning information must be acquired, and usually, highly accurate satellite positioning information is required. In order for the autonomous vehicle to realize an autonomous driving function in a high-speed driving state, information such as obstacle information on the road, road vehicle information, and road pedestrians must be updated in real time. The image acquired by the harvester of the present invention is image data information corresponding to agricultural crop grains in farmland, and the image is an image of the surroundings of the vehicle acquired based on the current position of the vehicle. The harvester does not require ultra-high accuracy satellite positioning information, and only general satellite positioning with meter-level accuracy (such as GPS positioning or Beidou positioning) is required. Therefore, the image acquired and processed by the harvester is different from that of an autonomous vehicle, and the route plan and driving mode formed by the harvester are also not the same. It can be understood that the function of the harvester of the present invention to visually recognize the area of the farmland and the autonomous driving function are different from the recognition mode of an autonomous vehicle.
[0095] The pan / tilt head 21 of the image acquisition device 20 further includes a pan / tilt head fixing member 211 and at least one pan / tilt head moving member 212, and the pan / tilt head moving member 212 is movably connected to the pan / tilt head fixing member 211. The pan / tilt head fixing member 211 is fixedly provided on the harvester body 10, and the video camera 22 is attached to the pan / tilt head moving member 212. The pan / tilt head moving member 212 of the pan / tilt head 21 movably supports the video camera 22, so that even if the harvester body 10 shakes, the video camera 22 maintains a stable relative position, thereby taking clear images.
[0096] That is, when the harvester body 10 vibrates or swings, for example, mechanically vibrates or swings when performing harvesting work in farmland, the camera head fixing member 211 of the camera head 21 vibrates together with the harvester body 10, and the camera head moving member 212 of the camera head 21 moves relative to the camera head fixing member 211 to neutralize the vibration caused by the camera head fixing member 211, thereby maintaining the positional stability of the video camera 22. In detail, the camera head moving member 212 neutralizes the shaking or vibration of the moving fixing member 211 in the up-down, left-right, and front-back directions, maintains the stability of the photographing position of the video camera 22, and captures more stable image information.
[0097] As shown in Fig. 14, the video camera 22 of the image acquisition device 20 is provided on the head moving member 212 of the camera head 21, and is fixedly or movably attached to the head moving member 212 of the camera head 21. Preferably, the video camera 22 is movably provided on the movable moving member 212, and is rotatable with respect to the upper end of the head moving member 212, and captures images in different viewing directions. Optionally, the video camera 22 is fixedly attached to the upper end of the head moving member 212, and captures images within a specified viewing range under the fixed and supported action of the camera head 21, for example, captures images within a forward viewing field of the harvester body 10.
[0098] The video camera 22 includes a video camera body 221 and at least one video camera driving device 222, and the video camera driving device 222 drives and moves the video camera body 221 so as to capture images of fields of view in different directions. The video camera body 221 is movably mounted on the pan head moving member 212 and can rotate up and down under the driving action of the video camera driving device 222, thereby capturing images of farmland and crops far and near the harvester body 10. It can be seen that when the video camera body 221 is driven by the video camera driving device 222 to rotate downward, it captures an image near the harvester body 10, thereby clearly recognizing information of the farm crops in the image. When the video camera body 221 is driven by the video camera driving device 222 to rotate upward, it captures an image far from the harvester body 10, thereby recognizing the work area and farmland boundary area of the farmland from the image.
[0099] The video camera driving device 222 drives the video camera body 221 to rotate it left and right, thereby causing the video camera body 221 to capture images of the left and right sides of the harvester body 10, thereby recognizing the unworked area 100, the worked area 200, and the farmland boundary area 300 of the farmland.
[0100] FIG. 15 of the accompanying drawings of the specification of the present invention shows some optional mounting forms and mounting positions of the image capture device 20 attached to the harvester body 10. In a second preferred embodiment of the present invention, the image capture device 20 of the harvester is provided at the front position, the top, the left side, the right side, and the rear position of the harvester body 10. It can be understood that the captured images are different and the information recognized from the images is not the same depending on the mounting position of the image capture device 20. The image capture device 20 provided at the front side of the harvester body 10 captures an image in front of the harvester body 10. It can be understood that when the harvester travels forward to work, the image capture device 20 at the front side of the harvester body 10 captures the working situation of the harvester body 10, and adjusts the travel path, working parameters, etc. of the harvester body 10 according to the photographed working situation.
[0101] The image capture device 20 provided at the rear of the harvester body 10 captures an image of the rear of the harvester body 10, and captures an image of the worked area 200 when the harvester travels forward to work. By recognizing the image of the worked area 200 captured by the image capture device 20 at the rear of the harvester body 10, it is possible to recognize whether the harvester body 10 has passed the harvesting work, and it is easy to adjust the work parameters of the harvester body 10. In addition, the harvester body 10 recognizes whether the agricultural products in the worked area 200 have been completely harvested and whether the fruits of the agricultural products remain, etc., based on the image captured by the image capture device 20 provided at the rear of the harvester body 10. The harvester body 10 adjusts the work parameters based on the information recognized from the image, and further improves the harvesting work. In addition, when traveling backward, the image captured by the image capture device 20 provides a back guide image to the driver.
[0102] The image capture device 20 provided at the top of the harvester body 10 captures an image of the harvester body 10 at a distance, and recognizes the work area of the farmland, the border area of the farmland, etc. based on the image. Preferably, the image capture device 20 provided at the top of the harvester body 10 is a rotatable pan-head video camera.
[0103] In addition, the image acquisition device 20 provided on the left or right side of the harvester body 10 captures an image of the left or right side of the harvester body 10. Based on the image of the left or right side of the harvester body 10, the state of the farm crops in the farmland in the image is recognized, thereby recognizing the unworked area 100, the worked area 200, and the farmland boundary area 300.
[0104] As shown in Figures 12 and 14, the harvester further includes an image processing system 30, a positioning device 40, and a navigation system 50, and the image processing system 30, the positioning device 40, and the navigation system 50 are provided in the harvester body 10. The positioning device 40 acquires position information of the harvester body 10 and transmits the acquired position information to the harvester body 10. The navigation system 50 provides navigation information to the harvester body 10 based on the positioning information of the positioning device 40. The image processing system 30 recognizes the unworked area 100, the worked area 200, and the farmland boundary area 300 from the image based on the image of the farmland acquired by the image acquisition device 20.
[0105] Preferably, the image processing system 30 recognizes the unworked area 100, the worked area 200, and the farmland boundary area 300 from the image using an image segmentation recognition technique. It may be understood that the image processing system 30 may recognize area and boundary information in the image in other forms. Therefore, in the second preferred embodiment of the present invention, the form in which the image processing system 30 recognizes the image is merely exemplary and not limiting.
[0106] As shown in Figures 16A and 16B, the image processing system 30 recognizes the farmland area and farmland boundaries in the image based on the image of the surroundings of the harvester body 10 captured by the image acquisition device 20, and recognizes information such as the type of crop in the farmland, the height of the crop, the fullness of the fruit, and the thickness and size of the stem.
[0107] The image processing system 30 selects one of the segmentation recognition methods, such as a segmentation method based on a threshold, a segmentation method based on an area, a segmentation method based on an edge, and a segmentation method based on a specific theory, and segments and recognizes the image acquired by the image acquisition device 20 to recognize the area and boundary in the image. Preferably, the image processing system 30 uses a deep learning algorithm to segment and recognize the image, and performs area segmentation and boundary definition for the image. That is, the image processing system 30 uses a deep learning algorithm to recognize the area and boundary of the corresponding farmland in the image, and the harvester body 10 travels and works according to the recognized and segmented area and boundary. More preferably, the image processing system 30 uses an image segmentation recognition technique of a convolutional neural network algorithm in the deep learning algorithm to recognize the unworked area 100, the worked area 200, and the farmland boundary area 300 in the corresponding farmland from the image.
[0108] The processing algorithm used by the image processing system 30 is merely illustrative and is not limiting. Therefore, the image processing system 30 may further use other algorithms to segment and recognize the acquired image and recognize the farmland area and boundaries in the image.
[0109] The image processing system 30 is an image processor provided in the harvester body 10, and the image processor receives images or videos captured by the image acquisition device 20 and recognizes information in the images or videos. It can be understood that the harvester body 10 is operated according to the information recognized by the image processing system 30 to control the travel path and adjust the operation parameters.
[0110] As shown in FIG. 12 and FIG. 14, the harvester body 10 further includes a vehicle body 11, a work system 12 provided on the vehicle body 11, and an operation control system 13. The work system 12 is connected to the vehicle body 11, and the vehicle body 11 drives the work system 12 to perform crop harvesting work. The operation control system 13 controls the travel of the vehicle body 11 and controls the work of the work system 12. The operation control system 13 has an unmanned operation mode and an operation operation mode. In the unmanned operation mode of the harvester, the operation control system 13 controls the vehicle body 11 to automatically operate and the work system 12 to operate. On the other hand, in the operation operation mode of the harvester, the operation control system allows a driver to manually operate the vehicle body 11 and control the work of the work system.
[0111] In a second preferred embodiment of the present invention, the driving control system 13 controls the traveling of the vehicle body 11 and the mowing work of the working system 12. That is, the driving control system 13 controls the adjustment of the work parameters of the working system 12 while the vehicle body 11 is traveling. The driving control system 13 acquires information such as the type of crop, the height of the crop, the fullness of the fruit, and the diameter of the stem of the crop in the image recognized by the image processing system 30, and adjusts the work parameters of the working system 12 based on the acquired information, for example, adjusting the working speed, working width, working height of the working system 12, and adjusting post-processing parameters, etc.
[0112] The work system 12 further includes at least one mowing device 121, at least one transport device 122, and at least one post-treatment device 123, and the transport device 122 is configured to receive crops mowed by the mowing device 121 and transport the crops to the post-treatment device 123 so that the post-treatment device 123 performs post-treatment on the crops. The mowing device 121, the transport device 122, and the post-treatment device 123 of the work system are respectively provided and connected to the vehicle body 11 by transmission, and the vehicle body 11 drives the mowing device 121, the transport device 122, and the post-treatment device 123 of the work system 12 to operate and perform work. It can be understood that the post-treatment device 123 is implemented as a subsequent treatment device after the crop is harvested, for example, as a threshing device if the harvester is a grain harvester, as a packing device if the harvester is a grass-cutting machine, and as a vegetable or fruit screening and storage device if the harvester is a vegetable or fruit harvesting machine.
[0113] The operation control system 13 controls the width, mowing height, and mowing speed of the mower 121 based on the image information recognized by the image processing system 30. It can be understood that when the density of crops in the farmland is high, the information of the crops in the farmland photographed by the image acquisition device 20 is recognized by the image processing system 30, and the operation control system 13 controls any of the work parameters of the mower 121, such as reducing the mowing width, increasing the mowing height, and reducing the mowing speed, based on the image information recognized by the image processing system 30.
[0114] The driving control system 13 controls the transport speed, transport power, etc. of the transport device 122 based on the image information recognized by the image processing system 30. It can be understood that when the stems of the crops in the farmland are coarse, the height of the crops is high, and the density is high, the information of the crops in the farmland photographed by the image acquisition device 20 is recognized by the image processing system 30, and the driving control system 13 controls the transport device 122 to improve the work parameters such as the transport speed, transport power, etc. based on the image information recognized by the image processing system 30.
[0115] The operation control system 13 controls the post-processing parameters of the post-processing device 123 based on the image information recognized by the image processing system 30. The image information includes the fullness of the fruit of the crop in the farmland, the size of the fruit, the moisture content, the humidity, the type of the fruit of the crop, etc. It can be understood that the image processing system 30 recognizes the crop information of the crop in the farmland, and the operation control system 13 adjusts the post-processing parameters of the post-processing device, such as the blowing power and the rotation speed of the post-processing chamber, based on the image information recognized by the image processing system 30.
[0116] 17 of the accompanying drawings of the specification of the present invention, another optional embodiment of the image capture device 20A of the harvester according to the second preferred embodiment of the present invention is described in the following description. In this optional embodiment, the image capture device 20A realizes lens photography and prevents shake by controlling the viewpoint and zoom of the lens inside a video camera.
[0117] In response to the above, the image acquisition device 20A includes a camera mounting means 21A and at least one video camera 22A, and the video camera 22A is mounted on the harvester body 10 by the camera mounting means 21A. The bottom end of the camera mounting means 21A is mounted on the harvester body 10, the camera mounting means 21A is fixed via the harvester body 10, and the upper end of the camera mounting means 21A is connected to the video camera 22A. The video camera 22A maintains a balanced state by being supported by the camera mounting means 21A, and stably captures or films images. The video camera 22A captures images or videos of the surroundings of the harvester body 10 by the supporting action of the camera mounting means 21A, and the video camera 22A captures images within the field of view of the harvester body 10 based on the mounting position of the camera mounting means 21A.
[0118] The video camera 22A of the image capture device 20A captures at least one visual image by taking a photograph based on the position of the harvester body 10. That is, it can be seen that the video camera 22A of the image capture device 20A captures the image within the field of view of the harvester body 10, thereby avoiding the problem of inaccurate image data due to changes in the position of the image capture device 20A and the harvester body 10 position.
[0119] According to another aspect of the present invention, there is further provided an automatic operation method for a harvester, the automatic operation method comprising: (a) acquiring at least one image and recognizing farmland areas and farmland boundaries in said image; (b) planning at least one driving plan route 603 based on the recognition information; and (c) controlling the harvester body 10 to automatically travel according to the travel plan route 603.
[0120] In the above-mentioned automatic driving method steps, the driving control system 13 controls the driving and operation of the harvester body 10 based on the area information and farmland boundaries recognized by the image processing system 30.
[0121] Step (a) of the above-mentioned automatic driving method further includes a step of recognizing information corresponding to the crops in the farmland in the image, the crop information including information such as the crop type, crop height, and fruit fullness.
[0122] The above-mentioned automatic driving method step (b) (b.1) recognizing and demarcating areas and boundaries of said image corresponding to agricultural land; and (b.2) planning at least one driving plan route 603 based on the recognized area.
[0123] Step (b.1) of the above-mentioned autonomous driving method further includes a step of segmenting the image using an image segmentation technique and recognizing and partitioning regions of the image.
[0124] In step (a) of the automatic driving method, image information of the surroundings of the harvester body 10 is captured in real time according to the position and traveling direction of the harvester body 10. That is, the image acquisition device 20 captures images near the harvester body 10 in real time.
[0125] In step (b) of the above-mentioned automatic driving method, the image processing system uses an image segmentation technique to segment the image information, and recognizes and divides the image area into the unworked area 100, the worked area 200, and the farmland boundary area 300. Accordingly, step (b.1) of the automatic driving method is as follows: Dividing the image into a plurality of pixel regions 301 and normalizing pixel values of the pixel regions 301 into an array; extracting features of the pixel regions 301 corresponding to each of the arrays; and outputting a classification label for the image based on features corresponding to the pixel region 301.
[0126] It can be seen that the classification labels correspond to the unworked area 100 , the worked area 200 , and the field boundary area 300 .
[0127] Step (b.2) of the above-mentioned automatic driving method further includes a step in which the image processing system 30 recognizes area planning information of the image and navigation information of the navigation system 50 based on positioning information of the harvester body 10, and obtains the driving planned route 603.
[0128] Step (b.2) of the above-mentioned automatic driving method further includes a step of adjusting the traveling direction of the harvester body 10 based on information of the agricultural crop in the image recognized by the image processing system 30, and forming a vehicle traveling path 604.
[0129] The above-mentioned autonomous driving method further includes a step (b.3) of comparing whether the area division and area boundary range recognized by the image processing system 30 match with the previous area boundary range, and if they do not match, adjusting the area division and area boundary range corresponding to the image, and if they match, keeping the area division and boundary range unchanged.
[0130] Corresponding to the above, in the above method step (c), the driving control system 13 controls the travel of the vehicle body 11 of the harvester body 10 based on the positioning information of the harvester body 10, the area planning information of the farmland obtained by the image processing system 30, and the navigation information.
[0131] The automatic driving method further includes a step (d) of adjusting work parameters of the work system 12 of the harvester body 10 based on the image recognition information.
[0132] It can be understood by those skilled in the art that the embodiments of the present invention shown in the above description and drawings are merely examples and are not intended to limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functions and structural principles of the present invention have been shown and explained in the examples, and the embodiments of the present invention can be arbitrarily modified or altered without departing from the above principles.
Claims
1. A harvester body; At least one image capture device provided on the harvester body for capturing images of the surroundings of the harvester body; and an image processing system that recognizes farmland information in the image based on the image captured by the image capture device, The image processing system includes: an image segmentation module for segmenting the image into a plurality of pixel regions, each pixel region including at least one pixel unit; a characterization module for normalizing pixel values of the pixel regions into arrays based on the pixel units of the pixel regions, extracting features of the pixel regions corresponding to each of the arrays, and selecting the features; a region partitioning module for recognizing and partitioning regions of the image based on characteristics of the pixel regions; The image processing system divides the farmland in the image into different areas by the image segmentation module, and the farmland information includes an area division and an area boundary range corresponding to the image; the image processing system acquires images captured by the image acquisition device in real time, and compares whether the recognized area division and area boundary range are consistent with the previous area boundary range, and if not consistent, adjusts the area division and area boundary range corresponding to the image; if consistent, keeps the area division and area boundary range unchanged; A harvester characterized in that the harvester body automatically controls operation based on the farmland information recognized by the image processing system.
2. Further comprising a route planning system that plans at least one travel route based on the farmland information recognized by the image processing system; The harvester according to claim 1 , wherein the harvester body controls operation according to the travel plan route planned by the route planning system.
3. The harvester according to claim 2 , wherein the image processing system uses an image segmentation recognition technique to recognize agricultural crop information in the image, and the harvester body automatically adjusts operation parameters based on the recognized information.
4. The harvester of claim 1 , wherein the image capture device is provided at the front of the harvester body, at the top of the harvester body, on the left side of the harvester body, on the right side of the harvester body, or at the rear of the harvester body.
5. The harvester of claim 2, further comprising a positioning device and a navigation system, the positioning device and the navigation system being provided in the harvester body, the positioning device acquiring position information of the harvester body, and the navigation system providing navigation information to the harvester body.
6. The path planning system comprises: a working area setting module for setting a boundary area of the farmland to obtain a working area and a working boundary of the farmland; A travel route planning module, The harvester according to claim 5 , wherein the route planning system obtains at least one travel planned route based on positioning information of the harvester body, area planning information of the image recognized by the image processing system, and navigation information of the navigation system.
7. 2. The harvester according to claim 1, wherein the harvester body includes a vehicle body, at least one work system provided on the vehicle body, and an operation control system, the vehicle body drives and operates the work system, and the operation control system controls the operation of the vehicle body and work parameters of the work system.
8. The harvester according to claim 7, wherein the driving control system acquires information of images recognized by the image processing system and captured by the image acquisition device, automatically controls a travel route of the vehicle body, and controls work parameters of the work system, thereby realizing unmanned automatic driving and mowing work.
9. In the automatic operation method of a harvester, (a) acquiring at least one image and recognizing farmland areas and farmland boundaries in said image; (b) planning at least one driving plan route based on the recognition information; and (c) a step of controlling the harvester body to automatically travel along the travel planned route, The step (b) (b.1) segmenting the image using image segmentation techniques to identify and demarcate areas and boundaries of the image corresponding to agricultural fields; (b.2) planning at least one travel plan route based on the recognized area; (b.3) comparing whether the area division and area boundary range recognized by the image processing system are consistent with the previous area boundary range, and if not, adjusting the area division and area boundary range corresponding to the image, and if consistent, keeping the area division and area boundary range unchanged; The step (b.1) comprises: Dividing the image into a number of pixel regions and normalizing pixel values of the pixel regions into an array; extracting features of the pixel regions corresponding to each of the arrays; and outputting a classification label for the image based on features corresponding to the pixel region.
10. The autonomous driving method of claim 9, wherein step (a) further includes a step of recognizing information corresponding to crops in the farmland in the image, and the crop information includes plant information such as crop type, crop height, and fruit fullness.
11. The autonomous driving method of claim 9, wherein in step (b), the image processing system uses image segmentation technology to segment the image and recognize and divide the areas of the image into unworked areas, worked areas, and farmland boundary areas.
12. The autonomous driving method of claim 9, wherein step (b.2) further includes a step of planning the travel plan route based on positioning information of the harvester body, area planning information of the image, and navigation information of a navigation system.
13. The automatic driving method according to claim 9, further comprising the step (d) of adjusting work parameters of a work system of the harvester body based on the image recognition information.
Citation Information
Patent Citations
Visual navigation path recognition system of grain combine harvester
CN103914071A
Apparatus for detecting row of crop in farm working machine
JP1989211408A
Apparatus for binarization processing of image information in steering control of mobile working machine
JP1991019607A
Image retrieval device, image classifying device, and computer readable recording medium recorded with program for functioning computer as these devices
JP2000285141A
Work vehicle for agriculture
JP2004008186A
