Combine

Abstract

Provided is a combine comprising: a threshing device (4) for threshing grain culms; a feed chain (4B) for supplying the grain culms to the threshing device (4); cameras (103, 104) and a ranging device (105) for monitoring manual threshing work by a worker; and a work state estimation device (120) for assessing the risk of the work performed by the worker from a video captured by the cameras (103, 104) and ranging information obtained by the ranging device (105). Due to this configuration, this combine is capable of appropriately assessing the risk of the manual threshing work.　In addition, the cameras (103, 104) are disposed at positions higher than a grain culm conveyance height of the feed chain (4B), and the ranging device (105) is disposed at a position lower than the grain culm conveyance height of the feed chain (4B).

Description

combine

[0001] The present invention relates to a combine harvester.

[0002] Conventionally, a combine harvester is known that is provided with a camera for capturing an image of a hand-threshing operation section in order to evaluate the risk of hand-threshing operation.

[0003] Japanese Patent Application Laid-Open No. 2013-9653

[0004] However, Patent Document 1 describes that image analysis processing is used to determine whether the worker has moved backward from the manual handling position for a predetermined period of time, but because it only evaluates the movement time, it cannot be said to be sufficient for assessing risk, and there is significant room for improvement.

[0005] Therefore, the main object of the present invention is to provide a combine harvester that can appropriately evaluate the danger of manual threshing work.

[0006] The present invention, which has solved the above problems, is as follows.

[0007] That is, the invention described in claim 1 is a combine harvester equipped with at least one camera (103, 104) for monitoring the manual threshing work of a worker, and a work state estimation device (120) for evaluating the risk of the manual threshing work from the images captured by the cameras (103, 104), wherein the work state estimation device (120) is equipped with a posture estimation unit (122) for outputting coordinate information of the worker in the image, and evaluates the risk of the manual threshing work based on the coordinate information of the worker output by the posture estimation unit (122), and the coordinate information includes coordinates corresponding to any of the shoulders, elbows, head, neck, and waist.

[0008] The invention described in claim 2 is a combine harvester according to claim 1, further comprising a distance measuring device (105), and the work state estimation device (120) is configured to evaluate the risk of manual threshing work from the distance information of an object obtained by the distance measuring device (105).

[0009] The invention described in claim 3 is the combine described in claim 1 or 2, characterized in that the work state estimation device (120) has a proximity determination unit (124) that evaluates the distance between a danger area defined in advance in the coordinate system of the imaging area and the position of the worker's trunk based on the worker's coordinate information output by the posture estimation unit (122).

[0010] The invention described in claim 4 is a combine described in claim 2, in which the cameras (103, 104) are arranged above the stalk transport height of the feed chain (4B) that supplies stalks to the threshing device (4), and the distance measuring device (105) is arranged below the stalk transport height of the feed chain (4B).

[0011] The invention described in claim 5 is the combine described in claim 2, in which the work state estimation device (120) is capable of evaluating the risk of manual threshing work using the estimation results of the posture estimation unit (122), and of evaluating the risk of manual threshing work without using the estimation results of the posture estimation unit (122).

[0012] According to the present invention, the risk of manual handling work can be appropriately evaluated.

[0013] Fig. 1 is a left side view of the combine harvester; Fig. 2 is a right side view of the combine harvester; Fig. 3 is a plan view of the left part of the control unit; Fig. 4 is a right side view of the side panel of the control unit; Fig. 5 is an explanatory diagram showing a control device of the combine harvester;

[0014] As shown in Figures 1 and 2, the combine harvester has a traveling device 2 consisting of a pair of left and right crawlers that travels on the soil surface on the underside of the body frame 1, a harvesting device 3 that harvests the stalks in the field on the front side of the body frame 1, a threshing device 4 that threshers and sorts the harvested stalks on the rear left side of the harvesting device 3, and a control unit 5 on which an operator rides on on the rear right side of the harvesting device 3.

[0015] An engine room 6 for mounting an engine E is provided below the control unit 5, a grain tank 7 for storing threshed and sorted grain is provided behind the control unit 5, and a discharge auger 8 consisting of a grain lifting section extending vertically and a horizontal discharge section extending longitudinally for discharging grain to the outside is provided behind the grain tank 7. The control unit 5 is also covered by a cabin 9 equipped with an illumination light on top.

[0016] As shown in FIG. 3, a front panel 45 is provided in front of the pilot's seat of the pilot section 5, and a side panel 50 is provided on the left side.

[0017] The front panel 45 is provided with a monitor 46 that displays the traveling speed of the traveling device 2 and the output rotation of the engine E. In addition, to the right of the monitor 46, there are provided operation levers (not shown) for turning the traveling device 2 and raising and lowering the reaping device 3.

[0018] The left portion of the side panel 50 is provided adjacent to the left front / rear frame 16D that supports the left wall of the cabin 9. A speed change lever 51 for operating the continuously variable transmission is provided at the front portion of the side panel 50, an emergency stop switch (also referred to as a "switch") 52 for emergency stopping the engine E is provided behind the speed change lever 51, and a reaper / thresh lever 53 for operating a reaping clutch that transmits the output rotation of the engine E to the reaping device 3 and a threshing clutch that transmits the output rotation of the engine E to the threshing device 4 is provided to the right of the emergency stop switch 52.

[0019] The emergency stop switch 52 is provided in a recess 55 of the side panel 50, and the top of the emergency stop switch 52 is provided at approximately the same position as the upper surface of the side panel 50. This makes it possible to prevent the emergency stop switch 52 from being pressed accidentally.

[0020] In a side view, the front wall of the recess 55 is formed with a front-to-rear inclination, and the rear wall is formed with a rear-to-rear inclination. In addition, in the front-to-rear direction, the emergency stop switch 52 is moved forward and located rearward of the grip portion of the mower release lever 53 when it is in a forward-inclined position, and moved rearward and located rearward of the grip portion of the mower release lever 53 when it is in a rear-inclined position. This allows the emergency stop switch 52 to be quickly pressed if the engine E overheats, etc.

[0021] A power port 71 is provided at the rear of the side panel 50 to extract power from the vehicle battery into the cabin 9. This power port 71 is of the USB (Universal Serial Bus) type and supplies power to devices such as a portable information terminal carried by the worker.

[0022] A power steering lever is also provided on the front right side of the cabin 9. This power steering lever is operated forward and backward to raise and lower the harvesting device 3, and left and right to steer the machine. A camera is also mounted on the upper front part of the cabin 9 to capture images of the area in front of the machine. This camera is used by a supervisor who remotely monitors the work status and for image analysis by evaluation systems such as machine learning. However, this type of camera cannot capture the intended destination of the operator when the machine is steered until the machine's direction of travel actually changes, leaving room for improvement in its convenience as a means of confirmation. Therefore, when the power steering lever is operated, it is preferable to orient the camera's optical axis in the direction of power steering lever operation depending on the angle and duration of operation. Conversely, it can also be oriented in the opposite direction to the power steering lever operation.

[0023] Next, the manual threshing section will be described. The manual threshing section is the section where an operator supplies hand-cut stalks to the threshing device 4. More specifically, the front part of the feed chain 4B that supplies stalks to the threshing device 4 has a conveying section exposed upward in the area forward of the front wall 4A of the threshing device 4. Therefore, the area in front of the threshing device 4, from the front wall 4A of the threshing device 4 to the front end of the feed chain 4B, is the manual threshing section.

[0024] The manual threshing (threshing) work of feeding straw into this manual threshing work section is dangerous if clothing or the human body interferes with the feed chain 4B, which moves at high speed. Therefore, when a dangerous event occurs during the work state, the work vehicle control device 110 (Figure 5) executes control accordingly to ensure the safety of the feed chain 4B, etc.

[0025] An emergency stop switch 101, a reaping unit clutch sensor 102, etc. are connected to the input interface of the work vehicle control device 110.

[0026] The output interface of the work vehicle control device 110 is connected to an engine control device 131, a reaping section clutch 132, a threshing section clutch 133, a warning output device 134, a threshing drum cover release mechanism 135, and the like.

[0027] A work state estimation device 120 is also connected to the work vehicle control device 110. A first camera (also referred to as "camera") 103 and a second camera (also referred to as "camera") 104 are connected to the work state estimation device 120, and captured images can be input. A LIDAR device (also referred to as "range finding device") 105 is also connected to the work state estimation device 120. The LIDAR device 105 is a known light detection and ranging device, and can be replaced with any other range finding device as long as it is capable of three-dimensionally searching for the distance to the surrounding environment.

[0028] Although not shown in detail, the emergency stop switch 101 is provided on the outer left side surface at the front of the threshing device 4 and is operated by the operator when he senses danger. The reaping unit clutch sensor 102 detects the connection state of the reaping clutch provided in the transmission path from the engine E to the reaping device 3.

[0029] For convenience, the engine control device 131 is depicted as being connected to an output interface. However, in reality, it is connected to the work vehicle control device 110 via a so-called CAN, and communicates command signals and the like to acquire status data of the engine E and control the rotational speed, etc. The reaping unit clutch 132 and the threshing unit clutch 133 connect and disconnect power from the engine E to the reaping device 3 and the threshing device 4. The warning output device 134 alerts the manual threshing worker to dangerous situations and may be installed anywhere, but is preferably located near the front of the threshing device 4. The threshing drum cover release mechanism 135 is an actuator located on the top of the threshing device 4 to forcibly open the threshing drum cover that covers the upper half of the threshing drum. The threshing drum cover is equipped with a clamping rod that faces the upper side of the feed chain 4B. Therefore, opening the threshing drum cover not only exposes the upper half of the threshing drum, but also releases the feed chain 4B from clamping the straw.

[0030] The first camera 103 and the second camera 104, which are connected to the work state estimation device 120 by wire or wirelessly, are both cameras for capturing images of the area around the hand-threshing work unit, and the first camera 103 is fixed to the left frame at the bottom of the left side of the cabin 9. The first camera 103 is installed with its optical axis facing left and right, and captures images of the area around the hand-threshing work unit from the cabin 9 side. The second camera 104 is provided at the rear of the reaping device 3, and is located in the space below the conveyor cover that covers the upper side of the straw conveying mechanism leading to the threshing device 4, and is fixed to the support frame of the conveyor cover. The optical axis of the second camera 104 faces forward and backward, and captures images of the area around the hand-threshing work unit from the reaping device 3 side, which is in front of the hand-threshing work unit.

[0031] The work state estimation device 120 evaluates the state of the manual threshing work based mainly on the video data captured by the first camera 103 and the second camera 104. Markers are provided on the front wall 4A of the threshing device 4 to identify the angle and position of the video captured by the camera relative to the camera. The markers are black-and-white images of several pixels in length and width, and are recognized by the work state estimation device 120 based on their arrangement. Multiple markers are provided, and the relationship between the real coordinate system and the camera coordinate system is evaluated based on the relative positions of the markers in the video. For example, markers are provided at each of the four vertices of a rectangle in a single plane on the front surface of the front wall 4A. Based on the shape of this rectangle in the camera video, it is possible to recognize how the real coordinate system appears, and evaluate the position and size of objects reflected in the video.

[0032] By performing this coordinate system identification process on each of the first camera 103 and the second camera 104 and taking into account their relative postures and positions, it is possible to more accurately evaluate the positions of objects captured by both cameras. In this way, the various functions of the work state estimation device 120, such as the person detection unit 121, posture estimation unit 122, clothing determination unit 123, proximity determination unit 124, and movement determination unit 125, perform various processes using the coordinate information obtained by the above coordinate system identification process together with the images from the first camera 103 and the second camera 104.

[0033] The various estimation units and judgment units provided in the work state estimation device 120 are trained models based on machine learning, and are given training data to perform basic learning for judgment, and then fine-tuned to optimize them for the vehicle body and camera.

[0034] The person detection unit 121 of the work state estimation device 120 detects areas in which people are captured in the images captured by each of the first camera 103 and the second camera 104. How the person detection unit 121 outputs the determined information is arbitrary, but the format is to output area coordinate information including the size and position of a rectangle surrounding the area in which a person is captured, together with a probability value that the area is a person.

[0035] The posture estimation unit 122 estimates the posture of the person based on the information of the person area extracted by the person detection unit 121. The posture estimation is performed by fitting the person in the video to a predetermined human body shape model. That is, positions corresponding to several key points such as the shoulders, elbows, and head are identified in the image, and a model of straight bone segments connecting these points is defined in an evaluation space of the real coordinate system, thereby estimating the position and posture of the worker. Note that because the range captured by the first camera 103 and the second camera 104 is generally limited to the upper body, posture estimation is performed only on the upper body to reduce the amount of calculation required for video analysis.

[0036] The posture estimation unit 122 outputs meta-evaluation data relating to the posture of the person that is a summary of the coordinate information of each key point.

[0037] The clothing determination unit 123 estimates and determines the state of clothing of the person in each of the images captured by the first camera 103 and the second camera 104. In particular, it estimates whether or not the person is wearing gloves, which have a significant impact on the risk of work, and whether or not the person's clothing is disheveled, which may increase the risk of getting caught in something.

[0038] The clothing determination unit 123 determines whether or not gloves are being worn. Specifically, based on the position information of the head or neck obtained by the posture estimation unit 122, the image pixel information of the area corresponding to that area is referenced to calculate the color of the worker's skin. Similarly, the skin color of the area around the hand joints is calculated. Whether gloves are being worn on the hands is determined using the difference in these colors and the evaluation results of a function that evaluates the skin color-likeness of a separately prepared color code (RGB values, etc.). The clothing determination unit 123 also determines whether a towel is being worn around the neck. As with the case of wearing gloves, this is preferably determined from the color of the neck.

[0039] The clothing determination unit 123 performs the determination as described above, but since information on the mere presence or absence of problems with clothing is insufficient for risk assessment, the degree of risk due to clothing is converted into a numerical value and output.

[0040] The proximity determination unit 124 determines how close the worker's body is to the moving parts of the manual threshing unit. During manual threshing, the worker passes both arms over the feed chain 4B and positions them above the manual threshing unit. Furthermore, the forearms are often hidden by the straw bundles, making it difficult to assess the dangerous proximity to the moving parts from the image. Therefore, the proximity determination unit 124 uses the output results of the posture estimation unit 122 to evaluate the dangerous proximity to the moving parts based on the proximity of the worker's trunk to the dangerous area. In other words, because the positions and shapes of the threshing device 4 and the feed chain 4B are known, it is possible to determine in advance the extent to which they exist within the coordinate system of the image capture area. The dangerous area is defined as the area within a certain distance from these components, and the risk level is evaluated based on the distance of the closest point of the trunk (roughly the bone segment connecting the neck keypoint and the waist keypoint). In defining the danger area, it does not necessarily have to be a uniform distance from the moving parts, and can be appropriately specified depending on the actual risk, such as defining it wider towards the rear of the machine body downstream of the feed chain 4B, or defining it wider around exposed parts of the feed chain 4B and the openings of the threshing device 4. The proximity determination unit 124 outputs the degree of danger due to proximity in a numerical value, and preferably further distinguishes between the degree of danger due to the threshing part (threshing drum) and the degree of danger due to the feed chain 4B.

[0041] The movement determination unit 125 determines whether the worker's body has made a dangerous movement. Dangerous movements are roughly divided into two types: the first is a movement of suddenly approaching a moving part, and the second is a movement of suddenly shifting downstream in the conveying direction of the feed chain 4B (toward the rear of the machine body). Both may indicate that the worker has been caught in a moving part, and it is reasonable to evaluate a movement of suddenly approaching a moving part as a highly dangerous work movement, even if it does not result in the worker being caught in the moving part.

[0042] Therefore, the movement determination unit 125 monitors the movement of the trunk using the output result of the attitude estimation unit 122, and evaluates whether there is any significant acceleration in the movement of the trunk in particular. In this way, the degree of danger regarding approaching the moving parts and the speed and acceleration of movement toward the rear of the aircraft is evaluated, and the evaluation is quantified and output.

[0043] With the above-described configuration, the work state estimation device 120 comprehensively evaluates the danger of the work based on the output results from the clothing determination unit 123, the proximity determination unit 124, and the movement determination unit 125, and outputs a countermeasure operation command to the work vehicle control device 110 specifying a risk elimination operation.

[0044] The countermeasure operation command may be, in descending order of danger level according to the degree of danger assessed by the work state estimation device 120, stopping the engine E and releasing the threshing drum cover using the threshing drum cover release mechanism 135, stopping the feed chain 4B, slowing down the feed chain 4B, issuing a warning using the warning output device 134, etc. A combination of the above-listed operations may also be performed. That is, for example, when performing an operation for a higher danger level, it may be possible to combine and execute an operation prepared for a lower danger level. Furthermore, the threshold for classifying the countermeasure operations may also be adjustable by the worker or mechanic.

[0045] In the above example, the work state estimating device 120 comprehensively considers the output results from the clothing determination unit 123, the proximity determination unit 124, and the movement determination unit 125. However, the evaluation can also be performed by taking the maximum value of the risk levels output by these units, by using a simple average, or by using an average value weighted by one of them. It is also possible to use only one of the determination units, in which case the amount of calculation by the work state estimating device 120 can be reduced and processing can be achieved by a device with low calculation resources.

[0046] Furthermore, although the work state estimating device 120 evaluates images from two cameras, the first camera 103 and the second camera 104, images from a single camera or three or more cameras may be used.

[0047] The posture estimation unit 122 estimates the posture of the person using images from the cameras (first camera 103 and second camera 104), but the posture may also be estimated using the cameras (first camera 103 and second camera 104) and the LIDAR device 105. In this case, it is preferable to install the LIDAR device 105 at a position lower than the stalk transport height of the feed chain 4B (the height of the stalk supply port formed on the side of the thresher 4) so ​​that distance measurement can be performed to the left side of the machine body. In other words, if the LIDAR device 105 is installed near the stalk transport height, the transported stalks may interfere with distance measurement, and it may be difficult to obtain distance measurement data necessary for estimating the posture of the person. The LIDAR device 105 can also be attached to the reaper 3, but it is necessary to consider that its relative position to the thresher 4 changes due to its raising and lowering operation.

[0048] By installing the LIDAR device 105 as described above, the first camera 103 and the second camera 104 mainly acquire information about people above the height at which the stalks are transported, and the LIDAR device 105 mainly acquires information about people below the height at which the stalks are transported. As described above, the posture of the trunk is important for the posture estimation unit 122, and the posture of the trunk is estimated by determining the neck position through image analysis of the cameras (the first camera 103 and the second camera 104) and determining the waist based on the distance measurement results of the LIDAR device 105.

[0049] With this configuration, above the stalk transport height, the camera can estimate the position of key points such as the transported stalks and straw, which are difficult to estimate using electromagnetic wave ranging methods due to noise, while also obtaining other risk assessment information such as clothing.Below the stalk transport height, accurate information on the posture of people in the camera's blind spot can be obtained, and the relative distance to the threshing device 4 can be accurately measured.

[0050] Furthermore, the LIDAR device 105 is used for posture determination by the posture estimation unit 122, but apart from the processing by the posture estimation unit 122, when an object is detected approaching in close range or approaching at high speed, the work state estimation device 120 can also output a countermeasure operation command without going through the proximity determination unit 124 or the operation determination unit 125. In this case, it is possible to quickly issue a countermeasure operation command with less delay due to information processing load, thereby improving worker safety.

[0051] 4 Threshing device 4B Feed chain 103 First camera (camera) 104 Second camera (camera) 105 LIDAR device (ranging device) 120 Work state estimation device 122 Attitude estimation unit

Claims

1. A combine harvester comprising at least one camera (103, 104) for monitoring the manual threshing work of a worker, and a work state estimation device (120) for evaluating the degree of danger of the manual threshing work from images captured by the cameras (103, 104), wherein the work state estimation device (120) comprises a posture estimation unit (122) for outputting coordinate information of the worker in the images, and evaluates the degree of danger of the manual threshing work based on the coordinate information of the worker output by the posture estimation unit (122), and the coordinate information includes coordinates corresponding to any of the shoulders, elbows, head, neck, and waist.

2. A combine harvester as claimed in claim 1, further comprising a distance measuring device (105), wherein the work state estimation device (120) is configured to evaluate the risk of manual threshing work from distance information of an object obtained by the distance measuring device (105).

3. A combine harvester as described in claim 1 or 2, characterized in that the work state estimation device (120) has a proximity determination unit (124) that evaluates the distance between a pre-defined danger area within the coordinate system of the imaging area and the position of the worker's torso based on the worker's coordinate information output by the posture estimation unit (122).

4. A combine harvester as described in claim 2, wherein the cameras (103, 104) are positioned above the stump transport height of a feed chain (4B) that supplies stumps to the threshing device (4), and the distance measuring device (105) is positioned below the stump transport height of the feed chain (4B).

5. A combine harvester as described in claim 2, wherein the work state estimation device (120) is capable of evaluating the risk of manual threshing work using the estimation results of the posture estimation unit (122) and evaluating the risk of manual threshing work without using the estimation results of the posture estimation unit (122).

Images