Robot work vehicle

Abstract

To prevent an accident by allowing an unmanned robot work vehicle to accurately distinguish a human in a field.SOLUTION: An imaging device is made to have a function that can measure a distance to an object, as well as a movement and shape of the object, and a function that can distinguish a tint of the object, so as to analyze an image of the object using artificial intelligence to distinguish a human, and allow a work vehicle to perform a safe work to prevent an accident.SELECTED DRAWING: Figure 15

Description

[Technical field]

[0001] The present invention relates to a work vehicle that determines whether a person is present through image analysis of an imaging device, thereby driving the vehicle and operating a work machine. [Background technology]

[0002] Robotic machinery and remote-controlled driving technology are desirable for agricultural and construction machinery, but there are cases where people approach work vehicles in the field, making safety measures necessary. In particular, the ability to distinguish between humans is important. (Patent Document 1) [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2022-123742 Summary of the Invention [Problem to be solved by the invention]

[0004] Conventional technology includes a technique for controlling image recognition of specific work clothes in a specific location and issuing an audible alarm to alert the wearer.

[0005] However, unless the location and clothing are limited, it is difficult to determine that a moving object is a human. The present invention aims to provide a work vehicle that can use a neural network to identify humans in a field based on the position of the human face, the movement of the limbs, and the color of the human body, and to control it safely. [Means for solving the problem]

[0006] The first aspect of the present invention is achieved by the following technical means.

[0007] The imaging device has a function of measuring the shape, movement and position of an object based on the distribution of the object, and a function of recognizing color, and when a position in a farm field that can be determined to be skin color is at a specified height from the ground of the farm field, it is compared with human-shaped data that is determined to be a human.

[0008] The second aspect of the present invention is achieved by the following technical means.

[0009] In identifying whether an imaged human-like shape is a human, the system is equipped with a size detection unit that detects the overall height and width of the human-like shape, the position and amplitude of the parts corresponding to the limbs, and the position and amplitude of the position corresponding to the head, and a color detection unit that detects the position of skin tones and the color of the parts corresponding to the limbs and detects the relative sensitivity of each part, and the system changes the state of the detection data from the size detection unit and color detection unit, and uses a neural network to perform image analysis to compare the movement of the human-like shape, the movement of the limbs of the human-like shape, and the amount of movement of the object itself and the amount of movement of the limbs, and compares the registered data with the state change data analyzed from the imaging device to identify whether it is a human.

[0010] The third aspect of the present invention is achieved by the following technical means.

[0011] The imaging device is provided with a lighting device near a device having a function of recognizing colors, and the color tone emitted from the lighting device can be modulated. When image matching between the detection data of the size detection unit and the color tone detection unit cannot be achieved, the modulation of the color tone emitted from the lighting device and the determination of the masking color are changed.

[0012] The fourth aspect of the present invention is achieved by the following technical means.

[0013] If the captured human-like shape is determined to be a human, and the distance between the human-like shape and the vehicle is less than a predetermined value α, the work equipment is stopped, driving is halted, and the acquired information is transmitted to the cloud system; if the distance between the human-like shape and the vehicle becomes shorter and falls below a predetermined distance β, the function of the vehicle operating unit is cut off and operations are only accepted via external communications.

[0014] The fifth aspect of the present invention is achieved by the following technical means.

[0015] An imaging device having a color recognition function is positioned below the lighting device, and a satellite positioning device is positioned above the imaging device and the lighting device, with both devices being provided in front of or further forward than the front of the vehicle operating unit, and positioned above or below the vehicle operating unit in a position that is not within the field of view of the vehicle operating unit, and facing parallel to the ground or diagonally downward. Effect of the Invention

[0016] According to the first aspect of the present invention, a configuration can be adopted in which a human-like shape in a farm field is determined.

[0017] According to the second aspect of the present invention, it is possible to determine whether a human-like shape in a field is a human or not by the action of a neural network.

[0018] According to the third aspect of the present invention, by taking measures when the detected data of the size detector and the color detector do not match, erroneous detection can be prevented.

[0019] The fourth invention enables safety control when the distance between a human and a work vehicle is within a danger range, and also enables response to accidents such as theft.

[0020] According to the fifth aspect of the present invention, it is possible to detect the placement position of the imaging device with higher accuracy. [Brief description of the drawings]

[0021] [Figure 1] FIG. 1 is an overall perspective view of a work vehicle according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a left side view of a work vehicle according to an embodiment of the present invention. [Diagram 3] FIG. 1 is a perspective view of the periphery of an imaging device of a work vehicle according to an embodiment of the present invention; [Figure 4] FIG. 1 is a top view of the vicinity of an imaging device of a work vehicle according to an embodiment of the present invention; [Diagram 5] Skin color matching diagram for human-shaped shapes in the conversion processing of captured images of the present invention [Figure 6]Analytical diagram of a human-shaped body in the conversion process of a captured image of the present invention [Figure 7] A linear transformation diagram of a human-shaped shape in the transformation process of a captured image of the present invention. [Figure 8] Amplitude diagram of a human-shaped arm in the conversion process of a captured image of the present invention [Figure 9] Foot amplitude diagram of a human-shaped object in the conversion process of a captured image according to the present invention [Figure 10] FIG. 1 is a diagram showing a comparison of a human-shaped face in the conversion process of a captured image according to the present invention; [Figure 11] A diagram showing the relationship between the color of the face of the human-shaped body and the height above ground according to the present invention. [Figure 12] Relationship between amplitude and ground height of the human-shaped object of the present invention [Figure 13] A diagram showing the relationship between the movement of the main body and limbs of the human-shaped device of the present invention. [Figure 14] FIG. 1 is a diagram showing the relationship between the reflectance of the light amount of the lighting device of the present invention and the height of a human-shaped figure. [Figure 15] A block diagram of human judgment and work process change by shape change analysis using neural network in an embodiment of the present invention. [Figure 16] FIG. 13 is a diagram showing another embodiment of the present invention in which the imaging device is disposed on the work machine when the work machine is located below the machine body. [Figure 17] FIG. 13 is a diagram showing another embodiment of the present invention in which the imaging device is disposed on the front frame when the work machine is located below the machine body. [Figure 18] FIG. 13 is a diagram showing another embodiment of the present invention in which an imaging device is disposed in the center of the cabin. [Figure 19] FIG. 13 is a diagram showing another embodiment of the present invention in which imaging devices are arranged at the four corners of the cabin. [Figure 20] FIG. 13 is a diagram showing an imaging device disposed on a safety frame in another embodiment of the present invention. [Figure 21] FIG. 13 is a diagram showing another embodiment of the present invention in which an imaging device is disposed on a top link. [Figure 22] A diagram showing an arrangement of a millimeter wave radar and an imaging device according to another embodiment of the present invention. [Diagram 23] FIG. 13 is a diagram showing another embodiment of the present invention in which an imaging device is disposed below a spare seedling stand. [Figure 24]FIG. 13 is a diagram showing an arrangement of an imaging device for checking the working state of a work machine in another embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] The present invention will now be described with reference to the embodiments shown in the drawings.

[0023] The work vehicle and related part diagrams shown in Figs. 1 to 15 show an example of this embodiment.

[0024] The background of the present invention will now be described.

[0025] In response to the labor shortage in agriculture, unmanned agricultural machinery, so-called robotic work vehicles, are being used. These robotic work vehicles are operated under the supervision of a human in a remote location. The main reason for the human supervision is to prevent accidents such as a human approaching the work vehicle and being caught in the vehicle during work.

[0026] To deal with this, a system is required in which the robotic work vehicle itself can identify humans and perform safe control. This invention uses an imaging device to perform accurate analysis of people and the robotic work vehicle itself performs safe control, thereby realizing an unmanned work vehicle.

[0027] The configuration of the work vehicle of the present invention will be described.

[0028] The present invention relates to the configuration and control of an imaging device that is generally provided on vehicles that travel and perform work in autonomous agricultural machinery, such as tractors that are attached with work implements for tilling fields, rice planters that plant rice and vegetable seedlings, vegetable transplanters, combines that harvest rice and vegetables, and dedicated vegetable harvesters.

[0029] The configuration of the tractor 100 in Fig. 1 will be described as the work vehicle to be described. As an agricultural work vehicle and a robot vehicle that pulls a work machine, the tractor 100 is equipped with a satellite positioning device 111 and an inertial positioning device 112. Below the device that confirms the vehicle's position, an imaging device, LIDAR (Light Detection and Ranging) 120, is equipped. This is one of the remote sensing technologies that uses light, and it is possible to analyze the shape and properties of an object by measuring scattered light in response to laser irradiation that emits pulses. It is also possible to measure the distance to an object, and it is possible to measure distances from long distances to short distances and the moving speed of an object.

[0030] The imaging device is provided at the center of the front part of the vehicle body. A left light 140 and a right light 150 are provided at a position behind the vehicle body with respect to the position of the CCD camera 130. As shown in Figs. 2, 3 and 4, the left light 140 and the right light 150 are positioned above the CCD camera 130 and at a rear position in the traveling direction of the vehicle. They are located at a distance 142 behind. Therefore, even if the CCD camera 130 has a wide-angle imaging range 131, the illumination range of the left light 140 is 141 and the illumination range of the right light 150 is 151, so it is possible to sufficiently illuminate the imaging range 131.

[0031] In FIG. 4, the illumination range 151 is configured symmetrically such that the spread angle thereof is smaller than the line connecting the position of the end 132 of the CCD camera 130 and the right end 152 of the right lighting 150. As a result, the illumination ranges 141 and 151 do not directly illuminate the lens position of the CCD camera 130, so that no halation or the like occurs and the color of the imaged object can be accurately acquired.

[0032] Furthermore, the CCD camera 130, which is an imaging device, is disposed below the left lighting 140 and the right lighting 150. This positional relationship also makes it possible to prevent halation.

[0033] It is well known that the positions of the CCD camera and lighting can be moved in the forward / backward, left / right, and up / down directions in this way, but it is important that the placement takes into consideration the design of the mobile device, and by locating the lighting unit in a recessed position, it is possible to achieve both functionality and design.

[0034] The satellite positioning device 111 is disposed above the imaging device 130 and the lighting devices, the left light 140 and the right light 150, all of which are provided forward of the operation seat 101, or in the form of an operation box, in front of the operation box. The satellite positioning device 111 may be disposed further forward of the operation seat 101 and the operation box, but is disposed above or below the operation seat 101 in a position that is not within the field of view of the operation seat 101, and is disposed parallel to the ground direction or diagonally downward. In the embodiment shown in FIG. 2, the satellite positioning device 111 is located in front of the cabin, which is the operation box.

[0035] A method for analyzing an imaged object using a combination of two types of imaging devices, a CCD camera 130 and a LIDAR 120, will be described.

[0036] 5 shows an image of a person (human-shaped), and the captured image 200 is an image of the CCD camera 130 showing the color and shape. Since the figure is not shown in color, it is shown by diagonal lines. Note that the relationship between the shape of the diagonal lines and the color is not particularly shown, and the difference in the diagonal lines means that the color has changed.

[0037] In this way, the work vehicle has basic data for recognizing humans registered in the data storage unit S15-20 in Fig. 15. The basic data has many registered shape images of human shapes, and has image data of humans walking in a field, standing upright, standing with legs spread apart in a natural manner, etc., and has data on the external shape, limb positions, head, amplitude ranges and colors of each part. In particular, skin color recognition of the head position is the basis for the human shape search of the present invention. The human shape may be controlled by roughly matching human shapes similar to the skin color recognition of the head position.

[0038] In human recognition, accurate judgment is possible if specific imaging conditions are set and the image is easy to analyze; however, the present invention relates to identifying humans in unexpected situations, so first the skin color at the head position of a human-like shape is recognized and the human shape is matched, focusing on the search target, and then imaging of that target is centrally performed and detailed analysis is performed; the detailed analysis will now be described.

[0039] The CCD camera 130 may be a monocular camera, but in order to perform accurate color recognition, the relationship with the lighting device is important as explained in the previous paragraph. In this image, captured image 201 detects a color tone corresponding to skin color, but the reflectance is medium to slightly high, and the object surface has fine irregularities, so the reflectance does not increase significantly. In contrast, captured images 202, 203, and 206 are of colors other than skin color, and compared to captured image 201, there are areas with higher and lower reflectance. Captured images 204 and 205 may recognize skin color or may detect other colors.

[0040] The captured image 210 is an image of the same object captured by the LIDAR 120. The image is point cloud data. In this figure, the points are displayed as large circles for easy understanding, but in the actual image, they are fine dots, and the point cloud shape can also be configured to have a smooth shape.

[0041] Shape recognition on a two-dimensional screen is almost the same as planar recognition by the CCD camera 130, but since the distance between the object and the LIDAR 120 can be calculated, it is possible to estimate the three-dimensional shape of the captured image 210. In addition, by detecting the movement of the object through point cloud movement detection, the detailed three-dimensional shape of the captured image 210 can be determined.

[0042] Therefore, by combining the captured image 200 and the captured image 210, the captured image 220 can be synthesized. The feature of the synthesized image is that it allows the stereoscopic image to have a color tone, and by analyzing the partial movement, the parts of the object that move instantly and the parts that are stably fixed can be identified. Also, from the way the object moves in a link-like manner, the parts that serve as the fulcrum of the link can be determined. By analyzing the difference in the speed of this movement and the link configuration, the main body of the object and the branches and leaves can be distinguished. For example, in the case of a human, the torso, hands, feet, and head can be distinguished. In the captured image 220, only the skin color is extracted and the point cloud data is subjected to color recognition. The captured image 221 is the part for which the skin color is recognized. In this way, the captured image 201 of the CCD camera is synthesized with the captured image 211 by the LIDAR to become the captured image 221. The tendency of this image analysis is the same for each part, such as the captured image 206 ⇒ 216 ⇒ 226 and the captured image 205 ⇒ 215 ⇒ 225. This image analysis method is a method for analyzing image data, corresponds to the first state change, and is the control section of S15-21 in FIG.

[0043] With regard to the recognition of human-like shapes, priority is given to detecting the skin-colored position from the field in the images captured by the LIDAR 120 and the CCD camera 130. If the skin-colored position is at a specified height from the ground in the field and the corresponding object is a moving object, it is analyzed as a human-like shape. If multiple human-like shapes are recognized, a method is adopted in which each individual is analyzed.

[0044] Next, in FIG. 6, the flow of analyzing the movement of an object from the movement of an image will be described. The left diagram in FIG. 6 is the same as the right diagram in FIG. 5, and is the captured image 220. The case where the captured image 220 moves and becomes the captured image 220A will be described. The total height of the object 230 ⇒ total height 230A, but there is no difference in the dimensions. Similarly, the height of the skin-colored part 231 ⇒ 231A, there is no difference in the dimensions. However, there is a difference in the dimensions between the dimension 232 ⇒ 232A and the dimension 233 ⇒ 233A. This difference is analyzed to be a change, even if it becomes longer or shorter. However, since the total height of the object does not change, it is possible to analyze from the change state of the first state change that the dimensions 232 and 233 are the branches and leaves of the object, and correspond to the hands and feet of a person.

[0045] By narrowing down the LIDAR images, it is also possible to change the point cloud data into linear data. Captured image 220B in Fig. 7 is a diagram obtained by linearly converting captured image 220. Similarly, captured image 220A is a diagram obtained by linearly converting captured image 220C. By converting either of them into a linear shape, the main body part and the peripheral part can be more clearly distinguished.

[0046] Dimension 232B changes to dimension 232C as the mounting angle changes from dimension 234B to dimension 234C. Dimension 233B changes to dimension 233C as the mounting angle changes from dimension 235B to dimension 235C. However, since dimensions 230B and 230C, which are the overall height, do not change, it is presumed that dimensions 232B and 233B are parts of branches and leaves that involve movement, and since there is little change or movement in the flesh-colored parts between dimensions 231B and 231C, it is presumed that captured image 220 is a human.

[0047] In Figure 8, we verify the judgement of a human arm. In the captured image 220D, the dimension 232D moves very fast. It moves smoothly and at a fast speed between 8S1 to 8S5. In contrast, the movement of 8S7 to 8S12 is also confirmed. This is because the movement of the dimension 232E, which is assumed to be the foot in Figure 9, moves more frequently and at a faster speed than the movement of 9S1 to 9S4.

[0048] In this way, the technology of judging a person using an image capture device can judge the balance and movement of the eyes, nose, mouth, ears, etc. on the face, and the color of the face, but the image is taken in an easy-to-understand state so that the person can recognize the image. However, there is no way to take an image in an easy-to-understand state for the purpose of preventing theft or accidents.

[0049] However, the work vehicle used in the present invention is for agricultural use, and is to be identified in fields such as rice fields. Therefore, human movement is often limited. When humans move in fields, they walk on two legs, and it is difficult to imagine them moving while sitting or crawling on all fours using their hands and feet. In houses, humans move in a variety of ways, such as sitting down, lying down, and using chairs and desks, so the range of conditions that change is wide, making it difficult to judge people. The difficult part of identifying humans in fields is wearing hats, gloves, boots, etc. In addition, wrapping a towel around the neck or wearing a mask can also make it difficult to identify the head, but walking in fields is often characteristic.

[0050] In the field, people do not sit on the soil, nor do they walk while crouching. It is difficult to maintain balance on soft fields, so people tend to walk with their arms and legs stretched out. This makes it easy to convert to Figures 5 to 7, and the arm position movements in Figure 8 and the leg movements in Figure 9 are often captured at the expected measurement positions, so image analysis and matching can be done with a small amount of calculation. Thus, in the field according to the present invention, human movement is limited, and analysis from captured images is possible.

[0051] In this way, when the imaging device is equipped with a LIDAR device and a camera that recognizes color, and the position that can be determined to be skin color from the ground is at a certain height, it is clear that it is easy to match a human shape.

[0052] 10, 11, 12, and 13 explain the control to deal with the misrecognition of animals other than humans. As described above, in the field, it is difficult to distinguish the head because people wear hats, gloves, boots, etc., towels wrapped around the neck, and masks. However, in the case of humans, since there is movement, it is possible to recognize the skin-colored parts by capturing an image from a certain direction. In FIG. 10, the dimensional ratio of the position of the dimension 231F and the ratio of the dimension 241 itself are calculated for the part corresponding to the face with respect to the total height 230F. Also, since the dimension 241 also moves, the allowable range is also set as the dimension 242, and this is also judged by the ratio. In this way, regardless of the actual dimension of the total height 230F, the certainty that it is a human is confirmed by dealing with the ratio.

[0053] Figure 11 shows subtle differences in the color of skin tones in relation to RGB color shades and total height 230F. The X-axis shows the red hue of RGB, and the Y-axis shows the total height 230F. Although it differs depending on weather conditions and field conditions, under the same conditions, when the total height 230F is small (S11-1), or when a person is short, red may appear slightly stronger in the field due to the reflected light from the field and the image from the imaging device being received slightly downward. Conversely, when the person is tall (S11-2), blue appears slightly stronger, and these color corrections also allow for accurate analysis of the face and its position.

[0054] 12 shows the relationship between the height of an object and the amplitude of the movement of the object by superimposing images of the object taken successively at object heights 230, 230A, 230B, 230C, 230D, 230E, and 230F. The X-axis shows the height of the object, and the Y-axis shows the amplitude of the object. In this embodiment, the amplitude of the image has a large amplitude at the low position 12S-1 and the intermediate position S12S-2. 12S-1 corresponds to the position of a person's foot, and 12S-2 corresponds to the position of a hand.

[0055] Furthermore, 12S-3 corresponds to the human torso, and 12S-4 corresponds to the human head, and the degree of amplitude is low. By synthesizing the outline of an object in this way, the amplitude of a specific part is large, and if the positions match, it is possible to estimate that it may be a human. In this embodiment, the solid line indicates the registered reference amplitude value, and the measured value is displayed as a two-dot chain line. The dashed line indicates a large quadrupedal animal, and it can be seen that the tendency is different. The analysis matching is handled by mathematical means, and if the matching rate is above a certain level, it is determined that it is a match.

[0056] Such human judgment is a motion judgment based on the specific actions taken by humans in the field, as described above. In the field, the field is soft after plowing, plowing, or immediately after rice planting, and in the case of bipedal walking, the arms and legs are often stretched out, so the accuracy of image judgment is high. However, on a general asphalt pavement, the footing is stable, and people often run with their arms bent or carry luggage in their hands, making it difficult to determine whether they are humans, and control that mainly recognizes them as moving objects is immediate. However, there are different requirements for recognizing people in the field. In emergency responses other than accidents involving people being caught during work, it is necessary to prevent theft and vandalism, and it is necessary to recognize people with high accuracy, not just moving objects.

[0057] For human judgment, it is necessary to increase the degree of external shape matching. This is the analysis of the amount of movement of the limbs mentioned above. The degree of amplitude can be the amount of movement of a simple external image, or it can be measured by measuring the movement of a specific color. In addition, after filtering out parts other than the moving parts, specific parts can be matched using a convolutional neural network and calculated from the degree of overlap. The basic human shapes to be compared are registered as cloud data, and can be handled using ICT technology on mobile devices to reduce the load on the CPU of the work vehicle.

[0058] In both methods, the center position and head position of the humanoid shape are extracted from the image, and the relative movement amount and absolute movement amount with respect to the limbs are calculated to improve the recognition of the humanoid shape, thereby performing image control to improve the accuracy of determining the human. This content will be explained with reference to FIG. 13. The same humanoid shape is continuously imaged, and after a predetermined time has passed, humanoid shape 220G moves to humanoid shape 220H. When the amount of movement is measured at the center of the humanoid shape in this image, dimension 251 is detected. Also, a movement amount of dimension 252 is detected for a part estimated to be the head of the humanoid shape. In either case, the amount of movement that is considered to be the main body of the humanoid shape is small. However, compared to that amount of movement, the amount of movement that is estimated to be the left and right arms is detected as dimension 253 and dimension 254. Furthermore, the amount of movement that is estimated to be the left and right feet is dimension 256 and dimension 257. In particular, the amount of movement of dimension 257 of the stepping out on the moving side of the human body is large. In this way, dimensions 251 and 252 are considered to be the amount of movement of the main body of the humanoid shape, and are absolute movement amounts. In comparison, dimensions 253, 254, 256, and 257 are the distance of movement including the amount of movement of the main body, and are the amount of relative movement. In this case, dimensions 253, 254, 256, and 257 are larger than dimensions 251 and 252, which are the amount of movement of the main body, and the parts of dimensions 253, 254, 256, and 257 are presumed to be human limbs, and since dimension 251 of the center of the main body is much smaller than dimension 252, dimension 252 is presumed to be the head, and it is determined that there is a high possibility that human-like shapes 220G and 220H are human.

[0059] Based on such technology, the fourth invention performs safety control for both the target object and the work vehicle. If it is determined that the target is a human, it is very dangerous for the target object to approach the work vehicle. If the distance to the work vehicle falls below a certain distance α, the work equipment is stopped if one is attached, and the vehicle is then stopped from traveling. In addition, information on the emergency stop is immediately uploaded to the cloud, allowing users to share the information.

[0060] After that, if the human-shaped figure approaches within the specified distance β even though the vehicle has stopped, this is a dangerous act and may be a prank or theft. In this case, manual operation of the vehicle is blocked. Simultaneous operation is blocked and only external communication operations are accepted, i.e., only remote operation is accepted.

[0061] If the operating seat is a cabin type, it is also possible to accommodate mechanical locks such as locking the doors, locking the steering wheel, and fixing the operating levers.

[0062] The values ​​of α and β can be changed depending on the vehicle model and the work speed. α is about 4m and β is about 2m, and can be set arbitrarily, but it is recommended to set upper and lower limits, and even if the safety control can be cut off, there is an automatic return function after a certain time has passed.

[0063] In determining whether a person has a humanoid shape, in addition to the color tone and the amount of movement (amplitude), the amount of light reflection described above is also used as a criterion. The amount of reflection is indicated by the relative sensitivity of the CCD camera 130, and is calculated by analyzing a specific wavelength when irradiated with near-infrared light. The left lighting 140 and right lighting 150 are composed of LED lights, and the color tone is modulated by changing the frequency or wavelength, thereby correcting and controlling the relative sensitivity of the CCD camera 130. When working in a farm field, the desired amount of light or wavelength of light may not be obtained depending on the working time or weather, so the left lighting 140 and right lighting 150 control the LED lights to correspond to the amount of light and wavelength of light that are the reference for the CCD camera 130.

[0064] As described above, captured image 201 detects a color tone corresponding to skin color, but the reflectance is medium to slightly high, and the object surface has fine irregularities, so the reflectance does not increase significantly. In contrast, captured images 202, 203, and 206 are of colors other than skin color, and compared to captured image 201, there are areas with higher reflectance and areas with lower reflectance. The areas with higher reflectance include boots, raincoats, hats, etc. made of resin or enamel. Conversely, the areas with lower reflectance are clothing made of materials such as cotton. Clothing made of synthetic fibers has a higher reflectance than cotton, but compared to the reflectance of skin, it is still low enough to be distinguished.

[0065] The relative sensitivity of the CCD camera 130 corresponds to the reflectance at a specific wavelength. Therefore, it is possible to identify human skin by extracting the intermediate value of the relative sensitivity. Identification of this intermediate value of relative sensitivity must be done for each wavelength, and it is possible to identify it in the work vehicle by registering it in advance in cloud data and performing a comparison calculation.

[0066] FIG. 14 shows the relationship between the reflectance of the light amount of the lighting device and the height of the human figure. The X-axis is the reflectance of the light amount of the lighting device, which is the same as the relative sensitivity of the CCD camera 130. The Y-axis is the height of the human figure. S14-1 is the shoe part. In the case of boots, the reflectance is extremely high. S14-2 is the lower body part, but the reflectance is low due to the clothing. S14-3 is the upper body and face part. The face part has an intermediate reflectance, but if the chest is open or the clothing is short sleeved, it may show a high reflectance. S14-4 is the hair part. If the hair is long and black, the reflectance is extremely low. Straw hats and the like have a rough surface, so they are likely to show the same tendency. S14-5 is the top edge of the outline, and may be detected as high depending on the angle of the sunlight.

[0067] Although it is technically possible to use the reflectance of an object to identify a human being in this way, unlike the conventional technology of JP 2022-123742 A, the condition is not limited to identifying a human being wearing the same clothes on the premises of a factory or the like. Therefore, in a farm field, if a person wears sunglasses or a black mask, has long black hair and does not wear a hat, the reflectance decreases, which may result in misrecognition. Therefore, if a person is recognized as a human being, it is necessary to issue a warning instruction by voice as a safety control. For example, if a voice saying "It is dangerous. Do not approach the vehicle. It may suddenly accelerate" is played, it is expected that a human will respond to the voice.

[0068] In another embodiment, a radiation thermometer may be added to detect the temperature of an object to allow estimation of the presence of a human.

[0069] Furthermore, as mentioned above, by modulating the color tone by changing the frequency or wavelength of the lighting device, it is possible to detect even when language cannot be recognized, and even if a large animal other than a human is recognized, it is possible to perform safety control by adjusting the lighting device accordingly.

[0070] The control of identifying humans from images captured by the imaging device described above is possible using conventional analysis technology, but doing this inside a work vehicle would be difficult as it would require calculations and analysis solely by the CPU responsible for controlling the vehicle.

[0071] In other words, capturing a large number of images and searching and matching them on a large number of cloud servers places a burden on the CPU and takes time, making it impossible to use for urgent analysis. To solve this problem, it is necessary to control the work machine to make the inferred judgment of humans alone, and it is also necessary to have a function that can learn the criteria for making such judgments.

[0072] By using a neural network, an artificial intelligence, as a means of identifying humans from image data, the accuracy can be further improved. This process is explained in the block control diagram of human identification and work process changes through shape change analysis using a neural network in Figure 15.

[0073] The function of measuring the shape, movement and position of an object based on the distribution of the object in the first invention is the LIDAR 120, which is the first imaging device S15-1, and the function of recognizing the color is the CCD camera 130, which is the second imaging device S15-7. By combining the images from both imaging devices, it is possible to determine whether the captured object is a moving object or a fixed object, and to detect specific movements from the size, shape and color of the object.

[0074] Although human detection is important, since humans wear various clothes, human detection can be estimated by color and shape, but it is not decisive. Therefore, in the present invention, limited control is performed in a specific location, namely, a farm field. As described above, when a human moves in a farm field, the human is walking on two legs, and the positions of the face and limbs are in a limited range of positions. In reality, the work vehicle works in the farm field, and the position in the farm field is confirmed by a satellite positioning device and controlled. As described above, the overall height and skin color position of the human-shaped shape can be detected from the position of the ground in the farm field using image data from the LIDAR 120 and the CCD camera 130. This enables the work vehicle to roughly determine whether it is a human.

[0075] The second invention further analyzes the human shape.

[0076] The first imaging device S15-1 performs analysis in the size detection unit S15-2 of the humanoid shape by combining the image captured by the LIDAR 120, data from the obstacle sensor 23, and data from various sensors of the vehicle not shown in this block diagram. A first state change S15-21 process is performed to analyze the humanoid shape through the strength of the connections (synapses) of each data by a neural network. The analysis contents are calculated as follows: the height S15-3 of the entire humanoid shape, the width S15-4 of the entire humanoid shape, the position and amplitude S15-5 of the limbs equivalent, and the position and amplitude S15-6 of the head equivalent. This state corresponds to the captured image 210 in FIG. 5.

[0077] The second imaging device S15-7 is a CCD camera 130 that recognizes color and shape, and performs a color detection unit S15-8 for human-shaped shapes. The captured image and data from various sensors of the vehicle (not shown in this block diagram) are combined, and a first state change S15-21 process is performed to analyze the human-shaped shape through the strength of the connections (synapses) of each data by a neural network. By changing the state of the image, which is the analysis work, the position of the skin color system S15-9, the color corresponding to the limbs S15-10, the masking color judgment S15-11, and the detection of relative sensitivity S15-12 are calculated. This state corresponds to the captured image 220 in FIG. 5. Note that the masking color judgment S15-11 is initially set to a standard setting value, but has a function of automatically changing the setting value while learning in the third invention.

[0078] By combining the state-changed images from the humanoid size detection unit S15-2 and the humanoid color detection unit S15-8, the combined image is further analyzed in the second state change S15-22. This image analysis process is the image change process shown in Figures 6 and 7. At this stage, it is possible to recognize and infer a humanoid human, but further image analysis of the limb amplitude is performed. The basic data in the data storage unit S15-20 is collated with the state-changed image to determine whether the person is a human S15-14.

[0079] This image analysis process is an analysis method that uses a neural network to perform image analysis of the movements of the humanoid shape, the movements of the limbs of the humanoid shape, and a comparison calculation between the amount of movement of the object itself and the amount of movement of the limbs, and identifies whether it is a human by matching the registered data with the state change data analyzed from the imaging device, and is judged by the image analysis processes in Figures 8, 9, 10, 11, 12, 13, and 14.

[0080] In this way, the data obtained from the imaging device and various sensor data are taken into account, the image state is changed repeatedly, and the data is compared with the data storage unit, the differences are registered, and the process of using them for the next data analysis is repeated. The means of analysis used is an artificial intelligence circuit, or so-called neural network analysis method.

[0081] In using the neural network of the present invention, the series of steps is explained using the block diagram in Figure 15, and the analysis method by changing the state of each image is described above in Figures 5 to 14. This neural network makes it possible to identify humans from captured images.

[0082] Furthermore, by utilizing the data of the neural network of the present invention, the above-mentioned emergency stop judgment S15-15 and anti-theft judgment S15-16 are performed. Also, there are cases where the human recognition is incorrect, and the erroneous recognition estimation S15-17 is always performed in the background control. For example, if the image is recognized as a human, but the image data captured after a predetermined time has elapsed after the emergency stop does not show any limb movements that would be recognized as a human, and it is determined that another animal has been mistakenly detected, the operation restart plan S15-41 is made and operation is restarted.

[0083] The learning function of this neural network is to automatically determine the function of the work process change and decision unit S15-30. Although it is also possible to manually respond by receiving instructions from a mobile terminal or remote control device S15-31, in the content of the present invention, the work process plan S15-37, operation resumption plan S15-41, and vehicle safety control S15-45 are performed by changing the weighting of information based on the learning degree from data from the cloud S15-34, the satellite positioning device S15-32, and the above-mentioned status change information S15-14 to S15-17.

[0084] In one example, in the driving route S15-38, a route is automatically created that does not pass through the location where a human was found, and this location is registered in map control S15-40. From the next time onwards, a route is created that does not allow the robot to pass through this location, thereby making the user aware of the situation.

[0085] In addition, the control timing and accuracy of S15-42 to S15-49 are further adjusted by the learning function using the functions described above.

[0086] The basic function of the third invention will be described.

[0087] There are cases where the image data detected by the humanoid shape size detection unit S15-2 and the humanoid shape color detection unit S15-8 do not match. In the captured image 220 of Fig. 5, the image of the LIDAR 120 of the part corresponding to the face and the skin color part of the CCD camera 130 are misaligned and do not match. This state may occur due to differences in the illuminance of external light, or when the reference color for color recognition is misaligned due to reflected light in the field, or when the background colors such as the color of the soil in the field and the color of the sky are similar and identification becomes unclear, or when the humanoid shape moves quickly.

[0088] To deal with this, the imaging device is provided with a lighting device near the device with the function of recognizing color, and the color tone emitted from the lighting device can be modulated. In this embodiment, the color tone of the right lighting device 140 and the left lighting device 150 can be modulated. When the image detection data of the human-shaped size detection unit S15-2 and the human-shaped color tone detection unit S15-8 cannot be matched, the modulation of the color tone emitted from the lighting device and the masking color determination S15-11 are changed to deal with this. Note that the masking color determination S15-11 has a learning function, and by registering data in the farm field map, if a change is required depending on a specific time and place, the lighting modulation and masking color determination are automatically changed, and the situation is automatically dealt with in future uses.

[0089] In this way, the function described in FIG. 15 can combine images from the imaging device and perform analysis using the calculation function in the CPU of the work vehicle, so even if a person suddenly appears in front of the work vehicle, a processing decision can be made immediately. In the conventional technology, even if a person suddenly appears, the only recognition possible is obstacle detection, so an emergency stop of the vehicle is possible, but processing such as stopping the work equipment or responding to theft cannot be performed. Analysis and matching of images through communication with the cloud takes time, so it is difficult to immediately anticipate a person with conventional control. In the present invention, by providing artificial intelligence functions, identification and judgment can be performed within the vehicle, making it possible to respond to an emergency.

[0090] However, this detection is limited to agricultural work and fields. However, since the images were not taken under conditions where the robots wanted to recognize themselves, the conditions for analysis are not easy. To address this issue, the imaging device is equipped with a LIDAR device and a camera that can recognize colors, the colors emitted from the lighting device are modulated, and the walking conditions of people in the fields are registered as a database for comparison, allowing for immediate analysis to determine whether they are human, although this is a guess under limited conditions.

[0091] However, even when matching skin color, there are differences in imaging conditions and individuals, and even creating a database through a large amount of learning may not be enough to cover everything. Also, although there is a function to change the matching criteria and tolerance range, there is a high possibility of false recognition in situations where the filtering function is turned off.

[0092] Considering such condition analysis, increasing the accuracy of the judgment criteria for a specific element may result in the inability to make judgment analysis or in the possibility of misrecognition. The invention that solves this problem is the content of the present invention, which makes a comprehensive judgment by measuring and analyzing various elements. In other words, it is not a method of detecting a single condition with high accuracy, but a method of increasing accuracy by combining various conditions in a cumulative manner.

[0093] It is desirable to have technology that can combine a large number of conditions like this and process data in a way that is similar to human recognition and judgment. This technology relies heavily on the specific technology of artificial intelligence as in the present invention, and is realized by incorporating control that processes images and responds to changes and outputs in the work process in a way that is similar to human judgment.

[0094] It should be noted that, although this is a response to situations other than farm fields that fall outside the conditions of the first invention, the discrimination function of this neural network also works outside farm fields. However, when it comes to identifying humans, people are in various states, and the accuracy of identifying them as humans decreases. However, although it is not possible to recognize them as people, it is possible to recognize them as moving objects, so it is possible to respond by recognizing them as moving objects or obstacles using an imaging device and taking evasive action.

[0095] With this function, when entering a farm road from a field, the foreign object and obstacle detection function will stop the vehicle. Since farm machinery is operating in the field, there is a risk of being caught in the machinery, so the accuracy of human identification is necessary, but in other cases, accidents can often be avoided by stopping the vehicle.

[0096] In addition, on farm roads and the like, driving speeds are fast and theft while moving is difficult, but when driving at low speeds in fields, it is possible for someone to get into the work vehicle, so human recognition is necessary to deal with theft.

[0097] Next, the mounting positions of the imaging device and satellite positioning device of the present invention in different models will be described.

[0098] 16 shows a model in which the working section is mounted in the center of the vehicle body, on the abdomen, and a working machine 282 such as a mower is disposed below the vehicle body. In this model, a stay is extended from below the power frame in a position that is not affected by the movement of the working machine, and an imaging device 291 is provided in front of the working machine and above the uppermost position of the working machine, so that the area in which the working machine will work can be imaged in advance, and in the case of a mower, the image can be used to obtain information such as the height of the grass, the mowing direction and height depending on the direction of the grass, and power control.

[0099] Fig. 17 shows a different form from Fig. 16, in which the imaging device is disposed in the center of the vehicle body in front of the front axle 301. In Fig. 16, the imaging device is positioned low to match the height of the work equipment, and imaging horizontal to the ground is possible, but changing the height of the work equipment can make it difficult to measure distances accurately. In Fig. 17, however, the imaging device 302 is disposed as low as possible in a location where the position can be fixed, making it possible to capture images horizontal to the ground under conditions where distances can be measured accurately.

[0100] Fig. 18 shows the case of imaging from a high position. In order to mount the imaging device on the top in a stable position, it is recommended to place it on top of the cabin 303 which is the operation room. Among these, when measuring while rotating 360 degrees like LIDAR 304, it is placed in one place in the center of the cabin.

[0101] Furthermore, since it is difficult to rotate the imaging device when taking continuous images, by arranging imaging devices 305 at the four corners of the cabin as shown in FIG. 19, continuous imaging in all directions is possible.

[0102] Figure 20 shows the safety frame specification. To ensure safety in the event of a vehicle tipping over, there is a safety frame 306 that protects the operator up to the head, and this frame is equipped with an imaging device 307. Since the safety frame has a folding function, the folding shape must be taken into consideration.

[0103] Figure 21 shows the mounting position of the rear image capture device on a model without a cabin. There is a link to which the work equipment is attached, and by placing the image capture device on this link, it is possible to capture images in accordance with the movement of the work equipment. In Figure 21, the image capture device 309 is placed on the top link 308.

[0104] FIG. 22 shows an arrangement in which the millimeter wave radar 260 is installed simultaneously with the imaging device 262. In agricultural machinery, millimeter waves can be used not only to measure the distance to an obstacle but also to detect the speed of the obstacle with high accuracy. Therefore, it is effective for detecting the movement of an obstacle at the edge of a field, cooperative work, and detecting crops in front. In terms of location, a lower front position is advantageous for crop detection, so it can be installed using a member 263 such as a frame for attaching and detaching the front weight. When the millimeter wave is provided at the lower front, the imaging device 262 is installed at a higher position than the millimeter wave radar so that it can be easily used to detect distant positions. In the imaging device 262, in order to deal with diffuse reflection of sunlight, rainwater, wind and snow, it is possible to accommodate it in a portion 261 that extends the ceiling part of the cabin and forms an eaves.

[0105] 23 shows the relationship between the satellite positioning device 272 and the imaging device 271 in the transplanter. The imaging device 272 is supported by a stay 273 of the satellite positioning device 272, and is installed at a lower position than the G satellite positioning device 272. Furthermore, if the transplanter has a control mechanism for detecting ridges or obstacles and stopping the vehicle, the imaging device 271 is installed under the spare seedling platform 272 and located behind the front end 273 of the spare seedling platform. This makes it easy to remove seedling frames, and the spare seedling platform 272 can protect the imaging device 271 in the event of a collision with an obstacle in front.

[0106] 24 is a diagram showing the position of an imaging device 281 for checking the condition of planted seedlings in a rice transplanter. The imaging device 281 is arranged at a position above the planting section 282, which is equivalent to the upper part of the seedling tank. This makes it possible to deal with mud splashes caused by the planting section 282, and also makes it easy to check the seedling tank. [Explanation of symbols]

[0107] 100 Tractor 120 Imaging device (LIDAR) 130 Imaging device (CCD camera) 140 Left lighting (LED light) 150 Right lighting (LED light) 200 captured images (CCD camera) 210 Images (LIDAR) 220 Captured image (composite) S15-21 First state change (neural network) S15-22 Second state change (neural network)

Claims

1. This work vehicle is equipped with an imaging device that has the function of measuring the shape, movement, and position of an object based on the distribution of the object, as well as the function of recognizing color, and has the function of matching a position in a field that can be determined to be skin color with human-shaped data that is determined to be a human when the position is at a specified height from the ground in the field.

2. a size detection unit that detects the height and width of the entire human shape, the position and amplitude of a portion corresponding to a limb, and the position and amplitude of a portion corresponding to a head in identifying whether an image of the human shape is a human; A color detection unit detects the position of the skin color system and the color of the limbs, and detects the relative sensitivity of each part. Change the state of the detection data of the size detection section and the color detection section, The work vehicle of claim 1, which uses a neural network to perform image analysis of the movements of a humanoid shape, the movements of the limbs of a humanoid shape, and a comparison calculation between the amount of movement of the object itself and the amount of movement of the limbs, and identifies humans by comparing registered data with state change data analyzed from an imaging device.

3. In the imaging device, a lighting device is provided near a device having a function of recognizing color, and the color emitted from the lighting device can be modulated. When the image of the detection data of the size detection unit and the color detection unit cannot be matched, 3. A work vehicle according to claim 1, further comprising a function for changing the modulation of the color tone emitted from the lighting device and the determination of the masking color.

4. If the captured human-like shape is determined to be a human, When the distance between the human-shaped shape and the vehicle is equal to or less than a predetermined value α, the work equipment is stopped, the vehicle stops traveling, and the acquired information is transmitted to the cloud system.

3. The work vehicle according to claim 1 or 2, wherein when the distance between the human-like shape and the vehicle becomes shorter and falls below a predetermined distance β, the function of the vehicle operation unit is cut off and only operations via external communication are accepted.

5. A work vehicle as claimed in claim 1 or claim 2, wherein an imaging device having a color recognition function is arranged below a lighting device, and a satellite positioning device is arranged above a imaging device and a lighting device, and both devices are provided in front of or further forward than the front of the vehicle operating unit, in a position that is not within the field of view of the vehicle operating unit, and are arranged above or below the vehicle operating unit, and are arranged parallel to the ground or diagonally downward.

Images