Garbage collection truck using object detection system based on deep learning etc.

Abstract

To achieve load mitigation of prior work and expansion of a detection object by performing object recognition and attitude recognition.SOLUTION: An object detection system 1 for detecting an object in an image, includes an object recognition unit 2 which recognizes etc. a person in an image as a real person, and an attitude recognition unit 3 which recognizes an attitude of a person. The object recognition unit 2 recognizes a position of a total person on the basis of a contour of the person in the image. The attitude recognition unit 3 may recognize a position of a front part of a prescribed joint of a person, and includes an object tracking unit 4 for tracking movement or the like of the person. The object tracking unit 4 may recognize a movement route of the total person and the front part of the prescribed joint. A garbage collection vehicle 10 using the object detection system 1 stops raking of a plate 13 when a part of a person enters into an input port 12 or a total person enters into a prescribed area. When a shielding object S occupies a first ratio α or more in the image, it may be determined that the object detection system 1 cannot perform detection, or the shielding object S may be a corrugated cardboard B' or an umbrella B''.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an object detection system that detects an object in an image captured by a camera, and a garbage collection truck using the same. [Background technology]

[0002] 2. Description of the Related Art Conventionally, there is known a person recognition device that uses a camera to capture an image of the vicinity of a refuse inlet of a refuse collection vehicle, and recognizes a person in the vicinity of the refuse inlet based on the image data (see Patent Document 1). This person recognition device comprises a person identification unit that detects color information of an object image in an image captured by the camera and compares the detected color information with a detection target color registered in a memory unit for an item attached to the person's hand or foot to identify whether the object image is a person's hand or foot, a color information registration unit that registers in the memory unit color information of the item attached in an image previously captured by the camera as the detection target color to be used for identification by the person identification unit, and a person position determination unit that determines whether the object image identified as a hand or foot by the person identification unit is within a specified area near the garbage inlet. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-156564 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, the person recognition device described in Patent Document 1 requires that detected color information be compared with the detection target color registered in a memory unit for items worn on a person's hands or feet. However, since the colors of gloves, work clothes, etc. worn by workers in garbage collection trucks vary from one municipality or organization to another, the detection target color registered in the memory unit must be changed for each municipality, organization, etc., which increases the workload of having to change the detection target color in advance, etc. Furthermore, the human recognition device of Patent Document 1 has the problem that it cannot recognize the entry of a member of the public other than a worker into a specified danger area near the garbage dump entrance if the member of the public is wearing clothing of a color other than the pre-registered detection target color.

[0005] In view of the above, the present invention aims to provide an object detection system and a garbage collection truck that realize "reduction of the burden of preparatory work" and "expansion of detection targets" by performing object recognition and pose recognition. [Means for solving the problem]

[0006] The present invention Garbage Truck 10 is an object detection system that detects an object in an image captured by a camera, the object including a person, the object detection system having an object recognition unit that at least recognizes the person in the image as an actual person, and a posture recognition unit that recognizes the posture of the person in the image, the object including an obstruction, and the object recognition unit of the object detection system also recognizes the obstruction in the image as an actual obstruction. A garbage collection truck using an object detection system, the garbage collection truck having a camera that captures the image, an inlet that is captured by the camera and through which garbage can be dropped, a plate that can push garbage into the inlet, and a control system that controls the plate, the control system stops the dumping of garbage into the plate when a part of a person in the image enters the inlet or when the entire person in the image enters a specified area, and the control system determines that the object cannot be detected by the object detection system when the area of ​​an obstruction in the image occupies a first percentage or more of the area of ​​the image. This is the first characteristic.

[0007] The present invention Garbage Truck 10 A second feature of the present invention is, in addition to the first feature, that the object recognition unit recognizes the position of the entire person in three-dimensional space based on the outline of the person in the image, and the posture recognition unit recognizes predetermined joints of the person based on the outline of the person in the image, and recognizes the position of the portion beyond the recognized predetermined joints in three-dimensional space.

[0008] The present invention Garbage Truck 10 The third feature of the present invention is that, in addition to the first or second feature, the present invention further comprises an object tracking unit that tracks at least the movement of a person within the image.

[0009] The present invention Garbage Truck 10 A fourth feature of the present invention is, in addition to the second feature, to have an object tracking unit that tracks at least a movement of the person in the image, and the object tracking unit recognizes a movement path of the entire person in three-dimensional space and / or a movement path of a portion beyond the specified joint in three-dimensional space based on an outline of the person in the image.

[0010] Due to these features, by having an object recognition unit 2 that recognizes person H in image G as an actual person, and a posture recognition unit 3 that recognizes the posture of person H, unlike Patent Document 1, when the object detection system 1 is used on a garbage collection truck 10, etc., even if the colors of gloves, work clothes, etc. worn by workers on the garbage collection truck 10, etc. differ depending on the municipality, organization, etc., there is no need to perform work such as changing the detection target color registered in the memory unit in advance, and naturally the burden of pre-work is reduced ("reduced burden of pre-work"). At the same time, it becomes possible to detect the approach of ordinary people other than workers on the garbage collection truck 10 or the like to a specified area (danger area, etc.) D, regardless of the color of clothing, etc. worn by such ordinary people, and this is not limited to people H, and it can be said that object B of any color can be detected by the object detection system 1, regardless of the color of object B. ("Expansion of detection targets" is realized.) The object detection system 1 has two units, an object recognition unit 2 and a posture recognition unit 3, and can be said to have a multi-layer structure (in other words, a deep neural network (DNN)) with depth, so it can also be said to be an "object detection system using deep learning, etc."

[0011] In addition, the object recognition unit 2 recognizes the position P of the entire person H based on the contour He, and the posture recognition unit 3 recognizes the position P of specific joints Hj of the person H and the part beyond the specific joints Hj based on the contour He of the person H, thereby improving the "detection accuracy" of an object B such as a person H.

[0012] Furthermore, by having an object tracking unit 4 that tracks the movements of the person H, and by using the object tracking unit 4 to recognize the movement path L of the entire person H and the part beyond a specified joint Hj based on the contour He, it is possible to further "improve detection accuracy."

[0015] The first embodiment of the garbage collection vehicle 10 according to the present invention 5 The features of the above are 1~4 In addition to the above features, the shielding object is a cardboard box and / or an umbrella.

[0016] Due to these features, the control system 14 stops the sweeping of the plate 13 when part of the person H enters the drop-in opening 12 or when the entire person H enters a specified area (such as a danger area) D, thereby enabling the person H, such as a worker on the garbage collection truck 10, to detect dangerous movements when dropping garbage into the drop-in opening 12 of the garbage collection truck 10, and when dangerous movements are detected, the level of danger to the person H, such as the worker, can be reduced ("reduction of risk").

[0017] Furthermore, when an obstruction S in image G occupies a first percentage or more of the area of ​​image G, the control system 14 determines that the object cannot be detected by the object detection system 1, and makes it possible to inform person H, such as the user of the garbage collection truck 10 or a worker, that the object cannot be detected by the object detection system 1. This allows risk avoidance measures to be taken using something other than the object detection system 1, thereby further "reducing the level of risk." The shield S may be a cardboard box B' or an umbrella B". Effect of the Invention

[0018] The object detection system and the garbage collection truck according to the present invention can realize "reduction of the burden of preparatory work" and "expansion of detection targets" by performing object recognition and pose recognition. [Brief description of the drawings]

[0019] [Figure 1] 1 is a schematic diagram showing an object detection system according to the present invention; [Diagram 2] 1 is a drawing substitute photograph illustrating a person in an image "after" detection by an object detection system. [Diagram 3] 1 is a schematic side view showing a designated area (danger area) in a garbage collection vehicle according to the present invention and a person's movements, in which (a) shows that the person's elbows and forwards have entered the designated area, and (b) shows that the person's knees and forwards have entered the designated area. [Figure 4] 1 is a schematic diagram illustrating an object detection system and a garbage collection truck using the same; [Diagram 5] 1A is a flowchart showing a control system for a garbage collection truck, and FIG. 1B is a flowchart showing a process for stopping a plate, and FIG. 1B is a flowchart showing a process for determining whether an object detection system is unable to detect an object. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS With reference to the drawings, embodiments of the present invention will be described in detail. <Overall configuration of object detection system 1> 1, 2 and 4 show an object detection system 1 according to the present invention. This object detection system 1 is a system that detects an object B in an image G captured by a camera.

[0021] The object detection system 1 includes at least an object recognition unit 2 and a posture recognition unit 3, which will be described later, and may be configured as a computer. The object detection system 1 may also include an object tracking unit 4. The object detection system 1 may have other components than the object recognition unit 2, the attitude recognition unit 3, and the object tracking unit 4, and may have, for example, a communication unit that communicates with vehicles (such as a garbage truck 10 described below), devices, equipment, etc. located away from the location where the object detection system 1 is installed, via the Internet or other communication networks, or directly, in a wired or wireless manner, and may transmit and receive images G and videos (such as a series of multiple images G) through this communication unit (see FIG. 4). Also, the object recognition unit 2, the attitude recognition unit 3, and the object tracking unit 4 may be configured as a server, or may be configured in cooperation with or sharing the responsibilities of the computer and server of the garbage truck 10. Next, image G and object B will be described below.

[0022] <Image G> As shown in FIGS. 1 and 2, an image G is captured by a specific camera. When image G is captured by a camera 11 attached to a garbage collection truck 10 described later, the image G may show the garbage inlet 12 of the garbage collection truck 10 described later or parts other than the garbage inlet 12, or a specified area D near the garbage inlet 12, etc., or it may show only the vicinity of the garbage collection truck 10 in image G (i.e., the garbage collection truck 10 itself is not shown in image G).

[0023] Furthermore, image G may be of a vehicle other than the garbage collection truck 10, or may be captured by a camera attached to a device, product, facility, equipment, etc. other than a vehicle. In this case, a part of the other vehicle, etc. may be captured in image G, or a specified area near a part of the other vehicle, etc. may be captured in image G, or only the vicinity of the other vehicle, etc. may be captured in image G (i.e., the other vehicle, etc. itself is not captured in image G). In addition to images captured by a camera 11 attached to a garbage collection truck 10, image G may also be images captured by an aerial vehicle such as a drone, or images captured by a (fixed) camera (such as a security camera or surveillance camera) installed in the city, in a store, a shopping district, a large building facility, a tenant building, an apartment building, etc.

[0024] Image G may be taken by a camera 11 or the like attached to the garbage collection vehicle 10, capturing an image of the drop inlet 12 or the vicinity thereof from approximately above, or from approximately the side or approximately behind the drop inlet 12 or the like. Below, it is described that image G is mainly captured by a camera 11 attached to a garbage collection truck 10 described later, and that the garbage collection truck 10's drop-in opening 12 and a specified area D in the vicinity of the drop-in opening 12 are captured in the image G.

[0025] Furthermore, one object detection system 1 may detect object B from two or more images G. In this case, for example, the images may be two or more images G captured by two or more cameras 11 (so-called stereo cameras) attached to a garbage collection truck 10, or two or more images G captured simultaneously by two or more cameras, such as a camera 11 attached to the garbage collection truck 10 and a camera attached to an aircraft, etc. Alternatively, image G may be a color image (an image that has hue and saturation in addition to brightness), a grayscale image (an image that has only brightness and no hue or saturation), or a black and white two-tone image such as a halftone image. The object B in such an image G will now be described.

[0026] <Object B> As shown in FIGS. 1 to 3, object B is a detection target of the object detection system 1 described above, and is captured in the image G described above. Object B includes person H, which will be described later. The object B also includes an obstruction S, which will be described later, and this obstruction S is a cardboard box B' and / or an umbrella B'', etc.

[0027] In addition, object B may be any object other than person H, cardboard box B', or umbrella B" described below, for example, a bag filled with garbage (garbage bag) or bulky garbage (garbage bags and bulky garbage can also be said to be included in obstructions S), or it may be the ground, buildings, or steps in the environment around person H (such as a pit in a factory (a step of about 1m)), or it may include ordinary people such as moving cars, motorcycles, bicycles, and pedestrians other than person H. Note that with regard to object B, things other than person H and obstructions S (such as the ground, buildings, steps, the car itself, etc.) can also be collectively said to be the environment (other things). Also, in image G, a red circle may be displayed to indicate that video is being recorded (recording in progress).

[0028] <Human H, contour He, joints Hj> As shown in Figures 1 to 3, person H is a person who appears in the image G described above, and may include not only the workers on the garbage collection truck 10 described above, but also drivers of cars, motorcycles, or bicycles, pedestrians, etc. who are approaching the drop-in opening 12 of the garbage collection truck 10 (those who are close to or have entered the specified area D). It can be said that in a single image G, there are cases where the person H and parts other than the person H (such as the garbage truck 10, the background, the cardboard box B', the umbrella B") are captured (present), and cases where only the person H is captured (because the person H or a part of him / her (such as a hand) approaches the camera 11 described above, for example). In the following description, it is assumed that person H is mainly captured in a single image G together with other parts of person H.

[0029] The contour (edge) He of a person H is the boundary between the person H and parts other than the person H in one image G, and can be said to represent the shape of the person H. The extraction of the contour He of the person H from the image G will be described later in relation to the object recognition unit 2 and the like.

[0030] Person H naturally has joints Hj, and the joints Hj include, specifically, the left and right shoulders, the left and right elbows, the left and right wrists, the joints in the palms of the left and right hands, the first and second joints of the thumbs, and the first to third joints of the index finger, middle finger, ring finger, and little finger of the left and right hands, the neck (neck joint), the waist (waist joint), the left and right hip joints (hip joints), the left and right knees, the left and right ankles, the joints of the left and right feet, the first and second joints of the thumbs (including the ball of the foot), and the first to third joints of the index finger, middle finger, ring finger, and little finger of the left and right feet, etc.

[0031] <Part of the human body (part from the elbow to the forehead HA, part from the knee to the forehead HB, etc.) As shown in FIGS. 1 to 3, the person H has the above-mentioned joints Hj, and the person H has a portion extending beyond a certain joint Hj. Specifically, this includes the left and right elbow parts HA (if it is from joint to joint, then it would be from the elbow to the wrist or from the elbow to the first joint of any finger), the left and right knee parts HB (similarly, from the knee to the ankle or from the knee to the first joint of any finger), as well as the left and right shoulder parts (from the shoulder to the wrist or from the shoulder to the first joint of any finger), the left and right hip joint parts (from the hip joint to the ankle or from the hip joint to the first joint of any finger), and the neck part (the head, so to speak).

[0032] With regard to the parts HA from the elbows down etc., if the object recognition unit 2 and posture recognition unit 3 described later can extract from image G the tips of each arm of person H, the tips of each finger on each arm (particularly the tips of the middle finger and thumb), the tips of each leg of person H, the toes of each leg, the top of the head, etc., then the part HA from the elbows down may be from the elbow to the tips of each arm, the part HB from the knees down may be from the knee to the tips of each leg, etc., and the part from the neck down may be from the neck to the top of the head, etc. Below, we will discuss object B other than person H.

[0033] <Shielding object S (cardboard B', umbrella B”, etc.)> As shown in Figure 1, the obstruction S is an object that blocks the image of the person H described above, and if the image G is captured by a camera 11 attached to a garbage collection truck 10, it can also be said that the obstruction S is located between the camera 11 and the drop-off opening 12 and blocks the image of the person H. The shield S may be, for example, a cardboard box B' or an umbrella B'', or may be a garbage bag or bulky trash.

[0034] As shown in FIG. 1, cardboard box B' is the cardboard box shown in image G described above. It can be said that in a single image G, there are cases where cardboard B' and parts other than cardboard B' (such as person H, garbage truck 10, background, umbrella B") are captured (present), and cases where only cardboard B' is captured (due to cardboard B' approaching camera 11 as mentioned above, etc.). Hereinafter, the cardboard box B' will be described as being mainly shown in a single image G together with parts other than the cardboard box B'.

[0035] The contour B'e of the cardboard box B' is also the boundary between the cardboard box B' and parts other than the cardboard box B' within a single image G, and can be said to represent the shape of the cardboard box B'. The extraction of the contour B'e of the cardboard box B' from the image G will also be described later in relation to the object recognition unit 2 and the like.

[0036] Similarly, as shown in FIG. 1, umbrella B” is an umbrella that appears in the above-mentioned image G, and it can be said that there are cases where, within a single image G, umbrella B” and parts other than umbrella B” (such as person H, garbage truck 10, background, cardboard box B', etc.) appear (exist), and cases where only umbrella B” appears (because umbrella B” approaches camera 11 as described above, etc.), and hereinafter umbrella B” will mainly be described as appearing together with parts other than umbrella B” within a single image G. The contour B"e of umbrella B" is also the boundary between the umbrella B" and parts other than the umbrella B" within a single image G, and can be said to represent the shape of the umbrella B". The extraction of the contour B"e of umbrella B" from image G will also be described in the object recognition unit 2, etc., which will be described later.

[0037] In addition, even if the obstructing object S is a garbage bag or bulky garbage, there are cases where the garbage bag, etc. and other parts (such as a person H, a garbage truck 10, the background, a cardboard box B', an umbrella B'', etc.) are captured (present) in a single image G, and cases where only the garbage bag, etc. is captured (because the garbage bag, etc. has approached the camera 11 described above, etc.), and in the following, the garbage bag, etc. will mainly be described as being captured together with other parts in a single image G. It should be noted that bulky garbage can also include cardboard boxes B' and umbrella B''. The contour of a garbage bag, etc. can also be said to be the boundary between the garbage bag, etc. and parts other than the garbage bag, etc. within a single image G, and the extraction of the contour of the garbage bag, etc. from image G will be described in the object recognition unit 2, etc., described later. Next, the object recognition unit 2 and the like will be described.

[0038] <Object recognition unit 2> As shown in FIG. 1, the object recognition unit 2 is a part that at least recognizes that the person H in the above-mentioned image G is an actual person. Of course, the object recognition unit 2 may also recognize objects other than person H. For example, it may recognize that the cardboard box B' in the image G is an actual cardboard box, and it may recognize that the umbrella B" in the image G is an actual umbrella. Furthermore, the object recognition unit 2 may recognize from the cardboard box B' and umbrella B" in image G whether the actual cardboard box is box-shaped or folded into a substantially rectangular shape, or whether the actual umbrella is open or closed, or may recognize a predetermined area D, described below, within image G (or may provide in advance the position and range (region) of the predetermined area D within image G, or the position P and range (region) in three-dimensional space to the object recognition unit 2, etc.).

[0039] The object recognition unit 2 may have any configuration as long as it recognizes a person H, etc. in an image G, but for example, it may recognize that a person H, etc. in the above-mentioned image G is an actual person, etc. based on the contour He of the person H, etc. In addition, the object recognition unit 2 may remove noise and distortion from the image G, emphasize the contour He of the person H in the image G, adjust the brightness and color (tone, hue, and saturation) of the image G, cut out an area of ​​the person H, etc. from the image G (area extraction), etc.

[0040] In the object recognition unit 2, any image processing may be performed as long as the contour He of person H, etc. can be extracted from image G. For example, the contour He of person H, etc. may be extracted (by gradient method) from maximum and minimum values ​​calculated by first derivative of changes in pixel values ​​(brightness of each pixel, etc.) of each pixel in image G, or extracted (by Laplacian method) from inflection points calculated by second derivative. In addition, the contour He of person H may be extracted from image G by gradient method using smoothing or weighted average, Canny method, etc. Furthermore, the extraction of the contour He of a person H from an image G may be performed using artificial intelligence (AI), machine learning (ML), or deep learning (DL), which will be described later.

[0041] The object recognition unit 2 recognizes, based on the contour He of person H extracted from image G, that person H in image G from which the contour He has been extracted, is an actual person, and recognizes that a cardboard box B' and an umbrella B" in image G from which the contours B'e, B"e, etc. have been extracted, are actual cardboard boxes and umbrellas, but the recognition of these people, etc. may be performed by deep learning. In more detail, the object recognition unit 2 uses a deep neural network (DNN) that is a neural network (NN) that mimics the structure of the human brain nerves, increasing the number of layers such as the "input layer," "hidden layer," and "output layer" to make it more multi-layered (deep).

[0042] It can be said that such an object recognition unit 2 can automatically learn the feature amount (pattern) of the person H or the entire person H simply by inputting the image G. Note that the object recognition unit 2 may be provided in advance with a plurality of images G (e.g., tens of thousands of images) labeled (with correct answers) as to which lines in the image G are the contour He of the person H, which (which area, etc.) are the person H, etc. (i.e., learning with a teacher (with correct answers) is performed), and the object recognition unit 2 automatically learns the feature amount, and for the image G input to the object recognition unit 2, it may output which lines are the contour He of the person H, etc., and which are the person H, etc., as a score (a ratio such as 96%) from the previously learned feature amount, or may output an image G with a display indicating that the input image G has been recognized as a person H (e.g., by enclosing the image G in a rectangle containing the word "person", see FIG. 2). Additionally, the object recognition unit 2 may perform unsupervised learning, reinforcement learning, or the like.

[0043] <Position P in three-dimensional space (position Ph of the entire person H), etc.> As shown in Figures 1 to 3 and described above, the object recognition unit 2 may recognize the position P (left / right position X, front / back position Y, up / down position Z, etc.) of the entire person H in three-dimensional space based on the contour He and area of ​​the person H in the image G. Furthermore, the object recognition unit 2 may also recognize the position P in three-dimensional space of the entire cardboard box B' or the entire umbrella B'', which are objects B other than person H, based on their contours B'e, B''e and areas.In addition, the object recognition unit 2 may also output which areas of the background other than object B are the background as a score, or output an image G with an indication that the cardboard box B', umbrella B'', etc. have been recognized from the input image G.

[0044] Here, as described above, the position P in three-dimensional space is recognized (calculated) based on the contours He of the person H, cardboard box B', etc. in the image G, but the position P in three-dimensional space may also be recognized based on the area of ​​the person H, etc. extracted from the image G. Of the positions P in three-dimensional space, the position Ph of person H as a whole in three-dimensional space may be determined as the position P in three-dimensional space of the center of gravity of person H in image G (the center of gravity of the area occupied by person H (figure of a specified shape)) based on the contour He and area of ​​person H from image G.

[0045] The position Ph of person H as a whole in three-dimensional space may be the position of the waist joint of person H based on the joints Hj of person H recognized by the posture recognition unit 3 described later, or it may be the position of the foot (such as the ball of the foot) that is in contact with the ground on which person H is standing. In addition, among the positions P in three-dimensional space, the position P in three-dimensional space of cardboard B', umbrella B", etc. may also be the position P in three-dimensional space of the center of gravity of cardboard B', umbrella B", etc. in image G (the center of gravity of the area occupied by cardboard B', umbrella B", etc. (for example, if cardboard B' is box-shaped, it is an approximately hexagonal shape, if cardboard B' is folded, it is an approximately parallelogram-shaped or approximately rectangular shape, if umbrella B" is open, it is an approximately crescent-shaped shape, and if umbrella B" is closed, it is an approximately linear or approximately "? (question mark)"-shaped shape (such as the gripping part), etc.)) based on the contours B'e, B"e and areas of cardboard B', umbrella B", etc. from image G.

[0046] <Posture recognition section 3> As shown in FIG. 1, the posture recognition unit 3 is a part that recognizes the posture of the person H in the image G described above, or the posture of the person H recognized by the object recognition unit 3 described above. The posture recognition unit 3 may recognize postures other than that of the person H. For example, the posture recognition unit 3 may recognize the postures of a cardboard box B' or an umbrella B'' in the image G.

[0047] The posture recognition unit 3 may have any configuration as long as it recognizes the posture of person H in image G. For example, the posture recognition unit 3 may recognize a specific joint Hj of person H in the above-mentioned image G based on a contour He of person H extracted by the posture recognition unit 3 itself, or may recognize a position P in three-dimensional space of the part beyond the recognized specific joint Hj. On the other hand, the posture recognition unit 3 may utilize the contour He of the person H extracted by the object recognition unit 2 described above, and based on the contour He of the person H extracted by the object recognition unit 2, recognize a specific joint Hj of the person H in the image G described above, or recognize the position P in three-dimensional space of the part beyond the recognized specific joint Hj. In addition, like the object recognition unit 2 described above, the posture recognition unit 3 may also remove noise and distortion from the image G, emphasize the contour He of the person H in the image G, adjust the brightness and color of the image G, cut out areas of the person H, etc. from the image G, etc.

[0048] In the posture recognition unit 3, any image processing may be performed as long as the contour He of person H, etc. can be extracted from the image G. As in the object recognition unit 2 described above, the contour He of person H, etc. may be extracted from the image G by, for example, a gradient method, a Laplacian method, a gradient method using smoothing or a weighted average, a Canny method, etc., and the extraction of the contour He of person H, etc. from the image G may itself be performed by artificial intelligence, machine learning, deep learning, etc. The posture recognition unit 3 also recognizes predetermined joints Hj of the person H in the image G and the position P of the part beyond the predetermined joints Hj in the three-dimensional space based on the contour He of the person H extracted by the posture recognition unit 3 itself (or the above-mentioned object recognition unit 2) from the image G, but the recognition of the predetermined joints Hj, etc. may be performed by deep learning. Details of the deep learning itself are the same as those of the above-mentioned object recognition unit 2.

[0049] It can be said that such a posture recognition unit 3 can automatically learn the feature amount (pattern) of a predetermined joint Hj of a person H simply by inputting an image G. Note that the posture recognition unit 3 is provided in advance with a plurality of images G labeled with information such as which parts of the person H in the image G are the predetermined joints Hj of the person H (i.e., the posture recognition unit 3 is made to perform supervised learning), and the posture recognition unit 3 automatically learns the feature amount, and may output, for the image G input to the posture recognition unit 3, which part is the predetermined joint Hj of the person H from the previously learned feature amount as a score, or may output an image G with a mark indicating the predetermined joint Hj of the person H recognized for the input image G with a mark such as "○", and other such information may be provided. In addition, the posture recognition unit 3 may perform unsupervised learning, reinforcement learning, and the like. In addition, for objects B other than person H, such as a cardboard box B' or an umbrella B", the posture recognition unit 3 may recognize characteristic parts or points in the cardboard box B', umbrella B", etc., based on the contours B'e, B"e of the cardboard box B', umbrella B", etc. extracted by the posture recognition unit 3 itself (or the above-mentioned object recognition unit 2), and may recognize the position P of the recognized parts or points in three-dimensional space.

[0050] <Position P in three-dimensional space (position Pj of the part beyond a specific joint Hj) etc.> As shown in Figures 1 to 3 and described above, the posture recognition unit 3 may recognize a predetermined joint Hj of the person H in the image G based on the contour He or area of ​​the person H in the image G extracted by the posture recognition unit 3 itself (or the above-mentioned object recognition unit 2), and may recognize the position P in three-dimensional space of the part beyond the recognized predetermined joint Hj. Furthermore, for objects B other than person H such as a cardboard box B' or an umbrella B", the posture recognition unit 3 may also recognize the position P in three-dimensional space of characteristic parts of the cardboard box B' or umbrella B", etc. (for example, in the case of cardboard B', the corners of a box-like, approximately parallelogram-like, approximately rectangular, etc. shape, or, if the umbrella B" is open, the tip of the convex arc side of an approximately crescent-shaped figure, both ends connecting the convex arc side and the concave arc side, or an approximately "?" shape (such as a gripping part) attached to the end of an approximately linear shape, or, if the umbrella B" is closed, the approximately "?" shape (such as a gripping part) attached to the end of the approximately linear shape and the tip on the opposite side) based on the contours B'e, B"e and areas of the cardboard box B', umbrella B", etc. extracted by the posture recognition unit 3 itself (or the above-mentioned object recognition unit 2).

[0051] It should be noted that the position P in the three-dimensional space recognized by the posture recognition unit 3 is not the overall position P of the person H or the cardboard box B' but the position P of a portion thereof. Among the positions P in three-dimensional space, the position Pj in three-dimensional space as the portion HA, HB, etc. beyond a specific joint Hj (such as an elbow or knee) of person H may be approximately the midpoint of the portion HA, HB, etc. beyond a specific joint Hj of person H in image G, based on the contour He and area of ​​person H from image G.

[0052] Specifically, if the joint Hj is an elbow, the part HA from the elbow Hj may be approximately the midpoint between the position Pj of the elbow Hj in three-dimensional space and the position Pj of the wrist in three-dimensional space; if the joint Hj is a knee, the part HB from the knee Hj may be approximately the midpoint between the position Pj of the knee Hj in three-dimensional space and the position Pj of the ankle in three-dimensional space. Alternatively, if the joint Hj is a shoulder, the part from the shoulder Hj may be approximately the midpoint between the position Pj of the shoulder Hj in three dimensional space and the position Pj of the wrist in three dimensional space, or the position Pj of the elbow in three dimensional space; if the joint Hj is a hip joint, the part from the hip joint Hj may be approximately the midpoint between the position Pj of the hip joint Hj in three dimensional space and the position Pj of the ankle in three dimensional space, or the position Pj of the knee in three dimensional space.

[0053] Here, the posture recognition unit 3 may recognize the posture of the person H as a whole, such as whether he or she is standing, bending, walking, or putting something into the opening 12 of the garbage collection truck 10 described later, from the positions Pj in three-dimensional space of each of the specific joints Hj of the person H in the image G and the relative positions of each of these joints Hj relative to each other. In addition, among the positions P in three-dimensional space, the position P in three-dimensional space as a part of the cardboard B', umbrella B'', etc., may be based on the contours B'e, B''e, or areas of the cardboard B', umbrella B'', etc. from image G. For example, in the case of cardboard B', it may be the position P in three-dimensional space of each corner portion itself, which is box-shaped or has a folded approximately parallelogram shape, or, if the umbrella B'' is open, it may be the position P in three-dimensional space of the tip portion on the convex arc side of the approximately crescent-shaped figure, or, if the umbrella B'' is closed, it may be the position P in three-dimensional space of the approximately ``?'' shape (such as the gripping part) attached to the end of an approximately linear shape.

[0054] In addition, the orientation recognition unit 3 may also recognize the orientation of cardboard B', umbrella B'', etc. For example, in the case of cardboard B', it may recognize whether the cardboard B' is box-shaped or folded from the positional relationship between the position P in three-dimensional space of the center of gravity of cardboard B' in image G described above in the object recognition unit 2 (the center of gravity as a figure such as an approximately hexagonal shape if cardboard B' is box-shaped, or an approximately parallelogram shape or approximately rectangular shape if cardboard B' is folded) and the position P in three-dimensional space of each corner, the number of corners, etc. Also, for example, in the case of an umbrella B", it may be possible to recognize whether the umbrella B" is open or closed based on the positional relationship between the position P in three-dimensional space of the center of gravity of the umbrella B" in image G described in the object recognition unit 2 above (if the umbrella B" is open, it is a roughly crescent-shaped figure, and if the umbrella B" is closed, it is the center of gravity as a figure such as a roughly linear or roughly "?"-shaped figure (such as the gripping part)) and the position P in three-dimensional space of the tip part of the convex arc side of the roughly crescent-shaped figure, or the presence or absence of a roughly crescent-shaped figure.

[0055] <Object Tracking Unit 4> As shown in Figure 1, the object tracking unit 4 is a part that at least tracks the movement of person H within the above-mentioned image G, or the movement of person H recognized by the above-mentioned object recognition unit 2 and posture recognition unit 3. Of course, the object tracking unit 4 may track objects other than the person H. For example, the object tracking unit 4 may track a cardboard box B' in the image G or an umbrella B'' in the image G.

[0056] The object tracking unit 4 may have any configuration as long as it tracks the person H in the image G. For example, the object tracking unit 4 may recognize the movement path L of the entire person H in three-dimensional space, or the movement path L of parts HA, HB, etc. beyond a specific joint Hj in three-dimensional space, based on the contour He of the person H extracted by the object tracking unit 4 itself. On the other hand, the object tracking unit 4 may utilize the contour He of the person H extracted by the object recognition unit 2 and posture recognition unit 3 described above, and recognize the movement path L of the entire person H in three-dimensional space, and the movement path L of the parts HA, HB, etc. beyond a specific joint Hj in three-dimensional space, based on the contour He of the person H extracted by the object recognition unit 2 and the posture recognition unit 3.

[0057] In addition, the object tracking unit 4 may be configured so that the camera that captures the image G (such as the camera 11 attached to the garbage collection truck 10 described later) itself moves so that the optical axis of the lens moves, so that a person H, etc. is always captured in the image G. In addition, like the object recognition unit 2 described above, the object tracking unit 4 may also remove noise and distortion from the image G, emphasize the contour He of the person H in the image G, adjust the brightness and color of the image G, and cut out areas of the person H, etc. from the image G.

[0058] In the object tracking unit 4, any image processing may be performed as long as the contour He of person H, etc. can be extracted from image G. As in the object recognition unit 2 and posture recognition unit 3 described above, the contour He of person H, etc. may be extracted from image G by, for example, a gradient method, a Laplacian method, a gradient method using smoothing or a weighted average, a Canny method, etc., and the extraction of the contour He of person H, etc. from image G may itself be performed by artificial intelligence, machine learning, deep learning, etc. The object tracking unit 4 also recognizes the movement path L of the entire person H in three-dimensional space, and the movement path L of parts HA, HB, etc. beyond a specific joint Hj in three-dimensional space, based on the contour He and area of ​​the person H extracted from the image G by the object tracking unit 4 itself (or the above-mentioned object recognition unit 2 or posture recognition unit 3). Furthermore, for objects B other than person H, such as a cardboard box B' or an umbrella B", the object tracking unit 4 may recognize the movement path L of the entire cardboard box B', umbrella B", etc. in three-dimensional space, or the movement path L of characteristic parts or points of the cardboard box B', umbrella B", etc. in three-dimensional space, based on the contours B'e, B"e and areas of the cardboard box B', umbrella B", etc. extracted by the object tracking unit 4 itself (or the above-mentioned object recognition unit 2 or posture recognition unit 3).

[0059] <Path of movement in three-dimensional space L> As shown in Figures 1 to 3 and described above, the object tracking unit 4 may recognize the movement path L of the entire person H in three-dimensional space, or the movement path L of parts HA, HB, etc. beyond a specific joint Hj in three-dimensional space, based on the contour He or area of ​​the person H in image G extracted by the object tracking unit 4 itself (or the above-mentioned object recognition unit 2 or posture recognition unit 3). Furthermore, for objects B other than person H such as a cardboard box B' or an umbrella B", the object tracking unit 4 may also recognize the movement path L in three-dimensional space of characteristic parts of the cardboard box B' or umbrella B" (for example, in the case of a cardboard box B', the corners of a box-like, approximately parallelogram-like, approximately rectangular, etc. shape, or, if the umbrella B" is open, the tip of the convex arc side of an approximately crescent-shaped figure, both ends connecting the convex arc side and the concave arc side, or an approximately "?" shape (such as a gripping part) attached to the end of an approximately linear shape, or, if the umbrella B" is closed, the approximately "?" shape (such as a gripping part) attached to the end of an approximately linear shape and the tip on the opposite side) based on the contours B'e, B"e and areas extracted by the object tracking unit 4 itself (or the above-mentioned object recognition unit 2 or posture recognition unit 3).

[0060] Here, in the present invention, a "movement path L" refers to an image G from one time to another time (an image from the starting point time to the ending point time; for example, if 30 images G are captured per second, a total of 300 images will be captured in 10 seconds), and a line segment (or straight line) showing the movement of the whole or a part of a person H, etc. (parts HA, HB, etc. beyond a specified joint Hj) in three-dimensional space, and the image G from the starting point time to the ending point time means that there are two or more images G, including the image G at the starting point time and the image G at the ending point time. Furthermore, in the present invention, a "line segment (or straight line) indicating movement in three-dimensional space" refers to a predetermined line segment or straight line in three-dimensional space that is recognized based on the position P (left / right position X, front / back position Y, up / down position Z, etc.) in three-dimensional space in each of images G at two or more times.

[0061] For example, if there are two times (i.e., if there are only two images, image G at the initial end time and image G at the end time), the line segment connecting position P in three-dimensional space at the initial end time and position P in three-dimensional space at the end time can be said to be the movement path L. In addition, since the position P in the three-dimensional space is expressed as a set of three values, such as the left-right position X, the front-back position Y, and the up-down position Z, it is possible to assume that the position P in the three-dimensional space at two times exists simultaneously in one three-dimensional space, and it can also be said that the line segment connecting these two assumed positions P is the travel path L.

[0062] Furthermore, if there are three or more times (that is, in addition to the two images G at the start end time and G at the end time, there is one or more images G at a time between the start end time and the end time; as described above, if 30 images G are captured per second, there will be 298 images at times between the start end time and the end time in 10 seconds), it is assumed that each of the positions P in three-dimensional space at the three or more times exist simultaneously in a single three-dimensional space, and the line connecting these three or more assumed positions P can be said to be the movement path L. Alternatively, it can be said that the movement path L is the straight line (a line segment regression line, so to speak) that minimizes the sum of the distances to each of the three or more assumed positions P, for example, using the least squares method. Furthermore, if this movement path L is a straight line, both ends of it can be extended infinitely, but since both ends of the line will move away from position P in three-dimensional space at the initial end time and position P in three-dimensional space at the terminal time, strictly speaking, this movement path L can also be said to be a line segment between the foot of a perpendicular line drawn from position P in three-dimensional space at the initial end time to the line (movement path L) to the foot of a perpendicular line drawn from position P in three-dimensional space at the terminal time to the line.

[0063] In addition, if the movement of the whole or part of a person H, etc. in three-dimensional space is curved, such as meandering, a line segment connecting position P in three-dimensional space at the starting point time and position P in three-dimensional space at the ending point time, or a straight line (or line segment) linearly regressed using the least squares method, etc., can be said to be far removed from the actual movement, so nonlinear regression or divided linear regression may be performed on each of the positions P in three-dimensional space at each assumed time that existed simultaneously in a single three-dimensional space from image G from the starting point time to the ending point time. Furthermore, regardless of whether the movement of the whole or part of the person H or the like in a three-dimensional space is curved or linear, the object tracking unit 4 may recognize the movement by deep learning.

[0064] To explain in more detail, the object tracking unit 4 also uses a multi-layered deep neural network, and by simply inputting image G at the start time to image G at the end time, it can automatically learn features (patterns) regarding the movement of the whole or part of a person H, etc. in three-dimensional space. In addition, the object tracking unit 4 may be provided in advance with multiple images G labeled with which lines are the movement path L in the image G from a specified start time to an end time (i.e., allowed to perform supervised learning), and the object tracking unit 4 may automatically learn the features and output, as a score, which line is the movement path L, etc., for the image G input to the object tracking unit 4 from the features learned in advance, or may output an image G with an indication of the movement path L recognized for the input image G. Additionally, the object tracking unit 4 may also perform unsupervised learning, reinforcement learning, or the like.

[0065] In this way, by the object tracking unit 4 recognizing the movement path L of the whole or part of a person H, etc. in three-dimensional space, it can be said that, for example, based on the movement path L of the part HA of the person H from the elbow Hj onwards, it can be determined with greater accuracy whether or not the part HA of the person H from the elbow Hj onwards has entered the drop-in opening 12 of the garbage collection truck 10 described below. Furthermore, when the object tracking unit 4 recognizes the movement path L of the whole or part of the person H, etc. in the three-dimensional space, it naturally recognizes the position P in the three-dimensional space at the start time and the position P in the three-dimensional space at the end time, so that the movement of the whole or part of the person H, etc. in the three-dimensional space can be recognized not only by the movement path L but also by the direction (direction, vector) of the movement from the position P in the three-dimensional space at the start time to the position P in the three-dimensional space at the end time. For this reason, for example, based on the direction of the movement of the part HA of the person H from the elbow Hj to another position P in the three-dimensional space and the movement path L, it can be said that it is possible to determine with even higher accuracy whether the part HA of the person H from the elbow Hj to the end has entered the input port 12 of the garbage truck 10 described later. The garbage collection truck 10 will be described below.

[0066] <Garbage Truck 10> As shown in FIGS. 3 to 5, a garbage collection truck 10 according to the present invention is a car (vehicle) that uses the object detection system 1 described above and collects garbage. Here, "garbage" in the present invention can also be said to be rubbish, and in this case, the garbage collection truck 10 can also be said to be a garbage collection vehicle.

[0067] In addition, in the present invention, "using the object detection system 1" does not only mean that the object detection system 1 is mounted on the garbage collection truck 10 (in other words, the garbage collection truck 10 has the object detection system 1), but also means that the garbage collection truck 10 is connected via the Internet to an object detection system 1 located in a different location from the garbage collection truck 10 (in other words, not mounted on the garbage collection truck 10), and object recognition and posture recognition and other object tracking are performed by the object detection system 1 (see Figure 4). The garbage collection vehicle 10 has a camera 11, a dump port 12, a plate 13, and a control system 14.

[0068] In addition, the garbage collection truck 10 may have a garbage collection box 22 having an inlet 12 at the rear side mounted on a body frame 21 of the garbage collection truck 10, and a driver's seat 23, etc., may be provided at the front. The garbage collection box 22 is pivotally supported on the vehicle frame 21 so as to be tiltable, and may be tilted toward the rear of the garbage collection vehicle 10 by extending a tilting cylinder connected between the garbage collection box 22 and the vehicle frame 21.

[0069] The garbage storage box 22 has a storage box main body 22a and a garbage insertion box 22b, and the garbage insertion box 22b, which has an insertion opening 12 (described later) on its rear end surface, is connected to the storage box main body 22a so as to cover the rear end side of the storage box main body 22a. The garbage insertion box 22b is pivotally supported on the rear upper end of the storage box main body 22a so as to be tiltable. In addition, the garbage collection truck 10 may have a communication unit that communicates with the object detection system 1, devices, equipment, etc. located at a location remote from the garbage collection truck 10, either via the Internet or directly, either wired or wirelessly, and this communication unit may send and receive images G and videos (such as multiple consecutive images G) (see Figure 4), or may have a recording device that records the images G and videos captured by the camera 11 (or a memory unit that stores data of images G, not shown), and these communication units and memory units may be provided within the control system 14 described later. The camera 11 and other components of the garbage collection truck 10 will be described below.

[0070] <Camera 11> As shown in FIGS. 3 and 4, the camera 11 is a device (imaging device) that captures the image G described above. The camera 11 may be attached to the garbage collection truck 10 at a position where an inlet 12, which will be described later, is included in the image G.

[0071] Two or more cameras 11 may be attached to one garbage collection truck 10 (so-called stereo cameras), and the lens of the camera 11 is not particularly limited, but may be, for example, a wide-angle lens. Camera 11 may be installed in any location as long as it captures images of drop inlet 12 etc. and its vicinity from approximately above, or from approximately the side, or from approximately the rear, so that drop inlet 12 or a specified area D near drop inlet 12 etc. is captured in image G captured by camera 11. For example, camera 11 may be installed on the upper side of the rear end face of waste collection box 22 (in other words, camera 11 may be attached above drop inlet 12, which will be described later, with the optical axis of the lens facing diagonally downward and rearward.

[0072] The range of image G captured by such a camera 11 is not particularly limited, but if a wide-angle lens is used, for example, the sky may be captured in part of the upper part of image G, or the left and right rear corners of the garbage truck 10 may be captured in part of the left and right sides of image G, and a point approximately 10 m behind the garbage truck 10 may also be captured in image G. Alternatively, the camera 11 may be used as a backup camera for the driver of the garbage collection truck 10 to monitor the rear. In addition, a work light (not shown) may be provided below or to the side of the location where the camera 11 is attached.

[0073] <Inlet port 12> 3 and 4, the insertion opening 12 is provided on the rear end surface of the waste collection box 22 (waste insertion box 22b) as described above, and is an opening portion through which waste can be inserted into the waste collection box 22. As described above, the insertion slot 12 is imaged by the camera 11, and the image G may include the insertion slot 12 and a predetermined area D in the vicinity of the insertion slot 12 or the like. Also, the insertion port 12 may be always open, but may also be openable and closable.

[0074] The insertion port 12 can be said to be an opening that connects the outside of the rear end side of the garbage collection box 22 to the inside of the garbage collection box 22, and when a person H, such as a worker, puts his / her elbow Hj and the part (hand, forearm, etc.) HA, etc., into the insertion port 12, the part HA, etc., from the elbow Hj and the part of the body is hidden in the image G taken by the camera 11 that captures the insertion port 12 from approximately above, etc. (i.e., the part HA, etc., from the elbow Hj and the part of the body is not visible in the image G). As a result, the object detection system 1 (particularly the posture recognition unit 3) determines that the part HA, etc., from the elbow H1 and the part of the body, which is a part of the person H, has entered the insertion port 12. In the following description, "a part of the person H entering the insertion port 12" may be expressed as "hidden". Furthermore, since the drop-in opening 12 is also shown in image G, the object recognition unit 2 may naturally recognize the position P of the drop-in opening 12 in three-dimensional space, and this "position P of the drop-in opening 12 in three-dimensional space" refers to the actual drop-in opening 12 having a predetermined width, and, based on the positional relationship between the camera 11 attached to the garbage collection truck 10 and the drop-in opening 12, the position P is below a predetermined line segment that runs approximately along the left-right direction in image G, and both ends of the predetermined line segment become the left and right edges of the actual drop-in opening 12. Furthermore, if the camera 11 is installed on the upper edge of the drop-in opening 12 or inside the drop-in opening 12 (inside the garbage dump box 22b), the person H (or part of the person, such as the part HA from the elbow H1 onwards) will not be hidden by the drop-in opening 12, so if there is part of the person H in an area overlapping the drop-in opening 12, it can be determined that part of the person H has entered the drop-in opening 12.

[0075] In addition, when the person H has his / her elbow Hj and / or other parts HA inserted in the insertion port 12, the plate 13 described below should be avoided from being moved. Plate 13 is described below.

[0076] <Plate 13> As shown in FIGS. 3 and 4, the plate 13 is a generally plate-shaped part (member) that can push the waste inserted into the insertion opening 12 into the depths of the waste collection box 22a. There are no particular limitations on the configuration of the plate 13 so long as it is capable of sweeping the garbage inside the drop-in opening 12 to the back of the garbage collection box 22a, but for example, it may be provided in two locations near the drop-in opening 12 of one garbage collection truck 10 (see Figure 3(a)), or it may be provided in only one location near the drop-in opening 12 of one garbage collection truck 10, and there are also no particular limitations on the shape of the plate 13 (see Figures 3(a) and (b)).

[0077] With regard to Figure 3(a), the plate 13 may be a first plate 13a provided in the above-mentioned garbage disposal box 22b and rotatable about a support shaft by a rotary motor in order to scoop up garbage disposed in the garbage disposal box 22b from the bottom to the upper front side within the garbage disposal box 22b, and a second plate 13b that can be swung back and forth about a support shaft by a pushing cylinder in order to push the garbage on the first plate 13a into the storage box main body 22a. In the case of FIG. 3(b), the plate 13 is only the first plate 13a described above, and it can be said that the first plate 13a plays a role in raking the dirt into the storage box main body 22a.

[0078] The plate 13 (first and second plates 13a and 13b) may have a sensor such as an encoder attached to its support shaft so that its rotation angle and swing angle can be detected. In this way, by detecting the rotation angle of the plate 13, the position of the plate 13 can be determined. A control system 14 for controlling such plates 13 etc. will be described below.

[0079] <Control System 14> As shown in FIGS. 4 and 5, the control system 14 is a system that controls at least the plate 13 described above, and may be configured by a computer. The control system 14 stops the sweeping of the above-mentioned plate 13 in any of the following cases <1-1> to <1-3>: <1-1> when a part of the person H in the above-mentioned image G (such as the part HA from the elbow Hj to the tip, or the part HB from the knee Hj to the tip, etc.) is hidden by the above-mentioned insertion port 12 for a first time period (e.g., 5 seconds) or more has elapsed; <1-2> when a part of the person H in the above-mentioned image G is hidden by the insertion port 12 for a second time period (e.g., 5 seconds) or more has elapsed and the above-mentioned plate 13 is in a predetermined position; <1-3> when the entire person H in the above-mentioned image G has entered the predetermined area D and a third time period (e.g., 15 seconds) or more has elapsed.

[0080] Here, in <1-1> and <1-2> of the present invention, "when a part of person H is hiding in the insertion port 12 and a time of 5 seconds or more has elapsed (in other words, the time when a part of person H is hiding in the insertion port 12)" does not only include the case where exactly 5 seconds have elapsed since hiding (the hiding time is exactly 5 seconds), but also the case where exactly 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, etc. have elapsed since hiding (the hiding time is exactly 6 seconds...), and may also be "when a part of person H is hiding in the insertion port 12 and a time of 10 seconds or more has elapsed" or "when a part of person H is hiding in the insertion port 12 and a time of 15 seconds or more has elapsed". In <1-3> of the present invention, "when the entire person H in image G has entered the designated area D and 15 seconds or more have passed (in other words, the entry time of the entire person H into the designated area D)" similarly includes not only the case where exactly 15 seconds have passed since entry, but also the case where exactly 16 seconds, 17 seconds, 18 seconds, 19 seconds, 20 seconds, 21 seconds, 22 seconds, 23 seconds, 24 seconds, 25 seconds, etc. have passed since entry (the entry time is exactly 16 seconds...), and may also be "when a part of person H is hidden by the insertion port 12 and 20 seconds or more have passed" or "when a part of person H is hidden by the insertion port 12 and 25 seconds or more have passed." Up to this point, the introduction of the plate 13 has been stopped after waiting for the first to third times to elapse, but the introduction of the plate 13 may be stopped immediately (without waiting for the time to elapse) when a part of the person H enters the insertion port 12.

[0081] In addition, in the present invention, "a part of the person H in image G is hidden by the insertion slot 12" means, with respect to a part of the person H and the insertion slot 12 in the same image G, that a part of the person H that was shown in that image G literally disappears from the image G (is no longer shown in the image G). This disappearance of a part of the person H from image G can be said to be recognized by the object recognition unit 2 and the posture recognition unit 3, and in the case where there are a pair of parts of the person H, one on the left and one on the right (for example, parts HA from the left and right elbows Hj, or parts HB from the left and right knees Hj, etc.), this means that at least one part of the person H, either on the left or right, is hidden by the insertion slot 12. Here, the object recognition unit 2 and the posture recognition unit 3 may recognize that a part of person H has disappeared from image G by determining whether or not the position P in three-dimensional space of a part of person H (such as the part HA from the elbow Hj to the knee Hj, or the part HB from the knee Hj to the knee Hj) recognized by the posture recognition unit 3 described above is located on the side of a specified area D, for example, below the line segment which is the input port 12 recognized by the object recognition unit 2 described above, in the same image G.

[0082] In addition, in the present invention, "when the plate 13 is in a predetermined position" means that the plate 13 which rotates or swings about the support shaft is in a predetermined rotational position or a predetermined swing position. Here, among the specified positions of the plate 13, in particular, a specified position (such as a specified rotation position) where the distance in a side view between the tip of the plate 13 and the upper rear end of the bottom of the above-mentioned garbage bin 22b (the lower end of the drop-in opening 12) is less than a specified value (where the tip of the plate 13 is close to the lower end of the drop-in opening 12) can also be said to be a dangerous position.

[0083] Furthermore, the "predetermined area D" in the present invention is an area of ​​a predetermined shape (approximately rectangular or cubic, etc.) and a predetermined size at a predetermined position in three-dimensional space, and can be said to be an area of ​​a predetermined shape (approximately rectangular or square, etc. in a strict planar view, and a curved, approximately rectangular or approximately trapezoidal, etc. when viewed from the camera 11 side (in other words, the shape of the predetermined area D reflected in the image G)) and a predetermined width near the lower end of the insertion port 12 when viewed in a planar view (or when viewed from the camera 11 side), and at the same time, an area of ​​a predetermined shape (approximately rectangular or approximately square, etc.) and a predetermined width near the insertion port 12 when viewed from the side. The position and size of this specified area D are not particularly limited, but for example, if the specified area D is approximately rectangular in shape in actual three-dimensional space, its frontmost surface may be located in a position that contacts the lower end of the insertion port 12 in a plan view (or a position several centimeters from the lower end), and its bottommost surface may be located several tens of centimeters (such as 10 cm to 70 cm) from the ground in a side view, and its size may be, for example, a front-to-back length of 10 cm to 100 cm (such as about 20 cm or about 50 cm), a top-to-bottom length of 30 cm to 300 cm (such as about 60 cm or about 150 cm), and a left-to-right length of 60 cm to 600 cm (such as about 240 cm or about 300 cm).

[0084] In addition, the predetermined area D, the forefront of which is in a position adjacent to the lower end of the insertion port 12 in a plan view, can also be said to be a danger area. In the present invention, "the entire person H in image G falls within the specified area D" literally means, with respect to the entire person H and the specified area D within the same image G, that the entire person H captured in that image G falls within the specified area D, and it can be said that the entry of the entire person H into the specified area D is recognized by the object recognition unit 2.

[0085] Here, the object recognition unit 2, etc. may recognize that the entire person H has entered the specified area D by checking whether or not the position P in three-dimensional space of the entire person H recognized by the object recognition unit 2, etc. described above, for the entire person H and the specified area D in the same image G, is within the region of the specified area D that was recognized by the object recognition unit 2, etc. For example, if the position P of the entire person H in three-dimensional space in image G is the center of gravity of the area (figure of a specified shape) occupied by person H, or the position of the waist joint recognized by the posture recognition unit 3, then the question is whether or not that position P is within a specified area D actually shown in image G; if the position P of the entire person H in three-dimensional space is the position of the foot (ball of the foot, etc.) that is in contact with the ground on which person H is standing, then the question is whether or not that position P is within the area obtained by projecting the specified area D onto the ground. In this invention, "stopping the introduction of plate 13" means, when any of the above-mentioned cases <1-1> to <1-3> occurs, stopping plate 13 so as to maintain its position (rotating position or swinging position) at that time, stopping plate 13 when the tip of plate 13 is slightly away from the lower end of feeding inlet 12 from the position of plate 13 at that time, or stopping plate 13 so that a worker or the like can manually move the tip of plate 13 away from the lower end of feeding inlet 12 while maintaining the position of plate 13 at that time.

[0086] The control system 14 may determine that the above-mentioned object detection system 1 is unable to detect the cardboard box B' and / or umbrella B'' in the above-mentioned image G in either of the following cases: <2-1> when the area of ​​the cardboard box B' and / or umbrella B'' in the above-mentioned image G occupies a first proportion (e.g., 30%) or more of the area of ​​the image G; and <2-2> when there is an abnormality in the above-mentioned image G. Here, with regard to the case in <2-1> of the present invention where "the area of ​​cardboard B' and / or umbrella B" in image G occupies 30% or more of the area of ​​image G (in other words, the area ratio of cardboard B' and / or umbrella B" to image G is 30% or more)", this also includes cases where the area of ​​cardboard B' and / or umbrella B" in image G occupies 40% or more of the area of ​​image G (the area ratio is 40% or more...), 50% or more of the area of ​​image G, 60% or more of the area of ​​image G, 70% or more of the area of ​​image G, 80% or more of the area of ​​image G, 90% or more of the area of ​​image G, or 100% of the area of ​​image G.

[0087] In addition, in the present invention, "the area of ​​cardboard B' and / or umbrella B" in image G" refers to the number of pixels (number of pixels) within the area of ​​cardboard B' and / or umbrella B" which has the same resolution in image G, which has a predetermined resolution (dpi, dots per inch) recognized by the object recognition unit 2 described above, or it can be said to be calculated from that number of pixels. Furthermore, in the present invention, the "area of ​​image G" is the product of the number of vertical pixels (number of pixels) of image G, which is the above-mentioned resolution, and the number of horizontal pixels (number of pixels), or it can be said to be calculated from that number of pixels. Therefore, in the present invention, the "area ratio of cardboard B' and / or umbrella B" to image G" can be said to be the value obtained by dividing the number of pixels, etc. within the area of ​​cardboard B' and / or umbrella B" having a specified resolution by the product, etc., of the number of vertical pixels and the number of horizontal pixels of image G having the same resolution, multiplied by 100.

[0088] Here, if position P of cardboard B' and / or umbrella B", etc. in image G is a position that covers a specified area D in image G or is near the specified area D, it can be said that even if the above-mentioned area rate is low, it is likely to be related to whether or not object B is incapable of being detected by object detection system 1. Therefore, if the area of ​​cardboard B' and / or umbrella B" in the specified area D accounts for a second proportion (e.g., 30%) or more of the area of ​​the specified area D (condition <2-1'>), it may be determined that it is incapable of being detected by the above-mentioned object detection system 1. In the present invention, "if there is an abnormality in image G" means a situation in which the object detection system 1 is unable to detect object B normally due to the condition of image G, including, for example, a situation in which the entire image G becomes completely black or completely white, a situation in which the camera 11 capturing image G breaks down, or a situation in which the cable between the camera 11 and the control system 14 is broken.

[0089] In addition, if the position of the camera 11 becomes abnormal, such as when the optical axis of the camera 11 becomes misaligned, the control system 14 may output a notification to a worker or the like that a camera misalignment has been detected. In addition to controlling plate 13, control system 14 may also control the on / off switching of camera 11, the direction of the optical axis of the lens in camera 11, and the opening and closing of insertion port 12.

[0090] <Other> The present invention is not limited to the above-described embodiment. The individual components or the overall structure, design, shape, dimensions, weight, etc. of the object detection system 1 and the garbage collection vehicle 10 can be appropriately changed in accordance with the spirit of the present invention. The object detection system 1 has the object recognition unit 2 and the posture recognition unit 3 described above, and also has the object tracking unit 4, and these object recognition unit 2, posture recognition unit 3, and object tracking unit 4 may always perform object recognition, posture recognition, and object tracking simultaneously, but conversely, there may be cases where only one of object recognition, posture recognition, and object tracking is performed without all of these units 2 to 4 functioning simultaneously, or where only two of object recognition, posture recognition, and object tracking are performed. Also, the object detection system 1 has only the object recognition unit 2 and the posture recognition unit 3, and may always perform object recognition and posture recognition simultaneously, but conversely, there may be cases where these units 2 and 3 are not functioning simultaneously, and where only one of object recognition and posture recognition is performed.

[0091] Furthermore, the object detection system 1 may first perform one or two of the above-mentioned object recognition, posture recognition, and object tracking, and then perform the remaining two or one (for example, perform two of object recognition and posture recognition simultaneously, and then perform the remaining object tracking.) Furthermore, the object detection system 1 may have only the object recognition unit 2 and posture recognition unit 3, and first perform one of object recognition and posture recognition, and then perform the remaining one. The object detection system 1 may have an integrated unit including the object recognition unit 2 and the attitude recognition unit 3, and also an object tracking unit 4, or may have an integrated unit including two of these units 2 to 4. Moreover, the object detection system 1 may have only the object recognition unit 2 and the attitude recognition unit 3, which are integrated together.

[0092] In addition to the above-mentioned object recognition unit 2, posture recognition unit 3, and object tracking unit 4, the object detection system 1 has a pre-processing unit (not shown) that performs at least one of the following: extracting the contour He and area of ​​person H etc. in image G, removing noise and distortion from image G, emphasizing the contour He of person H in image G, and adjusting the brightness and color tone (tone, hue, and saturation) of image G. Based on the contour He and area of ​​person H etc. extracted from image G by this pre-processing unit, the above-mentioned object recognition unit 2 may at least recognize person H in image G as an actual person, the posture recognition unit 3 may recognize the posture of person H in image G, and the object tracking unit 4 may at least track the movement of person H in image G. Furthermore, even when the object detection system 1 has a pre-processing unit, it can be said that the object recognition unit 2 recognizes the position P of the entire person H in three-dimensional space "based on the contour He of the person H in image G," the posture recognition unit 3 recognizes a specific joint Hj of the person H "based on the contour He of the person H in image G" and recognizes the position P in three-dimensional space of the parts HA, HB, etc. beyond the recognized specific joint Hj, and the object tracking unit 4 recognizes the movement path L of the entire person H in three-dimensional space, and / or the movement path L of the parts HA, HB, etc. beyond the specific joint Hj in three-dimensional space "based on the contour He of the person H in image G." The object detection system 1 has exemplified any one of the cases <1-1> to <1-3> as the conditions for stopping the above-mentioned sweeping of the plate 13, but it is sufficient to stop the above-mentioned sweeping of the plate 13 in at least any one of the conditions <1-1> and <1-3>, and the determination that detection is impossible in any one of the cases <2-1> and <2-2> may be omitted.

[0093] The object detection system 1 has the object recognition unit 2 and posture recognition unit 3 described above, as well as an object tracking unit 4, and as long as an image G containing a person H or the like is input, it does not need to have a camera that captures the image G (such as the camera 11 of the garbage collection truck 10) or the garbage collection truck 10. The object detection system 1 does not register the color to be detected in advance, and can detect the object B to be detected without using color information (hue, saturation, etc.) of the object B to be detected, but can determine whether the object B to be detected is wearing an accessory of a distinctive color. [Industrial Applicability]

[0094] The object detection system of the present invention detects objects by recognizing objects such as people and their posture in images, and by tracking other objects such as people, and can therefore be used to detect dangerous human behavior and dangerous conditions near the opening of a garbage truck, and can also be used to detect objects other than garbage trucks, such as dump trucks, sprinkler trucks, concrete pumps, mixer trucks, tailgate lifters, tanker trucks, water trucks, powder and granular material transport vehicles, detachable body vehicles, loader transporters, rail containers, labor-saving firefighting devices, forestry and biomass related products, facilities, equipment and devices, multi-story parking devices, coin parking devices, waterproof boards, etc., by recognizing objects such as people and their posture in images and tracking objects such as people near the vicinity of any vehicle (vehicle), device, product, facility, or equipment. The garbage collection truck of the present invention can be used by any local government or organization, regardless of the color of the gloves or work clothes worn by the workers operating the garbage collection truck. In addition, when members of the public other than workers throw garbage into the garbage collection truck's opening, the truck can be used to prevent dangerous actions or respond to dangerous situations by all people near the opening, regardless of the color of clothing they are wearing. [Explanation of symbols]

[0095] 1. Object Detection System 2 Object recognition section 3 Posture recognition section 4. Object Tracking Unit 10. Garbage Truck 11 Camera 12 Inlet 13 Plate 14 Control System G Image B Object H people He Human outline (edge) Hj: A specific joint in a human body P Position in 3D space L Path of movement in three-dimensional space D. Designated areas (danger areas, etc.) S shield

Claims

1. A garbage collection truck using an object detection system that detects an object in an image captured by a camera, the object including a person, the object having an object recognition unit that at least recognizes the person in the image as an actual person, and a posture recognition unit that recognizes the posture of the person in the image, the object including an obstruction, the object recognition unit of the object detection system also recognizing the obstruction in the image as an actual obstruction, The garbage collection vehicle includes a camera that captures the image, an inlet that is captured by the camera and through which garbage can be thrown, a plate that can push garbage into the inlet, and a control system that controls the plate; The control system includes: When a part of a person in the image enters the insertion slot, or when the entire person in the image enters a predetermined area, The scraping of the plate is stopped, The control system includes: When an area of ​​the obstruction in the image occupies a first proportion or more of an area of ​​the image, A garbage collection vehicle characterized in that it is determined that the object cannot be detected by the object detection system.

2. the object recognition unit recognizes a position of the entire person in a three-dimensional space based on a contour of the person in the image; The garbage collection truck according to claim 1, characterized in that the posture recognition unit recognizes a predetermined joint of the person based on an outline of the person in the image, and recognizes a position in three-dimensional space of a portion beyond the recognized predetermined joint.

3. 3. The garbage collection vehicle according to claim 1, further comprising an object tracking unit that tracks at least the movement of a person within the image.

4. an object tracking unit that tracks at least a movement of a person in the image; The garbage collection truck according to claim 2, characterized in that the object tracking unit recognizes the movement path of the entire person in three-dimensional space and / or the movement path of the portion beyond the specified joint in three-dimensional space based on the outline of the person in the image.

5. 5. The garbage collection vehicle according to claim 1, wherein the shielding object is a cardboard box and / or an umbrella.

Images