Moving object display system, moving object detection method

Abstract

The objective of this invention is to provide a moving object display system for efficiently detecting moving objects. [Solution] A moving object display system 1 comprising a camera 3, a control unit 11, and a display unit 5, wherein the camera 3 captures a front image including a plurality of bearings 50 and a rear image captured after the front image from the same direction, and the control unit 11 displays the difference between the front image and the rear image on the display unit 5 using a heat map.

Description

Technical Field

[0001] The present invention relates to a moving object display system and a moving object detection method.

Background Art

[0002] During a picking operation or the like, the objects loaded in a container or the like may collapse, causing damage to the objects. For example, since the bearings (objects) before heat treatment are easily damaged, it was necessary to search for the damaged bearings when the load collapsed and select them as defective products.

[0003] For example, Patent Document 1 discloses a detection method for detecting the collapse of rectangular parallelepiped articles stacked on a pallet. In Patent Document 1, images of the articles on the pallet before and after transfer are captured from above to determine the degree of change, or the overall shape of the article group is estimated by detecting the upper surface shape of the articles on the pallet by three-dimensional measurement means, thereby detecting the collapse of the articles.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] When the objects are regularly arranged as in Patent Document 1, if the system can detect the presence or absence of load collapse, it is easy for the operator to identify the objects that have moved due to load collapse. However, for example, when the bearings as objects are randomly stacked in a container, even if the system can determine that load collapse has occurred, it is difficult for the operator to determine which bearing has moved due to load collapse.

[0006] In view of the above circumstances, the present invention aims to efficiently detect moving objects. [Means for solving the problem]

[0007] To solve the above problems, the present invention provides a moving object display system comprising an imaging mechanism, a control unit, and a display unit, wherein the imaging mechanism captures a preceding image containing a plurality of objects and a succeeding image captured after the preceding image from the same direction, and the control unit displays the difference between the preceding image and the succeeding image on the display unit using a heat map.

[0008] As described above, the areas that have changed can be displayed in different colors using a heatmap. Therefore, by checking the heatmap, operators can quickly understand which objects have been displaced, and efficiently detect moved objects in a short amount of time.

[0009] The control unit may create a superimposed image by overlaying the heatmap with the rear image or an image captured by the imaging mechanism from the same direction as the rear image, and display it on the display unit. By overlaying the heatmap with the captured image, it becomes easier to visually understand the approximate location of the darkly colored areas in the heatmap in actual space. Therefore, moving objects can be detected more efficiently and in a shorter amount of time.

[0010] Furthermore, the present invention relates to a moving object display system comprising a detection mechanism, a control unit, and a display unit, wherein the detection mechanism detects the positions of multiple objects at different timings, and the control unit displays the displacement amounts of the multiple objects using a heat map based on the detection results of the detection mechanism. A heat map can also be created based on the object position detection results by the detection mechanism. This allows for the efficient detection of moving objects in a short time, similar to the above.

[0011] Furthermore, the present invention may include an imaging step of capturing a previous image containing multiple objects and a subsequent image captured after the previous image from the same direction; a display step of displaying the difference between the previous image and the subsequent image using a heat map; and a detection step of using the heat map to detect the objects that have moved between the time the previous image was captured and the time the subsequent image was captured. [Effects of the Invention]

[0012] According to the present invention, moving objects can be detected efficiently. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic side view showing the configuration of a moving object display system according to one embodiment of the present invention. [Figure 2] This is a schematic diagram showing a pre-picking image of the container's interior, captured by a camera. [Figure 3] This is a schematic diagram showing a post-picking image of the container, captured by a camera inside the container. [Figure 4] This is a schematic diagram showing an example of a heatmap. [Figure 5] This is a schematic diagram showing an example of an overlaid image. [Figure 6] This flowchart illustrates the procedure for detecting moving objects using a moving object display system. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below with reference to the drawings. In each figure, the same or corresponding parts are denoted by the same reference numerals, and redundant explanations will be simplified or omitted as appropriate.

[0015] Figure 1 is a schematic side view showing the configuration of a moving object display system 1 according to one embodiment of the present invention.

[0016] As shown in FIG. 1, the moving object display system 1 includes a robot arm 2, a camera 3 as an imaging mechanism, a gripping position detection camera 4, a display unit 5, a control unit 11, a gripping position calculation control unit 12, and the like.

[0017] The robot arm 2 has a gripping arm 2a for gripping a bearing as an object. The robot arm 2 is an industrial robot and can place the gripping arm 2a at a desired position to grip or release the gripped bearing.

[0018] In the container b, the bearings before heat treatment conveyed from the previous process are stacked in a scattered manner. Further, the bearings in the container 6 are picked up by the robot arm 2 and conveyed to the subsequent process.

[0019] The gripping position detection camera 4 is provided vertically above the container 6. The gripping position detection camera 4 is composed of a plurality of three-dimensional sensors and detects the position where the gripping arm 2a grips the bearing in the container 6.

[0020] The camera 3 images the container 6 and the bearings stacked in the container 6 from an obliquely upper direction. The camera 3 may be configured to image from vertically above the container 6 as long as there is no hindrance to the resolution of the captured image and the detection by the gripping position detection camera 4. The camera 3 in this embodiment is a two-dimensional camera.

[0021] The gripping position calculation control unit 12 controls the robot arm 2 based on the detection result by the gripping position detection camera 4, and moves the gripping arm 2a to a desired position to perform the bearing picking operation.

[0022] The control unit 11 performs control for detecting a moving object, such as displaying a superimposed image on the display unit 5 based on the image captured by the camera 3 (details will be described later). The display unit 5 in this embodiment is a display.

[0023] In the manufacturing process of such bearings, the bearings, which are loosely stacked in container 6, may shift during or after picking, causing damage to the fallen bearings or bearings that collide with the fallen bearings. However, it is difficult to identify the bearings that have shifted due to the shifting of the load from among the bearings haphazardly placed in container 6. In particular, when the object in question is a bearing, it is not possible to determine whether a bearing is damaged without checking the inner and outer rings around their circumference, and this determination of whether the bearing is damaged takes time.

[0024] Next, a method for detecting moving objects using the moving object display system 1 of this embodiment will be explained with reference to Figures 2 to 5.

[0025] Figure 2 schematically shows a first image, which is an example of an image taken inside the container 6 by camera 3. As shown in Figure 2, multiple bearings 50 are loosely stacked inside the container 6. The first image in this embodiment is an image taken before the picking operation. Figure 3 schematically shows a second image, which is an image taken inside the container 6 by camera 3 after the bearings 50 have been picked from Figure 2. These first and second images were taken from the same direction by a fixed camera 3.

[0026] In this embodiment, the control unit processes the image of the previous image in Figure 2 and the image of the next image in Figure 3 to create difference data and a difference image. The generated difference image is then displayed on the display unit as a heatmap. Thus, the control unit includes an image processing device that processes the previous and next images to generate a difference image and a heatmap.

[0027] Figure 4 schematically shows an example of a heatmap displayed in this way, and is a heatmap created from the before image in Figure 2 and the after image in Figure 3. This heatmap is an image that evaluates the intensity of the color of each pixel at each position in the before image and after image, and displays it in stages of color based on the difference. In other words, pixels with a large change in color intensity are shown in warm colors, and pixels with a small change or no change are shown in cool colors, each in stages. Warm and cool colors are just examples; it is sufficient to be visually distinguishable, such as by the intensity of a single color. In the actual image, some of the bearings 50A, 50B, and 50C shown in Figure 4 are shown in warm colors, indicating that these bearings 50 moved before and after picking. The bearings 50 other than those that moved are also displayed in color, but are displayed in a lighter cool color than bearings 50A to 50C. The reasons why these bearings 50 are colored in the heat map include errors in the camera 3 and control unit 11 due to disturbances, and differences in how light hits the bearings 50 when the previous and subsequent images are captured.

[0028] As shown in Figure 4, by displaying the difference between the previous and subsequent images using a heatmap, it is possible to instantly and visually recognize which parts have changed. Therefore, when a load collapse occurs, the worker can immediately identify the bearing 50 that has moved due to the collapse by checking the heatmap. As a result, by checking for damage to the moved bearing 50 or the surrounding bearings 50, defective products can be easily sorted out, improving the efficiency of the defective product sorting work and reducing the burden on the worker.

[0029] As shown in Figure 5, a superimposed image may be created by overlaying the heatmap and the captured image, and displayed on the display unit. In the case of an image that only shows the difference, as in Figure 4, it may be difficult to determine which bearing 50 in the actual container 6 corresponds to the darkly colored bearing in the image, for reasons such as the small number of areas that have changed. However, by overlaying the heatmap and the captured image as shown in Figure 5, it becomes easy to understand the approximate location of the bearing 50 displayed as a difference within the container 6, and it becomes easier to identify the bearing 50 that has moved within the container 6. The image to be overlaid with the heatmap may be the image used to create the heatmap, or it may be the current image if the camera 3 is continuously capturing images in real time. Alternatively, it may be an image taken from the same direction by another camera 3. Note that in Figure 5, the container 6 is shown with a dotted line for convenience.

[0030] Next, the procedure for detecting a moving object using the moving object display system of this embodiment will be explained with reference to the flowchart in Figure 6.

[0031] As shown in Figure 6, first, before performing the picking operation, the camera 3 captures a front image, which the control unit 11 acquires (step S1). Then, the robot arm 2 picks up the bearing 50 inside the container 6 (S2).

[0032] After picking, the camera 3 captures a rear image, which the control unit 11 acquires (S3). The control unit 11 then creates difference data and a difference image from the front image and the rear image (S4), and determines whether or not the item has fallen or collapsed from the difference data (S5). Steps S1 and S3 are the imaging process by the camera 3.

[0033] If the amount of change in the differential data exceeds a preset threshold, it is determined that a drop or collapse has occurred. This determination based on the amount of change and threshold may be based on the amount of change of any single pixel in the differential data exceeding the threshold, or on the number of pixels exceeding the threshold being considered as having occurred. Alternatively, it may be based on the number of pixels exceeding the threshold within a predetermined adjacent range in the captured image being considered as having occurred.

[0034] If it is determined that a drop or shifting of the load has occurred, the picking operation is stopped, and the control unit 11 creates a heatmap from the aforementioned difference images. Then, the control unit 11 generates a superimposed image by overlaying this heatmap with the current image captured by the camera 3 (S7).

[0035] The control unit 11 displays the superimposed image on the display unit 5 (display step S8). The operator checks the bearing 50 based on this superimposed image and detects the damaged bearing 50 as described above (detection step S9).

[0036] On the other hand, if it is determined in step S6 that there is no load collapse or fall, the above steps S1 to S5 are repeated for each picking operation.

[0037] The above description illustrates a case where a heatmap is created by capturing a front and back image using an imaging mechanism. However, the present invention is not limited to this, and a detection mechanism that detects the position of an object may be used instead of the imaging mechanism. Examples of detection mechanisms include laser sensors, ultrasonic sensors, and millimeter-wave radar sensors. In this case, multiple detection mechanisms are arranged (for example, in a grid) to detect the distance from the detection mechanism to the object or container at each position before and after picking, and calculate the change in that distance. The amount of vertical displacement at each position on a predetermined plane can then be converted into a heatmap and displayed on the display unit. This plane may be, for example, a plane parallel to the mounting surface of the bearing 50 of the container 6, or a plane from which the container 6 is viewed from diagonally above, as shown in Figures 2 and 3.

[0038] A three-dimensional camera may also be used as the detection mechanism. In this case, similar to the imaging mechanism described above, the amount of vertical displacement at each position on a predetermined plane can be converted into a heat map and displayed on the display unit. Alternatively, a three-dimensional camera can be used as the imaging mechanism to capture the aforementioned previous and subsequent images and create difference data (difference image) and a heat map. In this case, in addition to the difference data between the previous and subsequent images, the amount of vertical displacement of this image may be used to determine whether the load has fallen or collapsed. For example, if either the difference data between the previous and subsequent images or the amount of vertical displacement exceeds a threshold, it may be determined that the load has collapsed or fallen. Alternatively, the amount of change may be calculated using 3D point cloud data obtained from a three-dimensional camera, and the amount of change may be used to determine whether the load has fallen or collapsed.

[0039] The above description illustrates an embodiment in which the presence or absence of a fall or load collapse is determined using a single camera 3. However, the determination may also be made using multiple imaging mechanisms with different imaging directions. For example, the moving object display system 1 may have three cameras 3 with different imaging directions, each camera 3 capturing a front image and a rear image, and the control unit may create difference data and a difference image. If any one of these three difference data exceeds a threshold, it may be determined that a fall or load collapse has occurred. This prevents overlooking falls and load collapses and improves bearing quality. Alternatively, if, for example, two or more difference data exceed a threshold, it may be determined that a fall or load collapse has occurred. This improves the accuracy of the fall and load collapse detection.

[0040] Although embodiments of the present invention have been described above, the present invention is not limited to the embodiments described above, and various modifications can be made without departing from the spirit of the present invention.

[0041] In the embodiments described above, a bearing was used as the object, but the object of the present invention is not limited to this. Also, in the embodiments described above, the picking operation was performed before and after the preceding and succeeding images were taken, but the present invention is not limited to this. In other words, by capturing preceding and succeeding images in the time period before and after an operation in which the object may move, the moving object can be displayed and detected. [Explanation of Symbols]

[0042] 1. Moving Object Display System 3. Camera (imaging mechanism) 5 Display section 6 containers 11 Control Unit 12. Gripping position calculation control unit 50 Bearings (objects)

Claims

1. A moving object display system comprising an imaging mechanism, a control unit, and a display unit, The imaging mechanism captures a preceding image containing multiple objects and a subsequent image captured after the preceding image from the same direction. The control unit is characterized by displaying the difference between the previous image and the subsequent image on the display unit using a heat map, thereby providing a moving object display system.

2. The moving object display system according to claim 1, wherein the control unit creates a superimposed image by superimposing the heatmap and the rear image or an image captured by the imaging mechanism from the same direction as the rear image, and displays it on the display unit.

3. A moving object display system comprising a detection mechanism, a control unit, and a display unit, The detection mechanism detects the positions of multiple objects at different timings, The moving object display system is characterized in that the control unit displays the displacement amounts of multiple objects using a heat map based on the detection results of the detection mechanism.

4. An imaging step in which a preceding image containing multiple objects and a subsequent image captured after the preceding image are captured from the same direction, A display step that shows the difference between the aforementioned previous image and the aforementioned subsequent image using a heat map, A method for detecting moving objects, characterized by including a detection step of detecting the movement of the object between the time of capturing the previous image and the time of capturing the subsequent image using the heat map.

Images