Periphery monitoring system, periphery monitoring method, and periphery monitoring device

Abstract

To provide a periphery monitoring system, a periphery monitoring method, and a periphery monitoring device capable of appropriately monitoring a possibility of collision between a working machine and an obstacle including a peripheral person.SOLUTION: A periphery monitoring system 1 comprises: a camera 2 that captures a forklift and a periphery of the forklift; a recognition unit 12 that recognizes the forklift and an obstacle of a periphery of the forklift from an image captured by the camera 2; a calculation unit 14 that calculates a movement direction of the forklift and a movement direction of the obstacle from a recognition result of the recognition unit 12; and a determination unit 16 that determines a possibility of collision between the forklift and the obstacle based on the movement direction of the forklift and the movement direction of the obstacle calculated by the calculation unit 14.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a perimeter monitoring system, a perimeter monitoring method, and a perimeter monitoring device. [Background technology]

[0002] For example, a technique for detecting a moving object in a detection area is known for work in a warehouse performed using an industrial vehicle such as a forklift (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-033701 Summary of the Invention [Problem to be solved by the invention]

[0004] Work machines travel in places such as warehouses or factories where there are obstacles, including people, in the vicinity. If an alarm were to be output upon detecting all obstacles, including people, in the vicinity of the work machine, the alarm would be issued frequently. Therefore, it is desirable to appropriately determine the possibility of a collision between the work machine and nearby obstacles, including people.

[0005] An aspect of the present disclosure aims to provide a perimeter monitoring system, a perimeter monitoring method, and a perimeter monitoring device that appropriately monitor the possibility of a collision between a work machine and an obstacle, including a person in the vicinity. [Means for solving the problem]

[0006] According to an aspect of the present disclosure, there is provided a perimeter monitoring system comprising: a photographing unit that photographs a work machine and its surroundings; a recognition unit that recognizes the work machine and obstacles in its surroundings from the images photographed by the photographing unit; a calculation unit that calculates the movement direction of the work machine and the movement direction of the obstacle from the recognition results of the recognition unit; and a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the movement direction of the work machine and the movement direction of the obstacle calculated by the calculation unit.

[0007] According to an aspect of the present disclosure, there is provided a periphery monitoring method that includes photographing a work machine and its surroundings, recognizing the work machine and obstacles in its surroundings from the photographed image, calculating the movement direction of the work machine and the movement direction of the obstacle from the recognition results, and determining the possibility of a collision between the work machine and the obstacle based on the calculated movement direction of the work machine and the movement direction of the obstacle.

[0008] According to an aspect of the present disclosure, there is provided a periphery monitoring device comprising: a recognition unit that recognizes a work machine and obstacles in the vicinity of the work machine from video footage of the work machine and its surroundings; a calculation unit that calculates the direction of movement of the work machine and the direction of movement of the obstacle from the recognition results of the recognition unit; and a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the direction of movement of the work machine and the direction of movement of the obstacle calculated by the calculation unit. [Effects of the Invention]

[0009] According to aspects of the present disclosure, a perimeter monitoring system, a perimeter monitoring method, and a perimeter monitoring device are provided that can appropriately monitor the possibility of a collision between a work machine and an obstacle, including a person in the vicinity. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a schematic diagram illustrating an example of a location monitored by a perimeter monitoring system according to an embodiment of the present invention. [Figure 2]1 is a functional block diagram illustrating an example of a perimeter monitoring system according to an embodiment of the present invention. [Figure 3] FIG. 10 is a schematic diagram illustrating a recognition result of a forklift. [Figure 4] 3 is a schematic diagram illustrating an example of a monitoring image output by the perimeter monitoring system according to the present embodiment. FIG. [Figure 5] FIG. 2 is a schematic diagram illustrating an example of the positional relationship between a forklift and a person. [Figure 6] FIG. 6 is a schematic diagram illustrating the possibility of a collision between the forklift and the person in the positional relationship shown in FIG. 5. [Figure 7] FIG. 10 is a schematic diagram illustrating another example of the positional relationship between the forklift and the person. [Figure 8] FIG. 8 is a schematic diagram illustrating the possibility of a collision between the forklift and the person in the positional relationship shown in FIG. 7. [Figure 9] 4 is a flowchart illustrating an example of processing of the perimeter monitoring system according to the present embodiment. [Figure 10] FIG. 1 is a block diagram illustrating an example of a computer system according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings, but the present disclosure is not limited thereto. The components of each embodiment described below can be combined as appropriate. In addition, some components may not be used.

[0012] In this embodiment, a forklift 100 (see FIG. 4) driven by power from a battery will be described as an example of the work machine, but the work machine is not limited to this. For example, the work machine may be a wheel loader driven by power from a battery. Furthermore, the work machine is not limited to one driven by electricity, and may be a forklift driven by hydraulic power, etc.

[0013] <Perimeter monitoring system> The perimeter monitoring system 1 monitors the perimeter of the forklift 100 at a work site of the forklift 100. The perimeter monitoring system 1 appropriately monitors the possibility of a collision between the work machine and obstacles, including nearby people. In the following explanation, people will be described as an example of an obstacle. The perimeter monitoring system 1 monitors the perimeter of the forklift 100 in places where people 110 (see FIG. 4) may be working or moving around the forklift 100, such as an inspection area or a material handling area in a warehouse. The perimeter monitoring system 1 also monitors the perimeter of the forklift 100 in places with poor visibility on the travel path along which the forklift 100 and people 110 move, such as corners of a travel path in a factory.

[0014] FIG. 1 is a schematic diagram illustrating an example of a location monitored by a perimeter monitoring system 1 according to this embodiment. FIG. 1 shows a parking area S for a forklift 100 in a warehouse as an example of a location to be monitored. The warehouse has, for example, a height h [m] from the floor to the ceiling and a depth d [m]. In the warehouse, the forklift 100 is parked at the parking area S and lifts and lowers cargo. A person 110 also works and moves around in the warehouse. In the example shown in FIG. 1, the perimeter monitoring system 1 monitors the periphery of the forklift 100 in the vicinity of the parking area S.

[0015] 2 is a functional block diagram showing an example of the perimeter monitoring system 1 according to this embodiment. In this embodiment, the perimeter monitoring system 1 determines the possibility of a collision between the forklift 100 and the person 110 based on the moving direction of the forklift 100 and the moving direction of the person 110 around the forklift 100.

[0016] The perimeter monitoring system 1 includes a camera 2 as a photographing unit, a monitor 3, a vibration unit 4 as an alarm unit, a video memory unit 5, a learning memory unit 6, and a perimeter monitoring device 10. The camera 2, the monitor 3, the vibration unit 4, the video memory unit 5, the learning memory unit 6, and the perimeter monitoring device 10 are connected to each other wirelessly or via a wire so as to be able to communicate data.

[0017] Camera 2 is capable of capturing images of forklift 100 and person 110. Camera 2 is placed in a location, such as a warehouse, around forklift 100 where person 110 may be working or moving. In the example shown in FIG. 1, camera 2 is placed near the ceiling of the warehouse and captures images of parking lot S and the area around parking lot S. Camera 2 is placed on the path along which forklift 100 and people move, such as the corner of a path in a factory. Camera 2 transmits video data of the captured images to perimeter monitoring device 10. Camera 2 is capable of capturing images over a range of, for example, several meters.

[0018] In this embodiment, camera 2 is a monocular camera. Camera 2 has an optical system and an image sensor. The image sensor includes a CCD (Couple Charged Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The video captured by camera 2 includes multiple images per unit time depending on the frame rate.

[0019] The monitor 3 is placed, for example, in a warehouse or a remote monitoring room away from the travel path. The monitor 3 includes a flat panel display such as a liquid crystal display (LCD) or an organic electroluminescence display (OLED). The monitor 3 displays the video transmitted from the perimeter monitoring device 10. In this embodiment, as shown in FIG. 4, the monitor 3 displays a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 superimposed on the video 200 captured by the camera 2. The movement area 102 and the movement area 112 will be described later. The monitor 3 may be placed inside the forklift 100.

[0020] The vibration unit 4 is a vibration generating device. The vibration unit 4 generates vibrations to alert the operator. The vibration unit 4 generates vibrations in response to a control signal from the periphery monitoring device 10. In this embodiment, the vibration unit 4 generates vibrations when the periphery monitoring device 10 determines that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, the vibration unit 4 is worn on the body of the operator of the forklift 100. The vibration unit 4 may be installed in another position, such as on the seat of the driver's seat, as long as it can make the operator in the driver's seat aware of the vibrations.

[0021] After generating vibration, the vibration unit 4 may stop the vibration in response to a control signal from the periphery monitoring device 10. After generating vibration, the vibration unit 4 may stop the vibration based on an operation by an operator to stop the vibration. After generating vibration, the vibration unit 4 may stop the vibration after a predetermined time has elapsed.

[0022] In addition to the vibration unit 4, or instead of the vibration unit 4, another alarm unit (not shown) may be provided. The alarm unit is installed near the driver's seat of the forklift 100. The alarm unit may be installed in another location as long as it can issue an alarm to the operator in the driver's seat or around the forklift 100. The alarm unit uses, for example, a speaker or a lamp to emit a sound or light to warn the operator of the possibility of a collision, thereby alerting the operator. The alarm unit uses, for example, a display panel located in the driver's cab to display an image to warn the operator of the possibility of a collision, thereby alerting the operator.

[0023] The video storage unit 5 is a storage device that stores video data, etc., including data before and after the time when it is determined that there is a possibility of a collision. The video storage unit 5 stores video under the control of the video storage control unit 23. The video data to be stored will be described later. The video storage unit 5 uses, for example, at least one of non-volatile or volatile semiconductor memory such as RAM, ROM, flash memory, EPROM, EEPROM, magnetic disk, flexible disk, and magneto-optical disk.

[0024] The learning memory unit 6 is a storage device that stores learning image data, which is image data for learning. The learning memory unit 6 stores the learning image data under the control of the image storage control unit 24. The learning image data will be described later. The learning memory unit 6 uses, for example, at least one of non-volatile or volatile semiconductor memory such as RAM (Random Access Memory), ROM (Random Only Memory), flash memory, EPROM (Erasable Programmable Random Access Memory), EEPROM (Electrically Erasable Programmable Random Access Memory), a magnetic disk, a flexible disk, and a magneto-optical disk.

[0025] <Perimeter monitoring system control system> The periphery monitoring device 10 monitors the periphery of the forklift 100 and determines the possibility of contact between the forklift 100 and a person 110. If the periphery monitoring device 10 determines that there is a possibility of contact between the forklift 100 and a person 110, it outputs an alarm. A camera 2, a monitor 3, a vibration unit 4, a video memory unit 5, and a learning memory unit 6 are connected to the periphery monitoring device 10. The periphery monitoring device 10 includes a numerical calculation unit (processor) such as a CPU (Central Processing Unit). The periphery monitoring device 10 includes an image acquisition unit 11, a recognition unit 12, a tracking unit 13, a calculation unit 14, an image generation unit 15, a determination unit 16, an output control unit 21, an alarm output unit 22, a video storage control unit 23, an image storage control unit 24, and a storage unit 29.

[0026] The video acquisition unit 11 acquires video data of the video captured by the camera 2. The video data acquired by the video acquisition unit 11 includes the forklift 100 and the person 110 in the range to be monitored. The video acquisition unit 11 converts the acquired video data into image data for each frame and outputs it to the recognition unit 12. The video acquisition unit 11 stores the video data in the memory unit 29.

[0027] The recognition unit 12 recognizes the forklift 100 and the person 110 from the video data of the video acquired by the video acquisition unit 11. More specifically, the recognition unit 12 performs a recognition process for the forklift 100 and the person 110 on the image data of each frame extracted from the video data. The recognition unit 12 performs the recognition process using artificial intelligence (AI). The recognition unit 12 uses a learning model to recognize the forklift 100 and the person 110 as input data and outputs the recognition results.

[0028] The learning model is generated by learning the feature quantities of the forklift 100 and the person 110 from a plurality of learning image data. By performing machine learning using learning images including the forklift 100 and the person 110 as training data, a learning model is generated in which the feature quantities of the forklift 100 and the person 110 are input and the recognition results of the forklift 100 and the person 110 are output as data.

[0029] The training image data includes a forklift 100 and a person 110. The training image data includes, for example, the forklift 100 and the person 110 displayed at various positions in the image. The training image data includes, for example, the forklift 100 and the person 110 in various orientations and sizes. The training image data includes, for example, image data captured under various lighting conditions, weather conditions, seasons, and time periods. The training image data includes, for example, image data in which a portion of the forklift 100 and a portion of the person 110 are missing.

[0030] In this embodiment, the learning model may be trained using an average image obtained by averaging multiple images extracted from the video captured by the camera 2. Images extracted from the captured video may contain noise due to the forklift 100 and the person 110 being moving objects. By using the average image data as training image data, noise is reduced. The training image data includes, for example, average image data of an average image obtained by averaging multiple consecutive frame images to reduce noise.

[0031] The recognition unit 12 associates the detection frames and center points of the detection frames of the recognized forklift 100 and person 110 with the image on which the recognition process was performed, and stores them in the storage unit 29. In this embodiment, the recognition unit 12 associates the detection frame 101 and center point C1 of the detection frame 101 of the recognized forklift 100 (see FIG. 5 ) and the detection frame 111 and center point C2 of the detection frame 111 of the person 110 (see FIG. 5 ) with the image, and stores them in the storage unit 29. The detection frames of the recognized forklift 100 and person 110 are, for example, rectangular in shape surrounding the outlines of the forklift 100 and person 110.

[0032] FIG. 3 is a schematic diagram illustrating the recognition result of the forklift 100. FIG. 3 is a schematic diagram of an image of the forklift 100 viewed from directly above. Note that the image actually captured by the camera 2 is an image captured from diagonally above, as shown in FIG. 4. In the example of FIG. 3, the detection frame 101 of the forklift 100 is rectangular and surrounds the outer shape of the forklift 100. The center point of the detection frame 101 is designated as C1.

[0033] The tracking unit 13 tracks the forklift 100 and the person 110 recognized by the recognition unit 12. In this embodiment, the tracking unit 13 tracks the center point C1 of the forklift 100 and the center point C2 of the person 110 recognized by the recognition unit 12. The tracking unit 13 stores the tracking results in the memory unit 29.

[0034] The calculation unit 14 calculates the moving direction of the forklift 100 and the moving direction of the person 110 from the recognition result of the recognition unit 12. More specifically, the calculation unit 14 calculates the moving direction and speed of the forklift 100 and the person 110 recognized by the recognition unit 12. The calculation unit 14 stores the calculated moving direction and speed of the forklift 100 and the person 110 in the storage unit 29 in association with the image that has been subjected to the recognition process.

[0035] In this embodiment, the movement direction and speed of the forklift 100 are the movement direction and speed of the center point C1 of the detection frame 101 of the forklift 100. In this embodiment, the movement direction and speed of the person 110 are the movement direction and speed of the center point C2 of the detection frame 111 of the person 110. The movement direction and speed of the center point can be calculated by a known method using a Kalman filter.

[0036] The image generation unit 15 generates an image showing the detection frames, movement direction, and speed of the forklift 100 and the person 110 recognized by the recognition unit 12. The image generation unit 15 stores the generated image in the storage unit 29 in association with the image.

[0037] In this embodiment, the movement areas indicating the movement direction and speed of the forklift 100 and the person 110 are shown in the shape of a sector. The movement area of ​​the forklift 100 is the area through which the forklift 100 may move within a few seconds. The movement area of ​​the person 110 is the area through which the person 110 may move within a few seconds. By defining the movement area as a sector, errors in the movement direction are taken into consideration. The center line of the sector is a line extending from the center point along the movement direction. The radius is, for example, the distance traveled within a few seconds. The radius of the sector is determined according to the movement speed. The radius becomes longer as the movement speed increases. The central angle varies depending on the area of ​​the obstacle being monitored. In the case of the forklift 100, for example, the central angle is approximately 60°. In this embodiment, the central angle of the person 110 is, for example, approximately 10° because the area of ​​the person 110 is smaller than that of the forklift 100.

[0038] In this way, in this embodiment, the image generating unit 15 generates a rectangular image indicating the detection frame of the forklift 100 and the person 110, and a sector-shaped image indicating the movement area.

[0039] 3, the direction and speed of movement of the forklift 100 are indicated by a sector-shaped movement area 102 centered at a central point C1. The length of the sector-shaped movement area of ​​the forklift 100 in the front-to-rear direction is R1 [m]. The length R1 [m] is, for example, about several meters.

[0040] Fig. 4 is a schematic diagram illustrating an example of a monitoring image 200 output by the perimeter monitoring system 1 according to this embodiment. In the example shown in Fig. 4, an image showing a detection frame 101 of the forklift 100, an image showing a sector-shaped movement area 102 indicating the movement direction and speed of the forklift 100, an image showing a detection frame 111 of a person 110, and an image showing a sector-shaped movement area 112 indicating the movement direction and speed of the person 110 are superimposed and displayed on the image 200 captured by the camera 2.

[0041] When the judgment unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the image generation unit 15 may generate an image to be superimposed on the video 200, indicating the possibility of a collision using text, graphics, etc.

[0042] The determination unit 16 determines the possibility of a collision between the forklift 100 and the person 110 based on the moving direction of the forklift 100 and the moving direction of the person 110 calculated by the calculation unit 14. As a result, the determination unit 16 determines that there is no possibility of a collision, for example, even if the person 110 is present near the forklift 100, if the moving direction of the forklift 100 and the moving direction of the person 110 are opposite to each other.

[0043] In this embodiment, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when the moving direction of the forklift 100 and the person 110 intersect. For example, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when a line extending in the moving direction of the forklift 100 intersects with a line extending in the moving direction of the person 110. Furthermore, the determination unit 16 may make a determination taking into consideration the moving speed. For example, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when a line segment extending in the moving direction of the forklift 100 and having a length corresponding to the speed of the forklift 100 intersects with a line segment extending in the moving direction of the person 110 and having a length corresponding to the speed of the person 110. Thus, the determination unit 16 determines that there is no possibility of a collision when the moving directions of the forklift 100 and the person 110 do not intersect.

[0044] In this embodiment, the determination unit 16 may determine that there is a possibility of a collision between the forklift 100 and the person 110 when the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap. By using the movement area 102 and the movement area 112, the determination unit 16 can determine the possibility of a collision while taking into account an error in the calculated movement direction.

[0045] FIG. 5 is a schematic diagram illustrating an example of the positional relationship between the forklift 100 and the person 110. FIG. 6 is a schematic diagram illustrating the possibility of a collision in the positional relationship between the forklift 100 and the person 110 in FIG. 5. FIG. 6 is a diagram in which the movement area 102 of the forklift 100 and the movement area 112 of the person 110 are extracted from FIG. 5. In the example shown in FIGS. 5 and 6, the movement area 102 of the forklift 100 and the movement area 112 of the person 110 do not overlap. In this case, the determination unit 16 does not determine that there is a possibility of a collision between the forklift 100 and the person 110.

[0046] FIG. 7 is a schematic diagram illustrating another example of the positional relationship between the forklift 100 and the person 110. FIG. 8 is a schematic diagram illustrating the possibility of a collision in the positional relationship between the forklift 100 and the person 110 in FIG. 7. FIG. 8 is a diagram in which the movement area 102 of the forklift 100 and the movement area 112 of the person 110 are extracted from FIG. 7. In the example shown in FIGS. 7 and 8, the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap. In this case, the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110.

[0047] The output control unit 21 controls the video 200 captured by the camera 2 so that a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 are superimposed on the video 200 captured by the camera 2. In this embodiment, the marker indicating the movement area 102 and the marker indicating the movement area 112 are fan-shaped images. In this embodiment, the output control unit 21 controls the video 200 captured by the camera 2 so that an image generated by the image generation unit 15 is superimposed on the video 200. More specifically, the output control unit 21 controls the video 200 captured by the camera 2 so that a rectangular image indicating the detection frames of the forklift 100 and the person 110 and a fan-shaped image indicating the movement area are superimposed on the video 200 and output.

[0048] In this embodiment, the output control unit 21 controls the output to the monitor 3. The output control unit 21 may perform control so as to distribute the video to the monitor 3 via a network (not shown).

[0049] The alarm output unit 22 controls to output an alarm when the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, the alarm output unit 22 controls to vibrate the vibration unit 4 when the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110.

[0050] After outputting the alarm, the alarm output unit 22 may control the vibration unit 4 to stop vibrating when the possibility of a collision has disappeared. In this embodiment, after outputting the alarm, the alarm output unit 22 may control the vibration unit 4 to stop vibrating when the possibility of a collision has disappeared.

[0051] When the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the video storage control unit 23 controls the video storage unit 5 to store a video in which the image generated by the image generation unit 15 is superimposed on the video 200 captured by the camera 2. More specifically, the video storage control unit 23 controls the video storage unit 5 to store a rectangular image indicating the detection frame of the forklift 100 and the person 110 and a sector-shaped image indicating the movement area superimposed on the video 200 captured by the camera 2. The video storage control unit 23 may also control the video storage unit 5 to store the sector-shaped image indicating the movement area of ​​the forklift 100 and the person 110 without superimposing it.

[0052] When the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the video storage control unit 23 may perform control so that the video captured by the camera 2, including the time before and after the determination, is stored in the video storage unit 5. The video storage control unit 23 may perform control so that the video 200 captured by the camera 2, including the time before and after the determination, is stored in the video storage unit 5 with a rectangular image indicating the detection frame of the forklift 100 and the person 110 and a sector-shaped image indicating the movement area superimposed thereon.

[0053] In order to improve the accuracy of the learning model, the image storage control unit 24 controls the learning storage unit 6 to store images that have been recognized by the recognition unit 12. The image storage control unit 24 controls the learning storage unit 6 to store, for example, image data captured under predetermined conditions. The predetermined conditions are when conditions similar to those under which image data with low recognition accuracy of the forklift 100 and the person 110 were captured are met. The predetermined conditions are, for example, when brightness, weather, season, time of day, etc. meet predetermined conditions.

[0054] The image storage control unit 24 may process the captured image data and control the learning storage unit 6 to store a plurality of processed image data.

[0055] For example, if the perimeter monitoring system 1 determines that there is a possibility of a collision and the operator's vibration unit 4 vibrates, and the operator determines that there is no possibility of a collision, the recognition accuracy is predicted to be low. For example, if the perimeter monitoring system 1 determines that there is a possibility of a collision and the monitor monitoring the video displayed on the monitor 3 determines that there is no possibility of a collision, the recognition accuracy is predicted to be low.

[0056] The storage unit 29 includes at least one of a RAM, a ROM, a flash memory, and a hard disk drive. The storage unit 29 stores data used in the processing of the perimeter monitoring device 10.

[0057] <Perimeter monitoring system processing> It is assumed that the perimeter monitoring system 1 is activated during the operating hours of the forklift 100. While the perimeter monitoring system 1 is activated, the camera 2 captures images of the surroundings, and the perimeter monitoring device 10 determines the possibility of a collision between the forklift 100 and a person 110.

[0058] 9 is a flowchart showing an example of processing of the perimeter monitoring system 1 according to this embodiment. The perimeter monitoring device 10 acquires video (step S101). More specifically, the perimeter monitoring device 10 acquires video data of video captured by the camera 2 using the video acquisition unit 11. The perimeter monitoring device 10 converts the acquired video data into image data for each frame using the video acquisition unit 11 and outputs the image data to the recognition unit 12. The perimeter monitoring device 10 proceeds to step S102.

[0059] The periphery monitoring device 10 recognizes the forklift 100 and the person 110 (step S102). More specifically, the periphery monitoring device 10 recognizes the forklift 100 and the person 110 from the image data acquired from the video acquisition unit 11 by the recognition unit 12. The periphery monitoring device 10 proceeds to step S103.

[0060] The periphery monitoring device 10 tracks the forklift 100 and the person 110 (step S103). More specifically, the periphery monitoring device 10 causes the tracking unit 13 to track the forklift 100 and the person 110 recognized by the recognition unit 12. The periphery monitoring device 10 proceeds to step S104.

[0061] The periphery monitoring device 10 calculates the moving direction and speed (step S104). More specifically, the periphery monitoring device 10 calculates the moving direction and speed of the forklift 100 and the moving direction and speed of the person 110 using the calculation unit 14. The periphery monitoring device 10 proceeds to step S105.

[0062] The periphery monitoring device 10 generates an image (step S105). More specifically, the periphery monitoring device 10 generates, by the image generation unit 15, an image showing the detection frames of the forklift 100 and the person 110 recognized by the recognition unit 12, as well as the direction and speed of movement. The periphery monitoring device 10 proceeds to step S106.

[0063] The periphery monitoring device 10 controls to output the video (step S106). More specifically, the periphery monitoring device 10 controls the output control unit 21 to superimpose and display a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 on the video 200 captured by the camera 2. The periphery monitoring device 10 proceeds to step S107.

[0064] The periphery monitoring device 10 determines the possibility of a collision (step S107). More specifically, the periphery monitoring device 10 determines, via the determination unit 16, the possibility of a collision between the forklift 100 and the person 110 based on the movement direction of the forklift 100 and the movement direction of the person 110 calculated by the calculation unit 14. The periphery monitoring device 10 may determine, via the determination unit 16, that there is a possibility of a collision between the forklift 100 and the person 110 when the movement direction of the forklift 100 and the movement direction of the person 110 intersect. The periphery monitoring device 10 may also determine, via the determination unit 16, that there is a possibility of a collision between the forklift 100 and the person 110 when the movement area 102 of the forklift 100 and the movement area 112 of the person 110 overlap. When the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110 (Yes in step S107), the process proceeds to step S108. If the determining unit 16 does not determine that there is a possibility of a collision between the forklift 100 and the person 110 (Yes in step S107), the process proceeds to step S110.

[0065] If it is determined that there is a possibility of a collision (Yes in step S107), the periphery monitoring device 10 controls to output an alarm (step S108). More specifically, the periphery monitoring device 10 controls the alarm output unit 22 to vibrate the vibration unit 4. The periphery monitoring device 10 proceeds to step S109.

[0066] The operator recognizes that the vibration of the vibration unit 4 indicates that the forklift 100 may collide with the person 110. The operator then performs an operation to avoid the collision with the person 110. For example, the operator performs a braking operation, releases the accelerator pedal, or operates the steering wheel.

[0067] The periphery monitoring device 10 controls to store the video (step S109). More specifically, when the determination unit 16 determines that there is a possibility of a collision between the forklift 100 and the person 110, the periphery monitoring device 10 controls the video storage control unit 23 to superimpose a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 on the video 200 captured by the camera 2, including the time before and after the determination, and store the superimposed video in the video storage unit 5. Note that the periphery monitoring device 10 may also control the video storage control unit 23 to store the marker indicating the movement area 102 and the marker indicating the movement area 112 of the person 110 in the video storage unit 5 without superimposing the marker.

[0068] If it is determined that there is no possibility of a collision (No in step S107), the periphery monitoring device 10 controls the alarm output unit 22 to stop the vibration of the vibration unit 4 (step S110).

[0069] <Computer System> FIG. 10 is a block diagram showing an example of a computer system 1000 according to this embodiment. The above-described periphery monitoring device 10 includes the computer system 1000. The computer system 1000 has a processor 1001 such as a CPU, a main memory 1002 including a nonvolatile memory such as a ROM and a volatile memory such as a RAM, a storage 1003, and an interface 1004 including an input / output circuit. The functions of the above-described periphery monitoring device 10 are stored as a program in the storage 1003. The processor 1001 reads the program from the storage 1003, loads it into the main memory 1002, and executes the above-described processing in accordance with the program. The program may be distributed to the computer system 1000 via a network.

[0070] <Effects> As described above, in this embodiment, the possibility of a collision between the forklift 100 and the person 110 is determined based on the moving direction of the forklift 100 and the moving direction of the person 110. This embodiment can determine that there is no possibility of a collision, for example, even if the person 110 is present near the forklift 100, if the moving directions of the forklift 100 and the person 110 are opposite each other. This embodiment can appropriately determine the possibility of a collision between the forklift 100 and the nearby person 110. In this way, this embodiment can appropriately monitor the possibility of a collision between the forklift 100 and the nearby person 110.

[0071] In this embodiment, if the direction of movement of the forklift 100 and the direction of movement of the person 110 intersect, it is determined that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, if the direction of movement of the forklift 100 and the direction of movement of the person 110 do not intersect, it can be determined that there is no possibility of a collision. In this embodiment, it is possible to more appropriately determine the possibility of a collision between the forklift 100 and the nearby person 110.

[0072] In this embodiment, when the movement area 102 of the forklift 100 overlaps with the movement area 112 of the person 110, it is determined that there is a possibility of a collision between the forklift 100 and the person 110. This embodiment can more appropriately determine the possibility of a collision by taking into account an error in the calculated movement direction.

[0073] In this embodiment, an alarm is output when it is determined that there is a possibility of a collision between the forklift 100 and the person 110. In this embodiment, the alarm can notify the operator of the forklift 100 or the like of the possibility of a collision.

[0074] In this embodiment, when it is determined that there is a possibility of a collision between the forklift 100 and the person 110, a marker indicating the movement area 102 of the forklift 100 and a marker indicating the movement area 112 of the person 110 are superimposed on the video captured by the camera 2 and stored. This embodiment can store the video when it is determined that there is a possibility of a collision. This allows the validity of the determination of the possibility of a collision to be confirmed later.

[0075] In this embodiment, when it is determined that there is a possibility of a collision between the forklift 100 and the person 110, the video captured by the camera 2 is stored, including the period before and after the determination. This embodiment can store the video before and after the determination that there is a possibility of a collision. This allows the validity of the determination of the possibility of a collision to be confirmed later.

[0076] In this embodiment, a learning model is used to recognize the forklift 100 and the person 110 around the forklift 100 using an image cut out from a video as input data. This embodiment can improve the accuracy of recognizing the forklift 100 and the person 110.

[0077] In this embodiment, the learning model is trained using an average image obtained by averaging multiple images extracted from a video. According to this embodiment, learning can be performed using an average image with appropriately reduced noise. This embodiment can improve the recognition accuracy of the forklift 100 and the person 110.

[0078] <Modification> In the above, the camera 2 is placed near the ceiling of the warehouse or on the travel path, but this is not a limitation. The camera 2 may be placed on the forklift 100. In this case, the direction of movement of the person, etc. is calculated from the image captured by the camera 2 placed on the forklift 100. The direction of movement of the forklift 100 can be determined, for example, from the steering angle. The speed of the forklift 100 can be determined, for example, from a speedometer provided on the forklift 100. The movement area of ​​the forklift 100 can be determined from the direction of movement and vehicle speed determined as described above.

[0079] Step S105 in the flowchart shown in Fig. 9 is not essential and may be omitted. The movement direction and speed may be calculated without generating a detection frame and an image indicating the movement direction and speed.

[0080] Steps S106, S108, S109, and S110 in the flowchart shown in FIG. 9 are not essential steps and may be omitted.

[0081] In the above example, a person is an example of an obstacle. Other examples of obstacles include stationary objects such as other work machines and cargo in a factory. In the example of other work machines, the possibility of a collision may be determined when the movement direction of the forklift 100 intersects with the movement direction of the other work machine. Also, the possibility of a collision may be determined when the movement area of ​​the forklift 100 overlaps with the movement area of ​​the other work machine. In the example of cargo, the recognition unit 12 may recognize the cargo, and the possibility of a collision may be determined when the center point of the cargo intersects with the movement direction of the forklift 100. Also, the possibility of a collision may be determined when the center point of the cargo overlaps with the movement area of ​​the forklift 100.

[0082] Although the above description has been given assuming that the work machine is the forklift 100, the work machine is not limited to this. The work machine may also be a wheel loader, a hydraulic excavator, or the like that is driven by power from a battery or an engine, or the like. [Explanation of symbols]

[0083] 1...perimeter monitoring system, 2...camera (photographing unit), 3...monitor, 4...vibration unit (alarm unit), 5...video memory unit, 6...learning memory unit, 10...perimeter monitoring device, 11...video acquisition unit, 12...recognition unit, 13...tracking unit, 14...calculation unit, 15...image generation unit, 16...judgment unit, 21...output control unit, 22...alarm output unit, 23...video memory control unit, 24...image memory control unit, 29...memory unit, 100...forklift (work machine), 101...detection frame, 102...movement area, 110...person, 111...detection frame, 112...movement area.

Claims

1. an imaging unit that images the work machine and the surrounding area of ​​the work machine; a recognition unit that recognizes the work machine and obstacles around the work machine from the image captured by the imaging unit; a calculation unit that calculates the movement direction and speed of the work machine and the movement direction and speed of the obstacle from the recognition result of the recognition unit; a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the movement direction of the work machine and the movement direction of the obstacle calculated by the calculation unit; Equipped with the determination unit determines that there is a possibility of a collision between the work machine and the obstacle when a movement area indicating the movement direction and the speed of the work machine overlaps with a movement area indicating the movement direction and the speed of the obstacle; the movement area is a sector having a center line that extends from the work machine or the obstacle along the movement direction, The central angle of the sector changes depending on the area of ​​the obstacle to be monitored, The radius of the sector increases as the moving speed increases. Perimeter surveillance system.

2. the determination unit determines that there is a possibility of a collision between the work machine and the obstacle when the movement direction of the work machine and the movement direction of the obstacle intersect. The perimeter monitoring system according to claim 1 .

3. an alarm unit that outputs an alarm when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle; The perimeter monitoring system according to claim 1 or 2, comprising:

4. The alarm unit is a vibration unit that generates vibration to alert an operator, the vibration unit vibrates when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle. The perimeter monitoring system according to claim 3 .

5. an image storage control unit that, when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle, controls the image captured by the image capturing unit to store a marker indicating a movement area in the movement direction of the work machine and a marker indicating a movement area in the movement direction of the obstacle, superimposed on the image; The perimeter monitoring system according to claim 1 , comprising:

6. an image storage control unit that, when the determination unit determines that there is a possibility of a collision between the work machine and the obstacle, stores the image captured by the image capture unit, including the period before and after the time point at which the determination is made; The perimeter monitoring system according to claim 1 , comprising:

7. the recognition unit uses a learning model to recognize the work machine and obstacles around the work machine using an image cut out from the video as input data. The perimeter monitoring system according to any one of claims 1 to 6.

8. The learning model is trained using an average image obtained by averaging a plurality of images extracted from the video. The perimeter monitoring system according to claim 7 .

9. The obstacle is a person. The perimeter monitoring system according to any one of claims 1 to 8.

10. Taking an image of the work machine and the surrounding area of ​​the work machine; Recognizing the work machine and obstacles around the work machine from the captured video; calculating a moving direction and speed of the work machine and a moving direction and speed of the obstacle from the recognition result; determining, based on the calculated movement direction of the work machine and the movement direction of the obstacle, that there is a possibility of a collision between the work machine and the obstacle when a movement area indicating the movement direction and speed of the work machine overlaps with a movement area indicating the movement direction and speed of the obstacle; Including, the movement area is a sector having a center line that extends from the work machine or the obstacle along the movement direction, The central angle of the sector changes depending on the area of ​​the obstacle to be monitored, The radius of the sector increases as the moving speed increases. A perimeter monitoring method performed by a perimeter monitoring system.

11. a recognition unit that recognizes obstacles around the work machine and the surrounding area of ​​the work machine from images of the work machine and the surrounding area of ​​the work machine; a calculation unit that calculates the movement direction and speed of the work machine and the movement direction and speed of the obstacle from the recognition result of the recognition unit; a determination unit that determines the possibility of a collision between the work machine and the obstacle based on the movement direction of the work machine and the movement direction of the obstacle calculated by the calculation unit; Equipped with the determination unit determines that there is a possibility of a collision between the work machine and the obstacle when a movement area indicating the movement direction and the speed of the work machine overlaps with a movement area indicating the movement direction and the speed of the obstacle; the movement area is a sector having a center line that extends from the work machine or the obstacle along the movement direction, The central angle of the sector changes depending on the area of ​​the obstacle to be monitored, The radius of the sector increases as the moving speed increases. Perimeter monitoring device.

Images