Person position identification system, telepresence robot operation management device, telepresence robot operation management method, and program
Movable telepresence robots with environmental sensors and cameras enhance indoor positioning by reducing blind spots through SLAM mapping and image recognition, enabling real-time person location determination.
Patent Information
- Application Number
- JP2023080105
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-15
- Publication Date
- 2024-11-27
AI Technical Summary
Conventional multi-camera surveillance systems in indoor facilities have blind spots where the position of a person cannot be determined due to overlapping camera fields of view.
Employing movable telepresence robots equipped with environmental shape measurement sensors and cameras, which use SLAM technology to create maps, display positional markers, and track individuals based on image recognition, thereby reducing blind spots.
The system effectively reduces areas where a person's position cannot be specified by allowing telepresence robots to navigate and identify individuals in real-time, even in unpredictable indoor environments.
Smart Images

Figure 2024164543000001_ABST
Abstract
Description
[Technical field]
[0001] The present invention relates to a human position identification system, a telepresence robot operation management device, a telepresence robot operation management method, and a program. [Background technology]
[0002] As a method for identifying the location of a person moving within an indoor facility, for example, there is a method using a surveillance video camera installed on the ceiling or wall of the indoor facility. When a specific person crosses the field of view of the surveillance video camera, the person is photographed by the surveillance video camera, so that the observer can determine that the person was near the surveillance video camera that photographed the person. Furthermore, as a method for identifying the location of a person moving within an indoor facility, a method of linking a beacon and a smartphone is known. A beacon is installed on the ceiling or wall of the indoor facility, and the smartphone catches the radio waves emitted by the beacon to identify the location of the person carrying the smartphone.
[0003] A multiple camera monitoring system for such an indoor facility is disclosed in Patent Document 1. The multiple camera monitoring system of Patent Document 1 realizes an arrangement of multiple cameras that reduces the amount of overlap of the camera fields of view by providing feedback according to the amount of overlap of the camera fields of view between the cameras. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2011-114580 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, in the above-mentioned multiple camera surveillance system, since the multiple cameras are fixedly installed, there is a problem that there may be places where a person's position cannot be identified, for example, when the blind spots of the camera fields of view of these cameras overlap.
[0006] The present invention has been made in consideration of the above circumstances, and provides a person position identification system, a telepresence robot operation management device, a telepresence robot operation management method, and a program that can reduce the areas where a person's position cannot be identified more than in the past. [Means for solving the problem]
[0007] The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is a human position identification system comprising one or more mobile telepresence robots equipped with an environmental shape measurement sensor, and a telepresence robot operation management device, the telepresence robot operation management device comprising an environmental display unit that acquires measurement information indicating a position that is the measurement result of the environmental shape measurement sensor and displays a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the telepresence robot.
[0008] Another aspect of the present invention is the above-mentioned person position identification system, wherein the environmental display unit changes the color of the predetermined mark according to the distance from the telepresence robot to the position indicated by the measurement information.
[0009] Another aspect of the present invention is the above-mentioned person positioning system, wherein each of the one or more telepresence robots is equipped with a camera, and the operation management device is equipped with a detected person display unit that selects one image from each of the one or more robots in which the same person is detected from images captured by the camera, and displays the images in a row.
[0010] Another aspect of the present invention is the above-mentioned person positioning system, wherein each of the one or more telepresence robots is equipped with a camera, and the operation management device is equipped with a robot control unit that causes one of the one or more telepresence robots to track a person detected from an image captured by the camera when the person is not included in a pre-registered whitelist or is included in a pre-registered blacklist.
[0011] Another aspect of the present invention is a telepresence robot operation and management device that acquires measurement information indicating a position that is a measurement result obtained by an environmental shape measurement sensor equipped on one or more mobile telepresence robots, and has an environmental display unit that displays a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots.
[0012] Another aspect of the present invention is a telepresence robot operation and management method comprising the steps of acquiring measurement information indicating a position which is a measurement result obtained by an environmental shape measurement sensor provided on one or more mobile telepresence robots, and displaying a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots.
[0013] Another aspect of the present invention is a program for causing a computer to function as an environmental display unit that acquires measurement information indicating a position that is a measurement result obtained by an environmental shape measurement sensor equipped on one or more mobile telepresence robots, and displays a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots. Effect of the Invention
[0014] According to the present invention, the human position identification system, the telepresence robot operation management device, the telepresence robot operation management method, and the program can reduce the area in which the position of a person cannot be identified compared to the conventional art. [Brief description of the drawings]
[0015] [Figure 1] FIG. 1 is a schematic diagram showing a configuration of a person position identification system 10 according to an embodiment of the present invention. [Diagram 2] FIG. 13 is a diagram showing an example (part 1) of an image stored in the storage unit 900 in the embodiment. [Diagram 3] FIG. 13 is a diagram showing a second example of an image stored in the storage unit 900 in the embodiment. [Figure 4] FIG. 13 is a diagram showing a third example of an image stored in the storage unit 900 in the embodiment. [Diagram 5] FIG. 13 is a diagram showing an example (part 4) of an image stored in the storage unit 900 in the embodiment. [Figure 6] FIG. 5 is a diagram showing an example (part 5) of an image stored in the storage unit 900 in the embodiment. [Figure 7] 2 is a schematic block diagram showing the configuration of a telepresence robot operation management device 200 in the embodiment. FIG. [Figure 8] 11A to 11C are diagrams illustrating examples of display by an environment display unit 302 in the embodiment. [Figure 9] 11A and 11B are diagrams showing example images captured by a video camera provided in the telepresence robot in the embodiment. [Figure 10] FIG. 11 is a diagram showing an example of a screen UI displayed on the electronic display 300 in the embodiment. [Figure 11] 4 is a flowchart illustrating the operation of the person position identification system 10 in the embodiment. [Figure 12] 10 is a flowchart illustrating the operation of an environment display unit 302 in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Fig. 1 is a schematic diagram showing the configuration of a person position identification system 10 according to an embodiment of the present invention. The person position identification system 10 includes one or more telepresence robots 100, a telepresence robot operation management device 200, an electronic display 300, an information and communication network 400, an image input server 500, an image recognition server 600, an image extraction server 700, an image output server 800, and a storage unit 900.
[0017] Each device included in the person location identification system 10 may be realized by one or more computers reading and executing a program. The program is generally a machine code of 0s and 1s written in a programming language that is readable by a computer. There is no limitation on the type of machine code programming language as long as it can be used to construct the person location identification system 10, and it may be, for example, Basic, Visual-Basic, Fortran, C, C++, Visual-C++, C#, Java, Bash, Python, MATLAB (registered trademark), or a combination of these.
[0018] The telepresence robot 100 includes a video camera (camera), an environment shape measuring sensor, and a moving means. The telepresence robot 100 may include, for example, wheels for traveling as a moving means. The wheels are connected to a power source, and power generated by the power source is used to transmit power to the wheels. An electromagnetic motor is suitable as the power source, and examples of the power source include a DC motor, a brushless DC motor, a stepping motor, a servo motor, an induction motor, a PM motor, an AC motor, an in-wheel motor, and an ultrasonic motor. An internal combustion engine can also be used as the power source. There are no particular limitations on the number and installation positions of the wheels, as long as the number and positions allow the telepresence robot 100 to travel stably. The movement of the telepresence robot 100 can be remotely controlled by wireless communication. The means of wireless communication between the telepresence robot 100 and the telepresence robot operation management device 200 that remotely controls the telepresence robot 100 may be any of wireless LAN (Local Area Network), Wi-Fi (registered trademark), third-generation, fourth-generation, fifth-generation mobile communication systems, and LPWA (Low Power Wide Area), etc.
[0019] The video camera equipped in the telepresence robot 100 may be any one equipped with an electronic imaging element, and may be a video camera equipped with a CCD (Charge Coupled Devices) or a CMOS (Complementary Metal Oxcide Semiconductor). In order to utilize the image recognition technology, the camera parameters of the video camera may be acquired in advance and may be utilized for the image recognition technology in the image recognition server 600 or the like. As a method of using the image recognition technology, the following one or two methods are known on the premise that the camera parameters are known: 1. A method of knowing the body shape of a target person from the scale difference or the position of the image center in a test image of the target person taken in advance, and 2. A method of knowing the body shape of a target person from Structure from Motion (abbreviated as SfM). The body shape of a target person includes height, waist width, etc. In addition, the image taken by the video camera can be a still image or a video. In the case of a video, since a video is formed of a continuous flow of still images with slightly different scenes, any one or more still images from the video can be used. The video camera may be a Web video camera or a single-lens reflex video camera installed separately from the Web video camera, but it is preferable to select an appropriate one in terms of, for example, image quality, weight, size, and price.
[0020] The environmental shape measurement sensor provided in the telepresence robot 100 is used to realize the SLAM (Simultaneous Localization and Mapping) technology of the telepresence robot 100. SLAM is a method of estimating the current position information of the telepresence robot 100 while creating a map of the surroundings of the telepresence robot 100 acquired using the environmental shape measurement sensor, and is a method of completing the entire map while expanding a new measurement area by the telepresence robot 100 moving. Examples of methods for calculating SLAM include scan matching methods such as ICP (Iterative Closest Point) and NDT (Normal Distributions Transform), and Bayes filter methods, and any of these methods can be used in this embodiment. The environmental shape measurement sensor may be a sensor using any of light rays, radio waves, and ultrasonic waves. For example, LiDAR (Light Detection And Ranging) using a laser range scanner, ToF (Time of Flight) sensor, millimeter wave radar using millimeter waves, ultrasonic sensor using ultrasonic waves, etc., or a sensor combining these can be suitably used.
[0021] The environment shape measuring sensor not only measures the internal shape of the building, but also responds to a person moving near the telepresence robot 100. The telepresence robot operation management device 200 can identify the positional relationship (distance) between the telepresence robot 100 and a person using the detection result of the environment shape measuring sensor. The telepresence robot 100 may be manually driven via the telepresence robot operation management device 200 while an operator (human) checks an image from a video camera equipped in the telepresence robot 100 displayed by the telepresence robot operation management device 200. The telepresence robot operation management device 200 may determine a route from a start point designated by an operator to a goal point using a route search algorithm such as A*, and automatically drive the telepresence robot 100 according to the determined route. Manual driving and automatic driving may be used in combination.
[0022] The information and communication network 400 may include an Internet line network. The local server area and the remote server area communicate with each other via the information and communication network 400. The line network of the information and communication network 400 may be a wired or wireless network, or a combination of both.
[0023] The image input server 500 inputs images of people captured by the video camera of the telepresence robot 100 via the telepresence robot operation management device 200 and writes and stores the images in the storage unit 900. The image input server 500 is connected to the telepresence robot operation management device 200 via the information and communication network 400, but the connection may be by wired communication or wireless communication.
[0024] The image recognition server 600 is a server equipped with an algorithm for recognizing people in images stored in the storage unit 900. There are various types of objects for recognizing people, such as a face, height, body bone structure, walking posture and stride length, the pattern and color of the person's clothes, the position of a top and the positional relationship of bottoms, the presence or absence of accessories such as accessories and bags carried by the person, or a combination of these, and the image recognition server 600 recognizes (identifies) a person (individual). A general algorithm for recognizing a person can be used, such as YOLO (You Only Look Once), R-CNN (Convolutional Neural Network), SSD (Single Shot Multibox Detector), DETR (Detection Transformer), DCN (Deformed Convolutional Networks), and HOG (Histogram of Oriented Gradient). When the image recognition server 600 recognizes a person, it is preferable to digitally timestamp the image with the date and time when the image was acquired.
[0025] The image cutout server 700 cuts out and extracts only a specific portion (for example, a portion to be recognized) of the recognized person from the image stored in the storage unit 900 in a rectangular shape. The image cutout server 700 cuts out only the specific portion, not the entire captured image, to delete unnecessary image portions and reduce the amount of information. As a result, it plays a role in reducing communication delays in the information communication network 400 in the subsequent stage and improving real-time performance in identifying the person's position. If it is desired to further improve the real-time performance in identifying the person's position, an image compression server may be inserted in the subsequent stage of the image input server 500 and the image cutout server 700. In that case, an image decoding server is further inserted in the subsequent stage of the image compression server, and the compressed image data is decrypted using a previously provided decoding key in the image decoding server, and the original image is decoded.
[0026] The image output server 800 reads out the cut-out image output by the image cut-out server 700 from the storage unit 900 , and transmits it to the telepresence robot operation management device 200 via the information communication network 400 .
[0027] The storage unit 900 stores images captured by the video camera of the telepresence robot 100 and images extracted by the image extraction server 700, and provides them to the image recognition server 600 and the image output server 800. The storage unit 900 may be installed in both a local server area and a remote server area. A hard disk, a solid state drive (SSD), a non-volatile memory, or the like may be used as the storage unit 900.
[0028] Fig. 2 to Fig. 6 are diagrams showing examples of images stored in the storage unit 900 in this embodiment. Fig. 2 shows an example of an image captured by the video camera of the telepresence robot 100, i.e., an example of an image stored in the storage unit 900 by the image input server 500. Fig. 3 to Fig. 6 each show an example of an image cut out by the image cut-out server 700. Fig. 3 to Fig. 6 show an example of a specific part being a person's clothing, but instead of clothing, the specific part may be a face, a whole body, clothing, accessories, etc., just like the recognition target.
[0029] The image input server 500 is installed in the same premises where the telepresence robot 100 is installed, while the image recognition server 600, the image extraction server 700, the image output server 800, and the storage unit 900 may be installed in a place (remote server area) called the cloud, away from the premises (local server area) where the telepresence robot 100 is installed. Services that deploy the cloud include AWS (registered trademark) (provided by Amazon (registered trademark)), Azure (registered trademark) (provided by Microsoft (registered trademark)), GCP (registered trademark) (provided by Google (registered trademark)), etc., and any of them may be used.
[0030] In order for the image recognition server 600 to perform person recognition, a large number of person images may be trained in advance in the image recognition server 600. The person images (training data) to be trained are preferably one or more person images. In particular, in order to improve the image recognition rate, the person images to be trained may include a front image of a person, or may include a combination of two images, a front image and a side image of a person, or may include a combination of three images, a front image, a side image, and a rear image of a person.
[0031] Similarly, the test data (data for evaluating the generalizability of the trained model) preferably includes one or more images of a person. In particular, in order to improve the image recognition rate, the test data may include a front image of a person, a combination of two images, a front image and a side image of a person, or a combination of three images, a front image, a side image, and a rear image of a person.
[0032] The electronic display 300 displays the cut-out image received by the telepresence robot operation management device 200 from the image output server 800. The electronic display 300 may be installed in either the local server area or the remote server area, or in both locations, and may be appropriately selected according to the purpose. The electronic display 300 may be any device equipped with an electronic optical element, and in particular, a liquid crystal display, an organic light-emitting diode (OLED) display, a mini LED display, a micro LED display, or the like may be used. The electronic display 300 may be a monitor device dedicated to image display, or an electronic display such as a notebook computer, a tablet terminal, or a smartphone.
[0033] The telepresence robot operation management device 200 is used in cooperation with the telepresence robot 100 to display the operating status of the telepresence robot 100 and to remotely control the telepresence robot 100. The telepresence robot operation management device 200 may be an information processing terminal equipped with an electronic display. The telepresence robot operation management device 200 may be, for example, a desktop computer, a notebook computer, a tablet terminal, or a smartphone. The display contents by the monitoring display function of the telepresence robot operation management device 200 include the current position and direction of the telepresence robot 100, the direction of the video camera equipped in the telepresence robot 100, the travel path and trajectory, the travel speed, the remaining battery level, and the video captured by the video camera equipped in the telepresence robot 100. The control contents by the telepresence robot operation function of the telepresence robot operation management device 200 include the travel speed and direction of the telepresence robot 100, and the up, down, left, and right directions of the video camera equipped in the telepresence robot 100. These control contents may be directly specified by an operator, or may be automatically controlled by the telepresence robot operation management device 200. In addition, the telepresence robot 100 may travel autonomously based on route information specified by the telepresence robot operation management device 200.
[0034] 7 is a schematic block diagram showing the configuration of a telepresence robot operation management device 200 in this embodiment. The telepresence robot operation management device 200 includes a communication unit 301, an environment display unit 302, a robot control unit 303, and a detected person display unit 304. The communication unit 301 communicates with the telepresence robot 100 and with other devices via an information communication network 400.
[0035] The environment display unit 302 acquires measurement information indicating a position, which is a measurement result by the environmental shape measurement sensor equipped in the telepresence robot 100, and displays a cross at a position indicated by the measurement information on a background map including the periphery of the telepresence robot 100, as shown in FIG. 8. Here, a cross is used as the predetermined mark, but any mark such as a triangle, a circle, a square, a star, etc. may be used. The environment display unit 302 may change the color of the cross according to the distance from the telepresence robot 100 to the position indicated by the measurement information. For example, the environment display unit 302 may change the color of the cross to red when the distance is less than 1 m, orange when the distance is 1 m or more but less than 3 m, and light blue when the distance is 3 m or more. The environment display unit 302 may further display an image captured by a video camera of the telepresence robot 100, as shown in FIG. 9. By viewing these, the operator of the telepresence robot operation management device 200 can easily grasp the position of a person in the image captured by the video camera.
[0036] When a person detected from an image captured by the video camera of the telepresence robot 100 is not registered in the whitelist or is registered in the blacklist, the robot control unit 303 causes one or more telepresence robots 100 to track the detected person. The whitelist and the blacklist may be lists of people corresponding to registered images previously learned by the image recognition server 600. When tracking a person, the telepresence robot 100 may be configured to automatically travel so as to continue to capture the person with the video camera.
[0037] As shown in FIG. 10, the detected person display unit 304 selects at least one image from each of one or more telepresence robots 100 in which the same person is detected from among the images captured by the video cameras of the telepresence robots 100, and displays the images in a row.
[0038] The environment display unit 302 may store data of a background map of a virtual space drawn according to the dimensions of the real space by VR (Virtual Reality), CAD (Computer-Aided Design), BIM (Building Information Modeling), or the like, and may display the background map. Since this background map is scaled in advance, the distance (horizontal distance, vertical distance) between two points on the background map can be known. The environment display unit 302 may superimpose an icon indicating the current position and orientation of the telepresence robot 100 and an icon indicating the orientation of the video camera equipped in the telepresence robot 100 on the background map. This makes it easier for the operator to confirm the distance between the current position of the telepresence robot 100 and a person.
[0039] Since the background map is scaled, the environment shape map can be acquired by running a virtual telepresence robot equipped with an environment shape measurement sensor on the background map. Alternatively, the environment shape map can be acquired by running a real telepresence robot 100 equipped with a real environment shape measurement sensor in a real environment. By acquiring the environment shape map, it becomes possible to freely run the telepresence robot 100 in the environment using SLAM.
[0040] The environment display unit 302 may display the environment shape map and the background map in a superimposed manner. Since the environment shape map displays objects as crosses, if the object is a fixed object, it is difficult to determine what shape it is, but since the background map expresses the shape of the fixed object using VR, CAD, or BIM, the fixed object is visualized, which has the advantage of making it easier to distinguish.
[0041] In order to visualize how far an object detected by the environmental shape measuring sensor provided in the telepresence robot 100 is from the telepresence robot 100, it is preferable that the environment display unit 302 classifies the detected object by color according to distance. FIG. 8 is a diagram showing an example of display by the environment display unit 302 in this embodiment. In FIG. 8, the shape TR of the telepresence robot 100 is displayed on the environment display unit 302, and an object located far away from the telepresence robot 100 is displayed with a light blue cross CR1, an object located somewhat close to the telepresence robot 100 is displayed with an orange cross CR2, and an object located even closer than the orange object is displayed with a red cross CR3. This is preferable because the positional relationship between the telepresence robot 100 and the object is clear at a glance.
[0042] If the object detected by the environment shape measuring sensor is a fixed object, the cross does not move on the environment shape map or the background map, but remains at the same position. On the other hand, if the object detected by the environment shape measuring sensor is a moving object, the cross moves on the environment shape map or the background map, so that the moving object is highly likely to be a person. Therefore, the environment display unit 302 may display a virtual person that reproduces the position of the identified moving object (person) in a human form. Instead of a human form, a predetermined mark that is different from a cross, such as a triangle, and indicates that it is a moving object, may be used. This makes it easier for the operator to identify whether the object detected by the environment shape measuring sensor is a fixed object or a person (moving object).
[0043] Fig. 10 is a diagram showing an example of a screen UI (User Interface) displayed on the electronic display 300 in this embodiment. Screen G1 in Fig. 10 is an example of a screen UI that the detected person display unit 304 displays on the electronic display 300. Screen G1 is a screen that displays three people A to C using five telepresence robots A to E, but the number of telepresence robots and people is not limited to this, and can be any desired number depending on the application. Also, a person's clothing is used as an example of the recognition target, but as described above, the recognition target is not limited to only a person's clothing.
[0044] In the center of the screen G1, images captured by the video camera of the telepresence robot 100 are displayed in a grid pattern, with the person's clothing as the center. Each of the captured images is a captured image by the image capture server 700, with the person recognized by the image recognition server 600. The date and time of the image captured and acquired by the video camera of the telepresence robot 100 are digitally stamped on each of these captured images. The name or identification information (ID) of the telepresence robot 100 that captured the captured images is added to the top of the screen, indicating which telepresence robot 100 captured the corresponding vertical row of captured images. Furthermore, the identification information of the person whose clothing has been recognized is added to the left of the screen, and it is distinguished which person is detected (recognized, identified) in the corresponding horizontal row of captured images. The recognition rate of the person is displayed on the right of the screen. Note that the screen G1 is merely an example, and a screen UI similar to this may be adopted.
[0045] 11 is a flowchart explaining the operation of the person position identification system 10 in this embodiment. First, the telepresence robot operation management device 200 uses the video camera of the telepresence robot 100 to capture a registration image of the person (step Sa1). The camera used for capturing the image may be a portable general-purpose small camera, or a large camera that is fixedly installed on a ceiling, wall, etc.
[0046] The telepresence robot operation management device 200 makes the image recognition server 600 learn the registered image captured in step Sa1 (step Sa2). The telepresence robot operation management device 200 makes the telepresence robot 100 start patrol (step Sa3). The telepresence robot 100 captures people with a video camera during patrol (step Sa4). The telepresence robot operation management device 200 acquires the image captured in step Sa4 from the telepresence robot 100 and transmits it to the image input server 500 (step Sa5). The transmitted images may include images that do not include people, or may include only images that include people. When only images that include people are transmitted, the telepresence robot 100 or the telepresence robot operation management device 200 may determine whether or not a person is included in the image. The telepresence robot 100 may transmit the image captured in step Sa4 to the image input server 500 without going through the telepresence robot operation management device 200. When the image input server 500 receives the transmitted image, it stores the image in the storage unit 900 .
[0047] Next, the image recognition server 600 compares the image stored in the storage unit 900 with the registered image learned in step Sa2 and recognizes the person (step Sa6). Next, the image cutout server 700 cuts out the image of the area where the person was recognized in step Sa6, and sets it as a cutout image (step Sa7). Note that, if the person does not match the registered person in the comparison in step Sa6, the image cutout server 700 may cut out an image without the person.
[0048] Next, the image cutout server 700 transmits the cutout image together with the matching result in step Sa6 to the image output server 800 (step Sa8). Upon receiving the matching result with the cutout image, the image output server 800 causes the electronic display 300 to display the cutout image together with the matching result via the detected person display unit 304 of the telepresence robot operation management device 200 (step Sa9). Note that the image output server 800 may include the detected person display unit 304, which may cause the electronic display 300 to display the cutout image together with the matching result.
[0049] 12 is a flowchart for explaining the operation of the environment display unit 302 in this embodiment. First, the environment display unit 302 acquires the measurement results (measurement information) of the environment shape measurement sensor of the telepresence robot 100 (step Sb1). Next, the environment display unit 302 specifies the position of the telepresence robot 100 (step Sb2). The environment display unit 302 may match the measurement results with a background map to specify the position of the telepresence robot 100.
[0050] Next, the environment display unit 302 converts the coordinates of the measurement result acquired in step Sb1 based on the position of the telepresence robot 100 identified in step Sb2 (step Sb3). The measurement result indicates the position of the object detected by the environment shape measurement sensor in relative coordinates based on the telepresence robot 100, but this conversion converts it to absolute coordinates (for example, coordinates on a background map). Next, the environment display unit 302 displays an image of a cross indicating the measurement result after conversion in step Sb3 superimposed on the background map (step Sb4). Note that the environment display unit 302 may display an image of the measurement results after conversion by multiple telepresence robots 100 superimposed on the background map.
[0051] In the above embodiment, the telepresence robot 100 has been described as being equipped with a video camera, but may be equipped with a still camera.
[0052] The human position identification system 10 according to the present invention will be further described using an application scenario. The technical scope of the present invention is not to be interpreted as being limited solely on the basis of the specific content of the application scenario.
[0053] (Application scenario: security patrol in an office building) Ten telepresence robots 100 are placed in various locations in an office building to patrol and guard the office building. The telepresence robot operation management device 200 is a desktop PC, and the PC, the image input server 500, the SSD constituting part of the memory unit 900, and the electronic display 300 are installed in a control center that manages the telepresence robots 100. The other components, the image recognition server 600, the image extraction server 700, the image output server 800, and the hard disk constituting part of the memory unit 900, are installed on the cloud. The control center and the cloud are connected by wire through the Internet that constitutes the information and communication network 400. The camera parameters of each of the web video cameras provided on the ten telepresence robots 100 are measured in advance. Each of the ten telepresence robots 100 is also equipped with a LiDAR as an environment shape measurement sensor.
[0054] All employees working in this office building have three images of themselves taken in advance (front, side, and back) by the web video camera equipped in the telepresence robot 100 when they arrive at work, and these images are registered in the SSD as a whitelist. YOLOv5 is used as the algorithm for recognizing people, and the clothing worn by the person is used as the recognition target.
[0055] The telepresence robot operation management device 200 sets up ten telepresence robots 100 and makes these telepresence robots 100 travel automatically within an office building, concentrating on patrolling areas near the entrance, around pillars, toilets, and employee back yards that tend to be blind spots that cannot be monitored by video cameras fixed to the ceiling or walls.
[0056] The video camera A1 equipped on one of the ten telepresence robots 100, a telepresence robot A, captures an image of a person wearing clothes that do not exist in the clothing images of all employees stored and registered in the SSD. The image is stored in the storage unit 900 via the image input server 500. The image recognition server 600 reads the image from the storage unit 900, and after the person is recognized, the image cut-out server 700 cuts out only the clothing portion. The image output server 800 transmits the image cut out by the image cut-out server 700 (cut-out image) to the telepresence robot operation management device 200. The telepresence robot operation management device 200 displays the cut-out image on the electronic display 300 installed in the control center together with information indicating that a person who is not an employee has been detected.
[0057] When the monitor at the control center saw this display and visually checked the monitoring screen of the telepresence robot operation management device 200, he saw that the telepresence robot A was moving toward the right end of the entrance, and the video camera A1 was also facing toward the right end of the entrance. Therefore, the monitor inferred that the person entered from the left end of the entrance, and decided to be on guard against this person as a suspicious person. The telepresence robot operation management device 200 also estimated the height of the suspicious person to be 178 cm by SfM using the image from the video camera A1, and displayed this on the electronic display 300.
[0058] The control center then confirmed from the time information digitally time-stamped on the cut-out images that the suspicious person was photographed in the order of Telepresence Robot B, Telepresence Robot C, and Telepresence Robot D. The locations of the suspicious people could be identified from the monitoring screen of the Telepresence robot operation management device 200, which showed that Telepresence Robot B was located near Pillar A, Telepresence Robot C was located near Pillar B, Telepresence Robot C was located near the smoking area, and Telepresence Robot D was located near the employee back yard.
[0059] Next, the monitor at the control center decided to silently observe the suspicious person's behavior for a while. However, when the image of the suspicious person's clothes was next transmitted from the telepresence robot E and displayed on the electronic display 300, the monitor checked the monitoring screen of the telepresence robot operation management device 200 and found that the suspicious person was wandering around the employee backyard again. On the monitoring screen displayed by the telepresence robot operation management device 200, it was found that the distance between the suspicious person and the backyard measured by the LiDAR of the telepresence robot E was about 1 m. In addition, SfM using the image from the video camera E1 of the telepresence robot E confirmed that the height of the suspicious person was 177 cm, which was almost the same as the previous result from the video camera A1, and therefore it was confirmed that it was the same suspicious person. The monitor at the control center immediately contacted the security guard and removed the suspicious person from the backyard.
[0060] (Application Scenario 2: Finding a lost child) Department store F provides a customer service for finding lost preschool children. When a preschool child enters the department store, a video camera installed in the telepresence robot 100 captures three images of the preschool child's entire body, a front image, a side image, and a back image, and these images are registered as a blacklist. In this case, the recognition targets are the face of the preschool child, the pattern and color of the clothes, and the combination of accessories and bags. R-CNN is used as the algorithm for recognizing people. The camera parameters of the web video camera installed in each of the telepresence robots 100 in department store F are measured in advance. Each telepresence robot 100 is also equipped with a ToF as an environmental shape measurement sensor.
[0061] The central monitoring center of department store F received a report that preschool child G had gone lost, so the center deployed multiple telepresence robots 100 in locations that are easily overlooked during normal patrols, such as near pillars or objects, behind product display shelves, near restrooms, fitting rooms, and indoor parking lots, and began patrolling autonomously.
[0062] The telepresence robot operation management device 200 is a notebook computer, and this computer, image input server 500, a hard disk that constitutes part of the storage unit 900, and electronic display 300 are installed in a central monitoring center. The image recognition server 600, image extraction server 700, image output server 800, and a hard disk that constitutes the storage unit 900 are installed on the cloud, and the central monitoring center and the cloud are connected by wire for communication via the Internet that constitutes the information and communication network 400.
[0063] Since the preschool child G is included in the blacklist, after a while, the cut-out images of the recognized images of the preschool child G taken by the telepresence robot H, the telepresence robot I, and the telepresence robot J are displayed in sequence on the electronic display 300 installed in the central monitoring center. By comparing the time information digitally time-stamped on the cut-out images with the monitoring screen of the telepresence robot operation management device 200, it is found that the preschool child G has moved from the product display shelf to the toilet area and is currently near the object. On the monitoring screen displayed by the telepresence robot operation management device 200, the distance between the preschool child G and the object by the ToF of the telepresence robot J was about 0.5 m. In addition, in all of the images, the height of the person measured by SfM was always about 90 cm, so it was confirmed that this person was the same preschool child G. The central monitoring center dispatched security guards to the vicinity of the object, protected the preschool child G, and handed him over to his guardian.
[0064] Although each application scenario of the present invention has been described above, the specific scenarios are not limited to these application scenarios, and also include modifications and combinations of scenarios that do not deviate from the gist of the present invention.
[0065] Blind spots for human detection devices such as video cameras and beacons installed on the ceilings and walls of indoor facilities include building entrances, the backs of structures such as pillars and objects, the shadows of product display shelves, restrooms, smoking areas, fitting rooms, employee back areas, and indoor parking lots.
[0066] On the other hand, in the person position identification system 10 according to the present invention, the telepresence robot 100 equipped with a video camera and an environmental shape measuring sensor can move and patrol freely inside an indoor facility, thereby narrowing blind spots in the monitoring location. Also, the server group is divided into two server areas, a local server area and a remote server area, to distribute the calculation processing, and the amount of data transmitted to the information and communication network 400 by the image extraction server 700 is reduced, so that the recognized image is immediately displayed on the electronic display 300, thereby reducing blind spots during monitoring time.
[0067] As described above, the person position identification system of this embodiment can identify the position of a person in real time even in places that are blind spots for fixed video cameras or beacons, by freely patrolling a telepresent robot equipped with a video camera and an environmental shape measurement sensor within an indoor facility.
[0068] The following embodiment may also be adopted. (1) One embodiment is a human position identification system comprising: one or more mobile telepresence robots each equipped with an environmental shape measurement sensor; and a telepresence robot operation management device, the telepresence robot operation management device comprising an environmental display unit that acquires measurement information indicating a position that is a measurement result by the environmental shape measurement sensor and displays a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the telepresence robot.
[0069] (2) Another embodiment is the person position identification system described in (1), wherein the environmental display unit changes the color of the predetermined mark according to the distance from the telepresence robot to the position indicated by the measurement information.
[0070] (3) In another embodiment, the system for identifying human positions is as described in (1) or (2), wherein each of the one or more telepresence robots is equipped with a camera, and the operation management device is equipped with a detected human display unit that selects one image from each of the one or more robots in which the same human is detected from images captured by the camera, and displays the images in a row.
[0071] (4) Another embodiment is a person positioning system according to any one of (1) to (3), wherein each of the one or more telepresence robots is equipped with a camera, and the operation management device is equipped with a robot control unit that causes one of the one or more telepresence robots to track a person detected from an image captured by the camera when the person is not included in a pre-registered whitelist or is included in a pre-registered blacklist.
[0072] (5) Yet another embodiment is a telepresence robot operation management device that acquires measurement information indicating a position that is a measurement result obtained by an environmental shape measurement sensor equipped on one or more mobile telepresence robots, and includes an environmental display unit that displays a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots.
[0073] (6) Yet another embodiment is a telepresence robot operation and management method comprising the steps of acquiring measurement information indicating a position that is a measurement result obtained by an environmental shape measurement sensor provided on one or more mobile telepresence robots, and displaying a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots.
[0074] (7) Another embodiment is a program for causing a computer to function as an environmental display unit that acquires measurement information indicating a position that is a measurement result obtained by an environmental shape measurement sensor equipped on one or more mobile telepresence robots, and displays a predetermined mark at the position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots.
[0075] 1, the functions of the telepresence robot operation management device 200, image input server 500, image recognition server 600, image extraction server 700, and image output server 800 may be realized by recording the programs in a computer-readable recording medium and having a computer system read and execute the programs recorded in the recording medium. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.
[0076] In addition, if a WWW system is used, the "computer system" also includes the homepage providing environment (or display environment). In addition, "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, and storage devices such as hard disks built into computer systems. Furthermore, "computer-readable recording medium" also includes those that dynamically hold a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and those that hold a program for a certain period of time, such as volatile memory inside a computer system that serves as a server or client in such cases. Furthermore, the above program may be one that realizes part of the above-mentioned functions, or may be one that can realize the above-mentioned functions in combination with a program already recorded in the computer system.
[0077] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and design changes and the like that do not deviate from the gist of the present invention are also included. [Explanation of symbols]
[0078] 10 Person Location Identification System 100 Telepresence Robot 200 Telepresence robot operation management device 300 Electronic Display 301 Communications Department 302 Environment display section 303 Robot Control Unit 304 Detected Person Display Unit 400 Information and Communication Network 500 Image Input Server 600 Image Recognition Server 700 Image Extraction Server 800 Image output server 900 Storage section
Claims
1. One or more mobile telepresence robots equipped with environment shape measurement sensors; Telepresence robot operation management device and Equipped with The telepresence robot operation management device includes: an environment display unit that acquires measurement information indicating a position that is a measurement result by the environment shape measurement sensor, and displays a predetermined mark at the position indicated by the measurement information on a map including the periphery of the telepresence robot; A person location identification system comprising:
2. The human position identification system according to claim 1 , wherein the environment display unit changes the color of the predetermined mark according to a distance from the telepresence robot to a position indicated by the measurement information.
3. each of the one or more telepresence robots comprises a camera; The operation management device includes: a detected person display unit that selects images in which the same person is detected from each of the one or more robots among the images captured by the camera, and displays the images in a row. The person location system of claim 1 .
4. each of the one or more telepresence robots comprises a camera; The operation management device includes: A robot control unit that causes any one of the one or more telepresence robots to track a person detected from an image captured by the camera when the person is not included in a preregistered whitelist or is included in a preregistered blacklist. The person location system of claim 1 .
5. An environment display unit that acquires measurement information indicating a position that is a measurement result by an environment shape measurement sensor provided in one or more movable telepresence robots, and displays a predetermined mark at a position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots. A telepresence robot operation management device comprising:
6. Acquiring measurement information indicating a position, which is a measurement result by an environment shape measurement sensor provided in one or more mobile telepresence robots; displaying a predetermined mark at a position indicated by the measurement information on a map including the periphery of the one or more telepresence robots; A telepresence robot operation and management method comprising:
7. Computer, An environment display unit that acquires measurement information indicating a position that is a measurement result by an environment shape measurement sensor provided in one or more movable telepresence robots, and displays a predetermined mark at a position indicated by the measurement information on a map including the surroundings of the one or more telepresence robots. A program to function as a
Citation Information
Patent Citations
Multiple cameras monitoring system, mobile terminal apparatus, center apparatus, and method for monitoring multiple cameras
JP2011114580A
