Patrol robot control system

Abstract

The objective is to provide a patrol robot control system that can determine whether a target is a family member or not, for use as a patrol robot in security for buildings and apartments. [Solution] The control means photographs any person encountered during patrols using a camera, and queries an external server via a storage means or communication means to obtain information about the photographed person. The system then determines whether the person is a family member who is a building contract holder or a resident of the apartment building, and performs different actions depending on whether the person is a family member or not.

Description

Technical Field

[0001] The present invention relates to a system for controlling a patrol robot used in the security industry.

Background Art

[0002] In recent years, robots are expected to rapidly enter human society, including humanoid types. Also, the Three Laws of Robotics are proposed. According to this, it is a principle shown as what a robot should follow in the SF novels of SF writer Isaac Asimov. It is also called the Three Laws of Robotics. It consists of three principles with the aims of "safety to humans, obedience to commands, and self - defense".

[0003] Thus, although there are principles for robots to ensure safety to surrounding humans and obedience to their commands, different actions are required for security personnel who are on the protecting side and malicious humans. Therefore, Patent Document 1 discloses security systems, servers, security devices, and work robot technologies.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] According to this Patent Document 1, a security system 1 is disclosed that includes a work robot 30 that is autonomously mobile and performs predetermined tasks, a security device 20 that monitors the target of surveillance and has a setting means 231 that switches and sets multiple security modes according to the operator's operation, a determination means that determines whether or not the time conditions for the work robot 30 to perform the work are met, and a control means 132 that controls the work performance of the work robot 30 according to the determination result of the time conditions and the type of security mode set.

[0006] However, only the technology for setting and operating robots in various modes has been disclosed. While robots are expected to be rapidly integrated into human society, when used in the security industry, especially for security and patrol duties in buildings and apartments, they must not be something that frightens building owners or apartment residents. On the other hand, it is desirable that the robots be able to act resolutely against intruders in buildings or individuals with malicious intent towards apartment residents.

[0007] The inventors have been developing a patrol robot that is friendly and accepted by building tenants and apartment residents, while being intimidating to outsiders, with the intention of using it in the security industry, particularly for patrol duties within buildings and apartments. The present invention aims to provide a patrol robot control system for security in buildings and apartments that can determine whether a person is an insider who needs to be protected or not. [Means for solving the problem]

[0008] To solve the above problems, in a patrol security robot control system according to one embodiment of the present invention, the robot is provided with a drive means capable of driving the robot to patrol a building or apartment according to a program and to drive various movements of the robot itself, a camera means capable of photographing the surroundings, a communication means for sending and receiving information to and from the outside, a power supply means for supplying power to the various means, a storage means for storing the robot control program and various information, and a control means for controlling these drive means, camera means, communication means, power supply means and storage means. The control means is characterized in that, when encountering a person during patrol, it photographs the person with the camera means, queries an external server via the storage means or communication means for information on the person photographed, determines whether the person is a family member who is a tenant of the building or an apartment resident, and controls the robot to take different actions depending on whether the person is a family member or not.

[0009] Furthermore, the robot is equipped with a microphone that inputs sounds from its surroundings under the control of a control system, allowing it to input voice information from family members. This makes it possible to input information about family members' problems by voice, which will facilitate responses at disaster prevention centers and other locations.

[0010] Furthermore, the robot is equipped with a voice output device that outputs voices under the control of the control device, enabling it to greet and converse with family members.

[0011] Furthermore, if the robot encounters a person who is not a family member, it will issue a warning or ask a question to that person, and will also take a picture with the camera and transmit the information to an external server via the communication means. This allows for an instantaneous response at the disaster prevention center and can also be used as evidence in cases of criminal activity. [Effects of the Invention]

[0012] The patrol robot control system of the present invention maintains the safety for humans and obedience to commands as stipulated in the Three Laws of Robotics. When used as a patrol robot for security work in buildings and apartments, it is accepted with affection by building tenants and apartment residents, and can resolutely perform its patrol duties against malicious third parties. This provides a highly sophisticated patrol robot control system for patrol robots. [Brief explanation of the drawing]

[0013] [Figure 1] This is a schematic diagram of a patrol robot control system according to one embodiment of the present invention. [Figure 2] Figure 1 is a block diagram of the patrol robot control system. [Figure 3] This is a block diagram of the server shown in Figure 1. [Figure 4] This flowchart shows the state of the patrol robot control system of the present invention when it encounters a human. [Figure 5] This diagram shows security robots positioned within the floor and being filmed by multiple fixed cameras. [Figure 6] This is a floor plan showing the layout of the floor and the placement of multiple fixed cameras. [Figure 7] This flowchart shows the operation of map creation by a security robot control system according to one embodiment of the present invention. [Figure 8] This is a flowchart showing the operation of a security robot control system according to one embodiment of the present invention. [Figure 9] This is a flowchart showing the operation of the server related to movement control. [Figure 10] This is a flowchart showing the robot's movements related to motion control. [Modes for carrying out the invention]

[0014] Embodiments of the present invention will be described below.

[0015] (Embodiment) FIG. 1 is a schematic diagram of a patrol robot control system 10 according to an embodiment of the present invention. The patrol robot control system 10 can be used in buildings, condominiums, stores such as bookstores, convenience stores, drugstores, or shopping malls, etc. Hereinafter, the case of using it in a building will be described. The patrol robot control system 10 according to an embodiment of the present invention is provided, for example, with a robot 1A having a vehicle mounted on its legs when moving, or a humanoid robot 1B capable of two-legged walking. Also, as a backup function, it includes a server 2 including a management terminal device 24 and a plurality of fixed cameras 3 for monitoring robots 1A and 1B.

[0016] Robots 1A and 1B can patrol and guard stores, buildings, and condominiums. In the case of a store, there are a large number of people including customers. Also, at night, there are people in buildings and condominiums, and in case of emergency, it may be dangerous not to stop driving.

[0017] Robots 1A and 1B (hereinafter simply referred to as 1) patrol within the floor, perform monitoring, tracking, and voice calling for theft prevention, and provide advertising information to visitors. On the server 2 side including the management terminal device 24, it is an information processing device 24 such as a server device managed by, for example, an operator of the security robot control system 10, and transmits a movement instruction to robot 1. The plurality of fixed cameras 3 are provided in the upper layer part such as the ceiling of the bookstore floor. The plurality of fixed cameras 3 constantly photograph the inside of the floor and transmit the photographed images to the server 2. Robots 1, server 2, and the plurality of fixed cameras 3 are connected by a network N such as an intranet or the Internet. In FIG. 1, one robot 1 is illustrated, but the number of robots 1 is not limited to this, and a plurality of robots 1 may be provided. Also, in FIG. 1, five servers 2 are shown, but the number of servers 2 is not limited to this, and any number of servers 2 may be provided. For example, the number of servers may be one. Also, three fixed cameras 3 are illustrated, but the number of fixed cameras is not limited to this as long as it is two or more.

[0018] The outline of the patrol robot control system 10 according to this embodiment will be described based on FIGS. 1 and 2. The patrol robot control system 10 according to this embodiment is a security robot control system that immediately stops the operation of the driving means 12 of the robot in response to the voice information input to the microphone means 14 provided in the robot 1. The robot 1 further includes a communication means 11 for transmitting and receiving information through an appropriate network, a camera means 13 for photographing the surroundings, a power supply means 18 for supplying power to various means, and a control means 17 for controlling these voice input means 18, driving means 12, communication means 11, camera means 13, and power supply means 18. Basically, when the robot 1 is immediately stopped by the voice information from the microphone means 14, the control means 17 issues a signal indicating the immediate stop to the management terminal device 24 (see FIG. 3), and through the communication means 11, transmits the video information from the camera means 13 and the voice information from the microphone means 14 to the external management terminal device 24, and enters a standby state for receiving command signals from the management terminal device 24.

[0019] In this way, the patrol robot basically has a function of being immediately stopped by voice in accordance with the three laws of robotics. In the present invention, the functions of the patrol robot when patrolling inside a building or a condominium will be described. The patrol robot 1, based on the configuration of FIG. 2, will be described with reference to the server 2 of FIG. 3 and the flowchart of FIG. 4. It starts the patrol according to a predetermined program through the patrol. If there is an encounter with a human in S1, the camera means 13 photographs the human, and inquires about the video to the server storage unit 22 (S2). Then, it determines whether the human is on the side of the building contractor or a family member of the resident of the condominium (S3). If it is determined that the human is a family member, it salutes and, for example, operates to greet with "Good evening!" and responds by voice (step 4 (S4)). On the other hand, if it is determined that the human is not a family member, in step 5 (S5), it asks "Who are you?" or "Do you need anything?", and can also intimidate if it is suspicious. Also, the situation is photographed by the camera means and recorded together with the voice.

[0020] In this way, patrol robot 1 determines whether the person it encounters during its patrol is a family member or not, and responds accordingly by changing its actions and questions, thereby fulfilling its role as patrol robot 1.

[0021] Furthermore, each patrol robot is equipped with a microphone 14 that inputs sounds from its surroundings under the control of a control device, allowing it to input voice information from family members. This makes it possible to input information about family members' problems via voice, facilitating responses at disaster prevention centers and other locations.

[0022] Furthermore, the robot is equipped with a voice output device that outputs voices under the control of the control device, enabling it to greet and converse with family members. This data is also accumulated, increasing the reliability of the data indicating whether someone is a family member or not.

[0023] Furthermore, if the robot encounters a person who is not a family member, it will issue a warning or ask a question to that person, and will also take a picture with the camera and transmit the information to an external server via the communication means. This allows for an instantaneous response at the disaster prevention center and can also be used as evidence in cases of criminal activity.

[0024] Robot 1 is equipped with a voice output means 18, which receives voice information from the management terminal device 24 and enables Robot 1 to emit voice. This allows Robot 1, when its movement or operation is in an emergency stop state, to ask people nearby, "What's wrong?", and to intimidate people nearby who try to destroy or misuse Robot 1 with voice. Even if Robot 1's movement or operation stops, the camera means 13, microphone means 14, voice output means 18, display means 15, and control means 17 continue to function, waiting for commands from the management terminal device 24, allowing for analysis of the cause of the emergency stop and preparation of the next course of action.

[0025] The control means 17 analyzes the cause of the emergency stop based on the audio information from the microphone means 14 and the photographic information from the camera means 13, and transmits the cause of the emergency stop to the management terminal device 24. The management terminal device 24 can send commands to the robot 1 to the extent that it can confirm the cause and the information from the camera means 13.

[0026] Furthermore, the control means 17 is designed to emergency stop the operation of the robot 1 if communication with the management terminal device 24 is interrupted. While momentary interruptions may occur depending on the communication environment, it is best to design the system while considering the level of interruption depending on the situation.

[0027] While a highly designed, bipedal humanoid robot 2B would suffice, the security robot control system 10 according to this embodiment, based on the current wheeled mobile robot 1A, uses multiple fixed cameras 3 to photograph the robot 1 on the floor and estimates the robot's position at the time of photography (hereinafter referred to as the current position) based on the multiple images captured. The server 2 creates movement instructions for the robot 1 to move around the floor based on the floor map and the estimated current position, and transmits these instructions to the robot 1. The robot 1 receives the movement instructions transmitted from the server 2 and moves around the floor based on these instructions. While moving around the floor based on the movement instructions, the robot 1 also continuously captures images of its surroundings and performs actions such as monitoring, tracking, and calling out to shoplifters to prevent theft, and providing advertising information to customers. In this way, the robot 1, server 2, and multiple fixed cameras 3 work together to realize the security robot control system 10. The following describes each component of the security robot control system 10. Furthermore, this mobile robot 1A cannot perform actions such as saluting because its upper body cannot move its hands, but it can move and, since it can receive and generate voice input, it can also converse, enabling a certain degree of communication.

[0028] (Configuration of Robot 1) Robot 1A is a humanoid robot as shown in Figure 1, comprising a head including a face, a torso, and arms. Figure 2 shows a block diagram of Robot 1. Robot 1 comprises a communication means 11, a drive means (for movement) 12, a camera means 13, an audio input means 14, a display means 15 as an output unit, a storage unit 16, and a control means 17. The communication unit 11, the movement means 12, and the audio output means 18 are provided, and each means is connected by a data bus 28.

[0029] The communication means 11 is an interface 95 that communicates with an external device (e.g., a management terminal device 24) wirelessly or via a wired connection to send and receive information. In this embodiment, the communication means 11 can mutually send and receive information with the server 2 via the network N. When communicating with the server 2, the communication means 11 appropriately performs the robot 1's individual authentication process (login process, etc.). This allows the server 2 to determine which of the multiple robots is accessing the server if multiple robots exist. Furthermore, if the login is not legitimate, access to the server 2 is appropriately blocked.

[0030] The drive mechanism 12 moves and rotates the robot 1 body in all directions (forward, backward, left, and right), and is, for example, an omni-wheel. An omni-wheel is a tire that can move in all directions and has multiple rotatable rollers. By controlling the rotation of these rollers, it is possible to move in any direction and also rotate. In this embodiment, since the robot 1 is used in a bookstore, it is preferable to move slowly to avoid collisions with customers, etc. For example, the maximum movement speed by the drive mechanism 12 is set to 2 km / h. Note that the drive mechanism 12 is not limited to an omni-wheel, and any other means of movement can be used.

[0031] Furthermore, with recent advancements in humanoid robot research, robots capable of walking freely are perfectly acceptable. In fact, robots that can move smoothly, avoiding collisions with customers and other pedestrians, are preferable.

[0032] The camera system 13 is used to photograph customers and the like, and includes, for example, at least one 2D camera, which captures images at a predetermined screen resolution and frame rate. The screen resolution is, for example, full HD (1920 x 1080 pixels). The frame rate is, for example, 30fps. For example, there are two 2D cameras, which are installed at different positions on the face of the robot 1 (for example, on the forehead and around the mouth). By placing the 2D cameras at different positions, even if one of the 2D cameras cannot photograph customers and the like (for example, if there is an obstruction directly in front of the forehead and part or all of the shooting range is blocked by the obstruction), the other 2D camera can still photograph customers and the like.

[0033] Robot 1 detects the presence of customers based on images captured by a 2D camera. Specifically, it continuously captures images of the store interior using the 2D camera and transmits the captured images to Server 2 via the communication unit 11. The images may be still images or moving images. Preferably, the shooting unit 13 transmits compressed images to Server 2. For example, the vertical and horizontal dimensions of the image are compressed to 1 / 4 each, resulting in an output image of 480 x 270 pixels. The frame rate is also compressed to 1 / 2, resulting in 15 fps. This improves the efficiency of communication resources related to image transmission and the memory capacity of Server 2.

[0034] Furthermore, the camera means 13 can measure the distance to the object. For example, the camera means 13 may include at least one 3D camera. The 3D camera can obtain the distance to the object by irradiating it with infrared light and detecting the reflected light. The 3D camera acquires 3D images (images including distance information) at a predetermined screen resolution (e.g., 320 x 240 pixels) and a predetermined frame rate (e.g., 20 fps). For example, there are two 3D cameras, each installed at a different position on the face of the robot 1 (e.g., at the position of both eyes). By placing the 3D cameras at different positions, even if one 3D camera cannot photograph a customer (e.g., if there is an obstruction in front of the right eye and the shooting range is blocked by the obstruction), the other 3D camera (the 3D camera installed at the position of the left eye) can still photograph the customer. The robot 1 detects the distance to the object using the 3D camera and moves autonomously using the drive means 12 based on the detected distance.

[0035] Although this invention has been described as a patrol robot for the security industry, the camera means 13 of the robot 1 allows for the identification of a customer based on their facial image information. Based on the customer's pre-stored purchase information, advertising information for that customer can be output by the output means (display means 15) provided on the robot 1. Customer information has traditionally been obtained by distributing point cards, etc. By reading the registered point card (membership card) at the register and linking it with the customer's facial image, customer purchase information can be retrieved, enabling the efficient output of advertising information tailored to the customer's preferences.

[0036] Thus, by understanding the purchasing tendencies of individual customers, it becomes possible to provide more efficient advertising information. Customers will also be less inclined to pay attention to advertisements that are close to what they are looking for, resulting in more effective advertising. If robots can recognize individual customers, customers will proactively approach the robots and seek information. For example, once the robot recognizes a customer, it might say, "Customer, your favorite item, XX, is on sale today."

[0037] The voice input means 14 is a device that accepts voice input and operational input. For example, the voice input means 14 collects the voice of a customer's conversation. The voice input means 14 also accepts input operations such as touching by the customer. For example, the voice input means 14 may include a microphone and an input sensor such as a touch sensor as needed.

[0038] The display means 15 may consist of an audio output means 18 and a device that displays a screen, images, videos, etc. For example, the display means 15 may be an integrated speaker and display. The display may be, for example, a liquid crystal display or an organic EL display. The display means 15 outputs advertising information, etc., received from the server 2 via the communication unit 11. The advertising information may include product commercials, product descriptions, and announcements. The target products of the advertising information include products handled at the location where the robot 1 is placed. However, the target products are not limited to these, and may also be products or services of other stores. For example, it may relate to products and services of another store adjacent to the bookstore where the robot 1 is placed, or if there is a coffee shop adjacent to the bookstore, the menu and services of that coffee shop may be the target of the advertising information. In this way, it is possible to coordinate services between different stores, industries, etc.

[0039] In this embodiment, the robot 1 is equipped with a touch panel on its chest. The touch panel is an input device that accepts input operations from customers and is also an output device that displays various screens, images, videos, etc. In this embodiment, the input function is realized by the touch panel and microphone means 14, and the functions of the display means 15 and audio output means 18 are realized by the touch panel and speaker.

[0040] The storage means 16 includes, for example, a primary storage device and a secondary storage device, and stores various information provided by the server 2 and programs necessary for information processing. For example, the storage means 16 stores the application for the robot of the security robot control system 10 (hereinafter referred to as the robot application). The robot application can be obtained from a predetermined distribution server via, for example, the network N. The operation of the robot 1 according to this embodiment is realized when the robot application is executed (started).

[0041] The control means 17 includes a dedicated or general-purpose processor. The control means 17 controls the operation of the entire robot 1 using a robot application. For example, the control means 17 transmits and receives information via the communication means 11. Specifically, for example, the control means 17 transmits video captured by the camera means 13 to the server 2 via the communication means 11. The control means 17 also transmits the voices of customers collected by the voice input means 14 to the server 2.

[0042] The control means 17 also controls the movement of the robot 1. Based on movement instructions from the server 2, the control means 17 moves the robot 1 using the drive means 12. A movement instruction is, for example, a command to move the robot 1 to a first position different from its current position. Movement instructions include commands for movement and rotation of the robot 1 in the forward, backward, left, and right directions. Specifically, a movement instruction might be, for example, "move 3m forward." In this case, the first position different from the current position is a point 3m ahead in the direction in front of the robot, as viewed from the robot's current position. Movement instructions are not limited to this; they could also be, "rotate 90 degrees clockwise, and then move 4m forward after rotating," etc. Alternatively, a movement instruction may not be a command specifying the direction or distance of movement, but rather a command that directly specifies the control of the drive means 12. For example, a movement instruction may be a command that specifies the rotation speed of the rollers of an omni-wheel. In recent years, the widespread adoption of self-walking humanoid robots is expected, so either approach is acceptable as long as it does not obstruct the flow of customers.

[0043] Here, the state of robot 1 is one of two: waiting for a movement instruction from server 2 (hereinafter also referred to as the standby state) or moving based on a movement instruction from server 2 (hereinafter also referred to as the active state). Whether robot 1 is in the standby state or the active state is stored, for example, in the storage means 16 of robot 1. Alternatively, the state of robot 1 may be transmitted to server 2 and stored in server 2. If robot 1 is not in the standby state (i.e., robot 1 is in the active state), the control means 17 moves robot 1 using the movement means 12 based on a movement instruction received from server 2. On the other hand, if robot 1 is in the standby state, the control unit 17 autonomously moves robot 1 within a predetermined range based on robot 1's current position. In other words, the control means 17 moves robot 1 based on the robot 1's sensors (such as a camera as the imaging unit 13) without relying on a movement instruction. Here, the predetermined range based on the current position is, for example, a radius of 50 cm centered on the current position.

[0044] Furthermore, when movement based on a movement instruction is completed, the control means 17 puts the robot 1 into a standby state. When the robot enters the standby state, as described above, the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position.

[0045] Furthermore, if a predetermined event occurs while the robot 1 is moving based on a movement instruction, the control means 17 cancels the movement instruction and transitions the robot to a standby state. The predetermined event is, for example, an event such as the presence of an obstacle in the direction of the robot 1's movement, in a position that cannot be captured by the fixed camera 3. For example, the obstacle may be luggage, a shopping basket, or a person standing still. The control means 17 determines whether or not a predetermined event has occurred based on input from at least one of the camera means 13 or the voice input means 14. If a predetermined event occurs, the control means 17 cancels the movement instruction from the server 2 and transitions the robot 1 to a standby state. When the robot 1 transitions to a standby state, as described above, the control means 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position.

[0046] (Configuration of Server 2) Figure 3 is a block diagram of a server according to one embodiment of the present invention. Server 2 comprises a server communication unit 21, a server storage unit 22, and a server control unit 23. A management terminal device 24 is connected to Server 2 either directly or via the network N, and the server communication unit 21, the server storage unit 22, and the server control unit 23 are connected to each other by a bus 34.

[0047] The server communication unit 21 is an interface that communicates wirelessly or via wired connection with the management terminal device 24 and other terminal devices to send and receive information. In this embodiment, the server communication unit 21 can send and receive information with the robot 1 and multiple fixed cameras 3 via the network N.

[0048] The server storage unit 22 includes, for example, a primary storage device and a secondary storage device, and stores various information and programs necessary for providing and controlling the security robot control system 10.

[0049] The server control unit 23 includes a dedicated or general-purpose processor and performs various processes. The server control unit 23 controls the operation of the entire server 2. For example, the server control unit 23 sends and receives information via the server communication unit 21. Specifically, the server control unit 23 receives video and audio transmitted from the robot 1 and multiple fixed cameras 3 via the server communication unit 21. The server control unit 23 also stores the received information (video and audio) in the server storage unit 22. The stored information is kept for a certain period of time and then deleted. This period is, for example, one month. The amount of data stored in one month is approximately 1TB when storing video from one robot 1 or one fixed camera 3 (assuming 10 hours / day and 30 days of operation). The server storage unit 22 has the necessary storage capacity.

[0050] The server control unit 23 also authenticates customers based on the received video. When authenticating a customer based on the received video, the server control unit 23 extracts the customer's face image from the video and performs facial recognition based on the database. Artificial intelligence technology is used as appropriate for facial recognition, and the accuracy of authentication is improved through machine learning. The database stores information related to customers (hereinafter also referred to as user information) and is stored in the server storage unit 22. This database includes user information such as user ID, name, face image, gender, age, membership card ID, and preferences. A user ID is an identifier used to uniquely identify a customer. The database stores user information such as the customer's name, face image, gender, age, membership card ID, and preferences, associated with the user ID. User information may include other information, such as a mobile phone number. Information such as purchase history at other stores may also be associated with the mobile phone information as appropriate, in which case it becomes possible to use various information about the customer, such as purchase history at other stores. The server control unit 23 accesses the server storage unit 22 to search for records that match or are similar to the customer's face image included in the video received from the robot 1, and performs facial recognition. Preferably, such a database is encrypted. By encrypting the data, even if the information is leaked to the outside, it becomes virtually impossible to recover information such as the face image, name, and age, thereby appropriately protecting privacy.

[0051] The server control unit 23 also estimates the current position of the robot 1 based on multiple images captured by the multiple fixed cameras 3. Figure 5 shows the robot positioned in the store and being photographed by the multiple fixed cameras 3. In Figure 5, the vertical field of view of the three fixed cameras 3 is α, and the images captured by each fixed camera 3 show the robot 1. In Figure 5, the vertical field of view of all fixed cameras 3 is set to α, but the field of view may be changed for each fixed camera. As shown in Figure 5, the robot 1 moves through the aisles between the bookshelves 4. Therefore, it is preferable to set the height of the robot 1 according to the height of the bookshelves 4 so that it is captured by the multiple fixed cameras 3.

[0052] The server control unit 23 estimates the current position of robot 1 based on the one or more images in which robot 1 is visible, if robot 1 is visible in one or more images. The installation positions of the fixed cameras 3 are predetermined. If robot 1 is visible in two or more images, these two or more images are captured from different directions by different fixed cameras 3 at approximately the same time. Therefore, if the location of robot 1 in two or more images is determined, the current position of robot 1 can be estimated with high accuracy. That is, the current position of robot 1 can be estimated by measuring the horizontal and vertical angles from the images in which robot 1 can be detected within the field of view of each fixed camera 3. Any image recognition method can be used to detect whether or not robot 1 is visible in an image. Also, if the robot is visible in one image, the current position of robot 1 can be estimated from that one image. Specifically, the direction of robot 1 is detected from the image in which robot 1 is visible, and since the height of the fixed camera 3 from the floor is known, the distance from the fixed camera 3 to robot 1 can be calculated based on where robot 1 is located within the image's field of view. Based on this direction and distance, the current position of robot 1 can be estimated.

[0053] On the other hand, if robot 1 is not visible in one or more images, the server control unit 23 estimates the current position of robot 1 based on past movement instructions. In other words, it estimates the current position of robot 1 based on the time a movement instruction was issued in the past and the current time.

[0054] The server control unit 23 also determines whether robot 1 is in a standby state, and if robot 1 is in a standby state, it sends a movement instruction to robot 1. On the other hand, if robot 1 is not in a standby state, the server control unit 23 does not send a movement instruction to robot 1. Whether robot 1 is in a standby state is determined, for example, by receiving status information from robot 1. This determination may be made by determining whether robot 1 has moved to a first position based on a movement instruction within a predetermined time. Alternatively, this determination may be made by determining whether robot 1 is moving within a predetermined range based on its current position.

[0055] As described above, the positions of the multiple fixed cameras 3 are predetermined and are installed in designated locations on the upper part of the floor, such as the ceiling. Figure 6 is an example of a floor plan showing the layout of bookshelves 4 and the arrangement of the multiple fixed cameras 3 when viewed from above inside the store. As shown in Figure 6, the fixed cameras 3 are installed on the upper part of the floor, such as the ceiling, above the movement path of the robot 1, and the shooting directions of adjacent fixed cameras 3 are adjusted to be different. The horizontal field of view of each fixed camera 3 is β. With this arrangement and adjustment of the fixed cameras 3, the robot 1 can be photographed by one or more fixed cameras 3 in most of the floor, and the position of the robot 1 can be estimated with high accuracy. Note that the arrangement, shooting direction, and field of view of the fixed cameras 3 are not limited to this, and the arrangement, shooting angle, and field of view of the fixed cameras 3 can be appropriately set so that each location on the floor can be photographed by one or more fixed cameras 3. Also, although the horizontal field of view of all fixed cameras 3 is set to β, the field of view may be changed for each camera.

[0056] Furthermore, the server control unit 23 can create a map of the entire floor for moving the robot 1 using images captured by multiple fixed cameras 3. Map creation is performed when the server 2 receives a map creation instruction. The map creation instruction may be input to the server 2 by an administrator, or it may be an instruction from an administrator terminal different from the server 2. Alternatively, the server 2 may automatically generate the map creation instruction when the security robot control system 10 is initially started up or periodically. Upon receiving a map creation instruction, the server control unit 23 receives images captured by all fixed cameras 3 installed on the floor and creates a map based on these images. Based on this map, the server control unit 23 generates movement instructions for the locations and paths where the robot 1 can move.

[0057] Furthermore, the server control unit 23 may generate movement instructions based on images of the robot 1's surroundings. Specifically, the server control unit 23 determines from the images of the fixed camera 3 whether or not there are obstacles or people around the robot 1. If it determines that there are obstacles or people around the robot 1, the server control unit 23 generates movement instructions to avoid those obstacles or people. In other words, it does not generate movement instructions that would cause the robot to collide with those obstacles or people. This makes it possible for the robot 1 to move while avoiding obstacles and people.

[0058] (Configuration of Fixed Camera 3) Multiple fixed cameras 3 are installed at predetermined locations in the upper part of the floor, such as the ceiling, as described above. Conventional security cameras installed in the upper part of the floor, such as the ceiling, may be used as multiple fixed cameras 3. Multiple fixed cameras 3 may also be attached to pipes for installing security cameras. In this case as well, images of the floor can be captured from the upper part of the floor. Alternatively, multiple fixed cameras 3 may be installed in the upper part of the side walls of the floor. The fixed cameras 3 acquire images at a predetermined screen resolution (e.g., 1920 x 1080 pixels) and at predetermined time intervals, and transmit the acquired images to the server 2.

[0059] (Operation of the security robot control system 10) Next, the operation of the security robot control system 10 according to one embodiment of the present invention will be explained using the flowcharts shown in Figures 7 to 10.

[0060] Figure 7 is a flowchart illustrating the operation of map creation by a security robot control system 10 according to one embodiment of the present invention. First, the server 2 receives a map creation instruction via the server communication unit 21 (step S10). Upon receiving the map creation instruction, the server control unit 23 receives images captured by all fixed cameras 3 installed on the floor via the server communication unit 21 (step S20). Subsequently, the server control unit 23 creates a map based on the received images (step S30). Thus, with the security robot control system 10 according to this embodiment, a map of the floor can be created without operating the robot 1, and therefore, a map can be created at a higher speed compared to SLAM (Simultaneously Localization and Mapping) in conventional robots. Furthermore, since the installation locations of the multiple fixed cameras 3 are known in advance, a highly accurate map can be created.

[0061] Figure 8 is a flowchart showing the operations related to the estimation of the current position and movement control of the robot 1 by the security robot control system 10 according to one embodiment of the present invention.

[0062] First, the server 2 receives multiple images captured by multiple fixed cameras 3 via the server communication unit 21 (step S100).

[0063] Next, the server control unit 23 estimates the current position of the robot 1 based on multiple images captured by the multiple fixed cameras 3. First, the server control unit 23 determines whether or not the robot 1 is visible in one or more images (step S110). If the robot 1 is visible in one or more images (step S110: yes), the server control unit 23 estimates the current position of the robot 1 based on the one or more images in which the robot 1 is visible (step S120). On the other hand, if the robot 1 is not visible in one or more images (step S110: no), the server control unit 23 estimates the current position of the robot 1 based on past movement instructions (step S130).

[0064] After estimating the current position of robot 1, the server control unit 23 performs movement control of robot 1 (step S140). Movement control is mainly performed by movement instructions transmitted from the server control unit 23 to robot 1. Movement instructions are created based on the map created by server 2 and the estimated current position of robot 1.

[0065] Figures 8 to 10 are flowcharts showing the operation related to the drive control of robot 1. Figure 9 is a flowchart showing the operation of server 2, including emergency stop, and Figure 10 is a flowchart showing the operation of robot 1. First, the operation of server 2 will be explained with reference to Figure 9.

[0066] First, the server control unit 23 commands patrol and security (step S190), and when robot 1 starts moving toward patrol and security, robot 1 is configured to constantly determine whether it has received an emergency stop voice message (step S200). When it receives an emergency stop command by voice or information such as a scream from an external source, robot 1 performs an emergency stop, analyzes the cause of the emergency stop, transmits the analysis results to the management terminal device, and enters a standby state (steps S200, S205, S210).

[0067] Meanwhile, on the management terminal device 24 side, when robot 1 is in an emergency stop state and waiting to receive a command signal, depending on the cause of the emergency stop, robot 1 can report the cause, and if assistance is needed or the situation needs to be checked based on the camera means 13 and microphone means 14, it can communicate with people around it by voice, saying things like, "What's wrong?"

[0068] Furthermore, in cases where the cause of the stoppage is a malicious third party, such as illegally stopping the robot, illegally issuing stop commands or screams, or destroying the robot, the management terminal device 24 can restart the robot 1, and the voice output means 18 can also emit a threatening sound. Thus, as a security robot 1, it becomes necessary to control the robot 1 while anticipating all possible scenarios.

[0069] Next, the operation of robot 1 will be explained with reference to Figure 10. First, the control means 17 of robot 1 determines whether or not its own state is in standby mode (step S300). If it is not in standby mode (step S300: no), the control unit 17 moves robot 1 using the movement means 12 based on a movement instruction from server 2 (step S310).

[0070] Next, the control means 17 determines whether a predetermined event has occurred while the robot 1 is moving based on the movement instruction (step S320). If it is determined that no predetermined event has occurred (step S320: no), the process proceeds to step S330.

[0071] When movement based on the movement instruction is completed (step S330), the control means 17 puts the robot 1 into a standby state (step S340). Then the control means 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position (step S350).

[0072] If it is determined in step S300 that the robot 1 is in a standby state, the process proceeds to step S350, where the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position.

[0073] Furthermore, if it is determined in step S320 that a predetermined event has occurred, the process proceeds to step S340, where the robot 1 is put into a standby state, and then the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position (step S350).

[0074] As described above, according to the patrol robot control system 10 of one embodiment of the present invention, the current position of the robot 1 is estimated from multiple images captured by multiple fixed cameras 3, and movement instructions are transmitted to the robot 1 based on the estimated current position to control the movement of the robot 1. Therefore, the current position of the robot 1 and its movement can be estimated and controlled without installing RFID or markers.

[0075] Furthermore, according to the security robot control system 10 of one embodiment of the present invention, since the movement of the robot 1 is controlled after understanding the situation of most of the floor area using multiple fixed cameras 3, the robot 1 can be moved and positioned to a more suitable location, and it is expected that the effectiveness of actions performed by the robot 1 for shoplifting prevention, such as monitoring, tracking, calling out, and providing advertising information to customers will be enhanced. In addition, even when controlling the movement of multiple robots 1 on the floor, the positioning and movement control of the multiple robots can be performed while considering the situation of the entire floor.

[0076] Furthermore, in the security robot control system 10 according to one embodiment of the present invention, when the robot 1 is in a standby state, i.e., when it has not received a movement instruction, the robot 1 autonomously moves within a predetermined range based on its current position. As a result, the robot 1 appears to be constantly moving autonomously, which can enhance the effectiveness of shoplifting prevention and advertising. In addition, because the robot 1 is constantly moving, images and other data around the robot 1 can be acquired over a wider area.

[0077] Furthermore, if the robot 1 is not visible in one or more of the images captured by the multiple fixed cameras 3, the server control unit 23 of the server 2 may send a movement instruction to move the robot 1 into the shooting range that can be simultaneously captured by the multiple fixed cameras 3. In this way, the robot 1 will be captured by the multiple fixed cameras 3 after moving based on the movement instruction, and the current position of the robot 1 can be estimated with high accuracy.

[0078] Furthermore, if robot 1 does not move to the first position even after a predetermined number of movement instructions have been sent to robot 1, server 2 may send a movement instruction to move to a second position different from the first position. For example, if there is an obstacle in the path of robot 1, even if robot 1 tries to move based on the movement instruction, the same predetermined event may occur each time, potentially preventing it from moving to the first position. To prevent robot 1 from becoming unable to move due to this event, the number of times the same movement instruction can be resent is limited to a predetermined number of times. If movement based on the movement instruction is not performed even after the predetermined number of transmissions, the destination is changed to a second position different from the first position. This suppresses situations in which robot 1 is unable to move and allows for efficient movement control of robot 1.

[0079] Furthermore, if the robot 1 does not move to the first position even after the server 2 has sent a predetermined number of movement instructions to move to the first position, the server control unit 23 may estimate that there is an obstacle between the robot 1 and the first position. Based on this estimation, the server control unit 23 may also update the floor map to include information that an obstacle exists. By updating the map, the server control unit 23 can create movement instructions for the robot 1 taking into account the presence of the obstacle, thereby suppressing situations where the robot 1 is unable to move and enabling efficient movement control of the robot 1.

[0080] In this embodiment, the robot 1 is configured to move autonomously within a predetermined range based on its current position while in standby mode. However, the robot 1 may be configured to stop while in standby mode. If the robot 1 is not given the function of autonomous movement, its functions can be simplified.

[0081] In this embodiment, an example was described in which the server 2 estimates the position of robot 1 and creates movement instructions for movement control. However, robot 1 may, for example, have some or all of the functions of server 2. For example, server 2 may estimate the current position of robot 1, server 2 may transmit the current position information to robot 1, and robot 1 may perform movement control based on the current position information. If robot 1 has all the functions of server 2, a security robot control system can be configured with robot 1 and multiple fixed cameras 3 without providing server 2.

[0082] The present invention has been described above based on various drawings and embodiments. However, it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present invention. For example, the functions included in each means, step, etc., can be rearranged in a logically consistent manner, and multiple means or steps, etc., can be combined into one or divided. [Explanation of symbols]

[0083] 1A, 1B Robots 2 servers 3 Fixed camera 4 bookshelves 10. Security robot control system 11. Means of communication 12 Driving means 13 Camera methods 14 Microphone methods 15. Audio output means 16 Memory means 17 Control means Buses 18 and 24 21 Server Communication Section 22 Server Storage Unit 23 Server Control Unit 24 Management terminal device

Claims

1. The robot is equipped with drive means capable of driving the robot to patrol buildings and apartments according to a program and to drive various movements of the robot itself, camera means capable of photographing the surroundings, communication means for sending and receiving information to and from the outside, power means for supplying power to the various means, storage means for storing robot control programs and various information, and control means for controlling these drive means, camera means, communication means, power means, and storage means. The control means is characterized by taking a photograph of any person encountered during patrol using the camera means, querying an external server via the storage means or communication means for information about the person, determining whether the person is a building tenant or apartment resident, and then performing different actions depending on whether the person is a family member or not.

2. The patrol robot control system according to claim 1, characterized in that the robot is provided with a microphone means for inputting sounds from around the robot under the control of a control means, and is also capable of inputting voice information from within the robot.

3. The patrol robot control system according to claim 1 to 2, further characterized in that the robot is provided with a voice output means that outputs voice from the robot under the control of the control means, enabling it to converse with family members.

4. The patrol robot control system according to any one of claims 1 to 3, characterized in that the robot issues a warning or questions to the person it encounters if that person is not a family member, and also takes a photograph with the camera means and transmits the information to an external server via the communication means.

Images