Robot movement control system

Abstract

We provide a mobile control system that can estimate the current position of a robot and control its movement, as well as a mobile control system that collects information and displays advertisements within a store. [Solution] The system includes multiple fixed cameras installed on the upper part of the floor, such as the ceiling, to repeatedly photograph the inside of the floor, and a robotic camera installed on the robot itself to photograph its surroundings. The server estimates the current position of the robot based on multiple images taken from different directions substantially simultaneously by the multiple fixed cameras and the robotic camera, and controls the movement of the robot based on the current position estimated from the multiple images taken by the fixed cameras and the robotic camera.

Description

Technical Field

[0001] The present invention relates to a movement control system and a movement control method for estimating the current position of a robot and performing movement control of the robot.

Background Art

[0002] Conventionally, techniques for controlling the movement of a moving device such as a robot have been known. Conventionally, in order to confirm the current position of a moving device such as a robot, markers, RFID (Radio Frequency Identification System), etc. have been installed on the floor surface of the target location, and based on the information from them, the current position of the moving device has been confirmed and movement control has been performed (for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, when installing markers or RFID on the floor surface of the target location, it takes time and additional costs are required, increasing the cost. Also, when installing markers on the floor surface, if the markers wear due to aging deterioration, it becomes difficult to estimate the current position of the moving device, so the markers must be replaced, which is time-consuming and further increases the cost. In addition, the markers must be in a form that can be recognized by image recognition by the moving device, and there is also a possibility of affecting the aesthetics of the floor surface.

[0005] Therefore, an object of the present invention made in view of the above problems is to provide a movement control system that can estimate the current position of a robot and perform movement control without installing RFID, markers, etc., and a movement control system that performs information collection and advertisement display in a store. [Means for solving the problem]

[0006] To solve the above problems, a movement control system according to one embodiment of the present invention comprises a server, a robot that patrols within a floor, a plurality of fixed cameras provided on the upper part of the floor, such as the ceiling, that repeatedly photograph the inside of the floor, and a robot camera provided on the robot itself that photographs its surroundings. The server estimates the current position of the robot based on a plurality of images taken substantially simultaneously from different directions by the plurality of fixed cameras and the robot camera, and each time the current position of the robot is estimated, it determines whether the robot is in a standby state waiting for a movement instruction from the server. If the robot is in the standby state, it transmits a movement instruction to the robot to move it to a first position different from its current position based on the current position estimated from the plurality of images, and does not transmit the movement instruction if the robot is not in the standby state.

[0007] As an example, the robot camera is characterized by employing a 360-degree camera to capture images of the customer and their surroundings.

[0008] Furthermore, a key feature of this system is that it outputs advertising information or store information for the displayed products based on the robot's position. This allows for efficient output of product advertising information at predetermined product display locations, or output of store information at designated locations.

[0009] Furthermore, a key feature of this system is that, if the robot camera can identify a customer using their facial image information, it can output advertising information to that customer using an output means provided on the robot, based on the customer's pre-stored purchase information. At the register, by reading a pre-registered membership card and linking it with the customer's facial image, customer purchase information can be retrieved, enabling efficient output of advertising information tailored to the customer's preferences.

[0010] According to a movement control method of one embodiment of the present invention, the movement control system comprises a server, a robot that moves within a floor, a plurality of fixed cameras installed on the upper part of the floor such as the ceiling, and a robot camera installed on the robot itself, and includes the steps of accurately estimating the current position of the robot based on a plurality of images taken by the plurality of fixed cameras and the robot camera, and accurately recognizing the current position of the robot based on the estimated current position. [Effects of the Invention]

[0011] According to the movement control system of the present invention, it is possible to move by free walking without installing RFID or markers, and to easily estimate the robot's current position and control its movement. [Brief explanation of the drawing]

[0012] [Figure 1] This is a schematic diagram of a mobile control system according to embodiment of the present invention. [Figure 2] Figure 1 is a block diagram of the robot. [Figure 3] This is a block diagram of the server shown in Figure 1. [Figure 4] This diagram shows robots positioned within the floor and being filmed by multiple fixed cameras. [Figure 5] This is a floor plan showing the layout of the floor and the placement of multiple fixed cameras. [Figure 6] This flowchart shows the operation of map creation by a mobile control system according to one embodiment of the present invention. [Figure 7] A flowchart illustrating the operation of a motion control system according to one embodiment of the present invention. [Figure 8] This is a flowchart showing the operation of the server related to movement control. [Figure 9] This is a flowchart showing the robot's movements related to motion control. [Modes for carrying out the invention]

[0013] Embodiments of the present invention will be described below.

[0014] (Embodiment) Figure 1 is a schematic diagram of a mobile control system 10 according to one embodiment of the present invention. The mobile control system 10 can be used in stores such as bookstores, convenience stores, drugstores, and shopping malls, and the following will describe its use in a bookstore. The mobile control system 10 according to one embodiment of the present invention comprises a robot 1, a server 2, and a plurality of fixed cameras 3. The robot 1 patrols the floor and performs monitoring, tracking, and verbal communication to prevent shoplifting, as well as providing advertising information to customers. The server 2 is an information processing device such as a server device managed by, for example, the operator of the mobile control system 10, and transmits movement instructions to the robot 1. The plurality of fixed cameras 3 are installed in the upper part of the bookstore floor, such as the ceiling. The plurality of fixed cameras 3 constantly photograph the inside of the floor and transmit the captured images to the server 2. The robot 1, server 2, and plurality of fixed cameras 3 are connected by a network N such as an intranet or the internet. In Figure 1, one robot 1 is shown as an example, but the number of robots 1 is not limited to this, and multiple robots 1 may be provided. Furthermore, although Figure 1 shows five servers 2, the number of servers 2 is not limited to this, and any number of servers 2 may be provided; for example, there may be just one server. Also, although three fixed cameras 3 are shown as an example, the number of fixed cameras is not limited to two or more.

[0015] The outline of the movement control system 10 according to this embodiment will be described. The movement control system 10 according to this embodiment uses a plurality of fixed cameras 3 to photograph the robot 1 within the floor, and based on the plurality of photographed images, estimates the position where the robot exists at the time of photographing (hereinafter referred to as the current position). The server 2 creates a movement instruction for the robot 1 to move within the floor based on the map of the floor and the estimated current position, and transmits the movement instruction to the robot 1. The robot 1 receives the movement instruction transmitted from the server 2 and moves within the floor based on the movement instruction. Also, while moving within the floor based on the movement instruction, the robot 1 constantly photographs, for example, the surrounding video, and performs operations such as monitoring for theft prevention, tracking, and speaking, and providing advertising information to visitors. In this way, the robot 1, the server 2, and the plurality of fixed cameras 3 cooperate to realize the movement control system 10. Hereinafter, each component of the movement control system 10 will be described.

[0016] (Configuration of Robot 1) First, the configuration of the robot 1 will be described. Preferably, the robot 1 is a humanoid robot as shown in FIG. 1 and includes a head including a face, a torso, and arms. FIG. 2 shows a block diagram of the robot 1. The robot 1 includes a communication unit 11, a moving means 12, a photographing unit 13, an input unit 14, an output unit 15, a storage unit 16, and a control unit 17. The communication unit 11, the moving means 12, the photographing unit 13, the input unit 14, the output unit 15, the storage unit 16, and the control unit 17 are respectively connected by a bus 18.

[0017] The communication unit 11 is an interface that communicates with an external device wirelessly or by wire and performs transmission and reception of information. In this embodiment, the communication unit 11 can mutually transmit and receive information with the server 2 via the network N. When the communication unit 11 communicates with the server 2, it appropriately performs the solid authentication process (login process, etc.) of the robot 1. Thereby, when there are a plurality of robots, the server 2 can grasp from which robot among the plurality of robots the access is made. Also, when the login is not regular, the access to the server 2 is appropriately blocked.

[0018] The moving means 12 is for moving and rotating the robot 1 body in all directions (front, back, left, and right), and for example, it is an omni-wheel. The omni-wheel is a tire that can move in all directions and has a plurality of rotatable rollers. By controlling the rotation of the rollers, it is possible to move in any direction and also perform a rotating operation. In this embodiment, since the robot 1 is used in a bookstore, it is preferable to move slowly so as not to collide with customers or the like. For example, the maximum value of the moving speed by the moving means 12 is set to 2 km / h. Note that the moving means 12 is not limited to an omni-wheel, and any moving means can be adopted.

[0019] Recently, the research on humanoid robots has been progressing. Even a robot that can walk freely without problems, rather, it is preferable that it can move smoothly while avoiding collisions with customers or the like and bypassing customers.

[0020] The imaging unit 13 is for imaging customers or the like, and for example, includes at least one 2D camera and captures images at a predetermined screen resolution and frame rate. The screen resolution is, for example, full high vision (1920×1080 pixels). Also, the frame rate is, for example, 30 fps. For example, the number of 2D cameras is two, and they are respectively provided at different positions on the face of the robot 1 (for example, the position of the forehead and the position near the mouth). By arranging 2D cameras at different positions, even when one 2D camera cannot image customers or the like (for example, when there is an obstacle immediately in front of the forehead and a part or all of the imaging range is blocked by the obstacle), it is possible to image customers or the like with the other 2D camera.

[0021] Robot 1 detects the presence of customers based on images captured by a 2D camera. Specifically, it continuously captures images of the store interior using the 2D camera and transmits the captured images to Server 2 via the communication unit 11. The images may be still images or moving images. Preferably, the shooting unit 13 transmits compressed images to Server 2. For example, the vertical and horizontal dimensions of the image are compressed to 1 / 4 each, resulting in an output image of 480 x 270 pixels. The frame rate is also compressed to 1 / 2, resulting in 15 fps. This improves the efficiency of communication resources related to image transmission and the memory capacity of Server 2.

[0022] Furthermore, the imaging unit 13 can measure the distance to the object. For example, the imaging unit 13 may include at least one 3D camera. The 3D camera can acquire the distance to the object by irradiating it with infrared light and detecting the reflected light. The 3D camera acquires 3D images (images including distance information) at a predetermined screen resolution (e.g., 320 x 240 pixels) and a predetermined frame rate (e.g., 20 fps). For example, there are two 3D cameras, each installed at a different position on the face of the robot 1 (e.g., at the position of both eyes). By placing the 3D cameras at different positions, even if one 3D camera cannot photograph a customer (e.g., if there is an obstruction in front of the right eye and the shooting range is blocked by the obstruction), the other 3D camera (the 3D camera installed at the position of the left eye) can still photograph the customer. The robot 1 detects the distance to the object using the 3D camera and moves autonomously using the movement means 12 based on the detected distance.

[0023] Furthermore, if the robot camera (shooting unit 13) can identify a customer using their facial image information, the robot can output advertising information to that customer using an output means (display unit 15) provided on the robot, based on the customer's pre-stored purchase information. Customer information has traditionally been obtained by distributing point cards, and by reading the registered point card (membership card) at the register and linking it with the customer's facial image, customer purchase information can be retrieved, enabling efficient output of advertising information tailored to the customer's preferences.

[0024] In this way, understanding the purchasing tendencies of individual customers allows for the provision of more efficient advertising information. Customers will also be less inclined to pay attention to advertisements that are close to what they are looking for, resulting in more effective advertising. If robots can recognize individual customers, customers will proactively approach the robots and seek information. For example, once the robot recognizes a customer, it might say, "Customer, your favorite item, XX, is on sale today."

[0025] The input unit 14 is a device that accepts voice input and operation input. For example, the input unit 14 collects the voice of a customer's conversation. The input unit 14 also accepts input operations such as touch from the customer. For example, the input unit 14 is composed of a microphone and an input sensor such as a touch sensor.

[0026] The output unit 15 is a device that outputs audio and displays screens, images, videos, etc. For example, the output unit 15 is composed of a speaker and a display. The display is, for example, a liquid crystal display or an organic EL display. The output unit 15 outputs advertising information, etc., received from the server 2 via the communication unit 11. The advertising information includes product commercials, product descriptions, and announcements. The target products of the advertising information include products handled at the location where robot 1 is deployed. However, the target products are not limited to these, and may also be products or services of other stores. For example, it may relate to products and services of another store adjacent to the bookstore where robot 1 is deployed, or if there is a coffee shop adjacent to the bookstore, the menu and services of that coffee shop may be the target of the advertising information. In this way, it is possible to coordinate services between different stores, industries, etc.

[0027] In this embodiment, a touch panel is provided on the chest of robot 1. The touch panel is an input device that accepts input operations from customers and is also an output device that displays various screens, images, videos, etc. In this embodiment, the functions of the input unit 14 are realized by the touch panel and microphone, and the functions of the output unit 15 are realized by the touch panel and speaker.

[0028] The storage unit 16 includes, for example, a primary storage device and a secondary storage device, and stores various information provided by the server 2 and programs necessary for information processing. For example, the storage unit 16 stores the robot application for the mobile control system 10 (hereinafter referred to as the robot application). The robot application can be obtained from a predetermined distribution server via, for example, the network N. The operation of the robot 1 according to this embodiment is realized when the robot application is executed (started).

[0029] The control unit 17 includes a dedicated or general-purpose processor. The control unit 17 controls the operation of the entire robot 1 using a robot application. For example, the control unit 17 transmits and receives information via the communication unit 11. Specifically, for example, the control unit 17 transmits video captured by the imaging unit 13 to the server 2 via the communication unit 11. The control unit 17 also transmits the voice of a customer collected by the input unit 14 to the server 2.

[0030] The control unit 17 also controls the movement of the robot 1. Based on movement instructions from the server 2, the control unit 17 moves the robot 1 using the movement means 12. A movement instruction is, for example, a command to move the robot 1 to a first position different from its current position. Movement instructions include commands for movement and rotation of the robot 1 in the forward, backward, left, and right directions. Specifically, a movement instruction might be, for example, "move 3m forward." In this case, the first position different from the current position is a point 3m ahead in the direction in front of the robot, as viewed from the robot's current position. Movement instructions are not limited to this; they could also be, "rotate 90 degrees clockwise, and then move 4m forward after the rotation." Alternatively, a movement instruction may not be a command specifying the direction or distance of movement, but rather a command that directly specifies the control of the movement means 12. For example, a movement instruction may be a command that specifies the rotation speed of the rollers of an omni-wheel. In recent years, the widespread adoption of self-walking humanoid robots is expected, so either approach is acceptable as long as it does not obstruct the flow of customers.

[0031] Here, the state of robot 1 is one of two: waiting for a movement instruction from server 2 (hereinafter also referred to as the standby state) or moving based on a movement instruction from server 2 (hereinafter also referred to as the active state). Whether robot 1 is in the standby state or the active state is stored, for example, in the memory unit 16 of robot 1. Alternatively, the state of robot 1 may be transmitted to server 2 and stored in server 2. If robot 1 is not in the standby state (i.e., robot 1 is in the active state), the control unit 17 moves robot 1 using the movement means 12 based on a movement instruction received from server 2. On the other hand, if robot 1 is in the standby state, the control unit 17 autonomously moves robot 1 within a predetermined range based on robot 1's current position. In other words, the control unit 17 moves robot 1 based on the robot 1's sensors (such as a camera as the imaging unit 13) without relying on a movement instruction. Here, the predetermined range based on the current position is, for example, a radius of 50 cm centered on the current position.

[0032] Furthermore, when movement based on a movement instruction is completed, the control unit 17 puts the robot 1 into a standby state. When the robot enters the standby state, as described above, the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position.

[0033] Furthermore, if a predetermined event occurs while the robot 1 is moving based on a movement instruction, the control unit 17 cancels the movement instruction and transitions the robot to a standby state. The predetermined event is, for example, an event such as the presence of an obstacle in the direction of the robot 1's movement, in a position that cannot be captured by the fixed camera 3. For example, the obstacle may be luggage, a shopping basket, or a person standing still. The control unit 17 determines whether or not a predetermined event has occurred based on input from at least one of the imaging unit 13 or the input unit 14. If a predetermined event occurs, the control unit 17 cancels the movement instruction from the server 2 and transitions the robot 1 to a standby state. When the robot 1 transitions to a standby state, as described above, the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position.

[0034] (Configuration of Server 2) Figure 3 is a block diagram of a server according to one embodiment of the present invention. Server 2 comprises a server communication unit 21, a server storage unit 22, and a server control unit 23. The server communication unit 21, the server storage unit 22, and the server control unit 23 are connected to each other by a bus 24.

[0035] The server communication unit 21 is an interface that communicates with external devices wirelessly or via wired connections to send and receive information. In this embodiment, the server communication unit 21 can send and receive information with the robot 1 and multiple fixed cameras 3 via the network N.

[0036] The server storage unit 22 includes, for example, a primary storage device and a secondary storage device, and stores various information and programs necessary for providing and controlling the mobile control system 10.

[0037] The server control unit 23 includes a dedicated or general-purpose processor and performs various processes. The server control unit 23 controls the operation of the entire server 2. For example, the server control unit 23 sends and receives information via the server communication unit 21. Specifically, the server control unit 23 receives video and audio transmitted from the robot 1 and multiple fixed cameras 3 via the server communication unit 21. The server control unit 23 also stores the received information (video and audio) in the server storage unit 22. The stored information is kept for a certain period of time and then deleted. This period is, for example, one month. The amount of data stored in one month is approximately 1TB when storing video from one robot 1 or one fixed camera 3 (assuming 10 hours / day and 30 days of operation). The server storage unit 22 has the necessary storage capacity.

[0038] The server control unit 23 also authenticates customers based on the received video. When authenticating a customer based on the received video, the server control unit 23 extracts the customer's face image from the video and performs facial recognition based on the database. Artificial intelligence technology is used as appropriate for facial recognition, and the accuracy of authentication is improved through machine learning. The database stores information related to customers (hereinafter also referred to as user information) and is stored in the server storage unit 22. This database includes user information such as user ID, name, face image, gender, age, membership card ID, and preferences. A user ID is an identifier used to uniquely identify a customer. The database stores user information such as the customer's name, face image, gender, age, membership card ID, and preferences, associated with the user ID. User information may include other information, such as a mobile phone number. Information such as purchase history at other stores may also be associated with the mobile phone information as appropriate, in which case it becomes possible to use various information about the customer, such as purchase history at other stores. The server control unit 23 accesses the server storage unit 22 to search for records that match or are similar to the customer's face image included in the video received from the robot 1, and performs facial recognition. Preferably, such a database is encrypted. By encrypting the data, even if the information is leaked to the outside, it becomes virtually impossible to recover information such as the face image, name, and age, thereby appropriately protecting privacy.

[0039] The server control unit 23 also estimates the current position of the robot 1 based on multiple images captured by the multiple fixed cameras 3. Figure 4 shows the robot positioned in the store and being photographed by the multiple fixed cameras 3. In Figure 4, the vertical field of view of the three fixed cameras 3 is α, and the images captured by each fixed camera 3 show the robot 1. In Figure 4, the vertical field of view of all fixed cameras 3 is set to α, but the field of view may be changed for each fixed camera. As shown in Figure 4, the robot 1 moves through the aisles between the bookshelves 4. Therefore, it is preferable to set the height of the robot 1 according to the height of the bookshelves 4 so that it is visible to the multiple fixed cameras 3.

[0040] The server control unit 23 estimates the current position of robot 1 based on the one or more images in which robot 1 is visible, if robot 1 is visible in one or more images. The installation positions of the fixed cameras 3 are predetermined. If robot 1 is visible in two or more images, these two or more images are captured from different directions by different fixed cameras 3 at approximately the same time. Therefore, if the location of robot 1 in two or more images is determined, the current position of robot 1 can be estimated with high accuracy. That is, the current position of robot 1 can be estimated by measuring the horizontal and vertical angles from the images in which robot 1 can be detected within the field of view of each fixed camera 3. Any image recognition method can be used to detect whether or not robot 1 is visible in an image. Also, if the robot is visible in one image, the current position of robot 1 can be estimated from that one image. Specifically, the direction of robot 1 is detected from the image in which robot 1 is visible, and since the height of the fixed camera 3 from the floor is known, the distance from the fixed camera 3 to robot 1 can be calculated based on where robot 1 is located within the image's field of view. Based on this direction and distance, the current position of robot 1 can be estimated.

[0041] On the other hand, if robot 1 is not visible in one or more images, the server control unit 23 estimates the current position of robot 1 based on past movement instructions. In other words, it estimates the current position of robot 1 based on the time a movement instruction was issued in the past and the current time.

[0042] The server control unit 23 also determines whether robot 1 is in a standby state, and if robot 1 is in a standby state, it sends a movement instruction to robot 1. On the other hand, if robot 1 is not in a standby state, the server control unit 23 does not send a movement instruction to robot 1. Whether robot 1 is in a standby state is determined, for example, by receiving status information from robot 1. This determination may be made by determining whether robot 1 has moved to a first position based on a movement instruction within a predetermined time. Alternatively, this determination may be made by determining whether robot 1 is moving within a predetermined range based on its current position.

[0043] As described above, the positions of the multiple fixed cameras 3 are predetermined and are installed in designated locations on the upper part of the floor, such as the ceiling. Figure 5 is an example of a plan view showing the layout of bookshelves 4 and the arrangement of the multiple fixed cameras 3 when viewed from above inside the store. As shown in Figure 5, the fixed cameras 3 are installed on the upper part of the floor, such as the ceiling, above the movement path of the robot 1, and the shooting directions of adjacent fixed cameras 3 are adjusted to be different. The horizontal field of view of each fixed camera 3 is β. With this arrangement and adjustment of the fixed cameras 3, the robot 1 can be photographed by one or more fixed cameras 3 in most of the floor, and the position of the robot 1 can be estimated with high accuracy. Note that the arrangement, shooting direction, and field of view of the fixed cameras 3 are not limited to this, and the arrangement, shooting angle, and field of view of the fixed cameras 3 can be appropriately set so that each location on the floor can be photographed by one or more fixed cameras 3. Also, although the horizontal field of view of all fixed cameras 3 is set to β, the field of view may be changed for each camera.

[0044] Furthermore, the server control unit 23 can create a map of the entire floor for moving the robot 1 using images captured by multiple fixed cameras 3. Map creation is performed when the server 2 receives a map creation instruction. The map creation instruction may be input to the server 2 by an administrator, or it may be an instruction from an administrator terminal different from the server 2. Alternatively, the server 2 may automatically generate the map creation instruction when the movement control system 10 is initially started up or periodically. Upon receiving a map creation instruction, the server control unit 23 receives images captured by each fixed camera 3 installed on the floor and creates a map based on these images. Based on this map, the server control unit 23 generates movement instructions for the locations and paths where the robot 1 can move.

[0045] Furthermore, the server control unit 23 may generate movement instructions based on images of the robot 1's surroundings. Specifically, the server control unit 23 determines from the images of the fixed camera 3 whether or not there are obstacles or people around the robot 1. If it determines that there are obstacles or people around the robot 1, the server control unit 23 generates movement instructions to avoid those obstacles or people. In other words, it does not generate movement instructions that would cause the robot to collide with those obstacles or people. This makes it possible for the robot 1 to move while avoiding obstacles and people.

[0046] (Configuration of Fixed Camera 3) Multiple fixed cameras 3 are installed at predetermined locations in the upper part of the floor, such as the ceiling, as described above. Conventional security cameras installed in the upper part of the floor, such as the ceiling, may be used as multiple fixed cameras 3. Multiple fixed cameras 3 may also be attached to pipes for installing security cameras. In this case as well, images of the floor can be captured from the upper part of the floor. Alternatively, multiple fixed cameras 3 may be installed in the upper part of the side walls of the floor. The fixed cameras 3 acquire images at a predetermined screen resolution (e.g., 1920 x 1080 pixels) and at predetermined time intervals, and transmit the acquired images to the server 2.

[0047] (Operation of the movement control system 10) Next, the operation of the movement control system 10 according to one embodiment of the present invention will be explained using the flowcharts shown in Figures 6 to 9.

[0048] Figure 6 is a flowchart illustrating the operation of map creation by a mobile control system 10 according to one embodiment of the present invention. First, the server 2 receives a map creation instruction via the server communication unit 21 (step S10). Upon receiving the map creation instruction, the server control unit 23 receives images captured by all fixed cameras 3 installed on the floor via the server communication unit 21 (step S20). Subsequently, the server control unit 23 creates a map based on the received images (step S30). Thus, with the mobile control system 10 according to this embodiment, a map of the floor can be created without operating the robot 1, and therefore, a map can be created at a higher speed compared to SLAM (Simultaneously Localization and Mapping) in conventional robots. Furthermore, since the installation positions of the multiple fixed cameras 3 are known in advance, a highly accurate map can be created.

[0049] Figure 7 is a flowchart showing the operations related to the current position estimation and movement control of the robot 1 by the movement control system 10 according to one embodiment of the present invention.

[0050] First, the server 2 receives multiple images captured by multiple fixed cameras 3 via the server communication unit 21 (step S100).

[0051] Next, the server control unit 23 estimates the current position of the robot 1 based on multiple images captured by the multiple fixed cameras 3. First, the server control unit 23 determines whether or not the robot 1 is visible in one or more images (step S110). If the robot 1 is visible in one or more images (step S110: yes), the server control unit 23 estimates the current position of the robot 1 based on the one or more images in which the robot 1 is visible (step S120). On the other hand, if the robot 1 is not visible in one or more images (step S110: no), the server control unit 23 estimates the current position of the robot 1 based on past movement instructions (step S130).

[0052] After estimating the current position of robot 1, the server control unit 23 performs movement control of robot 1 (step S140). Movement control is mainly performed by movement instructions transmitted from the server control unit 23 to robot 1. Movement instructions are created based on the map created by server 2 and the estimated current position of robot 1.

[0053] Figures 8 and 9 are flowcharts illustrating the operation of robot 1's movement control. Figure 8 is a flowchart showing the operation of server 2, and Figure 9 is a flowchart showing the operation of robot 1. First, the operation of server 2 will be explained using Figure 8.

[0054] First, the server control unit 23 determines whether or not robot 1 is in a standby state (step S200). If it determines that robot 1 is in a standby state, the server control unit 23 sends a movement instruction to robot 1 via the server communication unit 21 (step S210). On the other hand, if it determines that robot 1 is not in a standby state, the server control unit 23 does not send a movement instruction to robot 1 via the server communication unit 21.

[0055] Next, the operation of robot 1 will be explained with reference to Figure 9. First, the control unit 17 of robot 1 determines whether or not its own state is in standby mode (step S300). If it is not in standby mode (step S300: no), the control unit 17 moves robot 1 using the movement means 12 based on a movement instruction from server 2 (step S310).

[0056] Next, the control unit 17 determines whether a predetermined event has occurred while the robot 1 is moving based on the movement instruction (step S320). If it is determined that no predetermined event has occurred (step S320: no), the process proceeds to step S330.

[0057] When movement based on the movement instruction is completed (step S330), the control unit 17 puts the robot 1 into a standby state (step S340). Then the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position (step S350).

[0058] If it is determined in step S300 that the robot 1 is in a standby state, the process proceeds to step S350, where the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position.

[0059] Furthermore, if it is determined in step S320 that a predetermined event has occurred, the process proceeds to step S340, where the robot 1 is put into a standby state, and then the control unit 17 autonomously moves the robot 1 within a predetermined range based on the robot 1's current position (step S350).

[0060] As described above, according to the movement control system 10 of one embodiment of the present invention, the current position of the robot 1 is estimated from multiple images captured by multiple fixed cameras 3, and movement instructions are transmitted to the robot 1 based on the estimated current position to control the movement of the robot 1. Therefore, the current position of the robot 1 and movement control can be performed without installing RFID or markers.

[0061] Furthermore, according to the movement control system 10 of one embodiment of the present invention, since the movement control of the robot 1 is performed after understanding the situation of most of the floor area using multiple fixed cameras 3, the robot 1 can be moved and positioned to a more suitable location, and it is expected that the effectiveness of actions performed by the robot 1, such as monitoring, tracking, calling out, and providing advertising information to customers to prevent shoplifting, can be enhanced. In addition, even when controlling the movement of multiple robots 1 on the floor, the positioning and movement control of the multiple robots can be performed while considering the situation of the entire floor.

[0062] Furthermore, in the movement control system 10 according to one embodiment of the present invention, when the robot 1 is in a standby state, i.e., when it has not received a movement instruction, the robot 1 autonomously moves within a predetermined range based on its current position. As a result, the robot 1 appears to be constantly moving autonomously, which can enhance the effectiveness of shoplifting prevention and advertising. In addition, because the robot 1 is constantly moving, images and other data around the robot 1 can be acquired over a wider area.

[0063] Furthermore, if the robot 1 is not visible in one or more of the images captured by the multiple fixed cameras 3, the server control unit 23 of the server 2 may send a movement instruction to move the robot 1 into the shooting range that can be simultaneously captured by the multiple fixed cameras 3. In this way, the robot 1 will be captured by the multiple fixed cameras 3 after moving based on the movement instruction, and the current position of the robot 1 can be estimated with high accuracy.

[0064] Furthermore, if robot 1 does not move to the first position even after a predetermined number of movement instructions have been sent to robot 1, server 2 may send a movement instruction to move to a second position different from the first position. For example, if there is an obstacle in the path of robot 1, even if robot 1 tries to move based on the movement instruction, the same predetermined event may occur each time, potentially preventing it from moving to the first position. To prevent robot 1 from becoming unable to move due to this event, the number of times the same movement instruction can be resent is limited to a predetermined number of times. If movement based on the movement instruction is not performed even after the predetermined number of transmissions, the destination is changed to a second position different from the first position. This suppresses situations in which robot 1 is unable to move and allows for efficient movement control of robot 1.

[0065] Furthermore, if the robot 1 does not move to the first position even after the server 2 has sent a predetermined number of movement instructions to move to the first position, the server control unit 23 may estimate that there is an obstacle between the robot 1 and the first position. Based on this estimation, the server control unit 23 may also update the floor map to include information that an obstacle exists. By updating the map, the server control unit 23 can create movement instructions for the robot 1 taking into account the presence of the obstacle, thereby suppressing situations where the robot 1 is unable to move and enabling efficient movement control of the robot 1.

[0066] In this embodiment, the robot 1 is configured to move autonomously within a predetermined range based on its current position while in standby mode. However, the robot 1 may be configured to stop while in standby mode. If the robot 1 is not given the function of autonomous movement, its functions can be simplified.

[0067] In this embodiment, an example was described in which the server 2 estimates the position of robot 1 and creates movement instructions for movement control. However, robot 1 may, for example, have some or all of the functions of server 2. For example, server 2 may estimate the current position of robot 1, server 2 may transmit the current position information to robot 1, and robot 1 may perform movement control based on the current position information. If robot 1 has all the functions of server 2, a movement control system can be configured using robot 1 and multiple fixed cameras 3 without providing server 2.

[0068] The present invention has been described above based on various drawings and embodiments. However, it should be noted that those skilled in the art will find it easy to make various modifications and alterations based on this disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present invention. For example, the functions included in each means, each step, etc., can be rearranged in a logically consistent manner, and multiple means or steps, etc., can be combined into one or divided. [Explanation of Symbols]

[0069] 1 Robot 2 servers 3 Fixed camera 4 bookshelves 10. Mobility control system 11 Communications Department 12 Means of Transportation 13. Photography Department 14 Input section 15 Output section 16 Memory section 17 Control Unit Buses 18 and 24 21 Server Communication Section 22 Server Storage Unit 23 Server Control Unit

Claims

1. The system comprises a server, a robot that patrols the floor, multiple fixed cameras installed on the upper part of the floor, such as the ceiling, that repeatedly photograph the floor, and a robot camera installed on the robot itself that photographs its surroundings. The server, Based on multiple images taken from different directions almost simultaneously by the multiple fixed cameras and the robot camera, the current position of the robot is estimated. A movement control system characterized by determining whether the robot is in a standby state waiting for a movement instruction from the server each time the current position of the robot is estimated, and if the robot is in the standby state, sending a movement instruction to the robot to move to a first position different from its current position based on the current position estimated from the plurality of images, and not sending the movement instruction if the robot is not in the standby state.

2. The robot movement control system according to claim 1, characterized in that the robot camera employs a 360-degree camera to capture images of the customer and the surrounding area.

3. The robot movement control system according to claim 1 or 2, characterized in that it outputs advertising information for displayed products or store information depending on the position of the robot.

4. The robot movement control system according to any one of claims 1 to 3, characterized in that, if the robot camera can identify a customer using the facial image information of the customer, the robot can output advertising information to the customer based on the customer's purchase information that has been stored in advance, using an output means provided on the robot.

Images