Robot control system and robot

Abstract

To provide a robot control system and a robot that can respond more appropriately to individuals of interest, thereby preventing problematic behavior by such individuals. [Solution] The robot control system includes a control unit that controls a self-propelled robot. The control unit determines whether a person of interest is included in a first image captured by a camera on the robot, and if it determines that a person of interest is included in the first image, it controls the robot's drive unit to track the person of interest and displays at least a portion of the first image or the real-time image captured by the camera on the robot's display unit.

Description

Technical Field

[0001] This disclosure relates to a robot control system and a robot.

Background Art

[0002] In Patent Document 1, information about a target person is acquired using a fixed camera, and the acquired information is compared with registered information that is information for identifying the target person to detect the target person. Based on the detection result, an abnormality of the target person is detected. When an abnormality of the target person is detected, a mobile robot is moved to a non-acquisition range of the fixed camera, and information about the target person is acquired using the mobile robot. A target person detection system is disclosed. This target person detection system detects the location and behavior of the target person not only when the target person is within the detection range of the fixed camera but also when the target person is within the non-detection range of the fixed camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In view of the above-described conventional situation, this disclosure has been devised to provide a robot control system and a robot that can more appropriately respond to a person of interest with a self-propelled robot and can prevent the occurrence of problematic behavior of the person of interest.

Means for Solving the Problems

[0005] This disclosure provides a robot control system comprising a control unit for controlling a self-propelled robot, wherein the control unit determines, based on a first image captured by a camera on the robot, whether or not a person of interest is included in the first image, and if it determines that a person of interest is included in the first image, it controls the drive unit of the robot to track the person of interest, and displays at least a portion of the first image or real-time image captured by the camera on a display unit of the robot.

[0006] This disclosure provides a robot control system comprising a control unit for controlling a self-propelled robot, wherein the robot control system comprises a communication unit capable of sending and receiving information via a network, the communication unit receives a third image captured by a surveillance camera installed in an area where the robot can self-propel, the control unit determines whether or not a person of interest is included in the first image based on a first image captured by a camera on the robot, and if it determines that a person of interest is included in the first image, it controls the drive unit of the robot to track the person of interest, and displays at least a portion of the third image on a display unit of the robot.

[0007] This disclosure provides a robot controlled by the robot control system described above, the robot having the drive unit and the display unit. [Effects of the Invention]

[0008] According to this disclosure, self-driving robots can be used to respond more appropriately to individuals of interest, thereby preventing problematic behavior by such individuals from occurring. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram showing an example configuration of a management system including a robot control system according to Embodiment 1. [Figure 2]Block diagram showing an example of the internal configuration of the system according to Embodiment 1. [Figure 3] This diagram illustrates the first warning operation of the robot in the system according to Embodiment 1. [Figure 4] This diagram illustrates the second vigilance operation of the robot in the system according to Embodiment 1. [Figure 5] This diagram illustrates the third warning action of the robot in the system according to Embodiment 1. [Figure 6] This diagram illustrates the fourth warning action of the robot in the system according to Embodiment 1. [Figure 7] A diagram illustrating the conversion table in the system according to Embodiment 1. [Figure 8] A diagram illustrating the conversion table in the system according to Embodiment 1. [Figure 9] This diagram illustrates the first operation in the system according to Embodiment 1, which involves a robot performing a predetermined action, and is a flowchart showing the flow of control processing in the operation of the robot control system. [Figure 10] This diagram illustrates the second operation in the system according to Embodiment 1, which involves a robot performing a predetermined action, and is a flowchart showing the flow of control processing in the first operation of the external system. [Figure 11] This diagram illustrates the second operation in the system according to Embodiment 1, which involves a robot performing a predetermined action, and is a flowchart showing the flow of control processing in the second operation of the external system. [Figure 12] This diagram illustrates the second operation in the system according to Embodiment 1, which involves the robot executing a predetermined action, and is a flowchart showing the flow of control processing in the operation of the robot control system. [Figure 13] A flowchart showing the flow of control processing in the robot's predetermined actions in the robot control system according to Embodiment 1. [Modes for carrying out the invention]

[0010] (Background leading to this disclosure) Traditionally, stores (or facilities) have sometimes introduced facial recognition systems for security purposes. This facial recognition system, for example, detects individuals within the store from images captured by surveillance cameras, extracts their characteristic features, and compares these features with registered information in a predetermined database to determine if the individual matches a person designated as a target for detection in that database. This predetermined database includes individuals who have previously caused problems such as shoplifting, or VIPs. When the facial recognition system determines that a person in the store is a target for detection, it transmits the identifier of the surveillance camera that captured the image to a store terminal. The store (or facility) then moves staff to the relevant area based on the surveillance camera identifier received from the facial recognition system to deal with the person identified as a target. However, this approach by store staff is unsuitable in various aspects, including human resources, response speed, and tracking capabilities. Furthermore, even using the technology described in Patent Document 1, there is room for improvement regarding the robot's actions towards the target individual (person of interest).

[0011] Therefore, the following embodiments describe a robot control system and a robot that can respond more appropriately to persons of interest and prevent problematic behavior by such persons.

[0012] Hereinafter, while appropriately referring to the drawings, each embodiment specifically disclosing the robot control system and the robot according to the present disclosure will be described in detail. However, detailed descriptions that are more than necessary may be omitted. For example, detailed descriptions of well-known matters and duplicate descriptions of substantially the same configurations may be omitted. This is to avoid making the following descriptions unnecessarily redundant and to facilitate the understanding of those skilled in the art. Note that the attached drawings and the following descriptions are provided for those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims thereby.

[0013] (Embodiment 1) <System Configuration> Hereinafter, while referring to FIGS. 1 to 2, a configuration example of the management system 1 according to Embodiment 1 will be described. FIG. 1 is a block diagram showing a configuration example of the management system 1 including the robot control system 100 according to Embodiment 1. FIG. 2 is a block diagram showing an internal configuration example of the system according to Embodiment 1. Note that the system configuration shown in FIG. 1 is an example, and one device or system may be divided into multiple components, or multiple devices or systems may be combined into one. Also, multiple instances of the device or system shown in FIG. 1 may be provided. Further, the configuration shown in FIG. 2 is an example, and some parts may be omitted or other parts may be added as necessary.

[0014] The management system 1 is a system that causes a robot RB arranged in a facility (for example, a shopping mall, a store, an airport, a hotel, a hospital, or a station) to perform a predetermined action on a person of interest. A person of interest is a person who matches a person of detection target registered in a detection target DB 300 (to be described later), or a person who has performed an action that should be noted (such as a suspicious action such as shoplifting). However, it is assumed here that the person of interest is a person imaged by a surveillance camera CA or a detection unit 15 (to be described later) of the robot RB within the facility.

[0015] The management system 1 includes a robot control system 100, an external system 200, a detection target DB (detection target database) 300, a surveillance camera CA, and a network NW. In the example shown in FIG. 1, an example is shown where the robot control system 100, the external system 200, the detection target DB 300, and the surveillance camera CA are each one unit, but they may each be plural.

[0016] Note that the management system 1 may have an access point or the like through which the robot control system 100 communicates. In this case, the communication method when the robot control system 100 communicates with the access point or the like may be any wireless communication. The wireless communication mentioned here may be communication via a wireless Local Area Network (LAN) such as Wi-Fi (registered trademark), Bluetooth Low Energy (hereinafter referred to as "BLE"), etc., or may be communication by an Internet of Things (IoT) network communication or protocol such as Matter, Z-Wave, or ZigBee.

[0017] [Robot control system] The robot control system 100 is a system that controls the robot RB based on information regarding a person of interest or the like identified based on the video of the surveillance camera CA or the video of the detection unit 15 of the robot RB.

[0018] The robot control system 100 includes a robot RB and a robot control server RS. Note that the control of the robot RB may be in a form where the robot control server RS generates a control command and the robot RB operates, a form where the robot RB and the robot control server RS cooperate to control, or a form where the robot RB is controlled alone. Note that if the robot control system 100 is in a form where the robot RB is controlled alone, it does not necessarily have to have the robot control server RS.

[0019] The robot RB is configured to be able to move autonomously within the facility where it is located. The robot RB includes a communication unit 10, a processor 11, a memory 12, an input unit 13, a monitor 14, a detection unit 15, and a drive unit 16.

[0020] The communication unit 10 is connected to the robot control server RS ​​and the external system 200 via a network NW (or access point) to enable data communication. Alternatively, the communication unit 10 may be directly connected to the robot control server RS ​​and the external system 200 to enable data communication. The communication unit 10 outputs various data transmitted from the robot control server RS ​​and the external system 200 to the processor 11. The communication unit 10 also transmits various data output from the processor 11 to the robot control server RS ​​and the external system 200.

[0021] The processor 11 is an example of a control unit of the robot control system 100, and is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU") or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and works in cooperation with the memory 12 to perform various processes and controls. Specifically, the processor 11 refers to the programs and data held in the memory 12 and executes those programs to realize the functions of the robot RB.

[0022] Memory 12 includes, for example, Random Access Memory (hereinafter referred to as "RAM") as work memory used when executing each process of the processor 11, and storage for storing programs and data that define the operation of the processor 11. Data or information generated or acquired by the processor 11 is temporarily stored in RAM. Programs that define the operation of the processor 11 are written to the storage. Memory 12 may record, for example, information on the patrol route that the robot RB travels, map information MP (or area information) of the area in which the robot RB can move autonomously, the robot RB's default actions, conversion table CT (see below), and information detected by the detection unit 15 (such as video).

[0023] The input unit 13 is an example of an input device that can accept user operations from facility staff, etc., and outputs the content of the input operation based on the user operation to the processor 11. The input unit 13 may be implemented as a touch panel integrated with the monitor 14, or as a physical button located near the housing of the robot RB or the monitor 14 and pressed by the user. The input unit 13 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice. Note that the input unit 13 is not a mandatory configuration and may be omitted.

[0024] The monitor 14 (input device) is configured using a display such as a Liquid Crystal Display (hereinafter referred to as "LCD"), an organic electroluminescence (hereinafter referred to as "EL"), or a touch panel. The monitor 14 may be movable. The monitor 14 displays various screens output from the processor 11.

[0025] The detection unit 15 is implemented by one or more cameras. The detection unit 15 is controlled by the processor 11 and captures images of the environment around the robot RB (faces, people, obstacles, or passages, etc.). The detection unit 15 also detects people present in the vicinity of the robot RB by performing one of the following detection methods: face recognition, object recognition, skeleton detection, clothing detection, etc. The detection unit 15 may be equipped with sensors, lasers, radar, etc. In this case, for example, multiple different devices such as cameras and radar may be combined. The robot RB may also be equipped with a Global Positioning System (GPS) or the like as the detection unit 15, which is capable of determining the robot RB's position. As detection results, the detection unit 15 outputs the captured video, the robot RB's own position information, and information about the environment around the robot RB to the processor 11.

[0026] The drive unit 16 is controlled by the processor 11 and makes the robot RB move autonomously based on control commands.

[0027] The robot control server RS ​​can be implemented using devices such as a smartphone, tablet, computer, or server. The robot control server RS ​​is managed within the facility where the robot RB is located and performs control of the robot RB. In other words, the robot control server RS ​​can be referred to as a "facility server." The robot control server RS ​​includes a communication unit 10, a processor 21, a memory 22, an input unit 23, and a monitor 24.

[0028] In the examples shown in Figures 1 and 2, the robot control server RS ​​is assumed to be managed within the facility, but it may be a server located outside the facility. Furthermore, the robot control server RS ​​may be shared as part of the robot control system 100 for multiple robots RB. Note that the multiple robots RB are not necessarily limited to those located within the same facility. Each robot RB may be controlled based on map information MP (or area information) or conversion table CT corresponding to its respective facility. Note that the map information MP (or area information) or conversion table CT may be different for each of the multiple robots RB within the same facility.

[0029] The communication unit 20 is connected to the robot RB and the external system 200 via a network NW (or access point) to enable data communication. The communication unit 20 may be connected to the robot RB and the external system 200 via an access point using Wi-Fi®, or it may be connected directly to the robot RB and the external system 200 via BLE without going through an access point to enable data communication. The communication unit 20 outputs various data transmitted from the robot RB or the external system 200 to the processor 21. The communication unit 20 also transmits various data output from the processor 21 to the robot RB or the external system 200.

[0030] The processor 21 is configured, for example, using a CPU or FPGA, and works in cooperation with the memory 22 to perform various processes and controls. Specifically, the processor 21 refers to the programs and data held in the memory 22 and executes those programs to realize the functions of the robot control server RS.

[0031] Memory 22 includes, for example, RAM as work memory used when executing each process of processor 21, and storage for storing programs and data that define the operation of processor 21. Data or information generated or acquired by processor 21 is temporarily stored in RAM. Programs that define the operation of processor 21 are written to the storage. Memory 22 may record, for example, information on the patrol route that robot RB travels, map information MP (or area information) of the area where robot RB can move autonomously, the robot RB's default actions, conversion table CT (see below), and information detected by the detection unit 15 (such as video).

[0032] The input unit 23 is capable of receiving operations and outputs the content of the input operations to the processor 21. The input unit 23 is implemented as a touch panel integrated with the monitor 24. The input unit 23 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice.

[0033] The monitor 24 is configured using a display such as an LCD, organic EL, or touch panel. The monitor 24 displays various screens (not shown) output from the processor 21.

[0034] [External System] External system 200 is a system that identifies individuals of interest based on footage from surveillance cameras (CA).

[0035] The external system 200 includes a camera management system 400 and a person detection system 500. In the example shown in Figure 1, there is one camera management system 400 and one person detection system 500, but there may be multiple units. Also, in the example shown in Figure 1, the external system 200 includes a camera management system 400 and a person detection system 500, but there may be only the camera management system 400 or the person detection system 500. Furthermore, some of the functions of the camera management system 400 may be implemented by the person detection system 500, and conversely, some of the functions of the person detection system 500 may be implemented by the camera management system 400. For example, some or all of the functions of the person detection system 500 may be implemented by the robot control server RS.

[0036] The camera management system 400 acquires video from the surveillance camera CA and extracts images containing people from the acquired video. The camera management system 400 then transmits the extracted images to the person detection system 500. The person detection system 500 determines whether the person in the image is the same person as the person to be detected, based on the image transmitted from the camera management system 400 and the detection target DB 300. If the person detection system 500 determines that the person in the image is the same person as the person to be detected, it identifies the person in the image as a person of interest. The person detection system 500 then identifies the characteristic information of the person of interest and the detection type in the detection target DB 300 as attribute information (see below). The person detection system 500 then transmits the identified attribute information and the camera identifier of the surveillance camera CA that captured the person of interest to the robot control system 100.

[0037] Furthermore, the camera management system 400 acquires video from the surveillance camera CA and extracts video containing people from the acquired video. The camera management system 400 then transmits the extracted video to the person detection system 500. The person detection system 500 determines whether a person in the video transmitted from the camera management system 400 has performed a noteworthy action by estimating their skeletal structure, etc. If the person detection system 500 determines that a person in the video has performed a noteworthy action, it identifies that person as a person of interest. The person detection system 500 then identifies the characteristic information of the person of interest as attribute information (see below). The person detection system 500 then transmits the identified attribute information and the camera identifier of the surveillance camera CA that captured the person of interest to the robot control system 100.

[0038] The external system 200 (camera management system 400 and person detection system 500) is implemented by devices 210 such as smartphones, tablet terminals, computers, and servers. The external system 200 is managed outside the facility where the robot RB is located and performs preliminary processing for controlling the robot RB. The external system 200 includes a communication unit 30, a processor 31, a memory 32, an input unit 33, and a monitor 34.

[0039] The communication unit 30 is connected to the robot control system 100 and each system constituting the external system 200 (camera management system 400 or person detection system 500) via a network NW (or access point) to enable data communication. The communication unit 20 may also be connected to the robot control system 100 and each system constituting the external system 200 (see above) via an access point using Wi-Fi®, or it may be connected to the robot control system 100 and each system constituting the external system 200 (see above) directly to enable data communication without going through an access point using BLE.

[0040] The processor 31 is configured, for example, using a CPU or FPGA, and works in cooperation with the memory 32 to perform various processes and controls. Specifically, the processor 31 refers to the programs and data held in the memory 32 and executes those programs to realize various functions.

[0041] Memory 32 includes, for example, RAM as work memory used when executing each process of the processor 21, and storage for storing programs and data that define the operation of the processor 21. Data or information generated or acquired by the processor 31 is temporarily stored in RAM. The storage contains programs that define the operation of the processor 31. Memory 32 may, for example, record the detection target DB 300 and the camera identifiers assigned to each surveillance camera CA.

[0042] The input unit 33 is capable of receiving operations and outputs the content of the input operations to the processor 21. The input unit 33 is implemented as a touch panel integrated with the monitor 34. The input unit 33 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice.

[0043] The monitor 34 is configured using a display such as an LCD, organic EL, or touch panel. The monitor 34 displays various screens (not shown) output from the processor 31.

[0044] [Databases to be detected] The detection target DB300 is a database in which individuals who should be subject to detection are registered, such as individuals who have committed shoplifting or other problematic acts in the past, or individuals who are suspected of having committed shoplifting or other problematic acts in the past. These individuals who should be subject to detection may be determined as appropriate by the administrator or users of the detection target DB.

[0045] In the detection target DB300, each person to be detected is associated with their facial image and facial features extracted from that image for facial recognition, and this is registered as detection target information. This detection target information may also include a person identifier, historical information such as past problematic behavior, etc.

[0046] The detection target DB300 is preferably stored in the memory 32 of the external system 200 (particularly the person detection system 500). However, the detection target DB300 may also be stored in the robot control system 100 or a server not shown, or in all of the robot control system 100, the external system 200, and the server not shown, or in any combination thereof.

[0047] [Surveillance camera] The surveillance camera CA monitors the facility by capturing images of the interior. The surveillance camera CA is preferably controlled by the processor 31 of the external system 200 (particularly the camera management system 400).

[0048] [Attribute information] Attribute information is information that indicates the attributes of a person of interest. Attribute information includes characteristic information, such as the person's clothing, bone structure, and gait, which are related to their appearance. Furthermore, attribute information may also include the detection type in the target DB300, the person's facial features, their status, and their category.

[0049] The detection type is, for example, a person identifier or historical information included in the information to be detected. The facial features of individuals of interest are the facial features used for facial recognition. The status of a person of interest is information that indicates the person's state or behavior, such as whether they are running, lying down, or moving towards a designated location. The categories for persons of interest include those requiring surveillance, under surveillance, unusual, lost, and VIP, among others, which are categories assigned to individuals of interest.

[0050] <Robot's warning actions> The following describes examples of the robot RB's warning actions with reference to Figures 3 to 6. Figure 3 is a diagram illustrating the first warning action of the robot RB in the system according to Embodiment 1. Figure 4 is a diagram illustrating the second warning action of the robot RB in the system according to Embodiment 1. Figure 5 is a diagram illustrating the third warning action of the robot RB in the system according to Embodiment 1. Figure 6 is a diagram illustrating the fourth warning action of the robot RB in the system according to Embodiment 1.

[0051] In its normal operation, the robot RB patrols a set route (also referred to as "normal patrol"). During this time, the robot RB may display advertisements or other information on the monitor 14. If a person captured by the detection unit 15 during patrol is identified as a person of interest by the robot control system 100, or if the robot control system 100 obtains information about a person of interest from an external system 200, the robot RB switches from normal operation to alert operation.

[0052] Figures 3-6 show the interior of the facility. Below, we will explain using the example of a visitor C1 being in a specific area of ​​the facility where product shelves S1-S2 and surveillance cameras CA1-CA2 are arranged as shown in Figures 3-6.

[0053] [First warning action] The first warning action will be explained using Figure 3. The robot control system 100 determines, based on the video captured by the detection unit 15, whether visitor C1 has performed a noteworthy action while robot RB is patrolling. Here, we assume that visitor C1 has performed a noteworthy action. In other words, the robot control system 100 identifies visitor C1 as a person of interest based on the video captured by the detection unit 15. Next, the robot control system 100 has robot RB approach and track visitor C1. Subsequently, the robot control system 100 has robot RB, which is tracking the person of interest, perform a predetermined action (see below). This makes visitor C1 aware that they are a target of vigilance and surveillance.

[0054] [Second warning action] The second operation will be explained using Figure 4. First, the external system 200 determines whether visitor C1 is a person of interest based on the video captured by the surveillance camera CA2. This determination determines whether visitor C1 is a person registered in the detection target DB 300, or whether visitor C1 has performed any noteworthy actions. Here, we assume that visitor C1 is a person of interest, or that visitor C1 has performed any noteworthy actions. In other words, the robot control system 100 identifies visitor C1 as a person of interest based on the video captured by the surveillance camera CA2 (see Figure 4(a)). Next, the external system 200 identifies the attribute information of visitor C1. Subsequently, the external system 200 transmits the attribute information of visitor C1 and the camera identifier of the surveillance camera CA1 that captured the image of visitor C1 to the robot control system 100.

[0055] Next, the robot control system 100 identifies the location information to move the robot RB based on the attribute information of visitor C1 obtained from the external system 200 and the camera identifier of the surveillance camera CA1. Subsequently, the robot control system 100 moves the robot RB to the location (destination) indicated by the identified location information.

[0056] Next, the robot RB moves towards its destination while using the detection unit 15 to image the surrounding environment and searching for visitor C1 by referring to attribute information acquired from the external system 200. Here, we assume that visitor C1 has been found.

[0057] Next, the robot control system 100 has robot RB approach and track visitor C1. Subsequently, the robot control system 100 has robot RB, which is tracking the person of interest, perform a predetermined action (see below). This makes visitor C1 aware that they are a target of vigilance and surveillance.

[0058] [Third warning action] The third warning action will be explained using Figure 5. Assume that visitor C1 is initially in a position captured by surveillance camera CA1, and then moves to a position captured by surveillance camera CA2. Furthermore, assume that visitor C1 has been identified as a person of interest by external system 200. In the third alert operation, robot control system 100 controls robot RB based on the most recent information obtained from external system 200, namely the camera identifier of surveillance camera CA2 (and visitor C1's attribute information). This increases the likelihood that robot RB will detect visitor C1. Note that in the third alert operation, the operation of external system 200, the operation of robot control system 100, and the alert operation of robot RB are the same as in the second alert operation.

[0059] [Fourth warning action] The fourth warning action will be explained using Figure 6. Visitor C1 is identified as a person of interest by the external system 200 based on video footage captured by the surveillance camera CA3 installed near the entrance of the facility. Furthermore, visitor C1 is registered in the detection target database 300 as having a history of suspected shoplifting at shelf S1. In the fourth alert action, the external system 200 transmits attribute information, including this history information (detection type), to the robot control system 100. The robot control system 100 then moves robot RB to the vicinity of shelf S1 based on this attribute information. Robot RB then patrols the vicinity of shelf S1 (security patrol) searching for visitor C1. This allows for focused monitoring of shelf S1, where shoplifting is highly likely to occur. Note that subsequent actions in the fourth alert action are the same as those in the second alert action.

[0060] <Conversion Table> The conversion table CT will be explained below with reference to Figures 7 and 8. Figure 7 is a diagram illustrating the conversion table TP in the system according to Embodiment 1. Figure 8 is a diagram illustrating the conversion table TP in the system according to Embodiment 1.

[0061] The conversion table CT is essentially a table used by the robot control system 100 to convert camera identifiers (and attribute information) obtained from the external system 200 into positional information for moving the robot RB.

[0062] As shown in Figure 7, the conversion table CT registers position information for moving the robot RB associated with the camera identifier. This position information may be the location of the surveillance camera CA corresponding to the camera identifier, or a predetermined location within the imaging range of the surveillance camera CA. The conversion table CT may also register attribute information, default actions to be performed by the robot RB, and priority for moving the robot RB. As shown in Figure 7, the conversion table CT may be created by referring to map information MP of the area in which the robot RB can move autonomously. In this case, as shown in Figure 8, the conversion table CT may register point information (points indicated by A to H) associated with the map information MP, and this point information may be linked to various other information. Furthermore, the position information for moving the robot RB is not limited to coordinate values; a patrol range or movement path for the robot RB to travel around may also be set.

[0063] <The first action the robot takes before executing a predetermined action> The following describes a first example of the operation of the robot RB until it performs a predetermined action, with reference to Figure 9. Figure 9 is a diagram illustrating the first operation of the robot RB until it performs a predetermined action in the system according to Embodiment 1, and is a flowchart showing the flow of control processing in the operation of the robot control system 100.

[0064] The first action leading up to the robot RB executing a predetermined action is performed by the robot control system 100 alone.

[0065] In this example, the processing entity is described as the robot control system 100, but the following control processing is performed by at least one of the robot RB and the robot control server RS, as described above (the same applies to the operation of the robot control system 100 in the second operation until the robot performs the predetermined action).

[0066] For example, when the robot control system 100 detects and images a person inside the facility using the detection unit 15 (camera), it starts the following processing flow.

[0067] The robot control system 100 (for example, processors 11, 21, and so on) acquires the captured video from the detection unit 15 (step Sp101).

[0068] The robot control system 100 extracts video footage containing the target person from the acquired video footage (step Sp102).

[0069] The robot control system 100 identifies the movements of the target person using skeletal detection on the extracted video (step Sp103).

[0070] The robot control system 100 determines whether the subject person's actions are noteworthy based on the identification results using skeletal detection (step Sp104). Noteworthy actions include, for example, placing goods in a bag instead of a basket in the facility, carrying a large bag on the shoulder, looking around nervously, quickly picking up goods, stiff or trembling limbs, putting hands in and out of pockets, or holding goods in one's hands while also holding a basket. If the subject person's actions are noteworthy (YES in step Sp104), the robot control system 100 proceeds to step Sp105. On the other hand, if the subject person's actions are not noteworthy (NO in step Sp104), this processing flow is terminated.

[0071] The robot control system 100 identifies the target person as a person of interest (step Sp105).

[0072] The robot control system 100 instructs the robot RB to perform a predetermined action on a person identified as a person of interest (step Sp106).

[0073] <Second action before the robot performs a predetermined action> The following describes a second example of the operation of the robot RB until it executes a predetermined action, with reference to Figures 10 to 12. Figure 10 is a diagram illustrating the second operation of the robot RB until it executes a predetermined action in the system according to Embodiment 1, and is a flowchart showing the flow of control processing in the first operation of the external system 200. Figure 11 is a diagram illustrating the second operation of the robot RB until it executes a predetermined action in the system according to Embodiment 1, and is a flowchart showing the flow of control processing in the second operation of the external system 200. Figure 12 is a diagram illustrating the second operation of the robot RB until it executes a predetermined action in the system according to Embodiment 1, and is a flowchart showing the flow of control processing in the operation of the robot control system 100.

[0074] The second action, before the robot RB performs its predetermined action, involves the coordinated operation of the robot control system 100 and the external system 200.

[0075] [First operation of the external system] The first operation of the external system 200 is to determine whether or not a person to be detected, who is registered in the detection target DB 300, is present within the facility.

[0076] In this example, the processing entity is described as the external system 200, but the functions may be divided and executed by the camera management system 400 and the person detection system 500. Furthermore, the following first and second operations may be executed by both the camera management system 400 and the person detection system 500, or by only one of them. In addition, the first and second operations may be executed simultaneously and in parallel, or sequentially.

[0077] The external system 200 (for example, the processor 31, and so on) acquires the captured video from each surveillance camera CA (step Sp201).

[0078] The external system 200 extracts images containing people from the acquired video (step Sp202).

[0079] The external system 200 compares the facial features of the person in the extracted image with the facial features of the person to be detected registered in the detection target database (step Sp203).

[0080] The external system 200 determines whether the person in the extracted image matches the person to be detected, that is, whether they are the same person (step Sp204). If the person in the extracted image is the same person as the person to be detected (YES in step Sp204), the external system 200 proceeds to step Sp205. On the other hand, if the person in the extracted image is not the same person as the person to be detected (NO in step Sp204), the external system 200 returns to step Sp201.

[0081] The external system 200 determines that the person in question is a person of interest and identifies attribute information including characteristic information of this person of interest and the detection type in the detection target DB 300 (step Sp205).

[0082] The external system 200 transmits attribute information of the person of interest and the camera identifier of the surveillance camera CA that captured the image of the person of interest to the robot control system 100 (step Sp206). Then, this processing flow ends.

[0083] [Second operation of external system 200] The second operation of the external system 200 is to determine whether or not a person exhibiting noteworthy behavior (such as suspicious behavior) is present within the facility.

[0084] The external system 200 (for example, processor 31, and so on) acquires the captured video from each surveillance camera CA (step Sp211).

[0085] The external system 200 extracts footage containing people from the acquired video (step Sp212).

[0086] The external system 200 analyzes the actions of the people captured in the extracted video by methods such as skeletal estimation (step Sp203).

[0087] The external system 200 determines whether the person in the extracted video has taken a noteworthy action (step Sp214). If it is determined that the person in the extracted video has taken a noteworthy action (YES in step Sp214), the external system 200 proceeds to step Sp215. If it is determined that the person in the extracted video has not taken a noteworthy action (NO in step Sp214), the external system 200 returns to step Sp211.

[0088] The external system 200 determines that the person in question is a person of interest and identifies attribute information including characteristic information of this person of interest (step Sp215).

[0089] The external system 200 transmits attribute information of the person of interest and the camera identifier of the surveillance camera CA that captured the image of the person of interest to the robot control system 100 (step Sp206). Then, this processing flow ends.

[0090] [Operation of the robot control system] When attribute information and a camera identifier are transmitted from the external system 200 during the first or second operation of the external system 200, the robot control system 100 starts the following processing flow.

[0091] The robot control system 100 (for example, processors 11, 21, and so on) obtains attribute information of the person of interest and the camera identifier of the surveillance camera CA that captured the person of interest from the external system 200 (step Sp221).

[0092] The robot control system 100 acquires position information for moving the robot RB based on the acquired attribute information and camera identifier (step Sp222). Here, the robot control system 100 identifies the position information based on, for example, a transformation table CT.

[0093] The robot control system 100 moves the robot RB to the location (destination) indicated by the identified location information (step Sp223). The path of the robot RB here does not have to be the shortest distance. For example, it may go to the destination via a priority monitoring area.

[0094] The robot control system 100 determines whether the detection unit 15 (camera) has detected a person of interest, that is, whether it has found a person of interest, while the robot RB is moving (step Sp224). Here, it is determined whether the person imaged by the detection unit 15 matches a person of interest by referring to the acquired attribute information. If the robot RB has found a person of interest (YES in step Sp224), the robot control system 100 proceeds to step Sp227. On the other hand, if the robot RB has not found a person of interest (NO in step Sp224), the robot control system 100 proceeds to step Sp225.

[0095] The robot control system 100 determines whether or not robot RB has arrived at its destination (step Sp225). If robot RB has arrived at its destination (YES in step Sp225), the robot control system 100 proceeds to step Sp226. On the other hand, if robot RB has not arrived at its destination (NO in step Sp225), the robot control system 100 returns to step Sp224.

[0096] The robot control system 100 instructs robot RB to patrol the vicinity of the destination (step Sp226). If a person of interest is found during the patrol (vigilant patrol) near the destination, it is advisable to have robot RB perform a predetermined action. Then this processing flow ends.

[0097] If robot RB finds a person of interest (YES in step Sp224), the robot control system 100 causes robot RB to perform a predetermined action (step Sp227). Then, this processing flow ends.

[0098] <Operation of the robot control system related to the robot's predetermined actions> The following describes an example of the operation of the robot control system 100 related to the predetermined actions of the robot RB, with reference to Figure 13. Figure 13 is a flowchart showing the flow of control processing in the operation related to the predetermined actions of the robot RB in the robot control system 100 according to Embodiment 1.

[0099] In this example, the processing entity is described as the robot control system 100, but the following control processing is performed by at least one of the robot RB and the robot control server RS, as described above.

[0100] The robot control system 100 (for example, processors 11, 21, and so on) moves the robot RB closer to the person of interest and starts tracking the person of interest (step Sp301).

[0101] The robot control system 100 controls the robot RB's monitor 14 so that the person of interest can see it (step Sp302). Here, the monitor 14 may be controlled to face the person of interest (by moving the monitor 14), or the drive unit 16 may be controlled to move the robot RB in front of the person of interest.

[0102] The robot control system 100 displays an image containing the person of interest on the monitor 14 (step Sp303). The image displayed here may be the image used to identify the person of interest, or it may be real-time image captured by the detection unit 15 or a surveillance camera CA near the person of interest. In this case, processing is applied to either of the aforementioned images so that the person of interest is clearly visible, for example, by enlarging the face of the person of interest. In addition, the background of the image may be set to a bright color and made to flash to signal to the person of interest that vigilance is being exercised. In addition, the person of interest may be made to visually confirm the change in the display.

[0103] The robot control system 100 determines whether the tracking termination conditions have been met (step Sp304). The tracking termination conditions here include, for example, the elapsed time since the start of tracking, the loss of the person of interest that was being tracked, or the person of interest that was being tracked leaving the facility. If the tracking termination conditions are met (YES in step Sp304), the robot control system 100 proceeds to step Sp305. On the other hand, if the tracking termination conditions are not met (NO in step Sp304), the robot control system 100 repeats step Sp304.

[0104] The robot control system 100 terminates tracking of the person of interest and transitions to normal patrol (step Sp305). Then, this processing flow ends.

[0105] Furthermore, the robot control system 100 may play voice messages to the person of interest while tracking them, or voice messages to indicate that it is on alert, and these voice messages may be played at a high volume (for example, louder than normal).

[0106] As described above, the system according to the embodiment allows the robot RB to respond more appropriately to persons of interest, and can prevent problematic behavior by persons of interest from occurring.

[0107] Furthermore, in the system according to this embodiment, if there are multiple persons of interest within the facility, the system prioritizes responding to persons of interest who match those registered in the detection target DB300, and prioritizes responding to persons of interest identified by noteworthy behavior (such as suspicious behavior like shoplifting).

[0108] (Other variations) This disclosure also applies to programs and storage media that supply programs that realize the functions of the apparatus of the above-described embodiment to the apparatus via a network or various storage media, and which are read and executed by a computer within the apparatus.

[0109] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to these examples. It will be clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can occur within the scope of the claims, and these will naturally fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention.

[0110] (Note) The following technologies are disclosed based on the above description of embodiments.

[0111] <Technology 1> A robot control system (e.g., 100) comprising a control unit (e.g., 11, 21) for controlling a self-propelled robot (e.g., RB), The control unit, Based on the first image captured by the camera (e.g., 15) of the robot, it is determined whether or not a person of interest is included in the first image. If it is determined that the first video contains a person of interest, the robot controls its drive unit (for example, 16) to track the person of interest. At least a portion of the first video or the real-time video captured by the camera is displayed on a display unit (for example, 14) of the robot. Robot control system.

[0112] This configuration automates monitoring by having the system recognize individuals of interest and track them with a robot, thereby reducing human resources and enabling efficient and continuous response. Furthermore, the robot's tracking of individuals of interest can prevent problematic behavior from occurring, and displaying at least a portion of the first video or real-time video on the display unit can alert individuals of interest and deter their actions. In short, this configuration allows for a more appropriate response from the self-driving robot to individuals of interest, thereby preventing problematic behavior from occurring.

[0113] <Technology 2> The control unit (for example, 11, 21) The drive unit or the display unit is controlled so that the person of interest can see the display unit. The robot control system described in Technology 1.

[0114] This configuration allows for direct visual warnings to individuals of interest.

[0115] <Technology 3> The control unit (for example, 11, 21) A portion including the person of interest is extracted from the first video or the real-time video, and the extracted video is displayed on the display unit. A robot control system as described in Technology 1 or Technology 2.

[0116] This configuration can create a strong sense of caution in individuals of interest, increasing the likelihood of deterring their actions.

[0117] <Technology 4> The control unit (for example, 11, 21) The extracted video is displayed with emphasis on the display unit. The robot control system described in Technology 3.

[0118] This configuration can make individuals of interest more vigilant, increasing the likelihood of deterring their actions.

[0119] <Technology 5> It further comprises a communication unit (e.g., 12, 22) capable of sending and receiving information over a network, The aforementioned communication unit (for example, 12, 22) The robot receives a camera identifier, which is the identification information of the surveillance camera that captured the person of interest, from an external system (e.g., 200, 400, 500) that processes surveillance video captured by a surveillance camera (e.g., CA) installed in an area where the robot can move autonomously. The control unit (for example, 11, 21) The drive unit is controlled to move the robot to a position within the area corresponding to the camera identifier. A robot control system described in any one of the technologies 1 through 4.

[0120] This configuration allows the robot to be moved to the appropriate position, enabling rapid and effective monitoring of individuals of interest. This expands the monitoring range and improves the accuracy of the monitoring system.

[0121] <Technology 6> The aforementioned communication unit (for example, 12, 22) The external system receives characteristic information of the person of interest, The control unit (for example, 11, 21) Based on the aforementioned feature information and the first video, it is determined whether or not the first video contains a person of interest. The robot control system described in Technology 5.

[0122] This configuration improves the accuracy of identifying individuals of interest and prevents misidentification.

[0123] <Technology 7> The aforementioned characteristic information is Information identified from the aforementioned surveillance footage, including information about the appearance of the person of interest, including their clothing. The robot control system described in Technology 6.

[0124] This configuration improves the accuracy of identifying individuals of interest and prevents misidentification.

[0125] <Technology 8> The control unit (for example, 11, 21) Based on the second video image captured by the aforementioned camera, it is determined whether the actions of the subject person included in the second video are noteworthy actions. If it is determined that the actions of the person concerned are noteworthy actions, the person concerned is identified as the person of interest. The robot controls the drive unit to track the target person identified as the person of interest. A robot control system described in any one of the technologies 1 through 7.

[0126] This configuration allows for the rapid tracking of a person of interest once noteworthy behavior is detected, thereby preventing problematic behavior from occurring.

[0127] <Technology 9> The control unit (for example, 11, 21) The movements of the subject person included in the second video are identified using skeletal detection. Based on the identification results using the aforementioned skeletal detection, it is determined whether the actions of the target person are the actions of interest. The robot control system described in Technology 8.

[0128] This configuration allows for highly accurate detection of a person's movements and rapid, precise evaluation of those movements.

[0129] <Technology 10> A robot control system (e.g., 100) comprising a control unit (e.g., 11, 21) for controlling a self-propelled robot (e.g., RB), The robot control system includes a communication unit (e.g., 10, 20) capable of sending and receiving information via a network. The aforementioned communications unit is The robot receives a third image captured by a surveillance camera (e.g., CA) installed in an area where it can move autonomously. The control unit, Based on the first image captured by the camera (e.g., 15) of the robot, it is determined whether or not a person of interest is included in the first image. If it is determined that the first video contains a person of interest, the robot controls its drive unit (for example, 16) to track the person of interest. At least a portion of the third image is displayed on a display unit (for example, 14) of the robot. Robot control system.

[0130] This configuration allows the system to recognize individuals of interest and have a robot track them, automating surveillance and enabling efficient and continuous response while reducing human resources. Furthermore, the robot's tracking of individuals of interest can prevent problematic behavior from occurring, and displaying at least a portion of the third video feed on the display unit can alert individuals of interest and deter their actions. In short, this configuration allows for a more appropriate response from the self-driving robot to individuals of interest, thereby preventing problematic behavior from occurring.

[0131] <Technology 11> At least a portion of the third video includes the aforementioned person of interest, A robot control system as described in Technology 10.

[0132] This configuration can create a strong sense of caution in individuals of interest, increasing the likelihood of deterring their actions.

[0133] <Technology 12> A robot (e.g., RB) controlled by a robot control system (e.g., 100) described in any one of Technology 1 to Technology 11, The device comprises the drive unit (for example, 16) and the display unit (for example, 14), robot.

[0134] This configuration allows for the prevention of problematic behavior by individuals of interest, controlled by a robotic control system. [Industrial applicability]

[0135] This disclosure provides a robot control system and a robot that are useful for enabling self-propelled robots to respond more appropriately to persons of interest and for preventing problematic behavior by such persons from occurring. [Explanation of Symbols]

[0136] 1 Management System 15 Detection unit 100 Robot Control Systems 200 External Systems 300 databases to be detected 400 Camera Management System 500 Person Detection System C1 Visitor CA, CA1~CA3 Surveillance Cameras CT Conversion Table S1~S2 Product shelf

Claims

1. A robot control system equipped with a control unit for controlling a self-propelled robot, The control unit, Based on the first image captured by the camera of the robot, it is determined whether or not a person of interest is included in the first image. If it is determined that the first video contains a person of interest, the robot controls its drive unit to track the person of interest. The robot displays at least a portion of the first video or the real-time video captured by the camera on a display unit. Robot control system.

2. The control unit, The drive unit or the display unit is controlled so that the person of interest can see the display unit. The robot control system according to claim 1.

3. The control unit, A portion including the person of interest is extracted from the first video or the real-time video, and the extracted video is displayed on the display unit. The robot control system according to claim 1.

4. The control unit, The extracted video is displayed with emphasis on the display unit. The robot control system according to claim 3.

5. It further includes a communication unit capable of sending and receiving information over a network, The aforementioned communications unit is The robot receives a camera identifier, which is the identification information of the surveillance camera that captured the person of interest, from an external system that processes surveillance images captured by surveillance cameras installed in an area where the robot can move autonomously. The control unit, The drive unit is controlled to move the robot to a position within the area corresponding to the camera identifier. The robot control system according to claim 1.

6. The aforementioned communications unit is The external system receives characteristic information of the person of interest, The control unit, Based on the aforementioned feature information and the first video, it is determined whether or not the first video contains a person of interest. The robot control system according to claim 5.

7. The aforementioned characteristic information is Information identified from the aforementioned surveillance footage, including information about the appearance of the person of interest, including their clothing. The robot control system according to claim 6.

8. The control unit, Based on the second video image captured by the aforementioned camera, it is determined whether the actions of the subject person included in the second video are noteworthy actions. If it is determined that the actions of the person concerned are noteworthy actions, the person concerned is designated as a person of interest. The robot controls the drive unit to track the target person identified as the person of interest. The robot control system according to claim 1.

9. The control unit, The movements of the subject person included in the second video are identified using skeletal detection. Based on the identification results using the aforementioned skeletal detection, it is determined whether the actions of the target person are the actions of interest. The robot control system according to claim 8.

10. A robot control system equipped with a control unit for controlling a self-propelled robot, The robot control system includes a communication unit capable of sending and receiving information via a network, The aforementioned communications unit is The robot receives a third image captured by a surveillance camera installed in an area where it can move autonomously. The control unit, Based on the first image captured by the camera of the robot, it is determined whether or not a person of interest is included in the first image. If it is determined that the first video contains a person of interest, the robot controls its drive unit to track the person of interest. At least a portion of the third image is displayed on a display unit of the robot. Robot control system.

11. At least a portion of the third video includes the aforementioned person of interest, The robot control system according to claim 10.

12. A robot controlled by a robot control system according to any one of claims 1 to 11, Having the aforementioned drive unit and the aforementioned display unit, robot.

Images