Robot control system, management system, camera management system, and robot control method

Abstract

To provide a robot control system, management system, camera management system, and robot control method that enable a self-propelled robot to move more appropriately towards a person of interest. [Solution] The robot control system comprises a control unit that controls a self-propelled robot and a communication unit that sends and receives information via a network. The communication unit receives attribute information of persons of interest and camera identifiers identified by an external system that processes images captured by multiple cameras. The control unit identifies location information, which is information related to the robot's movement, based on the received attribute information and camera identifiers, and moves the robot based on the identified location information. The attribute information is information indicating the attributes of persons of interest included in images captured by at least one of the multiple cameras, and the camera identifiers are identifiers assigned to each of at least one camera.

Description

Technical Field

[0001] The present disclosure relates to a robot control system, a management system, a camera management system, and a robot control method.

Background Art

[0002] In Patent Document 1, information about a target person is acquired using a fixed camera, the acquired information is compared with registered information that is information for identifying the target person to detect the target person, an abnormality of the target person is detected based on the detection result, when an abnormality of the target person is detected, a mobile robot is moved to a non-acquisition range of the fixed camera, and information about the target person is acquired using the mobile robot, and a target person detection system is disclosed. This target person detection system detects the location and behavior of the target person not only when the target person is located within the detection range of the fixed camera but also when the target person is located within the non-detection range of the fixed camera.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In view of the above-described conventional situation, the present disclosure has been devised, and an object thereof is to provide a robot control system, a management system, a camera management system, and a robot control method that can more appropriately move a self-propelled robot with respect to a person of interest.

Means for Solving the Problems

[0005] This disclosure provides a robot control system comprising a control unit for controlling a self-propelled robot and a communication unit for sending and receiving information via a network, wherein the communication unit receives attribute information of a person of interest and a camera identifier identified by an external system that processes images captured by a plurality of cameras, the control unit identifies position information which is information relating to the movement of the robot based on the received attribute information and the camera identifier, and moves the robot based on the identified position information, wherein the attribute information is information indicating the attributes of the person of interest included in the image captured by at least one of the plurality of cameras, and the camera identifier is an identifier assigned to each of the at least one cameras.

[0006] This disclosure provides a management system comprising the aforementioned robot control system, an external system for processing images captured by multiple cameras, and a detection target database in which detection target information, including facial images of persons to be detected and facial features extracted from the facial images, is registered, wherein the external system identifies persons of interest by referring to the detection target database.

[0007] The present invention provides a camera management system that can communicate with the robot control system, comprising: a first control unit that acquires video from a plurality of cameras and extracts images containing a person from the video; and a first communication unit that, if the person included in the extracted image is identified as the person of interest, transmits to the robot control system the camera identifier of at least the camera that captured the person of interest.

[0008] This disclosure provides a robot control method for controlling a self-propelled robot, wherein the method receives attribute information of a person of interest and a camera identifier identified by an external system that processes images captured by a plurality of cameras, identifies position information which is information relating to the movement of the robot based on the received attribute information and the camera identifier, and moves the robot based on the identified position information, wherein the attribute information is information indicating the attributes of the person of interest included in the image captured by at least one of the plurality of cameras, and the camera identifier is an identifier assigned to each of the at least one cameras. [Effects of the Invention]

[0009] According to this disclosure, self-propelled robots can be moved more effectively to target individuals of interest. [Brief explanation of the drawing]

[0010] [Figure 1] Block diagram showing an example configuration of a management system including a robot control system according to Embodiment 1. [Figure 2] Block diagram showing an example of the internal configuration of the system according to Embodiment 1. [Figure 3] This diagram illustrates the first warning operation of the robot in the system according to Embodiment 1. [Figure 4] This diagram illustrates the second vigilance operation of the robot in the system according to Embodiment 1. [Figure 5] This diagram illustrates the third warning action of the robot in the system according to Embodiment 1. [Figure 6] This diagram illustrates the fourth warning action of the robot in the system according to Embodiment 1. [Figure 7] A diagram illustrating the conversion table in the system according to Embodiment 1. [Figure 8] A diagram illustrating the conversion table in the system according to Embodiment 1. [Figure 9]A flowchart showing the control process flow in the first operation of the external system according to Embodiment 1. [Figure 10] A flowchart showing the control process flow in the second operation of the external system according to Embodiment 1. [Figure 11] A flowchart showing the control process flow in the operation of the robot control system according to Embodiment 1. [Figure 12] Block diagram showing an example configuration of a management system including a robot control system according to a modified example of Embodiment 1. [Figure 13] Block diagram showing an example configuration of a management system including a robot control system according to a modified example of Embodiment 1. [Figure 14] A block diagram showing an example of the internal configuration of a control terminal in a modified system according to Embodiment 1. [Modes for carrying out the invention]

[0011] (Background leading to this disclosure) Conventionally, in a store (or facility), a face recognition system may be introduced from the perspective of security measures or the like. This face recognition system, for example, detects a person in the store from the video captured by a surveillance camera installed in the store, extracts the feature amounts of the person, compares the extracted feature amounts with the registered information in a predetermined database, and determines whether the person matches the detection target person to be registered in the predetermined database. Here, the predetermined database is a database in which persons who have committed problem behaviors such as shoplifting in the past, or VIPs or the like are registered. Further, when the face recognition system determines that a person in the store is the detection target person, the face recognition system transmits the identifier of the surveillance camera that captured the person determined to be the detection target person to a store terminal or the like. Then, in the store (or facility), based on the identifier of the surveillance camera received from the face recognition system, a store employee is moved to the corresponding area, and a response to the person determined to be the detection target person is made. However, the response by the store employee is not suitable from various viewpoints such as human resources, response speed, and follow-up. Further, even if the technique described in Patent Document 1 is used, there is room for improvement in the operation of the robot with respect to the detection target person (person of interest).

[0012] Therefore, in the following embodiments, a robot control system, a management system, a camera management system, and a robot control method that can more appropriately move a self-propelled robot with respect to a person of interest will be described.

[0013] Hereinafter, each embodiment specifically disclosing the robot control system, the management system, the camera management system, and the robot control method according to the present disclosure will be described in detail with reference to the drawings as appropriate. However, a more detailed description than necessary may be omitted. For example, a detailed description of already well-known matters or a redundant description of substantially the same configuration may be omitted. This is to avoid making the following description unnecessarily redundant and to facilitate the understanding of those skilled in the art. The attached drawings and the following description are provided for those skilled in the art to fully understand the present disclosure, and it is not intended to limit the subject matter described in the claims by these.

[0014] (Embodiment 1) <System Configuration> Hereinafter, a configuration example of the management system 1 according to Embodiment 1 will be described while referring to FIGS. 1 to 2. FIG. 1 is a block diagram showing a configuration example of the management system 1 including the robot control system 100 according to Embodiment 1. FIG. 2 is a block diagram showing an internal configuration example of the system according to Embodiment 1. Note that the system configuration shown in FIG. 1 is an example, and one device or system may be divided into a plurality of parts, or a plurality of devices or systems may be combined into one. Also, a plurality of the one device or system shown in FIG. 1 may be provided. Also, the configuration shown in FIG. 2 is an example, and some parts may be omitted or other parts may be added as necessary.

[0015] The management system 1 is a system in which a robot RB arranged in a facility (for example, a shopping mall, a store, an airport, a hotel, a hospital, or a station) executes a predetermined action on a person of interest. A person of interest is a person who matches a person to be detected registered in a detection target DB 300 (to be described later), a person who has performed an action to be noted (suspicious actions such as shoplifting), a person who needs support by a facility staff, a lost child, or a VIP, etc. However, it is assumed here that the person of interest is a person imaged by the surveillance camera CA within the facility.

[0016] The management system 1 includes a robot control system 100, an external system 200, a detection target DB (detection target database) 300, a surveillance camera CA, and a network NW. In the example shown in FIG. 1, an example is shown in which the robot control system 100, the external system 200, the detection target DB 300, and the surveillance camera CA are each one, but each may be plural.

[0017] The management system 1 may also have access points, etc., through which the robot control system 100 communicates. In this case, the communication method used by the robot control system 100 when communicating with the access points, etc., may be any wireless communication. The wireless communication referred to here may be communication via a wireless Local Area Network (LAN) such as Wi-Fi (registered trademark), Bluetooth Low Energy (hereinafter referred to as "BLE"), etc., or communication via Internet of Things (IoT) network communication or protocols such as Matter, Z-Wave, or ZigBee.

[0018] [Robot control system] The robot control system 100 is a system that controls the robot RB based on information about persons of interest identified by an external system 200 based on video footage from a surveillance camera CA.

[0019] The robot control system 100 includes a robot RB and a robot control server RS. The control of the robot RB may be in one of the following ways: the robot control server RS ​​generates control commands for the robot RB to operate; the robot RB and the robot control server RS ​​cooperate in controlling the robot; or the robot RB controls the robot independently. If the robot control system 100 controls the robot RB independently, it does not need to include the robot control server RS.

[0020] The robot RB is configured to be able to move autonomously within the facility where it is located. The robot RB includes a communication unit 10, a processor 11, a memory 12, an input unit 13, a monitor 14, a detection unit 15, and a drive unit 16.

[0021] The communication unit 10 is connected to the robot control server RS ​​and the external system 200 via a network NW (or access point) to enable data communication. Alternatively, the communication unit 10 may be directly connected to the robot control server RS ​​and the external system 200 to enable data communication. The communication unit 10 outputs various data transmitted from the robot control server RS ​​and the external system 200 to the processor 11. The communication unit 10 also transmits various data output from the processor 11 to the robot control server RS ​​and the external system 200.

[0022] The processor 11 is an example of a control unit of the robot control system 100, and is configured using, for example, a Central Processing Unit (hereinafter referred to as "CPU") or a Field Programmable Gate Array (hereinafter referred to as "FPGA"), and works in cooperation with the memory 12 to perform various processes and controls. Specifically, the processor 11 refers to the programs and data held in the memory 12 and executes those programs to realize the functions of the robot RB.

[0023] Memory 12 includes, for example, Random Access Memory (hereinafter referred to as "RAM") as work memory used when executing each process of the processor 11, and storage for storing programs and data that define the operation of the processor 11. Data or information generated or acquired by the processor 11 is temporarily stored in RAM. Programs that define the operation of the processor 11 are written to the storage. Memory 12 may record, for example, information on the patrol route that the robot RB travels, map information MP (or area information) of the area in which the robot RB can move autonomously, the robot RB's default actions, conversion table CT (see below), and information detected by the detection unit 15.

[0024] The input unit 13 is an example of an input device that can accept user operations from facility staff, etc., and outputs the content of the input operation based on the user operation to the processor 11. The input unit 13 may be implemented as a touch panel integrated with the monitor 14, or as a physical button located near the housing of the robot RB or the monitor 14 and pressed by the user. The input unit 13 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice. Note that the input unit 13 is not a mandatory configuration and may be omitted.

[0025] The monitor 14 (input device) is configured using a display such as a Liquid Crystal Display (hereinafter referred to as "LCD"), an organic electroluminescence (hereinafter referred to as "EL"), or a touch panel. The monitor 14 displays various screens output from the processor 11.

[0026] The detection unit 15 is controlled by the processor 11 to detect the environment surrounding the robot RB (faces, people, obstacles, or passages, etc.). The detection unit 15 also detects people present in the vicinity of the robot RB by performing one of the following detection methods: face recognition, object recognition, skeleton detection, clothing detection, etc. The detection unit 15 may be implemented using one or more cameras, sensors, lasers, or radar, or it may be implemented using a combination of different devices, such as radar and cameras. The robot RB may also be equipped with a Global Positioning System (GPS) or the like as the detection unit 15, which is capable of determining the robot RB's position. As a result of the detection, the detection unit 15 outputs the robot RB's own position information, information about the environment surrounding the robot RB, etc., to the processor 11.

[0027] The drive unit 16 is controlled by the processor 11 and makes the robot RB move autonomously based on control commands.

[0028] The robot control server RS ​​can be implemented using devices such as a smartphone, tablet, computer, or server. The robot control server RS ​​is managed within the facility where the robot RB is located and performs control of the robot RB. In other words, the robot control server RS ​​can be referred to as a "facility server." The robot control server RS ​​includes a communication unit 10, a processor 21, a memory 22, an input unit 23, and a monitor 24.

[0029] In the examples shown in Figures 1 and 2, the robot control server RS ​​is assumed to be managed within the facility, but it may be a server located outside the facility. Furthermore, the robot control server RS ​​may be shared as part of the robot control system 100 for multiple robots RB. Note that the multiple robots RB are not necessarily limited to those located within the same facility. Each robot RB may be controlled based on map information MP (or area information) or conversion table CT corresponding to its respective facility. Note that the map information MP (or area information) or conversion table CT may be different for each of the multiple robots RB within the same facility.

[0030] The communication unit 20 is connected to the robot RB and the external system 200 via a network NW (or access point) to enable data communication. The communication unit 20 may be connected to the robot RB and the external system 200 via an access point using Wi-Fi®, or it may be connected directly to the robot RB and the external system 200 via BLE without going through an access point to enable data communication. The communication unit 20 outputs various data transmitted from the robot RB or the external system 200 to the processor 21. The communication unit 20 also transmits various data output from the processor 21 to the robot RB or the external system 200.

[0031] The processor 21 is configured, for example, using a CPU or FPGA, and works in cooperation with the memory 22 to perform various processes and controls. Specifically, the processor 21 refers to the programs and data held in the memory 22 and executes those programs to realize the functions of the robot control server RS.

[0032] Memory 22 includes, for example, RAM as work memory used when executing each process of processor 21, and storage for storing programs and data that define the operation of processor 21. Data or information generated or acquired by processor 21 is temporarily stored in RAM. Programs that define the operation of processor 21 are written to the storage. Memory 22 may record, for example, information on the patrol route that robot RB travels, map information MP (or area information) of the area where robot RB can move autonomously, the robot RB's default actions, conversion table CT (see below), and information detected by the detection unit 15.

[0033] The input unit 23 is capable of receiving operations and outputs the content of the input operations to the processor 21. The input unit 23 is implemented as a touch panel integrated with the monitor 24. The input unit 23 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice.

[0034] The monitor 24 is configured using a display such as an LCD, organic EL, or touch panel. The monitor 24 displays various screens (not shown) output from the processor 21.

[0035] [External System] External system 200 is a system that identifies individuals of interest based on footage from surveillance cameras (CA).

[0036] The external system 200 includes a camera management system 400 and a person detection system 500. In the example shown in Figure 1, there is one camera management system 400 and one person detection system 500, but there may be multiple units. Also, in the example shown in Figure 1, the external system 200 includes a camera management system 400 and a person detection system 500, but there may be only the camera management system 400 or the person detection system 500. Furthermore, some of the functions of the camera management system 400 may be implemented by the person detection system 500, and conversely, some of the functions of the person detection system 500 may be implemented by the camera management system 400. For example, some or all of the functions of the person detection system 500 may be implemented by the robot control server RS.

[0037] The camera management system 400 acquires video from the surveillance camera CA and extracts images containing people from the acquired video. The camera management system 400 then transmits the extracted images to the person detection system 500. The person detection system 500 determines whether the person in the image is the same person as the person to be detected, based on the image transmitted from the camera management system 400 and the detection target DB 300. If the person detection system 500 determines that the person in the image is the same person as the person to be detected, it identifies the person in the image as a person of interest. The person detection system 500 then identifies the characteristic information of the person of interest and the detection type in the detection target DB 300 as attribute information (see below). The person detection system 500 then transmits the identified attribute information and the camera identifier of the surveillance camera CA that captured the person of interest to the robot control system 100.

[0038] Furthermore, the camera management system 400 acquires video from the surveillance camera CA and extracts video containing people from the acquired video. The camera management system 400 then transmits the extracted video to the person detection system 500. The person detection system 500 determines whether a person in the video transmitted from the camera management system 400 has performed a noteworthy action by estimating their skeletal structure, etc. If the person detection system 500 determines that a person in the video has performed a noteworthy action, it identifies that person as a person of interest. The person detection system 500 then identifies the characteristic information of the person of interest as attribute information (see below). The person detection system 500 then transmits the identified attribute information and the camera identifier of the surveillance camera CA that captured the person of interest to the robot control system 100.

[0039] The external system 200 (camera management system 400 and person detection system 500) is implemented by devices 210 such as smartphones, tablet terminals, computers, and servers. The external system 200 is managed outside the facility where the robot RB is located and performs preliminary processing for controlling the robot RB. The external system 200 includes a communication unit 30, a processor 31, a memory 32, an input unit 33, and a monitor 34.

[0040] The communication unit 30 is connected to the robot control system 100 and each system constituting the external system 200 (camera management system 400 or person detection system 500) via a network NW (or access point) to enable data communication. The communication unit 20 may also be connected to the robot control system 100 and each system constituting the external system 200 (see above) via an access point using Wi-Fi®, or it may be connected to the robot control system 100 and each system constituting the external system 200 (see above) directly to enable data communication without going through an access point using BLE.

[0041] The processor 31 is configured, for example, using a CPU or FPGA, and works in cooperation with the memory 32 to perform various processes and controls. Specifically, the processor 31 refers to the programs and data held in the memory 32 and executes those programs to realize various functions.

[0042] Memory 32 includes, for example, RAM as work memory used when executing each process of the processor 21, and storage for storing programs and data that define the operation of the processor 21. Data or information generated or acquired by the processor 31 is temporarily stored in RAM. The storage contains programs that define the operation of the processor 31. Memory 32 may, for example, record the detection target DB 300 and the camera identifiers assigned to each surveillance camera CA.

[0043] The input unit 33 is capable of receiving operations and outputs the content of the input operations to the processor 21. The input unit 33 is implemented as a touch panel integrated with the monitor 34. The input unit 33 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice.

[0044] The monitor 34 is configured using a display such as an LCD, organic EL, or touch panel. The monitor 34 displays various screens (not shown) output from the processor 31.

[0045] [Databases to be detected] The detection target DB300 is a database in which individuals who should be subject to detection are registered, such as individuals who have committed shoplifting or other problematic acts in the past, individuals suspected of having committed shoplifting or other problematic acts in the past, or VIPs. These individuals who should be subject to detection may be determined as appropriate by the administrator or users of the detection target DB.

[0046] In the detection target DB300, each person to be detected is associated with their facial image and facial features extracted from that image for facial recognition, and this is registered as detection target information. This detection target information may also include a person identifier, historical information such as past problematic behavior, etc.

[0047] The detection target DB300 is preferably stored in the memory 32 of the external system 200 (particularly the person detection system 500). However, the detection target DB300 may also be stored in the robot control system 100 or a server not shown, or in all of the robot control system 100, the external system 200, and the server not shown, or in any combination thereof.

[0048] [Surveillance camera] The surveillance camera CA monitors the facility by capturing images of the interior. The surveillance camera CA is preferably controlled by the processor 31 of the external system 200 (particularly the camera management system 400).

[0049] [Attribute information] Attribute information is information that indicates the attributes of a person of interest. Attribute information includes characteristic information, such as the person's clothing, bone structure, and gait, which are related to their appearance. Furthermore, attribute information may also include the detection type in the target DB300, the person's facial features, their status, and their category.

[0050] The detection type is, for example, a person identifier or historical information included in the information to be detected. The facial features of individuals of interest are the facial features used for facial recognition. The status of a person of interest is information that indicates the person's state or behavior, such as whether they are running, lying down, or moving towards a designated location. The categories for persons of interest include those requiring surveillance, under surveillance, unusual, lost, and VIP, among others, which are categories assigned to individuals of interest.

[0051] <Robot's warning actions> The following describes examples of the robot RB's warning actions with reference to Figures 3 to 6. Figure 3 is a diagram illustrating the first warning action of the robot RB in the system according to Embodiment 1. Figure 4 is a diagram illustrating the second warning action of the robot RB in the system according to Embodiment 1. Figure 5 is a diagram illustrating the third warning action of the robot RB in the system according to Embodiment 1. Figure 6 is a diagram illustrating the fourth warning action of the robot RB in the system according to Embodiment 1.

[0052] As its normal operation, robot RB patrols a set route. During this time, robot RB may display advertisements or other information on monitor 14. When the robot control system 100 obtains information about a person of interest from the external system 200, robot RB switches from normal operation to alert operation.

[0053] Figures 3-6 show the interior of the facility. Below, we will explain using an example where there are four visitors C1-C4 in a facility arranged as shown in Figures 3-6, with an entrance FE, product shelves S1-S5, and surveillance cameras CA1-CA4.

[0054] [First warning action] The first alert action will be explained using the example of a case where visitor C3, captured by surveillance camera CA4, is determined to be a person of interest (see Figure 3).

[0055] The first alert action is "the movement of robot RB towards visitor C3, who is a person of interest."

[0056] First, the external system 200 determines that visitor C3 is a person of interest based on the video footage captured by the surveillance camera CA4. Next, the external system 200 identifies the attribute information of visitor C3. Subsequently, the external system 200 transmits this attribute information, along with the camera identifier of the surveillance camera CA4 that captured the footage of visitor C3, to the robot control system 100.

[0057] Next, the robot control system 100 determines the location information for moving the robot RB based on the attribute information of visitor C3 obtained from the external system 200 and the camera identifier of the surveillance camera CA4. Here, the destination of robot RB can be determined based on the location information linked to the camera identifier, or the conversion table CT, etc. (the same applies hereafter). Note that the location information here is, for example, the location information near visitor C3.

[0058] Next, robot RB moves to the destination indicated by the identified location information. Subsequently, while moving to the destination, robot RB determines whether the information (person) acquired by the detection unit 15 matches the person of interest, based on the attribute information of the person of interest. In other words, while moving to the destination, robot RB uses the detection unit 15 to determine whether or not a visitor C3 is present in the vicinity (the same applies hereafter). As a result, robot RB moves near visitor C3, who is the person of interest. After this, robot RB may either track visitor C3 or patrol the vicinity of the destination.

[0059] [Second warning action] The second alert action will be explained using the example of a case where visitor C3, captured by surveillance camera CA4, is determined to be a person of interest (see Figure 4).

[0060] The second alert action is "the default action of robot RB towards visitor C3, who is a person of interest." Note that the second alert action follows the same procedure as described above up to the point where robot RB moves near visitor C3, so the explanation will be omitted.

[0061] When the robot RB detects (discovers) visitor C3 using the detection unit 15 during or after travel to the destination, it approaches visitor C3 and performs a predetermined action. These predetermined actions include, for example, speaking to visitor C3, displaying information acquired by the detection unit 15 (such as camera footage) on the monitor 14 to indicate vigilance, approaching or moving away from visitor C3 to indicate vigilance, using gestures to indicate vigilance, or accepting input from the monitor 14 (input device). The predetermined actions are determined appropriately based on attribute information of the person of interest (visitor C3 in this example). Therefore, the predetermined actions may include tracking visitor C3 or patrolling the vicinity of the destination.

[0062] Furthermore, if the attribute information of a person of interest is "abnormal," for example, "fallen," the robot RB may speak to the person of interest. The robot RB may also play announcements or specific alarms at a relatively loud volume. This allows store staff notified by the person detection system 500 or the robot control system 100 to quickly go to the location of the person of interest. The same applies when the attribute information of a person of interest is "requires monitoring."

[0063] [Third warning action] The third type of alert action will be explained using the example of a case where visitor C1, captured by surveillance camera CA1, is determined to be a person of interest, and the merchandise shelf S4 is a priority monitoring area (see Figure 5). This priority monitoring area is, for example, an area where shoplifting is frequent, or an area identified based on attribute information such as the direction of movement or history information of the person of interest (the same applies hereinafter).

[0064] The third alert action is "the movement of robot RB to product shelf S4, which is a priority monitoring area based on the attribute information of visitor C1, who is a person of interest."

[0065] First, the external system 200 determines that visitor C1 is a person of interest based on the video footage captured by the surveillance camera CA1. Next, the external system 200 identifies the attribute information of visitor C1. Then, the external system 200 transmits this attribute information, along with the camera identifier of the surveillance camera CA1 that captured the footage of visitor C1, to the robot control system 100.

[0066] Next, the robot control system 100 determines the location information for moving the robot RB based on the attribute information of visitor C1 obtained from the external system 200 and the camera identifier of the surveillance camera CA1. The location information here is the location information of the product shelf S4, which is the priority monitoring area.

[0067] Next, robot RB moves to the identified destination. This means that robot RB moves to product shelf S4, which is a priority monitoring area based on the attribute information of visitor C1, who is a person of interest. After that, robot RB may detect and track visitor C1, or it may patrol the vicinity of the destination.

[0068] [Fourth warning action] The fourth alert action will be explained using the example of a case where visitor C3, captured by surveillance cameras CA4 and CA3, is determined to be a person of interest, and the product shelf S4 is a priority monitoring area (see Figure 6). In the example shown in Figure 6, visitor C3 is initially at position (a) and then moves to position (b).

[0069] The fourth alert action is "when visitor C3, who is a person of interest, is captured by multiple surveillance cameras CA4 and CA3, robot RB moves to the vicinity of visitor C3, or moves robot RB to the product shelf S4, which is a priority monitoring area based on visitor C3's attribute information (direction of movement and history information)."

[0070] First, assume that visitor C3 is at the position shown in (a). The external system 200 determines that visitor C3 is a person of interest based on the video footage captured by the surveillance camera CA4. Next, the external system 200 identifies the attribute information of visitor C3. Subsequently, the external system 200 transmits this attribute information, along with the camera identifier of the surveillance camera C4 that captured the footage of visitor C3, to the robot control system 100.

[0071] Next, assume that visitor C3 moves to the position shown in (b). As described above, the external system 200 determines that visitor C3, captured by the surveillance camera CA3, is a person of interest, identifies the attribute information of visitor C3, and transmits this attribute information and the camera identifier of the surveillance camera C3 to the robot control system 100.

[0072] In other words, the robot control system 100 receives attribute information and camera identifiers of visitor C3 before it moves, and attribute information and camera identifiers of visitor C3 after it moves, from the external system 200. The robot control system 100 identifies the location information to move robot RB based on the latest attribute information and camera identifiers (i.e., after visitor C3 moves) from this information. The location information here is either the location information near visitor C3, or the location information of the product shelf S4, which is a priority monitoring area.

[0073] Then, robot RB moves to the identified destination. This means that robot RB moves to the vicinity of visitor C3, or to the product shelf S4, which is a priority monitoring area based on the attribute information of visitor C3, who is a person of interest. After that, robot RB may detect and track visitor C3, or it may patrol the vicinity of the destination, or the area including surveillance cameras CA4 and CA3.

[0074] <Conversion Table> The following will explain the conversion table CT with reference to Figures 7 and 8. Figure 7 is a diagram illustrating the conversion table TP in the system according to Embodiment 1. Figure 8 is a diagram illustrating the conversion table TP in the system according to Embodiment 1.

[0075] The conversion table CT is essentially a table used by the robot control system 100 to convert information acquired from the external system 200 into actions for the robot RB. The information acquired from the external system 200 here refers to attribute information of persons of interest or camera identifiers.

[0076] The conversion table CT registers location information for moving the robot RB, default actions to be performed by the robot RB, and priority, all linked to a camera identifier or attribute information. The conversion table CT may also be created by referring to map information MP of the area in which the robot RB can move autonomously, as shown in Figure 7. In this case, the conversion table CT may also register point information (points indicated by A to H) linked to the map information MP, as shown in Figure 8, and this point information may be linked to various other information. Furthermore, the location information for moving the robot RB is not limited to coordinate values; a patrol range or movement path for the robot RB to travel around may also be set.

[0077] As a result, the camera management system 400 does not need to instruct the robot's guiding position for each of the multiple surveillance cameras CA, thus simplifying the interface between the robot control system 100 and the external system 200. Furthermore, for example, if the external system 200 has multiple camera management systems 400, the robot control system 100 can appropriately set the position information even if each of the multiple camera management systems 400 has a different interface specification.

[0078] Furthermore, the robot control system 100 is configured to receive camera identifiers and attribute information from an external system 200 and set position information based on them. This makes it easy to move the robot RB to different positions or adaptively change the behavior of the robot RB for the same surveillance camera CA, depending on the attribute information. This allows users to easily configure the control of the robot RB for newly added surveillance cameras CA, or to change the control of the robot RB for existing surveillance cameras CA, via the robot control system 100.

[0079] By allowing users to specify the patrol range or movement path of the robot RB as location information linked to a camera identifier, it is possible to set an appropriate patrol range or movement path according to the actual store layout and the placement of items that are prone to shoplifting. This is expected to make it easier to find suspicious individuals during patrols. Furthermore, even if the robot RB fails to find suspicious individuals, it is expected to deter shoplifting.

[0080] The conversion table CT is an example of a configuration for building a robot control system 100 with such features. Functions equivalent to the conversion table CT may be implemented using alternative configurations or means, such as software, scripts, or flow definition data.

[0081] <Operation of external systems> The operation examples of the external system 200 will be described below with reference to Figures 9 to 10. Figure 9 is a flowchart showing the flow of control processing in the first operation of the external system 200 according to Embodiment 1. Figure 10 is a flowchart showing the flow of control processing in the second operation of the external system 200 according to Embodiment 1.

[0082] In this example, the processing entity is described as the external system 200, but the functions may be divided and executed by the camera management system 400 and the person detection system 500. Furthermore, the following first and second operations may be executed by both the camera management system 400 and the person detection system 500, or by only one of them. Also, the first and second operations may be executed simultaneously and in parallel, or sequentially.

[0083] [First action] The first operation of the external system 200 is to determine whether or not a person to be detected, who is registered in the detection target DB 300, is present within the facility.

[0084] The external system 200 (for example, the processor 31, and so on) acquires the captured video from each surveillance camera CA (step Sp101).

[0085] The external system 200 extracts images containing people from the acquired video (step Sp102).

[0086] The external system 200 compares the facial features of the person in the extracted image with the facial features of the person to be detected registered in the detection target database (step Sp103).

[0087] The external system 200 determines whether the person in the extracted image matches the person to be detected, that is, whether they are the same person (step Sp104). If it is determined that the person in the extracted image is the same person as the person to be detected (YES in step Sp104), the external system 200 proceeds to step Sp105. On the other hand, if it is determined that the person in the extracted image is not the same person as the person to be detected (NO in step Sp104), the external system 200 returns to step Sp101.

[0088] The external system 200 determines that the person in question is a person of interest and identifies attribute information including characteristic information of this person of interest and the detection type in the detection target DB 300 (step Sp105).

[0089] The external system 200 transmits attribute information of the person of interest and the camera identifier of the surveillance camera CA that captured the image of the person of interest to the robot control system 100 (step Sp106). Then, this processing flow ends.

[0090] [Second action] The second operation of the external system 200 is to determine whether or not a person exhibiting noteworthy behavior (such as suspicious behavior) is present within the facility.

[0091] The external system 200 (for example, the processor 31, and so on) acquires the captured video from each surveillance camera CA (step Sp201).

[0092] The external system 200 extracts footage containing people from the acquired video (step Sp202).

[0093] The external system 200 analyzes the actions of the people captured in the extracted video by methods such as skeletal estimation (step Sp203).

[0094] The external system 200 determines whether the person in the extracted video has taken a noteworthy action (step Sp204). If it is determined that the person in the extracted video has taken a noteworthy action (YES in step Sp204), the external system 200 proceeds to step Sp205. If it is determined that the person in the extracted video has not taken a noteworthy action (NO in step Sp204), the external system 200 returns to step Sp201.

[0095] The external system 200 determines that the person in question is a person of interest and identifies attribute information including characteristic information of this person of interest (step Sp205).

[0096] The external system 200 transmits attribute information of the person of interest and the camera identifier of the surveillance camera CA that captured the image of the person of interest to the robot control system 100 (step Sp206). Then, this processing flow ends.

[0097] <Operation of the robot control system> The following describes an example of the operation of the robot control system 100 with reference to Figure 11. Figure 11 is a flowchart showing the flow of control processing in the operation of the robot control system 100 according to Embodiment 1.

[0098] In this example, the processing entity is described as the robot control system 100, but the following control processing is performed by at least one of the robot RB and the robot control server RS, as described above. Furthermore, the following control processing may be performed in a manner that coordinates multiple robots RB.

[0099] When attribute information and a camera identifier are transmitted from the external system 200 during the first or second operation of the external system 200, the robot control system 100 starts the following processing flow.

[0100] The robot control system 100 (for example, processors 11, 21, and so on) obtains attribute information of the person of interest and the camera identifier of the surveillance camera CA that captured the person of interest from the external system 200 (step Sp301).

[0101] The robot control system 100 acquires position information for moving the robot RB based on the acquired attribute information and camera identifier (step Sp302). Here, the robot control system 100 identifies the position information based on, for example, a transformation table CT.

[0102] The robot control system 100 moves the robot RB to the location (destination) indicated by the identified location information (step Sp303). The robot RB's movement path here does not have to be the shortest distance. For example, it may proceed to the destination via a priority monitoring area.

[0103] The robot control system 100 determines whether the detection unit 15 has detected a person of interest, that is, whether it has found a person of interest, while the robot RB is moving (step Sp304). Here, it refers to the acquired attribute information and determines whether the information (person) acquired by the detection unit 15 matches a person of interest. If the robot RB has found a person of interest (YES in step Sp304), the robot control system 100 proceeds to step Sp307. On the other hand, if the robot RB has not found a person of interest (NO in step Sp304), the robot control system 100 proceeds to step Sp305.

[0104] The robot control system 100 determines whether or not robot RB has arrived at its destination (step Sp305). If robot RB has arrived at its destination (YES in step Sp305), the robot control system 100 proceeds to step Sp306. On the other hand, if robot RB has not arrived at its destination (NO in step Sp305), the robot control system 100 returns to step Sp304.

[0105] The robot control system 100 instructs robot RB to patrol the vicinity of the destination (step Sp306). If a person of interest is found during patrolling the vicinity of the destination (vigilant patrol), it is advisable to have robot RB perform a predetermined action. Then this processing flow ends. Then this processing flow ends.

[0106] If robot RB finds a person of interest (YES in step Sp304), the robot control system 100 causes robot RB to perform a predetermined action (step Sp307). Then, this processing flow ends.

[0107] In summary, the system according to Embodiment 1 can raise awareness in individuals of interest who are suspicious, thereby preventing problematic behaviors such as shoplifting. Furthermore, the system according to Embodiment 1 can conduct a search for individuals of interest who are lost. In addition, the system according to Embodiment 1 can provide appropriate effects depending on the individual of interest.

[0108] Furthermore, in the system according to this embodiment, if there are multiple persons of interest within the facility, the system prioritizes responding to persons of interest who match those registered in the detection target DB300, and prioritizes responding to persons of interest identified by noteworthy behavior (such as suspicious behavior like shoplifting).

[0109] (Other variations) Figure 12 is a block diagram showing an example configuration of a management system 1 including a robot control system 100 according to a modified example of Embodiment 1. Figure 13 is a block diagram showing an example configuration of a management system 1 including a robot control system 100 according to a modified example of Embodiment 1. Figure 14 is a block diagram showing an example of the internal configuration of a control terminal 600 in a system according to a modified example of Embodiment 1. Note that the internal configuration of the robot control system 100 and the external system 200 is the same as in Embodiment 1, so the illustration is simplified in the example shown in Figure 14.

[0110] In the above-described embodiment 1, an example was described in which the management system 1 includes a robot control system 100, an external system 200, a detection target DB (detection target database) 300, a surveillance camera CA, and a network NW. On the other hand, in a modified example of the above-described embodiment 1, the management system 1 further includes a control terminal 600.

[0111] The control terminal 600 is configured to allow the operator of the control terminal 600 to operate the robot RB. The control terminal 600 can be implemented by a device such as a smartphone, tablet, or computer. The control terminal 600 may be managed outside the facility where the robot RB is located (for example, by an external system 200 or staff of a security company, etc.) as shown in Figure 12, or it may be managed inside the facility where the robot RB is located (for example, by a robot control system 100 or staff of the facility, etc.) as shown in Figure 13. The control terminal 600 includes a communication unit 40, a processor 41, a memory 42, an input unit 43, and a monitor 44, as shown in Figure 14.

[0112] The communication unit 40 is connected to the robot RB via a network NW (or access point) to enable data communication. The communication unit 20 may be connected to the robot RB via an access point using Wi-Fi®, or it may be connected directly to the robot RB via BLE without going through an access point to enable data communication. The communication unit 20 outputs various data transmitted from the robot RB to the processor 41. The communication unit 40 also transmits various data output from the processor 41 to the robot RB.

[0113] The processor 41 is configured, for example, using a CPU or FPGA, and works in cooperation with the memory 42 to perform various processes and controls. Specifically, the processor 41 refers to the programs and data held in the memory 42 and executes those programs to realize the functions of the control terminal 600.

[0114] Memory 42 includes, for example, RAM as work memory used when executing each process of the processor 41, and storage for storing programs and data that define the operation of the processor 41. Data or information generated or acquired by the processor 41 is temporarily stored in RAM. The storage contains programs that define the operation of the processor 41. Memory 42 may, for example, record information for manually operating the robot RB.

[0115] The input unit 43 is capable of receiving operations and outputs the content of the input operations to the processor 41. The input unit 43 is implemented as a touch panel integrated with the monitor 44. The input unit 43 may also be equipped with a microphone (not shown) and configured to accept voice input operations based on the user's voice.

[0116] The monitor 44 is configured using a display such as an LCD, organic EL, or touch panel. The monitor 44 displays various screens (not shown) output from the processor 41.

[0117] Here, we will explain an example of how to use the control terminal 600. For example, suppose there is a person in the facility wearing similar or identical clothing (such as a school uniform) to a person identified as a person of interest. In such a case, by using the control terminal 600, the operator of the control terminal 600 can decide which person to track, and based on that decision, control the robot RB by operating the robot RB or giving instructions to the robot RB. At this time, the control terminal 600 displays information acquired by the robot RB's detection unit 15 and operation items on the monitor 44 and accepts input operations, so that the operator of the control terminal 600 can intuitively operate the robot RB.

[0118] The operations that can be performed by the control terminal 600 include, for example, selecting a person to track, voice output by the robot RB, voice or video communication via the robot RB, movement of the robot RB, and selection of actions (e.g., default actions) to be performed by the robot RB.

[0119] In other words, by using the control terminal 600, the behavior of the robot RB can be adaptively changed.

[0120] Furthermore, this disclosure also applies to programs and storage media that supply programs that realize the functions of the apparatus of the above-described embodiment to the apparatus via a network or various storage media, and which are read and executed by a computer within the apparatus.

[0121] Although various embodiments have been described above with reference to the drawings, it goes without saying that this disclosure is not limited to these examples. It will be clear to those skilled in the art that various modifications, alterations, substitutions, additions, deletions, and equivalents can occur within the scope of the claims, and these will naturally fall within the technical scope of this disclosure. Furthermore, the components of the various embodiments described above can be combined arbitrarily without departing from the spirit of the invention.

[0122] (Note) The following technologies are disclosed based on the above description of embodiments.

[0123] <Technology 1> A robot control system (e.g., 100) comprising a control unit (e.g., 11, 21) for controlling a self-propelled robot (e.g., RB) and a communication unit (e.g., 12, 22) for sending and receiving information via a network, The aforementioned communication unit (for example, 12, 22) An external system (e.g., 200, 400, 500) that processes images captured by multiple cameras (e.g., CA) receives attribute information of persons of interest identified by the external system, as well as camera identifiers. The control unit (for example, 11, 21) Based on the received attribute information and camera identifier, the system identifies location information which is information related to the movement of the robot, and moves the robot based on the identified location information. The aforementioned attribute information is, This information indicates the attributes of the person of interest included in the video captured by at least one of the aforementioned multiple cameras. The aforementioned camera identifier is, The identifier assigned to each of the at least one of the aforementioned cameras is: Robot control system.

[0124] This configuration allows the robot to efficiently and accurately move towards a person of interest based on their attribute information and the camera identifier of the camera that captured the person of interest. This automates monitoring, reducing human resources while enabling efficient and continuous response. In short, this configuration allows the robot to move more appropriately towards a person of interest.

[0125] <Technology 2> The aforementioned robot, The robot has a detection unit (for example, 15) that detects the environment around it, The control unit (for example, 11, 21) Based on the detection information acquired by the detection unit, it is determined whether or not the person of interest is present around the robot. The robot control system described in Technology 1.

[0126] This configuration allows the robot to accurately perceive its surroundings and determine the presence of individuals of interest, enabling efficient response and monitoring.

[0127] <Technology 3> The control unit (for example, 11, 21) If it is determined that the person of interest is present in the vicinity of the robot, the robot is instructed to perform a predetermined action. The robot control system described in Technology 2.

[0128] This configuration allows for a swift and reliable response to individuals of interest.

[0129] <Technology 4> The control unit (for example, 11, 21) If it is determined that the person of interest is not present in the vicinity of the robot, it is determined whether the robot has arrived at the destination indicated by the location information identified by the robot. If it is determined that the robot has arrived at the destination, the robot is instructed to patrol the area around the destination. The robot control system described in Technology 2.

[0130] This configuration expands the monitoring range, enabling the rediscovery of individuals of interest and a rapid response to abnormal situations.

[0131] <Technology 5> Equipped with multiple such robots, The control unit (for example, 11, 21) To coordinate the multiple robots mentioned above, A robot control system described in any one of the technologies 1 through 4.

[0132] This configuration enables efficient work distribution and wide-area monitoring.

[0133] <Technology 6> A robot control system (e.g., 100) described in any one of Technology 1 to Technology 5, An external system (e.g., 200) that processes images captured by multiple cameras, The system comprises a detection target database (e.g., 300) in which detection target information, including facial images of individuals to be detected and facial features extracted from the facial images, The aforementioned external system By referring to the aforementioned database of detected individuals, we identify persons of interest. A management system (for example, 1).

[0134] This configuration allows for the rapid and accurate identification of individuals of interest, and enables the robot to move and respond to these individuals efficiently and accurately.

[0135] <Technology 7> The aforementioned external system A camera management system (e.g., 400) that acquires video from multiple cameras and extracts images containing people from the video, A person detection system (e.g., 500) is provided, which determines whether the person included in the image is the same person as the person to be detected, based on the extracted image and the detection target database, and if it is determined that they are the same person, identifies the person included in the image as the person of interest. The management system described in Technology 6.

[0136] This configuration allows for the rapid and accurate identification of individuals of interest, and enables the robot to move and respond to these individuals efficiently and accurately.

[0137] <Technology 8> The robot is further provided with a control terminal (e.g., 600) capable of operating the robot. The aforementioned control terminal is The robot accepts voice or video input from the operator of the control terminal so that the operator's voice or video can be output via the robot. A management system as described in Technology 6 or Technology 7.

[0138] This configuration allows the operator of the control terminal to communicate with the person of interest.

[0139] <Technology 9> A camera management system (e.g., 400) capable of communicating with a robot control system described in any one of Technology 1 to Technology 5, A first control unit (for example, 31) acquires video from the multiple cameras and extracts images containing people from the video, The system includes a first communication unit (e.g., 30) that, if the person included in the extracted image is identified as the person of interest, transmits at least the camera identifier of the camera that captured the person of interest to the robot control system. Camera management system.

[0140] This configuration allows the robot to move efficiently and accurately towards a person of interest based on the camera identifier of the camera that captured the person of interest. In other words, this configuration allows the robot to move more appropriately towards the person of interest.

[0141] <Technology 8> A robot control method for controlling a self-propelled robot (e.g., RB), An external system (e.g., 200, 400, 500) that processes images captured by multiple cameras (e.g., CA) receives attribute information of persons of interest identified by the external system, as well as camera identifiers. Based on the received attribute information and camera identifier, the location information, which is information related to the movement of the robot, is identified. Based on the identified position information, the robot is moved. The aforementioned attribute information is, This information indicates the attributes of the person of interest included in the video captured by at least one of the aforementioned multiple cameras. The aforementioned camera identifier is, The identifier assigned to each of the at least one of the aforementioned cameras is: Robot control methods.

[0142] This configuration allows the robot to move efficiently and accurately towards a person of interest, based on the attribute information of that person and the camera identifier of the camera that captured the image of that person. In other words, this configuration allows the robot to move more appropriately towards the person of interest. [Industrial applicability]

[0143] This disclosure is useful for robot control systems, management systems, camera management systems, and robot control methods that enable self-propelled robots to move more appropriately towards persons of interest. [Explanation of Symbols]

[0144] 1 Management System 15 Detection unit 100 Robot Control Systems 200 External Systems 300 databases to be detected 400 Camera Management System 500 Person Detection System C1-C4 Visitors CA, CA1~CA4 Surveillance Cameras CT Conversion Table S1-S5 Product Shelves

Claims

1. A robot control system comprising a control unit for controlling a self-propelled robot and a communication unit for sending and receiving information via a network, The aforementioned communications unit is From an external system that processes images captured by multiple cameras, the system receives attribute information of persons of interest identified by the external system, as well as camera identifiers. The control unit, Based on the received attribute information and camera identifier, the system identifies location information which is information related to the movement of the robot, and moves the robot based on the identified location information. The aforementioned attribute information is, This information indicates the attributes of the person of interest included in the video captured by at least one of the aforementioned multiple cameras. The aforementioned camera identifier is, The identifier assigned to each of the at least one of the aforementioned cameras is: Robot control system.

2. The aforementioned robot, The robot has a detection unit that detects the environment around it, The control unit, Based on the detection information acquired by the detection unit, it is determined whether or not the person of interest is present around the robot. The robot control system according to claim 1.

3. The control unit, If it is determined that the person of interest is present in the vicinity of the robot, the robot is instructed to perform a predetermined action. The robot control system according to claim 2.

4. The control unit, If it is determined that the person of interest is not present in the vicinity of the robot, it is determined whether the robot has arrived at the destination indicated by the location information identified by the robot. If it is determined that the robot has arrived at the destination, the robot is instructed to patrol the area around the destination. The robot control system according to claim 2.

5. Equipped with multiple such robots, The control unit, To coordinate the multiple robots mentioned above, The robot control system according to claim 1.

6. A robot control system according to any one of claims 1 to 5, An external system that processes images captured by multiple cameras, The system comprises a detection target database in which detection target information, including facial images of individuals to be detected and facial features extracted from the facial images, The aforementioned external system By referring to the aforementioned database of detected individuals, we identify persons of interest. Management system.

7. The aforementioned external system A camera management system that acquires video from multiple cameras and extracts images containing people from the video, A person detection system that determines, based on the extracted image and the detection target database, whether the person included in the image is the same person as the detection target person, and if it is determined that they are the same person, identifies the person included in the image as the person of interest, The management system according to claim 6.

8. The robot is further equipped with a control terminal capable of operating it. The aforementioned control terminal is The robot accepts voice or video input from the operator of the control terminal so that the operator's voice or video can be output via the robot. The management system according to claim 6.

9. A camera management system capable of communicating with a robot control system according to any one of claims 1 to 5, A first control unit acquires video from the aforementioned multiple cameras and extracts images containing people from the video, The system includes a first communication unit that, if the person included in the extracted image is identified as the person of interest, transmits at least the camera identifier of the camera that captured the person of interest to the robot control system. Camera management system.

10. A robot control method for controlling a self-propelled robot, From an external system that processes images captured by multiple cameras, the system receives attribute information of persons of interest identified by the external system, as well as camera identifiers. Based on the received attribute information and camera identifier, the location information, which is information related to the movement of the robot, is identified. Based on the identified position information, the robot is moved. The aforementioned attribute information is, This information indicates the attributes of the person of interest included in the video captured by at least one of the aforementioned multiple cameras. The aforementioned camera identifier is, The identifier assigned to each of the at least one of the aforementioned cameras is: Robot control methods.

Images