Data processing apparatus, data processing method, and robot

Abstract

The present technology relates to a data processing apparatus, a data processing method, and a robot, with which human sensing can be performed using an appropriate algorithm. A data processing apparatus according to one aspect of the present technology adaptively selects and executes a human sensing program in accordance with human sensing conditions, the human sensing program specifying a human sensing algorithm that performs human sensing based on sensor data output from a sensor mounted on a robot. The present technology is applicable to sensor apparatuses mounted on various devices.

Description

TECHNICAL FIELD

[0001] The present technology relates to a data processing apparatus, a data processing method, and a robot that can perform human sensing using an appropriate algorithm. BACKGROUND

[0002] Conventionally, various techniques for updating software of an apparatus have been proposed from the perspective of adding functions and ensuring compatibility with other apparatuses.

[0003] For example, Patent Literature 1 discloses a technique of determining a service that can be implemented by a combination of a camera and a communication apparatus and installing software that provides the service.

[0004] Further, Patent Literature 2 discloses a technique of updating firmware between an imaging apparatus and a host system in a case where it is detected that the firmware on the imaging apparatus is incompatible with the host system.

[0005] [LIST OF CITATIONS]

[0006] [Patent Literature]

[0007] Patent Literature 1: Japanese Patent Application Laid-Open No. 2007-286850

[0008] Patent Literature 2: Japanese Unexamined Patent Publication No. 2002-504724 SUMMARY

[0009] PROBLEMS TO BE SOLVED BY THE INVENTION

[0010] Although various techniques for changing software of an imaging apparatus such as a camera are disclosed, there is no disclosure of adaptively changing software of an apparatus that performs sensing in accordance with a condition of a sensed target or the like.

[0011] The present technology is achieved in view of such a situation, and an object thereof is to perform human sensing using an appropriate algorithm.

[0012] SOLUTION TO PROBLEM

[0013] A data processing apparatus according to a first aspect of the present technology is provided with a sensing control unit configured to adaptively select a human sensing program in which a human sensing algorithm that senses a human based on sensor data output from a sensor mounted on a robot is defined, in accordance with a human sensing condition.

[0014] A data processing apparatus according to a second aspect of the present technology is provided with a data processing unit configured to adaptively select a human sensing program in which a human sensing algorithm that senses a human based on sensor data output from a sensor mounted on a robot is defined, in accordance with a human sensing condition, and transmit the same to the robot.

[0015] The robot according to the third aspect of the present technology is provided with a sensor configured to output sensor data indicating a sensing result, a sensing control unit configured to adaptively select a person sensing program in which a person sensing algorithm for sensing a person based on sensor data output from the sensor is defined, to be executed in accordance with a person sensing condition, an operation plan setting unit configured to set an operation plan based on a result of execution of the person sensing program by the sensing control unit, and an operation unit configured to execute an operation in accordance with the operation plan set by the operation plan setting unit.

[0016] In the first aspect of the present technology, a person sensing program in which a person sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a robot is defined is adaptively selected to be executed in accordance with a person sensing condition.

[0017] In the second aspect of the present technology, a person sensing program in which a person sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a robot is defined is adaptively selected and transmitted to a robot in accordance with a person sensing condition.

[0018] In the third aspect of the present technology, a person sensing program in which a person sensing algorithm for sensing a person based on sensor data output from a sensor outputting sensor data indicating a sensing result is defined is adaptively selected to be executed in accordance with a person sensing condition, an operation plan is set based on a result of execution of the person sensing program, and an operation is executed in accordance with the set operation plan. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is a view showing a configuration example of a program providing system according to an embodiment of the present technology.

[0020] Figure 2 is a view showing a configuration example of a sensor device.

[0021] Figure 3 is a view showing an example of an appearance of a sensor device.

[0022] Figure 4 is a view showing an example of a sensing program.

[0023] Figure 5 is a view showing another example of a sensing program.

[0024] Figure 6 is a view showing still another example of a sensing program.

[0025] Figure 7 is a view showing an example of an update of a sensing program.

[0026] Figure 8 is a view showing an example of an update of a sensing program.

[0027] Figure 9 is a view showing another example of an update of a sensing program.

[0028] Figure 10 is a view showing an example of a sensing program set.

[0029] Figure 11 is a view showing an example of a sensing program set.

[0030] Figure 12 is a view showing an example of a sensing program set.

[0031] Figure 13 is a view showing an example of an update of a sensing program.

[0032] Figure 14 is a view showing an example of a provision source of a sensing program.

[0033] Figure 15 is a view showing an example of generation of a sensing program set.

[0034] Figure 16 is a view showing a conveyance state of a conveyance robot.

[0035] Figure 17 is an enlarged view of an appearance of a conveyance robot.

[0036] Figure 18 is a view showing an example of a posture of a conveyance robot when a dish is placed thereon.

[0037] Figure 19 is a plan view showing a layout of a space in which a conveyance robot moves.

[0038] Figure 20 is a view showing an example of a sensing algorithm.

[0039] Figure 21 is a block diagram showing a configuration example of hardware of a conveyance robot.

[0040] Figure 22 is a block diagram showing a functional configuration example of a conveyance robot.

[0041] Figure 23 is a flowchart for explaining a process of a conveyance robot.

[0042] Figure 24 is a flowchart for explaining an accuracy selection process performed at Step S1 in Figure 23

[0043] ​Figure 25 is a flowchart for explaining the indoor processing performed at step S14 in Figure 24

[0044] Figure 26 is a flowchart for explaining the outdoor processing performed at step S16 in Figure 24

[0045] Figure 27 is a flowchart for explaining the human sensing processing performed at step S2 in Figure 23

[0046] Figure 28 is a flowchart for explaining the human sensing processing performed at step S2 in Figure 23

[0047] Figure 29 is a view showing an application example of human sensing.

[0048] Figure 30 is a view showing an example of control of a sensing algorithm.

[0049] Figure 31 is a block diagram showing a configuration example of hardware of a computer that implements a program management server. DETAILED DESCRIPTION

[0050] <SUMMARY OF THE PRESENT TECHNOLOGY>

[0051] The present technology focuses on the following points: it is difficult to use the best algorithm to perform sensing in a sensor device installed on various devices such as robots, mobile bodies, and smartphones due to the following factors, and a solution thereof is implemented.

[0052] Factors

[0053] • There are many types of basic technologies.

[0054] • The maturity of the basic technologies is different.

[0055] • There are many variations in terms of cost.

[0056] • There are many variations in terms of applications.

[0057] • System design and implementation are challenging.

[0058] • There are many types of user requests.

[0059] • There are many installation restrictions, such as processor power, power consumption, and circuit size.

[0060] • There are many types of sensing targets.

[0061] ​​​​In particular, this technology enables sensor devices that perform human sensing to use an optimal algorithm as the sensing algorithm to perform sensing, which is an algorithm about sensing.

[0062] The following describes the modes used to implement this technology. The descriptions are given in the following order.

[0063] 1. Program provides system

[0064] 2. Examples of using sensing programs

[0065] 3. Configuration and operation of the transfer robot

[0066] 4. Other usage examples

[0067] 5. Variation

[0068] <Program Provider System>

[0069] Figure 1 This is a view illustrating a configuration example of a program providing system according to one embodiment of the present technology.

[0070] Figure 1 The program providing system is formed by connecting various devices such as mobile terminal 2-1, arm robot 2-2, mobile body 2-3, cooking robot 2-4 and conveyor robot 2-5 to program management server 1 via network 11 including the Internet.

[0071] Mobile terminal 2-1 is a smartphone.

[0072] Arm-type robot 2-2 is a dual-arm robot. A bracket is installed within the shell of arm-type robot 2-2. Arm-type robot 2-2 is a mobile robot.

[0073] Mobile vehicle 2-3 is a car. It is equipped with functions such as autonomous driving.

[0074] Cooking Robot 2-4 is a kitchen-type robot. It has the function of cooking by driving multiple cooking arms. These cooking arms reproduce operations similar to those performed by a human.

[0075] The transfer robot 2-5 is a robot that can place objects onto a top plate, which serves as a placement platform. In this state, the robot can move to the target location. The transfer robot 2-5 has wheels on its base.

[0076] Figure 1 Each device shown is equipped with sensor devices for sensing the environment, objects, people, etc.

[0077] Figure 2is a view showing a configuration example of the sensor device.

[0078] As Figure 2 The sensor device 21 is provided with a controller 31 and a sensor group 32, as shown in

[0079] The controller 31 controls each sensor forming the sensor group 32 to perform sensing of various targets, such as sensing of an environment, sensing of an object, and sensing of a person. The sensing of the controller 31 is performed based on sensor data output from each sensor forming the sensor group 32.

[0080] The controller 31 outputs a sensing result to a device on the host side. Various types of processing are performed by the device on the host side based on the sensing result of the controller 31. In a case where the sensor device 21 is mounted on the mobile terminal 2-1, a central processing unit (CPU) of the mobile terminal 2-1 functions as the device on the host side. The controller 31 also has a function of communicating with the device on the host side.

[0081] The sensor group 32 includes a plurality of sensors that perform sensing of various targets. In Figure 2 In the example in

[0082] The RGB camera 32A includes an image sensor for an RGB image. The RGB camera 32A images a surrounding state by driving the image sensor, and outputs an RGB image acquired by the imaging as sensor data.

[0083] The stereo camera 32B is a distance sensor of a stereo camera system, and includes two image sensors for a distance image. The stereo camera 32B outputs a distance image indicating a distance to a target as sensor data.

[0084] The ToF sensor 32C is a distance sensor of a time-of-flight (ToF) system. The ToF sensor 32C measures a distance to a target by the ToF system, and outputs distance information as sensor data.

[0085] The structured light sensor 32D is a distance sensor of a structured light system. The structured light sensor 32D measures a distance to a target by the structured light system, and outputs distance information as sensor data.

[0086] The laser radar (LiDAR) 32E measures a three-dimensional position of each point of a target, and outputs information indicating the measurement result as sensor data.

[0087] A sensor other than Figure 2The sensors forming the sensor group 32 are different sensors such as a positioning sensor, a gyro sensor, an acceleration sensor, a temperature sensor, and an illuminance sensor.

[0088] The type of the sensor forming the sensor group 32 is appropriately changed depending on the device on which the sensor device 21 is mounted. One sensor can form the sensor group 32.

[0089] The sensor device 21 can include a substrate on which the controller 31 and the sensor group 32 are arranged, or can be formed as a device in which a substrate on which each sensor is arranged is housed in a housing 21A as shown in Figure 3 .

[0090] In the sensor device 21 having such a configuration, the controller 31 executes a sensing program as a program for sensing, and realizes a sensing function of various targets such as an environment, an object, and a person. The sensing function of the controller 31 is realized based on the output of one sensor forming the sensor group 32 or based on a combination of the outputs of a plurality of sensors.

[0091] The environment sensing (sensing of an environment) includes, for example, the following.

[0092] • Imaging of an RGB image using an RGB camera 32A

[0093] • Measurement of a distance to a target using the outputs of a stereo camera 32B, a ToF sensor 32C, and a structured light sensor 32D

[0094] • Generation of a three-dimensional map using the output of a LiDAR 32E

[0095] • Estimation of a self position using the three-dimensional map

[0096] The environment sensed by the sensor device 21 includes various physical states that are states outside the sensor device 21 or the device on which the sensor device 21 is mounted, and which can be represented as quantized data by performing sensing.

[0097] The object sensing (sensing of an object) includes, for example, the following.

[0098] • Recognition and identification of a target using an RGB image imaged by an RGB camera 32A

[0099] • Measurement of a feature of a target, such as a shape, a size, a color, and a temperature

[0100] The object sensed by the sensor device 21 includes various stationary objects and moving objects around the sensor device 21 or the device on which the sensor device 21 is mounted.

[0101] Human sensing (sensing of a human) includes, for example, the following.

[0102] • recognizing a person, recognizing a face of a person, recognizing a person using an RGB image imaged by the RGB camera 32A

[0103] • recognizing a specific part of a person, such as a head, an arm, a hand, an eye, and a nose

[0104] • estimating a position of a specific part, including skeleton estimation

[0105] • estimating a physical feature of a person, such as a height and a weight

[0106] • estimating an attribute of a person, such as an age and a gender

[0107] The human sensed by the sensor device 21 includes a person around the sensor device 21 or around a device on which the sensor device 21 is mounted.

[0108] The controller 31 includes a plurality of programs having different algorithms as sensing programs for implementing respective sensing functions.

[0109] Figure 4 is a view showing examples of sensing programs prepared in the sensor device 21.

[0110] In the example in Figure 4 , the ranging program A, the ranging program B, and the ranging program C are prepared as firmware running on an operating system (OS). The ranging program A, the ranging program B, and the ranging program C are sensing programs implementing a ranging function as a human sensing function.

[0111] The ranging program A, the ranging program B, and the ranging program C are sensing programs implementing the same ranging function by different sensing algorithms. The ranging program A, the ranging program B, and the ranging program C define different sensing algorithms.

[0112] The ranging program A is a sensing program that performs ranging by a ranging algorithm A. The ranging program B is a sensing program that performs ranging by a ranging algorithm B. The ranging program C is a sensing program that performs ranging by a ranging algorithm C.

[0113] For example, the ranging algorithms A to C are sensing algorithms that perform ranging using different parameters: different parameters are set in the same sensor, and a distance is calculated by performing the same calculation based on the output of the sensor.

[0114] Further, the ranging algorithms A to C are sensing algorithms that perform ranging using different calculation methods: the same parameters are set in the same sensor, and a distance is calculated by performing different calculations based on the output of the sensor.

[0115] In a case where a plurality of distance sensors such as the stereo camera 32B, the ToF sensor 32C, and the structured light sensor 32D are prepared, the ranging algorithms A to C can be sensing algorithms that perform ranging using different distance sensors.

[0116] In this case, for example, the ranging algorithm A performs ranging based on the output of the stereo camera 32B, and the ranging algorithm B performs ranging based on the output of the ToF sensor 32C. Further, the ranging algorithm C performs ranging based on the output of the structured light sensor 32D.

[0117] In this way, in the sensor device 21, a plurality of programs having different sensing algorithms are prepared as sensing programs for implementing the same ranging function. A sensor used for ranging is associated with at least any one of each sensing algorithm or a sensing program that defines each sensing algorithm. In a case where a sensing program is executed, the operation of the associated sensor is controlled in conjunction with the sensing program.

[0118] For example, in the sensor device 21 mounted on the transport robot 2-5, a sensing algorithm corresponding to a sensing condition is selected and ranging is performed. The sensing condition is a selection condition of a sensing algorithm determined in accordance with the situation of the transport robot 2-5.

[0119] For example, in a case where the situation is suitable for the ranging algorithm A, the ranging program A is executed, and ranging is performed using the ranging algorithm A. Further, in a case where the situation is suitable for the ranging algorithm B, the ranging program B is executed, and ranging is performed using the ranging algorithm B. In a case where the situation is suitable for the ranging algorithm C, the ranging program C is executed, and ranging is performed using the ranging algorithm C.

[0120] Since ranging is performed by adaptively selecting a sensing algorithm (sensing program) corresponding to a sensing condition, ranging can be performed by the best sensing algorithm. This is equally applicable to a case where the sensing target is not distance.

[0121] A sensing program defines a sensing algorithm. Selecting a sensing program corresponds to selecting a sensing algorithm.

[0122] Note that adaptively selecting a sensing algorithm means selecting a sensing algorithm associated with a sensing condition when the condition is detected. A sensing algorithm considered appropriate is associated with a sensing condition corresponding to each assumed situation. The association between a sensing condition and a sensing algorithm can be dynamically changed.

[0123] Figure 5 is a view showing another example of a sensing program.

[0124] In Figure 5In the example, food identification programs A, B, and C are prepared as firmware to run on the OS. Food identification programs A through C are sensing programs that implement food identification functionality as an object sensing function.

[0125] Food identification program A is a sensing program that performs food identification using food identification algorithm A. Food identification program B is a sensing program that performs food identification using food identification algorithm B. Food identification program C is a sensing program that performs food identification using food identification algorithm C. For example, in sensor device 21 installed on cooking robot 2-4, a sensing algorithm corresponding to sensing conditions determined through the cooking process, etc., is selected and the food is identified.

[0126] Figure 6 This is a view that shows yet another example of a sensing procedure.

[0127] exist Figure 6 In the example, face recognition programs A, B, and C are prepared as firmware to run on the OS. Face recognition programs A through C are sensing programs that implement face recognition functionality as a human sensing function.

[0128] Face recognition program A performs a face recognition sensing procedure using face recognition algorithm A. Face recognition program B performs a face recognition sensing procedure using face recognition algorithm B. Face recognition program C performs a face recognition sensing procedure using face recognition algorithm C. For example, in sensor device 21 installed on cooking robot 2-4, a sensing algorithm corresponding to sensing conditions determined by the cooking process, etc., is selected and a face is recognized.

[0129] • Update of sensing software

[0130] exist Figure 1 The system provides a program that can update the sensing program prepared as firmware in the sensor device 21 of each device.

[0131] Figure 7 This is a view showing an example of an update to the sensing program.

[0132] like Figure 7 As indicated by the arrow, the program management server 1 provides sensing programs to each device. The program management server 1 includes a database (DB) of sensing programs to be provided to each device.

[0133] exist Figure 7 In the example, a ranging program D that performs ranging through a ranging algorithm D is provided to the mobile terminal 2-1, and a face recognition program H that performs face recognition through a face recognition algorithm H is provided to the arm robot 2-2.

[0134] Further, the mobile body 2-3 is provided with a self-position estimation program J that performs self-position estimation by the self-position estimation algorithm J, and the cooking robot 2-4 is provided with an object recognition program K that performs object recognition by the object recognition algorithm K. The transfer robot 2-5 is provided with a person recognition program M that performs person recognition by the person recognition algorithm M.

[0135] Figure 8 is a view showing an example of update of the sensing program.

[0136] In the sensor device 21 of each device, a sensing program can be added as shown in A of Figure 8 In the example in A of Figure 8 In the example in A of

[0137] In the default state, in the sensor device 21 of each device, a sensing program that defines a sensing algorithm corresponding to a general situation is prepared. Even in the case where the sensor device 21 of each device cannot respond to the situation with the sensing program prepared in advance, this can be responded to a special situation by adding a sensing program that defines a sensing algorithm corresponding to such a special situation.

[0138] Further, as shown in B of Figure 8 In the example in B of Figure 8 In the example in B of

[0139] Figure 9 is a view showing another example of update of the sensing program.

[0140] As shown in Figure 9 In the example in Figure 9 In the example in

[0141] In the DB of the program management server 1, as shown in Figure 10 In the example in

[0142] In the example in Figure 10 In the example in In the example in

[0143] The sensing program set corresponding to a place is, for example, a set used in the sensor device 21 installed on a device with a mobile function. In the same room, sensing program sets can be prepared in units of more refined places, such as a sensing program set for a kitchen and a sensing program set for a dining room.

[0144] Sensing program sets can also be prepared for various places, such as a sensing program set for a sea, a sensing program set for a mountain, and a sensing program set for inside a train.

[0145] Furthermore, in the example in Figure 10 , a sensing program set for distance measurement on a sunny day and a sensing program set for distance measurement on a rainy day are prepared. These sensing program sets are sensing program sets corresponding to weather.

[0146] The sensing program set corresponding to weather is, for example, a set used in the sensor device 21 installed on a device with a mobile function and which can move outdoors. Sensing program sets can also be prepared for various changing conditions, such as sensing program sets for each time of day, such as morning, noon, and night, sensing program sets for each brightness, and sensing program sets for each temperature.

[0147] Sensing program sets can also be prepared for various purposes, such as a sensing program set when running, a sensing program set when playing baseball, a sensing program set when cooking curry, and a sensing program set when cooking salad.

[0148] The sensor device 21 of each device can collectively add sensing programs by specifying the ID of the sensing program set corresponding to the use condition. The ID as identification data is set in each sensing program set. The ID as identification data is also set in each sensing program forming the sensing program set.

[0149] Instead of sensing program sets that implement the same distance measurement function by different sensing algorithms, sensing program sets that implement different functions, such as Figure 11 , can be added as shown in

[0150] In the example in Figure 11 , the sensing program set includes a distance measurement program D, a face recognition program H, and an object recognition program K. The distance measurement program D is a sensing program that performs distance measurement by a distance measurement algorithm D, and the face recognition program H is a sensing program that performs face recognition by a face recognition algorithm H. The object recognition program K is a sensing program that performs object recognition by an object recognition algorithm K.

[0151] Figure 12 is a view showing an example of a sensing program set.

[0152] Figure 12The sensing program set illustrated in FIG. 1 includes an algorithm manager that is a program that controls adaptive selection of an algorithm.

[0153] The sensor device 21 executes the algorithm manager and selects a sensing algorithm that corresponds to a sensing condition. In the algorithm manager, a combination of information that indicates a type of sensing program that controls execution and information that indicates an execution order of the sensing program is set. In this way, an algorithm manager can be prepared for each sensing program set.

[0154] Figure 13 is a view that illustrates an example of update of a sensing program.

[0155] A sensing program can be executed in each of the sensor device 21 and the controller 51 that is a device on the host side, and a predetermined function such as a distance measuring function can be implemented. In this case, the sensing program of the controller 51 can be updated similarly to the sensing program of the sensor device 21. The controller 51 is, for example, a data processing device on the host side such as a CPU of the mobile terminal 2-1 and a CPU of a PC installed on the arm-type robot 2-2.

[0156] A sensing program that updates firmware of the sensor device 21 and a sensing program that updates firmware of the controller 51 can be included in one sensing program set to be provided.

[0157] A sensing program and a sensing program set can be provided at a charge or for free. One sensing program set can include both a sensing program that is charged and a sensing program that is free.

[0158] When a sensing program is updated as described above, the sensor device 21 can be authenticated by the program management server 1 based on key information for authentication, and the update is executed in a case where it is confirmed that the sensor device is a legitimate device. The key information for authentication is prepared as unique information in each sensor device 21.

[0159] The authentication of the sensor device 21 using the key information for authentication can be executed not at the time of updating a sensing program but at the time of executing a sensing program.

[0160] • Provision source of sensing program

[0161] Figure 14 is a view that illustrates an example of a provision source of a sensing program.

[0162] As Figure 14 illustrated in FIG. 1, a sensing program that is provided from the program management server 1 to each device is developed by a developer who is registered by a user who executes a service in a program provision system, for example. By a service provider who operates a service using the program provision system, each developer is provided with information on specifications of the sensor device 21 and a development tool such as a software development kit (SDK).

[0163] Each developer develops a sensing program or a sensing program set by using an SDK or the like, and uploads it from his or her own computer to the program management server 1. The uploaded sensing program and sensing program set are stored in the sensing program DB so as to be managed.

[0164] The program management server 1 manages the usage status of each sensing program and sensing program set, such as the number of times of installation and the number of times of execution in each device. A predetermined incentive, such as the issuance of points and the payment of an amount corresponding to the usage status, can be provided from the service provider to the developer.

[0165] Figure 15 is a view showing an example of the generation of a sensing program set.

[0166] A sensing program set can be generated by any user by putting together sensing programs developed and uploaded by each developer.

[0167] In the example in Figure 15 , an indoor distance measuring program set is generated by putting together the distance measuring program D, the distance measuring program E, and the distance measuring program F among the distance measuring programs A to G.

[0168] The indoor distance measuring program set generated in this way is released by the program management server 1 as an installable sensing program set, and is installed on a predetermined device.

[0169] An incentive can be provided to a user who generates a sensing program set by putting together a plurality of sensing programs.

[0170] <Usage example of sensing program>

[0171] • Usage example of conveyance robot

[0172] Here, a usage example of person sensing is described.

[0173] For example, in a case where the conveyance robot 2-5 conveys a dish as a conveyance object, that is, serves a dish in a shop such as a restaurant, the sensor device 21 installed on the conveyance robot 2-5 performs person sensing according to a sensing program. In order to serve a dish to a person who ordered the dish, it is necessary to recognize a person around and specify the person who ordered the dish.

[0174] Figure 16 is a view showing the state of conveyance by the conveyance robot 2-5.

[0175] Figure 16 The state of the conveyance robot 2-5 moving inside a kitchen of a building is shown. A cooked dish is placed on a top plate prepared as a placement table for a conveyance object. In this example, the conveyance robot 2-5 is used to serve a dish.

[0176] The transport robot 2-5 plans a movement route, avoids obstacles, and the like based on the human sensing result of the sensor device 21, moves to a destination, and serves dishes. Furthermore, the transport robot 2-5 controls customer service based on the human sensing result of the sensor device 21.

[0177] Figure 17 is an enlarged view of the appearance of the transport robot 2-5.

[0178] As shown in Figure 17 , the transport robot 2-5 is formed by connecting a ring-shaped base 101 and a circular thin-plate-shaped top plate 102 with a thin rod-shaped support arm 103. A plurality of tires are provided on the bottom surface side of the base 101. The base 101 functions as a movement unit that realizes movement of the transport robot 2-5.

[0179] The radial length of the base 101 is substantially the same as the radial length of the top plate 102. In a case where the top plate 102 is substantially directly above the base 101, the support arm 103 is in a tilted state as shown in Figure 17 .

[0180] The support arm 103 includes an arm member 103-1 and an arm member 103-2. The diameter of the arm member 103-1 on the top plate 102 side is slightly smaller than the diameter of the arm member 103-2 on the base 101 side. When the arm member 103-1 is housed inside the arm member 103-2 at an extension / contraction unit 103A, the length of the support arm 103 is adjusted as indicated by the bidirectional arrow.

[0181] The angle of the support arm 103 can be adjusted at each of the connection between the base 101 and the support arm 103 and the connection between the top plate 102 and the support arm 103.

[0182] Figure 18 is a view showing an example of the posture of the transport robot 2-5 when a dish is placed thereon.

[0183] In the example in Figure 18 , by providing the support arm 103 substantially vertically and setting its length to the maximum length, the height of the top plate 102 is adjusted to be substantially the same as the height of the top plate of the cooking robot 2-4.

[0184] When the transport robot 2-5 is in this state, the cooking arm of the cooking robot 2-4 places a dish on the top plate 102. In the example in Figure 18 , the dish completed by the cooking operation of the cooking robot 2-4 is placed by the cooking arm.

[0185] As shown in Figure 18As shown, cooking robots 2-4 are equipped with multiple cooking arms that perform various cooking operations, such as cutting ingredients, baking ingredients, and arranging cooked ingredients. The cooking operations performed by the cooking arms are executed according to cooking data that defines the content and sequence of the cooking operations. The cooking data includes information about each cooking process before the dish is finished.

[0186] In this way, the dishes served by the delivery robots 2-5 are dishes cooked by the cooking robots 2-4. Dishes prepared by humans can be placed on the top plate 102 and served by humans.

[0187] Figure 19 This is a plan view showing the layout of the space in which the transfer robots 2-5 move.

[0188] like Figure 19 As shown, a kitchen #1 and a hall #2 are prepared in the restaurant in which the conveyor robots 2-5 move. There is a passageway #11 between the kitchen #1 and the hall #2.

[0189] exist Figure 19 The exterior of the building (whose boundaries are indicated by dashed lines) provides a garden #21 facing the lobby #2. Tables and other dining facilities are provided not only in the lobby #2 but also in the garden #21.

[0190] The text describes how a delivery robot 2-5 moves within a space and performs customer service. The customer services performed by the delivery robot 2-5 include receiving orders, serving food, and offering beverages.

[0191] Human sensing performed by the sensor device 21 mounted on the conveyor robot 2-5 is executed using an algorithm corresponding to human sensing conditions set depending on the conditions of the conveyor robot 2-5 (e.g., the location of the conveyor robot 2-5). The sensing program for human sensing is adaptively selected based on the human sensing conditions and executed by the sensor device 21.

[0192] Specific examples of sensing algorithms

[0193] Figure 20 This is a view showing an example of a sensing algorithm defined by the sensing program prepared in transport robots 2-5.

[0194] like Figure 20 As shown, in the conveyor robot 2-5, an individual recognition algorithm A1 and an attribute recognition algorithm A2 are prepared as sensing algorithms for human sensing.

[0195] The individual recognition algorithm Al is a sensing algorithm for recognizing who the target to be sensed is and recognizing attributes of the target. The attributes of the target include gender and age. In addition, the attributes of the target also include features of appearance, such as dominant hand and hair length.

[0196] In contrast, the attribute recognition algorithm A2 is a sensing algorithm for recognizing attributes of the target. With the attribute recognition algorithm A2, the target is not recognized but only the attributes of the target are recognized.

[0197] In addition, for each of the individual recognition algorithm Al and the attribute recognition algorithm A2, a sensing algorithm for each situation, such as an indoor sensing algorithm, an outdoor sensing algorithm, a sensing algorithm for a dark place, and the like, is prepared.

[0198] In Figure 20 For example, the individual recognition algorithm Al-1 is an indoor sensing algorithm, and the individual recognition algorithm Al-2 is an outdoor sensing algorithm. In addition, the individual recognition algorithm Al-3 is a sensing algorithm for a dark place.

[0199] Similarly, the attribute recognition algorithm A2-1 is an indoor sensing algorithm, and the attribute recognition algorithm A2-2 is an outdoor sensing algorithm. In addition, the attribute recognition algorithm A2-3 is a sensing algorithm for a dark place.

[0200] In this way, in the transport robot 2-5, a sensing program defining each individual recognition algorithm Al and a sensing program defining each attribute recognition algorithm A2 corresponding to a person sensing condition set according to a situation are prepared. In the transport robot 2-5, the individual recognition algorithm Al or the attribute recognition algorithm A2 corresponding to the person sensing condition is selected, and person sensing is performed.

[0201] Since the required sensing algorithm differs according to the purpose, such as whether who the target is needs to be specified together with the attributes or whether only the attributes need to be specified, the individual recognition algorithm Al and the attribute recognition algorithm A2 are prepared.

[0202] In addition, a sensing algorithm for each situation is prepared for each of the individual recognition algorithm Al and the attribute recognition algorithm A2, because it is necessary to change the processing according to the situation, such as where the target is, in order to ensure accuracy.

[0203] For example, in the case where person sensing is performed in a dark place, a sensing algorithm against a dark place (noise) is required.

[0204] In addition, in the case where person sensing is performed outdoors, a sensing algorithm against direct sunlight is required. Since the noise differs according to the weather, such as rain and cloudiness, a sensing algorithm corresponding to each weather is required.

[0205] Further, since the illuminance changes due to the passage of time and sudden changes in weather, a sensing algorithm corresponding to each illuminance is required.

[0206] Here, the switching of the individual recognition algorithm Al and the attribute recognition algorithm A2 for each use case is described.

[0207] • Use Case 1

[0208] Use Case 1 is a use case in which the delivery robot 2-5 confirms the entry of a customer into a restaurant. Figure 19

[0209] In Use Case 1, the attribute of the customer is recognized using the attribute recognition algorithm A2. Based on the attribute recognition result, a process of determining the seat to which the customer is guided is performed, for example, by the delivery robot 2-5.

[0210] • Use Case 2

[0211] Use Case 2 is a use case in which the amount and preference of cooking are confirmed when an order is received. In Use Case 2, there are a case of using the individual recognition algorithm Al and a case of using the attribute recognition algorithm A2.

[0212] In the case of using the attribute recognition algorithm A2, the attribute of the customer is recognized using the attribute recognition algorithm A2.

[0213] For example, the person who receives the order and the person who does not receive the order are recognized based on the attribute recognition result.

[0214] Further, information of the attribute of the customer is stored based on the attribute recognition result. The stored information is used for customer service at the next visit or the like.

[0215] Further, the way of receiving the order is determined based on the attribute recognition result. For example, in the case where the customer is a woman, it is confirmed whether or not to reduce the amount at the time of receiving the order. Further, in the case where the customer is a child, when a recommended drink is suggested, a drink other than an alcoholic drink is suggested.

[0216] In the case of using the individual recognition algorithm Al, the individual recognition algorithm Al is used to recognize who the target is together with the attribute.

[0217] Based on the individual recognition result, the way of receiving the order is determined from the order history of this person. For example, a process of presenting the order content in the previous visit or analyzing the preference of this person from the order history to suggest a recommended dish is performed. Further, a process of recommending a dish based on the preference of this person and information on intolerance or explaining the menu according to the background knowledge of this person is performed.

[0218] • Use Case 3

[0219] ​The use example 3 is a use example of delivering (serving) dishes to customers in a hall. In the use example 3, the attribute recognition algorithm A2 recognizes attributes of the customers.

[0220] For example, dishes are delivered to the order target based on the attribute recognition result. Further, in a case where attributes of a person to be preferentially served are set, dishes are preferentially delivered to a person having the set attributes. The processing of changing the serving direction is performed by the dominant hand of the target.

[0221] A series of processes corresponding to the use example as described above will be described later with reference to a flowchart.

[0222] <Configuration and operation of delivery robot>

[0223] • Configuration of delivery robot

[0224] Figure 21 is a block diagram showing a configuration example of hardware of the delivery robot 2-5.

[0225] The delivery robot 2-5 is formed by connecting the top plate lifting driving unit 122, the tire driving unit 123, the sensor group 124, and the communication unit 125 to the controller 121. The sensor device 21 is also connected to the controller 121.

[0226] The controller 121 includes a CPU, a ROM, a RAM, a flash memory, and the like. The controller 121 executes a predetermined program and controls the overall operation of the delivery robot 2-5 including the sensor device 21. The controller 121 corresponds to the controller 51 of the host side. Figure 13 ).

[0227] The top plate lifting driving unit 122 includes a motor or the like provided on a connecting member between the base 101 and the support arm 103, and on a connecting member between the top plate 102 and the support arm 103. The top plate lifting driving unit 122 drives the corresponding connecting member.

[0228] Further, the top plate lifting driving unit 122 includes a track or a motor provided inside the support arm 103. The top plate lifting driving unit 122 extends and retracts the support arm 103.

[0229] The tire driving unit 123 includes a motor that drives a tire provided on the bottom surface of the base 101.

[0230] The sensor group 124 includes various sensors such as a positioning sensor, a gyro sensor, an acceleration sensor, a temperature sensor, and an illuminance sensor. Sensor data indicating detection results of the sensor group 124 is output to the controller 121.

[0231] The communication unit 125 is a wireless communication module, such as a wireless LAN module or a mobile communication module. The communication unit 125 communicates with external devices, such as the program management server 1.

[0232] Figure 22 is a block diagram showing a functional configuration example of the transport robot 2-5.

[0233] Figure 22 At least a part of the functional units shown in FIG. 1 is realized by a CPU forming the controller 121 and a CPU forming the controller 31 of the sensor device 21 by executing a predetermined program.

[0234] In the controller 121, a route information acquisition unit 151, a positioning control unit 152, a movement control unit 153, a posture control unit 155, an environmental data acquisition unit 156, and a surrounding state recognition unit 157 are realized.

[0235] In contrast, in the controller 31 of the sensor device 21, a situation detection unit 201 and a sensing control unit 202 are realized. The sensor device 21 is a data processing device that controls a sensing algorithm.

[0236] The route information acquisition unit 151 of the controller 121 controls the communication unit 125 and receives information of a destination and a movement route transmitted from a control device not shown. The information received by the route information acquisition unit 151 is output to the movement control unit 153.

[0237] The route information acquisition unit 151 can plan a movement route based on a destination and a current position of the transport robot 2-5 when a transport object or the like is ready.

[0238] In this case, the route information acquisition unit 151 functions as an operation plan setting unit that plans an operation of the transport robot 2-5 and sets an operation plan.

[0239] The positioning control unit 152 detects a current position of the transport robot 2-5. For example, the positioning control unit 152 generates a map of a space in which the cooking robot 2-4 is installed based on a detection result of a distance sensor forming the sensor device 21. Sensor data output as the sensor device 21 is acquired by the environmental data acquisition unit 156 and provided to the positioning control unit 152.

[0240] The positioning control unit 152 detects the current position by specifying its own position on the generated map. Information on the current position detected by the positioning control unit 152 is output to the movement control unit 153. The detection of the current position by the positioning control unit 152 can be performed based on an output of a positioning sensor forming the sensor group 124. The current position of the transport robot 2-5 can be detected by the sensor device 21.

[0241] The movement control unit 153 controls the tire drive units 123 to control the movement of the conveyance robot 2-5 based on the information provided from the route information acquisition unit 151 and the current position detected by the positioning control unit 152.

[0242] Further, in a case where the information on the obstacles around it is provided by the surrounding state recognition unit 157, the movement control unit 153 controls the movement so as to avoid the obstacles. The obstacles include various moving objects and stationary objects such as people, furniture, and home appliances. In this way, the movement control unit 153 controls the movement of the conveyance robot 2-5 accompanying the conveyance of the conveyance object based on the people sensing result of the sensor device 21.

[0243] The posture control unit 155 controls the top plate lifting drive units 122 to control the posture of the conveyance robot 2-5. Further, in conjunction with the control of the movement control unit 153, the posture control unit 155 controls the posture of the conveyance robot 2-5 during the movement so as to keep the top plate 102 horizontal.

[0244] The posture control unit 155 controls the posture of the conveyance robot 2-5 in accordance with the surrounding state recognized by the surrounding state recognition unit 157. For example, the posture control unit 155 controls the posture of the conveyance robot 2-5 so that the height of the top plate 102 approaches the height of the top plate of the cooking robot 2-4 or the height of the top plate of the dining table recognized by the surrounding state recognition unit 157.

[0245] The environment data acquisition unit 156 controls the sensor device 21 to perform people sensing and acquire sensor data indicating the people sensing result. The sensor data acquired by the environment data acquisition unit 156 is provided to the positioning control unit 152 and the surrounding state recognition unit 157.

[0246] The surrounding state recognition unit 157 recognizes the surrounding state based on the sensor data indicating the people sensing result provided from the environment data acquisition unit 156. Information indicating the recognition result of the surrounding state recognition unit 157 is provided to the movement control unit 153 and the posture control unit 155.

[0247] In a case where the detection of the obstacles, the measurement of the distance to the obstacles, the estimation of the direction of the obstacles, the estimation of the own position, and the like are performed by the sensor device 21, the surrounding state recognition unit 157 outputs the information on the obstacles as the information indicating the recognition result of the surrounding state.

[0248] The detection of an obstacle, the measurement of the distance to the obstacle, the estimation of the direction of the obstacle, the estimation of the own position, and the like can be performed by the surrounding state recognition unit 157 based on the sensing results of the sensor device 21. In this case, the sensor data for each processing performed by the surrounding state recognition unit 157 is detected by the sensor device 21.

[0249] In this way, the content of the processing performed by the sensor device 21 is arbitrary. That is, the raw data detected by the sensor provided on the sensor device 21 can be directly provided to the controller 121 as the sensor data, or the processing and analysis of the raw data can be performed on the sensor device 21 side, and the result of the processing and analysis can be provided to the controller 121 as the sensor data.

[0250] The condition detection unit 201 on the sensor device 21 side detects the condition of the transport robot 2-5. The condition of the transport robot 2-5 is detected based on, for example, the sensor data output from the sensors forming the sensor group 124 or the sensor data output from the sensors provided on the sensor device 21.

[0251] The condition of the transport robot 2-5 includes, for example, the operation of the transport robot 2-5, such as the operation performed by the transport robot 2-5, the place where the transport robot 2-5 is located, the weather, the temperature, the humidity, and the brightness of the place where the transport robot 2-5 is located. Further, the condition of the transport robot 2-5 also includes external conditions, such as the condition of the person in communication with the transport robot 2-5 and the condition of the obstacle around the transport robot 2-5.

[0252] The condition detection unit 201 outputs information indicating such a condition of the transport robot 2-5 to the sensing control unit 202.

[0253] The sensing control unit 202 selects a sensing algorithm in accordance with the person sensing condition to be performed in the condition detected by the condition detection unit 201, and executes a sensing program defining the selected sensing algorithm.

[0254] For example, the sensing algorithm or the sensing program is associated with each person sensing condition. The sensing control unit 202 selects the sensing algorithm or the sensing program corresponding to the person sensing condition using the ID as the identification data. A set of sensing programs can be selected in accordance with the person sensing condition.

[0255] The sensing control unit 202 drives each sensor provided on the sensor device 21 by executing the sensing program, and performs person sensing based on the output of each sensor. The sensing control unit 202 outputs the sensor data indicating the person sensing result to the controller 121. The sensing control unit 202 appropriately outputs various types of sensor data other than the person sensing result.

[0256] • Operation of the transport robot

[0257] Referring to Figure 23 the flowchart in FIG. 10, the process of the transport robot 2-5 will be described.

[0258] At step S1, the sensing control unit 202 executes an accuracy selection process. By the accuracy selection process, a sensing algorithm for ensuring the accuracy of person sensing is selected. The accuracy selection process will be described later in detail with reference to the flowchart in FIG. 8. Figure 24

[0259] At step S2, the sensing control unit 202 executes a person sensing process. The person sensing process is a process corresponding to the use case described above. The person sensing process will be described later in detail with reference to the flowchart in FIG. 9. Figure 27

[0260] Next, the accuracy selection process executed at step S1 in FIG. 10 will be described with reference to the flowchart in FIG. 8. Figure 24 Figure 23 At step S11, the situation detection unit 201 detects the place of the transport robot 2-5 based on sensor data from the sensor group 124 or sensor data output from each sensor forming the sensor device 21. Both the sensor data from the sensor group 124 and the sensor data output from each sensor forming the sensor device 21 can be used to detect the situation of the transport robot 2-5.

[0261] At step S12, the sensing control unit 202 determines whether the transport robot 2-5 is located indoors based on the detection result of the situation detection unit 201.

[0262] In a case where it is determined at step S12 that the transport robot 2-5 is located indoors, the sensing control unit 202 selects an indoor basic algorithm at step S13 and executes person sensing.

[0263] The indoor basic algorithm is a sensing algorithm that adjusts the imaging parameters such as shutter speed and sensitivity of the RGB camera 32A according to the intensity of ambient light and executes person sensing. The shutter speed is set to a level of a lower speed, and the sensitivity is set to a level of a higher sensitivity.

[0264] At step S14, the sensing control unit 202 executes an indoor process. In the indoor process, a sensing algorithm is selected according to the indoor situation, and person sensing is executed. The sensing algorithm for person sensing is appropriately switched from the indoor basic algorithm to another sensing algorithm. The indoor process will be described later in detail with reference to the flowchart in FIG. 11.

[0265] Figure 25

[0266] ​​​​​In contrast, in a case where it is determined at step S12 that the transport robot 2-5 is not located in the room, i.e., is located outdoors, the sensing control unit 202 selects an outdoor basic algorithm and executes person sensing at step S15.

[0267] The outdoor basic algorithm is a sensing algorithm that adjusts the imaging parameters of the RGB camera 32A, such as shutter speed and sensitivity, according to the intensity of the ambient light and executes person sensing. The shutter speed is set to a higher speed level, and the sensitivity is set to a lower sensitivity level.

[0268] At step S16, the sensing control unit 202 executes outdoor processing. In the outdoor processing, a sensing algorithm is selected according to the outdoor situation, and person sensing is executed. The sensing algorithm used for person sensing is appropriately switched from the outdoor basic algorithm to another sensing algorithm. The outdoor processing is described in detail later with reference to the flowchart in Figure 26 .

[0269] After the indoor processing at step S14 is executed or after the outdoor processing at step S16 is executed, the process returns to step S1 in Figure 23 , and the subsequent processing is repeated.

[0270] Next, the indoor processing executed at step S14 in Figure 25 is described with reference to the flowchart in Figure 24 .

[0271] At step S21, the sensing control unit 202 determines whether the transport robot 2-5 is located in a dark place based on the detection result of the situation detection unit 201.

[0272] In a case where it is determined at step S21 that the transport robot 2-5 is located in a dark place, the sensing control unit 202 selects an algorithm for a dark place according to person sensing conditions for executing person sensing in a dark place at step S22, and executes person sensing. The algorithm for a dark place is, for example, a sensing algorithm that images an RGB image with the sensitivity of the RGB camera 32A set to be higher than a standard sensitivity, and executes person sensing based on the RGB image acquired by the imaging.

[0273] In a case where it is determined at step S21 that the transport robot 2-5 is not located in a dark place, the sensing control unit 202 selects an algorithm for a bright place according to person sensing conditions for executing person sensing in a bright place at step S23, and executes person sensing. The algorithm for a bright place is, for example, a sensing algorithm that images an RGB image with the sensitivity of the RGB camera 32A set to be lower than a standard sensitivity, and executes person sensing based on the RGB image acquired by the imaging.

[0274] After the person sensing is executed using the sensing algorithm selected in the place of the transport robot 2-5, the process returns to Figure 24 step S14 in FIG. 14, and the subsequent processing is executed.

[0275] Next, the outdoor processing executed at step S16 in FIG. 14 is described with reference to the flowchart in Figure 26 Figure 24

[0276] At step S31, the sensing control unit 202 determines whether the weather in the place where the transport robot 2-5 is located is sunny. Whether the weather is sunny is determined on the basis of the situation detection result of the situation detection unit 201.

[0277] In the case where it is determined at step S31 that the weather in the place where the transport robot 2-5 is located is sunny, at step S32, the sensing control unit 202 determines whether this is a place where a shadow is likely to occur.

[0278] In the case where it is determined at step S32 that this is a place where a shadow is likely to occur, at step S33, the sensing control unit 202 selects an algorithm that is resistant to shadow noise in accordance with the person sensing condition in which person sensing is executed in a place where a shadow is likely to occur. The sensing algorithm selected here is a sensing algorithm in which imaging is performed while the imaging parameters of the RGB camera 32A are adjusted so as to expand the dynamic range of luminance, and person sensing is executed on the basis of the RGB image acquired through the imaging.

[0279] After the sensing algorithm is selected, the sensing program that defines the selected sensing algorithm is executed, and person sensing is executed. The same applies in the case where another sensing algorithm is selected.

[0280] In the case where it is determined at step S32 that this is not a place where a shadow is likely to occur, at step S34, the sensing control unit 202 selects an algorithm that is resistant to direct sunlight in accordance with the person sensing condition in which person sensing is executed under direct sunlight. The sensing algorithm selected here is a sensing algorithm in which imaging is performed while the imaging parameters of the RGB camera 32A are adjusted so as to increase the shutter speed and reduce the sensitivity, and person sensing is executed on the basis of the RGB image acquired through the imaging.

[0281] In contrast, in the case where it is determined at step S31 that the weather is not sunny, the process branches to step S35.

[0282] At step S35, the sensing control unit 202 determines whether it is raining. Whether it is raining is determined on the basis of the situation detection result of the situation detection unit 201.

[0283] ​​If it is determined at step S35 that it is raining, then at step S36, the sensing control unit 202 selects a rain-resistant noise-resistant algorithm based on the human sensing conditions for performing human sensing in a rainy location. The sensing algorithm selected here is the following: performing image processing on the RGB image captured by the RGB camera 32A to remove noise, and then performing human sensing based on the RGB image acquired after noise removal.

[0284] Known techniques are used for noise removal. Techniques for noise removal are disclosed, for example, in "https: / / digibibo.com / blog-entry-3422.html and http: / / www.robot.tu-tokyo.ac.jp / ~yamashita / paper / A / A025Final.pdf".

[0285] If it is determined at step S35 that there is no rain, then at step S37, the sensing control unit 202 selects an algorithm suitable for the dark environment based on the human sensing conditions for performing human sensing in a dark environment. The sensing algorithm selected here is the following: imaging is performed while adjusting the imaging parameters of the RGB camera 32A to reduce the shutter speed and increase the sensitivity, and human sensing is performed based on the RGB image acquired through imaging.

[0286] After performing human sensing using the sensing algorithm selected based on locations 2-5 of the conveyor robot, the process returns to... Figure 24 Step S16 in the process, and then perform the subsequent processing.

[0287] Next, refer to Figure 27 The flowchart description in Figure 23 The human sensing process is performed at step S2.

[0288] Here, as described in Example 2, the processing of the delivery robots 2-5 that receive orders is described. Figure 27 For example, the processing in the process is performed after the conveyor robots 2-5 move to the vicinity of the target from which they received the order.

[0289] In step S51, the sensing control unit 202 determines whether to perform individual identification when receiving an order.

[0290] If it is determined in step S51 that individual identification will be performed, in step S52, the sensing control unit 202 performs human sensing using an individual identification algorithm based on the human sensing conditions for performing individual identification.

[0291] If it is determined in step S51 that individual identification will not be performed, in step S53, the sensing control unit 202 performs human sensing using an attribute identification algorithm based on the human sensing conditions for performing attribute identification.

[0292] After performing human sensing using an algorithm selected based on whether individual identification is performed, the process proceeds to step S54.

[0293] In step S54, the controller 121 (e.g., the surrounding state identification unit 157) determines, based on the human sensing results, whether the person specified by the human sensing is a customer visiting the store for the first time.

[0294] If it is determined in step S54 that the customer is a first-time visitor to the store, the controller 121 performs services for the first-time customer in step S55.

[0295] In step S56, the controller 121 receives the order based on the individual identification result or attribute recognition result. For example, if the person specified by the human sensor is female, it confirms whether to reduce the quantity.

[0296] In contrast, if it is determined at step S54 that the customer is not a first-time visitor to the store, at step S57, controller 121 performs the service corresponding to the number of store visits. After performing customer service at step S56 or step S57, the process proceeds to step S58.

[0297] In step S58, when using an individual recognition algorithm, the sensing control unit 202 switches the sensing algorithm used for human sensing to an attribute recognition algorithm. Afterward, the process returns to... Figure 23 Step S2 in the above process is repeated.

[0298] Next, refer to Figure 28 The flowchart description in Figure 23 Step S2 in the process involves another person sensing process.

[0299] Here, as described in Example 3, the handling of the food delivery robots 2-5 is described. Figure 28 The processing, for example, is performed after the delivery robots 2-5 move to the vicinity of the target (orderer) to whom the dishes are served.

[0300] In step S71, the sensing control unit 202 of the sensor device 21 performs human sensing using an attribute recognition algorithm based on the human sensing conditions for performing attribute recognition.

[0301] In step S72, the controller 121 determines, based on the human sensing results of the sensor device 21, whether there are any customers who should be given priority in supply.

[0302] If it is determined at step S72 that there is a customer who should be given priority in supply, then at step S73, the sensing control unit 202 searches for the customer who should be given priority in supply.

[0303] In contrast, in the case where it is determined at step S72 that there is no orderer to be preferentially served, at step S74, the sensing control unit 202 searches for the nearest orderer.

[0304] After the order is searched at step S73 or step S74, the process shifts to step S75.

[0305] At step S75, the controller 121 (for example, the movement control unit 153) moves based on the human sensing result of the sensor device 21. For example, the delivery robot 2-5 is moved so as to approach the orderer found by the search.

[0306] At step S76, the controller 121 determines whether the serving direction is changed based on the human sensing result of the sensor device 21. For example, in the case where the delivery robot 2-5 is located at a position on the side of the hand opposite to the dominant hand of the orderer, it is determined that the serving direction is changed. In contrast, in the case where the delivery robot 2-5 is located at a position on the side of the dominant hand of the orderer, it is determined that the serving direction is not changed.

[0307] In the case where it is determined at step S76 that the serving direction is changed, at step S77, the controller 121 causes the delivery robot 2-5 to move so as to change the serving direction. In the case where it is determined at step S76 that the serving direction is not changed, the processing at step S77 is skipped.

[0308] At step S78, for example, the posture control unit 155 of the controller 121 controls the posture of the delivery robot 2-5, adjusts the height of the top plate 102 to the height of the table used by the customer, and serves the dishes to the orderer.

[0309] At step S79, the sensing control unit 202 determines whether all the dishes are served.

[0310] In the case where it is determined at step S79 that there is a dish that has not been served, the process returns to step S72, and the above-described processing is repeated.

[0311] In contrast, in the case where it is determined at step S79 that all the dishes are served, at step S80, the sensing control unit 202 performs human sensing using the attribute recognition algorithm. Thereafter, the process returns to step S2 in Figure 23 , and the above-described processing is repeated.

[0312] Through the above-described processing, the delivery robot 2-5 can select a sensing algorithm according to the situation, such as where it is located, and perform human sensing. Further, the delivery robot 2-5 can control various operations, such as serving, based on the result of human sensing performed using the sensing algorithm selected according to the situation.

[0313] <Other use examples>

[0314] Other use examples of the person sensing by the delivery robot 2-5 are described.

[0315] • Use example 4

[0316] Use example 4 is a use example of supplying a beverage.

[0317] In use example 4, the attribute of the customer is recognized using the attribute recognition algorithm A2. Based on the attribute recognition result, a process of determining a beverage to be supplied is executed, for example, by the delivery robot 2-5. For example, in a case where the attribute of the person to be supplied is a child, a beverage other than an alcoholic beverage is supplied.

[0318] • Use example 5

[0319] Use example 5 is a use example of providing a sweet such as a chewing gum and a candy according to a dish ordered.

[0320] In use example 5, the attribute of the customer is recognized using the attribute recognition algorithm A2. Based on the attribute recognition result, a process of managing the attribute of the orderer and the dish ordered by the person in association with each other is executed by the delivery robot 2-5, for example. In a case where a person leaves the store after finishing a meal, a sweet is provided according to the dish ordered.

[0321] Although the use examples of the person sensing by the delivery robot 2-5 are described above, various use examples are similarly assumed for the person sensing by other devices.

[0322] Here, use cases of the person sensing by the cooking robot 2-4 are described.

[0323] • Use example 6

[0324] Use example 6 is a use example when food materials are arranged. In use example 6, there are a case of using the individual recognition algorithm Al and a case of using the attribute recognition algorithm A2.

[0325] Note that the arrangement of the food materials by the cooking robot 2-4 is executed by driving the cooking arm to arrange the cooked food materials at predetermined positions of the tableware. The position of the cooked food materials, the position of the tableware, and the like are recognized based on the result of the object sensing by the sensor device 21 or the like and the arrangement is executed.

[0326] In the case of using the attribute recognition algorithm A2, the attribute of the person who orders the dish scheduled is recognized using the attribute recognition algorithm A2.

[0327] The arrangement method is changed based on the attribute recognition result: for example, in a case where the target of the order is a woman, tableware for women is used, and in a case where the target is a child, an arrangement for children is executed.

[0328] In the case of using the individual recognition algorithm Al, who the target is is recognized together with attributes using the individual recognition algorithm Al.

[0329] Based on the individual recognition result, for example, the cooking robot 2-4 performs processing of changing the arrangement or changing the food material to be used according to the preference of the person.

[0330] • Use example 7

[0331] Use example 7 is a use example when a menu is determined. Cooking performed by the cooking robot 2-4 is performed according to cooking data corresponding to the menu. The cooking data includes information defining the contents and order of the cooking operation of the cooking arm in each cooking process before the dish is completed.

[0332] In use example 7, who the target is is recognized together with attributes using the individual recognition algorithm Al.

[0333] For example, based on the individual recognition result, the history of the meal content of the target is specified, and a menu considering the nutritional balance is determined.

[0334] • Use example 8

[0335] Use example 8 is a use example when cooking at home. In use example 8, who the target is is recognized together with attributes using the individual recognition algorithm Al.

[0336] For example, in a case where only family members are specified based on the individual recognition result, cooking is first performed using older food materials. In addition, processing of changing the cooking content (the strength of the taste, the degree of baking, the degree of boiling, etc.) is performed according to the health condition of the family members.

[0337] • Use example 9

[0338] Use example 9 is a use example when washing hands. A recess for washing hands is prepared at a predetermined position of the top plate of the cooking robot 2-4. The recess is provided with a configuration for spraying water or a washing liquid toward the hands.

[0339] In use example 9, there are a case of using the individual recognition algorithm Al and a case of using the attribute recognition algorithm A2.

[0340] In the case of using the attribute recognition algorithm A2, the attribute of the person washing hands is recognized using the attribute recognition algorithm A2.

[0341] Based on the attribute recognition result, for example, in a case where the person washing hands is a woman or a child, the hands are washed with water having a suppressing strength.

[0342] In the case of using the individual recognition algorithm Al, who the target is is recognized together with attributes using the individual recognition algorithm Al.

[0343] Based on the individual recognition result, for example, the strength of cleaning and the type of cleaning liquid are changed in accordance with the preference of the person.

[0344] • Use example 10

[0345] Use example 10 is a use example when the cooking robot 2-4 is used. In use example 10, the attribute of the target is recognized using the attribute recognition algorithm A2.

[0346] Based on the attribute recognition result, for example, control is performed by the cooking robot 2-4 so as to prevent use by a child.

[0347] <Variant>

[0348] • Application example of other systems

[0349] Figure 29 is a view showing an application example of person sensing.

[0350] In a case where person sensing is performed in each of a plurality of systems provided in one store, person sensing corresponding to the use example is defined for each system, as shown in Figure 29 .

[0351] Figure 29 The supply system shown in Figure 23 is a system including a delivery robot 2-5. In the supply system, person sensing corresponding to the use example is performed by the process described with reference to .

[0352] Figure 29 The store monitoring system shown in Figure 23 is a system that uses a sensor device 21 as a monitoring camera. The sensor device 21 is attached to each position in the store. Also in the store monitoring system, person sensing corresponding to the use example is performed by a process similar to the process described with reference to .

[0353] • Example of instance where the sensing algorithm is selected from the outside

[0354] The selection of the sensing algorithm corresponding to the person sensing condition is performed in the sensor device 21, but this can be performed by a device external to the device on which the sensor device 21 is installed.

[0355] Figure 30 is a view showing an example of control of the sensing algorithm.

[0356] In the example in Figure 30 , the selection of the sensing algorithm corresponding to the sensing condition is performed by the program management server 1 as an external device. In this case, Figure 22The configuration of the controller 31 in the above-described embodiment is implemented in the program management server 1. The program management server 1 is a data processing apparatus that controls a sensing program executed by the sensor device 21 installed on the transport robot 2-5.

[0357] As indicated by an arrow #1, sensor data for detecting a situation is transmitted from the transport robot 2-5 to the program management server 1, and execution of a sensing program is requested.

[0358] A situation detection unit 201 of the program management server 1 detects a situation of the transport robot 2-5 on the basis of the sensor data transmitted from the transport robot 2-5. Further, a human sensing condition corresponding to the situation of the transport robot 2-5 is determined by a sensing control unit 202, and a sensing algorithm is selected.

[0359] The sensing control unit 202 of the program management server 1 transmits a sensing program defining a sensing algorithm corresponding to the human sensing condition to the sensor device 21 installed on the transport robot 2-5, and causes it to execute the sensing program.

[0360] In this way, the sensing algorithm can be controlled by an apparatus external to the sensor device 21. For example, the controller 121 of the transport robot 2-5 on which the sensor device 21 is installed can form the external apparatus, and the sensing algorithm can be controlled by the controller 121.

[0361] The sensing program defining the sensing algorithm corresponding to the human sensing condition can be executed by the program management server 1 or the controller 121 as the external apparatus, and information indicating an execution result can be transmitted to the sensor device 21.

[0362] Figure 31 is a block diagram showing a configuration example of hardware of a computer that implements the program management server 1.

[0363] A central processing unit (CPU) 1001, a read only memory (ROM) 1002, and a random access memory (RAM) 1003 are connected to each other through a bus 1004.

[0364] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006 including a keyboard, a mouse, and the like, and an output unit 1007 including a display, a speaker, and the like are connected to the input / output interface 1005. Further, a storage unit 1008 including a hard disk, a non-volatile memory, and the like, a communication unit 1009 including a network interface, and the like, and a drive 1010 that drives a removable medium 1011 are connected to the input / output interface 1005.

[0365] The control of the sensing algorithm as described above is implemented by execution of a predetermined program by the CPU 1001.

[0366] • Examples of programs

[0367] The series of processes described above can be executed by hardware or can be executed by software. In the case where the series of processes is executed by software, a program forming the software is installed on a computer incorporated with a dedicated hardware, a general-purpose personal computer, or the like.

[0368] A program to be installed is recorded in a removable medium 1011 shown in FIG. 1, which includes an optical disk (Compact Disc-Read Only Memory (CD-ROM), Digital Versatile Disc (DVD), or the like), a semiconductor memory, or the like, to be provided. Further, this can be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed on the ROM 1002 and the storage unit 1008. Figure 31

[0369] Note that the program executed by the computer can be a program that executes processing in time series in the order described in this specification, or can be a program that executes processing in parallel or at a time when needed (for example, when a call is made).

[0370] Note that in this specification, a system is intended to mean a collection of a plurality of components (devices, modules (components), or the like), and it does not matter whether all the components are in the same housing. Thus, both a plurality of devices housed in different housings and connected via a network and one device in which a plurality of modules are housed in one housing are a system.

[0371] The effects described in this specification are merely illustrative; the effects are not limited thereto and there can be other effects.

[0372] Embodiments of the present technology are not limited to the above-described embodiments, and various modifications can be made without departing from the spirit of the present technology.

[0373] For example, the present technology can be configured as cloud computing in which a plurality of devices share one function via a network to process together.

[0374] Further, each step described in the flowchart above can be executed by one device, or in a shared manner by a plurality of devices.

[0375] Further, in the case where a plurality of processes are included in one step, the plurality of processes included in one step can be executed by one device or in a shared manner by a plurality of devices.

[0376] [List of reference numerals]

[0377] 1 program management server

[0378] 2-1 mobile terminal ​

[0379] 2-2 Arm-type robot

[0380] 2-3 Mobile body

[0381] 2-4 Cooking robot

[0382] 2-5 Conveyance robot

[0383] 21 Sensor device

[0384] 31 Controller

[0385] 32 Sensor group

[0386] 121 Controller

[0387] 124 Sensor group

[0388] 201 Condition detection unit

[0389] 202 Sensing control unit

Claims

1. A data processing apparatus, comprising: A condition detection unit is configured to detect the condition of the robot based on sensor data output from sensors mounted on the robot. as well as The sensing control unit is configured to: Based on the robot's condition, determine whether to execute an individual recognition algorithm or an attribute identification algorithm, and If the individual identification algorithm is to be executed, a first person sensing procedure is selected to be executed, which is used to perform individual identification based on sensor data output from the sensors mounted on the robot.

2. The data processing apparatus according to claim 1, wherein, The sensing control unit selects the first person sensing program obtained via the network.

3. The data processing apparatus according to claim 1, wherein, The sensing control unit selects the first person sensing program to be executed from a set of first person sensing programs that includes a combination of multiple first person sensing programs.

4. The data processing apparatus according to claim 3, wherein, The human sensing program set includes a combination of information indicating the type of human sensing algorithm defined in the first human sensing program and information indicating the execution order of the plurality of first human sensing programs.

5. The data processing apparatus according to claim 3, wherein, The sensing control unit selects the first human sensing program included in the human sensing program set obtained via the network.

6. The data processing apparatus according to claim 5, wherein, The sensing control unit selects the human sensing program set using identification data that identifies the human sensing program set.

7. The data processing apparatus according to claim 3, wherein, The human sensing algorithms defined in the multiple first-person sensing programs are algorithms applicable to the output sensor data when different parameters are set for the same sensor.

8. The data processing apparatus according to claim 3, wherein, The human sensing algorithms defined in the multiple first-person sensing programs are algorithms applicable to the output sensor data when the same parameters are set on the same sensor.

9. The data processing apparatus according to claim 3, wherein, The human sensing algorithms defined in the multiple first-person sensing programs are algorithms applicable to sensor data output from different sensors.

10. The data processing apparatus according to claim 9, wherein, At least one of the first person sensing program and the person sensing algorithm defined in the first person sensing program is associated with the sensor, and The sensing control unit controls the operation of multiple sensors by combining the selection and execution of the first person's sensing program.

11. The data processing apparatus according to claim 1, further comprising: A movement control unit is configured to control the movement state of the movement unit accompanying the transmission of the object based on the execution result of the sensing control unit on the first person sensing program.

12. The data processing apparatus according to claim 11, further comprising: The top plate on which the object to be transported is placed is placed; A retractable support unit that supports the top plate; as well as The movable unit connected to the support unit, wherein, The motion control unit controls the posture state of the top plate and the support unit, as well as the movement state of the motion unit, based on the execution result of the first person sensing program by the sensing control unit.

13. The data processing apparatus according to claim 12, wherein, The conveying object, placed by the cooking arm of the cooking system driven according to the cooking process or by a person, is placed on the top plate.

14. A data processing method, comprising: The following operations are performed using a data processing device: The robot's condition is detected based on sensor data output from sensors mounted on the robot. Based on the robot's condition, it is determined whether to execute an individual identification algorithm or an attribute recognition algorithm. If it is determined to execute the individual identification algorithm, a first person sensing procedure is selected to execute, which is used to perform individual identification based on sensor data output from the sensors installed on the robot.

15. A data processing apparatus, comprising: The data processing unit is configured as follows: The robot's condition is detected based on sensor data output from sensors mounted on the robot. Based on the robot's condition, determine whether to execute an individual recognition algorithm or an attribute identification algorithm, and If the individual identification algorithm is to be executed, a first person sensing program is selected and sent to the robot. The first person sensing program is used to perform individual identification based on sensor data output from the sensors mounted on the robot.

16. The data processing apparatus according to claim 15, wherein, The data processing unit sends the first human sensing program in response to a request from the robot.

17. The data processing apparatus according to claim 15, wherein, The data processing unit executes the first human sensing procedure in response to a request from the robot.

18. A data processing method, comprising: The following operations are performed using a data processing device: The robot's condition is detected based on sensor data output from sensors mounted on the robot. Based on the robot's condition, determine whether to execute an individual recognition algorithm or an attribute identification algorithm; If it is determined that the individual identification algorithm will be executed, a first person sensing program is selected, which is used to perform individual identification based on the sensor data output from the sensor installed on the robot; as well as The selected first-person sensing program is sent to the robot.

19. A robot comprising: A sensor, which is configured to output sensor data indicating the results of sensing; A condition detection unit is configured to detect the condition of the robot based on sensor data output from the sensor; The sensing control unit is configured to: Based on the robot's condition, determine whether to execute an individual recognition algorithm or an attribute identification algorithm, and If it is determined that the individual identification algorithm will be executed, a first person sensing procedure will be selected to be executed, which is used to perform individual identification based on the sensor data output from the sensor; An operation plan setting unit is configured to set an operation plan based on the execution result of the sensing control unit on the first person sensing program; as well as An operation unit is configured to perform operations according to the operation plan set by the operation plan setting unit.

Images