Data processing apparatus, data processing method, and mobile body

Abstract

The present technique relates to a data processing apparatus, a data processing method, and a robot with which it is possible to perform human sensing using an appropriate algorithm. The data processing apparatus according to an aspect of the present technique selects and executes a human sensing program adaptively in accordance with human sensing conditions, the human sensing program specifying a human sensing algorithm that performs human sensing on the basis of sensor data output from a sensor installed on a robot. The present technique is applicable to sensor devices mounted on various apparatuses.

Description

Technical Field

[0001] This technology particularly relates to a data processing device, a data processing method, which can perform human sensing by using an appropriate algorithm. Moving body It relates to.

Background Art

[0002] Conventionally, various technologies for updating the software of a device have been proposed from the viewpoints of adding functions and ensuring compatibility with other devices.

[0003] For example, Patent Document 1 discloses a technology for determining a service that can be realized by a combination of a camera and a communication device and installing software for providing the service.

[0004] Also, Patent Document 2 discloses a technology for updating the firmware between an imaging device and a host system when it is detected that the firmware on the imaging device is incompatible with the host system.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] Although various technologies for changing the software of an imaging device such as a camera have been disclosed, there is no disclosure about adaptively changing the software of a device that performs sensing according to the situation of the sensing target and the like.

[0007] This technology has been made in view of such a situation, and it enables human sensing by using an appropriate algorithm.

Means for Solving the Problem

[0008] The data processing device according to the first aspect of the present technology determines a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having a movable operation unit with respect to the moving body main body, according to human sensing conditions, Determined the human sensing algorithm The defined human sensing program is selected and executed from a set of human sensing programs that are combinations of a plurality of the human sensing programs and include a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, controls the movement of the moving body and the operation of the operation unit.

[0009] The data processing device according to the second aspect of the present technology determines a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having a movable operation unit with respect to the moving body main body, according to human sensing conditions, and the determined human sensing algorithm The defined human sensing program is selected from a set of human sensing programs that are combinations of a plurality of the human sensing programs and include a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, transmits to the moving body.

[0010] The third aspect of the present technology Moving body includes a sensor that outputs sensor data representing a sensing result, a sensing control unit that adaptively selects and executes a human sensing program in which a human sensing algorithm for sensing a person based on the sensor data output from the sensor is defined, according to human sensing conditions, an operation plan setting unit that sets an operation plan based on the execution result of the human sensing program by the sensing control unit, and an operation unit that operates according to the operation plan set by the operation plan setting unit The operating part is movable with respect to the moving body main body .

[0011] In the first aspect of the present technology, a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having a movable operation unit with respect to the moving body main body is determined according to human sensing conditions, Decided the human sensing algorithm The defined human sensing program is selected and executed from a set of human sensing programs that are combinations of a plurality of the human sensing programs and include a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program,The movement of the moving body and the operation of the operating unit are controlled.

[0012] In a second aspect of the present technology, a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having a movable operating unit with respect to the moving body main body is determined according to human sensing conditions, and the determined human sensing algorithm The defined human sensing program is selected from a set of human sensing programs that are combinations of a plurality of the human sensing programs and include a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, is transmitted to the moving body.

[0013] In a third aspect of the present technology, a human sensing program in which a human sensing algorithm for sensing a person based on sensor data output from a sensor that outputs sensor data representing a sensing result is defined is adaptively selected and executed according to human sensing conditions, and based on the execution result of the human sensing program, an operation plan is set, and according to the set operation plan Of the operating part that is movable with respect to the moving body main body operations are performed.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Figure 23

Figure 24

Figure 25

Figure 26

Figure 27

Figure 28

Figure 29

Figure 30

Figure 31

Embodiments for Carrying Out the Invention

[0015] <Summary of the Present Technology> The present technology focuses on the fact that it is difficult to perform sensing using an optimal algorithm in sensor devices mounted on various devices such as robots, mobile bodies, and smartphones, and realizes a solution method therefor.

[0016] Factors · There are many types of elemental technologies. · The maturity levels of elemental technologies vary. · There are many cost variations. · There are many application variations. · The difficulty of system design and implementation is high. · There are many types of user requirements. · There are many implementation constraints such as the power, power consumption, and circuit scale of the processor. · There are many types of sensing targets.

[0017] In particular, the present technology enables sensing to be performed using an optimal algorithm as a sensing algorithm related to sensing in a sensor device that performs human sensing.

[0018] Hereinafter, embodiments for carrying out the present technology will be described. The description will be made in the following order. 1. Program Provision System 2. Use Case of Sensing Program 3. Configuration and Operation of the Transport Robot 4. Other Use Cases 5. Variations

[0019] <Program Provision System> FIG. 1 is a diagram showing a configuration example of a program provision system according to an embodiment of the present technology.

[0020] The program provision system in FIG. 1 is configured by connecting various devices such as a portable terminal 2-1, an arm robot 2-2, a mobile body 2-3, a cooking robot 2-4, and a transport robot 2-5 to a program management server 1 via a network 11 composed of the Internet or the like.

[0021] The portable terminal 2-1 is a smartphone.

[0022] The arm robot 2-2 is a dual-arm robot. A cart is provided on the housing of the arm robot 2-2. The arm robot 2-2 is a movable robot.

[0023] The mobile body 2-3 is an automobile. The mobile body 2-3 is equipped with functions such as an automatic driving function.

[0024] The cooking robot 2-4 is a kitchen-type robot. The cooking robot 2-4 has a function of performing cooking by driving a plurality of cooking arms. The cooking arms reproduce the same operations as those performed by a person.

[0025] The transport robot 2-5 is a robot that can place an object to be transported on a top plate prepared as a mounting table and move to a target position in that state. Wheels are provided on the base portion of the transport robot 2-5.

[0026] Each device shown in FIG. 1 is equipped with a sensor device used for environmental sensing, object sensing, human sensing, etc.

[0027] FIG. 2 is a diagram showing a configuration example of a sensor device.

[0028] As shown in FIG. 2, a controller 31 and a sensor group 32 are provided in the sensor device 21.

[0029] The controller 31 controls each sensor constituting the sensor group 32 to perform sensing of various targets such as environmental sensing, object sensing, and human sensing. The sensing by the controller 31 is performed based on the sensor data output by each sensor constituting the sensor group 32.

[0030] The controller 31 outputs the sensing result to the device on the host side. Based on the sensing result by the controller 31, various processes are performed on the device on the host side. When the sensor device 21 is mounted on the mobile terminal 2-1, the CPU (Central Processing Unit) of the mobile terminal 2-1 serves as the device on the host side. The controller 31 is also provided with a function for communicating with the device on the host side.

[0031] The sensor group 32 is composed of a plurality of sensors that perform sensing of various targets. In the example of FIG. 2, the sensor group 32 is composed of an RGB camera 32A, a stereo camera 32B, a ToF sensor 32C, a structured-light sensor 32D, and a LiDAR 32E.

[0032] The RGB camera 32A has an image sensor for RGB images. The RGB camera 32A drives the image sensor to photograph the surrounding situation and outputs the obtained RGB image as sensor data.

[0033] The stereo camera 32B is a distance sensor using the stereo camera method and has two image sensors for distance images. The stereo camera 32B outputs a distance image representing the distance to the object as sensor data.

[0034] The ToF sensor 32C is a distance sensor using the ToF (Time Of Flight) method. The ToF sensor 32C measures the distance to an object using the ToF method and outputs the distance information as sensor data.

[0035] The structured-light sensor 32D is a distance sensor using the structured-light method. The structured-light sensor 32D measures the distance to an object using the structured-light method and outputs the distance information as sensor data.

[0036] LiDAR (Light Detection and Ranging) 32E measures the three-dimensional position of each point of an object and outputs the information representing the measurement result as sensor data.

[0037] Sensors different from the sensors shown in FIG. 2, such as a positioning sensor, a gyro sensor, an acceleration sensor, a temperature sensor, and an illuminance sensor, may be included in the sensor group 32.

[0038] The types of sensors constituting the sensor group 32 are appropriately changed by the device on which the sensor device 21 is mounted. The sensor group 32 may be constituted by one sensor.

[0039] The sensor device 21 may be constituted by a substrate on which the controller 31 and the sensor group 32 are arranged, or may be constituted as a device in which a substrate on which each sensor is arranged is housed in the housing 21A as shown in FIG. 3.

[0040] In the sensor device 21 having such a configuration, the controller 31 executes a sensing program, which is a program for sensing, and realizes the sensing functions for various objects such as the environment, objects, and people. The sensing function of the controller 31 is realized based on the output of one sensor constituting the sensor group 32 or based on a combination of the outputs of a plurality of sensors.

[0041] Environmental sensing (sensing of the environment) includes, for example, the following. · Shooting of RGB images using the RGB camera 32A · Measurement of the distance to an object using the outputs of the stereo camera 32B, the ToF sensor 32C, and the structured-light sensor 32D · Generation of a three-dimensional map using the output of the LiDAR 32E · Estimation of the self-position using the three-dimensional map

[0042] The environment to be sensed by the sensor device 21 includes various physical states outside the sensor device 21 or outside the device on which the sensor device 21 is mounted, which can be expressed as quantitative data by sensing.

[0043] Object sensing (sensing of an object) includes, for example, the following. · Recognition and identification of an object using the RGB image captured by the RGB camera 32A · Measurement of the characteristics of an object, such as shape, size, color, and temperature

[0044] The objects to be sensed by the sensor device 21 include various stationary and moving objects around the sensor device 21 or around the device on which the sensor device 21 is mounted.

[0045] Human sensing (sensing of a person) includes, for example, the following. · Recognition of a person, recognition of a person's face, and identification of a person using the RGB image captured by the RGB camera 32A · Recognition of specific parts of a person, such as the head, arms, hands, eyes, and nose · Estimation of the position of a specific part, including bone estimation · Estimation of physical characteristics of a person, such as height and weight · Estimation of a person's attributes, such as age and gender

[0046] The person to be sensed by the sensor device 21 includes a person around the sensor device 21 or around the device on which the sensor device 21 is mounted.

[0047] The controller 31 has a plurality of programs with different algorithms as sensing programs for realizing respective sensing functions.

[0048] FIG. 4 is a diagram showing an example of a sensing program prepared for the sensor device 21.

[0049] In the example of FIG. 4, a distance measurement program A, a distance measurement program B, and a distance measurement program C are prepared as firmware that operates on an OS (Operating System). The distance measurement program A, the distance measurement program B, and the distance measurement program C are sensing programs that realize a distance measurement function as a human sensing function.

[0050] The distance measurement program A, the distance measurement program B, and the distance measurement program C are sensing programs that realize the same distance measurement function by different sensing algorithms. The distance measurement program A, the distance measurement program B, and the distance measurement program C define different sensing algorithms.

[0051] The distance measurement program A is a sensing program that performs distance measurement by the distance measurement algorithm A. The distance measurement program B is a sensing program that performs distance measurement by the distance measurement algorithm B. The distance measurement program C is a sensing program that performs distance measurement by the distance measurement algorithm C.

[0052] For example, the distance measurement algorithms A to C are sensing algorithms that perform distance measurement using different parameters, such as setting different parameters for the same sensor and calculating the distance by performing the same calculation based on the output of the sensor.

[0053] Furthermore, ranging algorithms A to C are sensing algorithms that perform ranging using different calculation methods, such as setting the same parameters for the same sensor and calculating different distances based on the output of the sensor.

[0054] When a plurality of distance sensors such as stereo camera 32B, ToF sensor 32C, and structured-light sensor 32D are provided, ranging algorithms A to C may be sensing algorithms that perform ranging using different distance sensors.

[0055] In this case, for example, ranging algorithm A performs ranging based on the output of stereo camera 32B, ranging algorithm B performs ranging based on the output of ToF sensor 32C, and ranging algorithm C performs ranging based on the output of structured-light sensor 32D.

[0056] In this way, a plurality of programs with different sensing algorithms are provided in sensor device 21 as sensing programs for realizing the same ranging function. At least one of each sensing algorithm and the sensing program that defines each sensing algorithm is associated with a sensor used for ranging. When the sensing program is executed, the operation of the associated sensor is controlled in conjunction with that.

[0057] For example, in sensor device 21 mounted on transport robot 2-5, ranging is performed by selecting a sensing algorithm according to the sensing conditions. The sensing conditions are the conditions for selecting a sensing algorithm, which are determined according to the situation of transport robot 2-5.

[0058] For example, when a situation suitable for ranging algorithm A occurs, ranging program A is executed, and ranging is performed using ranging algorithm A. Also, when a situation suitable for ranging algorithm B occurs, ranging program B is executed, and ranging is performed using ranging algorithm B. When a situation suitable for ranging algorithm C occurs, ranging program C is executed, and ranging is performed using ranging algorithm C.

[0059] Since ranging is performed by adaptively selecting a sensing algorithm (sensing program) according to the sensing conditions, ranging can be performed with the optimal sensing algorithm. The same applies when the sensing target is other than distance.

[0060] One sensing program defines one sensing algorithm. Selecting a sensing program is equivalent to selecting a sensing algorithm.

[0061] Note that adaptively selecting a sensing algorithm means selecting the sensing algorithm associated with the sensing conditions when it is detected that the conditions have been met. For each assumed sensing condition corresponding to the situation, a sensing algorithm considered to be suitable is associated. The association between the sensing conditions and the sensing algorithm may be dynamically changed.

[0062] FIG. 5 is a diagram showing another example of a sensing program.

[0063] In the example of FIG. 5, food recognition program A, food recognition program B, and food recognition program C are prepared as firmware that operates on the OS. Food recognition programs A to C are sensing programs that realize a food recognition function as an object sensing function.

[0064] The food ingredient recognition program A is a sensing program that performs food ingredient recognition using the food ingredient recognition algorithm A. The food ingredient recognition program B is a sensing program that performs food ingredient recognition using the food ingredient recognition algorithm B. The food ingredient recognition program C is a sensing program that performs food ingredient recognition using the food ingredient recognition algorithm C. For example, in the sensor device 21 mounted on the cooking robot 2-4, food ingredient recognition is performed by selecting a sensing algorithm according to the sensing conditions determined by the cooking process or the like.

[0065] Figure 6 is a diagram showing yet another example of the sensing program.

[0066] In the example of Figure 6, the face recognition programs A, B, and C are provided as firmware that operates on the OS. The face recognition programs A to C are sensing programs that implement the face recognition function as a human sensing function.

[0067] The face recognition program A is a sensing program that performs face recognition using the face recognition algorithm A. The face recognition program B is a sensing program that performs face recognition using the face recognition algorithm B. The face recognition program C is a sensing program that performs face recognition using the face recognition algorithm C. For example, in the sensor device 21 mounted on the cooking robot 2-4, face recognition is performed by selecting a sensing algorithm according to the sensing conditions determined by the cooking process or the like.

[0068] · Update of the sensing program In the program providing system of Figure 1, it is possible to update the sensing program provided as firmware in the sensor device 21 of each device.

[0069] Figure 7 is a diagram showing an example of the update of the sensing program.

[0070] As shown by the arrow in FIG. 7, the program management server 1 provides a sensing program for each device. The program management server 1 has a DB (Data Base) of the sensing programs provided to each device.

[0071] In the example of FIG. 7, a ranging program D that performs ranging by the ranging algorithm D is provided to the mobile terminal 2-1, and a face recognition program H that performs face recognition by the face recognition algorithm H is provided to the arm robot 2-2.

[0072] Also, a self-position estimation program J that performs self-position estimation by the self-position estimation algorithm J is provided to the moving body 2-3, and an object recognition program K that performs object recognition by the object recognition algorithm K is provided to the cooking robot 2-4. A person recognition program M that performs person recognition by the person recognition algorithm M is provided to the transport robot 2-5.

[0073] FIG. 8 is a diagram showing an example of the update of the sensing program.

[0074] In the sensor device 21 of each device, as shown in A of FIG. 8, it is possible to add a sensing program. In the example of A in FIG. 8, a ranging program D that performs ranging by the ranging algorithm D is added to the ranging programs A to C that perform ranging by the ranging algorithms A to C.

[0075] In the sensor device 21 of each device, in the default state, a sensing program that defines a sensing algorithm according to a general situation is prepared. Even when the sensor device 21 of each device encounters a situation that cannot be handled by the pre-prepared sensing program, it can cope with special situations by adding a sensing program that defines a sensing algorithm according to such special situations.

[0076] Also, as shown in B of FIG. 8, it is also possible to delete (uninstall) an unnecessary program. In the example of B in FIG. 8, as shown by the dashed frame, the distance measurement program C among the distance measurement programs A to C has been deleted.

[0077] FIG. 9 is a diagram showing another example of the update of the sensing program.

[0078] As shown in FIG. 9, it is also possible to perform an update in units of a sensing program set composed of a plurality of sensing programs. In the example of FIG. 9, a sensing program set composed of a distance measurement program D that performs distance measurement by a distance measurement algorithm D, a distance measurement program E that performs distance measurement by a distance measurement algorithm E, and a distance measurement program F that performs distance measurement by a distance measurement algorithm F is provided and added from the program management server 1.

[0079] As shown in FIG. 10, a plurality of sensing program sets that group a plurality of sensing programs according to usage conditions such as location, situation, and purpose are prepared in the DB of the program management server 1.

[0080] In the example of FIG. 10, a sensing program set for indoor distance measurement and a sensing program set for outdoor distance measurement are prepared. These sensing program sets are sensing program sets according to location.

[0081] The sensing program set according to location becomes, for example, a set used in the sensor device 21 mounted on a device having a moving function. Even within the same indoor area, sensing program sets may be prepared in more detailed location units such as a sensing program set for the kitchen and a sensing program set for the dining room.

[0082] It is also possible to prepare sensing program sets for various locations, such as a sensing program set for marine use, a sensing program set for mountain use, and a sensing program set for use inside a train.

[0083] Also, in the example of FIG. 10, a sensing program set for distance measurement in sunny weather and a sensing program set for distance measurement in rainy weather are prepared. These sensing program sets are sensing program sets according to the weather.

[0084] The sensing program set according to the weather is, for example, a set used in the sensor device 21 mounted on a device that has a moving function and may move outdoors. It is also possible to prepare sensing program sets for various changing situations, such as a sensing program set for each time zone like morning, day, and night, a sensing program set for each brightness level, and a sensing program set for each temperature.

[0085] It is also possible to prepare sensing program sets for various purposes, such as a sensing program set for running, a sensing program set for baseball, a sensing program set for cooking curry, and a sensing program set for cooking salad.

[0086] The sensor device 21 of each device can collectively add sensing programs by specifying the ID of the sensing program set according to the usage conditions. An ID is set as identification data for each sensing program set. An ID is also set as identification data for each sensing program that constitutes the sensing program set.

[0087] Instead of a set of sensing programs that realize the same distance measurement function by different sensing algorithms, as shown in FIG. 11, a set of sensing programs that realize different functions may be added.

[0088] In the example of FIG. 11, a sensing program set is constituted by a distance measurement program D, a face recognition program H, and an object recognition program K. The distance measurement program D is a sensing program that performs distance measurement by a distance measurement algorithm D, and the face recognition program H is a sensing program that performs face recognition by a face recognition algorithm H. The object recognition program K is a sensing program that performs object recognition by an object recognition algorithm K.

[0089] FIG. 12 is a diagram showing an example of a sensing program set.

[0090] The sensing program set shown in FIG. 12 includes an algorithm manager that is a program for controlling an adaptive selection of algorithms.

[0091] The sensor device 21 executes the algorithm manager and selects a sensing algorithm according to the sensing conditions. In the algorithm manager, a combination of information indicating the type of the sensing program for controlling the execution and information indicating the execution order of the sensing program is set. In this way, an algorithm manager may be prepared for each sensing program set.

[0092] FIG. 13 is a diagram showing an example of the update of a sensing program.

[0093] The sensing program may be executed in each of the sensor device 21 and the controller 51 which is a host-side device, so that a predetermined function such as a distance measurement function is realized. In this case, the sensing program of the controller 51 can also be updated in the same manner as the sensing program of the sensor device 21. The controller 51 is a host-side data processing device such as the CPU of the mobile terminal 2-1 or the CPU of the PC mounted on the arm robot 2-2.

[0094] A sensing program for updating the firmware of the sensor device 21 and a sensing program for updating the firmware of the controller 51 may be included in one sensing program set and provided.

[0095] The provision of the sensing program and the sensing program set may be provided for a fee or free of charge. In one sensing program set, a paid sensing program and a free sensing program may be mixed and included.

[0096] When updating the sensing program performed as described above, the authentication of the sensor device 21 is performed by the program management server 1 based on the key information for authentication, and the update may be performed when it is confirmed that it is a legitimate device. For each sensor device 21, key information for authentication is prepared as unique information.

[0097] The authentication of the sensor device 21 using the key information for authentication may be performed not at the time of updating the sensing program but at the time of executing the sensing program.

[0098] · Provider of the sensing program FIG. 14 is a diagram showing an example of the provider of the sensing program.

[0099] The sensing program provided from the program management server 1 to each device is developed, for example, by a developer who has performed user registration of a service in the program providing system as shown in FIG. 14. For each developer, information regarding the specifications of the sensor device 21 and development tools such as an SDK (Software Development Kit) are provided by a service provider who operates a service using the program providing system.

[0100] Each developer develops a sensing program or a set of sensing programs by using the SDK or the like, and uploads them from their own computer to the program management server 1. The uploaded sensing programs and sets of sensing programs are stored and managed in the sensing program DB.

[0101] The program management server 1 manages the usage status of each sensing program and set of sensing programs, such as the number of installations and the number of executions on each device. The service provider may provide predetermined incentives to the developers, such as payment of an amount according to the usage status and issuance of points.

[0102] FIG. 15 is a diagram showing an example of the generation of a set of sensing programs.

[0103] A set of sensing programs may be generated by any user collecting the sensing programs developed and uploaded by each developer.

[0104] In the example of FIG. 15, among the ranging programs A to G, an indoor ranging program set is generated by collecting three sensing programs: ranging program D, ranging program E, and ranging program F.

[0105] The indoor ranging program set generated in this way is published in the program management server 1 as an installable set of sensing programs and installed on a predetermined device.

[0106] Incentives may also be provided to the user who generates a set of sensing programs by collecting a plurality of sensing programs.

[0107] <Use case of sensing program> · Use case of transport robot Here, the use cases of people sensing will be described.

[0108] For example, when the transport robot 2-5 transports dishes as objects to be transported inside a restaurant or the like, that is, when serving dishes, in the sensor device 21 mounted on the transport robot 2-5, people sensing is performed according to the sensing program. In order to serve the ordered person with the dish, it is necessary to recognize the people around and identify the person who ordered the dish.

[0109] FIG. 16 is a diagram showing the state of transportation by the transport robot 2-5.

[0110] FIG. 16 shows the state of the transport robot 2-5 moving in the kitchen inside the building. Cooked dishes are placed on the top plate prepared as a mounting table for the object to be transported. In this example, the transport robot 2-5 is used for the purpose of serving dishes.

[0111] Based on the result of people sensing by the sensor device 21, the transport robot 2-5 plans the movement route, avoids obstacles, etc., moves to the destination, and serves the dishes. Also, based on the result of people sensing by the sensor device 21, the transport robot 2-5 controls the way of customer service.

[0112] FIG. 17 is a diagram showing an enlarged view of the appearance of the transport robot 2-5.

[0113] As shown in FIG. 17, the transport robot 2-5 is composed of connecting an annular base portion 101 and a circular thin plate-shaped top plate 102 with a thin rod-shaped support arm 103. A plurality of tires are provided on the bottom surface side of the base portion 101. The base portion 101 functions as a moving portion for realizing the movement of the transport robot 2-5.

[0114] The radial length of the base portion 101 and the radial length of the top plate 102 are substantially the same length. When the top plate 102 is substantially directly above the base portion 101, as shown in FIG. 17, the support arm 103 is in an inclined state.

[0115] The support arm 103 is composed of an arm member 103-1 and an arm member 103-2. The diameter of the arm member 103-1 on the top plate 102 side is slightly smaller than the diameter of the arm member 103-2 on the base portion 101 side. By the arm member 103-1 being housed in the telescopic portion 103A inside the arm member 103-2, as shown by the double-headed arrow, the length of the support arm 103 is adjusted.

[0116] The support arm 103 can adjust the angle at each of the connecting portion between the base portion 101 and the support arm 103, and the connecting portion between the top plate 102 and the support arm 103.

[0117] FIG. 18 is a diagram showing an example of the posture of the transport robot 2-5 when placing a dish.

[0118] In the example of FIG. 18, by making the support arm 103 substantially vertical and setting the length to the maximum length, the height of the top plate 102 is adjusted to be substantially the same height as the top plate of the cooking robot 2-4.

[0119] When the transport robot 2-5 is in such a state, the dish is placed on the top plate 102 by the cooking arm of the cooking robot 2-4. In the example of FIG. 18, the dish completed by the cooking operation of the cooking robot 2-4 is placed by the cooking arm.

[0120] As shown in FIG. 18, the cooking robot 2-4 is provided with a plurality of cooking arms that perform various cooking operations such as cutting ingredients, grilling ingredients, and plating cooked ingredients. The cooking operations by the cooking arms are performed according to cooking data that defines the content and order of the cooking operations. The cooking data includes information on each cooking process until the completion of the dish.

[0121] In this way, the dish served by the transport robot 2-5 is a dish made by the cooking robot 2-4. Dishes made by a person may be placed on the top plate 102 by a person and served.

[0122] Figure 19 is a plan view showing the layout of the space where the transport robot 2-5 moves.

[0123] As shown in Figure 19, in the restaurant where the transport robot 2-5 moves, Kitchen #1 and Hall #2 are prepared. There is a corridor #11 between Kitchen #1 and Hall #2.

[0124] Outside the building in Figure 19, the range of which is indicated by a dashed line, a garden #21 is provided facing Hall #2. Tables and the like for customers to have meals are prepared not only in Hall #2 but also in garden #21.

[0125] The case where the transport robot 2-5 moves in such a space and provides customer service will be described. The customer service performed by the transport robot 2-5 includes taking orders, serving dishes, providing drinks, and the like.

[0126] Human sensing by the sensor device 21 mounted on the transport robot 2-5 is performed using an algorithm according to the human sensing conditions set according to the situation of the transport robot 2-5, such as the location where the transport robot 2-5 is. A sensing program for performing human sensing is adaptively selected according to the human sensing conditions and executed in the sensor device 21.

[0127] ·Specific examples of sensing algorithms Figure 20 is a diagram showing an example of a sensing algorithm defined by the sensing program prepared for the transport robot 2-5.

[0128] As shown in Figure 20, for the transport robot 2-5, a personal identification algorithm A1 and an attribute recognition algorithm A2 are prepared as sensing algorithms for human sensing.

[0129] The personal identification algorithm A1 is a sensing algorithm used to identify who the person being sensed is and to recognize the attributes of the person. The attributes of the person include gender and age. In addition, the attributes of the person also include features that appear in the appearance, such as the dominant hand and the length of the hair.

[0130] On the other hand, the attribute recognition algorithm A2 is a sensing algorithm used to recognize the attributes of the person. Depending on the attribute recognition algorithm A2, the person may not be identified, and only the recognition of the attributes of the person is performed.

[0131] In addition, for each of the personal identification algorithm A1 and the attribute recognition algorithm A2, sensing algorithms for each situation, such as an indoor sensing algorithm, an outdoor sensing algorithm, a sensing algorithm for use in the dark, etc., are prepared.

[0132] In FIG. 20, for example, the personal identification algorithm A1-1 is an indoor sensing algorithm, and the personal identification algorithm A1-2 is an outdoor sensing algorithm. In addition, the personal identification algorithm A1-3 is a sensing algorithm for use in the dark.

[0133] Similarly, the attribute recognition algorithm A2-1 is an indoor sensing algorithm, and the attribute recognition algorithm A2-2 is an outdoor sensing algorithm. In addition, the attribute recognition algorithm A2-3 is a sensing algorithm for use in the dark.

[0134] In this way, for the transport robot 2-5, a sensing program that defines each personal identification algorithm A1 according to the human sensing conditions set according to the situation, and a sensing program that defines each attribute recognition algorithm A2 are prepared. In the transport robot 2-5, the personal identification algorithm A1 or the attribute recognition algorithm A2 according to the human sensing conditions is selected, and human sensing is performed.

[0135] The personal identification algorithm A1 and the attribute recognition algorithm A2 are prepared because the required sensing algorithms differ depending on the purpose, such as whether it is necessary to identify who the target person is along with the attributes or only to identify the attributes.

[0136] In addition, for each of the personal identification algorithm A1 and the attribute recognition algorithm A2, the situation-specific sensing algorithms are prepared because it is necessary to change the processing to ensure accuracy depending on the situation such as the location where the target person is.

[0137] For example, when the place where human sensing is performed is a dark place, a sensing algorithm that is strong against dark places (noise) is required.

[0138] Also, when the place where human sensing is performed is outdoors, a sensing algorithm that is strong against direct sunlight is required. Since the noise varies depending on the weather such as rain or cloudy, sensing algorithms corresponding to each weather are required.

[0139] Furthermore, since the illuminance changes due to the passage of time and sudden changes in weather, sensing algorithms corresponding to each illuminance are required.

[0140] Here, the switching for each use case of the personal identification algorithm A1 and the attribute recognition algorithm A2 will be described.

[0141] · Use Case 1 Use Case 1 is a use case in which the transport robot 2-5 confirms the entry of a customer in the restaurant in FIG. 19.

[0142] In Use Case 1, the attributes of the customer are recognized using the attribute recognition algorithm A2. Based on the recognition result of the attributes, for example, the process of determining the seat to guide is performed by the transport robot 2-5.

[0143] · Use Case 2 Use case 2 is a use case that confirms the quantity and cooking preferences when receiving an order. In use case 2, there are cases where the individual identification algorithm A1 is used and cases where the attribute recognition algorithm A2 is used.

[0144] In the case of using the attribute recognition algorithm A2, the customer's attributes are recognized using the attribute recognition algorithm A2.

[0145] Based on the recognition result of the attributes, for example, the person who has received the order and the person who has not are identified.

[0146] Also, based on the recognition result of the attributes, the customer's attribute information is saved. The saved information is used for customer service such as the next visit.

[0147] Furthermore, based on the recognition result of the attributes, the way of receiving the order is determined. For example, when the other party is a woman, confirmation is made as to whether to reduce the quantity when receiving the order. Also, when the other party is a child, when proposing a recommended drink, a drink other than an alcoholic drink is proposed.

[0148] In the case of using the individual identification algorithm A1, using the individual identification algorithm A1, who the target person is is recognized together with the attributes.

[0149] Based on the individual identification result, according to the person's order history, the way of receiving the order is determined. For example, the order content at the previous visit is presented, or the person's preferences are analyzed from the order history and recommended dishes are proposed. Also, dishes are proposed based on information about the person's preferences and allergies, and the menu is explained according to the person's knowledge background.

[0150] · Use case 3 Use case 3 is a use case of delivering (serving) dishes to customers in the hall. In use case 3, the customer's attributes are recognized using the attribute recognition algorithm A2.

[0151] Based on the recognition result of the attribute, for example, the prepared food is delivered to the ordered recipient. Also, when the attribute of the person to whom the food is preferentially served is set, the food is preferentially delivered to the person with the set attribute. A process of changing the serving direction is performed according to the dominant hand of the recipient.

[0152] A series of processes corresponding to the above use cases will be described later with reference to the flowchart.

[0153] <Configuration and Operation of the Transport Robot> · Configuration of the Transport Robot FIG. 21 is a block diagram showing a hardware configuration example of the transport robot 2-5.

[0154] The transport robot 2-5 is configured by connecting a top plate lifting drive unit 122, a tire drive unit 123, a sensor group 124, and a communication unit 125 to a controller 121. A sensor device 21 is also connected to the controller 121.

[0155] The controller 121 includes a CPU, ROM, RAM, flash memory, etc. The controller 121 executes a predetermined program and controls the overall operation of the transport robot 2-5 including the sensor device 21. The controller 121 corresponds to the host-side controller 51 (FIG. 13).

[0156] The top plate lifting drive unit 122 is composed of motors provided at the connecting portion between the base portion 101 and the support arm 103, the connecting portion between the top plate 102 and the support arm 103, etc. The top plate lifting drive unit 122 drives each connecting portion.

[0157] Also, the top plate lifting drive unit 122 is composed of rails and motors provided inside the support arm 103. The top plate lifting drive unit 122 expands and contracts the support arm 103.

[0158] The tire drive unit 123 is composed of a motor that drives the tire provided on the bottom surface of the base unit 101.

[0159] The sensor group 124 is composed of various sensors such as a positioning sensor, a gyro sensor, an acceleration sensor, a temperature sensor, and an illuminance sensor. The sensor data representing the detection results by the sensor group 124 is output to the controller 121.

[0160] The communication unit 125 is a wireless communication module such as a wireless LAN module and a mobile communication module. The communication unit 125 communicates with an external device such as the program management server 1.

[0161] FIG. 22 is a block diagram showing a functional configuration example of the transport robot 2-5.

[0162] Among the functional units shown in FIG. 22, at least a part is realized by a CPU constituting the controller 121 and a CPU constituting the controller 31 of the sensor device 21 executing a predetermined program.

[0163] In the controller 121, a path information acquisition unit 151, a positioning control unit 152, a movement control unit 153, an attitude control unit 155, an environmental data acquisition unit 156, and a surrounding state recognition unit 157 are realized.

[0164] On the other hand, in the controller 31 of the sensor device 21, a situation detection unit 201 and a sensing control unit 202 are realized. The sensor device 21 is a data processing device that controls a sensing algorithm.

[0165] The path information acquisition unit 151 of the controller 121 controls the communication unit 125 and receives information on the destination and the movement path transmitted from a control device (not shown). The information received by the path information acquisition unit 151 is output to the movement control unit 153.

[0166] At the timing when the object to be transported is prepared or the like, the movement route may be planned by the route information acquisition unit 151 based on the destination and the current position of the transport robot 2-5.

[0167] In this case, the route information acquisition unit 151 functions as an operation plan setting unit that plans the operation of the transport robot 2-5 and sets the operation plan.

[0168] The positioning control unit 152 detects the current position of the transport robot 2-5. For example, the positioning control unit 152 generates a map of the space where the cooking robot 2-4 is installed based on the detection results of the distance sensors that make up the sensor device 21. The sensor data output from the sensor device 21 is acquired by the environment data acquisition unit 156 and supplied to the positioning control unit 152.

[0169] The positioning control unit 152 detects the current position by identifying its own position in the generated map. The information on the current position detected by the positioning control unit 152 is output to the movement control unit 153. The detection of the current position by the positioning control unit 152 may be performed based on the output of the positioning sensors that make up the sensor group 124. The detection of the current position of the transport robot 2-5 may be performed by the sensor device 21.

[0170] The movement control unit 153 controls the movement of the transport robot 2-5 based on the information supplied from the route information acquisition unit 151 and the current position detected by the positioning control unit 152 by controlling the tire drive unit 123.

[0171] Also, when information about surrounding obstacles is supplied from the surrounding state recognition unit 157, the movement control unit 153 controls the movement to avoid the obstacles. The obstacles include various moving and stationary objects such as people, furniture, and household appliances. In this way, the movement control unit 153 controls the movement of the transport robot 2-5 accompanying the transport of the object to be transported based on the result of human sensing by the sensor device 21.

[0172] The attitude control unit 155 controls the top plate lifting drive unit 122 to control the attitude of the transport robot 2-5. Also, the attitude control unit 155 controls the attitude of the transport robot 2-5 during movement so as to keep the top plate 102 horizontal in conjunction with the control by the movement control unit 153.

[0173] The attitude control unit 155 controls the attitude of the transport robot 2-5 according to the surrounding state recognized by the surrounding state recognition unit 157. For example, the attitude control unit 155 controls the attitude of the transport robot 2-5 so as to bring the height of the top plate 102 closer to the height of the top plate of the cooking robot 2-4 or the top plate of the dining table recognized by the surrounding state recognition unit 157.

[0174] The environment data acquisition unit 156 controls the sensor device 21 to perform human sensing and acquires sensor data representing the result of human sensing. The sensor data acquired by the environment data acquisition unit 156 is supplied to the positioning control unit 152 and the surrounding state recognition unit 157.

[0175] The surrounding state recognition unit 157 recognizes the surrounding state based on the sensor data representing the result of human sensing supplied from the environment data acquisition unit 156. Information representing the recognition result by the surrounding state recognition unit 157 is supplied to the movement control unit 153 and the attitude control unit 155.

[0176] When the sensor device 21 performs detection of obstacles, measurement of the distance to the obstacles, estimation of the direction of the obstacles, estimation of the self-position, etc., the surrounding state recognition unit 157 outputs information regarding the obstacles as information representing the recognition result of the surrounding state.

[0177] Detection of obstacles, measurement of the distance to the obstacles, estimation of the direction of the obstacles, estimation of the self-position, etc. may be performed by the surrounding state recognition unit 157 based on the sensing result by the sensor device 21. In this case, the sensor data used for each process performed by the surrounding state recognition unit 157 will be detected by the sensor device 21.

[0178] Thus, the content of the processing performed in the sensor device 21 is arbitrary. That is, the raw data detected by the sensors provided in the sensor device 21 may be directly supplied to the controller 121 as sensor data, or the raw data may be processed and analyzed on the sensor device 21 side, and the results of the processing and analysis may be supplied to the controller 121 as sensor data.

[0179] The situation detection unit 201 on the sensor device 21 side detects the situation of the transport robot 2-5. The situation of the transport robot 2-5 is detected based on, for example, the sensor data output by the sensors constituting the sensor group 124, or the sensor data output by the sensors provided in the sensor device 21.

[0180] The situation of the transport robot 2-5 includes, for example, the operation of the transport robot 2-5 such as what operation it is performing, the location where the transport robot 2-5 is located, the weather, temperature, humidity, and brightness at the location where the transport robot 2-5 is located. In addition, the situation of the transport robot 2-5 also includes external situations such as the situation of the person with whom the transport robot 2-5 is communicating and the situation of obstacles around the transport robot 2-5.

[0181] The situation detection unit 201 outputs information representing such a situation of the transport robot 2-5 to the sensing control unit 202.

[0182] The sensing control unit 202 selects a sensing algorithm according to the human sensing condition of performing human sensing in the situation detected by the situation detection unit 201, and executes a sensing program that defines the selected sensing algorithm.

[0183] For example, for each human sensing condition, a sensing algorithm or a sensing program is associated. The sensing control unit 202 selects a sensing algorithm or a sensing program according to the human sensing condition using an ID as identification data. A sensing program set may be selected according to the human sensing condition.

[0184] The sensing control unit 202 drives each sensor provided in the sensor device 21 by executing a sensing program, and performs human sensing based on the output of each sensor. The sensing control unit 202 outputs sensor data representing the result of human sensing to the controller 121. The sensing control unit 202 appropriately outputs various types of sensor data other than the result of human sensing.

[0185] · Operations of the transport robot With reference to the flowchart of FIG. 23, the processing of the transport robot 2-5 will be described.

[0186] In step S1, the sensing control unit 202 performs accuracy selection processing. By the accuracy selection processing, a sensing algorithm for ensuring the accuracy of human sensing is selected. Details of the accuracy selection processing will be described later with reference to the flowchart of FIG. 24.

[0187] In step S2, the sensing control unit 202 performs human sensing processing. The human sensing processing is processing according to the use case as described above. Details of the human sensing processing will be described later with reference to the flowchart of FIG. 27.

[0188] Next, with reference to the flowchart of FIG. 24, the accuracy selection processing performed in step S1 of FIG. 23 will be described.

[0189] In step S11, the situation detection unit 201 detects the location of the transport robot 2-5 based on the sensor data from the sensor group 124 or the sensor data output by each sensor constituting the sensor device 21. The situation of the transport robot 2-5 may be detected using both the sensor data from the sensor group 124 and the sensor data output by each sensor constituting the sensor device 21.

[0190] In step S12, the sensing control unit 202 determines whether the location where the transport robot 2-5 is located is indoors based on the detection result by the situation detection unit 201.

[0191] If it is determined in step S12 that the location where the transport robot 2-5 is located is indoors, then in step S13, the sensing control unit 202 selects the indoor basic algorithm and performs human sensing.

[0192] The indoor basic algorithm is a sensing algorithm that adjusts the shooting parameters of the RGB camera 32A, such as the shutter speed and sensitivity, according to the intensity of the ambient light and performs human sensing. The shutter speed is set to a slower speed than the standard, and the sensitivity is set to a higher sensitivity than the standard.

[0193] In step S14, the sensing control unit 202 performs indoor processing. In the indoor processing, a sensing algorithm is selected according to the indoor situation, and human sensing is performed. The sensing algorithm used for human sensing is appropriately switched from the indoor basic algorithm to other sensing algorithms. Details of the indoor processing will be described later with reference to the flowchart of FIG. 25.

[0194] On the other hand, if it is determined in step S12 that the location of the transport robot 2-5 is not indoors, that is, outdoors, then in step S15, the sensing control unit 202 selects the outdoor basic algorithm and performs human sensing.

[0195] The outdoor basic algorithm is a sensing algorithm that adjusts the shooting parameters of the RGB camera 32A, such as shutter speed and sensitivity, according to the intensity of the ambient light and performs human sensing. The shutter speed is set to a speed faster than the standard, and the sensitivity is set to a sensitivity lower than the standard.

[0196] In step S16, the sensing control unit 202 performs outdoor processing. In the outdoor processing, a sensing algorithm is selected according to the outdoor situation, and human sensing is performed. The sensing algorithm used for human sensing can be appropriately switched from the outdoor basic algorithm to other sensing algorithms. Details of the outdoor processing will be described later with reference to the flowchart of FIG. 26.

[0197] After the indoor processing is performed in step S14, or after the outdoor processing is performed in step S16, the process returns to step S1 of FIG. 23, and the subsequent processing is repeated.

[0198] Next, with reference to the flowchart of FIG. 25, the indoor processing performed in step S14 of FIG. 24 will be described.

[0199] In step S21, the sensing control unit 202 determines whether the location of the transport robot 2-5 is a dark place based on the detection result by the situation detection unit 201.

[0200] If it is determined in step S21 that the location of the transport robot 2-5 is a dark place, then in step S22, the sensing control unit 202 selects a dark place algorithm according to the human sensing condition of performing human sensing in a dark place and performs human sensing. The dark place algorithm is, for example, a sensing algorithm that sets the sensitivity of the RGB camera 32A higher than the standard sensitivity to capture an RGB image and performs human sensing based on the captured RGB image.

[0201] If it is determined in step S21 that the location of the transport robot 2-5 is not a dark place, in step S23, the sensing control unit 202 selects a bright place algorithm according to the human sensing condition of performing human sensing in a bright place, and performs human sensing. The bright place algorithm is, for example, a sensing algorithm that sets the sensitivity of the RGB camera 32A lower than the standard sensitivity to capture an RGB image, and performs human sensing based on the RGB image obtained by the capture.

[0202] After human sensing is performed using the sensing algorithm selected according to the location of the transport robot 2-5, the process returns to step S14 in FIG. 24, and subsequent processes are performed.

[0203] Next, with reference to the flowchart of FIG. 26, the outdoor process performed in step S16 of FIG. 24 will be described.

[0204] In step S31, the sensing control unit 202 determines whether the weather at the location where the transport robot 2-5 is located is clear and sunny. The determination of whether the weather is clear and sunny is made based on the result of the situation detection by the situation detection unit 201.

[0205] If it is determined in step S31 that the weather at the location of the transport robot 2-5 is clear and sunny, in step S32, the sensing control unit 202 determines whether it is a place where shadows are likely to occur.

[0206] If it is determined in step S32 that it is a place where shadows are likely to occur, in step S33, the sensing control unit 202 selects an algorithm that is resistant to shadow noise according to the human sensing condition of performing human sensing in a place where shadows are likely to occur. The sensing algorithm selected here is a sensing algorithm that adjusts the shooting parameters of the RGB camera 32A to expand the dynamic range of luminance to perform shooting, and performs human sensing based on the RGB image obtained by the shooting.

[0207] After a sensing algorithm is selected, a sensing program that defines the selected sensing algorithm is executed, and human sensing is performed. The same applies when another sensing algorithm is selected.

[0208] If it is determined in step S32 that the location is not a place where shadows are likely to occur, in step S34, the sensing control unit 202 selects an algorithm that is strong against direct sunlight according to the human sensing condition of performing human sensing under direct sunlight. The sensing algorithm selected here is a sensing algorithm that adjusts the shooting parameters of the RGB camera 32A to increase the shutter speed and decrease the sensitivity, performs shooting, and performs human sensing based on the RGB image obtained by the shooting.

[0209] On the other hand, if it is determined in step S31 that the weather is not clear and sunny, the process proceeds to step S35.

[0210] In step S35, the sensing control unit 202 determines whether it is raining. The determination of whether it is raining is made based on the result of the situation detection by the situation detection unit 201.

[0211] If it is determined in step S35 that it is raining, in step S36, the sensing control unit 202 selects an algorithm that is strong against rain noise according to the human sensing condition of performing human sensing in a place where it is raining. The sensing algorithm selected here is a sensing algorithm that performs noise removal image processing on the RGB image captured by the RGB camera 32A and then performs human sensing based on the RGB image after noise removal.

[0212] Known techniques are used for noise removal. Regarding noise removal techniques, for example, they are disclosed in "https: / / digibibo.com / blog-entry-3422.html" and "http: / / www.robot.t.u-tokyo.ac.jp / ~yamashita / paper / A / A025Final.pdf".

[0213] If it is determined in step S35 that it is not raining, in step S37, the sensing control unit 202 selects an algorithm adaptable to a dim place according to the human sensing condition of performing human sensing in a dim place. The sensing algorithm selected here is a sensing algorithm that adjusts the shooting parameters of the RGB camera 32A to slow down the shutter speed and increase the sensitivity, performs shooting, and performs human sensing based on the RGB image obtained by the shooting.

[0214] After human sensing is performed using the sensing algorithm selected according to the location of the transport robot 2-5, the process returns to step S16 in FIG. 24, and the subsequent processing is performed.

[0215] Next, with reference to the flowchart of FIG. 27, the human sensing process performed in step S2 of FIG. 23 will be described.

[0216] Here, as described in use case 2, the process of the transport robot 2-5 receiving an order will be described. The process in FIG. 27 is performed, for example, after the transport robot 2-5 has moved to the vicinity of the person receiving the order.

[0217] In step S51, the sensing control unit 202 determines whether to perform personal identification when receiving an order.

[0218] If it is determined in step S51 that personal identification is to be performed, in step S52, the sensing control unit 202 performs human sensing using a personal identification algorithm according to the human sensing condition of performing personal identification.

[0219] If it is determined in step S51 that personal identification is not performed, in step S53, the sensing control unit 202 performs human sensing using an attribute recognition algorithm according to the human sensing condition of performing attribute recognition.

[0220] After human sensing is performed using the algorithm selected according to whether personal identification is performed or not, the process proceeds to step S54.

[0221] In step S54, the controller 121 (for example, the surrounding state recognition unit 157) determines based on the result of human sensing whether the person identified by human sensing is a customer who has come to the store for the first time.

[0222] If it is determined in step S54 that the person is a customer who has come to the store for the first time, in step S55, the controller 121 performs a corresponding operation for the first-time customer.

[0223] In step S56, the controller 121 accepts an order based on the result of personal identification or the result of attribute recognition. For example, if the person identified by human sensing is a female, confirmation is made as to whether to reduce the quantity.

[0224] On the other hand, if it is determined in step S54 that the person is not a customer who has come to the store for the first time, in step S57, the controller 121 performs a corresponding operation according to the number of visits to the store. After the customer is dealt with in step S56 or step S57, the process proceeds to step S58.

[0225] In step S58, when the personal identification algorithm is used, the sensing control unit 202 switches the sensing algorithm used for human sensing to the attribute recognition algorithm. Then, it returns to step S2 in FIG. 23, and the above-described process is repeated.

[0226] Next, referring to the flowchart of FIG. 28, other person sensing processing performed in step S2 of FIG. 23 will be described.

[0227] Here, as described in use case 3, the processing of the delivery robot 2-5 for serving dishes will be described. The processing in FIG. 28 is performed, for example, after the delivery robot 2-5 has moved to the vicinity of the target person (order placer) who is the recipient of the dish.

[0228] In step S71, the sensing control unit 202 of the sensor device 21 performs person sensing using an attribute recognition algorithm according to the person sensing condition of performing attribute recognition.

[0229] In step S72, the controller 121 determines whether there is an order placer who should be preferentially served based on the result of person sensing by the sensor device 21.

[0230] If it is determined in step S72 that there is an order placer who should be preferentially served, in step S73, the sensing control unit 202 searches for the order placer who should be preferentially served.

[0231] On the other hand, if it is determined in step S72 that there is no order placer who should be preferentially served, in step S74, the sensing control unit 202 searches for the nearest order placer.

[0232] After the search for the order placer is performed in step S73 or step S74, the process proceeds to step S75.

[0233] In step S75, the controller 121 (for example, the movement control unit 153) moves based on the result of person sensing by the sensor device 21. For example, the delivery robot 2-5 is moved to approach the order placer found by the search.

[0234] In step S76, the controller 121 determines whether to change the food delivery direction based on the result of human sensing by the sensor device 21. For example, if the position of the transport robot 2-5 is on the side opposite to the dominant hand of the order placer, it is determined that the food delivery direction should be changed. Conversely, if the position of the transport robot 2-5 is on the side of the dominant hand of the order placer, it is determined that the food delivery direction should not be changed.

[0235] If it is determined in step S76 that the food delivery direction is to be changed, in step S77, the controller 121 moves the transport robot 2-5 to change the food delivery direction. If it is determined in step S76 that the food delivery direction is not to be changed, the process of step S77 is skipped.

[0236] In step S78, for example, the posture control unit 155 of the controller 121 controls the posture of the transport robot 2-5, adjusts the height of the top plate 102 to the height of the table being used by the customer, and delivers the food to the order placer.

[0237] In step S79, the sensing control unit 202 determines whether all the food has been delivered.

[0238] If it is determined in step S79 that there is undelivered food, the process returns to step S72 and the above-described process is repeated.

[0239] On the other hand, if it is determined in step S79 that all the food has been delivered, in step S80, the sensing control unit 202 performs human sensing using an attribute recognition algorithm. Then, the process returns to step S2 in FIG. 23 and the above-described process is repeated.

[0240] Through the above processes, the transport robot 2-5 can select a sensing algorithm according to the situation such as its own location and perform human sensing. Also, the transport robot 2-5 can control various operations such as food delivery based on the result of human sensing performed using the sensing algorithm selected according to the situation.

[0241] <Other use cases> Other use cases of human sensing by the transport robot 2-5 will be described.

[0242] · Use case 4 Use case 4 is a use case of serving drinks.

[0243] In use case 4, the attributes of the customer are recognized using the attribute recognition algorithm A2. Based on the recognition result of the attributes, for example, the transport robot 2-5 performs a process of determining the type of drink to be served. For example, if the attribute of the person to be served is a child, a drink other than an alcoholic drink is served.

[0244] · Use case 5 Use case 5 is a use case of passing out snacks such as gum and candy according to the ordered dishes.

[0245] In use case 5, the attributes of the customer are recognized using the attribute recognition algorithm A2. Based on the recognition result of the attributes, for example, the transport robot 2-5 performs a process of associating and managing the attribute of the orderer with the dish ordered by that person. When there is a person leaving the store after finishing eating, snacks are passed out according to the ordered dishes.

[0246] In the above, the use cases of human sensing by the transport robot 2-5 have been described, but for human sensing by other devices, various use cases are also assumed in the same way.

[0247] Here, the use cases of human sensing by the cooking robot 2-4 will be described.

[0248] · Use case 6 Use case 6 is a use case when plating food ingredients. In use case 6, there are a case of using the personal identification algorithm A1 and a case of using the attribute recognition algorithm A2.

[0249] Incidentally, the plating of food by the cooking robot 2-4 is performed by driving the cooking arm and placing the cooked food at a predetermined position on the tableware. Based on the result of object sensing by the sensor device 21 or the like, the position of the cooked food, the position of the tableware, etc. are recognized, and plating is performed.

[0250] In the case of using the attribute recognition algorithm A2, the attribute of the person who ordered the dish being plated is recognized using the attribute recognition algorithm A2.

[0251] Based on the recognition result of the attribute, for example, when the person who placed the order is a woman, tableware suitable for women is used, and when the person is a child, plating for children is performed. Thus, the plating method is changed.

[0252] In the case of using the personal identification algorithm A1, who the target person is is recognized together with the attribute using the personal identification algorithm A1.

[0253] Based on the individual identification result, for example, processing such as changing the plating method or the ingredients to be used according to the preferences of that person is performed by the cooking robot 2-4.

[0254] · Use Case 7 Use Case 7 is a use case when determining the menu. The cooking by the cooking robot 2-4 is performed according to the cooking data corresponding to the menu. The cooking data includes information that defines the content and order of the cooking operations of the cooking arm in each cooking step until the completion of the dish.

[0255] In Use Case 7, who the target person is is recognized together with the attribute using the personal identification algorithm A1.

[0256] Based on the individual identification result, for example, the history of the target person's diet content is specified, and a menu considering the nutritional balance is determined.

[0257] · Use Case 8 Use Case 8 is a use case when cooking at home. In Use Case 8, using the personal identification algorithm A1, who the target person is is recognized together with the attributes.

[0258] Based on the individual identification result, for example, when it is specified that there are only family members, cooking is performed using the oldest ingredients first. Also, according to the family's health condition, processing is performed to change the cooking content (intensity of taste, degree of cooking, adjustment with hot water, etc.).

[0259] · Use Case 9 Use Case 9 is a use case when washing hands. At a predetermined position on the top plate of the cooking robot 2-4, a groove for washing hands is provided. The groove is provided with a configuration for spraying water and cleaning liquid toward the hands.

[0260] In Use Case 9, there are cases of using the personal identification algorithm A1 and cases of using the attribute recognition algorithm A2.

[0261] In the case of using the attribute recognition algorithm A2, using the attribute recognition algorithm A2, the attributes of the person washing hands are recognized.

[0262] Based on the recognition result of the attributes, for example, when the person washing hands is a woman or a child, the water pressure is reduced and hand washing is performed.

[0263] In the case of using the personal identification algorithm A1, using the personal identification algorithm A1, who the target person is is recognized together with the attributes.

[0264] Based on the individual identification result, for example, according to that person's preference, the strength of washing and the type of cleaning liquid are changed.

[0265] · Use Case 10 Use case 10 is a use case when using the cooking robot 2-4. In use case 10, the attributes of the target person are recognized using the attribute recognition algorithm A2.

[0266] Based on the recognition result of the attributes, for example, control such that children cannot use it is performed by the cooking robot 2-4.

[0267] <Modification example> ·Example of application to other systems Figure 29 is a diagram showing an example of the application of human sensing.

[0268] When human sensing is performed in each of a plurality of systems provided in one store, as shown in Figure 29, human sensing according to the use case is defined for each system.

[0269] The food distribution system shown in Figure 29 is a system composed of the transport robot 2-5. In the food distribution system, human sensing according to the use case is performed by the process described with reference to Figure 23.

[0270] The in-store monitoring system shown in Figure 29 is a system that uses the sensor device 21 as a monitoring camera. The sensor device 21 is attached to each position in the store. Also in the in-store monitoring system, human sensing according to the use case is performed by the same process as the process described with reference to Figure 23.

[0271] ·Example when the selection of the sensing algorithm is performed externally Although the selection of the sensing algorithm according to the human sensing condition is performed within the sensor device 21, it may be performed by a device external to the device on which the sensor device 21 is mounted.

[0272] Figure 30 is a diagram showing an example of the control of the sensing algorithm.

[0273] In the example of FIG. 30, the selection of the sensing algorithm according to the sensing conditions is performed by the program management server 1 which is an external device. In this case, the configuration of the controller 31 in FIG. 22 is realized in the program management server 1. The program management server 1 is a data processing device that controls the sensing program executed by the sensor device 21 mounted on the transport robot 2-5.

[0274] As shown by the arrow #1, sensor data used for situation detection is transmitted from the transport robot 2-5 to the program management server 1, and the execution of the sensing program is requested.

[0275] In the situation detection unit 201 of the program management server 1, the situation of the transport robot 2-5 is detected based on the sensor data transmitted from the transport robot 2-5. Also, the human sensing conditions according to the situation of the transport robot 2-5 are determined by the sensing control unit 202, and the sensing algorithm is selected.

[0276] The sensing control unit 202 of the program management server 1 transmits a sensing program that defines the sensing algorithm according to the human sensing conditions to the sensor device 21 mounted on the transport robot 2-5 and causes it to be executed.

[0277] In this way, the control of the sensing algorithm may be performed by a device external to the sensor device 21. For example, it is also possible to use the controller 121 of the transport robot 2-5 on which the sensor device 21 is mounted as an external device and perform the control of the sensing algorithm by the controller 121.

[0278] A sensing program that defines the sensing algorithm according to the human sensing conditions may be executed by an external device such as the program management server 1 or the controller 121, and information representing the execution result may be transmitted to the sensor device 21.

[0279] FIG. 31 is a block diagram showing a hardware configuration example of a computer that realizes the program management server 1.

[0280] A CPU (Central Processing Unit) 1001, a ROM (Read Only Memory) 1002, and a RAM (Random Access Memory) 1003 are interconnected by a bus 1004.

[0281] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006 composed of a keyboard, a mouse, etc., and an output unit 1007 composed of a display, a speaker, etc. are connected to the input / output interface 1005. Also, a storage unit 1008 composed of a hard disk, a non-volatile memory, etc., a communication unit 1009 composed of a network interface, etc., and a drive 1010 for driving a removable medium 1011 are connected to the input / output interface 1005.

[0282] Control of the sensing algorithm as described above is realized by a predetermined program being executed by the CPU 1001.

[0283] · Example of Program The above-described series of processes can be executed by hardware or by software. When the series of processes are executed by software, the program constituting the software is installed in a computer in which the program is incorporated into dedicated hardware, or a general-purpose personal computer, etc.

[0284] The program to be installed is provided by being recorded on a removable medium 1011 shown in FIG. 31, which consists of an optical disk (such as a CD-ROM (Compact Disc-Read Only Memory), a DVD (Digital Versatile Disc)), a semiconductor memory, or the like. Also, it may be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital broadcasting. The program can be pre-installed in the ROM 1002 or the storage unit 1008.

[0285] Note that the program executed by the computer may be a program that is processed in time series in accordance with the order described in this specification, or a program that is processed in parallel or at a necessary timing such as when a call is made.

[0286] Note that in this specification, the system means a collection of a plurality of components (devices, modules (parts), etc.), and it does not matter whether all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.

[0287] The effects described in this specification are merely examples and are not limiting, and there may be other effects.

[0288] The embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0289] For example, the present technology can take a configuration of cloud computing in which one function is shared and jointly processed by a plurality of devices via a network.

[0290] Also, each step described in the above flowchart can be executed by one device or shared and executed by a plurality of devices.

[0291] Furthermore, when a single step includes a plurality of processes, the plurality of processes included in that single step can be executed not only by a single device but also by being shared and executed by a plurality of devices.

Explanation of Signs

[0292] 1 Program management server, 2-1 Mobile terminal, 2-2 Arm robot, 2-3 Mobile body, 2-4 Cooking robot, 2-5 Transport robot, 21 Sensor device, 31 Controller, 32 Sensor group, 121 Controller, 124 Sensor group, 201 Situation detection unit, 202 Sensing control unit

Claims

1. A human sensing algorithm for sensing a person based on sensor data output from a sensor implemented in a moving body having a movable operation unit with respect to the moving body main body is determined according to human sensing conditions, The determined human sensing algorithm is a human sensing program defined by a combination of a plurality of the human sensing programs, and is selected and executed from a human sensing program set including a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, To control the movement of the moving body and the operation of the operation unit Data processing device.

2. The human sensing program defined by the human sensing algorithm is acquired via a network The data processing device according to claim 1.

3. The human sensing program set is acquired via a network The data processing device according to claim 1.

4. The human sensing program set is selected using identification data for identifying the human sensing program set The data processing device according to claim 3.

5. The human sensing algorithms defined in the plurality of human sensing programs are algorithms applied to sensor data output by setting different parameters for the same sensor The data processing device according to claim 1.

6. The human sensing algorithms defined in the plurality of human sensing programs are algorithms applied to sensor data output by setting the same parameters for the same sensor The data processing device according to claim 1.

7. The human sensing algorithms defined in the plurality of human sensing programs are algorithms applied to sensor data output from different sensors The data processing device according to claim 1.

8. At least one of the human sensing program and the human sensing algorithm defined in the human sensing program is associated with a sensor, In conjunction with selecting and executing the human sensing program, the operations of a plurality of sensors are controlled The data processing device according to claim 7.

9. Based on the execution result of the human sensing program, the state of the movement by the moving part accompanying the transportation of the object to be transported is controlled The data processing device according to claim 1.

10. The operation unit includes a top plate on which the object to be transported is placed, and a telescopic support unit that supports the top plate. The moving unit is connected to the support unit. Based on the execution result of the human sensing program, it controls the posture state including the state of the top plate and the state of the support unit, and the moving state by the moving unit. The data processing device according to claim 9.

11. The top plate places the cooking arm of the cooking system that is driven according to the cooking process or the object to be transported placed by a person. The data processing device according to claim 10.

12. A data processing device Determines a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having an operation unit movable with respect to the moving body main body according to human sensing conditions, Selects and executes a human sensing program in which the determined human sensing algorithm is defined from a set of human sensing programs that is a combination of a plurality of the human sensing programs and includes a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, Controls the movement of the moving body and the operation of the operation unit. A data processing method.

13. Determines a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having an operation unit movable with respect to the moving body main body according to human sensing conditions, Selects a human sensing program in which the determined human sensing algorithm is defined from a set of human sensing programs that is a combination of a plurality of the human sensing programs and includes a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, and transmits it to the moving body. A data processing device.

14. In response to a request from the moving body, transmits the human sensing program in which the human sensing algorithm is defined. The data processing device according to claim 13.

15. A data processing device Determines a human sensing algorithm for sensing a person based on sensor data output from a sensor mounted on a moving body having an operation unit movable with respect to the moving body main body according to human sensing conditions, Select the determined human sensing algorithm-defined human sensing program from a set of human sensing programs that is a combination of a plurality of the human sensing programs and includes a combination of information representing the type of the human sensing algorithm and information representing the execution order of the human sensing program, and transmit it to the mobile object. Data processing method.

16. A sensor that outputs sensor data representing a sensing result, A sensing control unit that adaptively selects and executes a human sensing program in which a human sensing algorithm for sensing a person based on the sensor data output from the sensor is defined, according to human sensing conditions, An operation plan setting unit that sets an operation plan based on the execution result of the human sensing program by the sensing control unit, An operation unit that operates according to the operation plan set by the operation plan setting unit Comprising The operation unit is movable with respect to the mobile body main body Mobile object.

Images