Control method, control device, program, and moving object control system

Abstract

To make it possible to display a recognition level of an object recognized from a photographed image imaged by a moving object.SOLUTION: An instruction device receives an imaged photographed image from a robot having a camera. The instruction device causes a display screen of the instruction device to display an object recognized from the photographed image and information indicating the recognition level of the object in association with the object. The present disclosure can be applied to a robot control system comprising a robot having a camera and a display device for controlling the robot.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present disclosure relates to a control method, a control device, a program, and a mobile body control system, and, for example, to a control method, a control device, a program, and a mobile body control system that are capable of displaying the degree of recognition of an object recognized from an image captured by a mobile body. [Background technology]

[0002] For example, autonomous robots that act (move) autonomously in response to the surrounding situation and general instructions from a user are becoming commonplace.

[0003] For example, if a user instructs this autonomous robot to bring a bottle of tea without specifying the product name, the robot will autonomously search for and bring the bottle of tea in response to the instruction.

[0004] That is, for example, if the autonomous robot finds multiple bottles of tea, it will use its own judgment to select the bottle of tea that the user desires and bring it to the user.

[0005] There is a technology in which a user specifies an item and a destination for the item, and a robot moves the item to the destination (see, for example, Patent Document 1).

[0006] However, since an autonomous robot acts autonomously, it may pass through the search range desired by the user and search other ranges. Therefore, it is desirable that the user specify a search range and have the autonomous robot search the specified search range. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2004-268148 Summary of the Invention [Problem to be solved by the invention]

[0008] It is desirable to display the degree of recognition of an object recognized from an image captured by a moving object.

[0009] The present disclosure has been made in consideration of such circumstances, and makes it possible to display the degree of recognition of an object recognized from an image captured by a moving body. [Means for solving the problem]

[0010] A control method according to a first aspect of the present disclosure is a control method in which a control device receives an image captured from a moving body having an imaging unit, and displays, on a display screen of the control device, an object recognized from the image and degree information corresponding to the object and indicating a degree of recognition of the object. A control device according to a first aspect of the present disclosure is a control device including a receiving unit that receives an image captured from a moving body having an imaging unit, and a display control unit that displays on a display screen an object recognized from the image and degree information that corresponds to the object and indicates the degree of recognition of the object. A program according to a first aspect of the present disclosure is a program for causing a computer to execute processing including receiving an image captured from a moving body having an imaging unit, and displaying on a display screen an object recognized from the image and degree information indicating the degree of recognition of the object in association with the object. In a first aspect of the present disclosure, a captured image is received from a moving body having an imaging unit, and an object recognized from the captured image and degree information indicating the degree of recognition of the object, associated with the object, are displayed on a display screen.

[0011] A mobile body control system according to a second aspect of the present disclosure is a mobile body control system including a control device having a receiving unit that receives an image captured from a mobile body having an imaging unit, and a display control unit that displays, on a display screen of the control device, an object recognized from the image and degree information that corresponds to the object and indicates the degree of recognition of the object. In a second aspect of the present disclosure, a control device and a mobile body are provided, the control device having a receiving unit that receives an image captured from the mobile body having an imaging unit, and a display control unit that displays, on a display screen of the control device, an object recognized from the image and degree information that corresponds to the object and indicates the degree of recognition of the object. [Effects of the Invention]

[0012] According to the present disclosure, it is possible to display the degree of recognition of an object recognized from an image captured by a moving body. [Brief explanation of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a robot control system according to the present disclosure. [Figure 2] FIG. 10 is a diagram showing an example of a display screen on which a plurality of items are displayed. [Figure 3] FIG. 1 is a diagram for explaining an outline of processing performed by a robot. [Figure 4] FIG. 10 is a diagram illustrating an example of a recognition state of an object. [Figure 5] FIG. 2 is a block diagram showing an example of the configuration of an instruction device. [Figure 6] FIG. 11 is a first diagram showing an example of a display screen on which a display based on information from a robot is displayed. [Figure 7] FIG. 2 is a second diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 8] FIG. 3 is a third diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 9]FIG. 4 is a fourth diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 10] FIG. 5 is a fifth diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 11] FIG. 6 is a sixth diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 12] FIG. 7 is a seventh diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 13] FIG. 8 is an eighth diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 14] FIG. 9 is a ninth diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 15] FIG. 10 is a first diagram showing an example of displaying changes in the score of an object using arrows. [Figure 16] FIG. 2 is a second diagram showing an example of displaying changes in the score of an object using arrows. [Figure 17] FIG. 10 is a tenth diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 18] 10 is a flowchart illustrating a display process performed by the instruction device. [Figure 19] FIG. 2 is a block diagram showing an example of the configuration of a robot. [Figure 20] FIG. 10 is a state transition diagram showing an example of transition of an object recognition state. [Figure 21] 10A and 10B are diagrams for explaining a method for generating robot recognition information according to the recognition state of an object. [Figure 22] 10 is a flowchart illustrating a score calculation process performed by a robot. [Figure 23] FIG. 11 is an eleventh figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 24] FIG. 12 is a twelfth figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 25]FIG. 13 is a thirteenth figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 26] FIG. 14 is a fourteenth figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 27] FIG. 15 is a fifteenth figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 28] FIG. 16 is a 16th figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 29] FIG. 17 is a 17th figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 30] FIG. 18 is an 18th figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 31] FIG. 10 is a diagram illustrating a feedback operation by a user. [Figure 32] 10A and 10B are diagrams illustrating an example of how the recognition state of an object transitions in response to a feedback operation. [Figure 33] FIG. 19 is a 19th diagram showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 34] FIG. 20 is a twentieth figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 35] FIG. 21 is a 21st figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 36] FIG. 22 is a 22nd figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 37] FIG. 23 is a 23rd figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 38] FIG. 24 is a 24th figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 39] 10 is a flowchart illustrating a feedback process performed by the instruction device. [Figure 40]10 is a flowchart for explaining a score recalculation process performed by a robot. [Figure 41] FIG. 25 is a 25th figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 42] FIG. 26 is a 26th figure showing an example of a display screen on which a display based on information from the robot is displayed. [Figure 43] FIG. 10 is a diagram illustrating an example of a case where a recognition state determined to be undetected is transitioned to another state. [Figure 44] 10A and 10B are diagrams illustrating an example of specifying a search range or an object using a captured image. [Figure 45] FIG. 10 is a diagram showing an example of specifying a search range using a spatial diagram of an interior of a room. [Figure 46] 10A and 10B are diagrams illustrating an example of reducing the amount of data of a captured image transmitted from a robot to an instruction device. [Figure 47] 10 is a flowchart illustrating an area designation process performed by the instruction device. [Figure 48] 10 is a flowchart for explaining a search process performed by a robot. [Figure 49] FIG. 1 is a block diagram illustrating an example of the configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0014] An embodiment of the present disclosure (hereinafter referred to as the present embodiment) will be described below. The description will be made in the following order. 1. First embodiment (an example of a case where information for predicting the behavior of a robot is displayed on a display screen of a command device) 2. Modification of the First Embodiment 3. Second embodiment (an example of a case where a user gives feedback to a robot using a pointing device) 4. Modification of the Second Embodiment 5. Third embodiment (an example of specifying a search range for a robot using a pointing device)

[0015] <1. First embodiment> [Example of robot control system configuration] FIG. 1 shows an example of the configuration of a robot control system 1 according to the first embodiment.

[0016] The robot control system 1 comprises an instruction device 21 having a display screen 21 a and an autonomous robot 22 .

[0017] The instruction device 21 generates instruction information for causing the robot 22 to take a desired action (for example, bringing a specified object to the user) in response to the user's operation, and transmits the information to the robot 22 via wireless communication or the like.

[0018] That is, for example, as shown in FIG. 2, the instruction device 21 displays a plurality of items selectable by the user, such as "tea," "carbonated drink," "sweets," etc., on the display screen 21a.

[0019] 2, the user selects (specifies) an item of an object that the user wants the robot 22 to bring from among a plurality of items. Note that, for example, categories of objects that the user wants the robot 22 to bring are displayed as items on the display screen 21a.

[0020] In response to this, the instruction device 21 generates instruction information for targeting an object belonging to the category (e.g., "tea") as the item selected by the user and for causing the object to be brought to the user, and transmits the instruction information to the robot 22 by wireless communication or the like. The instruction information includes category information indicating the category as the item selected by the user.

[0021] Based on instruction information from the instruction device 21, the robot 22 autonomously performs an operation of searching for an object as a target that belongs to a category as an item selected by the user.

[0022] The method of giving instructions to the robot 22 is not limited to the above-mentioned method using the instruction device 21, and there are other methods, for example, if the robot 22 is capable of recognizing the user's voice through voice recognition, the user can give instructions to a target, etc., by voice.

[0023] In this case, the robot 22 recognizes the instruction from the user as a voice, and based on the recognition result, autonomously executes the operation of searching for the target instructed by the user. Furthermore, for example, if the robot 22 is capable of recognizing the user's gestures, posture, etc., the user can indicate a target, etc., by using gestures, posture, etc.

[0024] The robot 22 is a bipedal walking robot that autonomously performs various actions based on the surrounding situation and general instructions from the user (for example, a general instruction to bring "tea").

[0025] The robot 22 is provided with sensors for detecting external stimuli, such as microphones equivalent to "ears" that detect sound, and cameras equivalent to "eyes" that detect light, at predetermined positions. Details of the robot 22 will be described later with reference to FIG. 19.

[0026] Next, FIG. 3 shows an overview of the processing performed by the robot 22.

[0027] Figure 3 shows an ambient image 31 obtained by capturing an image of the area around the robot 22, an object image 32 representing an object that the robot 22 recognizes as a target, an object image 33 representing an object that the robot 22 recognizes as not a target, an object image 34 representing an object that the robot 22 recognizes as possibly being a target, and a partial image 35 representing a portion of the captured image obtained by capturing an image of the robot 22.

[0028] The robot 22 autonomously moves within the room 36 in which the user resides, for example, to search for the target.

[0029] While searching for a target, the robot 22 appropriately transmits a surrounding image 31 and a partial image 35 captured by a built-in camera to the instruction device 21. The surrounding image 31 and the partial image 35 are used when the user specifies a search range for the robot 22 using the instruction device 21. This will be described mainly with reference to Figs. 43 to 47.

[0030] Furthermore, the robot 22 moves within the room 36 and, based on the captured image obtained by capturing an image of an object with the built-in camera, calculates a score representing the degree to which the object is a target, that is, a score representing the degree to which the object is to be processed by the robot 22. Note that a classifier (a function representing) for identifying whether or not an object is a target is used to calculate the score.

[0031] Then, the robot 22 determines the recognition state of the object based on the score of the object. In the following description, the robot 22 will be described as using an image obtained by a built-in camera to calculate the score and transmitting the image to the instruction device 21.

[0032] However, the images used for calculating the score and the images to be transmitted to the instruction device 21 are not limited to images obtained by a camera built into the robot 22, and may be images obtained by a camera installed on the ceiling of the room 36, for example. In this case, the camera installed on the ceiling of the room 36 transmits the captured images to the robot 22 via wireless communication.

[0033] Next, FIG. 4 shows an example of a recognition state.

[0034] FIG. 4A shows two cases where the recognition state of the robot 22 is "target" and "not target."

[0035] FIG. 4B shows three cases where the recognition state of the robot 22 is "target", "not sure yet", and "not target".

[0036] FIG. 4C shows a case where the recognition states of the robot 22 are four types: "target", "possibly target", "not yet known", and "not target".

[0037] In the first embodiment, the robot 22 shall recognize as one of the four types of recognition states as shown in FIG. 4C.

[0038] For example, when the score of an object is equal to or greater than the threshold th1, the robot 22 recognizes the object as "target", and when the score of the object is less than the threshold th1 and equal to or greater than the threshold th2 (<th1), it recognizes the object as "possibly target".

[0039] Also, for example, when the score of an object is less than the threshold th2 and equal to or greater than the threshold th3 (<th2), the robot 22 recognizes the object as "not yet known" whether it is a target, and when the score of the object is less than the threshold th3, it recognizes the object as "not target".

[0040] Here, the thresholds th1 to th3 are assumed to be determined in advance and held by the robot 22. Although the robot 22 compares the score with the threshold to determine the recognition information of the object, the method for determining the recognition information is not limited to this.

[0041] Returning to FIG. 3, for example, when the robot 22 determines that the score of the object "ABC tea" is equal to or greater than the threshold th1, it causes the display screen 21a of the indicating device 21 to display that the object "ABC tea" is recognized as "target".

[0042] That is, for example, on the display screen 21a of the indicating device 21, as shown in FIG. 3, an object image 32 representing the object "ABC tea" is displayed, for example, surrounded by a thick-line rectangle. This thick rectangle indicates that the robot 22 recognizes the object "ABC tea" as the "target."

[0043] Furthermore, for example, when the robot 22 determines that the score of the object "dog-shaped robot" is less than the threshold value th3, as shown in FIG. 3, the robot 22 displays on the display screen 21a of the instruction device 21 that the robot 22 recognizes the object "dog-shaped robot" as "not a target."

[0044] That is, for example, on the display screen 21a of the instruction device 21, an object image 33 representing the object "dog-type robot" is displayed with, for example, a diagonal line added thereto, as shown in Fig. 3. This diagonal line indicates that the robot 22 recognizes the object "dog-type robot" as "not a target."

[0045] Furthermore, for example, when the robot 22 determines that the score of the object "orange juice" is less than the threshold th1 and greater than or equal to the threshold th2, as shown in FIG. 3, the robot 22 displays on the display screen 21a of the instruction device 21 that it recognizes the object "orange juice" as "possibly a target."

[0046] That is, for example, on the display screen 21a of the instruction device 21, an object image 34 representing a plastic bottle with "orange juice" written on it is displayed surrounded by a thin-line rectangle, for example, as shown in Fig. 3. This thin-line rectangle indicates that the robot 22 recognizes the object "orange juice" as "possibly a target."

[0047] Also, for example, if the robot 22 determines that the score of a specified object is less than the threshold value th2 and greater than or equal to the threshold value th3, it displays on the display screen 21a of the instruction device 21 that it recognizes the specified object as "not yet clear" whether it is a target or not.

[0048] That is, for example, an object image representing a predetermined object is displayed as it is (without being surrounded by a rectangle or having diagonal lines added) on the display screen 21a of the instruction device 21. When the object image is displayed as it is, it indicates that the robot 22 recognizes that it is "still unclear" whether the predetermined object is a target or not.

[0049] In this way, the display screen 21a of the instruction device 21 displays information recognized by the robot 22, so that the user can easily predict the behavior of the robot 22.

[0050] The display method is not limited to this, and any display method may be used as long as it allows the user to intuitively understand the behavior of the robot 22.

[0051] Furthermore, when the user determines by referring to the display screen 21a that an object that is not desirable as a target has been recognized as a target, the user can perform a feedback operation on the pointing device 21 to the effect that the object is not a target.

[0052] In this case, the robot 22 recalculates the score of the object in response to the user's feedback operation and acts autonomously based on the recalculated score. Note that the user's feedback operation will be described later mainly with reference to Figures 31 to 42.

[0053] [Configuration example of indicator device 21] Next, FIG. 5 shows an example of the configuration of the instruction device 21. As shown in FIG.

[0054] The instruction device 21 comprises an operation unit 41, a control unit 42, a display unit 43 having a display screen 21a, a communication unit 44, and a storage unit 45.

[0055] The operation unit 41 is composed of operation keys, operation buttons, etc., and is operated by a user, for example, when the user gives instructions to the robot 22. In response to an operation by the user, the operation unit 41 supplies an operation signal corresponding to the user's operation to the control unit 42.

[0056] The control unit 42 generates a robot recognition image including an object image based on the robot recognition information from the communication unit 44, and supplies the generated image to the display unit 43 to display on the display screen 21a. The robot recognition information includes, for example, the score of the object as information necessary for display on the display screen 21a. Display examples of the display screen 21a will be described later with reference to FIGS. 6 to 17.

[0057] Furthermore, the control unit 42 generates instruction information, feedback information, and designated range information in response to an operation signal from the operation unit 41 and supplies them to the communication unit 44 .

[0058] Here, the feedback information refers to information in the instruction device 21 that indicates the content of the user's feedback.

[0059] The user can use the operation unit 41 to perform a feedback operation to indicate that an object that the robot 22 has recognized as a "target" is not a target, for example. This will be described in detail with reference to Figs. 31 to 42.

[0060] The designated range information indicates, for example, the position of an area designated by a user's area designation operation within the entire area of ​​the surrounding image 31 or partial image 35 displayed on the display screen 21a of the pointing device 21.

[0061] The user can use the operation unit 41 to perform an area designation operation to designate a search range in which the robot 22 searches for a target, for example, from the entire area of ​​the surrounding image 31 or the partial image 35. This will be described in detail with reference to Figs. 43 to 47.

[0062] The control unit 42 supplies the robot recognition information from the communication unit 44 to the storage unit 45 for storage.

[0063] The display unit 43 displays an image on the display screen 21a under the control of the control unit .

[0064] Although the instruction device 21 is configured to include the display unit 43, it may be configured, for example, not to include the display unit 43 in the instruction device 21. In this case, the control unit 42 and the externally provided display unit 43 are connected by a cable or the like.

[0065] The communication unit 44 receives robot recognition information transmitted from the robot 22 and supplies it to the control unit 42. The communication unit 44 also transmits instruction information, feedback information, and designated range information from the control unit 42 to the robot 22 using wireless communication or the like.

[0066] The storage unit 45 stores, for example, a control program executed by the control unit 42 in advance. The storage unit 45 also stores, for example, robot recognition information from the control unit 42.

[0067] [Example of display screen 21a] Next, FIGS. 6 to 14 show an example of the display screen 21a when the instruction device 21 displays a robot recognition image based on robot recognition information from the robot 22. FIG.

[0068] When the robot 22 is instructed to bring the object "tea" based on the instruction information from the instruction device 21, the robot 22 goes out to look for the object "tea."

[0069] The robot 22 is exploring the room, but has not yet detected any object, so the robot 22 does not transmit any robot recognition information to the instruction device 21.

[0070] Therefore, as shown in FIG. 6, the control unit 42 causes, for example, a blank robot recognition image to be displayed on the display screen 21a.

[0071] When the robot 22 detects the object "bird figurine" from the captured image obtained by capturing an image of the interior of the room 36, it extracts the object image 51 representing the object "bird figurine" from the captured image, includes it in the robot recognition information, and transmits it to the communication unit 44.

[0072] In response to this, the communication unit 44 supplies the robot recognition information from the robot 22 to the control unit 42. The control unit 42 supplies the object image 51 included in the robot recognition information from the communication unit 44 to the display unit 43, causing it to be displayed on the display screen 21a.

[0073] As a result, the object image 51 is displayed on the left side of the display screen 21a as shown in Fig. 7. Therefore, the user can easily predict that the robot 22 has detected the bird figurine as the object.

[0074] The robot 22 acts to approach the object "bird figurine" in order to identify whether the object "bird figurine" is a target. Then, the robot 22 extracts an object image 51 representing the "bird figurine" from a captured image obtained by capturing an image of the object "bird figurine" from nearby, includes the extracted object image 51 in robot recognition information, and transmits the robot recognition information to the communication unit 44.

[0075] In response to this, the communication unit 44 supplies the robot recognition information from the robot 22 to the control unit 42. The control unit 42 supplies the object image 51 included in the robot recognition information from the communication unit 44 to the display unit 43, causing it to be displayed on the display screen 21a.

[0076] As a result, as shown in Fig. 8, an object image 51 larger than that in Fig. 7 is displayed on the left side of the display screen 21a. Note that the robot 22 has not yet calculated the score for the object "bird figurine," and therefore recognizes that it "does not yet know" whether the object "bird figurine" is a target. Therefore, in Fig. 8, the object image 51 is displayed as is (without being surrounded by a rectangle or having diagonal lines added).

[0077] The robot 22 calculates a score for the object "bird figurine" from a captured image obtained by capturing an image of the object "bird figurine" from a close distance. Then, when the robot 22 recognizes that the object "bird figurine" is "not a target" based on the calculated score, it transmits to the communication unit 44 robot recognition information including (information representing) a recognition state that indicates "not a target."

[0078] In response to this, the communication unit 44 supplies the robot recognition information from the robot 22 to the control unit 42. The control unit 42 generates an object image 51 with diagonal lines added based on the recognition state included in the robot recognition information from the communication unit 44, supplies the object image 51 to the display unit 43, and causes it to be displayed on the display screen 21a.

[0079] As a result, an object image 51 with diagonal lines added thereto is displayed on the display screen 21a, as shown in FIG.

[0080] The diagonal lines added to the object image 51 indicate that the object "bird figurine" on the object image 51 is recognized by the robot 22 as "not a target."

[0081] Therefore, the user can easily predict that the robot 22 recognizes the "bird figurine" on the object image 51 as "not a target."

[0082] When the robot 22 detects the object "Juhachicha" from the captured image obtained by capturing an image of the room, the robot 22 extracts an object image 52 representing the object "Juhachicha" from the captured image.

[0083] The robot 22 transmits the robot recognition information including the extracted object image 52 to the communication unit 44.

[0084] In response to this, the communication unit 44 supplies the robot recognition information from the robot 22 to the control unit 42. The control unit 42 supplies the object image 52 included in the robot recognition information from the communication unit 44 to the display unit 43, causing it to be displayed on the display screen 21a.

[0085] 10, an object image 52 of the object "Juhachicha" detected by the robot 22 is displayed on the left side of the display screen 21a. Therefore, the user can easily predict that the robot 22 has detected the object "Juhachicha."

[0086] The robot 22 also calculates a score for the object "Juhachicha" from an image captured near the object "Juhachicha." If the robot 22 recognizes the object "Juhachicha" as a "target" based on the calculated score, it transmits robot recognition information including a recognition state indicating that the object "Juhachicha" is a "target" to the communication unit 44.

[0087] In response to this, the communication unit 44 supplies the robot recognition information from the robot 22 to the control unit 42. Based on the robot recognition information from the communication unit 44, the control unit 42 generates an object image 52 surrounded by a rectangle and indicated by a triangle mark, and an object image 51 with a diagonal line added, and supplies them to the display unit 43 to be displayed on the display screen 21a.

[0088] As a result, on the display screen 21a, as shown in Figure 11, an object image 52 surrounded by a thick rectangle and indicated by a triangle mark (▲ shown in Figure 11) is displayed on the left side of the figure, and an object image 51 with diagonal lines added is displayed to the right of the object image 52.

[0089] The thick rectangle surrounding the object image 52 indicates that the object "Juhachicha" in the object image 52 is recognized by the robot 22 as a "target." The triangle pointing at the object image 52 indicates that the robot 22 intends to take the object "Juhachicha" in the object image 52 home with him.

[0090] Furthermore, in this case, the control unit 42 displays object images representing the objects from the left to the right on the display screen 21a in descending order of the object scores.

[0091] The display position of the object image is determined based on the score of the object. Therefore, for example, the display interval between the first object image and the second object image represents the difference in the object scores between the first object image and the second object image.

[0092] Therefore, the user can intuitively grasp the score of the object represented by the object image according to the position on the display screen 21a where the object image is displayed.

[0093] From the display screen 21a as shown in FIG. 11, it can be easily predicted that the robot 22 will carry out the action of bringing the object "Juhachicha."

[0094] In addition, when the robot 22 moves autonomously and detects a new object from the images captured by the built-in camera, it generates robot recognition information and transmits it to the communication unit 44 in the same manner as described above.

[0095] In response to this, the control unit 42 supplies the robot recognition information from the communication unit 44 to the display unit 43, causing it to be displayed on the display screen 21a.

[0096] In this way, the control unit 42 updates the display on the display screen 21a based on the robot recognition information transmitted from the robot 22 via the communication unit 44.

[0097] That is, for example, the display screen 21a is then updated to newly display an object image 53 representing the object "ABC tea" as shown in Fig. 12 in accordance with the behavior of the robot 22. Furthermore, the display screen 21a is updated to newly display an object image 54 representing the object "C carbonated juice" and an object image 55 representing the object "USB memory" as shown in Fig. 13 in accordance with the behavior of the robot 22. Furthermore, for example, the display screen 21a is updated to change the position of each object image according to the score as shown in Fig. 14 in accordance with the behavior of the robot 22.

[0098] It should be noted that the object image 51 displayed on the display screen 21a shown in FIG. 13 is not displayed on the display screen 21a in FIG.

[0099] In FIG. 14, the further to the left an object is displayed on the display screen 21a, the higher its score, and the score of the object is represented by the position where the object image is displayed.

[0100] Therefore, when the object images 52 to 55 are displayed at positions according to the scores of the corresponding objects, the object image 51 cannot be displayed due to the size of the display screen 21a and the object image.

[0101] In FIG. 14, the score of the object "bird figurine" in object image 51 is assumed to be sufficiently smaller than the score of the object "USB memory" in object image 55.

[0102] 14, the score of the object "ABC Tea" in the object image 53 is smaller than the score of the object "Juhachi Tea" in the object image 52.

[0103] However, the triangle mark points to the object "ABC Tea" in the object image 53, and the robot 22 is trying to take home the object "ABC Tea." This is because the robot 22 determines the object to take home by taking into consideration not only the object's score but also the location where the object is located and the user's preferences.

[0104] In the first embodiment, the objects are arranged and displayed in descending order of their scores from left to right on the display screen 21a as shown in Figures 6 to 14. However, the objects can be displayed in any direction (for example, from right to left) other than the right direction.

[0105] Also, for example, in the first embodiment, as shown in Figures 6 to 14, the objects are displayed in an arbitrary direction in descending order of their scores, and the object images are displayed at positions according to the object scores.

[0106] However, the object images may be displayed on the display screen 21a using only the order of the scores. Alternatively, for example, the object images may be displayed on the display screen 21a at positions according to the scores of the objects, without using the order of the scores.

[0107] Furthermore, for example, at least one of the order, position, size, brightness (luminance, etc.), clarity, and color of the object images may be changed and displayed according to the score of the object.

[0108] Furthermore, for example, the score of an object may be displayed in association with the object image. The score may be displayed, for example, as a bar graph or a pie chart. When a pie chart is used to display the score, the percentage of the score of the object currently having the highest score may be displayed as a pie chart.

[0109] Furthermore, for example, the change in the score of the object may be displayed by an arrow or the like on the display screen 21a.

[0110] Specifically, for example, as shown in Figure 15, when the score of the object "C carbonated juice" corresponding to the object image 54 is changing to increase, a white arrow indicating an increase in the score of the object "C carbonated juice" is displayed on the display screen 21a in association with the object image 54.

[0111] Also, for example, as shown in Figure 15, if the score of the object "Juhachi Tea" corresponding to object image 52 is changing to decrease, a black arrow indicating a decrease in the score of the object "Juhachi Tea" is displayed on display screen 21a in association with object image 52.

[0112] Furthermore, for example, as shown in FIG. 15, if the score of the object "USB memory" corresponding to object image 55 is changing to decrease, a black arrow indicating the decrease in the score of the object "USB memory" is displayed on display screen 21a in association with object image 55.

[0113] 15 is displayed on the display screen 21a, the user can predict that the robot 22 will mistakenly bring the object "C carbonated juice" instead of the object "Juhachi tea." In this case, the user can clearly instruct the robot 22 to bring the object "Juhachi tea," for example, before the robot 22 mistakenly brings the object "C carbonated juice."

[0114] Furthermore, when the content as shown in Fig. 16 is displayed on the display screen 21a, the user can predict that the robot 22 will bring the object "Juhachicha" that the user desires. In this case, the user can concentrate on work or the like until the robot 22 brings the object "Juhachicha" without giving any instructions to the robot 22. Note that Fig. 16 has the same configuration as Fig. 15, and therefore a description of Fig. 16 will be omitted.

[0115] Furthermore, for example, on the display screen 21a, the change in the score of the object may be displayed as a bar graph or a line graph instead of an arrow, etc. In addition, for example, the score history may be displayed together with or instead of the change in the score of the object.

[0116] When score changes and history are displayed, the user can understand the changes and history of the object's score by referring to the display screen 21a, making it easy to predict changes in the score ranking and ultimately predicting the behavior of the robot 22 in advance.

[0117] These displays are realized by the control unit 42 referring to the robot recognition information stored in the storage unit 45.

[0118] Furthermore, when a change in the ranking of the score occurs, the indicator device 21 can notify the user by making a sound, vibrating, or flashing a lamp.

[0119] In the first embodiment, as shown in Figures 6 to 14, the object image of an object recognized by the robot 22 as a "target" is surrounded by a thick rectangle, and the object image of an object recognized as a "possible target" is displayed as is.

[0120] Furthermore, for example, the object image of an object that is recognized by the robot 22 as "not yet known" whether it is a target is displayed surrounded by a thin-line rectangle, and the object image of an object that is recognized as "not a target" is displayed with diagonal lines added.

[0121] However, the display method for representing each recognition state is not limited to rectangles, diagonal lines, and the like.

[0122] Furthermore, for example, a display method for representing the recognition state and a display method for representing the object's score may be used in combination, or only one of the display method for representing the recognition state and the display method for representing the object's score may be used to display on the display screen 21a.

[0123] Next, FIG. 17 shows an example of a case where detailed information of a selected object image is displayed in response to a user's selection operation on the object image on the display screen 21a.

[0124] The display screen 21a shown in Figure 17 mainly displays object image 52a representing the bottom portion of the object "Juhachicha" (a plastic bottle), and object images 52b to 52e obtained by capturing images of the object "Juhachicha" from different directions as detailed information about the object "Juhachicha."

[0125] For example, the robot 22 extracts an object image from the latest captured image of the object as the object image to be displayed on the display screen 21a, includes the object image in the robot recognition information, and transmits the information to the instruction device 21.

[0126] In addition, the robot 22 can use, as the object image to be included in the robot recognition information, an object image extracted from the most recent captured image, as well as an object image extracted from the highest resolution captured image among multiple captured images that show the object.

[0127] Furthermore, the object image to be included in the robot recognition information may be an extracted image from among a plurality of object images that clearly represents the entire object, or a typical extracted image may be used. Note that a typical extracted image may be, for example, an extracted image obtained by capturing an image of the object from a 45-degree angle diagonally above.

[0128] For example, if the robot 22 has acquired an image in the past that includes part of the object images 52b to 52e, but has acquired an image in the most recent image that includes part of the object image 52a, a robot recognition image including the object image 52a, which shows the bottom part of the object "Juhachicha", is transmitted to the instruction device 21.

[0129] 17, the pointing device 21 displays an object image 52a showing the bottom part of the object "Juhachicha" on the display screen 21a. In this case, when the user sees the object image 52a displayed on the display screen 21a, it may not be possible for the user to understand that the object represented by the object image 52a is "Juhachicha."

[0130] Therefore, the robot 22 transmits robot recognition information including the object image 52b to 52e in addition to the object image 52a to the communication unit 44 of the instruction device 21. In this case, the communication unit 44 supplies the robot recognition information from the robot 22 to the control unit 42.

[0131] Based on the robot recognition information from the communication unit 44, the control unit 42 causes the display screen 21a of the display unit 43 to display, from the left in the figure, an object image of the object "glue stick," an object image 52a of the object "Juhachicha," and an object image of the object "wallet," as shown in Figure 17.

[0132] Furthermore, the control unit 42 supplies the robot recognition information from the communication unit 44 to the storage unit 45 to store it.

[0133] In the pointing device 21, when the user uses the operation unit 41 to perform a selection operation to select the object image 52a on the display screen 21a, the operation unit 41 supplies an operation signal corresponding to the user's selection operation to the control unit 42.

[0134] Then, in response to an operation signal from the operation unit 41, the control unit 42 reads out the object images 52b to 52e included in the robot recognition information stored in the memory unit 45, supplies them to the display unit 43, and displays them on the display screen 21a.

[0135] As a result, as shown in FIG. 17, object images 52b to 52e are displayed on the display screen 21a as detailed information about the object "Juhachicha."

[0136] Therefore, even if the user cannot identify the object in object image 52a from object image 52a, the user can easily identify that the object in object image 52a is "Juhachicha" from object images 52b to 52e that are displayed in response to the user's selection operation.

[0137] Additionally, the control unit 42 can also display an enlarged version of the object image 52a as detailed information when, for example, a user performs a selection operation in response to an operation signal from the operation unit 41. In this case, for example, the control unit 42 reads out the object image 52a included in the robot recognition information stored in the storage unit 45, enlarges it at a predetermined magnification rate, and displays it on the display screen 21a.

[0138] Alternatively, for example, the control unit 42 may cause the display screen 21a to display the three-dimensional position where the object "Juhachi Tea" on the object image 52a is detected as detailed information of the object image 52a. In this case, the storage unit 45 stores robot recognition information including the three-dimensional position of the object "Juhachi Tea" on the object image 52a, along with the object image 52a.

[0139] Incidentally, when the instruction device 21 is, for example, a personal computer to which a mouse is connected as the operation unit 41, the user uses the mouse to perform a selection operation (mouse-over) to select the object image 52a by moving a cursor or the like over the object image 52a on the display screen 21a.

[0140] Of the multiple object images, only a predetermined number of object images according to the screen size of the display screen 21 a are displayed on the display screen 21 a. For this reason, the display screen 21 a is configured to display only object images with sufficiently high scores, object images with the top n scores, and object images that are determined to be at least one of "is a target" and "may be a target."

[0141] Therefore, since there are object images that are not displayed on the display screen 21a, it is desirable to configure the display screen 21a so that such object images can be displayed on the display screen 21a in response to a user operation.

[0142] That is, for example, the user can perform a scrolling operation by rotating the scroll button on the mouse to reduce the size of each object image displayed on the display screen 21a, thereby displaying more object images on the display screen 21a.

[0143] In this case, in response to the supply of an operation signal corresponding to the scroll operation from the operation unit 41, the control unit 42 reads out the robot recognition information already stored in the storage unit 45. Then, based on the read out robot recognition information, the control unit 42 generates object images reduced in size in response to the scroll operation, supplies the images to the display unit 43, and causes them to be displayed on the display screen 21a.

[0144] Furthermore, for example, the user can enlarge each object image displayed on the display screen 21a by performing a scroll operation, thereby displaying fewer object images on the display screen 21a.

[0145] Further, on the display screen 21a, for example, in the center thereof, an object image with a score of x1 is displayed, an object image with a score greater than x1 is displayed on the left side, and an object image with a score smaller than x1 is displayed on the right side.

[0146] On the display screen 21a, according to a user operation, an object image with a score of x2 (<x1) smaller than the score x1 can be displayed at the center of the display screen 21a, an object image with a score greater than the score x2 can be displayed on the left side, and an object image with a score smaller than the score x2 can be displayed on the right side.

[0147] That is, for example, when the user operates the operation unit 41 so that the cursor or the like is moved to the position where the object image with the score x2 is displayed and that position is set as the center position of the display screen 21a, in the control unit 42, a robot recognition image for displaying the object image with the score x2 at the center position of the display screen 21a is generated and displayed on the display screen 21a of the display unit 43.

[0148] In addition, for example, when the user operates the operation unit 41 so that the position where the object image with the score x1 is displayed is set as the center of the display screen 21a, the display on the display screen 21a is set to the original display in which the object image with the score x1 is displayed at the center thereof.

[0149] Also, for example, when the user performs an enlargement operation using the operation unit 41 to enlarge and display a predetermined area among all the areas on the display screen 21a, the predetermined area on the display screen 21a is enlarged and displayed.

[0150] [Explanation of the operation when the pointing device 21 displays a robot recognition image] Next, referring to the flowchart of FIG. 18, the display process performed by the pointing device 21 will be described.

[0151] This display process is started, for example, when robot recognition information is transmitted from the robot 22.

[0152] In step S21, the communication unit 44 receives robot recognition information from the robot 22 under the control of the control unit 42, and supplies the robot recognition information to the control unit 42.

[0153] In step S22, the control unit 42 generates a robot recognition image based on the robot recognition information from the communication unit 44, supplies it to the display unit 43, and causes it to be displayed on the display screen 21a.

[0154] Furthermore, the control unit 42 supplies the robot recognition information from the communication unit 44 to the storage unit 45 for storage.

[0155] In step S23, the control unit 42 determines whether or not a selection operation has been performed by the user, depending on whether or not an operation signal corresponding to a selection operation by the user has been supplied from the operation unit 41.

[0156] Then, in step S23, if the control unit 42 determines that the user has not performed a selection operation, it returns the process to step S21 and repeats the same processes thereafter.

[0157] Furthermore, in step S23, if the control unit 42 determines that a selection operation has been performed by the user, depending on whether an operation signal corresponding to the selection operation by the user has been supplied from the operation unit 41, the control unit 42 proceeds to step S24. In this case, for example, it is assumed that a selection operation has been performed to display detailed information about the object image 52a shown in Fig. 17.

[0158] In this case, in step S24, the control unit 42 reads out the robot recognition information stored in the storage unit 45, and supplies the object images 52b to 52e included in the read out robot recognition information to the display unit 43 to display them on the display screen 21a. Thereafter, the process returns to step S21, and the same processes are performed thereafter. Note that this display process is ended, for example, when the robot 22 no longer transmits robot recognition information.

[0159] As described above, according to the display processing, the display screen 21a as shown in Figures 6 to 17 is displayed based on the robot recognition information from the robot 22, so that the behavior of the robot 22 can be easily predicted.

[0160] Furthermore, according to the display processing, in step S23, in response to a selection operation to select object image 52a in FIG. 17, for example, object images 52b to 52e of the object shown in object image 52a as viewed from other directions are displayed in step S24.

[0161] This allows the user to more accurately grasp the object of the object image displayed on the display screen 21a.

[0162] [Example of robot 22 configuration] Next, FIG. 19 shows an example of the configuration of the robot 22.

[0163] The robot 22 is composed of a communication unit 61 , a camera 62 , a distance sensor 63 , a microphone 64 , a speaker 65 , a control unit 66 , a driving unit 67 , and a memory unit 68 .

[0164] The communication unit 61 receives instruction information, feedback information, designated range information, etc. from the instruction device 21 and supplies them to the control unit 66.

[0165] The communication unit 61 transmits the robot recognition information from the control unit 66 to the instruction device 21 .

[0166] The camera 62 corresponds to an "eye" that senses light, and is configured, for example, with a CCD (Charge Coupled Device) image sensor, etc. The camera 62 captures images of the surroundings of the robot 22 and supplies the captured images to the control unit 66.

[0167] The distance sensor 63 is a sensor that measures the distance from the robot 22 to an object, and supplies the measured distance to the object to the control unit 66 .

[0168] The microphone 64 corresponds to an “ear” that detects sound, collects sound, and supplies the collected sound signal to the control unit 66 .

[0169] The speaker 65 corresponds to the “mouth” of the robot 22 and outputs a predetermined sound according to the control of the control unit 66 .

[0170] The control unit 66 controls the communication unit 61 to the speaker 65 and the drive unit 67. That is, for example, the control unit 66 grasps the surrounding situation and the like based on the captured image from the camera 62, the distance from the distance sensor 63, the audio signal from the microphone 64, etc., and controls the drive unit 67 according to the grasped surrounding situation.

[0171] Furthermore, the control unit 66 controls the driving unit 67 based on the instruction information from the communication unit 61 to autonomously perform the action instructed by the user. That is, for example, the control unit 66 controls the driving unit 67 to cause the robot 22 to autonomously search for an object belonging to a category indicated by the category information included in the instruction information as a target.

[0172] Furthermore, the control unit 66 detects an object in the captured image based on the captured image from the camera 62, and extracts an object image representing the detected object. The control unit 66 also calculates a score for the detected object based on the captured image from the camera 62.

[0173] That is, for example, the control unit 66 reads out a classifier for a targeted object from the storage unit 68 based on instruction information from the communication unit 61. The control unit 66 also extracts feature quantities representing the characteristics of the detected object from the captured image captured by the camera 62.

[0174] Then, the control unit 66 calculates a score for the detected object using the read classifier based on the extracted feature amount. That is, for example, the control unit 66 can use the classification result (score) obtained from the classifier as the score for the detected object as is, or can calculate a final score by integrating the classification results obtained from the classifier in a time series.

[0175] The control unit 66 determines the recognition state of the object based on the calculated score of the object. The determination of the recognition state will be described later with reference to FIG.

[0176] Furthermore, the control unit 66 determines an object image to be displayed on the display screen 21a based on the calculated object score and the determined recognition state of the object. Note that a method for determining an object image to be displayed on the display screen 21a will be described later with reference to FIG.

[0177] The control unit 66, for example, changes the object image decided to be displayed on the display screen 21a (for example, by surrounding it with a rectangle or adding diagonal lines) depending on the recognition state of the object, generates robot recognition information including the changed object image, the object score, display target information, etc., and supplies it to the communication unit 61.

[0178] Furthermore, the control unit 66 controls the distance sensor 63 to measure the distance to the detected object, and the distance sensor 63 then supplies the control unit 66 with the distance to the object detected by the control unit 66.

[0179] The control unit 66 detects the three-dimensional position of the camera 62 (robot 22) based on the captured image from the camera 62 and the attitude (position and direction) of the camera 62. Details of the method for detecting the three-dimensional position of the robot 22 are described in, for example, Japanese Patent Application Laid-Open No. 2008-304268.

[0180] The control unit 66 may detect the three-dimensional position of the robot 22 using a positioning technique that uses, for example, a global positioning system (GPS) or Wi-Fi.

[0181] The control unit 66 calculates the three-dimensional position of the detected object based on the detected three-dimensional position and the distance supplied from the distance sensor 63, associates it with the captured image in which the object appears, and supplies it to the memory unit 68 for storage.

[0182] In addition, the control unit 66 reads out captured images (for example, the surrounding image 31 and partial image 35 in Figure 3) stored in the memory unit 68, generates a robot recognition image that includes the read captured images, and supplies it to the communication unit 61.

[0183] The control unit 66 detects a designated area representing an area designated by a designation operation by the user, based on the designated range information from the communication unit 61 and the captured image stored in the storage unit 68.

[0184] Furthermore, the control unit 66 reads out the three-dimensional position associated with the detected object in the designated area from the storage unit 68, and calculates a search range corresponding to the designated area based on the read three-dimensional position.

[0185] Then, the control unit 66 controls the drive unit 67 to drive the parts of the robot 22 corresponding to the hands and feet, and causes the robot 22 to perform an operation to search for a target (an object belonging to the category represented by the category information) within the calculated search range.

[0186] The driving unit 67 drives the parts of the robot 22 that correspond to the "hands" and "feet" under the control of the control unit 66. This allows the robot 22 to act autonomously.

[0187] The storage unit 68 stores, for example, a control program to be executed by the control unit 66 in advance, and also stores (holds) data that the control unit 66 instructs to write.

[0188] In addition, the memory unit 68 stores an identifier for identifying each object of a plurality of items. This identifier is a function that takes the feature amount of an object as input and outputs the score of the object. Note that this identifier is assumed to be generated and stored by pre - performed learning.

[0189] Furthermore, the memory unit 68 stores the captured image from the control unit 66 (a captured image in which the three - dimensional position of an object on the captured image is associated, such as the surrounding image 31 or the partial image 35, etc.).

[0190] Next, FIG. 20 shows an example of a state transition diagram of the recognition state of an object.

[0191] In FIG. 20, as recognition states recognized by the control unit 66, "not detected (detected)", "not yet known", "may be a target", "is a target", and "is not a target" are shown.

[0192] The control unit 66 recognizes the object as being in any of the states of "not yet known", "may be a target", "is a target", or "is not a target" according to the calculated score of the object.

[0193] That is, for example, when the score of an object is greater than or equal to the threshold th1, the control unit 66 recognizes the object as "being a target", and when the score of the object is less than the threshold th1 and greater than or equal to the threshold th2 (<th1), it recognizes it as "may be a target".

[0194] Also, for example, when the score of an object is less than the threshold th2 and greater than or equal to the threshold th3 (<th2), the control unit 66 recognizes the object as "not yet known" whether it is a target or not, and when the score of the object is less than the threshold th3, it recognizes it as "is not a target".

[0195] In addition, except when a feedback operation is performed by the user, when the control unit 66 recognizes an object as a "target," it always goes through a state in which it is recognized as a "possibly target."

[0196] Therefore, on the display screen 21a, it is possible to prevent an object that is recognized as "not yet known" whether it is a target or not (corresponding to an object image surrounded by a thin-line rectangle) from suddenly becoming an object that is recognized as "a target" (corresponding to an object image surrounded by a thick-line rectangle).

[0197] Therefore, the user only needs to pay attention to whether there are any objects on the display screen 21a that are recognized as "possible targets" (corresponding to the object images displayed as they are) that are not suitable as targets.

[0198] In this way, when recognizing an object as "a target," it must first be recognized as "possibly a target." This is because, if an object that is recognized on display screen 21a as "still unclear" as to whether it is a target or not, suddenly becomes an object recognized as "a target," the user would have to pay attention to all objects on display screen 21a, which would be extremely troublesome.

[0199] Next, FIG. 21 shows an example of a method by which the control unit 66 determines an object image to be displayed on the display screen 21a according to the score of the object.

[0200] In FIG. 21, an object 71, an object 72, an object 73, . . . an object 74 are shown in descending order of score from the left side of the drawing.

[0201] The control unit 66 recognizes the object 71 as a "target" based on the calculated score of the object 71. The control unit 66 also recognizes the object 72 as a "possible target" based on the score of the object 72.

[0202] Furthermore, the control unit 66 recognizes that it is "still unclear" whether the object 73 is a target or not based on the score of the object 73. Furthermore, the control unit 66 recognizes that the object 74 is "not a target" based on the score of the object 74.

[0203] For example, as shown in FIG. 21, the control unit 66 can determine that, among the multiple detected objects, object images corresponding to the top N objects with the highest scores are to be displayed on the display screen 21a of the indicating device 21.

[0204] Furthermore, for example, the control unit 66 may determine that, among the multiple detected objects, object images corresponding to objects that are recognized as "targets" and "possibly targets" are to be displayed on at least the display screen 21a of the indication device 21, as shown in FIG. 21.

[0205] [Explanation of how Robot 22 calculates the score] Next, the score calculation process performed by the robot 22 will be described with reference to the flowchart of FIG.

[0206] This score calculation process is started, for example, when the robot 22 searches the room based on instruction information from the instruction device 21.

[0207] In step S41, the camera 62 captures an image of the surroundings of the robot 22 and supplies the captured image to the control unit 66.

[0208] In step S42, the control unit 66 attempts to detect an object in the captured image based on the captured image from the camera 62.

[0209] Then, the control unit 66 proceeds to step S43 in response to detecting an object in the captured image based on the captured image from the camera 62. Note that if the control unit 66 does not detect an object in the captured image based on the captured image from the camera 62, the control unit 66 returns the process to step S41, and similar processes are performed thereafter.

[0210] The control unit 66 determines a target representing a processing object of the robot 22 based on instruction information supplied from the instruction device 21 via the communication unit 61, and reads out a classifier for the determined target from the storage unit 68.

[0211] In step S43, the control unit 66 calculates the feature amount of the detected object from the image captured by the camera 62. Then, the control unit 66 calculates the score of the object using the classifier read out from the storage unit 68 based on the calculated feature amount.

[0212] In step S44, the control unit 66 generates robot recognition information including the calculated score and the captured image stored in the storage unit 68, and supplies the information to the communication unit 61.

[0213] In step S45, the communication unit 61 transmits the robot recognition information from the control unit 66 to the instruction device 21, and the process returns to step S41, and the same processes are carried out thereafter.

[0214] The score calculation process is terminated, for example, when the robot 22 finds the target and brings it back to the user.

[0215] As described above, according to the score calculation process, the score of the detected object is calculated, and robot recognition information including the calculated score and the like is transmitted to the instruction device 21.

[0216] Therefore, since it becomes possible to display a robot recognition image such as those shown in FIGS. 6 to 17 on the display screen 21a of the instruction device 21, the user can easily predict the behavior of the robot 22.

[0217] <2. Modification of the First Embodiment> [Other examples of robot recognition images] Incidentally, the robot recognition image in which an object with a higher score is displayed on the left side of the display screen 21a has been described with reference to FIGS. 6 to 17, but the robot recognition image is not limited to this.

[0218] Next, FIG. 23 shows another example of the display of the robot recognition image.

[0219] 23, on an upper screen 81 representing the upper screen of the display screen 21a, the object images of objects with higher scores are displayed in order of the robot recognition images located to the left of the display screen 21a. In this case, the scores are highest in the order of object 91, object 92, object 93, and object 94.

[0220] Furthermore, the latest captured image obtained by the camera 62 is displayed on a lower screen 82 representing the upper screen of the display screen 21a. As shown in Fig. 23, this captured image shows an object 91 ("apple juice") and an object 92 ("C carbonated juice") displayed on the upper screen 81.

[0221] On the display screen 21a, the object 91 on the upper screen 81 and the object 91 ("apple juice") on the lower screen 82 are connected by a dotted line, so that the correspondence between the objects 91 displayed on the upper screen 81 and the lower screen 82 can be seen.

[0222] The same applies to the object 92. In this case, the robot recognition information includes the latest captured image obtained by the camera 62.

[0223] That is, in FIG. 23, the latest captured image obtained by imaging with the camera 62 is displayed on the lower screen 82, which is different from the cases described with reference to FIGS.

[0224] 23, when the upper screen 81 is viewed, the object images of the objects 91 and 92 are each surrounded by a thick rectangle, and therefore it is predicted that the robot 22 recognizes the objects 91 and 92 as targets. Also, when the lower screen 82 is viewed in FIG. 23, it is predicted that the robot 22 is located near the objects 91 and 92.

[0225] Therefore, by referring to the upper screen 81 and the lower screen 82 of Figure 23, the user can more easily guess that the robot 22 will take action to bring back either the object 91 or the object 92 as a target, compared to when referring to only the upper screen 81 of Figure 23.

[0226] 23, the latest captured image obtained by the camera 62 is displayed on the lower screen 82, but it is also possible to display, for example, a captured image obtained by the camera 62 at any time. Also, a composite image (such as a panoramic view image) created by combining a plurality of captured images obtained by the camera 62 may be displayed.

[0227] Furthermore, for example, the display screen 21a may display a thick rectangle or the like superimposed directly on the captured image obtained by the camera 62 (for example, the apple juice in the captured image is surrounded by a thick rectangle). In this case, the image displayed on the upper screen 81 shown in Fig. 23 is not displayed on the display screen 21a, and only the captured image is displayed.

[0228] 24, instead of the latest captured image captured by the camera 62, a spatial map representing the interior of the room may be displayed on the lower screen 82 of the display screen 21a. In this case, the robot recognition information includes spatial information representing the spatial map of the room. Note that the spatial information is assumed to be stored in advance in the storage unit 68 of the robot 22, for example.

[0229] 24, the spatial map displayed on the lower screen 82 shows the path along which the robot 22 will move with a dotted line 101. In Fig. 24, the objects displayed on the upper screen 81 and the positions on the spatial map where the objects exist are connected by dotted lines, making it possible to see the correspondence between them.

[0230] Therefore, by referring to the upper screen 81 and the lower screen 82 of the display screen 21a of FIG. 24, the user can infer that the robot 22 will take home the object 91 ("apple juice") as a target, compared to when the user refers to only the upper screen 81 of the display screen 21a of FIG. 24.

[0231] When displaying the display screen 21a as shown in Figures 23 and 24, it is desirable to preferentially display objects that exist near the robot 22 (objects that exist within a predetermined range from the position of the robot 22) as object images.

[0232] Furthermore, on the display screen 21a, for example, the captured image is displayed on the lower screen 82 shown in FIG. 23, and the spatial diagram is displayed on the lower screen 82 shown in FIG. 24, but it is also possible to display, for example, both the captured image and the spatial diagram.

[0233] In this case, for example, the display screen 21a displays a display such as that shown in FIG. 23, as well as a space diagram including a dotted line 101 indicating the path along which the robot 22 will move.

[0234] Next, FIG. 25 shows yet another display example of the robot recognition image.

[0235] 25, the display screen 21a displays the latest captured image obtained by the camera 62 of the robot 22. This captured image mainly shows objects 111 to 114 detected by the robot 22.

[0236] The actions that the robot 22 is about to take with respect to the objects 111 and 113 are displayed in speech bubbles 111a to 113a, respectively.

[0237] The speech bubble 111a displays the action "throw away", the speech bubble 112a displays the action "bring away", the speech bubble 113a displays the action "bring away", and the speech bubble 114a displays the action "throw away". When the display screen 21a as shown in Fig. 25 is displayed, the robot recognition information from the robot 22 includes processing information representing the processing (e.g., "throw away") of the robot 22 that targets an object (e.g., a "bird figurine").

[0238] The size of the speech bubbles 111a to 113a is determined according to the score of the object to which the speech bubble is attached, for example. The score may be represented by the color of the speech bubble in addition to the size of the speech bubble.

[0239] In the first embodiment, the user uses the instruction device 21 to instruct the robot 22 to bring "tea" as a target, but the instruction is not limited to this and the user can also instruct the robot 22 to, for example, throw the target in a trash can.

[0240] In addition, for example, the user can use the instruction device 21 to give instructions such as "grab," "pick up," and "put away" the target.

[0241] Figure 25 shows an example of a display that is displayed on the display screen 21a of the instruction device 21 when, for example, a user uses the instruction device 21 to instruct the robot 22 to bring "juice" and to throw away "small items" in a trash can.

[0242] That is, the robot recognition image displayed on the display screen 21a may be displayed in any manner as long as it allows the user to intuitively predict the behavior of the robot 22.

[0243] Furthermore, for example, the user can select a person as a target in addition to "tea" or the like, using the instruction device 21. In this case, the user can use the instruction device 21 to give an instruction such as "call" the person as the target.

[0244] In this case, for example, the name of the detected person may be displayed on the display screen 21a together with the person's name. Note that the person's name is assumed to be stored in advance in the storage unit 68 of the robot 22 in association with a classifier for identifying the person.

[0245] Furthermore, if a different person's name is displayed on the display screen 21a as the name representing the detected person, the user can use the operation unit 41 to change the name to the name of the detected person.

[0246] Incidentally, for example, the instruction device 21 displays a robot recognition image including object images 121 to 124 on the display screen 21a, as shown in FIG. 26, based on robot recognition information from the robot 22.

[0247] The display screen 21a shown in FIG. 26 displays that the robot 22 recognizes the object "Juhachicha" corresponding to the object image 121 as "possibly a target."

[0248] Furthermore, it is displayed that the robot 22 recognizes the object "C carbonated juice" corresponding to the object image 122 and the object "potted plant" corresponding to the object image 123 as "not sure yet" whether they are targets or not, and that it recognizes the object "stuffed toy" corresponding to the object image 124 as "not a target."

[0249] 26, the user can infer the behavior of the robot 22 as follows: For example, the user can infer that the robot 22 will eventually recognize the object "Juhachi Tea" that it has recognized as "a possible target" as "a target" and will take the object "Juhachi Tea" home with it.

[0250] In this case, the user may wait until the robot 22 returns with the object "Juhachicha."

[0251] Also, for example, on the display screen 21a shown in FIG. 27, a thick rectangle is displayed indicating that the robot 22 recognizes both the object "ABC Tea" corresponding to the object image 141 and the object "S Tea" corresponding to the object image 142 as "targets."

[0252] Furthermore, for example, on the display screen 21a shown in FIG. 27, a triangle mark is displayed indicating that the robot 22 is about to take home the object "ABC tea."

[0253] Therefore, the user can infer that the robot 22 is unsure whether to take home the object "ABC tea" or the object "S tea", but is planning to take home the object "ABC tea" for the time being.

[0254] If the user is happy with either the object "ABC tea" or the object "S tea," the user does not need to operate the instruction device 21 and simply waits for the robot 22 to bring back either the object "ABC tea" or the object "S tea."

[0255] For example, when the triangle mark is not displayed on the display screen 21a of FIG. 27 and the user wants either the object "ABC tea" or the object "S tea" to be brought quickly, it is desirable to be able to instruct the robot 22 to bring, for example, the object "ABC tea."

[0256] Therefore, for example, as shown in FIG. 28, the instruction device 21 allows the user to indicate, by operation of the user, the object image 162 corresponding to the object "Juhachicha" that the user wants to be brought, from among the object images 161 to 165 displayed on the display screen 21a.

[0257] In FIG. 28, for example, when a triangle mark is not displayed on the display screen 21a, the object image 162 corresponding to the object "Juhachicha" that the user wants to be brought is surrounded by a thick rectangle in response to the instruction given by the user's operation, and the object image 162 indicated by the triangle mark is displayed.

[0258] Also, for example, on the display screen 21a as shown in FIG. 29, when a triangle mark indicating the object "ABC Tea" corresponding to the object image 182 among the object images 181 to 187 is displayed, it is inferred that the robot 22 will take the object "ABC Tea" home with it.

[0259] If the user wants to take home the object "S Tea" instead of the object "ABC Tea", the user can operate the operation unit 41 of the instruction device 21 to instruct the object "S Tea" corresponding to the object image 184 to be brought instead of the object "ABC Tea".

[0260] That is, for example, as shown in FIG. 30, the user can specify the object image 184 on the display screen 21a using a pointer (arrow) or the like, and instruct the object "S Brown" corresponding to the object image 184 to be brought.

[0261] As a result, on the display screen 21a, instead of the triangular mark (Figure 29) that pointed to the object "ABC Tea" corresponding to the object image 182, a triangular mark (Figure 30) that points to the object "S Tea" corresponding to the object image 184 is displayed.

[0262] 3. Second Embodiment [Example of a case where a user looking at the display screen 21a gives instructions to the robot 22] Next, FIGS. 31 to 38 show an example of a case where the user operates the operation unit 41 of the instruction device 21 to perform a feedback operation for giving an instruction to the robot 22. FIG.

[0263] FIG. 31 shows an example in which the user performs a feedback operation using the operation unit 41 of the instruction device 21 while referring to the display screen 21a of the instruction device 21.

[0264] FIG. 31 shows object images 201 to 204 displayed on the display screen 21a. The object images 201 and 202 are surrounded by a rectangle indicating that the recognition state of the robot 22 is "target."

[0265] Also, the object image 203 is displayed as is, indicating that the robot 22's recognition state for the object "C carbonated juice" is "not yet known." Furthermore, the object image 204 has a diagonal line added to it, indicating that the robot 22's recognition state is "not a target."

[0266] The user can use the operation unit 41 of the instruction device 21 to perform a feedback operation to change the recognition state of the object "C carbonated juice" represented by the object image 203, for example, from "not sure yet" to "it is the target," as shown by the arrow 221.

[0267] Also, for example, the user can use the operation unit 41 of the instruction device 21 to perform a feedback operation to change the recognition state of the object "Juhachicha" represented by the object image 202 from "is the target" to "is not the target," as shown by the arrow 222.

[0268] The control unit 42 generates corresponding feedback information in response to an operation signal corresponding to a feedback operation from the operation unit 41, and supplies the generated feedback information to the communication unit 44. The communication unit 44 supplies the feedback information from the control unit 42 to the robot 22.

[0269] The robot 22 recalculates the scores of the objects based on the feedback information from the communication unit 44, and changes the recognition state of each object based on the resulting scores.

[0270] Next, FIG. 32 shows an example in which the control unit 66 of the robot 22 changes the recognition state of an object in response to feedback information from the instruction device 21.

[0271] For example, when a user performs a feedback operation using the operation unit 41 in the instruction device 21, the operation unit 41 supplies an operation signal corresponding to the feedback operation to the control unit .

[0272] The control unit 42 generates feedback information based on the operation signal from the operation unit 41 and supplies it to the communication unit 44. The communication unit 44 transmits the feedback information from the control unit 42 to the robot 22.

[0273] In this case, as shown in Figure 31, consider the case where, for example, the user uses the operation unit 41 of the instruction device 21 to perform a feedback operation to change the recognition state of the object "C carbonated juice" represented by the object image 203 from "still don't know" to "it's the target," as shown by the arrow 221.

[0274] In this case, in the robot 22, the control unit 66 recalculates the score of the object “C carbonated juice” based on the feedback information supplied from the communication unit 44 of the instruction device 21 via the communication unit 61.

[0275] As a result, based on the new score obtained by recalculation, the control unit 66 determines (changes) the recognition state of the object "C carbonated juice" from "still unknown" to "target", as shown by arrow 2211.

[0276] When a feedback operation is performed to change the recognition state of an object from "may be a target" to "is a target," the control unit 66 performs the following process. That is, for example, the control unit 66 changes the recognition state of the object from "may be a target" to "is a target," as indicated by arrow 2212, based on feedback information from the communication unit 61.

[0277] Furthermore, when a feedback operation is performed to change the recognition state of an object from "not a target" to "a target," the control unit 66 performs the following process. That is, for example, based on feedback information from the communication unit 61, the control unit 66 changes the recognition state of the object from "not a target" to "a target," as indicated by arrow 2213.

[0278] Furthermore, for example, as shown in Figure 31, consider a case where the user uses the operation unit 41 of the instruction device 21 to perform a feedback operation to change the recognition state of the object "Juhachicha" representing the object image 202 from "is the target" to "is not the target" as indicated by the arrow 222.

[0279] In this case, as shown in Figure 32, based on feedback information from the communication unit 61, the control unit 66 determines the recognition state of the object "Juhachicha" from "is a target" to "is not a target", as indicated by arrow 222.

[0280] [An example of feedback operation] Next, with reference to FIGS. 33 to 38, a case where the user performs a feedback operation while referring to the display screen 21a of the instruction device 21 will be described.

[0281] In the instruction device 21, the control unit 42 displays a robot recognition image including object images 241 to 244, for example, as shown in Figure 33, on the display screen 21a of the display unit 43 based on robot recognition information supplied from the robot 22 via the communication unit 44.

[0282] Furthermore, when the robot 22 recognizes the object "sports drink" represented by the object image 242 as "a target," the control unit 42 displays a robot recognition image on the display screen 21a based on the robot recognition information transmitted from the robot 22 via the communication unit 44.

[0283] As a result, a robot recognition image including object images 241 to 244 is displayed on the display screen 21a, as shown in FIG.

[0284] On the display screen 21a shown in Figure 34, it is assumed that the robot 22 recognizes as a target an object "Sports Drink" (object represented by object image 242) that is different from the object desired by the user, "Juhachicha" (object represented by object image 245).

[0285] When the robot 22 takes action to take home the object "sports drink" as a target, the control unit 42 displays a robot recognition image on the display screen 21a based on the robot recognition information transmitted from the robot 22 via the communication unit 44.

[0286] As a result, the display screen 21a displays an object image 242 with a triangle mark added thereto, indicating that the object "sports drink" is to be taken home as a target, as shown in FIG.

[0287] In this case, the user performs a feedback operation to indicate that the object "Sports Drink" is not a target, using the operation unit 41 of the instruction device 21, as shown in Fig. 35. At this time, the operation unit 41 supplies an operation signal corresponding to the user's feedback operation to the control unit 42.

[0288] The control unit 42 generates feedback information based on the operation signal from the operation unit 41 and supplies it to the communication unit 44. The communication unit 44 transmits the feedback information from the control unit 42 to the robot 22. In this case, for example, the robot 22 (control unit 66) recalculates the score of the object "Sports Drink" based on the feedback information from the communication unit 44.

[0289] As a result, the score of the object "Sports Drink" is set to be lower than the score of the object "Juhachi Tea", for example, and the robot 22 acts to take the object "Juhachi Tea" home as a target. Also, based on the recalculated score, the robot 22 changes the recognition state of the object "Sports Drink" from "is a target" to "is not a target".

[0290] In this case, in the instruction device 21, the control unit 42 displays a robot recognition image on the display screen 21a based on the robot recognition information transmitted from the robot 22 via the communication unit 44. The robot recognition information includes at least a score recalculated based on feedback information corresponding to the user's feedback operation.

[0291] As a result, the display screen 21a displays an object image 245 with a triangle mark added thereto, indicating that the object "Juhachicha" is to be taken home as a target, as shown in FIG.

[0292] In addition to performing a feedback operation to indicate that the object "Sports Drink" is not the target, the user may also perform a feedback operation to indicate that the object "Juhachicha" is the target, as shown in Figure 37, for example.

[0293] In this case, for example, the robot 22 (control unit 66) recalculates the score of the object "Juhachi Tea" based on feedback information from the communication unit 44. As a result, the score of the object "Juhachi Tea" becomes higher than the score of the object "Sports Drink", for example, and the robot 22 acts to take the object "Juhachi Tea" home as a target.

[0294] In response to this, the control unit 42 of the instruction device 21 displays a robot recognition image on the display screen 21 a based on the robot recognition information transmitted from the robot 22 via the communication unit 44 .

[0295] As a result, the display screen 21a displays an object image 245 with a triangle mark added thereto, indicating that the object "Juhachicha" is to be taken home as a target, as shown in FIG.

[0296] For example, when using a feedback operation to indicate that an object is not a target, the user may use the operation unit 41 to specify an object (object image) that is not a target, and treat the specified object as the target.

[0297] In this case, the control unit 42 generates feedback information indicating that the object designated by the user is not the target.

[0298] Also, for example, when using a feedback operation to indicate that an object is not a target, the user may use the operation unit 41 to specify the target object (object image), and treat any object that is not specified as not being a target.

[0299] In this case, the control unit 42 generates feedback information indicating that the object not specified by the user is not a target.

[0300] Note that an object may be specified by surrounding the object on the display screen 21a that is to be specified with a frame, etc. Furthermore, if the operation unit 41 includes a mouse, the object may be specified by moving a cursor or the like to the object on the display screen 21a that is to be specified and left-clicking (single-clicking or double-clicking) the mouse.

[0301] Furthermore, if the pointing device 21 is a personal computer or the like and the operation unit 41 is provided with control keys and shift keys, multiple objects can be designated simultaneously by left-clicking in combination with the control key or shift key.

[0302] Furthermore, for example, after specifying an object, the user may perform a feedback operation indicating that the specified object is not a target by selecting the option "not a target" from among options such as "is a target" and "is not a target" that are displayed by right-clicking with the mouse.

[0303] Furthermore, for example, the user can cancel the designation of an object by again left-clicking the designated object with the mouse. In addition, for example, the user can select an object by left-clicking and perform a throwing operation such as throwing the object out of the frame of the display screen 21a as a feedback operation indicating that the object is "not a target."

[0304] For example, if the operation unit 41 includes an "Enter" key, the pressing operation of pressing the "Enter" key when an object is specified can be used as a feedback operation indicating that the specified object is the target.

[0305] Also, for example, if the operation unit 41 includes a "Delete" key, the operation of pressing the "Delete" key when an object is specified can be used as a feedback operation indicating that the specified object is not the target.

[0306] Additionally, for example, if the operation unit 41 includes a "Tab" key, the user can move the focus displayed on the display screen 21a each time the "Tab" key is pressed.

[0307] The same applies to the case where a feedback operation is performed to indicate that a designated object is a target.

[0308] [Feedback processing performed by the instruction device 21] Next, the feedback process performed by the instruction device 21 will be described with reference to the flowchart of FIG.

[0309] This feedback process is started, for example, when the power supply of the pointing device 21 is turned on. Also, it is assumed that an object image is displayed on the display screen 21a.

[0310] In step S61, the control unit 42 determines whether or not a feedback operation has been performed by the user based on whether or not an operation signal corresponding to the feedback operation has been supplied from the operation unit 41, and repeats the processing of step S61 until it determines that a feedback operation has been performed by the user.

[0311] In step S61, if the control unit 42 determines that a feedback operation has been performed by the user based on whether an operation signal corresponding to the feedback operation has been supplied, the control unit 42 advances the process to step S62.

[0312] In step S62, the control unit 42 generates feedback information based on the feedback operation by the user and supplies it to the communication unit 44.

[0313] In step S63, the communication unit 44 transmits the feedback information from the control unit 42 to the robot 22 by wireless communication or the like, and the process returns to step S61, where the same processes are performed thereafter. Note that this feedback process is terminated, for example, when the power of the instruction device 21 is turned off.

[0314] As described above, the feedback process allows the user to indicate an object displayed on the display screen 21a as a target, etc., in response to a feedback operation by the user.

[0315] Therefore, for example, when the robot 22 recognizes an object that is not a target as a target, the instruction device 21 can provide feedback to the robot 22 that the object is not a target.

[0316] [About the score recalculation process performed by Robot 22] Next, the score recalculation process performed by the robot 22 will be described with reference to the flowchart of FIG.

[0317] This score recalculation process is started, for example, when the robot 22 is powered on.

[0318] In step S81, the communication unit 61 repeats the process of step S81 until feedback information is received from the instruction device 21. Then, in step S81, if the communication unit 61 receives feedback information from the instruction device 21, it supplies the received feedback information to the control unit 66 and proceeds to step S82.

[0319] In step S82, the control unit 66 recalculates the score of the object based on the feedback information from the communication unit 61, and the process proceeds to step S83.

[0320] In step S83, the control unit 66 generates robot recognition information based on the recalculated score, supplies it to the communication unit 61, and the process proceeds to step S84.

[0321] In step S84, the communication unit 61 transmits the robot recognition information from the control unit 66 to the instruction device 21 by wireless communication or the like, and the process returns to step S81, where the same processes are performed thereafter. Note that this score recalculation process is terminated, for example, when the power of the robot 22 is turned off.

[0322] As described above, according to the score recalculation process, the score is recalculated based on feedback information from the instruction device 21, so that it is possible to calculate a score that reflects feedback from the user.

[0323] <4. Modification of the Second Embodiment> In the second embodiment, the user refers to the display screen 21a and performs a feedback operation to indicate whether a predetermined object is a target or not, but the feedback operation by the user is not limited to this.

[0324] That is, for example, when objects 261 to 263 are displayed as one object on the display screen 21a as shown in FIG. 41, the robot 22 recognizes the objects 261 to 263 as one object.

[0325] In this case, the user can use the operation unit 41 of the instruction device 21 to perform a feedback operation, for example, to specify the area in which one object is displayed, so that the robot 22 recognizes the objects 261 to 263 as one object each.

[0326] In addition, the feedback operation of specifying the area where one object is displayed is performed, for example, when the operation unit 41 includes a mouse, by using the mouse to specify the area where one object is displayed and selecting the option "Separate" that is displayed by right-clicking.

[0327] Then, in response to the supply of an operation signal corresponding to the feedback operation from the operation unit 41, the control unit 42 generates feedback information and supplies it to the communication unit 44. The communication unit 44 transmits the feedback information from the control unit 42 to the robot 22 by wireless communication or the like.

[0328] In the robot 22, the communication unit 61 supplies feedback information from the communication unit 44 to the control unit 66. Based on the feedback information from the communication unit 61, the control unit 66 recognizes the objects 261 to 263, which were previously recognized as a single object, as single objects 261, 262, and 263, respectively.

[0329] In addition, the control unit 66 calculates scores for objects 261, 262, and 263 that have been recognized as a single object, and generates robot recognition information based on the calculation results, etc., and transmits it to the instruction device 21 via the communication unit 61.

[0330] In the instruction device 21, the control unit 42 displays a robot recognition image such as that shown in Figure 42 on the display screen 21a of the display unit 43 based on the robot recognition information transmitted from the communication unit 61 of the robot 22 via the communication unit 44.

[0331] Furthermore, for example, when something that should be treated as a single object (such as a set of knife, fork, and spoon that are used at the same time) is displayed separately as multiple objects on the display screen 21a, the user can use the operation unit 41 of the instruction device 21 to perform a feedback operation that specifies an area that includes the multiple objects that are displayed separately, so that the robot 22 recognizes the multiple objects that are displayed separately as a single object.

[0332] In addition, the feedback operation of specifying an area containing multiple objects displayed separately is performed, for example, when the operation unit 41 includes a mouse, by using the mouse to specify the area in which each object is displayed and then selecting the option "Merge" that is displayed by right-clicking.

[0333] Incidentally, for example, in the second embodiment, a feedback operation is performed on an object displayed on the display screen 21a.

[0334] However, in other cases, for example, when an object in the captured image is not detected by the control unit 66, an area designation operation can be performed to designate an area in the captured image that includes the undetected object, so that the robot 22 can detect (recognize) the undetected object as an object.

[0335] As a result, the robot 22 detects an undetected (detected) object as shown in FIG. 43, and changes the recognition state of the detected object from "undetected" to "not yet known."

[0336] That is, for example, the user can use the operation unit 41 of the instruction device 21 to perform an area designation operation that specifies a search range for the robot 22 to search for an object, thereby causing the robot 22 to search within the search range.

[0337] 5. Third Embodiment Next, FIG. 44 shows an example of specifying a search range using a captured image.

[0338] For example, if the robot recognition information includes a surrounding image 31 as shown in FIG. 44A as a captured image, the control unit 42 of the instruction device 21 will display the surrounding image 31 on the display screen 21a based on the robot recognition information.

[0339] In this case, the user can use the operation unit 41 of the instruction device 21 to perform an area designation operation to designate a partial area 281 on the surrounding image 31 displayed on the display screen 21a as an area corresponding to the search range in which to search for the target, as shown in Figure 44A.

[0340] As a result, the operation unit 41 supplies a corresponding operation signal to the control unit 42 in response to the user's area designation operation.

[0341] In response to an operation signal from the operation unit 41, the control unit 42 generates designated range information representing a partial area 281 designated by the user's area designation operation out of the entire area on the surrounding image 31, and supplies the designated range information to the communication unit 44. The communication unit 44 transmits the designated range information from the control unit 42 to the robot 22.

[0342] In response to this, the communication unit 61 of the robot 22 receives the designated range information from the communication unit 44 of the instruction device 21 and supplies it to the control unit 66. The control unit 66 detects a partial area 281 on the surrounding image 31 stored in the memory unit 68 based on the designated range information from the communication unit 61.

[0343] Then, the control unit 66 reads out from the memory unit 68 a plurality of three-dimensional positions associated with each object on the partial region 281, and calculates (specifies) a search range for searching for the target based on the read out plurality of three-dimensional positions.

[0344] Note that the method by which the control unit 66 calculates the search range is not limited to this. That is, for example, even if the partial region 281 is not associated with a three-dimensional position or the like, the control unit 66 can calculate the search range from parameters that represent the partial region 281 and the effective viewing angle, number of effective pixels, position, orientation, and the like of the camera 62.

[0345] In this case, the control unit 66 stores in the memory unit 68, for example, a first parameter representing the position and orientation of the camera 62 at the time of capturing the surrounding image 31, along with the image capturing time at which the surrounding image 31 was obtained by capturing the image, in association with the surrounding image 31. Also, it is assumed that second parameters representing the effective viewing angle and the number of effective pixels of the camera 62 are stored in advance in the storage unit 68.

[0346] That is, for example, the control unit 66 reads out from the memory unit 68 a first parameter representing the position and orientation of the camera 62 at the imaging time when the surrounding image 31 was obtained by imaging, and a second parameter representing the effective viewing angle and number of effective pixels of the camera 62.

[0347] Then, by using the first and second parameters, the control unit 66 identifies the three-dimensional field of view range corresponding to the partial area 281 from the partial area 281 occupying the surrounding image 31, that is, the area in real space that was captured as the partial area 281 at the time of capture.

[0348] The search range is an area within the field of view in which the robot 22 can act.

[0349] The control unit 66 controls the driving unit 67 to drive the parts of the robot 22 corresponding to the hands and feet, and causes the robot 22 to perform an operation of searching for a target within the calculated search range.

[0350] Also, for example, if the robot recognition information includes a partial image 35 as shown in FIG. 44B, the control unit 42 of the instruction device 21 will display the partial image 35 on the display screen 21a based on the robot recognition information.

[0351] In this case, the user can operate the operation unit 41 of the instruction device 21 to perform an area designation operation to designate a partial area 282 on the partial image 35 so as to detect the object "apple juice" on the partial image 35 displayed on the display screen 21a.

[0352] As a result, the robot 22 performs an operation of searching for a target within the search range corresponding to the partial region 282 on the partial image 35. Thus, the robot 22 can be made to detect the object "apple juice."

[0353] The partial image 35 may be, for example, an image obtained by capturing an image of an area specified by the user's area designation operation at a closer distance, or an image of a portion of an image captured in a previous capture.

[0354] Next, FIG. 45 shows an example of specifying a search range using a spatial diagram of an interior room.

[0355] For example, if the robot recognition information includes a spatial image 301 (an image showing a spatial diagram of a room) as shown in Fig. 45, the control unit 42 of the instruction device 21 will display the spatial image 301 on the display screen 21a based on the robot recognition information. If the storage unit 45 is configured to store the spatial image 301 in advance, the control unit 42 will read out the spatial image 301 from the storage unit 45 and display it on the display screen 21a.

[0356] In this case, the user can operate the operation unit 41 of the pointing device 21 to perform an area designation operation to designate a partial area 321 on the spatial image 301 displayed on the display screen 21a as a search range.

[0357] As a result, the operation unit 41 supplies a corresponding operation signal to the control unit 42 in response to the user's area designation operation.

[0358] In response to an operation signal from the operation unit 41, the control unit 42 generates designated range information representing a partial area 321 designated by the user's area designation operation out of the entire area on the spatial image 301, and supplies the information to the communication unit 44. The communication unit 44 transmits the designated range information from the control unit 42 to the robot 22. The designated range information is used by the robot 22 as identification information for identifying (understanding) a search range for searching for a target.

[0359] In response to this, the communication unit 61 of the robot 22 receives the designated range information from the communication unit 44 of the instruction device 21 and supplies it to the control unit 66. The control unit 66 detects a partial area 321 on the spatial image 301 stored in the storage unit 68 based on the designated range information from the communication unit 61.

[0360] Then, the control unit 66 reads out from the storage unit 68 a plurality of three-dimensional positions associated with each of the small areas that make up the partial area 321, and calculates a search range based on the read out plurality of three-dimensional positions.

[0361] The control unit 66 controls the drive unit 67 to drive parts of the robot 22 corresponding to the hands and feet, causing the robot 22 to perform an operation of searching for a target within the calculated search range. Note that the control unit 66 causes the robot 22 to perform an operation of searching within the search range, using an object belonging to a category represented by category information included in the instruction information as a target, based on instruction information from the communication unit 61. Furthermore, for example, when the control unit 66 has not received instruction information from the instruction device 21 via the communication unit 61, it autonomously determines a target and causes the robot 22 to perform an operation of searching within the search range.

[0362] As a result, the robot 22 performs an operation of searching for a target within the search range corresponding to the partial region 321 on the spatial image 301.

[0363] In addition, when a spatial image 301 is displayed on the display screen 21a, if the user uses the operation unit 41 to select, for example, a partial area 322 from the entire area on the spatial image 301, the captured image 341 obtained by capturing an image within the partial area 322 can be displayed on the display screen 21a.

[0364] That is, for example, in the instruction device 21, the control unit 42 supplies the robot recognition information transmitted from the robot 22 via the communication unit 44 to the storage unit 45 for storage.

[0365] Therefore, the storage unit 45 stores the captured image 341 in association with the partial region 322 on the spatial image 301, together with the spatial image 301 as shown in FIG.

[0366] Therefore, when the user selects a partial area 322, the control unit 42 reads out the captured image 341 corresponding to the partial area 322 selected by the user from the memory unit 45 and displays it on the display screen 21a as shown in Figure 45.

[0367] Then, the user can use the operation unit 41 to perform an area designation operation on the captured image 341 displayed on the display screen 21a, in the same manner as when referring to Figure 44B, to designate a partial area 361 on the captured image 341.

[0368] As a result, the robot 22 performs an operation of searching for a target within the search range corresponding to the partial area 361 on the captured image 341.

[0369] The captured image 341 is transmitted from the robot 22 to the instruction device 21 in a state where it is included in the robot recognition information.

[0370] Incidentally, it is desirable that the robot recognition information transmitted from the robot 22 to the instruction device 21 be a small amount of data in order to avoid data congestion in wireless communication and the like.

[0371] Therefore, for example, when the robot 22 transmits the captured image 341 together with the robot recognition information, it may transmit a low-quality captured image 341' as shown in FIG. 46A, which is obtained by reducing the data amount of the high-quality captured image 341 shown in FIG. 46B. Note that this captured image 341' is not used for the user to specify an area, etc., but is used, for example, when displaying an object image of the object "apple juice" on the display screen 21a. It is also desirable that the captured image used for the user to specify an area, etc., is transmitted in high quality as is and displayed on the display screen 21a.

[0372] Of the entire area of ​​the captured image 341', a partial area 381 showing an object (a plastic bottle of "apple juice") is of high image quality to be used as an object image, and the other areas (shown by diagonal lines) are of low image quality.

[0373] That is, for example, the partial area 381 is high resolution or color, and the area other than the partial area 381 is low resolution or monochrome.

[0374] [Regarding the area designation process performed by the indicator device 21] Next, the area designation process performed by the pointing device 21 will be described with reference to the flowchart of FIG.

[0375] In step S101, the communication unit 44 receives robot recognition information from the robot 22 and supplies it to the control unit 42. It is assumed that the robot recognition information includes, for example, the surrounding image 31 as a captured image.

[0376] In step S102, the control unit 42 supplies the surrounding image 31 included in the robot recognition information from the communication unit 44 to the display unit 43, and causes the display unit 43 to display the surrounding image 31 on the display screen 21a.

[0377] In step S103, the control unit 42 determines whether or not the user has performed an area designation operation, depending on whether or not an operation signal corresponding to the user's area designation operation has been supplied from the operation unit 41. Note that the area designation operation refers to, for example, an operation of designating a predetermined partial area 281 from the entire area of ​​the surrounding image 31 displayed on the display screen 21a.

[0378] Then, in step S103, the control unit 42 repeats the process of step S103 in response to an operation signal from the operation unit 41 until it determines that the user has performed an area designation operation.

[0379] In step S103, if the control unit 42 determines that the user has performed an area designation operation, the process proceeds to step S104. In this case, an operation signal corresponding to the user's area designation operation is supplied from the operation unit 41 to the control unit 42.

[0380] In step S104, the control unit 42 generates designated range information representing the partial area 281 on the surrounding image 31 designated by the area designation operation in response to the operation signal from the operation unit 41, and supplies the designated range information to the communication unit 44. The designated range information is used by the robot 22 as identification information for identifying (understanding) the search range for searching for a target.

[0381] In step S105, the communication unit 44 supplies the designated range information from the control unit 42 to the robot 22 using wireless communication or the like, and causes the robot 22 to search within the search range. This completes the area designation process.

[0382] As described above, the area designation process allows the user to specify the search range in which the robot 22 searches for a target by performing an area designation operation, so that, for example, the user can have the robot 22 search within a desired range.

[0383] In particular, for example, by the user performing an area designation operation, it is possible to designate an area that includes at least an object that has not been detected by the robot 22 as a search area, thereby making it possible for the robot 22 to detect the undetected object.

[0384] [Search process performed by robot 22] Next, the search process performed by the robot 22 will be described with reference to the flowchart of FIG.

[0385] In step S121, the communication unit 61 receives the designated range information from the communication unit 44 of the instruction device 21 and supplies it to the control unit 66. Note that step S121 is performed in parallel with step a122 and step S123, which will be described later. That is, for example, the communication unit 61 can receive the designated range information from the communication unit 44 of the instruction device 21 even while the processing of step S122 or step S123 is being executed.

[0386] In step S122 , the control unit 66 detects the partial area 281 on the surrounding image 31 stored in the storage unit 68 based on the specified range information from the communication unit 61 .

[0387] Then, the control unit 66 reads out from the storage unit 68 a plurality of three-dimensional positions associated with each of the small areas that make up the partial area 281, and calculates (specifies) a search range based on the read out plurality of three-dimensional positions.

[0388] In step S123, the control unit 66 drives the driving unit 67 so that the robot 22 can search for the target within the calculated search range. Note that the control unit 66 targets, for example, an object that belongs to a category indicated by category information included in instruction information, based on instruction information from the communication unit 61. This completes the search process.

[0389] As described above, in the search process, a target is searched for within a search range specified by a user's area specification operation.

[0390] Therefore, for example, when a range that includes at least an object that has not been detected by the robot 22 is designated as a search range by a region designation operation by the user, the robot 22 can detect the undetected object.

[0391] The reason why the search range is specified by the user's area specifying operation, rather than the object itself, is that the robot 22 cannot specify an object that has not yet been detected.

[0392] Therefore, by specifying a search range that includes an object that has not yet been detected by the robot 22, the user can cause the robot 22 to search the specified search range and detect the undetected object.

[0393] Furthermore, in step S103 of the area designation process, for example, the user designates the search range by designating a predetermined partial area 281 from the entire area on the surrounding image 31 displayed on the display screen 21a, but the method of designating the search range is not limited to this.

[0394] That is, for example, if the instruction device 21 can recognize the user's gestures, posture, etc. (hereinafter referred to as gestures, etc.) from a captured image obtained by capturing an image of the user, the user can specify a search range using gestures, etc.

[0395] Furthermore, for example, if the instruction device 21 can recognize voice, the user can specify the search range by voice. In this case, the user can specify the search range by speaking, for example, "kitchen" or "my room" as the search range.

[0396] In such a case, the user can also specify (indicate) the category of the target object by, for example, voice, gesture, etc. As a result, the indicating device 21 generates instruction information including category information indicating the category specified by the user.

[0397] Furthermore, for example, if the robot 22 can recognize gestures and the like from a captured image of the user, the user can directly specify a search range for the robot 22 by gestures and the like.

[0398] In this case, for example, the control unit 66 recognizes the user's gesture based on the captured image from the camera 62, and acquires (generates) the recognition result as specification information used to specify the search range. Then, the control unit 66 specifies the search range using the acquired specification information, and controls the driving unit 67 to cause the robot 22 to search in the specified search range.

[0399] Furthermore, for example, if the robot 22 can recognize voice, the user can directly specify a search range for the robot 22 by voice.

[0400] In this case, for example, the control unit 66 recognizes the user's voice based on the user's voice input from the microphone 64, and acquires (generates) the recognition result as identification information. Then, the control unit 66 uses the acquired identification information to specify a search range, and controls the driving unit 67 to cause the robot 22 to search within the specified search range.

[0401] In such a case, the user can also directly specify (instruct) the category of the target object to the robot 22 by voice, gesture, or the like, for example.

[0402] As a result, the control unit 66 generates instruction information including category information representing the category specified by the user, and causes the robot 22 to search for an object belonging to the category represented by the category information included in the generated instruction information as a target.

[0403] Incidentally, as the instruction device 21, for example, a personal computer or the like can be applied.

[0404] The above-described series of processes can be executed by hardware or software. When the series of processes are executed by software, the programs constituting the software are installed from a program recording medium into a computer incorporated in dedicated hardware, or into, for example, a general-purpose computer that can execute various functions by installing various programs.

[0405] [Example of computer configuration] FIG. 49 is a block diagram showing the hardware T of a computer that executes the above-described series of processes by a program.

[0406] A CPU (Central Processing Unit) 501 executes various processes in accordance with programs stored in a ROM (Read Only Memory) 502 or a storage unit 508. A RAM (Random Access Memory) 503 stores programs and data executed by the CPU 501 as appropriate. The CPU 501, ROM 502, and RAM 503 are interconnected by a bus 504.

[0407] An input / output interface 505 is also connected to the CPU 501 via a bus 504. An input unit 506 including a keyboard, mouse, microphone, etc., and an output unit 507 including a display, speaker, etc. are connected to the input / output interface 505. The CPU 501 executes various processes in response to commands input from the input unit 506. Then, the CPU 501 outputs the results of the processes to the output unit 507.

[0408] A storage unit 508 connected to the input / output interface 505 is formed of, for example, a hard disk, and stores various data and programs executed by the CPU 501. A communication unit 509 communicates with external devices via a network such as the Internet or a local area network.

[0409] Alternatively, the program may be acquired via the communication unit 509 and stored in the storage unit 508 .

[0410] When removable media 511 such as a magnetic disk, optical disk, magneto-optical disk, or semiconductor memory is attached, a drive 510 connected to the input / output interface 505 drives the media and acquires programs, data, etc. recorded thereon. The acquired programs and data are transferred to and stored in a storage unit 508 as necessary.

[0411] As shown in Fig. 49, a recording medium for recording (storing) a program that is installed in a computer and made executable by the computer may be a magnetic disk (including a flexible disk), an optical disk (including a CD-ROM (Compact Disc-Read Only Memory) and a DVD (Digital Versatile Disc)), a magneto-optical disk (including an MD (Mini-Disc)), or removable media 511, which is a package medium made up of a semiconductor memory or the like, or a ROM 502 in which the program is temporarily or permanently stored, or a hard disk constituting storage unit 508. The program is recorded on the recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting, via communication unit 509, which is an interface such as a router or modem, as needed.

[0412] In this specification, the steps describing the above-mentioned series of processes include not only processes that are performed chronologically in the order described, but also processes that are not necessarily performed chronologically but are performed in parallel or individually.

[0413] In this specification, a system refers to an entire device made up of multiple devices.

[0414] The present disclosure is not limited to the first to third embodiments, and various modifications are possible within the scope of the gist of the present disclosure. [Explanation of symbols]

[0415] 1 Robot control system, 21 Instruction device, 21a Display screen, 22 Robot, 41 Operation unit, 42 Control unit, 43 Display unit, 44 Communication unit, 45 Memory unit, 61 Communication unit, 62 Camera, 63 Distance sensor, 64 Microphone, 65 Speaker, 66 Control unit, 67 Drive unit, 68 Memory unit

Claims

1. The control device receiving a captured image from a moving object having an imaging unit; The object recognized from the captured image and degree information indicating the degree of recognition of the object in association with the object are displayed on a display screen of the control device. Control method.

2. The degree information is included in the recognition information recognized by the moving body. The control method according to claim 1 .

3. The control device displays the object on the display screen based on the recognition score and a threshold value. The control method according to claim 1 .

4. When the score related to the recognition is equal to or greater than the threshold value, the control device displays the object on the display screen as a processing target for the moving object. The control method according to claim 3 .

5. The control device changes at least one of the order, position, size, color, brightness, and clarity of the objects based on the degree information included in the recognition information and displays the objects on the display screen. The control method according to claim 2 .

6. The control device displays the object in the captured image captured by the moving object on the display screen as a processing target of the moving object. The control method according to claim 5.

7. The control device displays the object that exists within a predetermined range from the position of the moving body on the display screen. The control method according to claim 6.

8. The control device receiving the recognition information, which also includes spatial information representing a spatial diagram including a path traveled by the moving object; The spatial diagram is also displayed on the display screen based on the spatial information. The control method according to claim 7.

9. The control device displays the object associated with the position of the object on the space map on the display screen together with the space map. The control method according to claim 8.

10. The control device further displays at least one of history information indicating a history of the degree information and change information indicating a change in the degree information on the display screen. The control method according to claim 5.

11. The control device receives recognition information including the degree information newly calculated in response to a feedback operation performed by a user on the object displayed on the display screen. A control method according to any one of claims 1 to 10.

12. The control device displays detailed information about the object in response to the object being selected by a user's selection operation. A control method according to any one of claims 1 to 10.

13. The control device Also receiving processing information representing processing of the moving body that is a processing target of the object; The object associated with the processing of the moving object is displayed on the display screen based on the processing information. A control method according to any one of claims 1 to 10.

14. The control device receives the degree information from the moving object. A control method according to any one of claims 1 to 10.

15. The control device is a computer A control method according to any one of claims 1 to 10.

16. a receiving unit that receives an image captured from a moving object having an imaging unit; a display control unit that displays, on a display screen, an object recognized from the captured image and degree information that indicates a degree of recognition of the object in association with the object; A control device comprising:

17. On the computer, receiving a captured image from a moving object having an imaging unit; The object recognized from the captured image and degree information indicating the degree of recognition of the object in association with the object are displayed on a display screen. A program for executing processes including the above.

18. a receiving unit that receives an image captured from a moving object having an imaging unit; a display control unit that displays, on a display screen of the control device, an object recognized from the captured image and degree information that indicates a degree of recognition of the object in association with the object; the control device having The moving body A mobile object control system comprising:

Images