Processing system and processing method

Abstract

The present invention provides a gesture recognition device, a mobile object, a gesture recognition method, and a storage medium that can improve user convenience. The gesture recognition device includes: an acquisition unit that acquires an image captured by a user; and a recognition unit that recognizes the area in which the user was present when the image was captured. If the user was present in a first area when the image was captured, the recognition unit recognizes the user's gesture based on the image and first information for recognizing the user's gesture. If the user was present in a second area when the image was captured, the recognition unit recognizes the user's gesture based on the image and second information for recognizing the user's gesture.

Description

Technical Field

[0001] The present invention relates to a gesture recognition device, a mobile object, a gesture recognition method and a storage medium. Background Art

[0002] Robots that guide users to desired locations or deliver goods are known. For example, a mobile robot that maintains a predetermined distance from people while providing such services is disclosed (Japanese Patent No. 5617562). Summary of the Invention

[0003] However, the above-mentioned technology may not provide sufficient convenience to users.

[0004] The present invention has been made in consideration of such circumstances, and one object of the present invention is to provide a gesture recognition device, a mobile object, a gesture recognition method, and a storage medium that can improve user convenience.

[0005] Solutions to Problems

[0006] The gesture recognition device, mobile object, gesture recognition method, and storage medium of the present invention adopt the following structures.

[0007] (1): A gesture recognition device comprises: an acquisition unit that acquires an image captured by a user; and a recognition unit that recognizes an area where the user is present when the image is captured, and recognizes the user's gesture based on the image and first information for recognizing the user's gesture when the user is present in a first area when the image is captured; and recognizes the user's gesture based on the image and second information for recognizing the user's gesture when the user is present in a second area when the image is captured.

[0008] (2): In the above-mentioned aspect (1), the first area is an area within a predetermined distance from an imaging device that captures the image, and the second area is an area set at a position farther from the imaging device than the predetermined distance.

[0009] (3): In the above-mentioned configuration of (1) or (2), the first information is information for recognizing a gesture based on movement of a hand or a finger, excluding movement of an arm.

[0010] (4): In any one of the above-mentioned aspects (1) to (3), the second information is information for recognizing a gesture including movement of an arm.

[0011] (5): In the above aspect (4), the first area is an area where the recognition unit cannot recognize the movement of the user's arm from an image captured of the user present in the first area, or where it is difficult to recognize the movement of the user's arm.

[0012] (6): In any of the above schemes (1) to (5), the recognition unit recognizes the user's gesture based on the image, the first information and the second information when the user is present in a third area spanning the first area and a second area adjacent to the first area outside the first area, or in a third area between the first area and a second area farther than the first area when the image is captured.

[0013] (7): In the scheme of (6) above, when the recognition unit recognizes the user's gesture based on the image, the first information, and the second information, the recognition unit prioritizes the recognition result based on the image and the first information over the recognition result based on the image and the second information to recognize the user's gesture.

[0014] (8): The mobile object includes the gesture recognition device according to any one of the above (1) to (7).

[0015] (9): In the above-mentioned scheme (8), the mobile body further includes: a storage device that stores reference information that associates the user's gesture with the movement of the mobile body; and a control unit that controls the mobile body based on the movement of the mobile body associated with the user's gesture recognized by the recognition unit, with reference to the reference information.

[0016] (10): In the above-mentioned scheme (9), the mobile body includes: a first camera unit that captures the surroundings of the mobile body; and a second camera unit that captures a user who remotely operates the mobile body, the recognition unit attempts to recognize the user's gesture based on a first image captured by the first camera unit and a second image captured by the second camera unit, and the recognition result based on the second image is preferably used over the recognition result based on the first image, and the control unit controls the mobile body based on the surrounding conditions obtained from the image captured by the first camera unit and the action associated with the gesture recognized by the recognition unit.

[0017] (11): In any one of the above schemes (8) to (10), the mobile body includes: a first camera unit that captures the surroundings of the mobile body; and a second camera unit that captures a user who remotely operates the mobile body, the recognition unit recognizes the user's gesture based on the second image captured by the second camera unit with reference to the first information when the user is present in the first area and the user's gesture cannot be recognized based on the first image captured by the first camera unit, and the mobile body includes a control unit that controls the mobile body based on the image captured by the first camera unit according to the gesture recognized by the recognition unit.

[0018] (12): In any of the above schemes (8) to (11), the recognition unit tracks the user of the target based on the captured image, recognizes the gesture of the user being tracked, and does not process the gesture of the untracked person, and the mobile body has a control unit that controls the mobile body based on the gesture of the user being tracked.

[0019] (13): A gesture recognition method according to one embodiment of the present invention causes a computer to perform the following processing: obtaining an image captured by a user; identifying the area in which the user was present when the image was captured; if the user was present in a first area when the image was captured, recognizing the user's gesture based on the image and first information for recognizing the user's gesture; and if the user was present in a second area when the image was captured, recognizing the user's gesture based on the image and second information for recognizing the user's gesture.

[0020] (14): A program stored in a storage medium according to one embodiment of the present invention causes a computer to perform the following processing: obtaining an image captured by a user; identifying an area where the user was present when the image was captured; if the user was present in a first area when the image was captured, identifying the user's gesture based on a plurality of the images and first information for identifying the user's gesture; and if the user was present in a second area when the image was captured, identifying the user's gesture based on the image and second information for identifying the user's gesture.

[0021] Effects of the Invention

[0022] According to (1) to (14), the recognition unit recognizes a gesture using the first information or the second information according to the position of the user, thereby improving user convenience.

[0023] According to (6), the gesture recognition device can recognize the gesture with higher accuracy by recognizing the gesture using the first information and the second information.

[0024] According to (8) to (11), the mobile object can perform actions that reflect the user's intention. For example, the user can easily move the mobile object with a simple instruction.

[0025] According to (10) or (11), since the mobile body performs an action corresponding to the recognized gesture based on the images obtained by the camera for recognizing the surroundings and the camera for remote operation, the gesture can be recognized with better accuracy and the action corresponding to the user's intention can be further performed.

[0026] According to (12), the user of the mobile object tracking service can reduce the processing load and improve the convenience of the user by focusing on the gesture of the user being tracked and performing processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 This is a diagram showing an example of a mobile object including a control device according to an embodiment.

[0028] Figure 2 This is a diagram showing an example of a functional configuration included in the main body of a mobile object.

[0029] Figure 3 This is a diagram showing an example of a track.

[0030] Figure 4 This is a flowchart showing an example of the flow of tracking processing.

[0031] Figure 5 This is a diagram for explaining the process of extracting a user's feature amount and the process of registering the feature amount.

[0032] Figure 6 This is a diagram for explaining a process in which the recognition unit tracks a user.

[0033] Figure 7 This is a diagram for explaining tracking processing using feature amounts.

[0034] Figure 8 This is a diagram for explaining a process of specifying a user to be tracked.

[0035] Figure 9 This is a diagram for explaining another example of the process of tracking the user by the recognition unit.

[0036] Figure 10 This is a diagram for explaining a process of specifying a user to be tracked.

[0037] Figure 11 This is a flowchart showing an example of the flow of action control processing.

[0038] Figure 12 This is a diagram for explaining a gesture recognition process.

[0039] Figure 13 This diagram shows users existing in the first area.

[0040] Figure 14 This diagram shows users existing in the second area.

[0041] Figure 15 3 is a diagram for explaining the second gesture A.

[0042] Figure 16 3 is a diagram for explaining the second gesture B.

[0043] Figure 17 3 is a diagram for explaining the second gesture C.

[0044] Figure 18 3 is a diagram for explaining the second gesture D.

[0045] Figure 19 3 is a diagram for explaining the second gesture E.

[0046] Figure 20 3 is a diagram for explaining the second gesture F.

[0047] Figure 21 3 is a diagram for explaining the second gesture G.

[0048] Figure 22 3 is a diagram for explaining the second gesture H.

[0049] Figure 23 is a diagram for explaining the first gesture a.

[0050] Figure 24 3 is a diagram for explaining the first gesture b.

[0051] Figure 25 3 is a diagram for explaining the first gesture c.

[0052] Figure 26 3 is a diagram for explaining the first gesture d.

[0053] Figure 27 is a diagram for explaining the first gesture e.

[0054] Figure 28 3 is a diagram for explaining the first gesture f.

[0055] Figure 29 is a diagram for explaining the first gesture g.

[0056] Figure 30 This is a flowchart showing an example of a process in which the control device recognizes a gesture.

[0057] Figure 31This is a diagram showing the third area (part 1).

[0058] Figure 32 This is a diagram showing the third region (part 2).

[0059] Figure 33 This is a diagram for explaining an example of the functional configuration of the main body of the mobile object according to the second embodiment.

[0060] Figure 34 This is a flowchart showing an example of the flow of processing executed by the control device of the second embodiment.

[0061] Figure 35 3 is a diagram for explaining a modification example of the second gesture G.

[0062] Figure 36 3 is a diagram for explaining a modification example of the second gesture H.

[0063] Figure 37 3 is a diagram for explaining a modified example of the second gesture F.

[0064] Figure 38 It is a diagram for explaining the second gesture FR.

[0065] Figure 39 3 is a diagram for explaining the second gesture FL. DETAILED DESCRIPTION

[0066] Hereinafter, a gesture recognition device, a mobile object, a gesture recognition method, and a storage medium according to embodiments of the present invention will be described with reference to the accompanying drawings.

[0067] <First embodiment>

[0068] [Overall structure]

[0069] Figure 1 This diagram shows an example of a mobile object 10 equipped with a control device according to an embodiment. Mobile object 10 is an autonomous mobile robot. Mobile object 10 supports the actions of users. For example, mobile object 10 supports customers' shopping or customer service, or supports employees' work, based on instructions from store clerks, customers, or facility staff (hereinafter referred to as "users").

[0070] The mobile object 10 includes a main body 20, a container 92, and one or more wheels 94 (wheels 94A and 94B in the figure). The mobile object 10 moves based on user gestures, voice, operation of an input unit (a touch panel described later) of the mobile object 10, or instructions via a terminal device (e.g., a smartphone). The mobile object 10 recognizes gestures based on images captured by a camera 22 provided on the main body 20, for example.

[0071] For example, the mobile body 10 drives wheels 94 to follow the user's movements, or to guide the user. In this case, the mobile body 10 explains products and tasks to the user, or guides the user to the products or objects they are looking for. The user can store the products or goods they intend to purchase in the storage container 92.

[0072] In this embodiment, the case where the mobile body 10 is provided with a storage container 92 is described, but instead (or in addition to this), the mobile body 10 may be provided with a seat for the user to sit on, a frame for the user to sit on, a pedal for the user to place his feet, etc., so that the user can move with the mobile body 10.

[0073] Figure 2 1 is a diagram showing an example of a functional configuration included in the main body 20 of the mobile object 10. The main body 20 includes a camera 22, a communication unit 24, a position determination unit 26, a speaker 28, a microphone 30, a touch panel 32, a motor 34, and a control device 50.

[0074] The camera 22 captures the surroundings of the moving object 10. For example, the camera 22 is a fisheye camera capable of capturing a wide-angle (e.g., 360-degree) image of the surroundings of the moving object 10. The camera 22 is mounted, for example, on the upper portion of the moving object 10 to capture a wide-angle image of the surroundings of the moving object 10 in the horizontal direction. The camera 22 may also be implemented by combining multiple cameras (e.g., cameras capable of capturing a 120-degree or 60-degree horizontal image). The camera 221 is not limited to a single camera; it may be installed on multiple moving objects 10.

[0075] The communication unit 24 is a communication interface for communicating with other devices using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), or DSRC (Dedicated Short Range Communication).

[0076] The position determination unit 26 determines the position of the mobile object 10. The position determination unit 26 obtains the position information of the mobile object 10 using a GPS (Global Positioning System) device (not shown) built into the mobile object 10. The position information may be, for example, two-dimensional map coordinates or longitude and latitude information.

[0077] The speaker 28 outputs, for example, a predetermined sound. The microphone 30 receives input of, for example, a user's voice.

[0078] The touch panel 32 is composed of a display unit, such as an LCD (liquid crystal display) or organic EL (electroluminescence), and an input unit that can detect the operator's touch position via a coordinate detection mechanism. The display unit displays GUI (Graphical User Interface) switches for operation. When the input unit detects a touch operation, flick operation, or slide operation on a GUI switch, it generates an operation signal indicating the touch operation on the GUI switch and outputs it to the control device 50. In response to the operation, the control device 50 causes the speaker 28 to output sound or causes the touch panel 32 to display an image. The control device 50 can also move the movable body 10 in response to the operation.

[0079] The motor 34 drives the wheels 94 to move the vehicle 10. The wheels 94 include, for example, drive wheels driven in the rotational direction by the motor 34 and non-drive wheels, or steering wheels, driven in the yaw direction. By adjusting the angle of the steering wheels, the vehicle 10 can change its path or rotate.

[0080] In this embodiment, the movable body 10 includes wheels 94 as a mechanism for achieving movement, but this embodiment is not limited to this structure. For example, the movable body 10 may be a multi-legged walking robot.

[0081] The control device 50 includes, for example, an acquisition unit 52, an identification unit 54, a trajectory generation unit 56, a travel control unit 58, an information processing unit 60, and a storage unit 70. Part or all of the acquisition unit 52, identification unit 54, trajectory generation unit 56, travel control unit 58, and information processing unit 60 are implemented, for example, by a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Part or all of these functional units may also be implemented using hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or through a combination of software and hardware. The program may be pre-stored in the storage unit 70 (a storage device including a non-transitory storage medium), such as an HDD (Hard Disk Drive) or flash memory, or may be stored on a removable storage medium (non-transitory storage medium) such as a DVD or CD-ROM and installed by attaching the storage medium to a drive device. The acquisition unit 52, recognition unit 54, trajectory generation unit 56, travel control unit 58, or information processing unit 60 may be provided in a device separate from the control device 50 (mobile vehicle 10). For example, the recognition unit 54 may be provided in another device, and the control device 50 may control the mobile vehicle 10 based on the processing results of the other device. Some or all of the information stored in the storage unit 70 may also be stored in another device. A configuration including one or more functional units among the acquisition unit 52, recognition unit 54, trajectory generation unit 56, travel control unit 58, or information processing unit 60 may also constitute a system.

[0082] Storage unit 70 stores map information 72, gesture information 74, and user information 80. Map information 72 is information that represents the shape of roads and pathways within facilities, for example, by using links and nodes connected by these links. Map information 72 may also include information such as road curvature and POI (Point of Interest) information.

[0083] Gesture information 74 is information that associates gesture-related information (template features) with the motion of mobile object 10. Gesture information 74 includes first gesture information 76 (first information, reference information) and second gesture information 78 (second information, reference information). User information 80 is information indicating user features. Details of gesture information 74 and user information 80 will be discussed later.

[0084] The acquisition unit 52 acquires an image (hereinafter referred to as a “surrounding image”) captured by the camera 22 . The acquisition unit 52 stores the acquired surrounding image as pixel data in a fisheye camera coordinate system.

[0085] The recognition unit 54 recognizes the user U's body movements (hereinafter referred to as "gestures") based on one or more surrounding images. The recognition unit 54 recognizes the gestures by comparing the features of the user's gestures extracted from the surrounding images with the features of a template (features representing the gesture). Examples of these features include data representing characteristic features such as a person's fingers, finger joints, wrists, arms, and bones, as well as lines connecting these features and the inclination or position of these lines.

[0086] The trajectory generator 56 generates a trajectory that the moving object 10 should travel in the future based on the user's gesture, the user's set destination, surrounding objects, the user's position, map information 72, etc. The trajectory generator 56 combines multiple arcs to generate a trajectory that allows the moving object 10 to smoothly move to the target location. Figure 3 This is a diagram showing an example of a track. For example, the track is generated by combining three arcs. Each arc has a different curvature radius R. m1 、R m2 、R m3 , respectively relative to the forecast period T m1 、T m2 、T m3 The position of the end point is defined as Z m1 、Z m2 、Z m3 . Relative to the forecast period T m1 The track (the track during the first prediction period) is divided into three equal parts, and the positions thereof are Z m11 、Z m12 、Z m13 The moving direction of the moving body 10 at the reference point is defined as the X direction, and the direction intersecting the X direction at right angles is defined as the Y direction. m1 The first tangent line is the target point direction, and the direction that intersects the X' direction at right angles is the Y' direction. The angle between the first tangent line and the line segment extending in the X direction is θ. m1 The angle between the line segment extending in the Y direction and the line segment extending in the Y' direction is θ m1 The point where the line segment extending in the Y direction intersects the line segment extending in the Y' direction is the center of the arc of the trajectory during the first prediction period. m2 On the second tangent line, the target location direction is the X'' direction, and the direction that intersects the X'' direction at right angles is the Y'' direction. The angle between the second tangent line and the line segment extending along the X direction is θ m1 +θ m2The angle between the line segment extending along the Y direction and the line segment extending along the Y'' direction is θ m2 The point where the line segment extending in the Y direction intersects the line segment extending in the Y'' direction is the center of the arc of the track during the second prediction period. The arc of the track during the third prediction period is the arc passing through the Z m2 and Z m3 The center angle of the arc is θ3. The trajectory generation unit 56 can also calculate the trajectory by fitting a geometric model such as a Bezier curve. The trajectory is actually generated as a set of a finite number of trajectory points.

[0087] The trajectory generation unit 56 performs coordinate transformation between the orthogonal coordinate system and the fisheye camera coordinate system. A one-to-one relationship exists between the coordinates of the orthogonal coordinate system and the fisheye camera coordinate system, and this relationship is stored as correspondence information in the storage unit 70. The trajectory generation unit 56 generates a trajectory in the orthogonal coordinate system (orthogonal coordinate system trajectory) and transforms the coordinates of this trajectory into a trajectory in the fisheye camera coordinate system (fisheye camera coordinate system trajectory). The trajectory generation unit 56 calculates the risk of the fisheye camera coordinate system trajectory. Risk is an indicator value that indicates the likelihood that the mobile object 10 will approach an obstacle. The risk tends to increase as the distance from the trajectory (orbital point on the trajectory) to the obstacle decreases, and decreases as the distance from the trajectory (orbital point on the trajectory) to the obstacle increases.

[0088] When the total risk value and the risk of each track point meet predetermined criteria (for example, when the total risk value is less than or equal to a threshold value Th1 and the risk of each track point is less than or equal to a threshold value Th2), the trajectory generating unit 56 adopts the trajectory meeting the criteria as the trajectory of the moving object.

[0089] If the above-mentioned trajectory does not meet the pre-set criteria, the following processing can also be performed. The trajectory generation unit 56 detects a drivable space in the fisheye camera coordinate system and transforms the detected drivable space in the fisheye camera coordinate system into a drivable space in the orthogonal coordinate system. The drivable space refers to the space obtained by excluding obstacles and the surrounding areas of the obstacles (areas with set risks or areas with risks above a threshold value) from the area in the moving direction of the mobile body 10. The trajectory generation unit 56 corrects the trajectory so that the trajectory is contained within the drivable space transformed into the orthogonal coordinate system. The trajectory generation unit 56 transforms the orthogonal coordinate system trajectory into a fisheye camera coordinate system trajectory and calculates the risk of the fisheye camera coordinate system trajectory based on the surrounding image and the fisheye camera coordinate system trajectory. This process is repeated to search for a trajectory that meets the above-mentioned pre-set criteria.

[0090] The travel control unit 58 causes the vehicle 10 to travel along a track that satisfies a predetermined standard. The travel control unit 58 outputs a command value to the motor 34 for causing the vehicle 10 to travel along the track. The motor 34 rotates the wheels 94 according to the command value, causing the vehicle 10 to move along the track.

[0091] The information processing unit 60 controls various devices and equipment included in the main body 20. For example, the information processing unit 60 controls the speaker 28, the microphone 30, and the touch panel 32. The information processing unit 60 recognizes the sound input to the microphone 30 and the operation performed on the touch panel 32. The information processing unit 60 operates the mobile object 10 based on the recognition results.

[0092] In the above example, the recognition unit 54 recognizes the user's body movements based on images captured by the camera 22 installed on the mobile object 10. However, the recognition unit 54 may recognize the user's body movements based on images captured by a camera not installed on the mobile object 10 (a camera installed at a different location from the mobile object 10). In this case, the images captured by the camera are transmitted to the control device 50 via communication, and the control device 50 obtains the transmitted images and recognizes the user's body movements based on the obtained images. The recognition unit 54 may also recognize the user's body movements based on multiple images. For example, the recognition unit 54 may recognize the user's body movements based on images captured by the camera 22 and multiple images captured by cameras installed at different locations from the mobile object 10. For example, the recognition unit 54 may recognize the user's body movements from each image, apply the recognition results to a predetermined benchmark, recognize the user's body movements, or perform image processing on multiple images to generate one or more images and recognize the user's intended body movements from the generated images.

[0093] [Support Processing]

[0094] The mobile object 10 performs a support process for supporting the user's shopping. The support process includes a process related to tracking and a process related to behavior control.

[0095] [Processing related to tracking (Part 1)]

[0096] Figure 4 This is a flowchart showing an example of the tracking process. First, the control device 50 of the mobile body 10 accepts the user's login (step S100). Next, the control device 50 tracks the user who logged in at step S100 (step S102). Next, the control device 50 determines whether the tracking is successful (step S104). If the tracking is successful, the process proceeds to the following step. Figure 11 If the tracking is unsuccessful, the control device 50 identifies the user (step S106 ).

[0097] (Processing for logged-in users)

[0098] The user registration process in step S100 will be described. The control device 50 of the mobile object 10 confirms the user's intention to register based on a specific gesture, voice, or operation on the touch panel 32 by the user (e.g., a customer visiting a store). If the user's intention to register is confirmed, the recognition unit 54 of the control device 50 extracts user features and registers the extracted features.

[0099] Figure 5 This figure illustrates the process of extracting and registering user features. The recognition unit 54 of the control device 50 identifies the user from an image IM1 captured of the user and identifies the user's joints (performs skeleton processing). For example, the recognition unit 54 estimates the user's face, facial features, neck, shoulders, elbows, wrists, waist, ankles, and other parts of the user from image IM1, and performs skeleton processing based on the positions of the estimated parts. For example, the recognition unit 54 performs skeleton processing using a known method (such as human posture recognition) that uses deep learning to estimate the user's joints. Next, based on the results of the skeleton processing, the recognition unit 54 identifies the user's face, upper body, and lower body, extracts features for each of the identified face, upper body, and lower body parts, and stores the extracted features in the storage unit 70 as user features. Examples of facial features include male gender, female gender, hairstyle, and facial features. Examples of upper body features include upper body color. Examples of lower body features include lower body color.

[0100] (Tracking user processing)

[0101] The process of tracking the user in step S102 will be described. Figure 6 This is for explaining the process of tracking the user by the recognition unit 54 ( Figure 4 (See the processing of step S104 in the figure). The recognition unit 54 detects a user from image IM2 captured at time T. The recognition unit 54 detects the detected person from image IM3 captured at time T+1. Based on the user's position and movement direction at and before time T, the recognition unit 54 estimates the user's position at time T+1 and identifies users near the estimated position as the users to be tracked (tracking targets). Tracking is considered successful if the user can be identified.

[0102] In the tracking process, the recognition unit 54 may track the user using not only the position of the user at time T+1 as described above but also the feature amount of the user. Figure 7This diagram illustrates tracking processing using feature values. For example, the recognition unit 54 estimates the user's position at time T+1, identifies users near the estimated position, and further extracts feature values for those users. If the extracted feature values match the registered feature values by a threshold or more, the control device 50 estimates the identified user as the tracking target user and determines that tracking is successful.

[0103] For example, even when a user to be tracked overlaps or intersects with another person, the user can be tracked with higher accuracy based on the change in the user's position and the user's feature value as described above.

[0104] (Determine the user's processing)

[0105] The process of determining the user in step S106 will be described. If the user tracking is unsuccessful, the identification unit 54 Figure 8 As shown, the feature values of nearby people are compared with the feature values of the registered user to identify the user to be tracked. For example, the recognition unit 54 extracts the feature values of each person included in the image. The recognition unit 54 compares the feature values of each person with the feature values of the registered user and identifies people whose feature values match those of the registered user by at least a threshold. The recognition unit 54 identifies the identified user as the user to be tracked.

[0106] Through the above-described processing, the recognition unit 54 of the control device 50 can track the user with higher accuracy.

[0107] [Processing related to tracking (Part 2)]

[0108] In the above example, the user is a customer visiting a store. However, the following processing can also be performed when the user is a store clerk or a facility employee (eg, a medical practitioner in the facility).

[0109] (Processing for logged-in users)

[0110] The process of logging in the user in step S102 may also be performed as follows. Figure 9 This is for explaining the process of tracking the user by the recognition unit 54 ( Figure 4 FIG. 1 shows another example of the process of step S102 of FIG. The recognition unit 54 extracts facial features of a person from the captured image. The recognition unit 54 compares the extracted facial features with facial features of a user being tracked, which is pre-registered in the user information 80. If they match, the recognition unit 54 determines that the person in the image is the user being tracked.

[0111] (Determine the user's processing)

[0112] The process of determining the user in step S106 can also be performed as follows. If the user tracking is unsuccessful, the identification unit 54 Figure 10 As shown, the facial feature values of the people in the surrounding area are compared with the feature values of the registered user, and the person whose feature value matches by a threshold value or more is determined as the user to be tracked.

[0113] As described above, the recognition unit 54 of the control device 50 can track the user with higher accuracy.

[0114] [Processing related to action control]

[0115] Figure 11 This is a flowchart showing an example of the process of action control processing. Figure 4 The control device 50 recognizes the user's gesture (step S200) and controls the movement of the mobile body 10 based on the recognized gesture (step S202). Next, the control device 50 determines whether to end the service (step S204). If the service is not ended, return to Figure 4 The process of step S102 is continued. When the service is terminated, the control device 50 deletes the registered information related to the user, such as the user's feature quantity (step S206). Thus, one routine of this flowchart ends.

[0116] The processing of step S200 will be described. Figure 12 This figure illustrates the gesture recognition process. The control device 50 extracts a region encompassing one or both of the arm and hand (hereinafter referred to as the target region) from the results of skeletal processing and extracts feature quantities representing the state of one or both of the arm and hand within the extracted target region. The control device 50 identifies feature quantities that match the feature quantities representing the aforementioned state from the feature quantities included in the gesture information 74. The control device 50 causes the mobile object 10 to perform the motion associated with the feature quantities identified in the gesture information 74.

[0117] (Gesture recognition processing)

[0118] The control device 50 determines whether to refer to the first gesture information 76 or the second gesture information 78 of the gesture information 74 based on the relative position between the mobile body 10 and the user. Figure 13 As shown, when the user is within the predetermined distance from the moving object, in other words, when the user is within the first area AR1 set based on the moving object 10, the control device 50 determines whether the user is performing the same gesture as that included in the first gesture information 76. Figure 14As shown, when the user is separated from the moving body by a specified distance, in other words, when the user is in the second area set based on the moving body 10 (when the user is not in the first area AR1), the control device 50 determines whether the user is performing a gesture that is the same as the gesture included in the second gesture information 78.

[0119] The first gesture included in the first gesture information 76 is a gesture that uses the hand instead of the arm, and the second gesture included in the second gesture information 78 is a gesture that uses both the arm (the arm between the elbow and the hand) and the hand. The first gesture can be a body movement smaller than the second gesture, such as a body vibration or a small hand movement. A small body movement means that, when causing the mobile object 10 to perform a certain movement (such as walking straight), the body movement of the first gesture is smaller than that of the second gesture. For example, the first gesture can be a gesture using the hand or fingers, while the second gesture can be a gesture using the arm. For example, the first gesture can be a gesture using the leg below the knee, while the second gesture can be a gesture using the lower body. For example, the first gesture can be a gesture using the hand or foot, while the second gesture can be a gesture using the entire body, such as jumping.

[0120] When the camera 22 of the mobile object 10 captures a user present in the first area AR1, as shown in FIG. Figure 13 The arm portion shown is difficult to be included in the image, and the hand and fingers are included in the image. The first area AR1 is an area where the recognition unit 54 cannot recognize the user's arm from the image captured by the user in the first area AR1, or it is difficult to recognize the user's arm. When the camera 22 of the mobile object 10 captures the user in the second area AR2, as shown in FIG. Figure 14 As shown, the arm portion is included in the image. Therefore, as described above, when the user is in the first area AR1, the recognition unit 54 recognizes the gesture using the first gesture information 76, and when the user is in the second area AR2, the recognition unit 54 recognizes the gesture using the second gesture information 78. This allows for more accurate recognition of the user's gesture. The second gesture and the first gesture are described below in order.

[0121] [Gestures and actions included in the second gesture information]

[0122] Hereinafter, the user's front direction (forward direction) is referred to as the X direction, the direction intersecting the front direction is referred to as the Y direction, and the direction intersecting the X and Y directions and opposite to the vertical direction is referred to as the Z direction. Hereinafter, gestures for moving the mobile object 10 are described using the right arm and right hand. However, even when using the left arm and left hand, the same movements are also gestures for moving the mobile object 10.

[0123] (Second gesture A)

[0124] Figure 15 3 is a diagram for explaining the second gesture A. Figure 15 The left side of represents the gesture, Figure 15 The right side of represents the behavior of the mobile object 10 corresponding to the gesture (this also applies to the following figures). The gesture is made by, for example, user P1 (store clerk), and is described below (this also applies to the following figures). In the figure, P2 is a customer.

[0125] Gesture A is a gesture in which the user pushes their arm and hand out from near their body to move in front of it, causing the mobile object 10 behind them to move in front of them. The hand is rotated so that the arm and hand are roughly parallel to the negative Y direction and the thumb faces the positive Z axis (A1 in the figure). In this state, the shoulder or elbow joints are moved to move the hand in the positive X direction (A2 in the figure), and the fingertips are then roughly parallel to the positive X direction (A3 in the figure). In this state, the palm of the hand faces the positive Z direction. Then, with the fingertips roughly parallel to the X direction, the hand and arm are rotated so that the palm faces the negative Z direction (A4 and A5 in the figure). When performing the second gesture A, the mobile object 10 behind user P moves in front of user P1.

[0126] (Second gesture B)

[0127] Figure 16 This diagram illustrates the second gesture B. The second gesture B involves extending an arm and hand forward to advance the mobile object 10. With the arm and hand extended and the palm facing the negative Z direction, the arm and hand are extended parallel to the direction of movement of the mobile object 10 (e.g., the positive X direction) (B1 to B3 in the diagram). When the second gesture B is performed, the mobile object 10 moves in the direction indicated by the fingertips.

[0128] (Second gesture C)

[0129] Figure 17 This diagram illustrates the second gesture C. The second gesture C is a gesture (C1 and C2 in the figure) in which the palms of the arms and hands, extended forward, face each other in the X-direction to stop the moving object 10. When the second gesture C is performed, the moving object 10 stops moving.

[0130] (Second gesture D)

[0131] Figure 18This diagram illustrates the second gesture D. Second gesture D involves moving the arm and hand leftward to cause the mobile object 10 to move leftward. Starting with the arm and hand projecting forward (D1 in the diagram), the palm is rotated approximately 90 degrees clockwise, with the thumb facing the positive Z direction (D2 in the diagram). Using this position as a starting point, the arm and hand are repeatedly swung in the positive Y direction, returning to their starting point (D3 and D4 in the diagram). When performing second gesture D, the mobile object 10 moves leftward. When the arm and hand return to position D1 in the diagram, the mobile object 10 continues forward without moving leftward.

[0132] (Second gesture E)

[0133] Figure 19 This diagram illustrates the second gesture E. The second gesture E involves moving the arm and hand rightward to move the mobile object 10 rightward. Starting with the arm and hand projecting forward (E1 in the diagram), the palm is rotated counterclockwise, with the thumb facing the ground (E2 in the diagram). Using this position as a starting point, the arm and hand are repeatedly swung in the negative Y direction and returned to their starting point (E3 and E4 in the diagram). When the second gesture E is performed, the mobile object 10 moves rightward. When the arm and hand return to the position E1 in the diagram, the mobile object 10 continues forward without moving rightward.

[0134] (Second gesture F)

[0135] Figure 20 This diagram illustrates the second gesture F. The second gesture F is a gesture of waving to cause the mobile object 10 to move backward. The gesture involves repeatedly facing the palm in the positive Z direction (F1 in the diagram) and moving the arm or wrist so that the fingertips face the user (F2 to F5 in the diagram). When the second gesture F is performed, the mobile object 10 moves backward.

[0136] (Second gesture G)

[0137] Figure 21 This diagram illustrates the second gesture G. The second gesture G involves protruding the index finger (or a predetermined finger) and rotating the protruding finger to the left, causing the mobile object 10 to rotate leftward. The palm is oriented toward the negative Z direction (G1 in the diagram), the index finger is protruded, and the other fingers are gently clenched (bent) (G2 in the diagram). After moving the wrist or arm so that the fingertips face the positive Y direction, the arm and hand are returned to the position shown in G1 (G3 and G4 in the diagram). When the second gesture G is performed, the mobile object 10 rotates leftward.

[0138] (Second gesture H)

[0139] Figure 22 This diagram illustrates the second gesture H. The second gesture H involves protruding the index finger (or a predetermined finger) and rotating the protruding finger to the right, thereby rotating the mobile object 10 to the right. The palm is oriented toward the negative Z direction (H1 in the diagram), the index finger is protruded, and the other fingers are gently clenched (bent) (H2 in the diagram). After moving the wrist or arm so that the fingertips face the negative Y direction, the arm and hand are returned to the position shown in H1 (H3 and H4 in the diagram). When the second gesture H is performed, the mobile object 10 rotates to the right.

[0140] [Gestures included in the first gesture information]

[0141] (First gesture a)

[0142] Figure 23 This figure illustrates the first gesture a. First gesture a involves extending the hand forward to move the mobile object 10 forward. The thumb is oriented in the positive Z direction, with the back of the hand parallel to the Z direction (a in the figure). When performing first gesture a, the mobile object 10 moves in the direction indicated by the fingertips.

[0143] (First gesture b)

[0144] Figure 24 This figure illustrates the first gesture b. The first gesture b is a gesture for stopping a moving object 10 by facing the palm in the X direction (b in the figure). When the first gesture b is performed, the moving object 10 stops from moving forward.

[0145] (First gesture c)

[0146] Figure 25 1 is a diagram for explaining the first gesture c. The first gesture c is an action of moving the hand to the left so as to move the mobile body 10 to the left. Figure 23 As shown in FIG. 1 and FIG. 2 , starting from the state where the hand is extended forward ( c1 in the figure), the fingertips are repeatedly directed toward the positive Y direction and then returned to the starting point ( c2 and c3 in the figure). When the first gesture c is performed, the mobile body 10 moves to the left.

[0147] (First gesture d)

[0148] Figure 26 d is a diagram for explaining the first gesture d. The first gesture d is an action of moving the hand in the right direction so as to move the moving body 10 in the right direction. Figure 23 As shown in FIG. 1 and FIG. 2 , the hand is extended forward ( d1 in the figure) as the starting point, and the fingertip is repeatedly directed toward the negative Y direction and then returned to the starting point ( d2 and d3 in the figure). When the first gesture d is executed, the moving body 10 moves to the right.

[0149] (First gesture e)

[0150] Figure 27 This diagram illustrates the first gesture e. The first gesture e involves waving with fingertips to cause the mobile object 10 to move backward. The gesture involves repeatedly pointing the palm toward the positive Z direction (e1 in the diagram) and moving the fingertips toward the user (bringing the fingertips closer to the palm) (e2 and e3 in the diagram). When the first gesture e is executed, the mobile object 10 moves backward.

[0151] (First gesture f)

[0152] Figure 28 This figure illustrates the first gesture f. The first gesture f involves protruding the index finger and thumb (or a specified finger) and rotating the protruding finger to the left, thereby rotating the mobile object 10 to the left. The hand is rotated (f2 in the figure) with the palm facing the positive X direction, the index finger and thumb protruding, and the other fingers gently clenched (bent) (f1 in the figure). The palm faces the negative X direction, and the back of the hand faces the positive X direction. The rotated hand is then returned to its original position (f3 in the figure). When the first gesture f is performed, the mobile object 10 rotates to the left.

[0153] (First gesture g)

[0154] Figure 29 This figure illustrates the first gesture g. The first gesture g involves protruding the index finger and thumb (or a specified finger) and rotating the protruding finger to the right, thereby rotating the mobile body 10 to the right. With the index finger and thumb protruding and the other fingers gently grasped (bent), the index finger is oriented in the positive X direction or in a direction intermediate between the positive X and positive Y directions (g1 in the figure). In this state, the index finger is rotated in the positive Z direction or in a direction intermediate between the positive Z and negative Y directions (g2 in the figure). The rotated hand is then returned to its original position (g3 in the figure). When the first gesture g is performed, the mobile body 10 rotates to the right.

[0155] [flow chart]

[0156] Figure 30 This is a flowchart illustrating an example of gesture recognition processing by the control device 50. First, the control device 50 determines whether the user is present in the first area (step S300). If the user is present in the first area, the control device 50 identifies the user's behavior based on the acquired images (step S302). For example, the behavior is the user's movement identified from images acquired continuously over time.

[0157] Next, the control device 50 identifies a gesture that matches the behavior recognized in step S302 by referring to the first gesture information 76 (step S304). If a gesture that matches the behavior recognized in step S302 is not included in the first gesture information 76, it is determined that the gesture for controlling the movement of the mobile object 10 has not been executed. Next, the control device 50 performs an action corresponding to the identified gesture (step S306).

[0158] If the user is not in the first area (or if they are in the second area), the control device 50 identifies the user's behavior based on the acquired image (step S308), and references the second gesture information 78 to identify a gesture that matches the behavior identified in step S308 (step S310). Next, the control device 50 performs an action corresponding to the identified gesture (step S312). This concludes the processing of one routine in this flowchart.

[0159] For example, in the above process, the recognition unit 54 may recognize the gesture of the user being tracked and not perform the gesture recognition process for the untracked person. This allows the control device 50 to control the mobile object based on the gesture of the user being tracked with reduced processing load.

[0160] As described above, the control device 50 switches the gesture to be recognized based on the area where the user is present, thereby being able to recognize the user's gesture with higher accuracy and operate the mobile object 10 according to the user's intention. As a result, user convenience is improved.

[0161] The control device 50 is as follows Figure 31 As shown, a gesture can be recognized in the third area AR3 with reference to the first gesture information 76 and the second gesture information 78. Figure 31 The third area AR3 is an area between the outer edge of the first area AR1 and a position at a predetermined distance from the outer edge of the first area AR1. The second area AR2 is an area outside the third area AR3.

[0162] When the user is in the first area AR1, the recognition unit 54 recognizes a gesture by referring to the first gesture information 76. When the user is in the second area AR2, the recognition unit 54 recognizes a gesture by referring to the first gesture information 76 and the second gesture information 78. Specifically, the recognition unit 54 determines whether the user is performing the first gesture included in the first gesture information 76 or the second gesture included in the second gesture information 78. When the user is performing the first gesture or the second gesture in the third area AR3, the control device 50 controls the mobile object 10 based on the user's action associated with the first gesture or the second gesture. When the user is in the second area AR2, the recognition unit 54 recognizes a gesture by referring to the second gesture information 78.

[0163] The third area AR3 is Figure 32 The third area AR3 may be a region divided by a boundary that is a predetermined distance from the outer edge of the first area AR1 to the inner side, i.e., the outer edge, and a boundary that is a predetermined distance from the outer edge of the first area AR1 to the outer side, i.e., the outer edge (the boundary is divided by a predetermined distance from the outer edge of the first area AR1 to the outer side, i.e., the outer edge). Figure 31 The third area AR3 and Figure 32 The area formed by the combination of the third area AR3 can be the third area).

[0164] For example, in the third area AR3, even when both a first gesture and a second gesture are recognized, the first gesture may be prioritized over the second gesture. Priority refers to situations where, for example, the movement of the mobile object 10 indicated by the first gesture differs from the movement of the mobile object 10 indicated by the second gesture, the first gesture is prioritized, or the second gesture is disregarded. Even when the user is unintentionally moving their arm, this is recognized as the second gesture. However, this is because small gestures using the hand or fingers are less likely to be performed unintentionally, while the user is more likely to be moving their hand or fingers with intent. By prioritizing the first gesture in this way, the user's intention can be more accurately recognized.

[0165] In the above example, the recognition unit 54 recognizes the user's body movements based on multiple images captured continuously (multiple images or animations captured at predetermined intervals). However, the recognition unit 54 may alternatively recognize the user's body movements based on a single image. In this case, for example, the recognition unit 54 compares the feature quantity representing the user's body movement contained in the single image with the feature quantity contained in the first gesture information 76 or the second gesture information 78, and recognizes that the user is performing a gesture for which the feature quantity matches closely or exceeds a predetermined level.

[0166] In the above example, when the recognition unit 54 uses an image captured by a camera (camera device) set at a position different from the mobile body 10 to recognize the user's body movements, the first area is an area within a specified distance from the camera device that captures the image, and the second area is an area set at a position farther than the specified distance from the camera device.

[0167] In the above example, the second area is located farther from the first area. However, the second area may be located at a different location from the first and second areas. For example, the first area may be located in a first direction, and the second area may be located in a direction different from the first direction.

[0168] According to the first embodiment described above, the control device 50 switches the gesture to be recognized according to the user's position relative to the moving object, thereby recognizing the user's gesture with higher accuracy and appropriately operating the moving object 10. As a result, user convenience is improved.

[0169] <Second embodiment>

[0170] The following describes a second embodiment. In the second embodiment, the main body 20 of the mobile object 10 includes a first camera (first imaging unit) and a second camera (second imaging unit). Gestures are recognized using images captured by these cameras. The following description focuses on the differences from the first embodiment.

[0171] Figure 33 This diagram illustrates an example of the functional configuration of the main body 20A of the mobile object 10 according to the second embodiment. The main body 20A includes a first camera 21 and a second camera 23 in place of the camera 22. The first camera 21 is the same as the camera 22. The second camera 23 is a camera that captures images of a user who remotely operates the mobile object 10. The second camera 23 captures images for recognizing user gestures. Remote operation is performed using gestures. The second camera 23 can control its imaging direction, for example, using a mechanical mechanism. The second camera 23 captures images centered on the user being tracked. The information processing unit 60 controls the mechanical mechanism, for example, so that the second camera 23's imaging direction faces the user being tracked.

[0172] The recognition unit 54 attempts to recognize a user's gesture based on the first image captured by the first camera 21 and the second image captured by the second camera 23. The recognition unit 54 prioritizes the recognition result based on the second image (the second recognition result) over the recognition result based on the first image (the first recognition result). The trajectory generation unit 56 generates a trajectory based on the surrounding conditions obtained from the first image and the motion associated with the recognized gesture. The travel control unit 58 controls the mobile object 10 based on the trajectory generated by the trajectory generation unit 56.

[0173] [flow chart]

[0174] Figure 34 This is a flowchart illustrating an example of the processing flow executed by the control device 50 of the second embodiment. First, the acquisition unit 52 of the control device 50 acquires a first image and a second image (step S400). Next, the recognition unit 54 attempts to recognize a gesture in each of the first and second images and determines whether the gesture can be recognized in both images (step S402). In this process, if the user is in the first area, the first gesture information 76 is referenced, while if the user is outside the first area, the second gesture information 78 is referenced.

[0175] If gestures can be recognized from both images, recognition unit 54 determines whether the recognized gestures are the same (step S404). If the recognized gestures are the same, recognition unit 54 adopts the recognized gesture (step S406). If the recognized gestures are different, recognition unit 54 adopts the gesture recognized from the second image (step S408). Thus, the second recognition result takes precedence over the first recognition result.

[0176] If a gesture cannot be recognized from either image in step S402, the recognition unit 54 adopts a recognizable gesture (either the gesture recognized from the first image or the gesture recognized from the second image) (step S406). For example, if the user is present in the first area and the user's gesture cannot be recognized based on the first image captured by the first camera 21, the recognition unit 54 refers to the first gesture information 76 and recognizes the user's gesture based on the second image captured by the second camera 23. The mobile object 10 is then controlled to perform an action corresponding to the adopted gesture. This concludes the processing of one routine in this flowchart.

[0177] According to the above-described processing, the control device 50 can recognize the user's gesture with higher accuracy.

[0178] In the second embodiment, the first gesture information 76 or the second gesture information 78 can be referenced regardless of the user's position, or gesture information (information in which the characteristic amount of the gesture is associated with the action of the mobile body 10) different from the first gesture information 76 or the second gesture information 78 (for example, regardless of the user's position) can be referenced.

[0179] According to the second embodiment described above, the control device 50 can recognize gestures using images captured by two or more cameras, thereby recognizing gestures with higher accuracy and controlling the moving object 10 based on the recognition results. As a result, user convenience can be improved.

[0180] [Variation of the Second Gesture]

[0181] Instead of the second gesture described above, the second gesture can be implemented as follows. For example, the second gesture can be an upper arm gesture, regardless of palm movement. This allows control device 50 to accurately recognize the second gesture even when performed from a distance. While examples are provided below, different implementations are also possible.

[0182] (Second gesture G)

[0183] Figure 35This diagram illustrates a variation of the second gesture G. The second gesture G involves bending the elbow, pointing the palm upward, and rotating the upper arm leftward to rotate the mobile object 10 leftward (G# in the diagram). When the second gesture G is performed, the mobile object 10 rotates leftward.

[0184] (Second gesture H)

[0185] Figure 36 This diagram illustrates a variation of the second gesture H. The second gesture H involves bending the elbow, pointing the palm upward, and rotating the upper arm to the right, thereby rotating the mobile object 10 to the right (H# in the diagram). When the second gesture H is performed, the mobile object 10 rotates to the right.

[0186] (Second gesture F)

[0187] Figure 37 This figure illustrates a modified example of the second gesture F. The second gesture F is an action of bending the elbow and pointing the palm upward to move the mobile object 10 backward (F# in the figure). When the second gesture F is performed, the mobile object 10 moves backward.

[0188] (Second gesture FR)

[0189] Figure 38 This diagram illustrates the second gesture FR. The second gesture FR involves bending the elbow, pointing the palm upward, and moving the movable body 10 backward while moving rightward (FR in the diagram) based on the rightward inclination of the upper arm. When the second gesture FR is performed, the movable body 10 moves backward while moving rightward based on the rightward inclination of the upper arm.

[0190] Figure 39 This diagram illustrates the second hand gesture FL. The second hand gesture FL involves bending the elbow, pointing the palm upward, and moving the mobile object 10 leftward (FR in the diagram) by determining the amount of movement based on the leftward inclination of the upper arm. When the second hand gesture FL is performed, the mobile object 10 moves leftward based on the leftward inclination of the upper arm.

[0191] As described above, the control device 50 controls the mobile object 10 based on the second upper arm gesture. For example, even when a person at a distance performs the second gesture, the control device 50 can recognize the second gesture with greater accuracy and control the mobile object 10 in accordance with the person's intention.

[0192] The above-described embodiment can be expressed as follows.

[0193] A gesture recognition device is configured to include:

[0194] a storage device storing a program; and

[0195] Hardware processor,

[0196] The hardware processor executes the program stored in the storage device to perform the following processing:

[0197] Get the image captured by the user;

[0198] identifying an area in which the user was present when the image was captured;

[0199] If the user is present in the first area when the image is captured, recognizing the user's gesture based on the image and first information for recognizing the user's gesture,

[0200] If the user is present in the second area when the image is captured, the user's gesture is recognized based on the plurality of images captured successively in time and second information for recognizing the user's gesture.

[0201] The above-described embodiment can be expressed as follows.

[0202] The gesture recognition device comprises:

[0203] a first camera unit for capturing images of the surroundings of the moving object;

[0204] a second camera unit configured to capture a user who remotely operates the mobile object;

[0205] a storage device storing a program; and

[0206] Hardware processor,

[0207] The hardware processor executes the program stored in the storage device.

[0208] performing a trial process of recognizing the user's gesture based on a first image captured by the first camera unit and a second image captured by the second camera unit, and prioritizing a recognition result based on the second image over a recognition result based on the first image;

[0209] The moving object is controlled based on a surrounding situation obtained from the image captured by the first imaging unit and a motion associated with the gesture recognized by the recognition unit.

[0210] The above-described embodiment can be expressed as follows.

[0211] The gesture recognition device has:

[0212] a first camera unit for capturing images of the surroundings of the moving object;

[0213] a second camera unit configured to capture a user who remotely operates the mobile object;

[0214] a storage device storing a program; and

[0215] Hardware processor,

[0216] The hardware processor executes the program stored in the storage device.

[0217] When the user is present in the first area and the user's gesture cannot be recognized based on the first image captured by the first camera unit, the user's gesture is recognized based on the second image captured by the second camera unit with reference to the first information;

[0218] The moving object is controlled based on the image captured by the first imaging unit according to the recognized gesture.

[0219] While specific embodiments of the present invention have been described above, the present invention is not limited to these embodiments at all, and various modifications and substitutions can be made without departing from the spirit of the present invention.

Claims

1. A processing system, wherein: The processing system comprises: an acquisition unit that acquires an image captured by a user; and a recognition unit that recognizes an area in which the user was present when the image was captured, recognizes a gesture of the user based on the image and first information for recognizing the gesture of the user when the user was present in a first area when the image was captured, and recognizes a gesture of the user based on the image and second information for recognizing the gesture of the user when the user was present in a second area when the image was captured, The recognition unit attempts to recognize the user's gesture based on a first image captured by a first camera unit and a second image captured by a second camera unit. Compared with a recognition result based on the first image, a recognition result based on the second image is preferentially used to recognize the gesture. The first camera unit captures the periphery of a moving object controlled based on the recognition result of the gesture, and the second camera unit captures the user who remotely operates the moving object.

2. A processing system, wherein: The processing system comprises: an acquisition unit that acquires an image captured by a user; and a recognition unit that recognizes an area in which the user was present when the image was captured, recognizes a gesture of the user based on the image and first information for recognizing the gesture of the user when the user was present in a first area when the image was captured, and recognizes a gesture of the user based on the image and second information for recognizing the gesture of the user when the user was present in a second area when the image was captured, When the user is present in the first area and the user's gesture cannot be recognized based on the first image captured by the first camera unit, the recognition unit recognizes the user's gesture based on the second image captured by the second camera unit with reference to the first information. The first camera unit captures the periphery of the moving body controlled based on the recognition result of the gesture, and the second camera unit captures the user who remotely operates the moving body.

3. A processing system, wherein: The processing system comprises: an acquisition unit, which acquires an image captured by a user; a recognition unit that recognizes an area in which the user was present when the image was captured, recognizes a gesture of the user based on the image and first information for recognizing the gesture of the user if the user was present in a first area when the image was captured, and recognizes a gesture of the user based on the image and second information for recognizing the gesture of the user if the user was present in a second area when the image was captured; a first camera unit for capturing images of the surroundings of the moving object; a second camera unit configured to capture a picture of the user who remotely operates the mobile object; a storage device storing reference information associating the user's gesture with the movement of the moving object; as well as a control unit that controls the mobile body based on the movement of the mobile body associated with the gesture of the user recognized by the recognition unit, with reference to the reference information; The recognition unit attempts to recognize the user's gesture based on a first image captured by the first camera unit and a second image captured by the second camera unit, and prioritizes a recognition result based on the second image over a recognition result based on the first image. The control unit controls the moving object based on a surrounding situation obtained from an image captured by the first imaging unit and a motion associated with the gesture recognized by the recognition unit.

4. A processing system, wherein: The processing system comprises: an acquisition unit, which acquires an image captured by a user; a recognition unit that recognizes an area in which the user was present when the image was captured, recognizes a gesture of the user based on the image and first information for recognizing the gesture of the user if the user was present in a first area when the image was captured, and recognizes a gesture of the user based on the image and second information for recognizing the gesture of the user if the user was present in a second area when the image was captured; a first camera unit for capturing images of the surroundings of the moving object; a second camera unit configured to capture a picture of the user who remotely operates the mobile object; a storage device storing reference information associating the user's gesture with the movement of the moving object; as well as a control unit that controls the mobile body based on the movement of the mobile body associated with the gesture of the user recognized by the recognition unit, with reference to the reference information; When the user is present in the first area and the user's gesture cannot be recognized based on the first image captured by the first camera unit, the recognition unit recognizes the user's gesture based on the second image captured by the second camera unit with reference to the first information. The control unit controls the moving object based on the image captured by the first imaging unit according to the gesture recognized by the recognition unit.

5. The processing system according to any one of claims 1 to 4, wherein: The first area is an area within a predetermined distance from an imaging device that captures the image, and the second area is an area set at a position farther from the imaging device than the predetermined distance.

6. The processing system according to any one of claims 1 to 4, wherein: The first information is information for recognizing a gesture based on movement of a hand or a finger, excluding movement of an arm.

7. The processing system according to any one of claims 1 to 4, wherein: The second information is information for recognizing a gesture including a movement of an arm.

8. The processing system according to claim 7, wherein: The first area is an area where the recognition unit cannot recognize the movement of the user's arm from an image captured of the user present in the first area, or where it is difficult to recognize the movement of the user's arm.

9. The processing system according to any one of claims 1 to 4, wherein: The recognition unit recognizes the user's gesture based on the image, the first information and the second information when the user is present in a third area spanning the first area and a second area adjacent to the first area outside the first area, or in a third area between the first area and a second area farther than the first area when the image is captured.

10. The processing system according to claim 9, wherein: When recognizing the user's gesture based on the image, the first information, and the second information, the recognition unit prioritizes a recognition result based on the image and the first information over a recognition result based on the image and the second information, and recognizes the user's gesture.

11. A processing method, wherein: The processing method enables the computer to perform the following processing: Get the image captured by the user; identifying an area in which the user was present when the image was captured; recognizing a gesture of the user based on the image and first information for recognizing the gesture of the user, in a case where the user is present in the first area when the image is captured; recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user, in a case where the user is present in the second area when the image is captured; as well as A process of attempting to recognize the user's gesture is performed based on a first image captured by a first camera unit and a second image captured by a second camera unit, wherein the recognition result based on the second image is preferentially used to recognize the gesture over the recognition result based on the first image, the first camera unit captures the periphery of a moving object controlled based on the recognition result of the gesture, and the second camera unit captures the user who remotely operates the moving object.

12. A processing method, wherein: The processing method enables the computer to perform the following processing: Get the image captured by the user; identifying an area in which the user was present when the image was captured; recognizing a gesture of the user based on the image and first information for recognizing the gesture of the user, in a case where the user is present in the first area when the image is captured; recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user, in a case where the user is present in the second area when the image is captured; as well as When the user is present in the first area and the user's gesture cannot be recognized based on the first image captured by the first camera unit, the user's gesture is recognized based on the second image captured by the second camera unit with reference to the first information, the first camera unit captures the periphery of the moving body controlled based on the recognition result of the gesture, and the second camera unit captures the user who remotely operates the moving body.

13. A processing method, wherein: The processing method enables the computer to perform the following processing: Get the image captured by the user; identifying an area in which the user was present when the image was captured; recognizing a gesture of the user based on the image and first information for recognizing the gesture of the user, in a case where the user is present in the first area when the image is captured; recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user, in a case where the user is present in the second area when the image is captured; referring to reference information stored in a storage device that associates the user's gesture with the movement of the moving object, and controlling the moving object based on the movement of the moving object associated with the recognized user's gesture; performing a process of tentatively recognizing a gesture of the user based on a first image captured by a first camera unit capturing a periphery of the mobile object and a second image captured by a second camera unit capturing the user remotely operating the mobile object, and prioritizing a recognition result based on the second image over a recognition result based on the first image; as well as The moving object is controlled based on a surrounding situation obtained from the image captured by the first imaging unit and a motion associated with the recognized gesture.

14. A processing method, wherein: The processing method enables the computer to perform the following processing: Get the image captured by the user; identifying an area in which the user was present when the image was captured; recognizing a gesture of the user based on the image and first information for recognizing the gesture of the user, in a case where the user is present in the first area when the image is captured; recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user, in a case where the user is present in the second area when the image is captured; referring to reference information stored in a storage device that associates the user's gesture with the movement of the moving object, and controlling the moving object based on the movement of the moving object associated with the recognized user's gesture; recognizing the user's gesture based on a second image captured by a second imaging unit that captures the user remotely operating the mobile object, with reference to the first information, when the user is present in the first area and the user's gesture cannot be recognized based on a first image captured by a first imaging unit that captures the periphery of the mobile object; as well as The moving object is controlled based on the image captured by the first imaging unit according to the recognized gesture.

Images