Processing system and processing method
The gesture recognition device enhances user convenience by using multiple imaging units and position-based gesture recognition to accurately control mobile objects, addressing the limitations of existing robot systems in gesture recognition and user interaction.
Patent Information
- Application Number
- JP2021031630
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-01
- Publication Date
- 2024-11-11
- Estimated Expiration
- 2041-03-01
AI Technical Summary
Existing robot systems do not provide sufficient user convenience in guiding and transporting luggage, as they may not accurately recognize gestures due to limitations in imaging and processing user movements.
A gesture recognition device and method that utilizes multiple imaging units to distinguish between different areas based on user proximity, employing specific gesture information sets for accurate recognition, and prioritizing recognition results based on user position to control mobile objects effectively.
Improves user convenience by enabling accurate gesture recognition and precise control of mobile objects, allowing users to easily operate and interact with robots through intuitive gestures.
Smart Images

Figure 0007580302000001 
Figure 0007580302000002 
Figure 0007580302000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a gesture recognition device, a moving object, a gesture recognition method, and a program. [Background technology]
[0002] Conventionally, robots that guide users to desired locations or transport luggage are known. For example, a mobile robot that moves while maintaining a predetermined distance from people when providing the above-mentioned services has been disclosed (for example, see Patent Document 1). [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 5617562 Summary of the Invention [Problem to be solved by the invention]
[0004] However, the above techniques may not provide sufficient convenience for users.
[0005] The present invention has been made in consideration of the above circumstances, and one of its objects is to provide a gesture recognition device, a moving object, a gesture recognition method, and a program that can improve user convenience. [Means for solving the problem]
[0006] The gesture recognition device, the moving object, the gesture recognition method, and the program according to the present invention employ the following configuration. (1): A gesture recognition device includes an acquisition unit that acquires an image of a user, and a recognition unit that recognizes an area in which the user is present when the image is captured, and, if the user is present in a first area when the image is captured, recognizes a gesture of the user based on the image and first information for recognizing a gesture of the user, and, if the user is present in a second area when the image is captured, recognizes a gesture of the user based on the image and second information for recognizing a gesture of the user.
[0007] (2): In the above aspect (1), the first area is an area within a predetermined distance from an imaging device that captures the image, and the second area is an area set at a position farther than the predetermined distance from the imaging device.
[0008] (3): In the above aspect (1) or (2), the first information is information for recognizing a gesture based on hand or finger movements, but not including arm movements.
[0009] (4): In any one of the above aspects (1) to (3), the second information is information for recognizing a gesture including an arm movement.
[0010] (5): In the above aspect (4), the first area is an area in which the recognition unit cannot or has difficulty recognizing the arm movement of a user present in the first area from an image captured by the user.
[0011] (6): In any of the above aspects (1) to (5), the recognition unit recognizes a gesture of the user based on the image, the first information, and the second information when the user is present in a third area that spans the first area and a second area adjacent to the first area outside the first area, or a third area between the first area and the second area farther away than the first area, when the image is captured.
[0012] (7): In the above aspect (6), when the recognition unit recognizes the gesture of the user based on the image, the first information, and the second information, the recognition unit recognizes the gesture of the user by giving priority to the result of recognition based on the image and the first information over the result of recognition based on the image and the second information.
[0013] (8): The mobile object is equipped with a gesture recognition system according to any one of the above aspects (1) to (7).
[0014] (9): In the aspect (8) above, the device further includes a storage device storing reference information associating the user's gesture with the movement of the moving object, and a control unit that controls the moving object based on the movement of the moving object associated with the user's gesture recognized by the recognition unit by referring to the reference information.
[0015] (10): In the above aspect (9), a first imaging unit is provided which captures an image of the surroundings of a moving body, and a second imaging unit which captures an image of a user remotely operating the moving body, and the recognition unit attempts a process of recognizing a gesture of the user based on a first image captured by the first imaging unit and a second image captured by the second imaging unit, and adopts a recognition result based on the second image in preference to a recognition result based on the first image, and the control unit controls the moving body based on the surrounding situation obtained from the image captured by the first imaging unit and an operation associated with the gesture recognized by the recognition unit.
[0016] (11): In any of the above aspects (8) to (10), a first imaging unit that images the surroundings of a moving body and a second imaging unit that images a user remotely operating the moving body are provided, and the recognition unit, when the user is present in a first area and the recognition unit cannot recognize the gesture of the user based on the first image captured by the first imaging unit, recognizes the gesture of the user based on the second image captured by the second imaging unit by referring to the first information, and includes a control unit that controls the moving body based on the image captured by the first imaging unit in accordance with the gesture recognized by the recognition unit.
[0017] (12): In any of the above aspects (8) to (11), the recognition unit includes a control unit that tracks a target user based on a captured image, recognizes the gestures of the tracked user, does not perform processing to recognize gestures of persons not being tracked, and controls the moving object based on the gestures of the tracked user.
[0018] (13): A gesture recognition method according to one embodiment of the present invention includes a computer acquiring an image of a user, recognizing an area in which the user was present when the image was captured, and, if the user was present in a first area when the image was captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user, and, if the user was present in a second area when the image was captured, recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user.
[0019] (14): A program according to one embodiment of the present invention causes a computer to acquire an image of a user, recognize an area in which the user was present when the image was captured, and, if the user was present in a first area when the image was captured, recognize a gesture of the user based on a plurality of the images and first information for recognizing a gesture of the user, and, if the user was present in a second area when the image was captured, recognize a gesture of the user based on the images and second information for recognizing a gesture of the user. Effect of the Invention
[0020] According to (1)-(14), the recognition unit recognizes a gesture using the first information or the second information depending on the position of the user, thereby improving convenience for the user.
[0021] According to (6), the gesture recognition device can recognize a gesture with higher accuracy by recognizing the gesture using the first information and the second information.
[0022] According to (8)-(11), a moving object can perform an action that reflects a user's intention. For example, a user can easily operate a moving object by issuing a simple instruction.
[0023] According to (10) or (11), the mobile object performs an action according to the gesture recognized based on the images acquired by the camera for acquiring images to recognize the surroundings and the camera for remote control, thereby enabling the mobile object to recognize gestures more accurately and perform an action according to the user's intention.
[0024] According to (12), a mobile object can track a user to whom a service is being provided and perform processing by focusing on the gestures of the tracked user, thereby reducing the processing load and improving user convenience. [Brief description of the drawings]
[0025] [Figure 1] FIG. 1 is a diagram illustrating an example of a moving object 10 including a control device according to an embodiment. [Diagram 2] 2 is a diagram showing an example of a functional configuration included in a main body 20 of a moving object 10. FIG. [Diagram 3] FIG. 13 is a diagram showing an example of a trajectory. [Figure 4] 13 is a flowchart showing an example of the flow of a tracking process. [Diagram 5] 11 is a diagram for explaining a process for extracting a feature amount of a user and a process for registering the feature amount. FIG. [Figure 6] 4 is a diagram for explaining the process of tracking a user by the recognition unit 54 (the process of step S104 in FIG. 3). FIG. [Figure 7] FIG. 11 is a diagram for explaining a tracking process using a feature amount. [Figure 8] FIG. 11 is a diagram for explaining a process for identifying a user to be tracked. [Figure 9] 11 is a diagram for explaining another example of the process (the process of step S104 in FIG. 3) in which the recognition unit 54 tracks a user. FIG. [Figure 10] FIG. 13 is a diagram for explaining a process for identifying a user as a tracking target. [Figure 11] 13 is a flowchart showing an example of the flow of a behavior control process. [Figure 12] FIG. 11 is a diagram for explaining a process of recognizing a gesture. [Figure 13] FIG. 13 is a diagram showing users present in a first area. [Figure 14] FIG. 13 is a diagram showing users present in a second area. [Figure 15] FIG. 13 is a diagram for explaining a second gesture A. [Figure 16] FIG. 13 is a diagram for explaining a second gesture B. [Figure 17] FIG. 13 is a diagram for explaining a second gesture C. [Figure 18]13 is a diagram for explaining a second gesture D. FIG. [Figure 19] 13 is a diagram for explaining a second gesture E. FIG. [Figure 20] FIG. 13 is a diagram for explaining a second gesture F. [Figure 21] 13 is a diagram for explaining a second gesture G. FIG. [Figure 22] 13 is a diagram for explaining a second gesture H. FIG. [Diagram 23] FIG. 13 is a diagram for explaining a first gesture a. [Figure 24] FIG. 13 is a diagram for explaining a first gesture b. [Diagram 25] FIG. 11 is a diagram for explaining a first gesture c. [Figure 26] FIG. 13 is a diagram for explaining a first gesture d. [Figure 27] FIG. 13 is a diagram for explaining a first gesture e. [Figure 28] FIG. 11 is a diagram for explaining a first gesture f. [Figure 29] FIG. 11 is a diagram for explaining a first gesture g. [Diagram 30] 10 is a flowchart showing an example of a process for the control device 50 to recognize a gesture. [Diagram 31] FIG. 1 is a diagram showing the third region. [Diagram 32] FIG. 2 is a diagram showing the third region. [Diagram 33] FIG. 11 is a diagram for explaining an example of a functional configuration of a main body 20A of a moving object 10 according to a second embodiment. [Diagram 34] 10 is a flowchart showing an example of a flow of processes executed by a control device 50 of a second embodiment. [Diagram 35] 13 is a diagram for explaining a modified example of the second gesture G. FIG. [Diagram 36] 13 is a diagram for explaining a modified example of the second gesture H. FIG. [Figure 37]13 is a diagram for explaining a modified example of the second gesture F. FIG. [Figure 38] FIG. 13 is a diagram for explaining a second gesture FR. [Figure 39] 11 is a diagram for explaining a second gesture FL. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0026] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A gesture recognition device, a moving object, a gesture recognition method, and a program according to embodiments of the present invention will be described below with reference to the drawings.
[0027] First Embodiment [Overall configuration] FIG. 1 is a diagram showing an example of a mobile body 10 equipped with a control device according to an embodiment. The mobile body 10 is an autonomous mobile robot. The mobile body 10 supports the actions of a user. For example, the mobile body 10 supports shopping or customer service of a customer according to instructions from a store clerk, a customer, a facility staff member (hereinafter, these persons are referred to as "users"), etc., and supports the staff member's work.
[0028] The moving object 10 includes a main body 20, a container 92, and one or more wheels 94 (wheels 94A and 94B in the figure). The moving object 10 moves in response to instructions based on a user's gestures or voice, operations on an input unit (a touch panel to be described later) of the moving object 10, and operations on a terminal device (e.g., a smartphone). The moving object 10 recognizes gestures based on images captured by a camera 22 provided on the main body 20, for example.
[0029] For example, the mobile unit 10 drives the wheels 94 to move so as to follow the customer in accordance with the movement of the user, or to move so as to lead the customer. At this time, the mobile unit 10 explains the product or work to the user, or guides the user to the product or object that the user is looking for. In addition, the user can store the product or baggage that he or she plans to purchase in the container 92 that stores them.
[0030] In this embodiment, the moving body 10 is described as having a container 92, but instead of (or in addition to) this, the moving body 10 may be provided with a seating section on which the user sits, a housing on which the user climbs, a step on which the user places their feet, etc., so that the user can move together with the moving body 10.
[0031] 2 is a diagram showing an example of a functional configuration included in the main body 20 of the moving object 10. The main body 20 includes a camera 22, a communication unit 24, a position identification unit 26, a speaker 28, a microphone 30, a touch panel 32, a motor 34, and a control device 50.
[0032] Camera 22 captures images of the periphery of moving body 10. Camera 22 is, for example, a fisheye camera capable of capturing images of the periphery of moving body 10 at a wide angle (e.g., 360 degrees). Camera 22 is attached, for example, to the upper part of moving body 10, and captures images of the periphery of moving body 10 at a wide angle in the horizontal direction. Camera 22 may be realized by combining multiple cameras (multiple cameras capturing images in a horizontal range of 120 degrees or 60 degrees). The number of cameras 22 is not limited to one, and multiple cameras may be provided on moving body 10.
[0033] The communication unit 24 is a communication interface for communicating with other devices using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like.
[0034] The position identification unit 26 identifies the position of the mobile body 10. The position identification unit 26 acquires position information of the mobile body 10 by a GPS (Global Positioning System) device (not shown) built into the mobile body 10. The position information may be, for example, two-dimensional map coordinates or latitude and longitude information.
[0035] The speaker 28 outputs, for example, a predetermined sound. The microphone 30 receives, for example, an input of a sound uttered by the user.
[0036] The touch panel 32 is configured by superimposing a display unit such as an LCD (liquid crystal display) or an organic EL (electroluminescence) on an input unit capable of detecting a touch position of an operator by a coordinate detection mechanism. The display unit displays GUI (graphical user interface) switches for operation. When the input unit detects a touch operation, a flick operation, a swipe operation, or the like on the GUI switch, it generates an operation signal indicating that a touch operation on the GUI switch has been performed, and outputs the operation signal to the control device 50. In response to the operation, the control device 50 causes the speaker 28 to output a sound or causes the touch panel 32 to display an image. The control device 50 may also cause the moving object 10 to move in response to the operation.
[0037] The motor 34 drives the wheels 94 to move the mobile body 10. The wheels 94 include, for example, drive wheels that are driven in a rotational direction by the motor 34, and steering wheels that are non-driven wheels that are driven in a yaw direction. By adjusting the angle of the steering wheels, the mobile body 10 can change its course or rotate on its own axis.
[0038] In this embodiment, the moving body 10 has wheels 94 as a mechanism for realizing movement, but this embodiment is not limited to this configuration. For example, the moving body 10 may be a multi-legged robot.
[0039] The control device 50 includes, for example, an acquisition unit 52, a recognition unit 54, a trajectory generation unit 56, a travel control unit 58, an information processing unit 60, and a storage unit 70. Some or all of the acquisition unit 52, the recognition unit 54, the trajectory generation unit 56, the travel control unit 58, and the information processing unit 60 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these functional units may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage unit 70 (a storage device having a non-transient storage medium) such as a hard disk drive (HDD) or a flash memory, or may be stored in a removable storage medium (a non-transient storage medium) such as a DVD or a CD-ROM, and may be installed by mounting the storage medium in a drive device. The acquisition unit 52, the recognition unit 54, the trajectory generation unit 56, the traveling control unit 58, or the information processing unit 60 may be provided in a device different from the control device 50 (the moving body 10). For example, the recognition unit 54 may be provided in another device, and the control device 50 may control the moving body 10 based on the processing result of the other device. In addition, a part or all of the information stored in the storage unit 70 may also be stored in the other device. A configuration including one or more functional units among the acquisition unit 52, the recognition unit 54, the trajectory generation unit 56, the traveling control unit 58, or the information processing unit 60 may be configured as a system.
[0040] The storage unit 70 stores map information 72, gesture information 74, and user information 80. The map information 72 is information that expresses the shapes of roads and passages in a facility using, for example, links indicating the roads and passages in the facility and nodes connected by the links. The map information 72 may include information such as the curvature of the road and POI (Point Of Interest) information.
[0041] The gesture information 74 is information in which information about gestures (feature amount of a template) and the motion of the moving object 10 are associated with each other. The gesture information 74 includes first gesture information 76 (first information, reference information) and second gesture information 78 (second information, reference information). The user information 80 is information indicating the feature amount of a user. The gesture information 74 and the user information 80 will be described in detail later.
[0042] The acquisition unit 52 acquires an image (hereinafter referred to as a "peripheral image") captured by the camera 22. The acquisition unit 52 holds the acquired peripheral image as pixel data in the fish-eye camera coordinate system.
[0043] The recognition unit 54 recognizes a body movement (hereinafter referred to as a "gesture") by the user U based on one or more peripheral images. The recognition unit 54 recognizes a gesture by matching a feature amount of the user's gesture extracted from the peripheral image with a feature amount of a template (a feature amount indicating a gesture). The feature amount is data representing, for example, characteristic parts such as a person's fingers, finger joints, wrists, arms, and bones, links connecting them, and the inclination, position, etc. of the link.
[0044] The trajectory generation unit 56 generates a trajectory along which the moving body 10 should travel in the future, based on the user's gestures, a destination set by the user, surrounding objects, the user's position, map information 72, etc. The trajectory generation unit 56 generates a trajectory that allows the moving body 10 to move smoothly to the destination point by combining multiple arcs. FIG. 3 is a diagram showing an example of a trajectory. For example, the trajectory is generated by combining three arcs. Each arc has a different radius of curvature R m1 , R m2, R m3 For each of them, the forecast period T m1 , T m2 , T m3 The position of the end point is Z m1 , Z m2 , Z m3 In addition, the forecast period T m1 The orbit for (first prediction period orbit) is divided into three equal parts, for example, and the positions of each are Z m11 , Z m12 , Z m13 The moving direction of the moving body 10 at the reference point is defined as the X direction, and the direction perpendicular to the X direction is defined as the Y direction. The first tangent is Z m1 The first tangent is a tangent to the target point. The direction of the first tangent is the X' direction, and the direction perpendicular to the X' direction is the Y' direction. The angle between the first tangent and a line segment extending in the X direction is θ m1 The angle between the line segment extending in the Y direction and the line segment extending in the Y' direction is θ m1 The point where the line segment extending in the Y direction intersects with the line segment extending in the Y' direction is the center of the arc of the first prediction period orbit. The second tangent is Z m2 The second tangent is a tangent to the target point. The direction of the second tangent is the X' direction, and the direction perpendicular to the X' direction is the Y' direction. The angle between the second tangent and the line segment extending in the X direction is θ m1 +θ m2 The angle between the line segment extending in the Y direction and the line segment extending in the Y" direction is θ m2 The point where the line segment extending in the Y direction intersects with the line segment extending in the Y" direction is the center of the arc of the second prediction period trajectory. The arc of the third prediction period trajectory is Z m2 and Z m3 The central angle of this arc is θ3. The trajectory generating unit 56 may perform the calculation by fitting the state to a geometric model such as a Bezier curve. The trajectory is actually generated as a collection of a finite number of trajectory points, for example.
[0045] The trajectory generating unit 56 performs coordinate conversion between the Cartesian coordinate system and the fisheye camera coordinate system. A one-to-one relationship is established between the coordinates of the Cartesian coordinate system and the fisheye camera coordinate system, and the relationship is stored in the storage unit 70 as correspondence information. The trajectory generating unit 56 generates a trajectory in the Cartesian coordinate system (Cartesian coordinate system trajectory) and converts the trajectory into a trajectory in the fisheye camera coordinate system (fisheye camera coordinate system trajectory). The trajectory generating unit 56 calculates the risk of the fisheye camera coordinate system trajectory. The risk is an index value indicating the likelihood that the moving body 10 will approach an obstacle. The risk tends to be higher the smaller the distance between the trajectory (trajectory point of the trajectory) and the obstacle, and lower the risk tends to be higher the larger the distance between the trajectory (trajectory point) and the obstacle.
[0046] If the total risk value or the risk of each trajectory point meets a predetermined criterion (for example, if the total risk value is less than or equal to threshold value Th1 and the risk of each trajectory point is less than or equal to threshold value Th2), the trajectory generation unit 56 adopts the trajectory that meets the criterion as the trajectory along which the moving body will move.
[0047] If the above trajectory does not satisfy the preset criteria, the following process may be performed. The trajectory generating unit 56 detects a drivable space in the fisheye camera coordinate system, and converts the detected drivable space in the fisheye camera coordinate system into a drivable space in the Cartesian coordinate system. The drivable space is a space in the moving direction of the moving body 10 excluding obstacles and the area around the obstacles (areas where a risk is set or areas where the risk is equal to or greater than a threshold). The trajectory generating unit 56 corrects the trajectory so that the trajectory falls within the drivable space that has been coordinate-converted into the Cartesian coordinate system. The trajectory generating unit 56 converts the Cartesian coordinate system trajectory into a fisheye camera coordinate system trajectory, and calculates the risk of the fisheye camera coordinate system trajectory based on the surrounding image and the fisheye camera coordinate system trajectory. This process is repeated to search for a trajectory that satisfies the above preset criteria.
[0048] The travel control unit 58 causes the moving body 10 to travel along a trajectory that satisfies a preset standard. The travel control unit 58 outputs a command value for causing the moving body 10 to travel along the trajectory to the motor 34. The motor 34 rotates the wheels 94 in accordance with the command value, and moves the moving body 10 along the trajectory.
[0049] The information processing unit 60 controls various devices and equipment included in the main body 20. The information processing unit 60 controls, for example, the speaker 28, the microphone 30, and the touch panel 32. The information processing unit 60 also recognizes voice input to the microphone 30 and operations performed on the touch panel 32. The information processing unit 60 operates the moving object 10 based on the result of the recognition.
[0050] In the above example, the recognition unit 54 has been described as recognizing the user's body motion based on an image captured by the camera 22 provided on the moving body 10. However, the recognition unit 54 may recognize the user's body motion based on an image captured by a camera not provided on the moving body 10 (a camera provided at a position different from the moving body 10). In this case, the image captured by the camera is transmitted to the control device 50 via communication, and the control device 50 acquires the transmitted image and recognizes the user's body motion based on the acquired image. The recognition unit 54 may also recognize the user's body motion based on a plurality of images. For example, the recognition unit 54 may recognize the user's body motion based on an image captured by the camera 22 or a plurality of images captured by a camera provided at a position different from the moving body 10. For example, the recognition unit 54 may recognize the user's body motion from each image, apply the recognition result to a predetermined criterion, and recognize the user's body motion, or perform image processing on a plurality of images to generate one or more images, and recognize the user's intended body motion from the generated images.
[0051] [Support process] The mobile object 10 executes a support process for supporting the user in shopping. The support process includes a process related to tracking and a process related to behavior control.
[0052] [Tracking process (part 1)] FIG. 4 is a flowchart showing an example of the flow of the tracking process. First, the control device 50 of the moving object 10 accepts user registration (step S100). Next, the control device 50 tracks the user registered in step S100 (step S102). Next, the control device 50 judges whether or not tracking is successful (step S104). If tracking is successful, the process proceeds to step S200 in FIG. 11, which will be described later. If tracking is not successful, the control device 50 identifies the user (step S106).
[0053] (User registration process) The process of registering a user in step S100 will now be described. The control device 50 of the mobile object 10 confirms the user's intention to register based on a specific gesture, voice, or operation on the touch panel 32 of the user (e.g., a customer visiting a store). If the user's intention to register is confirmed, the recognition unit 54 of the control device 50 extracts the user's feature amount and registers the extracted feature.
[0054] FIG. 5 is a diagram for explaining a process for extracting a feature amount of a user and a process for registering a feature amount. The recognition unit 54 of the control device 50 identifies a user from an image IM1 captured by the user, and recognizes the joint points of the identified user (performs skeleton processing). For example, the recognition unit 54 estimates the user's face, facial parts, neck, shoulders, elbows, wrists, waist, ankles, etc. from the image IM1, and performs skeleton processing based on the positions of each estimated part. For example, the recognition unit 54 performs skeleton processing using a known method (e.g., a method such as open pose) for estimating the user's joint points and skeleton using deep learning. Next, the recognition unit 54 identifies the user's face, upper body, lower body, etc. based on the result of the skeleton processing, extracts feature amounts for each of the identified face, upper body, and lower body, and registers the extracted feature amounts in the storage unit 70 as the user's feature amounts. The facial feature amounts are, for example, male, female, hairstyle, and facial feature amounts. The upper body feature amount is, for example, the color of the upper body part. The feature amount of the lower body is, for example, the color of the lower body.
[0055] (User tracking process) The process of tracking a user in step S102 will be described. FIG. 6 is a diagram for explaining the process of tracking a user by the recognition unit 54 (the process of step S104 in FIG. 4). The recognition unit 54 detects a user from an image IM2 captured at time T. The recognition unit 54 detects this detected person from an image IM3 captured at time T+1. The recognition unit 54 estimates the position of the user at time T+1 based on the position and movement direction of the user at time T and before time T, and identifies a user present near the estimated position as a user to be tracked (tracking target). If the user can be identified, tracking is deemed to have been successful.
[0056] In the tracking process, the recognition unit 54 may track the user using the feature amount of the user in addition to the position of the user at time T+1 as described above. FIG. 7 is a diagram for explaining the tracking process using the feature amount. For example, the recognition unit 54 estimates the position of the user at time T+1, identifies a user who is present near the estimated position, and further extracts the feature amount of the user. When the extracted feature amount matches the registered feature amount by a threshold value or more, the control device 50 estimates that the identified user is the user to be tracked, and determines that tracking has been successful.
[0057] For example, even if the user to be tracked overlaps or intersects with another person, the user can be tracked more accurately based on the change in the user's position and the user's features as described above.
[0058] (Process to identify users) The process of identifying the user in step S106 will be described. When the recognition unit 54 is not successful in tracking the user, as shown in FIG. 8, it compares the feature amount of people in the vicinity with the feature amount of the registered user to identify the user to be tracked. For example, the recognition unit 54 extracts the feature amount of each person included in the image. The recognition unit 54 compares the feature amount of each person with the feature amount of the registered user to identify a person who matches the feature amount of the registered user by a threshold value or more. The identified user is regarded as the user to be tracked.
[0059] Through the above process, the recognition unit 54 of the control device 50 can track the user with greater accuracy.
[0060] [Tracking process (part 2)] In the above example, the user is described as a customer visiting a store, but if the user is a store clerk or facility staff member (e.g., a medical professional within the facility), the following processing may be performed.
[0061] (User registration process) The process of registering a user in step S102 may be performed as follows. Fig. 9 is a diagram for explaining another example of the process of tracking a user by the recognition unit 54 (the process of step S102 in Fig. 4). The recognition unit 54 extracts features of a person's face from a captured image. The recognition unit 54 compares the extracted features of the face with features of the face of a user to be tracked that is registered in advance in the user information 80, and if they match, determines that the person included in the image is the user to be tracked.
[0062] (Process to identify users) The process of identifying the user in step S106 may be performed as follows: If the recognition unit 54 is not successful in tracking the user, as shown in Fig. 10, it compares the facial features of people in the vicinity with the features of registered users, and identifies a person having features that match a threshold or more as the user to be tracked.
[0063] As described above, the recognition unit 54 of the control device 50 can track the user with greater accuracy.
[0064] [Processing related to behavior control] FIG. 11 is a flowchart showing an example of the flow of the behavior control process. This process is executed after the process of step S104 in FIG. 4. The control device 50 recognizes the user's gesture (step S200), and controls the behavior of the moving object 10 based on the recognized gesture (step S202). Next, the control device 50 judges whether or not to end the service (step S204). If the service is not to be ended, the process returns to the process of step S102 in FIG. 4 and continues tracking. If the service is to be ended, the control device 50 erases the registered information related to the user, such as the user's feature amount (step S206). This ends one routine of this flowchart.
[0065] The process of step S200 will be described. FIG. 12 is a diagram for explaining the process of recognizing a gesture. The control device 50 extracts an area including one or both of the arm and the hand (hereinafter, the target area) from the skeleton processing result, and extracts a feature amount indicating the state of one or both of the arm and the hand in the extracted target area. The control device 50 specifies a feature amount matching the feature amount indicating the above-mentioned state from the feature amount included in the gesture information 74. The control device 50 causes the moving object 10 to execute a motion of the moving object 10 associated with the specified feature amount in the gesture information 74.
[0066] (Gesture recognition processing) The control device 50 determines whether to refer to the first gesture information 76 or the second gesture information 78 of the gesture information 74 based on the relative positions of the moving object 10 and the user. As shown in FIG. 13, when the user is not a predetermined distance away from the moving object, in other words, when the user is in the first area AR1 set based on the moving object 10, the control device 50 determines whether the user is making the same gesture as the gesture included in the first gesture information 76. As shown in FIG. 14, when the user is a predetermined distance away from the moving object, in other words, when the user is in the second area set based on the moving object 10 (when the user is not in the first area AR1), the control device 50 determines whether the user is making the same gesture as the gesture included in the second gesture information 78.
[0067] The first gesture included in the first gesture information 76 is a gesture using the hand without using the arm, and the second gesture included in the second gesture information 78 is a gesture using the arm (the arm between the elbow and the hand) and the hand. The first gesture may be a body movement such as a gesture or a small hand movement smaller than the second gesture. A small body movement means that when the moving body 10 is made to perform a certain movement (the same movement such as moving straight), the body movement of the first gesture is smaller than the body movement of the second gesture. For example, the first movement may be a gesture using the hand or fingers, and the second gesture may be a gesture using the arm. For example, the first movement may be a gesture using the leg below the knee, and the second gesture may be a gesture using the lower half of the body. For example, the first movement may be a gesture using the hand or foot, and the second gesture may be a gesture using the whole body such as jumping.
[0068] When the camera 22 of the moving body 10 captures an image of a user in the first area AR1, the arm portion is difficult to fit in the image, but the hand and fingers fit in the image, as shown in FIG. 13. The first area AR1 is an area in which the recognition unit 54 cannot or has difficulty recognizing the arm of the user in the image captured of the user in the first area AR1. When the camera 22 of the moving body 10 captures an image of a user in the second area AR2, the arm portion fits in the image, as shown in FIG. 14. Therefore, as described above, when the user is in the first area AR1, the recognition unit 54 recognizes the gesture using the first gesture information 76, and when the user is in the second area AR2, the recognition unit 54 recognizes the gesture using the second gesture information 78, thereby making it possible to recognize the user's gesture with higher accuracy. Below, the second gesture and the first gesture will be described in that order.
[0069] [Gestures and actions included in the second gesture information] Hereinafter, the front direction (forward direction) of the user is referred to as the X direction, the direction intersecting the front direction is referred to as the Y direction, and the direction intersecting the X direction and the Y direction and opposite to the vertical direction is referred to as the Z direction. Below, gestures for moving the moving body 10 will be described using the right arm and right hand, but the same movements are also possible when the left arm and left hand are used.
[0070] (Second gesture A) Fig. 15 is a diagram for explaining the second gesture A. The left side of Fig. 15 shows the gesture, and the right side of Fig. 15 shows the action of the moving object 10 corresponding to the gesture (this also applies to the following figures). The gesture will be described below assuming that it is performed by a user P1 (a store clerk), for example (this also applies to the following figures). In the figure, P2 is a customer.
[0071] Gesture A is a gesture in which the user pushes his / her arm and hand from near the body forward so as to move the moving object 10 located behind the user in front of the user. The hand is rotated so that the arm and hand are approximately parallel to the negative Y direction and the thumb points in the positive Z axis direction (A1 in the figure), and in this state, the shoulder or elbow joint is moved to move the hand in the positive X direction (A2 in the figure), and further the fingertips are approximately parallel to the positive X direction (A3 in the figure). In this state, the palm faces the positive Z direction. Then, the hand and arm are rotated so that the palm faces the negative Z direction with the fingertips approximately parallel to the X direction (A4, A5 in the figure). When the second gesture A is performed, the moving object 10 located behind the user P moves in front of the user P1.
[0072] (Second gesture B) 16 is a diagram for explaining the second gesture B. The second gesture B is a gesture in which the arm and hand are thrust forward to move the moving object 10 forward. With the arm and hand stretched out with the palm facing in the negative Z direction, the arm and hand are thrust out so as to be parallel to the direction in which the moving object 10 is to move (for example, the positive X direction) (B1 to B3 in the figure). When the second gesture B is performed, the moving object 10 moves in the direction indicated by the fingertips.
[0073] (Second gesture C) 17 is a diagram for explaining the second gesture C. The second gesture C is a gesture in which the arm and hand are extended forward with the palm facing in the X direction to stop the moving body 10 moving forward (in the figure, C1 and C2). When the second gesture C is performed, the moving body 10 goes from a moving forward state to a stopped state.
[0074] (Second gesture D) FIG. 18 is a diagram for explaining the second gesture D. The second gesture D is an action of moving the arm and hand to the left so as to move the moving object 10 to the left. From a state in which the arm and hand are extended forward (D1 in the figure), the palm is rotated approximately 90 degrees clockwise to point the thumb in the positive Z direction (D2 in the figure), and the arm and hand are swung in the positive Y direction from this state as a starting point, and the action of returning the arm and hand to the starting point is repeated (D3, D4 in the figure). When the second gesture D is performed, the moving object 10 moves to the left. When the arm and hand are returned to the state of D1 in the figure described above, the moving object 10 moves forward without moving to the left.
[0075] (Second gesture E) FIG. 19 is a diagram for explaining the second gesture E. The second gesture E is an action of moving the arm and hand to the right so as to move the moving body 10 to the right. From a state in which the arm and hand are extended forward (E1 in the figure), the palm is rotated counterclockwise with the thumb facing the ground (E2 in the figure), and the arm and hand are swung in the negative Y direction from this state as a starting point, and the action of returning the arm and hand to the starting point is repeated (E3, E4 in the figure). When the second gesture E is performed, the moving body 10 moves to the right. When the arm and hand are returned to the state of E1 in the figure described above, the moving body 10 moves forward without moving to the right.
[0076] (Second gesture F) 20 is a diagram for explaining the second gesture F. The second gesture F is a motion of waving so as to move the moving object 10 backward. The palm is turned in the positive Z direction (F1 in the figure), and the arm or wrist is moved repeatedly so that the fingertips point toward the user (F2 to F5 in the figure). When the second gesture F is performed, the moving object 10 moves backward.
[0077] (Second gesture G) FIG. 21 is a diagram for explaining the second gesture G. The second gesture G is an action of sticking out an index finger (or a specified finger) and rotating the finger sticking out to the left so as to rotate the moving object 10 to the left. The palm is turned in the negative Z direction (G1 in the figure), the index finger is stuck out and the other fingers are lightly clenched (bent) (G2 in the figure), the wrist or arm is moved to point the fingertips in the positive Y direction, and then the arm and hand are returned to the state of G1 in the figure (G3, G4 in the figure). When the second gesture G is performed, the moving object 10 rotates to the left.
[0078] (Second gesture H) 22 is a diagram for explaining the second gesture H. The second gesture H is an action of sticking out an index finger (or a specified finger) and rotating the finger sticking out to the right so as to rotate the moving object 10 to the right. The palm of the hand is turned in the negative Z direction (H1 in the figure), the index finger is stuck out and the other fingers are lightly clenched (bent) (H2 in the figure), the wrist or arm is moved to point the fingertips in the negative Y direction, and then the arm and hand are returned to the state of H1 in the figure (H3, H4 in the figure). When the second gesture H is performed, the moving object 10 rotates to the right.
[0079] [Gesture included in the first gesture information] (First gesture a) 23 is a diagram for explaining the first gesture a. The first gesture a is a gesture in which the hand is thrust forward to move the moving object 10 forward. The thumb is pointed in the positive Z direction and the back of the hand is parallel to the Z direction (a in the figure). When the first gesture a is made, the moving object 10 moves in the direction indicated by the fingertip.
[0080] (First gesture b) 24 is a diagram for explaining the first gesture b. The first gesture b is a gesture in which the palm is directed in the X direction to stop the moving body 10 moving forward (b in the figure). When the first gesture b is performed, the moving body 10 goes from a moving forward state to a stopped state.
[0081] (First gesture c) Fig. 25 is a diagram for explaining the first gesture c. The first gesture c is an action of moving the hand leftward so as to move the moving object 10 leftward. Starting from a state in which the hand is thrust forward as shown in Fig. 23, a (c1 in the figure), the action of pointing the fingertips in the positive Y direction and returning to the starting point is repeated (c2, c3 in the figure). When the first gesture c is performed, the moving object 10 moves leftward.
[0082] (First gesture d) Fig. 26 is a diagram for explaining the first gesture d. The first gesture d is an action of moving the hand to the right so as to move the moving object 10 to the right. Starting from a state in which the hand is thrust forward as shown in Fig. 23, a (d1 in the figure), the fingertips are directed in the negative Y direction and the action of returning to the starting point is repeated (d2, d3 in the figure). When the first gesture d is performed, the moving object 10 moves to the right.
[0083] (First gesture e) 27 is a diagram for explaining the first gesture e. The first gesture e is a motion of beckoning with the fingertips so as to move the moving object 10 backward. The palm is turned in the positive Z direction (e1 in the figure), and the fingertips are repeatedly moved so as to point toward the user (to bring the fingertips closer to the palm) (e2, e3 in the figure). When the first gesture e is performed, the moving object 10 moves backward.
[0084] (First gesture f) FIG. 28 is a diagram for explaining the first gesture f. The first gesture f is an action of extending the index finger and thumb (or a specified finger) and rotating the extended fingers to the left so as to rotate the moving body 10 to the left. The palm is directed to the positive X direction, the index finger and thumb are extended, and the other fingers are lightly clenched (bent) (f1 in the figure), and the hand is rotated so that the palm faces the negative X direction and the back of the hand faces the positive X direction (f2 in the figure). Then, the rotated hand is returned to its original state (f3 in the figure). When the first gesture f is performed, the moving body 10 rotates to the left.
[0085] (First gesture g) FIG. 29 is a diagram for explaining the first gesture g. The first gesture g is an action of extending the index finger and thumb (or a specified finger) and rotating the extended fingers to the right so as to rotate the moving body 10 to the right. The index finger and thumb are extended, the other fingers are lightly clenched (bent), and the index finger is directed in the positive X direction or in a direction intermediate between the positive X direction and the positive Y direction (g1 in the figure). In this state, the index finger is rotated in the positive Z direction or in a direction intermediate between the positive Z direction and the negative Y direction (g2 in the figure). Then, the rotated hand is returned to its original state (g3 in the figure). When the first gesture g is performed, the moving body 10 rotates to the right.
[0086] [flowchart] 30 is a flowchart showing an example of a process in which the control device 50 recognizes a gesture. First, the control device 50 determines whether or not a user is present in the first area (step S300). If a user is present in the first area, the control device 50 recognizes the behavior of the user based on the acquired images (step S302). The behavior is, for example, the movement of the user recognized from images acquired continuously in time.
[0087] Next, the control device 50 refers to the first gesture information 76 to identify a gesture that matches the behavior recognized in step S302 (step S304). If the first gesture information 76 does not include a gesture that matches the behavior recognized in step S302, it is determined that a gesture that controls the movement of the moving object 10 is not performed. Next, the control device 50 performs an action corresponding to the identified gesture (step S306).
[0088] If the user does not exist in the first area (if the user exists in the second area), the control device 50 recognizes the user's behavior based on the acquired image (step S308), and identifies a gesture that matches the behavior recognized in step S308 by referring to the second gesture information 78 (step S310). Next, the control device 50 performs an action corresponding to the identified gesture (step S312). This ends the processing of one routine of this flowchart.
[0089] For example, in the above process, the recognition unit 54 recognizes the gestures of a user being tracked, and does not need to perform processing to recognize the gestures of a person not being tracked. This allows the control device 50 to control the moving object based on the gestures of a user being tracked with reduced processing load.
[0090] As described above, the control device 50 can more accurately recognize the user's gestures by switching the gestures to be recognized based on the area in which the user is present, and can operate the moving object 10 according to the user's intention. As a result, the user's convenience is improved.
[0091] As shown in Fig. 31, the control device 50 may recognize a gesture in the third area AR3 by referring to the first gesture information 76 and the second gesture information 78. In Fig. 31, the third area AR3 is an area between the outer edge of the first area AR1 and a position outside the first area AR1 and a predetermined distance from the outer edge. The second area AR2 is an area outside the third area AR3.
[0092] When the user is in the first area AR1, the recognition unit 54 recognizes the gesture with reference to the first gesture information 76. When the user is in the second area AR2, the recognition unit 54 recognizes the gesture with reference to the first gesture information 76 and the second gesture information 78. That is, the recognition unit 54 determines whether the user is performing the first gesture included in the first gesture information 76 or the second gesture included in the second gesture information 78. When the user is performing the first gesture or the second gesture in the third area AR3, the control device 50 controls the moving object 10 based on the user's operation associated with the first gesture or the second gesture. When the user is in the second area AR2, the recognition unit 54 recognizes the gesture with reference to the second gesture information 78.
[0093] Also, the third region AR3 may be a region between the outer edge of the first region AR1 and a position inside the first region AR1 at a predetermined distance from the outer edge, as shown in Fig. 32. Also, the third region AR3 may be a region defined by a boundary inside the outer edge of the first region AR1 at a predetermined distance from the outer edge, and a boundary outside the outer edge of the first region AR1 at a predetermined distance from the outer edge (the third region AR3 in Fig. 31 and the third region AR3 in Fig. 32 may be combined to form the third region).
[0094] For example, when both the first gesture and the second gesture are recognized in the third area AR3, the first gesture may be adopted in preference to the second gesture. The priority means, for example, that when the movement of the moving body 10 indicated by the first gesture is different from the movement of the moving body 10 indicated by the second gesture, the movement of the first gesture is prioritized, or the second gesture is not considered. When the user moves his / her arm unintentionally, it may be recognized as the second gesture, but a small gesture using the hand or finger is unlikely to be performed unintentionally by the user, and is more likely to be moved with the intention of performing the gesture. In this way, by prioritizing the first gesture, the intention of the user can be recognized more accurately.
[0095] In the above example, the recognition unit 54 has been described as recognizing the user's body movement based on a plurality of images captured in succession (a plurality of images or videos captured at a predetermined interval), but instead (or in addition) the recognition unit 54 may recognize the user's body movement based on one image. In this case, the recognition unit 54 compares, for example, a feature amount indicating the user's body movement included in one image with a feature amount included in the first gesture information 76 or the second gesture information 78, and recognizes that the user is making a gesture with a feature amount that matches highly or is equal to or greater than a predetermined degree.
[0096] Also, in the above example, when the recognition unit 54 recognizes the user's body movements using an image captured by a camera (imaging device) installed at a position different from the moving body 10, the first area is an area within a predetermined distance from the imaging device that captures the image, and the second area is an area set at a position farther than the predetermined distance from the imaging device.
[0097] In the above example, the second region is described as being located farther away than the first region, but instead, the second region may be set at a different position from the first region and the second region. For example, the first region may be set in a first direction, and the second region may be set in a direction different from the first direction.
[0098] According to the first embodiment described above, the control device 50 switches the gesture to be recognized depending on the position of the user relative to the moving body, thereby enabling the control device 50 to recognize the gesture of the user with higher accuracy and appropriately operate the moving body 10. As a result, the user's convenience is improved.
[0099] <Second embodiment> The second embodiment will be described below. The main body 20 of the moving object 10 in the second embodiment includes a first camera (first imaging unit) and a second camera (second imaging unit), and recognizes gestures using images captured by these cameras. The following description will focus on differences from the first embodiment.
[0100] FIG. 33 is a diagram for explaining an example of a functional configuration of the main body 20A of the moving body 10 of the second embodiment. The main body 20A includes a first camera 21 and a second camera 23 instead of the camera 22. The first camera 21 is a camera similar to the camera 22. The second camera 23 is a camera that captures an image of a user who remotely operates the moving body 10. The second camera 23 is a camera that captures an image for recognizing a gesture of the user. The remote operation is performed by a gesture. The imaging direction of the second camera 23 can be controlled by, for example, a mechanical mechanism. The second camera 23 captures an image centered on the user to be tracked. The information processing unit 60 controls the mechanical mechanism, for example, so that the imaging direction of the second camera 23 is directed toward the user to be tracked.
[0101] The recognition unit 54 attempts to perform a process of recognizing a gesture of the user based on the first image captured by the first camera 21 and the second image captured by the second camera 23. The recognition unit 54 prioritizes the result of recognition based on the second image (second recognition result) over the result of recognition based on the first image (first recognition result). The trajectory generation unit 56 generates a trajectory based on the surrounding situation obtained from the first image and the movement associated with the recognized gesture. The traveling control unit 58 controls the mobile object 10 based on the trajectory generated by the trajectory generation unit 56.
[0102] [flowchart] FIG. 34 is a flowchart showing an example of the flow of processing executed by the control device 50 of the second embodiment. First, the acquisition unit 52 of the control device 50 acquires a first image and a second image (step S400). Next, the recognition unit 54 attempts processing to recognize a gesture in each of the first image and the second image, and determines whether or not the gesture has been recognized from both images (step S402). In this processing, if the user is present in the first area, the first gesture information 76 is referenced, and if the user is present outside the first area, the second gesture information 78 is referenced.
[0103] If the gesture can be recognized from both images, the recognition unit 54 determines whether the recognized gestures are the same (step S404). If the recognized gestures are the same, the recognition unit 54 adopts the recognized gesture (step S406). If the recognized gestures are not the same, the recognition unit 54 adopts the gesture recognized from the second image (step S408). This gives the second recognition result priority over the first recognition result.
[0104] If the recognition unit 54 is unable to recognize a gesture from both images in the process of step S402, the recognition unit 54 adopts the recognized gesture (the gesture recognized from the first image or the gesture recognized from the second image) (step S406). For example, if the user is present in the first area and the recognition unit 54 is unable to recognize the user's gesture based on the first image captured by the first camera 21, the recognition unit 54 refers to the first gesture information 76 and recognizes the user's gesture based on the second image captured by the second camera 23. Then, the moving object 10 is controlled to perform an action according to the adopted gesture. This ends the process of one routine of this flowchart.
[0105] Through the above-described processing, the control device 50 can recognize the user's gesture with higher accuracy.
[0106] In the second embodiment, the first gesture information 76 or the second gesture information 78 may be referenced regardless of the user's position, or gesture information different from the first gesture information 76 or the second gesture information 78 (for example, information that does not take into account the user's position) (information that associates gesture features with the behavior of the moving body 10) may be referenced.
[0107] According to the second embodiment described above, the control device 50 recognizes gestures using images captured by two or more cameras, thereby enabling the control device 50 to recognize gestures more accurately and control the moving object 10 based on the recognition result. As a result, the user's convenience can be improved.
[0108] [Variations of the second gesture] The second gesture may be in the following form instead of the above-mentioned second gesture. For example, the second gesture may be a gesture of the upper arm without considering the movement of the palm. This allows the control device 50 to recognize the second gesture with high accuracy even if it is performed at a long distance. Examples are given below, but other forms may be used.
[0109] (Second gesture G) 35 is a diagram for explaining a modified example of the second gesture G. The second gesture G is an action (G# in the figure) of bending the elbow, turning the palm upward, and rotating the upper arm to the left so as to rotate the moving object 10 to the left. When the second gesture G is performed, the moving object 10 rotates to the left.
[0110] (Second gesture H) 36 is a diagram for explaining a modified example of the second gesture H. The second gesture H is an action (H# in the figure) of bending the elbow, turning the palm upward, and rotating the upper arm to the right so as to rotate the moving object 10 to the right. When the second gesture H is performed, the moving object 10 rotates to the right.
[0111] (Second gesture F) 37 is a diagram for explaining a modified example of the second gesture F. The second gesture F is an action of bending the elbow and turning the palm up so as to move the moving object 10 backward (F# in the figure). When the second gesture F is performed, the moving object 10 moves backward.
[0112] (Second gesture FR) 38 is a diagram for explaining the second gesture FR. The second gesture FR is an action (FR in the figure) in which the elbow is bent, the palm is turned up, and the amount of movement of the moving body 10 to the right is determined by the degree of tilt of the upper arm to the right, so that the moving body 10 moves back while moving to the right. When the second gesture FR is performed, the moving body 10 moves back while moving to the right according to the degree of tilt of the upper arm to the right.
[0113] 39 is a diagram for explaining the second gesture FL. The second gesture FL is an action (FR in the figure) in which the elbow is bent, the palm is turned up, and the amount of leftward movement of the moving body 10 is determined by the degree of leftward inclination of the upper arm so as to move the moving body 10 back while moving leftward, so that the moving body 10 moves back while moving leftward according to the degree of leftward inclination of the upper arm. When the second gesture FL is performed, the moving body 10 moves back while moving leftward according to the degree of leftward inclination of the upper arm.
[0114] As described above, the control device 50 controls the moving body 10 based on the second gesture made by the upper arm. For example, even if a person far away makes the second gesture, the control device 50 can recognize the second gesture with higher accuracy and control the moving body 10 according to the person's intention.
[0115] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, A user acquires a captured image, Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; and when the user is present in a second area when the image is captured, recognizing a gesture of the user based on a plurality of the images captured successively in time and second information for recognizing a gesture of the user. Gesture recognizer.
[0116] The above-described embodiment can be expressed as follows. A first imaging unit that captures an image of the periphery of the moving object; A second imaging unit that captures an image of a user remotely operating the moving object; A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, performing a process of recognizing a gesture of the user based on a first image captured by the first imaging unit and a second image captured by the second imaging unit, and adopting a result of the recognition based on the second image with priority over a result of the recognition based on the first image; controlling the moving object based on a surrounding situation obtained from an image captured by the first imaging unit and an action associated with the gesture recognized by the recognition unit; Gesture recognizer.
[0117] The above-described embodiment can be expressed as follows. A first imaging unit that captures an image of the periphery of the moving object; A second imaging unit that captures an image of a user remotely operating the moving object; A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, when the user is present in a first area and a gesture of the user cannot be recognized based on a first image captured by the first imaging unit, recognizing a gesture of the user based on a second image captured by the second imaging unit with reference to the first information; controlling the moving object based on an image captured by the first imaging unit in response to the recognized gesture; Gesture recognizer.
[0118] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]
[0119] REFERENCE SIGNS LIST 10 mobile object, 20 main body, 21 first camera, 22 camera, 23 second camera, 50 control device, 52 acquisition unit, 54 recognition unit, 56 trajectory generation unit, 58 driving control unit, 60 information processing unit, 70 storage unit, 74 gesture information, 76 first gesture information, 78 second gesture information, 80 user information
Claims
1. An acquisition unit that acquires an image captured by a user; Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; a recognition unit that recognizes a gesture of the user based on the image and second information for recognizing a gesture of the user when the user is present in a second area when the image is captured, The recognition unit is a process of recognizing a gesture of the user is attempted based on a first image captured by a first imaging unit that captures an image of the periphery of a moving object controlled based on a recognition result of the gesture and a second image captured by a second imaging unit that captures an image of a user remotely operating the moving object, and the gesture is recognized by giving priority to and adopting a result of recognition based on the second image over a result of recognition based on the first image; Processing system.
2. An acquisition unit that acquires an image captured by a user; Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; a recognition unit that recognizes a gesture of the user based on the image and second information for recognizing a gesture of the user when the user is present in a second area when the image is captured, The recognition unit is when the user is present in a first area and a gesture of the user cannot be recognized based on a first image captured by a first imaging unit that captures an image of the periphery of a moving object controlled based on a recognition result of the gesture, the gesture of the user is recognized based on a second image captured by a second imaging unit that captures an image of the user remotely operating the moving object by referring to the first information; Processing system.
3. An acquisition unit that acquires an image captured by a user; Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; a recognition unit that recognizes a gesture of the user based on the image and second information for recognizing a gesture of the user when the user is present in a second area when the image is captured; A first imaging unit that captures an image of the periphery of the moving object; A second imaging unit that captures an image of a user remotely operating the moving object; a storage device storing reference information in which a gesture of the user is associated with a motion of the mobile object; a control unit that refers to the reference information and controls the moving object based on a motion of the moving object associated with the gesture of the user recognized by the recognition unit, the recognition unit attempts a process of recognizing a gesture of the user based on a first image captured by the first imaging unit and a second image captured by the second imaging unit, and adopts a result of the recognition based on the second image in preference to a result of the recognition based on the first image; The control unit controls the moving object based on a surrounding situation obtained from an image captured by the first imaging unit and an action associated with the gesture recognized by the recognition unit. Processing system.
4. An acquisition unit that acquires an image captured by a user; Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; a recognition unit that recognizes a gesture of the user based on the image and second information for recognizing a gesture of the user when the user is present in a second area when the image is captured; A first imaging unit that captures an image of the periphery of the moving object; A second imaging unit that captures an image of a user remotely operating the moving object; a storage device storing reference information in which a gesture of the user is associated with a motion of the mobile object; a control unit that refers to the reference information and controls the moving object based on a motion of the moving object associated with the gesture of the user recognized by the recognition unit, the recognition unit, when the user is present in a first area and a gesture of the user cannot be recognized based on a first image captured by the first imaging unit, recognizes a gesture of the user based on a second image captured by the second imaging unit with reference to the first information; a control unit that controls the moving object based on an image captured by the first imaging unit in response to the gesture recognized by the recognition unit, Processing system.
5. the first area is an area within a predetermined distance from an imaging device that captures the image, The second area is an area set at a position farther than the predetermined distance from the imaging device. A processing system according to any one of claims 1 to 4.
6. The first information is information for recognizing a gesture by a movement of a hand or a finger, not including a movement of an arm. A processing system according to any one of claims 1 to 5.
7. The second information is information for recognizing a gesture including an arm movement. A processing system according to any one of claims 1 to 6.
8. The first region is a region in which the recognition unit cannot recognize or has difficulty in recognizing the arm movement of a user present in the first region from an image captured by the recognition unit. The processing system of claim 7 .
9. the recognition unit recognizes a gesture of the user based on the image, the first information, and the second information when the user is present in a third area spanning the first area and a second area adjacent to the first area outside the first area, or a third area between the first area and the second area farther than the first area, when the image is captured. A processing system according to any one of claims 1 to 8.
10. When the recognition unit recognizes a gesture of the user based on the image, the first information, and the second information, the recognition unit recognizes the gesture of the user by giving a result of the recognition based on the image and the first information a higher priority than a result of the recognition based on the image and the second information. The processing system of claim 9 .
11. The computer A user acquires a captured image, Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; When the user is present in a second area when the image is captured, recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user; a process of recognizing a gesture of the user is attempted based on a first image captured by a first imaging unit that captures an image of the periphery of a moving object controlled based on a recognition result of the gesture and a second image captured by a second imaging unit that captures an image of a user remotely operating the moving object, and the gesture is recognized by giving priority to and adopting a result of recognition based on the second image over a result of recognition based on the first image; Processing methods.
12. The computer A user acquires a captured image, Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; When the user is present in a second area when the image is captured, recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user; when the user is present in a first area and a gesture of the user cannot be recognized based on a first image captured by a first imaging unit that captures an image of the periphery of a moving object controlled based on a recognition result of the gesture, the gesture of the user is recognized based on a second image captured by a second imaging unit that captures an image of the user remotely operating the moving object by referring to the first information; Processing methods.
13. The computer A user acquires a captured image, Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; When the user is present in a second area when the image is captured, recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user; referring to reference information in which the gesture of the user and the motion of the moving object are associated with each other and stored in a storage device, and controlling the moving object based on the motion of the moving object associated with the recognized gesture of the user; performing a process of recognizing a gesture of the user based on a first image captured by a first imaging unit that captures an image of the periphery of the moving object and a second image captured by a second imaging unit that captures an image of a user remotely operating the moving object, and adopting a result of the recognition based on the second image in preference to a result of the recognition based on the first image; controlling the moving object based on a surrounding situation obtained from an image captured by the first imaging unit and an action associated with the recognized gesture; Processing methods.
14. The computer A user acquires a captured image, Recognizing an area in which the user exists when the image is captured; When the user is present in a first area when the image is captured, recognizing a gesture of the user based on the image and first information for recognizing a gesture of the user; When the user is present in a second area when the image is captured, recognizing a gesture of the user based on the image and second information for recognizing a gesture of the user; referring to reference information in which the gesture of the user and the motion of the moving object are associated with each other and stored in a storage device, and controlling the moving object based on the motion of the moving object associated with the recognized gesture of the user; when the user is present in a first area and a gesture of the user cannot be recognized based on a first image captured by a first imaging unit that captures an image of the periphery of a moving object, a gesture of the user is recognized based on a second image captured by a second imaging unit that captures an image of the user remotely operating the moving object by referring to the first information; controlling the moving object based on an image captured by the first imaging unit in response to the recognized gesture; Processing methods.
Citation Information
Patent Citations
Sending system of charging pulse
JP1981017562A
Protection apparatus and protection system
JP2015048025A
Vertically taking off and flying table
JP2016101774A
Method and apparatus for controlling a device having an image acquisition unit and a distance measurement unit
JP2019522853A
Interaction mode selection based on detected distance between user and machine interface
JP2019533846A
