Processing device, mobile object, processing method, and program

Abstract

To provide a processing device, a mobile object, a processing method, and a program that accurately identify a target while reducing a processing load.SOLUTION: In a main body 20 of an autonomous mobile type robot, a controller 50 has a recognition unit 54. The recognition unit 54 includes: a first processing unit configured to convert a first image into a second image having a resolution lower than that of the first image; and a second processing unit configured to identify, based on the second image, a target area including a predetermined object target in the second image and identify, based on the identified target area in the second image, a target area including the object target in the first image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a processing device, a mobile object, a processing method, and a program. [Background technology]

[0002] Conventionally, an information processing device has been disclosed that analyzes images captured by two cameras to track an object (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2018-88234 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the above-mentioned technology, the configuration of the device used may be complicated, or the processing load may be high.

[0005] The present invention has been made in consideration of the above circumstances, and one of its objectives is to provide a processing device, a moving body, a processing method, and a program that can accurately identify targets while reducing the processing load. [Means for solving the problem]

[0006] The processing device, the moving object, the processing method, and the program according to the present invention employ the following configuration. (1): A processing device according to one embodiment of the present invention includes a first processing unit that converts a first image into a second image having a lower resolution than the first image, and a second processing unit that identifies a target area including a predetermined target object in the second image based on the second image, and identifies a target area including the target object in the first image based on the identified target area in the second image.

[0007] (2): In the above aspect (1), the second processing unit analyzes the second image converted from the first image captured at a first time and the second image converted from the first image captured at a second time later than the first time, and tracks a target object included in the target area of ​​the second image corresponding to the first time in the second image corresponding to the second time.

[0008] (3): In the above aspect (1) or (2), the second processing unit tracks the target object in a time series of second images into which each of the first images captured in time series is converted based on a change in position of the target object in the second images.

[0009] (4): In any of the above aspects (1) to (3), the target object is a person, and the second processing unit tracks the person in the second image based on one or more pieces of information on the color of the person's clothes, the gender of the person, or the hairstyle of the person, which are obtained from the second image converted from the first image captured in the past or stored in a memory unit.

[0010] (5): In any of the above aspects (1) to (4), the second processing unit identifies a target area including the target object in the first image before the second image is converted based on the target area in the second image, and performs a predetermined image recognition process on the identified target area in the first image.

[0011] (6): In the above aspect (5), the predetermined image recognition process includes a process of recognizing a gesture of the target object based on information of a specific area of ​​the target area in the first image.

[0012] (7): In the aspect (6) above, the predetermined image recognition process includes identifying the specific region based on recognizing a skeleton or joint points with respect to the target region in the first image.

[0013] (8): In the aspect (7) above, the predetermined image recognition process includes a process of determining an area including an arm or hand of the target object as the specific area based on the recognition result of the skeleton or joint points.

[0014] (9): In any of the above aspects (6) to (8), the predetermined image recognition process includes a process of recognizing a gesture by preferentially using information from a specific area, among two or more specific areas of the target area in the first image, in which the degree of change in the movement of the target object over time is large.

[0015] (10): In the above aspect (9), the two or more specific areas include at least a specific area identified as an area including the right arm or right hand of the target object and a specific area identified as an area including the left arm or left hand of the target object.

[0016] (11): In any of the above aspects (1) to (10), the target object is a person, and the second processing unit extracts an area including the person's face in the first image, and tracks the target object by comparing facial features included in the extracted area with facial features of the person previously stored in a memory unit.

[0017] (12): In any of the above aspects (1) to (10), the target object is a person, and the second processing unit tracks the person included in the target area of ​​the second image converted from the first image captured at a first time in the second image converted from the first image captured at a second time that is later than the first time, extracts an area including an arm or hand of the tracked person in the first image captured at the second time based on a tracking result, and recognizes a gesture of the person based on the extracted area.

[0018] (13) In any one of the above aspects (6), (7), and (12), the device further includes a control unit that controls a moving object based on the gesture recognized by the second processing unit.

[0019] (14): A mobile object equipped with a processing device according to any one of (1) to (13) above.

[0020] (15): A processing method according to one embodiment of the present invention includes a computer converting a first image into a second image having a lower resolution than the first image, identifying a target area in the second image based on the second image that includes a predetermined target object, and identifying a target area in the first image that includes the target object based on the identified target area in the second image.

[0021] (16): A program according to one embodiment of the present invention causes a computer to convert a first image into a second image having a lower resolution than the first image, identify a target area in the second image based on the second image, the target area including a predetermined target object, and identify a target area in the first image including the target object based on the identified target area in the second image. Effect of the Invention

[0022] According to (1)-(16), the processing device converts a first image into a second image having a lower resolution than the first image, and obtains a target area including the target in the first image based on the target area including the target in the second image, thereby enabling accurate identification of targets while reducing the processing load.

[0023] According to (4), the processing device can track a person more accurately by using the person's characteristics.

[0024] According to (6), the processing device can more accurately recognize gestures of objects that are the subject of the action, such as people, robots, and animals.

[0025] According to (9) or (10), the processing device can recognize gestures with higher accuracy by preferentially using information from a specific region that exhibits a large degree of change over time.

[0026] According to (11), the processing device can track a person with higher accuracy by using the facial features of the person.

[0027] According to (13), the processing device can control the moving object as intended by the person. [Brief description of the drawings]

[0028] [Figure 1] FIG. 1 is a diagram illustrating an example of a moving object 10 including a control device according to an embodiment. [Diagram 2] 2 is a diagram showing an example of a functional configuration included in a main body 20 of a moving object 10. FIG. [Diagram 3] 13 is a flowchart showing an example of the flow of a tracking process. [Figure 4] 11 is a diagram for explaining a process for extracting a feature amount of a user and a process for registering the feature amount. FIG. [Diagram 5] 11 is a diagram for explaining a process in which a recognition unit 54 tracks a user. FIG. [Figure 6] FIG. 11 is a diagram for explaining a tracking process using a feature amount. [Figure 7] FIG. 11 is a diagram for explaining a tracking process using a feature amount. [Figure 8] FIG. 13 is a diagram for explaining another example of the process in which the second processing unit 55B tracks a user. [Figure 9] FIG. 13 is a diagram showing an example of a process for extracting an area including a person present in the vicinity from a high-resolution image. [Figure 10] 13 is a flowchart showing an example of the flow of a behavior control process. [Figure 11] FIG. 1 is a diagram (part 1) for explaining a process of recognizing a gesture. [Figure 12] FIG. 2 is a diagram (part 2) for explaining the gesture recognition process. [Figure 13] FIG. 11 is a diagram (part 3) for explaining the gesture recognition process. [Figure 14]FIG. 11 is a diagram for explaining a process of recognizing a gesture. [Figure 15] 13 is a diagram showing an example of a first gesture included in first gesture information 76. FIG. [Figure 16] FIG. 11 is a diagram for explaining a process of recognizing a gesture. [Figure 17] 13 is a diagram showing an example of a second gesture included in second gesture information 78. FIG. [Figure 18] 10 is a flowchart showing an example of a process for the control device 50 to recognize a gesture. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0029] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS A processing device, a mobile object, a processing method, and a program according to an embodiment of the present invention will be described below with reference to the drawings.

[0030] First Embodiment [Overall configuration] 1 is a diagram showing an example of a mobile body 10 equipped with a processing device according to an embodiment. The mobile body 10 is an autonomous mobile robot. The mobile body 10 supports the actions of a user. For example, the mobile body 10 supports shopping or customer service of a customer according to instructions from a store clerk, a customer, a facility staff member (hereinafter, these persons are referred to as "users"), etc., and supports the work of the staff.

[0031] The moving object 10 includes a main body 20, a container 92, and one or more wheels 94 (wheels 94A and 94B in the figure). The moving object 10 moves in response to instructions based on a user's gestures or voice, operations on an input unit (a touch panel to be described later) of the moving object 10, and operations on a terminal device (e.g., a smartphone). The moving object 10 recognizes gestures based on images captured by a camera 22 provided on the main body 20, for example.

[0032] For example, the mobile unit 10 drives the wheels 94 to move so as to follow the customer in accordance with the movement of the user, or to move so as to lead the customer. At this time, the mobile unit 10 explains the product or work to the user, or guides the user to the product or object that the user is looking for. In addition, the user can store the product or baggage that he or she plans to purchase in the container 92 that stores them.

[0033] In this embodiment, the moving body 10 is described as having a container 92, but instead of (or in addition to) this, the moving body 10 may be provided with a seating section on which the user sits, a housing on which the user climbs, a step on which the user places their feet, etc., so that the user can move together with the moving body 10.

[0034] 2 is a diagram showing an example of a functional configuration included in the main body 20 of the moving object 10. The main body 20 includes a camera 22, a communication unit 24, a position identification unit 26, a speaker 28, a microphone 30, a touch panel 32, a motor 34, and a control device 50 (an example of a "processing device").

[0035] Camera 22 captures images of the periphery of moving body 10. Camera 22 is, for example, a fisheye camera capable of capturing images of the periphery of moving body 10 at a wide angle (e.g., 360 degrees). Camera 22 is attached, for example, to the upper part of moving body 10, and captures images of the periphery of moving body 10 at a wide angle in the horizontal direction. Camera 22 may be realized by combining multiple cameras (multiple cameras capturing images in a horizontal range of 120 degrees or 60 degrees). The number of cameras 22 is not limited to one, and multiple cameras may be provided on moving body 10.

[0036] The communication unit 24 is a communication interface for communicating with other devices using a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like.

[0037] The position identification unit 26 identifies the position of the mobile body 10. The position identification unit 26 acquires position information of the mobile body 10 by a GPS (Global Positioning System) device (not shown) built into the mobile body 10. The position information may be, for example, two-dimensional map coordinates or latitude and longitude information.

[0038] The speaker 28 outputs, for example, a predetermined sound. The microphone 30 receives, for example, an input of a sound uttered by the user.

[0039] The touch panel 32 is configured by superimposing a display unit such as an LCD (liquid crystal display) or an organic EL (electroluminescence) on an input unit capable of detecting a touch position of an operator by a coordinate detection mechanism. The display unit displays GUI (graphical user interface) switches for operation. When the input unit detects a touch operation, a flick operation, a swipe operation, or the like on the GUI switch, it generates an operation signal indicating that a touch operation on the GUI switch has been performed, and outputs the operation signal to the control device 50. In response to the operation, the control device 50 causes the speaker 28 to output a sound or causes the touch panel 32 to display an image. The control device 50 may also cause the moving object 10 to move in response to the operation.

[0040] The motor 34 drives the wheels 94 to move the mobile body 10. The wheels 94 include, for example, drive wheels that are driven in a rotational direction by the motor 34, and steering wheels that are non-driven wheels that are driven in a yaw direction. By adjusting the angle of the steering wheels, the mobile body 10 can change its course or rotate on its own axis.

[0041] In this embodiment, the moving body 10 has wheels 94 as a mechanism for realizing movement, but this embodiment is not limited to this configuration. For example, the moving body 10 may be a multi-legged robot.

[0042] The control device 50 includes, for example, an acquisition unit 52, a recognition unit 54, a trajectory generation unit 56, a travel control unit 58, an information processing unit 60, and a storage unit 70. Some or all of the acquisition unit 52, the recognition unit 54, the trajectory generation unit 56, the travel control unit 58, and the information processing unit 60 are realized by, for example, a hardware processor such as a CPU (Central Processing Unit) executing a program (software). Some or all of these functional units may be realized by hardware (including circuitry) such as an LSI (Large Scale Integration), an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a GPU (Graphics Processing Unit), or may be realized by cooperation between software and hardware. The program may be stored in advance in a storage unit 70 (a storage device having a non-transient storage medium) such as a hard disk drive (HDD) or a flash memory, or may be stored in a removable storage medium (a non-transient storage medium) such as a DVD or a CD-ROM, and may be installed by mounting the storage medium in a drive device. The acquisition unit 52, the recognition unit 54, the trajectory generation unit 56, the traveling control unit 58, or the information processing unit 60 may be provided in a device different from the control device 50 (the moving body 10). For example, the recognition unit 54 may be provided in another device, and the control device 50 may control the moving body 10 based on the processing result of the other device. In addition, a part or all of the information stored in the storage unit 70 may also be stored in the other device. A configuration including one or more functional units among the acquisition unit 52, the recognition unit 54, the trajectory generation unit 56, the traveling control unit 58, or the information processing unit 60 may be configured as a system.

[0043] The storage unit 70 stores map information 72, gesture information 74, and user information 80. The map information 72 is information that expresses the shapes of roads and passages in a facility using, for example, links indicating the roads and passages in the facility and nodes connected by the links. The map information 72 may include information such as the curvature of the road and POI (Point Of Interest) information.

[0044] The gesture information 74 is information in which information about a gesture (feature amount of a template) and a motion of the moving object 10 are associated with each other. The gesture information 74 includes first gesture information 76 and second gesture information 78. The user information 80 is information indicating the feature amount of a user. The gesture information 74 and the user information 80 will be described in detail later.

[0045] The acquisition unit 52 acquires an image (hereinafter referred to as a "peripheral image") captured by the camera 22. The acquisition unit 52 holds the acquired peripheral image as pixel data in the fish-eye camera coordinate system, for example.

[0046] The recognition unit 54 includes, for example, a first processing unit 55A and a second processing unit 55B. The first processing unit 55A converts a first image (for example, a high-resolution image) captured by the camera 22 into a second image (low-resolution image) having a lower resolution than the first image. The second processing unit 55B specifies a target area including a predetermined target in the second image based on the second image, and specifies a target area including the target in the first image based on the specified target area in the second image. The target is, for example, a target to be tracked. The processing of the first processing unit 55A and the second processing unit 55B will be described in detail later.

[0047] Further, the second processing unit 55B recognizes a body movement (hereinafter referred to as a "gesture") by the user based on one or more peripheral images. The recognition unit 54 recognizes the gesture by matching the feature amount of the user's gesture extracted from the peripheral image with the feature amount of the template (feature amount indicating the gesture). The feature amount is data representing, for example, characteristic parts such as a person's fingers, finger joints, wrists, arms, and bones, links connecting them, and the inclination, position, etc. of the link.

[0048] The trajectory generating unit 56 generates a trajectory along which the moving body 10 should travel in the future based on the user's gestures, a destination set by the user, surrounding objects, the user's position, map information 72, and the like. The trajectory generating unit 56 generates a trajectory that allows the moving body 10 to move smoothly to the destination point by combining a plurality of arcs. The trajectory generating unit 56 generates a trajectory by combining, for example, three arcs. The trajectory generating unit 56 may generate a trajectory by fitting a state to a geometric model such as a Bezier curve. The trajectory is actually generated as a collection of a finite number of trajectory points, for example.

[0049] The trajectory generating unit 56 performs coordinate conversion between the Cartesian coordinate system and the fisheye camera coordinate system. A one-to-one relationship is established between the coordinates of the Cartesian coordinate system and the fisheye camera coordinate system, and the relationship is stored in the storage unit 70 as correspondence information. The trajectory generating unit 56 generates a trajectory in the Cartesian coordinate system (Cartesian coordinate system trajectory) and converts the trajectory into a trajectory in the fisheye camera coordinate system (fisheye camera coordinate system trajectory). The trajectory generating unit 56 calculates the risk of the fisheye camera coordinate system trajectory. The risk is an index value indicating the likelihood that the moving body 10 will approach an obstacle. The risk tends to be higher the smaller the distance between the trajectory (trajectory point of the trajectory) and the obstacle, and lower the risk tends to be higher the larger the distance between the trajectory and the obstacle.

[0050] If the total risk value or the risk of each trajectory point meets a predetermined criterion (for example, if the total risk value is less than or equal to threshold value Th1 and the risk of each trajectory point is less than or equal to threshold value Th2), the trajectory generation unit 56 adopts the trajectory that meets the criterion as the trajectory along which the moving body will move.

[0051] If the above trajectory does not satisfy the preset criteria, the following process may be performed. The trajectory generating unit 56 detects a drivable space in the fisheye camera coordinate system, and converts the detected drivable space in the fisheye camera coordinate system into a drivable space in the Cartesian coordinate system. The drivable space is a space in the moving direction of the moving body 10 excluding obstacles and the area around the obstacles (areas where a risk is set or areas where the risk is equal to or greater than a threshold). The trajectory generating unit 56 corrects the trajectory so that the trajectory falls within the drivable space that has been coordinate-converted into the Cartesian coordinate system. The trajectory generating unit 56 converts the Cartesian coordinate system trajectory into a fisheye camera coordinate system trajectory, and calculates the risk of the fisheye camera coordinate system trajectory based on the surrounding image and the fisheye camera coordinate system trajectory. This process is repeated to search for a trajectory that satisfies the above preset criteria.

[0052] The travel control unit 58 causes the moving body 10 to travel along a trajectory that satisfies a preset standard. The travel control unit 58 outputs a command value for causing the moving body 10 to travel along the trajectory to the motor 34. The motor 34 rotates the wheels 94 in accordance with the command value, and moves the moving body 10 along the trajectory.

[0053] The information processing unit 60 controls various devices and equipment included in the main body 20. The information processing unit 60 controls, for example, the speaker 28, the microphone 30, and the touch panel 32. The information processing unit 60 also recognizes voice input to the microphone 30 and operations performed on the touch panel 32. The information processing unit 60 operates the moving object 10 based on the result of the recognition.

[0054] In the above example, the recognition unit 54 has been described as using the image captured by the camera 22 provided on the moving body 10 for various processes, but the recognition unit 54 may perform various processes using an image captured by a camera not provided on the moving body 10 (a camera provided at a position different from the moving body 10). In this case, the image captured by the camera is transmitted to the control device 50 via communication, and the control device 50 acquires the transmitted image and performs various processes based on the acquired image. The recognition unit 54 may also perform various processes using multiple images. For example, the recognition unit 54 may perform various processes based on an image captured by the camera 22 or multiple images captured by a camera provided at a position different from the moving body 10.

[0055] [Support process] The mobile object 10 executes a support process for supporting the user in shopping. The support process includes a process related to tracking and a process related to behavior control.

[0056] [Tracking process (part 1)] FIG. 3 is a flowchart showing an example of the flow of the tracking process. First, the control device 50 of the moving object 10 accepts user registration (step S100). Next, the control device 50 tracks the user registered in step S100 (step S102). Next, the control device 50 judges whether or not tracking is successful (step S104). If tracking is successful, the process proceeds to step S200 in FIG. 10, which will be described later. If tracking is not successful, the control device 50 identifies the user (step S106).

[0057] (User registration process) The process of registering a user in step S100 will now be described. The control device 50 of the mobile object 10 confirms the user's intention to register based on a specific gesture, voice, or operation on the touch panel 32 of the user (e.g., a customer visiting a store). When the user's intention to register is confirmed, the recognition unit 54 of the control device 50 extracts the user's feature amount and registers the extracted feature amount.

[0058] FIG. 4 is a diagram for explaining a process for extracting a feature amount of a user and a process for registering the feature amount. The second processing unit 55B of the control device 50 identifies a user from an image IM1 captured by the user, and recognizes the joint points and skeleton of the identified user (performs skeleton processing). For example, the second processing unit 55B estimates the user's face, facial parts, neck, shoulders, elbows, wrists, waist, ankles, etc. from the image IM1, and performs skeleton processing based on the positions of the estimated parts. For example, the second processing unit 55B performs skeleton processing using a known method (e.g., a method such as open pose) for estimating the user's joint points and skeleton using deep learning. Next, the second processing unit 55B identifies the user's face, upper body, lower body, etc. based on the result of the skeleton processing, extracts feature amounts for each of the identified face, upper body, and lower body, and registers the extracted feature amounts in the storage unit 70 as the user's feature amounts. The facial feature amounts are, for example, male, female, hairstyle, and facial feature amounts. The feature amount indicating male / female is a feature amount indicating the shape of the head, etc., and the hairstyle is information indicating the length of the hair (short hair, long hair, etc.) obtained from the shape of the head. The feature amount of the upper body is, for example, the color of the upper body. The feature amount of the lower body is, for example, the color of the lower body.

[0059] (User tracking process) The first processing unit 55A converts each of the high-resolution images captured per unit time into a low-resolution image. For example, a high resolution means that the number of pixels per unit area in the image is greater than the number of pixels per unit area in the low-resolution image (high dpi). The first processing unit 55A converts the high-resolution image into a low-resolution image by thinning out the pixels in the high-resolution image IM, or converts the high-resolution image into a low-resolution image by applying a predetermined algorithm.

[0060] The second processing unit 55B analyzes the low-resolution image obtained by converting the high-resolution image captured at the first time and the low-resolution image obtained by converting the high-resolution image captured at the second time after the first time, and tracks the target included in the target area including the target to be tracked in the low-resolution image corresponding to the first time in the low-resolution image corresponding to the second time. The second processing unit 55B tracks the target in the low-resolution image based on the change in the position of the target in the time-series low-resolution images into which each of the high-resolution images captured in time series is converted. The low-resolution image used for this tracking is, for example, a low-resolution image obtained by converting the high-resolution image captured most recently. A specific description will be given below.

[0061] The process of tracking the user in step S102 will now be described. Fig. 5 is a diagram for explaining the process of tracking the user by the recognition unit 54 (the process of step S102 in Fig. 3). The first processing unit 55A of the recognition unit 54 acquires an image captured at time T. This image is an image captured by the camera 22 (hereinafter, high-resolution image IM2).

[0062] The first processing unit 55A of the recognition unit 54 converts the high-resolution image IM2 into a low-resolution image IM2# having a lower resolution than the high-resolution image IM2. Next, the second processing unit 55B detects a person and a person detection area including the person from the low-resolution image IM2#.

[0063] The second processing unit 55B estimates the position of the user at time T (person detection area) based on the position of the person detected at time T-1 (before time T) (person detection area of ​​the tracked user at time T-1) and the moving direction of the person. If the user detected in the low-resolution image IM2 obtained at time T is near a position estimated from the position or moving direction of the tracked user before time T-1, the second processing unit 55B identifies the user detected at time T as the user to be tracked (tracking target). If the user can be identified, tracking is deemed to have been successful.

[0064] As described above, the control device 50 tracks the user using the low-resolution image IM2#, and therefore the processing load is reduced.

[0065] In the tracking process, the second processing unit 55B may track the user using the feature amount of the user in addition to the user's position at time T and time T-1 as described above. FIG. 6 is a diagram for explaining the tracking process using the feature amount. For example, the second processing unit 55B estimates the user's position at time T, identifies a user who exists near the estimated position, and further extracts the feature amount of the user. When the extracted feature amount matches the registered feature amount by a threshold value or more, the control device 50 estimates that the identified user is a user to be tracked, and determines that tracking has been successful.

[0066] For example, when extracting the feature amount of a user, the second processing unit 55B extracts an area including a person, and performs skeleton processing on the image (high resolution image) of the extracted area to extract the feature amount of the person. This reduces the processing load.

[0067] The second processing unit 55B may estimate that the identified user is a user to be tracked when the feature amount obtained from the low-resolution image, instead of the feature amount obtained from the high-resolution image, matches the registered feature amount by a threshold value or more. In this case, the storage unit 70 stores a feature amount to be compared with the feature amount obtained from the low-resolution image in advance, and this feature amount is used. The second processing unit 55B may also identify the user by comparing, for example, a feature amount extracted from an image obtained during tracking with a feature amount obtained from an image captured this time, instead of (or in addition to) the registered feature amount.

[0068] For example, even if the user to be tracked overlaps or intersects with another person, the user can be tracked more accurately based on the change in the user's position and the user's features as described above.

[0069] (Process to identify users) The process of identifying the user in step S106 will be described. When the second processing unit 55B does not succeed in tracking the user, as shown in FIG. 7, it compares the feature amount of the person in the vicinity with the feature amount of the registered user to identify the user to be tracked. For example, the second processing unit 55B extracts the feature amount of each person included in the image. The second processing unit 55B compares the feature amount of each person with the feature amount of the registered user to identify the person who matches the feature amount of the registered user by a threshold value or more. The second processing unit 55B determines the identified user as the user to be tracked. At this time, the feature amount used may be the feature amount obtained from a low-resolution image or may be the feature amount obtained from a high-resolution image.

[0070] Through the above-described processing, the second processing unit 55B of the control device 50 can track the user with higher accuracy.

[0071] [Tracking process (part 2)] In the above example, the user is described as a customer visiting a store, but if the user is a store clerk or facility staff member (e.g., a medical professional within the facility), the following processing may be performed.

[0072] (User registration process) The process of tracking the user in step S102 may be performed as follows. FIG. 8 is a diagram for explaining another example of the process of tracking the user by the second processing unit 55B (the process of step S102 in FIG. 3). The second processing unit 55B extracts an area including a person from the low-resolution image, and extracts an area (an area including the person) corresponding to the area extracted from the high-resolution image. The second processing unit 55B further extracts an area including the face of the person from the area extracted from the high-resolution image, and extracts a feature amount of the face of the person. The second processing unit 55B compares the feature amount of the extracted face part with the feature amount of the face part of the user to be tracked registered in advance in the user information 80, and if they match, determines that the person included in the image is the user to be tracked.

[0073] (Process to identify users) The process of identifying the user in step S106 may be performed as follows. If the second processing unit 55B is not successful in tracking the user, it extracts an area including a person in the vicinity from the high-resolution image as shown in Fig. 9. The second processing unit 55B extracts an area including a face of the person from the extracted area, extracts features of the face of the person, compares the face features of the person in the vicinity with the features of registered users, and identifies a person having features that match a threshold or more as a tracking target user.

[0074] As described above, the control device 50 can track the user with higher accuracy. In addition, the control device 50 extracts a person using a low-resolution image and further extracts a person using a high-resolution image as necessary, so that the processing load can be reduced.

[0075] [Processing related to behavior control] FIG. 10 is a flowchart showing an example of the flow of the behavior control process. This process is executed after the process of step S104 in FIG. 3. The control device 50 recognizes the gesture of the user (step S200), and controls the behavior of the moving object 10 based on the recognized gesture (step S202). Next, the control device 50 judges whether or not to end the service (step S204). If the service is not to be ended, the process returns to the process of step S102 in FIG. 3, and tracking is continued. If the service is to be ended, the control device 50 erases the registered information related to the user, such as the user's feature amount (step S206). For example, if the user makes a gesture or performs an operation indicating an intention to end the service, or inputs a voice, the service ends. Also, if the user or the moving object 10 reaches the boundary with the outside of the area where the service is provided, the provision of the service ends. This ends one routine of this flowchart.

[0076] The process of step S200 will be described. FIG. 11 is a diagram (part 1) for describing the process of recognizing a gesture. The second processing unit 55B specifies, in the high-resolution image IM corresponding to the time T, a person detection area (target area) that is the same as the person detection area including the tracked user detected in the low-resolution image IM2# corresponding to the time T. Then, the second processing unit 55B cuts out (extracts) the person detection area (target area) in the specified high-resolution image IM. The person detection area (target area) that is specified or cut out is not limited to the same person detection area as the person detection area including the tracked user, but may be a person detection area (target area) including the person detection area including the user. For example, in addition to the person detection area including the user, an area including another area may be specified and cut out, and this may be the target area.

[0077] The second processing unit 55B executes image recognition processing on the cut-out human detection area. The image recognition processing includes processing for recognizing a person's gesture, skeleton processing, processing for identifying an area including a person's arm or hand, or processing for extracting an area where the degree of change in the user's movement (for example, an arm or hand) is large. These will be described below.

[0078] FIG. 12 is a diagram (part 2) for explaining the process of recognizing a gesture. The second processing unit 55B performs skeleton processing on the image of the user included in the extracted person detection area. The second processing unit 55B extracts an area including one or both of the arms and the hands (hereinafter, the target area) from the skeleton processing result, and extracts a feature amount indicating the state of one or both of the arms and the hands in the extracted target area. The target area (an example of a "specific area") is, for example, an area used for recognizing a gesture. The second processing unit 55B specifies a feature amount matching the feature amount indicating the above-mentioned state from the feature amount included in the gesture information 74. The control device 50 causes the moving body 10 to perform the motion of the moving body 10 associated with the specified feature amount in the gesture information 74. Note that whether to extract the area including the hands or the area including the arms is determined depending on the position of the user relative to the moving body 10. For example, if the user is not at a predetermined distance from the moving object 10, a region including the hands is extracted, and if the user is at a predetermined distance from the moving object 10, a region including the arms is extracted.

[0079] FIG. 13 is a diagram (part 3) for explaining the process of recognizing gestures. The second processing unit 55B may recognize a gesture by preferentially using information on an area where the degree of change in the movement of a person in time series is large (an area including a part with a large degree of change among each part). The second processing unit 55B extracts an area (specific area) including an arm or hand with a large degree of change among a first degree of change in the left arm or left hand of the user and a second degree of change in the right arm or right hand of the user based on the result of skeleton processing on high-resolution images captured in time series, and recognizes a gesture of the user performed by the arm or hand included in the extracted area. That is, the second processing unit 55B recognizes a gesture by preferentially using information on an area with a large degree of change in time series (for example, the degree of change in the arm or hand) among two or more areas (specific areas). The two or more areas include at least a specific area identified as an area including the right arm or right hand of the target object and a specific area identified as an area including the left arm or left hand of the target object.

[0080] The second processing unit 55B extracts, as a target region, a region including the left arm or left hand that has a larger degree of change in the time series of the change in the right arm or right hand of the user and the time series of the change in the left arm or left hand, as shown in Fig. 13. The second processing unit 55B recognizes, for example, a gesture of the left arm or left hand that has a larger degree of change.

[0081] Alternatively, the second processing unit 55B may determine whether the user's right arm or right hand is making a gesture to control the moving body 10, and whether the user's left arm or left hand is making a gesture to control the moving body 10, and recognize the gesture based on the result of the determination.

[0082] In the above example, the tracking target has been described as a person, but instead of (or in addition to) this, the tracking target may be an object that is the subject of an action, such as a robot or an animal. In this case, the second processing unit 55B recognizes the gesture of the object, such as the robot or the animal.

[0083] (Gesture recognition processing) The control device 50 determines whether to refer to the first gesture information 76 or the second gesture information 78 of the gesture information 74, based on the relative positions of the moving object 10 and the user. As shown in Fig. 14, when the user is not a predetermined distance away from the moving object 10, in other words, when the user is within a first area AR1 set based on the moving object 10, the control device 50 determines whether the user is making the same gesture as the first gesture included in the first gesture information 76.

[0084] 15 is a diagram showing an example of a first gesture included in the first gesture information 76. The first gesture is, for example, a gesture using the hands without using the arms, as shown below. Gesture for moving the mobile object 10 forward: This gesture is a gesture in which the hand is thrust forward. Gesture for stopping the moving body 10 moving forward: This gesture is a gesture in which the palm of the hand is directly facing forward of the user. Gesture for moving the mobile object 10 to the left: This gesture is a gesture for moving the hand to the left. Gesture for moving the mobile object 10 to the right: This gesture is a gesture for moving the hand to the right. Gesture for moving the moving object 10 backward: This gesture is a gesture (beckoning gesture) in which the palm is turned in the opposite vertical direction and the fingertips are repeatedly moved so that they point toward the user (bringing the fingertips closer to the palm). Gesture for rotating the mobile object 10 to the left: This gesture is made by sticking out the index finger and thumb (or a specified finger) and rotating the fingers sticking out to the left. Gesture for rotating the mobile object 10 to the right: This gesture is made by pointing out the index finger and thumb (or a specified finger) and rotating the pointed fingers to the right.

[0085] As shown in FIG. 16, when the user is a predetermined distance away from the moving body 10, in other words, when the user is in the second area AR2 set based on the moving body 10 (when the user is not in the first area AR1), the control device 50 determines whether the user is performing the same gesture as the second gesture included in the second gesture information 78.

[0086] The second gesture is a gesture using the arm (the arm between the elbow and the hand) and the hand. The second gesture may be a body movement such as a larger gesture or a larger hand movement than the first gesture. A larger body movement means that when the moving body 10 is made to perform a certain movement (the same movement such as moving straight), the body movement of the second gesture is larger than the body movement of the first gesture. For example, the first movement may be a gesture using the hand or fingers, and the second gesture may be a gesture using the arm. For example, the first movement may be a gesture using the leg below the knee, and the second gesture may be a gesture using the lower body. For example, the first movement may be a gesture using the hand or foot, and the second gesture may be a gesture using the whole body such as jumping.

[0087] When the camera 22 of the moving body 10 captures an image of a user in the first area AR1 as shown in FIG. 14, the arm portion is difficult to fit in the image, but the hand and fingers fit in the image. The first area AR1 is an area in which the recognition unit 54 cannot or has difficulty recognizing the arm of the user from the image captured of the user in the first area AR1. When the camera 22 of the moving body 10 captures an image of a user in the second area AR2 as shown in FIG. 16, the arm portion fits in the image. Therefore, as described above, when the user is in the first area AR1, the recognition unit 54 recognizes the gesture using the first gesture information 76, and when the user is in the second area AR2, the recognition unit 54 recognizes the gesture using the second gesture information 78, thereby making it possible to recognize the user's gesture with higher accuracy.

[0088] FIG. 17 is a diagram showing an example of the second gesture included in the second gesture information 78. As shown in FIG. A gesture for moving the moving body 10 located behind the user to the front of the user: This gesture is made by the user pushing his / her arm and hand from near the body to the front of the body. Gesture for moving the mobile object 10 forward: This gesture is made by thrusting the arm and hand forward. Gesture for stopping the moving body 10 moving forward: This gesture is made by extending the arm and hand forward with the palm facing forward. Gesture for moving the mobile object 10 to the left: This gesture is for moving the arm and hand to the left. Gesture for moving the mobile object 10 to the right: This gesture is a gesture for moving the arm and hand to the right. A gesture for moving backward the moving object 10. This gesture is a gesture (beckoning gesture) in which the palm is turned in the opposite vertical direction and the arm or wrist is repeatedly moved so that the fingertips point toward the user. Gesture for rotating the mobile object 10 to the left: This gesture is made by sticking out an index finger (or a specified finger) and rotating the finger to the left. Gesture for rotating the mobile object 10 to the right: This gesture is made by sticking out an index finger (or a specified finger) and rotating the finger to the right.

[0089] [flowchart] 18 is a flowchart showing an example of a process in which the control device 50 recognizes a gesture. First, the control device 50 determines whether or not a user is present in the first area (step S300). If a user is present in the first area, the control device 50 recognizes the behavior of the user based on the acquired images (step S302). The behavior is, for example, the movement of the user recognized from images acquired continuously in time.

[0090] Next, the control device 50 refers to the first gesture information 76 to identify a gesture that matches the behavior recognized in step S302 (step S304). If the first gesture information 76 does not include a gesture that matches the behavior recognized in step S302, it is determined that a gesture that controls the movement of the moving object 10 is not performed. Next, the control device 50 performs an action corresponding to the identified gesture (step S306).

[0091] If the user does not exist in the first area (if the user exists in the second area), the control device 50 recognizes the user's behavior based on the acquired image (step S308), and identifies a gesture that matches the behavior recognized in step S308 by referring to the second gesture information 78 (step S310). Next, the control device 50 performs an action corresponding to the identified gesture (step S312). This ends the processing of one routine of this flowchart.

[0092] For example, in the above process, the recognition unit 54 recognizes the gestures of a user being tracked, and does not need to perform processing to recognize the gestures of a person not being tracked. This allows the control device 50 to control the moving object based on the gestures of a user being tracked with reduced processing load.

[0093] As described above, the control device 50 can more accurately recognize the user's gestures by switching the gestures to be recognized based on the area in which the user is present, and can operate the moving object 10 according to the user's intention. As a result, the user's convenience is improved.

[0094] According to the first embodiment described above, the control device 50 converts a first image into a second image having a lower resolution than the first image, acquires a target area including the target object to be tracked in the second image, and acquires a target area including the target object in the first image based on the target area in the acquired second image, thereby making it possible to accurately identify targets while reducing the processing load.

[0095] The above-described embodiment can be expressed as follows. A storage device storing a program; a hardware processor; The hardware processor executes the program stored in the storage device, converting a first image into a second image having a lower resolution than the first image; identifying a target area including a predetermined target in the second image based on the second image, and identifying a target area including the target in the first image based on the identified target area in the second image; Processing unit.

[0096] The above describes the form for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention. [Explanation of symbols]

[0097] REFERENCE SIGNS LIST 10 mobile object, 20 main body, 22 camera, 50 control device, 52 acquisition unit, 54 recognition unit, 55A first processing unit, 55B second processing unit, 56 trajectory generation unit, 58 driving control unit, 60 information processing unit, 70 storage unit, 74 gesture information, 76 first gesture information, 78 second gesture information, 80 user information

Claims

1. a first processing unit that converts a first image into a second image having a lower resolution than the first image; a second processing unit that identifies a target area including a predetermined target in the second image based on the second image, and identifies a target area including the target in the first image based on the identified target area in the second image, performing an image recognition process on a target area of ​​the first image corresponding to the target area specified by the second processing unit, and recognizing a gesture of the target object based on information of a specific area of ​​the target area in the first image; Processing unit.

2. a first processing unit that converts a first image into a second image having a lower resolution than the first image; a second processing unit that identifies a target area including a predetermined target in the second image based on the second image, and identifies a target area including the target in the first image based on the identified target area in the second image, the second processing unit identifies a target area including the object target in the first image before the second image is converted based on the target area of ​​the second image, and executes a predetermined image recognition process on the identified target area in the first image; The predetermined image recognition process includes a process of recognizing a gesture by preferentially using information of a specific region in which a degree of change in the movement of the target object over time is large among two or more specific regions of the target region in the first image. Processing unit.

3. The second processing unit analyzes the second image obtained by converting the first image captured at a first time and the second image obtained by converting the first image captured at a second time later than the first time, and tracks a target object included in the target area of ​​the second image corresponding to the first time in the second image corresponding to the second time. The processing device according to claim 1 or 2.

4. The second processing unit tracks the target object in the second images based on a change in a position of the target object in a time-series second image obtained by converting each of the first images captured in the time series. The processing device according to any one of claims 1 to 3.

5. the target object is a person, The second processing unit tracks the person in the second image based on one or more pieces of information of the color of the person's clothes, the gender of the person, and the hairstyle of the person, which are obtained from the second image obtained by converting the first image captured in the past or stored in a storage unit. The processing device according to any one of claims 1 to 4.

6. The predetermined image recognition process includes a process of recognizing a gesture of the target object based on information of a specific area of ​​the target area in the first image. The processing device according to claim 2 .

7. The predetermined image recognition process includes identifying the specific region based on recognizing a skeleton or a joint point with respect to the target region in the first image. The processing device of claim 6.

8. The predetermined image recognition process includes a process of determining an area including an arm or a hand of the target object as the specific area based on a recognition result of the skeleton or joint points. The processing device of claim 7.

9. The two or more specific regions include at least a specific region identified as a region including a right arm or a right hand of the target object and a specific region identified as a region including a left arm or a left hand of the target object. The processing device according to claim 2 .

10. the target object is a person, the second processing unit extracts an area including a face of a person in the first image, and tracks the person as the target object by comparing a facial feature amount included in the extracted area with a facial feature amount of the person previously stored in a storage unit.

10. The processing device according to any one of claims 1 to 9.

11. the target object is a person, The second processing unit is Tracking a person included in the target area of ​​the second image obtained by converting the first image captured at a first time in the second image obtained by converting the first image captured at a second time later than the first time; extracting an area including an arm or a hand of the tracked person in the first image captured at the second time based on a result of the tracking, and recognizing a gesture of the person based on the extracted area. The processing device according to any one of claims 1 to 10.

12. The device further includes a control unit that controls a moving object based on the gesture recognized by the second processing unit.

12. The processing device of claim 6, 7 or 11.

13. A mobile object having the processing device according to any one of claims 1 to 12 mounted thereon.

14. The computer Converting a first image into a second image having a lower resolution than the first image; Identifying a target area including a predetermined target in the second image based on the second image, and identifying a target area including the target in the first image based on the identified target area in the second image; performing an image recognition process on the target region of the first image corresponding to the specified target region, and recognizing a gesture of the target object based on information of the specified region of the target region in the first image; Processing methods.

15. On the computer, Transforming a first image into a second image having a lower resolution than the first image; Identifying a target area including a predetermined target in the second image based on the second image, and identifying a target area including the target in the first image based on the identified target area in the second image; performing an image recognition process on the target area of ​​the first image corresponding to the specified target area, and recognizing a gesture of the target object based on information of the specified area of ​​the target area in the first image; program.

Images