Processing device, mobile object, processing method and storage medium

By converting high-resolution images into low-resolution images and combining character features and gesture recognition, the problems of complex equipment structure and high processing load in the prior art are solved, and precise object target determination and moving body control are achieved.

CN115052103BActive Publication Date: 2024-11-05HONDA MOTOR CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210184131.4
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2021-03-08
Filing Date
2022-02-25
Publication Date
2024-11-05
Estimated Expiration
2042-02-25

AI Technical Summary

Technical Problem

In the prior art, the equipment used to track object information processing devices has a complex structure and a high processing load, making it difficult to accurately determine object targets.

Method used

By converting high-resolution images into low-resolution images and determining the object target area based on low-resolution images, the object target area is determined in high-resolution images, reducing processing load, and using character features and gesture recognition for accurate results Track and control.

Benefits of technology

It realizes that while reducing the processing load, the object target is determined with good accuracy, improves the accuracy of character tracking and gesture recognition, and can control the moving object according to the character's intention.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115052103B_ABST
    Figure CN115052103B_ABST
Patent Text Reader

Abstract

The present invention provides a processing device, a mobile object, a processing method, and a storage medium that can reduce processing load and accurately determine an object target. The processing device includes: a first processing unit that converts a first image into a second image with a lower resolution than the first image; and a second processing unit that determines an object target area in the second image that includes a specified target object target based on the second image, and determines an object target area in the first image that includes the target object target based on the determined object target area in the second image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a processing device, a mobile object, a processing method and a storage medium. Background Art

[0002] In the past, an information processing device that analyzes images captured by two cameras to track an object has been disclosed (Japanese Patent Publication No. 2018-88234). Summary of the invention

[0003] However, in the above-mentioned techniques, there are cases where the structure of the equipment used is complicated or the processing load is high.

[0004] The present invention has been made in consideration of such circumstances, and one of its objects is to provide a processing device, a moving object, a processing method, and a storage medium that can reduce the processing load and identify an object with high accuracy.

[0005] Solutions to Solve Problems

[0006] The processing device, the mobile object, the processing method, and the storage medium of the present invention adopt the following structures.

[0007] (1): A processing device according to one embodiment of the present invention comprises: a first processing unit, which transforms a first image into a second image having a lower resolution than the first image; and a second processing unit, which determines an object target area in the second image including a specified object object target based on the second image, and determines an object target area in the first image including the object object target based on the determined object target area in the second image.

[0008] (2): In the above-mentioned scheme (1), the second processing unit analyzes the second image transformed from the first image captured at the first moment and the second image transformed from the first image captured at a second moment later than the first moment, and tracks the object target contained in the object target area of ​​the second image corresponding to the first moment in the second image corresponding to the second moment.

[0009] (3): In the above-mentioned scheme (1) or (2), the second processing unit tracks the object target in the second image based on the change in the position of the object target in the second image of the time series which is transformed from the first images captured in the time series.

[0010] (4): In any of the above schemes (1) to (3), the target object is a person, and the second processing unit tracks the person in the second image based on one or more information of the color of the person's clothes, the gender of the person, and the hairstyle of the person, obtained from the second image transformed from the first image taken in the past or stored in the storage unit.

[0011] (5): In any of the above schemes (1) to (4), the second processing unit determines the object target area including the object object target in the first image before being transformed into the second image based on the object target area of ​​the second image, and performs prescribed image recognition processing on the determined object target area in the first image.

[0012] (6): In the above-mentioned aspect (5), the predetermined image recognition processing includes a processing of recognizing a gesture of the target object based on information of a specific area in the object area in the first image.

[0013] (7): In the scheme of (6) above, the prescribed image recognition processing includes a process of determining the specific area based on identifying bones or joint points of the object target area in the first image.

[0014] (8): In the above scheme (7), the prescribed image recognition processing includes processing of treating a region including an arm or a hand of the target object as the specific region based on the recognition result of the bones or joints.

[0015] (9): In any of the above schemes (6) to (8), the prescribed image recognition processing includes preferentially utilizing information of two or more specific areas in the object target area in the first image, in which the degree of change in the movement of the object target in the time series is large, to perform gesture recognition processing.

[0016] (10): In the above scheme (9), the two or more specific areas include at least a specific area determined to include the right arm or right hand of the target object and a specific area determined to include the left arm or left hand of the target object.

[0017] (11): In any of the above schemes (1) to (10), the target object is a person, and the second processing unit extracts the area including the face of the person in the first image, and compares the feature value of the face contained in the extracted area with the feature value of the face of the person pre-stored in the storage unit to track the person of the target object.

[0018] (12): In any of the above schemes (1) to (10), the target object is a person, and the second processing unit tracks the person included in the object target area of ​​the second image transformed from the first image captured at the first moment in the second image transformed from the first image captured at a second moment later than the first moment, and based on the tracking result, extracts the area including the tracked arm or hand of the person in the first image captured at the second moment, and recognizes the gesture of the person based on the extracted area.

[0019] (13): In any one of the above-mentioned schemes (6), (7) or (12), the processing device further includes a control unit that controls the moving body based on the gesture recognized by the second processing unit.

[0020] (14): A mobile object comprising a processing device according to any one of the above-mentioned schemes (1) to (13).

[0021] (15): A processing method of one embodiment of the present invention enables a computer to perform the following processing: transforming a first image into a second image having a lower resolution than the first image; and determining an object target area in the second image that includes a specified object target based on the second image, and determining an object target area in the first image that includes the object target based on the determined object target area in the second image.

[0022] (16): A program stored in a storage medium in one embodiment of the present invention causes a computer to perform the following processing: transforming a first image into a second image having a lower resolution than the first image; and determining an object target area in the second image that includes a specified object target based on the second image, and determining an object target area in the first image that includes the object target based on the determined object target area in the second image.

[0023] Effects of the Invention

[0024] According to (1)-(16), the processing device transforms the first image into a second image having a lower resolution than the first image, and obtains an object target area including the object object target in the first image based on the object target area including the object object target in the second image, thereby reducing the processing load and determining the object target with good accuracy.

[0025] According to (4), the processing device can track the person with higher accuracy by using the characteristics of the person.

[0026] According to (6), the processing device can recognize the gesture of an object such as a person, a robot, or an animal that is the subject of the action with higher accuracy.

[0027] According to (9) or (10), the processing device can recognize the gesture with higher accuracy by preferentially utilizing information of a specific area having a large degree of change in time series.

[0028] According to (11), the processing device can track the person with higher accuracy by using the feature value of the person's face.

[0029] According to (13), the processing device can control the moving object according to the character's intention. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a diagram showing an example of a moving object provided with a control device according to an embodiment.

[0031] Figure 2 This is a diagram showing an example of a functional configuration included in the main body of a moving object.

[0032] Figure 3 This is a flowchart showing an example of the flow of tracking processing.

[0033] Figure 4 This is a diagram for explaining a process of extracting a user's feature amount and a process of registering the feature amount.

[0034] Figure 5 This is a diagram for explaining a process in which the recognition unit tracks a user.

[0035] Figure 6 This is a diagram for explaining tracking processing using feature amounts.

[0036] Figure 7 This is a diagram for explaining tracking processing using feature amounts.

[0037] Figure 8 This is a diagram for explaining another example of the process of tracking the user by the second processing unit.

[0038] Fig. 9 This is a diagram showing an example of a process of extracting a region including a person located in the periphery from a high-resolution image.

[0039] Fig.10 This is a flowchart showing an example of the flow of action control processing.

[0040] Fig.11 This is a diagram (part 1) for explaining a process of recognizing a gesture.

[0041] Fig.12 This is a diagram (part 2) for explaining the process of recognizing a gesture.

[0042] Fig.13This is a diagram (part 3) for explaining the process of recognizing a gesture.

[0043] Fig.14 It is a diagram for explaining a process of recognizing a gesture.

[0044] Fig.15 : is a diagram showing an example of a first gesture included in the first gesture information.

[0045] Fig.16 It is a diagram for explaining a process of recognizing a gesture.

[0046] Fig.17 : is a diagram showing an example of the second gesture included in the second gesture information.

[0047] Fig.18 This is a flowchart showing an example of a process in which the control device recognizes a gesture. DETAILED DESCRIPTION

[0048] Hereinafter, a processing device, a moving object, a processing method, and a storage medium according to embodiments of the present invention will be described with reference to the drawings.

[0049] <First Embodiment>

[0050] [Overall structure]

[0051] Figure 1 1 is a diagram showing an example of a mobile body 10 having a processing device according to an embodiment. The mobile body 10 is an autonomous mobile robot. The mobile body 10 supports the actions of a user. For example, the mobile body 10 supports the shopping or reception of a customer or supports the work of a staff member according to instructions of a store clerk, a customer, or a staff member of a facility (hereinafter referred to as a "user").

[0052] The mobile body 10 includes a main body 20, a container 92, and one or more wheels 94 (wheels 94A and 94B in the figure). The mobile body 10 moves according to the user's gesture, voice, operation of the input unit (touch panel described later) of the mobile body 10, and instruction of operation of a terminal device (such as a smartphone). The mobile body 10 recognizes the gesture based on an image captured by a camera 22 provided on the main body 20, for example.

[0053] For example, the mobile body 10 drives the wheels 94 to follow the movement of the user or guide the customer. At this time, the mobile body 10 explains the goods and operations to the user or guides the user to the goods and objects he is looking for. The user can store the goods and goods to be purchased in the storage container 92.

[0054] In the present embodiment, the case where the mobile body 10 is provided with the storage container 92 is described, but instead of (or on the basis of) this, the mobile body 10 may be provided with a seat for the user to sit on, a frame for the user to sit on, a pedal for the user to put his feet, etc., so that the user can move with the mobile body 10.

[0055] Figure 2 This is a diagram showing an example of a functional configuration included in the main body 20 of the mobile object 10. The main body 20 includes a camera 22, a communication unit 24, a position determination unit 26, a speaker 28, a microphone 30, a touch panel 32, a motor 34, and a control device 50 (an example of a "processing device").

[0056] The camera 22 captures the surroundings of the moving body 10. The camera 22 is, for example, a fisheye camera that can capture the surroundings of the moving body 10 at a wide angle (e.g., 360 degrees). The camera 22 is, for example, mounted on the upper portion of the moving body 10, and captures the surroundings of the moving body 10 at a wide angle in the horizontal direction. The camera 22 may also be realized by combining a plurality of cameras (a plurality of cameras that capture a range of 120 degrees or a range of 60 degrees in the horizontal direction). The camera 221 is not limited to one, and may be installed on a plurality of moving bodies 10.

[0057] The communication unit 24 is a communication interface for communicating with other devices using, for example, a cellular network, a Wi-Fi network, Bluetooth (registered trademark), DSRC (Dedicated Short Range Communication), or the like.

[0058] The position determination unit 26 determines the position of the moving object 10. The position determination unit 26 obtains the position information of the moving object 10 using a GPS (Global Positioning System) device (not shown) built into the moving object 10. The position information may be, for example, two-dimensional map coordinates or longitude and latitude information.

[0059] The speaker 28 outputs, for example, a predetermined sound. The microphone 30 receives input of, for example, a sound uttered by a user.

[0060] The touch panel 32 is composed of a display unit such as LCD (liquid crystal display) and organic EL (electroluminescence), and an input unit that can detect the touch position of the operator through a coordinate detection mechanism. The display unit displays a GUI (Graphical User Interface) switch for operation. When the input unit detects a touch operation, a flick operation, a slide operation, etc. on the GUI switch, it generates an operation signal indicating that the touch operation on the GUI switch has been performed, and outputs it to the control device 50. The control device 50 causes the speaker 28 to output sound or causes the touch panel 32 to display an image according to the operation. The control device 50 can also move the movable body 10 according to the operation.

[0061] The motor 34 drives the wheels 94 to move the vehicle 10. The wheels 94 include, for example, driving wheels driven in the rotation direction by the motor 34 and non-driving wheels, i.e., steering wheels, driven in the yaw direction. By adjusting the angle of the steering wheels, the vehicle 10 can change the route or rotate.

[0062] In the present embodiment, the moving body 10 includes wheels 94 as a mechanism for achieving movement, but the present embodiment is not limited to this structure. For example, the moving body 10 may be a multi-legged walking robot.

[0063] The control device 50 includes, for example, an acquisition unit 52, an identification unit 54, a track generation unit 56, a travel control unit 58, an information processing unit 60, and a storage unit 70. Part or all of the acquisition unit 52, the identification unit 54, the track generation unit 56, the travel control unit 58, and the information processing unit 60 are implemented by, for example, executing a program (software) by a hardware processor such as a CPU (Central Processing Unit). Part or all of these functional units may also be implemented by hardware (including a circuit unit; circuitry) such as LSI (Large Scale Integration), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), or by the coordination of software and hardware. The program may be pre-stored in a storage unit 70 (a storage device having a non-temporary storage medium) such as a HDD (Hard Disk Drive) or a flash memory, or may be stored in a removable storage medium (non-temporary storage medium) such as a DVD or CD-ROM, and installed by assembling the storage medium in a drive device. The acquisition unit 52, the recognition unit 54, the track generation unit 56, the travel control unit 58, or the information processing unit 60 may be provided in a device different from the control device 50 (mobile body 10). For example, the recognition unit 54 may be provided in another device, and the control device 50 controls the mobile body 10 based on the processing result of the other device. A part or all of the information stored in the storage unit 70 may be stored in another device. The configuration including one or more functional units in the acquisition unit 52, the recognition unit 54, the track generation unit 56, the travel control unit 58, or the information processing unit 60 may be configured as a system.

[0064] The storage unit 70 stores map information 72, gesture information 74, and user information 80. The map information 72 is information that represents the shape of roads and passages in facilities by lines and nodes connected by the lines. The map information 72 may also include road curvature, POI (Point Of Interest) information, and the like.

[0065] The gesture information 74 is information that establishes a correspondence between information related to the gesture (feature quantity of the template) and the movement of the mobile body 10. The gesture information 74 includes first gesture information 76 and second gesture information 78. The user information 80 is information indicating the feature quantity of the user. Details of the gesture information 74 and the user information 80 will be described later.

[0066] The acquisition unit 52 acquires an image (hereinafter referred to as a “surrounding image”) captured by the camera 22. The acquisition unit 52 stores the acquired surrounding image as pixel data in a fisheye camera coordinate system, for example.

[0067] The recognition unit 54 includes, for example, a first processing unit 55A and a second processing unit 55B. The first processing unit 55A converts a first image (for example, a high-resolution image) captured by the camera 22 into a second image (low-resolution image) having a lower resolution than the first image. The second processing unit 55B determines an object target area of ​​the second image including a specified target object target based on the second image, and determines an object target area including the target object target in the first image based on the determined object target area in the second image. The target object target refers to, for example, an object target of a tracking or pursuit object. Details of the processing of the first processing unit 55A and the second processing unit 55B will be described later.

[0068] The second processing unit 55B recognizes the user's body movement (hereinafter referred to as "gesture") based on one or more surrounding images. The recognition unit 54 recognizes the gesture by comparing the feature quantity of the user's gesture extracted from the surrounding image with the feature quantity of the template (the feature quantity representing the gesture). The feature quantity is, for example, data representing characteristic parts such as a person's fingers, finger joints, wrists, arms, bones, etc., lines connecting the characteristic parts, and the inclination or position of the line.

[0069] The track generation unit 56 generates a track that the moving body 10 should travel in the future based on the user's gesture, the destination set by the user, the surrounding objects, the user's position, the map information 72, etc. The track generation unit 56 combines a plurality of arcs to generate a track that allows the moving body 10 to move smoothly to the target location. The track generation unit 56 generates the track by combining, for example, three arcs. The track generation unit 56 can generate the track by, for example, fitting a geometric model such as a Bezier curve. The track is actually generated as a collection of a finite number of track points, for example.

[0070] The track generation unit 56 performs coordinate transformation between the orthogonal coordinate system and the fisheye camera coordinate system. A one-to-one relationship is established between the coordinates of the orthogonal coordinate system and the fisheye camera coordinate system, and the relationship is stored in the storage unit 70 as corresponding information. The track generation unit 56 generates a track in the orthogonal coordinate system (orthogonal coordinate system track), and transforms the track coordinates into a track in the fisheye camera coordinate system (fisheye camera coordinate system track). The track generation unit 56 calculates the risk of the fisheye camera coordinate system track. The risk is an index value indicating the magnitude of the possibility that the moving body 10 approaches an obstacle. The risk has a tendency that the smaller the distance between the track (track point of the track) and the obstacle is, the higher the risk is, and the larger the distance between the track and the obstacle is, the lower the risk is.

[0071] When the total risk value and the risk of each track point satisfy a preset criterion (for example, when the total risk value is less than a threshold value Th1 and the risk of each track point is less than a threshold value Th2), the trajectory generating unit 56 adopts the trajectory satisfying the criterion as the trajectory of the moving body.

[0072] In the case where the above-mentioned track does not satisfy the pre-set benchmark, the following processing may be performed. The track generation unit 56 detects the drivable space in the fisheye camera coordinate system, and transforms the detected drivable space coordinates in the fisheye camera coordinate system into the drivable space in the orthogonal coordinate system. The drivable space refers to the space obtained by removing obstacles and the surrounding areas of the obstacles (areas where risks are set or areas where risks are above a threshold) in the area of ​​the moving direction of the mobile body 10. The track generation unit 56 corrects the track in such a way that the track is contained in the drivable space transformed into the orthogonal coordinate system. The track generation unit 56 transforms the coordinates of the orthogonal coordinate system track into the fisheye camera coordinate system track, and calculates the risk of the fisheye camera coordinate system track based on the surrounding image and the fisheye camera coordinate system track. This process is repeated to search for a track that satisfies the above-mentioned pre-set benchmark.

[0073] The travel control unit 58 causes the vehicle 10 to travel along a track that satisfies a preset reference. The travel control unit 58 outputs a command value for causing the vehicle 10 to travel along the track to the motor 34. The motor 34 rotates the wheels 94 according to the command value to cause the vehicle 10 to move along the track.

[0074] The information processing unit 60 controls various devices and equipment included in the main body 20. The information processing unit 60 controls, for example, the speaker 28, the microphone 30, and the touch panel 32. The information processing unit 60 recognizes the sound input to the microphone 30 and the operation performed on the touch panel 32. The information processing unit 60 operates the moving body 10 based on the recognition result.

[0075] In the above example, although it has been described that the recognition unit 54 uses the image captured by the camera 22 provided on the mobile body 10 for various processes, the recognition unit 54 may also use the image captured by a camera not provided on the mobile body 10 (a camera provided at a position different from the mobile body 10) to perform various processes. In this case, the image captured by the camera is transmitted to the control device 50 via communication, and the control device 50 obtains the transmitted image and performs various processes based on the obtained image. The recognition unit 54 may perform various processes using a plurality of images. For example, the recognition unit 54 may also perform various processes based on the image captured by the camera 22 and a plurality of images captured by a camera provided at a position different from the mobile body 10.

[0076] [Support Processing]

[0077] The mobile body 10 performs a support process for supporting the user's shopping. The support process includes a process related to tracking and a process related to behavior control.

[0078] [Processing related to tracking (Part 1)]

[0079] Figure 3 1 is a flowchart showing an example of the tracking process. First, the control device 50 of the mobile body 10 accepts the user's login (step S100). Next, the control device 50 tracks the user who logged in at step S100 (step S102). Next, the control device 50 determines whether the tracking is successful (step S104). If the tracking is successful, the process proceeds to the following step. Fig.10 If the tracking is unsuccessful, the control device 50 determines the user (step S106).

[0080] (Processing for logged-in users)

[0081] The process of user registration in step S100 is described below. The control device 50 of the mobile body 10 confirms the user's intention to register based on a specific gesture, voice, or operation of the touch panel 32 by the user (for example, a customer visiting a store). When the user's intention to register can be confirmed, the recognition unit 54 of the control device 50 extracts the user's feature quantity and registers the extracted feature quantity.

[0082] Figure 4 5 is a diagram for explaining the process of extracting the feature quantity of the user and the process of logging the feature quantity. The second processing unit 55B of the control device 50 determines the user from the image IM1 taken by the user, and identifies the joint points and bones of the determined user (performs skeleton processing). For example, the second processing unit 55B estimates the user's face, facial parts, neck, shoulders, elbows, wrists, waist, ankles, etc. from the image IM1, and performs skeleton processing based on the positions of the estimated parts. For example, the second processing unit 55B uses a known method (such as a method such as human posture recognition) to perform skeleton processing, and the known method is a method of estimating the user's joint points and estimating by using deep learning. Next, the second processing unit 55B determines the user's face, upper body, lower body, etc. based on the result of the skeleton processing, and extracts the feature quantity of each determined face, upper body, and lower body, and logs the extracted feature quantity in the storage unit 70 as the user's feature quantity. The feature quantity of the face is, for example, the feature quantity of male, female, hairstyle, and face. The feature quantity indicating male or female is a feature quantity indicating the shape of the head, etc., and the hairstyle is information indicating the length of the hair (short hair, long hair, etc.) obtained from the shape of the head. The feature quantity of the upper body is, for example, the color of the upper body. The feature quantity of the lower body is, for example, the color of the lower body.

[0083] (Tracking user processing)

[0084] The first processing unit 55A converts the high-resolution images captured per unit time into low-resolution images. For example, high resolution means that the number of pixels per unit area in the image is greater than the number of pixels per unit area in the low-resolution image (high dpi). The first processing unit 55A performs pixel thinning processing on the high-resolution image IM to convert the high-resolution image into the low-resolution image, or converts the high-resolution image into the low-resolution image by applying a predetermined algorithm.

[0085] The second processing unit 55B analyzes the low-resolution image converted from the high-resolution image captured at the first moment and the low-resolution image converted from the high-resolution image captured at the second moment later than the first moment, and tracks the object target included in the object target area of ​​the low-resolution image corresponding to the first moment, which includes the object target to be tracked. The second processing unit 55B tracks the object target in the low-resolution image based on the change in the position of the object target in the low-resolution images of the time series converted from the high-resolution images captured in the time series. The low-resolution image used for the tracking is, for example, a low-resolution image converted from the most recently captured high-resolution image. The following is a specific description.

[0086] The process of tracking the user in step S102 will be described. Figure 5 This is used to explain the process of tracking the user by the identification unit 54 ( Figure 3 The first processing unit 55A of the recognition unit 54 acquires an image captured at time T. This image is an image captured by the camera 22 (hereinafter referred to as a high-resolution image IM2).

[0087] First processing unit 55A of recognition unit 54 converts high-resolution image IM2 into low-resolution image IM2# having a lower resolution than high-resolution image IM2. Next, second processing unit 55B detects a person and a person detection area including the person from low-resolution image IM2#.

[0088] The second processing unit 55B estimates the position of the user at time T (person detection area) based on the position of the person detected at time T-1 (before time T) (person detection area of ​​the user being tracked at time T-1) and the moving direction of the person. When the user detected in the low-resolution image IM2 obtained at time T exists near the position estimated from the position or moving direction of the user of the tracking target before time T-1, the second processing unit 55B determines the user detected at time T as the user of the target to be tracked (tracking target). If the user can be identified, tracking is considered successful.

[0089] As described above, control device 50 uses low-resolution image IM2# to track the user, thereby reducing the processing load.

[0090] In the tracking process, the second processing unit 55B may track the user using the user's feature amount in addition to the position of the user at time T and time T-1 as described above. Figure 6 2 is a diagram for explaining tracking processing using feature quantities. For example, the second processing unit 55B estimates the position of the user at time T, identifies the user existing near the estimated position, and further extracts the feature quantity of the user. When the extracted feature quantity and the registered feature quantity match by more than a threshold value, the control device 50 estimates the identified user as the user of the tracking target and determines that the tracking is successful.

[0091] For example, when extracting the user's feature quantity, the second processing unit 55B extracts a region including a person, and performs skeleton processing on the image (high-resolution image) of the extracted region to extract the feature quantity of the person. This reduces the processing load.

[0092] In place of the feature quantity obtained from the high-resolution image, when the feature quantity obtained from the low-resolution image matches the registered feature quantity by a threshold or more, the second processing unit 55B may infer that the determined user is the user of the tracking target. In this case, the storage unit 70 pre-stores the feature quantity for comparison with the feature quantity obtained from the low-resolution image, and uses the feature quantity. The second processing unit 55B may also compare the feature quantity extracted from the image obtained during tracking with the feature quantity obtained from the image captured this time, for example, instead of (or in addition to) the registered feature quantity, to determine the user.

[0093] For example, even when a user who is a tracking target overlaps or intersects with another person, the user can be tracked more accurately based on the change in the user's position and the user's feature amount as described above.

[0094] (Determine the user's process)

[0095] The process of determining the user in step S106 is described below. If the tracking of the user is unsuccessful, the second processing unit 55B Figure 7As shown, the feature quantity of the people located in the surrounding area is compared with the feature quantity of the logged-in user to determine the user to be tracked. The second processing unit 55B, for example, extracts the feature quantity of each person included in the image. The second processing unit 55B compares the feature quantity of each person with the feature quantity of the logged-in user to determine the person whose feature quantity matches the feature quantity of the logged-in user by more than a threshold. The second processing unit 55B uses the determined user as the user to be tracked. At this time, the feature quantity used may be a feature quantity obtained from a low-resolution image, or may be a feature quantity obtained from a high-resolution image.

[0096] Through the above-described processing, the second processing unit 55B of the control device 50 can track the user with higher accuracy.

[0097] [Processing related to tracking (Part 2)]

[0098] In the above example, the case where the user is a customer visiting a store is described, but the following processing can also be performed when the user is a store clerk or a facility employee (for example, a person engaged in medical treatment in the facility).

[0099] (Processing for logged-in users)

[0100] The process of tracking the user in step S102 may also be performed as follows. Figure 8 This is used to explain the process of tracking the user by the second processing unit 55B ( Figure 3 The second processing unit 55B extracts the area including the person from the low-resolution image, and extracts the area corresponding to the area extracted from the high-resolution image (the area including the person). The second processing unit 55B further extracts the area including the face of the person from the area extracted from the high-resolution image, and extracts the feature amount of the face of the person. The second processing unit 55B compares the feature amount of the extracted face with the feature amount of the face of the user of the tracking target pre-registered in the user information 80, and when they match, determines that the person included in the image is the user of the tracking target.

[0101] (Determine the user's process)

[0102] The process of determining the user in step S106 can also be performed as follows. When the tracking of the user is unsuccessful, the second processing unit 55B Fig. 9 As shown, an area including a person located in the periphery is extracted from the high-resolution image. The second processing unit 55B extracts an area including a facial part of the person from the extracted area, extracts a feature amount of the facial part of the person, and compares the feature amount of the face of the person located in the periphery with the feature amount of the logged-in user, so as to determine a person having a feature amount that matches a threshold value or more as a user to be tracked.

[0103] As described above, the control device 50 can track the user with higher accuracy. The control device 50 extracts the person using the low-resolution image and further extracts the person using the high-resolution image as needed, thereby reducing the processing load.

[0104] [Processing related to action control]

[0105] Fig.10 is a flowchart showing an example of the process of action control processing. Figure 3 The control device 50 recognizes the user's gesture (step S200), and controls the behavior of the mobile body 10 based on the recognized gesture (step S202). Next, the control device 50 determines whether to end the service (step S204). If the service is not ended, return to Figure 3 The processing of step S102 is continued, and tracking is continued. When the service is terminated, the control device 50 deletes the registered login information related to the user, such as the user's feature quantity (step S206). For example, when the user performs a gesture, operation, etc. indicating the intention to terminate the service, or inputs a voice, the service is terminated. When the user or the mobile body 10 reaches the boundary outside the area where the service is provided, the provision of the service is terminated. Thus, one routine of this flowchart ends.

[0106] The process of step S200 will be described. Fig.11 : is a figure (1) for illustrating the processing of recognizing gestures. The second processing unit 55B determines, in the high-resolution image IM corresponding to the time T, a person detection area (object target area) that is the same as the person detection area including the user being tracked, detected in the low-resolution image IM2# corresponding to the time T. Then, the second processing unit 55B cuts out (extracts) the person detection area (object target area) in the determined high-resolution image IM. The determined or cut-out person detection area (object target area) is not limited to the person detection area that is the same as the person detection area including the above-mentioned user being tracked, but may also be a person detection area (object target area) that includes the person detection area including the above-mentioned user. For example, in addition to the person detection area including the above-mentioned user, an area including other areas may also be determined and cut out and used as the object target area.

[0107] The second processing unit 55B performs image recognition processing on the cut-out person detection area. The image recognition processing includes processing for recognizing a person's gesture, skeleton processing, processing for determining an area including a person's arm or hand, or processing for extracting an area where the user's motion (such as an arm or hand) varies greatly. These are described below.

[0108] Fig.12 : is a figure (part 2) for explaining the processing of recognizing gestures. The second processing unit 55B performs skeleton processing on the image of the user included in the cut-out person detection area. The second processing unit 55B extracts a region including one or both of the arm and the hand (hereinafter referred to as the target region) from the result of the skeleton processing, and extracts a feature quantity representing the state of one or both of the arm and the hand in the extracted target region. The target region (an example of a "specific region") is, for example, a region used for recognition of gestures. The second processing unit 55B determines a feature quantity that matches the feature quantity representing the above-mentioned state from the feature quantity included in the gesture information 74. The control device 50 causes the mobile body 10 to perform an action of the mobile body 10 associated with the feature quantity determined in the gesture information 74. Whether to extract the region including the hand or the region including the arm is determined according to the position of the user relative to the mobile body 10. For example, when the user is not separated from the mobile body 10 by more than a specified distance, the region including the hand is extracted, and when the user is separated from the mobile body 10 by more than a specified distance, the region including the arm is extracted.

[0109] Fig.13 : is a figure (part 3) for explaining the processing of identifying gestures. The second processing unit 55B can identify gestures by preferentially using information of areas with large changes in the degree of motion of the person in the time series (including areas of parts with large changes in each part). The second processing unit 55B extracts the areas (specific areas) including the arms or hands with large changes in the first degree of change of the left arm or left hand of the user and the second degree of change of the right arm or right hand of the user based on the results of skeleton processing of the high-resolution images captured in the time series, and identifies the gestures of the user performed by the arms or hands contained in the extracted areas. That is, the second processing unit 55B identifies gestures by preferentially using information of areas with large changes in the degree of change (for example, the degree of change of the arms or hands) in the time series among two or more areas (specific areas). The two or more areas include at least a specific area determined as an area including the right arm or right hand of the target object and a specific area determined as an area including the left arm or left hand of the target object.

[0110] The second processing unit 55B is, for example, Fig.13 As shown, the area including the left arm or left hand with a greater degree of change in the time series of the right arm or right hand of the user is extracted as the target area. The second processing unit 55B, for example, recognizes the gesture of the left arm or left hand with a greater degree of change.

[0111] Alternatively, the second processing unit 55B may determine whether the user's right arm or right hand is performing a gesture for controlling the mobile object 10 and whether the user's left arm or left hand is performing a gesture for controlling the mobile object 10 and recognize the gesture based on the determination result.

[0112] In the above example, the case where the object to be tracked is a person is described, but the object to be tracked may be an object that is the subject of the action, such as a robot or an animal, instead of (or in addition to) this. In this case, the second processing unit 55B recognizes the gesture of the object, such as the robot or the animal.

[0113] (Gesture recognition processing)

[0114] The control device 50 determines whether to refer to the first gesture information 76 of the gesture information 74 or the second gesture information 78 based on the relative position between the mobile body 10 and the user. Fig.14 As shown, when the user is not separated from the moving body 10 by a specified distance, in other words, when the user is within the first area AR1 set based on the moving body 10, the control device 50 determines whether the user is performing a gesture identical to the first gesture included in the first gesture information 76.

[0115] Fig.15 : is a figure which shows an example of the 1st gesture included in the 1st gesture information 76. The 1st gesture is a gesture which uses a hand instead of an arm as shown below, for example.

[0116] ·Hand gesture for moving the moving object 10 forward: This hand gesture is a hand gesture of extending the hand forward.

[0117] · Gesture for stopping the moving object 10 that is moving forward: This gesture is a gesture in which the palm of the hand faces the front direction of the user.

[0118] · Gesture for moving the mobile object 10 to the left: This gesture is a gesture for moving the hand to the left.

[0119] · Gesture for moving the mobile object 10 to the right: This gesture is a gesture for moving the hand to the right.

[0120] · Gesture for moving the moving object 10 backward: This gesture is a gesture of repeatedly moving the fingertips (a gesture of waving) with the palm facing in the opposite direction of the vertical direction and the fingertips facing the user (bringing the fingertips close to the palm).

[0121] ·Hand gesture for rotating the moving body 10 in the left direction: This hand gesture is a hand gesture for rotating the fingers protruding in the left direction by protruding the index finger and thumb (or predetermined fingers).

[0122] ·Hand gesture for rotating the moving body 10 in the right direction: This hand gesture is a hand gesture for rotating the fingers protruding in the right direction by protruding the index finger and thumb (or predetermined fingers).

[0123] like Fig.16 As shown, when the user is separated from the mobile body 10 by a specified distance, in other words, when the user is present in the second area AR2 set based on the mobile body 10 (when the user is not present in the first area AR1), the control device 50 determines whether the user is performing a gesture that is the same as the second gesture included in the second gesture information 78.

[0124] The second gesture included in the second gesture information 78 is a gesture using an arm (the arm between the elbow and the hand) and a hand. The second gesture may be a body movement such as a body vibration larger than the first gesture, a large hand movement, etc. The large body movement is when the moving body 10 is made to perform a certain action (the same action as walking straight, etc.), and the body movement of the second gesture is larger than the body movement of the first gesture. For example, the first action may be a gesture using a hand or a finger, and the second gesture may be a gesture using an arm. For example, the first action may be a gesture using the leg below the knee, and the second gesture may be a gesture using the lower body. For example, the first action may be a gesture using a hand or a foot, etc., and the second gesture may be a gesture using the whole body such as jumping.

[0125] The camera 22 of the mobile object 10 is as described above. Fig.14 As shown in FIG. 1 , when photographing a user present in the first area AR1, the arm portion is difficult to be included in the image, and the hand and fingers are included in the image. The first area AR1 is an area where the recognition unit 54 cannot recognize the user's arm from the image photographed by the user present in the first area AR1, or it is difficult to recognize the user's arm. Fig.16 As shown, when the user existing in the second area AR2 is photographed, the arm portion is included in the image. Therefore, as described above, when the user exists in the first area AR1, the recognition unit 54 recognizes the gesture by using the first gesture information 76, and when the user exists in the second area AR2, the recognition unit 54 recognizes the gesture by using the second gesture information 78, thereby being able to recognize the user's gesture with higher accuracy.

[0126] Fig.17 3 is a diagram showing an example of the second gesture included in the second gesture information 78 .

[0127] · A gesture for moving the mobile object 10 located behind the user to the front of the user: This gesture is a gesture in which the user pushes the arm and hand from the vicinity of the body to the front of the body.

[0128] ·Hand gesture for moving the moving body 10 forward: This hand gesture is a hand gesture in which the arm and hand are extended forward.

[0129] ·Hand gesture for stopping the moving object 10 that is moving forward: This hand gesture is a hand gesture in which the arm and the hand protruding forward are facing the palm in the forward direction.

[0130] ·Hand gesture for moving the mobile body 10 to the left: This hand gesture is a hand gesture for moving the arm and hand to the left.

[0131] ·Hand gesture for moving the mobile body 10 to the right: This hand gesture is a hand gesture for moving the arm and hand to the right.

[0132] · Gesture for moving the moving object 10 backward. This gesture is a gesture of repeatedly moving the arm or wrist so that the palm faces the vertical direction opposite to the direction in which the fingertips face the user (a gesture of waving).

[0133] ·Hand gesture for rotating the moving body 10 in the left direction: This hand gesture is a hand gesture for protruding the index finger (or a predetermined finger) and rotating the finger protruding in the left direction.

[0134] ·Hand gesture for rotating the moving body 10 in the right direction: This hand gesture is a hand gesture for protruding the index finger (or a predetermined finger) and rotating the finger protruding in the right direction.

[0135] [flow chart]

[0136] Fig.18 1 is a flowchart showing an example of gesture recognition processing by the control device 50. First, the control device 50 determines whether the user is present in the first area (step S300). If the user is present in the first area, the control device 50 recognizes the user's behavior based on the acquired image (step S302). The behavior is, for example, the movement of the user recognized from the images acquired continuously in time.

[0137] Next, the control device 50 identifies a gesture that matches the behavior recognized in step S302 by referring to the first gesture information 76 (step S304). When the gesture that matches the behavior recognized in step S302 is not included in the first gesture information 76, it is determined that the gesture for controlling the movement of the mobile body 10 is not performed. Next, the control device 50 performs an action corresponding to the identified gesture (step S306).

[0138] If the user is not present in the first area (if the user is present in the second area), the control device 50 recognizes the user's behavior based on the acquired image (step S308), and refers to the second gesture information 78 to determine a gesture that matches the behavior recognized in step S308 (step S310). Next, the control device 50 performs an action corresponding to the determined gesture (step S312). Thus, the processing of one routine of this flowchart ends.

[0139] For example, in the above process, the recognition unit 54 may recognize the gesture of the user being tracked and not perform the process of recognizing the gesture of the person not being tracked. Thus, the control device 50 can control the mobile object based on the gesture of the user being tracked with reduced processing load.

[0140] As described above, the control device 50 switches the gesture to be recognized based on the area where the user is present, thereby being able to recognize the user's gesture with higher accuracy and operate the moving object 10 according to the user's intention. As a result, user convenience is improved.

[0141] According to the first embodiment described above, the control device 50 transforms the first image into a second image having a lower resolution than the first image, obtains an object target area including the object target of the object to be tracked in the second image, and based on the object target area of ​​the obtained second image, obtains an object target area including the object target in the first image, thereby reducing the processing load and determining the object target with good accuracy.

[0142] The above-described embodiment can be expressed as follows.

[0143] A processing device is configured to include:

[0144] a storage device storing a program; and

[0145] Hardware processor,

[0146] The hardware processor executes the program stored in the storage device to perform the following processing:

[0147] transforming a first image into a second image having a lower resolution than the first image; and

[0148] An object target region including a specified target object target in the second image is determined based on the second image, and an object target region including the target object target in the first image is determined based on the determined object target region in the second image.

[0149] Although specific embodiments of the present invention have been described above using the embodiments, the present invention is not limited to such embodiments at all, and various modifications and substitutions can be made without departing from the gist of the present invention.

Claims

1. A processing device, wherein: The processing device comprises: a first processing unit that transforms a first image into a second image with a lower resolution than the first image; as well as a second processing unit that determines an object target region including a specified target object target in the second image based on the second image, and determines an object target region including the target object target in the first image based on the determined object target region in the second image, The second processing unit analyzes the second image transformed from the first image captured at the first moment and the second image transformed from the first image captured at a second moment later than the first moment, and tracks the target object included in the target object area of ​​the second image corresponding to the first moment in the second image corresponding to the second moment, The second processing unit determines an object target region including the target object in the first image before being transformed into the second image based on the object target region in the second image, and performs a predetermined image recognition process on the determined object target region in the first image. The predetermined image recognition processing includes a process of recognizing a gesture of the target object based on information of a specific area in the object area in the first image, The processing device further includes a control unit for controlling the moving object based on the gesture recognized by the second processing unit. The extracted specific area is determined according to the positions of the moving object and the user.

2. The processing device according to claim 1, wherein: The second processing unit tracks the target object in the second image based on a change in position of the target object in a time-series second image converted from each of the first images captured in time series.

3. The processing device according to claim 1 or 2, wherein: The target object is a person, The second processing unit tracks the person in the second image based on one or more information of the color of the person's clothes, the person's gender, and the person's hairstyle obtained from the second image transformed from the first image captured in the past or stored in the storage unit.

4. The processing device according to claim 1 or 2, wherein: The prescribed image recognition processing includes a process of determining the specific area based on identifying bones or joint points of the object target area in the first image.

5. The processing device according to claim 4, wherein: The predetermined image recognition processing includes a process of determining a region including an arm or a hand of the target object as the specific region based on the recognition result of the skeleton or the joint point.

6. The processing device according to claim 1 or 2, wherein: The prescribed image recognition processing includes a process of performing gesture recognition by preferentially utilizing information of a specific area in which the degree of change in the motion of the target object target in time series is large among two or more specific areas in the object target area in the first image.

7. The processing device according to claim 6, wherein: The two or more specific areas include at least a specific area determined to include the right arm or right hand of the target object and a specific area determined to include the left arm or left hand of the target object.

8. The processing device according to claim 1 or 2, wherein: The target object is a person, The second processing unit extracts a region including a person's face in the first image, and compares the facial feature values ​​contained in the extracted region with the facial feature values ​​of the person pre-stored in the storage unit to track the person of the target object.

9. The processing device according to claim 1 or 2, wherein: The target object is a person, The second processing unit tracks a person included in the object target area of ​​the second image transformed from the first image captured at the first moment in the second image transformed from the first image captured at a second moment later than the first moment, Based on the tracking result, a region including the tracked arm or hand of the person in the first image captured at the second moment is extracted, and the gesture of the person is recognized based on the extracted region. 10 . A mobile body equipped with the processing device according to claim 1 .

11. A processing method, wherein: The processing method enables the computer to perform the following processing: transforming a first image into a second image having a lower resolution than the first image; and determining an object target region including a specified object target in the second image based on the second image, and determining an object target region including the object target in the first image based on the determined object target region in the second image, analyzing the second image transformed from the first image captured at a first moment and the second image transformed from the first image captured at a second moment later than the first moment, and tracking the target object included in the target object area of ​​the second image corresponding to the first moment in the second image corresponding to the second moment, determining an object target region including the target object in the first image before transformation into the second image based on the object target region of the second image, and performing a prescribed image recognition process on the determined object target region in the first image, The predetermined image recognition processing includes a process of recognizing a gesture of the target object based on information of a specific area in the object area in the first image, Controlling a moving object based on the recognized gestures, The extracted specific area is determined according to the positions of the moving object and the user.

12. A storage medium, wherein: The storage medium stores a program that causes a computer to execute the following processing: transforming a first image into a second image having a lower resolution than the first image; and determining an object target region including a specified object target in the second image based on the second image, and determining an object target region including the object target in the first image based on the determined object target region in the second image, analyzing the second image transformed from the first image captured at a first moment and the second image transformed from the first image captured at a second moment later than the first moment, and tracking the target object included in the target object area of ​​the second image corresponding to the first moment in the second image corresponding to the second moment, determining an object target region including the target object in the first image before transformation into the second image based on the object target region of the second image, and performing a prescribed image recognition process on the determined object target region in the first image, The predetermined image recognition processing includes a process of recognizing a gesture of the target object based on information of a specific area in the object area in the first image, Controlling a moving object based on the recognized gestures, The extracted specific area is determined according to the positions of the moving object and the user.