Information processing device, program and method

Abstract

An information processing device having a control unit which performs tracking using the tracking result of having tracked, using an image inputted in time series, a subject in the image in units of a tracking region that corresponds to a specific portion of the subject. FIG. 1

Description

[Technical Field]

[0001] The present technology relates to an information processing device, a program, and a method. [Background technology]

[0002] There are known techniques for detecting subjects such as human bodies from images. For example, Non-Patent Document 1 listed below discloses a technique for detecting part points of a human body from an image using a convolutional neural network (CNN) and detecting the human body in a bottom-up manner from connection information between the detected part points. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-En Wei, Yaser Sheikh, "Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields", Proceeding of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.7291-7299, 2017 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional subject detection techniques impose a heavy processing load, making it difficult to detect and track a subject in real time (e.g., 30 fps (frames per second)) using the conventional detection techniques.

[0005] An object of the present technology is to provide an information processing device, a program, and a method that can reduce the processing load when detecting and tracking a subject from an image. [Means for solving the problem]

[0006] This technology is via imaging device a control unit that uses images input in a time series manner to track a subject in the image using a tracking result obtained by tracking the subject in units of tracking areas corresponding to specific parts of the subject; The control unit further Extracting components of a subject in the image using the image, and detecting the subject in the image using the extraction result; A predetermined number of frames or more required to detect the subject in the image. Frame The object in the image is detected every few frames. specified Frame Number of images One or more frames Image of and information to counteract the movement of the imaging device. generating correction information including After the subject is detected, When tracking begins , using an image in which the position of the subject in the image included in the correction information has been corrected using information included in the correction information for canceling out the movement of the imaging device. Tracking the subject It is an information processing device.

[0007] This technology is The control unit via imaging device Tracking is performed using tracking results obtained by tracking a subject in an image in units of tracking areas corresponding to specific parts of the subject using images input in a time series; The control unit further Extracting components of a subject in the image using the image, and detecting the subject in the image using the extraction result; A predetermined number of frames or more required to detect the subject in the image. Frame The object in the image is detected every few frames. specified Frame Number of images One or more frames Image of and information to counteract the movement of the imaging device. generating correction information including After the subject is detected, When tracking begins , using an image in which the position of the subject in the image included in the correction information has been corrected using information included in the correction information for canceling out the movement of the imaging device. Tracking the subject The information processing method is a program that causes an information processing device to execute the information processing method.

[0008] This technology is The control unit via imaging device Tracking is performed using tracking results obtained by tracking a subject in an image in units of tracking areas corresponding to specific parts of the subject using images input in a time series; The control unit further Extracting components of a subject in the image using the image, and detecting the subject in the image using the extraction result; A predetermined number of frames or more required to detect the subject in the image. Frame The object in the image is detected every few frames. specified Frame Number of images One or more frames Image of and information to counteract the movement of the imaging device. generating correction information including After the subject is detected, When tracking begins , using an image in which the position of the subject in the image included in the correction information has been corrected using information included in the correction information for canceling out the movement of the imaging device. Tracking the subject It is an information processing method. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a tracking system according to an embodiment of the present technology. [Figure 2] FIG. 2 is a diagram for explaining the human body detection process. [Figure 3] FIG. 3 is a diagram for explaining a specific example of the tracking feature dictionary. [Figure 4] FIG. 4 is a diagram for explaining the human body tracking process. [Figure 5] FIG. 5 is a diagram for explaining the necessity of time delay compensation processing at the start of tracking. [Figure 6] FIG. 6 is a diagram for explaining the time delay compensation process at the start of tracking. [Figure 7] FIG. 7 is a diagram for explaining the movement of a human body in an image when the camera is not moving. [Figure 8] FIG. 8 is a diagram for explaining the movement of a human body in an image when the camera is moving. [Figure 9]FIG. 9 is a diagram for explaining the time delay compensation process according to the camera movement. [Figure 10] FIG. 10 is a diagram for explaining a specific example of the classification feature dictionary. [Figure 11] FIG. 11 is a diagram for explaining the human body identification process. [Figure 12] FIG. 12 is a diagram for explaining the importance determination process. [Figure 13] FIG. 13 is a diagram showing a specific example of an index for determining the importance. [Figure 14] FIG. 14 is a diagram for explaining the resetting of the tracking area. [Figure 15] FIG. 15 is a diagram showing a specific example of a UI for specifying a tracking area. [Figure 16] FIG. 16 is a diagram showing another specific example of the UI for specifying the tracking area. [Figure 17] FIG. 17 is a flowchart showing the overall flow of processing by the arithmetic processing device. DETAILED DESCRIPTION OF THE INVENTION

[0010] The embodiments described below are preferred specific examples of the present technology, and various technically preferable limitations are applied. However, the scope of the present technology is not limited to the following embodiments unless otherwise specified in the following description to the effect that the present technology is limited. The present technology will be described in the following order. <1. Embodiment> [1-1. Tracking system configuration] [1-2. Configuration of the Processing Unit] "Human body detection processing" "Body tracking processing" "Time delay compensation processing at the start of tracking" "Time delay compensation processing according to camera movement" "Human body identification processing" "Importance determination process" "UI for specifying tracking area" [1-3. Overall flow of processing by the processor] <2. Modifications> <3. Application Examples>

[0011] <1. Embodiment> [1-1. Tracking system configuration] Fig. 1 is a block diagram showing an example of the configuration of a tracking system. The tracking system 1 shown in Fig. 1 tracks a human body in an image. The tracking system 1 includes a distance sensor 2, an image sensor 3, an IMU (Inertial Measurement Unit) 4, an input IF (Interface) 5, and a processing unit 6. The distance sensor 2, the image sensor 3, the IMU 4, and the input IF 5 are each connected to the processing unit 6. This connection may be either wired or wireless.

[0012] The distance sensor 2 measures the distance to an object and outputs the measurement result as distance information. The distance sensor 2 is configured, for example, by a ToF (Time Of Flight) or a stereo camera. The image sensor 3 outputs image information captured at a predetermined time interval (for example, 30 fps (frames per second)). The image sensor 3 is configured, for example, by a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The IMU 4 outputs various measurement data (hereinafter referred to as IMU information) such as three-axis angular velocity, acceleration, and self-position. The IMU 4 is configured, for example, by a group of sensors such as a positioning sensor, a gyro sensor, a geomagnetic sensor, and an acceleration sensor.

[0013] The input IF 5 allows the user to input various information as input information, and is configured by, for example, an input device such as a touch panel device in a smartphone, or a mouse, keyboard, etc. in a personal computer. Specifically, the input IF 5 is configured to include a UI (user interface) described later.

[0014] The arithmetic processing device 6, which is an example of a control unit, has a function of executing a tracking program (hereinafter simply referred to as a program). The arithmetic processing device 6 is composed of a processor such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit). By executing this program, the arithmetic processing device 6 processes various pieces of information input from the distance sensor 2, the image sensor 3, the IMU 4, and the input IF 5 in accordance with the program, and outputs the processing results as output information.

[0015] The arithmetic processing device 6, together with the input IF 5, constitutes an information processing device 7. Specifically, the information processing device 7 is constituted by a computer such as a smartphone, a personal computer, or a microcomputer. More specifically, the information processing device 7 is constituted by including, in addition to the input IF 5 and the arithmetic processing device 6, a storage device and an output IF (interface), not shown. The storage device and output IF are each connected to the arithmetic processing device 6. This connection may also be either wired or wireless.

[0016] The storage device is configured to store the above-mentioned programs and various information (for example, information on various dictionaries described below) used in processing by executing the programs. This storage device is configured to include, for example, a ROM (Read Only Memory) and a RAM (Random Access Memory). Note that the programs and various information may be stored in a storage device separate from the storage device constituting the information processing device 7.

[0017] The output IF outputs information based on the processing result of the arithmetic processing unit 6 in the information processing device 7. Specifically, the output IF is configured by an output device such as a display in a smartphone or a personal computer.

[0018] Specifically, in this information processing device 7, the arithmetic processing device 6 reads a program stored in a storage device and executes the read program. The program may be provided or executed via a telecommunications line. By executing the program, the information processing device 7 tracks a human body in an image input in time series, using a tracking result obtained by tracking the human body in the image in units of tracking regions corresponding to specific parts of the human body (e.g., the head, neck, waist, knee centers, etc.). In other words, the program causes the information processing device 7 to execute a process of tracking a human body in an image input in time series, using a tracking result obtained by tracking the human body in the image in units of tracking regions corresponding to specific parts of the human body. Note that the information processing device 7 may include one or more of the distance sensor 2, the image sensor 3, and the IMU 4.

[0019] [1-2. Configuration of the Processing Unit] 1, the arithmetic processing device 6 has, as functional blocks that function by executing the above-mentioned programs, a distance information acquisition unit 11, an image acquisition unit 12, an IMU information acquisition unit 13, a human body detection unit 14, a camera motion estimation unit 15, a human body tracking unit 16, a time delay compensation unit 17, a tracking region determination unit 18, a human body identification unit 19, an importance determination unit 20, a situation determination unit 21, and a human motion estimation unit 22. Note that the functions of the functional blocks of the arithmetic processing device 6 may be realized not only by executing the programs, but also by combining dedicated hardware devices, circuits, etc. that have the functions.

[0020] The distance information acquisition unit 11 acquires distance information from the distance sensor 2 and supplies the acquired distance information to the human motion estimation unit 22. The image acquisition unit 12 acquires image information of the captured video from the imaging sensor 3 (e.g., image information at 30 fps) and supplies the acquired image information to the human body detection unit 14, the camera motion estimation unit 15, and the human body tracking unit 16. The IMU information acquisition unit 13 acquires IMU information from the IMU 4 and supplies the acquired IMU information to the camera motion estimation unit 15.

[0021] The human body detection unit 14 extracts components of a human body in an image using images input in a time series, and detects a human body in the image using the extraction results. The human body detection unit 14 also detects a human body in the image every predetermined number of frames (e.g., 10 frames) that is equal to or greater than the number of frames required to detect a human body in the image (specifically, from the start of detection to the completion of detection). Here, the human body detection process for detecting this human body will be described in detail.

[0022] "Human body detection processing" FIG. 2 is a diagram for explaining the human body detection process in the human body detection unit 14. Specifically, the human body detection unit 14 detects a human body using an image analysis technique that uses deep learning. For example, as shown in the figure, the human body detection unit 14 executes a first CNN (Convolutional Neural Network) process on image information of an image P1 of the entire image or a partially cropped region captured at time t=T1 supplied from the image acquisition unit 12. This first CNN process detects human body parts from the image information using a trained model and estimates their positions. Here, the human body parts refer to parts that make up the human body, such as the skeleton (constituent elements of the human body).

[0023] The human body detection unit 14 estimates the positions of specific parts of the human body (for example, four parts: the head, neck, waist, and knee centers) using the detected human body parts. The estimation results are used in the initialization process of the tracking process described later. Specifically, they are used to set tracking areas (areas indicated by dashed lines in FIG. 2) corresponding to the specific parts of the human body.

[0024] Specifically, the human body detection unit 14 estimates the positions of human body parts as follows: The human body detection unit 14 extracts part points (for example, 19 points: left eye, right eye, nose, left ear, right ear, mouth, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip joint, right hip joint, left knee, right knee, left ankle, and right ankle) for identifying human body parts from the image information. The human body detection unit 14 detects the human body parts based on the extraction results and estimates their positions. Then, when the required number or more of human body parts are detected, it is determined that a human body has been detected.

[0025] If there are multiple people in the image, the human body detection unit 14 detects each person. When it is determined that a human body has been detected, the human body detection unit 14 supplies detection information based on the detection result to the human body tracking unit 16 and the human body identification unit 19. For example, the detection information may include information on the aforementioned human body part points, human body parts, specific human body parts, human body regions (specifically, two-dimensional rectangular positions), etc.

[0026] For example, the human body detection process in the human body detection unit 14 is realized using image analysis technology such as "OpenPose" disclosed in the aforementioned Non-Patent Document 1. The human body detection process in the human body detection unit 14 imposes a heavy processing load, making it difficult to perform real-time processing for each frame. Therefore, as described above, the human body detection unit 14 performs this human body detection process every predetermined number of frames. In other words, the human body detection unit 14 periodically performs the human body detection process even while the human body tracking unit 16 (described later) is tracking a human body, and provides the human body tracking unit 16 with highly reliable detection information actually detected from the image. Note that the identification of human body parts and specific human body regions described above is not limited to this. For example, specific human body regions may be identified directly from feature points.

[0027] 1 estimates the motion of the camera moving between images using the image information supplied from the image acquisition unit 12 and the IMU information supplied from the IMU information acquisition unit 13, and supplies camera motion information based on the estimation result to the time delay compensation unit 17. For example, the camera motion estimation unit 15 may use image analysis processing such as visual odometry to determine the camera motion.

[0028] The human body tracking unit 16 uses images input in a time series to track a human body in the image using tracking results obtained by tracking the human body in units of tracking areas corresponding to specific parts of the human body. The human body tracking unit 16 also tracks the human body in the image for each frame using real-time processing. Note that one or more tracking areas are used for tracking, and the human body tracking unit 16 selects a tracking area to be used for tracking in units of tracking areas from multiple candidates. When tracking is performed using multiple tracking areas, the human body tracking unit 16 tracks the human body in the image based on the tracking results obtained in each tracking area. Here, the human body tracking process for tracking this human body will be described in detail.

[0029] "Body tracking processing" FIG. 3 is a diagram for explaining a specific example of a tracking feature dictionary used in the human body tracking process, and FIG. 4 is a diagram for explaining the human body tracking process in the human body tracking unit 16. The human body tracking unit 16 shown in FIG. 1 tracks the human body detected by the human body detection unit 14. Specifically, the human body tracking unit 16 performs tracking by estimating, using a correlation calculation, where the tracking area in the previous frame has moved to in the next frame. As described above, the human body tracking unit 16 tracks not the entire human body, but each tracking area corresponding to a specific part of the human body, and tracks the human body by integrating the tracking results for each tracking area.

[0030] When tracking a human body, the human body tracking unit 16 first performs initialization processing to set a tracking area for tracking and create a tracking feature dictionary. For example, as shown in FIG. 3, when a human body is detected in image P1 at time t=T1, the human body tracking unit 16 sets a tracking area based on image information, detection information, setting information, etc. Here, the setting information is information indicating which part of the human body a tracking area corresponding to is to be used for tracking. For example, in the example shown in FIG. 3, the human body tracking unit 16 sets four tracking areas corresponding to the specific parts of the human body described above (parts corresponding to the head, neck, waist, and knees, respectively, indicated by dashed lines). For example, as shown in FIG. 3, the tracking area corresponding to the head includes six feature points: the left eye, the right eye, the nose, the left ear, the right ear, and the mouth. Here, a tracking area includes one or more feature points, and the number of tracking areas usable for tracking is set to be smaller than the number of feature points.

[0031] Then, the human body tracking unit 16 executes a second CNN process for each set tracking region. The second CNN process extracts features that determine the likelihood of a specific part of the human body (for example, the likelihood of a head in the case of a head) using a trained model. The human body tracking unit 16 then registers each of the extracted features of the tracking region in a tracking feature dictionary.

[0032] The human body tracking unit 16 performs tracking using this tracking feature dictionary. When tracking a head, for example, the human body tracking unit 16 determines the head center position by taking into account the arrangement of feature points, and since the scale can also be determined by comparing image information, the human body tracking unit 16 tracks the head by correlation from the position and scale. For example, as shown in FIG. 4, the human body tracking unit 16 crops (extracts) a human body region from an image P2 captured at time t=T2 (a time after time t=T1), and executes a second CNN process corresponding to each tracking region at the position of t=T1 to extract each feature. The human body tracking unit 16 then updates the position of the tracking region by passing the feature and the feature in the tracking feature dictionary through a correlation filter. The human body tracking unit 16 then again executes a second CNN process corresponding to each tracking region at the updated position to extract each feature. The human body tracking unit 16 updates the tracking feature dictionary using the extracted feature. The human body tracking unit 16 supplies tracking information based on the tracking results of this human body to the human body identification unit 19. In this way, the human body tracking unit 16 essentially tracks the human body by updating the tracking feature dictionary that integrates the tracking results of the tracking in each tracking area.

[0033] For example, the human body tracking process in this human body tracking unit 16 is realized using a general tracking method such as KCF (Kernelized Correlation Filter). As described above, the human body tracking process in this human body tracking unit 16 performs tracking in units of tracking regions, so the processing load is low, and real-time processing can be performed for each frame even when tracking multiple people simultaneously. Therefore, the human body tracking unit 16 performs this human body tracking process for each frame as described above.

[0034] As described above, the human body detection unit 14 performs human body detection processing every predetermined number of frames and supplies detection information to the human body tracking unit 16. The human body tracking unit 16 performs the initialization processing described above each time. As a result, the human body tracking unit 16 updates the tracking feature dictionary as a check using highly reliable detection information, and uses the tracking feature dictionary to update the tracking feature dictionary described above and correct the tracking position using correction information described below.

[0035] If the human body detection process by the human body detection unit 14 takes a long time, a non-negligible time delay occurs between the frame in which the human body is detected and the frame in which the human body is being tracked. Therefore, a large time delay results in a large difference between the position of the human body detected by the human body detection unit 14 and the position of the human body actually captured when the human body tracking unit 16 starts tracking. Therefore, the human body tracking unit 16 compensates for the time delay at the start of tracking based on correction information supplied from the time delay compensation unit 17 (described later), corrects the human body position, and updates the tracking feature dictionary. Furthermore, the human body tracking unit 16 also compensates for the time delay according to camera movement in the case of a moving camera and according to estimated human body movement, using the correction information supplied from the time delay compensation unit 17. Note that the human body tracking unit 16 may only update the tracking feature dictionary without correcting the human body position.

[0036] 1 performs a compensation process to compensate for human movement from the start of human body detection by the human body detection unit 14 to the completion of detection. Specifically, the time lag compensation unit 17 predicts the current position of the human body using camera movement information supplied from the camera movement estimation unit 15 and movement information supplied from the human movement estimation unit 22. The time lag compensation unit 17 then generates correction information based on the prediction and supplies the generated correction information to the human body tracking unit 16. Here, the time lag compensation process to compensate for this time lag will be described in detail.

[0037] "Time delay compensation processing at the start of tracking" FIG. 5 is a diagram illustrating the necessity of time delay compensation processing at the start of tracking. As described above, the human body detection processing by the human body detection unit 14 is performed every predetermined number of frames (every nine frames in the example shown in FIG. 5). The frame in which the first human body detection processing for a certain person is performed is designated frame F0, and subsequent frames are designated frames F1, F2, F3, and so on. In this case, if time delay compensation processing at the start of tracking is not performed, the human body tracking unit 16 will use images from frame F9 onward, which is the current frame at the start of tracking. Starting tracking from frame F9 as is is equivalent to skipping frames F1 to F8. Because the amount of movement of a person in the image is likely to be large between frames F0 to F9, there is a problem in that the person being tracked is likely to be lost during the human body tracking processing in frame F9. Therefore, the time delay compensation unit 17 performs time delay compensation at the start of tracking as follows.

[0038] FIG. 6 is a diagram illustrating the time lag compensation process at the start of tracking. The time lag compensation unit 17 generates correction information, which causes the human body tracking unit 16 to store intermediate images and perform tracking using the intermediate images, and supplies the correction information to the human body tracking unit 16. For example, as shown in the figure, human body detection is started at frame F0, human body detection is completed at frame F8, and human body tracking is started at frame F9. In this case, the time lag compensation unit 17 causes the human body tracking unit 16 to store frame images during the execution of human body detection (images of frames F3 and F6 in the example shown) as intermediate images. Then, in the human body tracking process at frames F9 and F10 immediately after the start of tracking, the time lag compensation unit 17 causes sequential tracking to be performed using the stored intermediate images (images of frames F3 and F6) instead of the images of frames F9 and F10. In other words, when tracking starts, the human body tracking unit 16 initially performs sequential tracking using the images stored as intermediate images. After tracking has been performed using all of the saved intermediate images (in the example shown, from frame F11 onwards), tracking is performed using the current frame image. This reduces the number of frames skipped at the start of tracking. Reducing the number of frames skipped in this way can reduce the rate of loss immediately after the start of tracking, as mentioned above.

[0039] "Time delay compensation processing according to camera movement" Fig. 7 is a diagram for explaining the movement of a human body in an image when the camera is stationary, and Fig. 8 is a diagram for explaining the movement of a human body in an image when the camera is moving. As shown in Fig. 7, if the camera 30 having the image sensor 3 is a fixed camera or the like and does not move, there is no effect of time delay due to the movement of the camera 30. For example, in the example shown in Fig. 6, the movement of the person in the image is small from frame F0 to frame F3, so the rate of loss is low and it is easy to transition from detection to tracking.

[0040] In contrast, as shown in FIG. 8, when the camera 30 is a moving camera, the effect of the time delay caused by the camera movement becomes greater. For example, assume that the dashed lines in the figure represent the positions of the person and camera 30 in frame F0 in FIG. 6. In this case, if time delay compensation according to the camera movement is not performed, as shown in FIG. 8, when the person and camera 30 move to the position indicated by the solid lines by frame F3, the movement of the person in the image will be greater than the actual movement of the person, making it difficult to transition from detection to tracking. In other words, the rate of loss will increase. Therefore, the time delay compensation unit 17 performs time delay compensation processing according to the camera movement to prevent the movement of the camera 30 from affecting the movement of the person in the image.

[0041] Specifically, the time lag compensation unit 17 generates correction information for causing the human body tracking unit 16 to correct the human body position so as to cancel out the camera movement through time lag compensation processing according to the camera movement, and supplies the correction information to the human body tracking unit 16. More specifically, the time lag compensation unit 17 also stores information regarding the movement of the camera 30 when saving the intermediate images described above. Then, the time lag compensation unit 17 uses the correction information to set the human body detection position used when the human body tracking unit 16 starts tracking to a position where the movement of the camera 30 has been canceled. This can also reduce the rate of loss due to time delays caused by camera movement. This is particularly effective when the movement of the camera 30 is large.

[0042] FIG. 9 is a diagram illustrating the time lag compensation process according to camera motion. For example, in the example shown in FIG. 9, in addition to the time lag compensation process at the start of tracking described above, the following is performed. When the human body tracking unit 16 performs tracking on an image (image of frame F3) saved as an intermediate image, the time lag compensation unit 17 adjusts the position of the human body detection in frame F0 to a position obtained by canceling the movement m1 of the camera 30. In other words, the time lag compensation unit 17 calculates the movement m1 of the camera 30 from frame F0 to frame F3 and moves the detection position of the person in frame F0 in a direction (position) that cancels the movement m1. This reduces the movement of the person in the image, making it easier to transition from detection to tracking. Similarly, when performing tracking on the image of frame F6, the detection position of the person in frame F3 is adjusted to a position obtained by canceling the camera movement m2. Furthermore, when performing tracking on frame F11, the position is adjusted to a position obtained by canceling the camera movement m3. Note that the time lag compensation unit 17 may perform the time lag compensation process according to camera motion at times other than the start of tracking.

[0043] Now, returning to Fig. 1, the tracking area determination unit 18 shown in Fig. 1 determines a tracking area to be used for tracking for each human body based on input information supplied from the input IF 5 or importance information supplied from the importance determination unit 20, and supplies setting information based on the determination to the human body tracking unit 16.

[0044] The human body identification unit 19 identifies whether a new person in the image is a predetermined person based on the state (specifically, posture) represented by multiple body parts of each person. Specifically, the human body identification unit 19 identifies detected human bodies and human bodies being tracked based on the detection information supplied from the human body detection unit 14 and the tracking information supplied from the human body tracking unit 16. The human body identification unit 19 then supplies identification information based on the identification results to the situation determination unit 21 and the human movement estimation unit 22. More specifically, the human body identification unit 19 identifies a newly detected person whose tracking has been interrupted due to occlusion, cut-off, or the like in the image. Here, the human body identification process for identifying this human body will be described in detail.

[0045] "Human body identification processing" FIG. 10 is a diagram illustrating a specific example of a classification feature dictionary used in human body classification processing, and FIG. 11 is a diagram illustrating human body classification processing in the human body classification unit 19. When classifying a human body, the human body classification unit 19 first extracts posture-related features from the human body region being tracked and registers multiple features per person. For example, as shown in FIG. 10, the human body classification unit 19 crops the human body region of the tracked person A from image P1 at time t=T1, performs third CNN processing, and registers feature X for a forward-facing posture. Note that the third CNN processing extracts features for determining the posture of the human body using a trained model.

[0046] Similarly, the human body identification unit 19 crops the human body region of the same tracked person A from the image Pn at time t=Tn (later than time t=T1), performs a third CNN process, and registers a feature Y for the sideways posture. In this way, the human body identification unit 19 accumulates the feature amounts X, Y,... related to various postures of each person in the identification feature dictionary. Note that each feature amount X, Y,... is registered in association with a human body identification ID that can identify each human body.

[0047] In this state, human body identification processing is performed by matching with a classification feature dictionary. For example, as shown in FIG. 11, assume that a new person M appears at time t = Tm after the person being tracked leaves the frame. In this case, the human body identification unit 19 crops the human body region from the image Pm captured at time t = Tm and performs a third CNN process to extract features. The human body identification unit 19 then matches the extracted features with features registered in the classification feature dictionary to determine whether or not the person M is the person being tracked (human body identification). The human body identification unit 19 then supplies identification information based on the identification result to the situation determination unit 21 and the human movement estimation unit 22.

[0048] The timing of registration (accumulation) in the classification feature dictionary by the human body identification unit 19 can be set as appropriate, for example, every time tracking is performed in each frame image, but it is preferable to perform the registration at the timing of detection of human body parts by the human body detection unit 14. This makes it possible to accurately determine the posture from the positions of the human body parts shown in Fig. 2, and to accumulate multiple highly accurate features for each posture.

[0049] Returning to FIG. 1 again, the importance determination unit 20 shown in FIG. 1 determines the setting of a tracking area to be used for tracking in units of tracking areas according to the importance of tracking based on a predetermined index. Specifically, the importance determination unit 20 determines the importance based on situation information supplied from the situation determination unit 21 and input information supplied from the input IF 5, and supplies the importance information based on the determination result to the tracking area determination unit 18. The situation determination unit 21 determines various tracking situations and supplies the situation information based on the determination result to the importance determination unit 20. Specifically, the situation determination unit 21 understands the situation from the attributes of the person (human body) to be tracked, the relationship between the human body being tracked and the background, etc., based on the identification information supplied from the human body identification unit 19, and supplies the result of the understanding to the importance determination unit 20 as situation information. The process of determining the importance (importance determination process) performed by the importance determination unit 20 using this situation information and input information will be described in detail later.

[0050] The human movement estimation unit 22 generates output information based on the distance information supplied from the distance information acquisition unit 11 and the identification information supplied from the human body identification unit 19, and outputs the generated output information. Specifically, the human movement estimation unit 22 estimates the current position of the human body being tracked (for example, the position in a two-dimensional image or a position in three-dimensional space) using each feature in the tracking feature dictionary, and outputs the estimation result as output information in association with a tracking ID.

[0051] The output information output from the human motion estimation unit 22 is specifically used in the output IF described above. Specifically, the output IF displays information about the position of the human body being tracked based on this output information on a display. For example, the information about the human body position may include information representing the two-dimensional rectangular position of the human body and the tracking area used for tracking. Note that this output information may also be used for tracking a specific person using a moving camera, specifically, for camera movement for tracking. Furthermore, the human motion estimation unit 22 estimates the motion of the human body based on the identification information and distance information described above, and supplies motion information based on the estimation result to the time delay compensation unit 17. This motion information is used to generate correction information for time delay compensation according to the human body motion estimation by the time delay compensation unit 17.

[0052] "Importance determination process" Here, the importance determination process mentioned above will be explained. At the start of tracking, the importance determination unit 20 sets the setting of the tracking area for tracking (specifically, at least one of the number and location) to a predetermined default setting. That is, the importance determination unit 20 supplies information representing the default setting as importance information to the tracking area determination unit 18. After the start of tracking, the importance determination unit 20 also determines the importance of tracking, and supplies importance information representing the setting of the tracking area according to the determination result to the tracking area determination unit 18. When tracking multiple people, the importance determination unit 20 determines the importance of each person. In this way, the importance determination unit 20 changes the setting of the tracking area for tracking according to the determined importance.

[0053] Fig. 12 is a diagram for explaining the importance determination process. For example, in the example of tracking three persons A, B, and C shown in Fig. 12, the importance determination unit 20 sets (default setting) that all persons use two tracking regions, that is, the head and the waist, at the start of tracking. Then, if the importance determination unit 20 determines that the importance of person C is high after tracking starts, it sets the tracking region of person C to four regions, that is, the head, neck, waist, and knees. Note that the tracking regions of human bodies A and B, which have normal importance, remain two regions.

[0054] In this way, by setting a large number of tracking areas to track human bodies determined to be of high importance and a small number of tracking areas to track human bodies determined not to be of high importance, it is possible to effectively track human bodies while suppressing the overall processing load. In other words, for human bodies whose tracking importance is high, the number of tracking areas used for tracking is increased to improve robustness. Note that the settings may be changed not only when tracking multiple people but also when tracking only one person, and the settings of the locations of the tracking areas may be changed depending on the importance. The number and locations of the tracking areas are not limited to the example shown in the figure.

[0055] FIG. 13 is a diagram showing a specific example of an index for determining importance. As shown in FIG. 13, the importance determination unit 20 determines importance using indices such as the attributes of a person (human body), the background of the person, CPU load, and information about the person's movements. When the importance determination unit 20 uses the attributes of a person as an index, it determines the importance according to the person's attributes, such as gender and age. For example, as shown in the figure, the importance determination unit 20 determines the attributes of a person based on a gender-related parameter called "Gender" expressed as a numerical value from 0 to 100 (0 to 50 is male, and 50 to 100 is female) and an age-related parameter called "Age" expressed as a numerical value from 0 to 100 (estimated age value). For example, if a setting is made to focus on tracking children, the importance determination unit 20 sets the tracking area to be larger when the person being tracked is a child, and sets the tracking area to be smaller when the person being tracked is an adult. For example, these parameters can be extracted using machine learning.

[0056] Furthermore, when the background is used as an index, the importance determination unit 20 determines the importance according to the similarity between the person and the background. For example, as shown in the figure, the importance determination unit 20 determines the similarity using a parameter related to the similarity score obtained from the correlation during tracking, called "Parts Score." If the score is low, the person may resemble the background. For example, a person who is present in a background that is easily mistaken for a person (e.g., a background of the same color as the person's clothes, a complex background, etc.) is considered difficult to track. In such a case, the importance determination unit 20 determines that the similarity is high and allocates a larger tracking area.

[0057] The importance determination unit 20 may use the CPU load as an index and determine the importance according to the CPU load. For example, as shown in the figure, the importance determination unit 20 determines the overall CPU load using a parameter related to the CPU load of the entire system called "CPU Load." For example, the importance determination unit 20 allocates more tracking areas when the CPU load is equal to or less than a threshold, and reduces the number of tracking areas when the CPU load exceeds the threshold, such as when the CPU load increases due to other processing.

[0058] Furthermore, the importance determination unit 20 may use the person's movement information as an index and determine the importance according to the magnitude of the person's movement (human body movement information). For example, as shown in the figure, the importance determination unit 20 determines the magnitude of the person's movement based on a parameter called "Velocity" that represents the speed at which the person is moving in three-dimensional space and a parameter called "Pose Velocity" that represents the magnitude of a change in posture or a relative change in the position of a human body part. For example, the importance determination unit 20 determines the magnitude of movement based on the amount of movement of the human body part in this way, and since people who move a lot (for example, children) are difficult to track, the importance determination unit 20 tracks them using a large number of tracking areas.

[0059] FIG. 14 is a diagram for explaining the resetting of tracking areas. Suppose a person who has been tracked using tracking areas according to importance leaves the frame and reappears in the frame. In this case, the importance determination unit 20 performs tracking using default settings at the start of tracking, and when it is identified that the person is person C being tracked, sets tracking to be performed again using tracking areas according to importance. In the example shown in FIG. 14, person C, who has a high level of importance and has been tracked in four areas, head, neck, waist, and knees, leaves the frame and reappears in the frame. In this case, when tracking starts, first, two tracking areas, head and waist, are set (default settings), and when it is identified that the person who has appeared in the frame is person C, four tracking areas are set again.

[0060] The tracking area for tracking is automatically set by the importance determination unit 20, but the setting of the tracking area can also be changed by input information from the user. In other words, the user can specify the setting himself / herself. Specifically, the specification of this tracking area is performed via the UI of the input IF5 described above. The UI used to specify this tracking area will be described below.

[0061] "UI for specifying tracking area" FIG. 15 is a diagram showing a specific example of a UI for specifying a tracking area. Specifically, the UI for specifying a tracking area is configured by a touch panel device (touch panel display) incorporated into a display serving as an output IF. A camera image is displayed on the display. The output IF also displays part icons 50 that can be operated by the user using the touch panel device, superimposed on the camera image. The part icons 50 represent the setting of the tracking area for a person in the camera image, and are icons that allow the user to change the setting of the tracking area.

[0062] Specifically, the UI for specifying the tracking area is configured so that the user can control the setting of the tracking area by dragging and dropping the part icons 50 superimposed on the human body being tracked. That is, operation information of the part icons 50 is supplied as input information from the input IF 5 shown in Fig. 1 to the tracking area determination unit 18. The tracking area determination unit 18 generates setting information based on this input information and supplies it to the human body tracking unit 16, and sets the tracking area for tracking to match the display of the part icons 50.

[0063] 15, for example, of the four part icons 50 of the head, neck, waist, and knees superimposed on human body C, the knee part icon 50 is moved by drag and drop by the user to the knee part of human body B, which is superimposed on the two part icons 50 of the head and waist. In this way, the information processing device 7 has a UI for specifying the tracking area, allowing the user to change the settings of the tracking area easily and intuitively through the UI.

[0064] Fig. 16 is a diagram showing another specific example of a UI for specifying a tracking area. In the example shown in Fig. 16, a display window W separate from the camera image is displayed on the display, and a human body in the camera image is extracted and displayed within the display window W. Part icons 50 are then superimposed on the human body within the display window W, and the setting of the tracking area can be controlled by dragging and dropping the part icons 50 within the display window W. In this way, by providing a display window W for controlling the setting of the tracking area, it is possible to prevent the part icons 50 from obscuring the camera image.

[0065] 15 and 16, the part icon 50 is a circle surrounded by a dashed line at a location on the human body that corresponds to the tracking area, but the shape and size of the part icon 50 are not limited to this. For example, the part icon 50 may be surrounded by various shapes or represented by a dot. Furthermore, the UI for specifying the tracking area is not limited to the one using the touch panel device described above, and the part icon 50 may be moved using a mouse or keyboard.

[0066] [1-3. Overall flow of processing by the processor] Fig. 17 is a flowchart showing the overall flow of processing by the arithmetic processing device 6. The processing shown in Fig. 17 is performed for each frame. First, when the processing starts, camera motion estimation is performed (step S1). Specifically, as described above, the camera motion estimation unit 15 shown in Fig. 1 generates camera motion information using image information, IMU information, etc., and supplies it to the time delay compensation unit 17.

[0067] Next, it is determined whether the frame being processed is a frame for human body detection (step S2), and if it is determined that the frame is a frame for human body detection (YES), human body detection processing is performed (step S3). Specifically, the human body detection unit 14 shown in Fig. 1 performs human body detection processing on the image information to detect a human body, and supplies detection information based on the detection result to the human body tracking unit 16 and the human body identification unit 19.

[0068] If a human body is detected in the human body detection process of step S3, a human body identification process is performed (step S4), and it is determined whether the detected human body is a human body that has been tracked in the past (step S5). Specifically, the human body identification unit 19 shown in Fig. 1 performs the human body identification process using the detection information, etc., and supplies identification information based on the processing result to the situation determination unit 21 and the human movement estimation unit 22.

[0069] Next, if it is determined in step S5 that the human body has been tracked in the past (YES), a time lag compensation process is performed according to the estimated human body motion (step S6). Specifically, the time lag compensation unit 17 shown in Fig. 1 generates correction information based on the motion information and the like, and supplies it to the human body tracking unit 16.

[0070] Then, in step S6, a time delay compensation process is performed according to the estimated human body motion, and then the human body ID is reassigned (step S7). By reassigning the human body ID, the tracking area for tracking is set to correspond to the reassigned human body ID.

[0071] After the process in step S7, or if it is determined in step S2 that the frame is not one for which human body detection is to be performed (NO), human body tracking processing is performed (step S8), and the tracking feature dictionary used in the human body tracking processing is updated (step S9). Specifically, the human body tracking unit 16 shown in FIG. 1 performs human body tracking processing using image information, identification information, setting information, correction information, etc., and supplies tracking information based on the processing result to the human body identification unit 19, while updating the tracking feature dictionary. Note that in this human body tracking processing, the above-mentioned time delay compensation processing at the start of tracking and time delay compensation processing according to camera movement are performed as necessary. Specifically, the time delay compensation unit 17 shown in FIG. 1 generates correction information based on camera movement information, etc., and supplies it to the human body tracking unit 16.

[0072] On the other hand, if it is determined in step S5 that the human body has not been tracked in the past (NO), a new human body tracking process is started by the human body tracking unit 16 shown in FIG. 1 (step S10), and a new tracking feature dictionary for the human body detected in the human body detection process is created (step S11).

[0073] After updating the tracking feature dictionary in step S9 or creating the human body tracking feature dictionary in step S11, features for human body identification are registered in the identification feature dictionary used in the human body identification process (step S12). Specifically, the human body identification unit 19 shown in Fig. 1 accumulates features of the human body being tracked in the identification feature dictionary using a human body identification ID or the like.

[0074] Then, output information based on the tracking result is supplied to the output IF (step S13). Specifically, the human motion estimation unit 22 shown in Fig. 1 generates output information such as the two-dimensional rectangular position of the human body being tracked based on the information of the feature amounts stored in the tracking feature dictionary, and supplies the output information to the output IF.

[0075] Next, the importance determination unit 20 shown in Fig. 1 performs importance determination processing (step S14), and the tracking area determination unit 18 determines a tracking area according to the importance (step S15), and the processing ends. Specifically, the situation determination unit 21 generates situation information based on the identification information and supplies it to the importance determination unit 20. The importance determination unit 20 generates importance information based on the input information, situation information, etc., and supplies it to the tracking area determination unit 18. Then, the tracking area determination unit 18 generates setting information based on the importance information or the input information, and supplies it to the human body tracking unit 16. This setting information is used in tracking in the next frame processing.

[0076] 17, and can be changed as appropriate as long as similar processing is possible. For example, the processing of steps S13 and S14 does not necessarily have to be performed after tracking (after steps S8 and S10) as described above, but may be performed before tracking (before steps S8 and S10).

[0077] An information processing device 7 according to an embodiment of the present technology uses images input in a time series to track human bodies in the images, using tracking results obtained by tracking human bodies in units of tracking regions corresponding to specific parts of the human body. For example, when tracking the entire human body, it is necessary to track all of the above-mentioned part points, which results in a very large number of tracking points. In contrast, the information processing device 7 performs tracking using a smaller number of tracking regions than the number of part points. Therefore, the processing load can be reduced compared to tracking the entire human body. This makes it possible to track human bodies (e.g., tracking multiple people) detected in real time (e.g., 30 fps) on a mobile device, for example.

[0078] Furthermore, tracking in units of tracking areas enables tracking with high accuracy that is robust to changes in appearance, posture, etc., compared to tracking the entire human body. Furthermore, as the number of people to be tracked and the number of tracking areas increase, the processing load increases, but since the number of tracking areas can be adjusted according to the importance based on the aforementioned index, the processing load can be reduced by making appropriate settings.

[0079] <2. Modifications> Although the embodiments of the present technology have been specifically described above, the present technology is not limited to the above-described embodiments, and various modifications based on the technical ideas of the present technology are possible. For example, various modifications such as those described below are possible. Furthermore, one or more of the following modification aspects can be arbitrarily selected and combined as appropriate. Furthermore, the configurations, methods, processes, shapes, materials, and numerical values ​​of the above-described embodiments can be combined with each other as long as they do not deviate from the spirit and scope of the present technology.

[0080] In the above-described embodiment, the subject to be tracked is a human body, but it is not limited to a human body and may be any other subject from which components can be extracted (for example, other living things such as animals or moving objects such as cars). For example, when a car is to be tracked, windows, doors, tires, etc. can be set as the tracking region.

[0081] In the above-described embodiment, a human body is tracked in image information supplied in real time from the image sensor 3, but the images for tracking the human body are not limited to this, and the human body may also be tracked in a recorded image.

[0082] In the embodiment described above, the human body detection unit 14 performs human body detection every predetermined number of frames, but the timing of human body detection is not limited to this. For example, human body detection may be performed at any timing after the first human body detection as a check on the result, or only the first human body detection may be performed and the check on the result may be omitted.

[0083] In the embodiment described above, the human body tracking unit 16 tracks human bodies for each frame, but the timing of human body tracking is not limited to this. Human body tracking may be performed at any timing that allows tracking at a shorter cycle than the cycle at which human body detection can be performed. For example, human body tracking may be performed once every two frames for a 60 fps image.

[0084] In the above-described embodiment, a case has been described in which human body identification is performed when a new human body is detected, and if that person is the human body being tracked, the tracking area is returned to the previous setting, but the purpose of human body identification is not limited to this. For example, an arbitrary human body designated by a user or the like may be tracked, and human body identification may be performed to determine whether or not that human body is the detected human body.

[0085] <3. Application Examples> Next, an electronic device to which the information processing device according to the above-described embodiment, modification, etc. is applied will be described. The information processing device is suitable for use in grasping the movement of a subject in an image. Therefore, the information processing device can be applied to electronic devices such as controller devices for mobile objects such as drones, camera devices such as surveillance cameras and in-vehicle cameras, robots such as security robots and monitoring robots, and game consoles.

[0086] The present technology can also be configured as follows. (1) The image processing device includes a control unit that uses images input in a time series to track a subject in the image using tracking results obtained by tracking the subject in units of tracking areas corresponding to specific parts of the subject. Information processing device. (2) The control unit extracts components of a subject in the image using the image, and detects the subject in the image using the extraction result. An information processing device according to (1). (3) The control unit detects the subject in the image for each predetermined number of frames equal to or greater than the number of frames required to detect the subject in the image. (2) An information processing device according to the present invention. (4) The control unit tracks the subject in the image in real time for each frame. An information processing device according to any one of (1) to (3). (5) The control unit performs tracking in units of one or more tracking areas, and when tracking is performed in units of the tracking areas using a plurality of tracking areas, tracks the subject in the image based on the tracking results of the tracking in each tracking area. An information processing device according to any one of (1) to (4). (6) The control unit selects a tracking area to be used for tracking in the tracking area unit from a plurality of candidates. An information processing device according to any one of (1) to (5). (7) The control unit determines a setting of a tracking area to be used for tracking in units of tracking areas according to the importance of tracking based on a predetermined index. An information processing device according to any one of (1) to (6). (8) The predetermined index includes at least one of the attribute of the subject, the background of the subject, the CPU load, or the magnitude of the motion of the subject. (7) An information processing device according to (7). (9) The control unit changes the setting of the tracking area used for tracking in units of tracking areas based on input information from a user. An information processing device according to any one of (1) to (8). (10) The control unit identifies whether a new object in the image is a predetermined object based on a state represented by a plurality of components of each object. An information processing device according to any one of (1) to (9). (11) The control unit detects a subject in the image using the image, and when tracking the subject in the image, performs a compensation process to compensate for movement of the subject from the start of detection of the subject in the image to the completion of detection. An information processing device according to any one of (1) to (10). (12) The control unit Using images input in a time series, a subject in the image is tracked in units of tracking regions corresponding to specific parts of the subject, and the tracking results are used to track the subject. A program that causes an information processing device to execute an information processing method. (13) An icon that allows a user to change the setting of the tracking area used for tracking in units of tracking areas is displayed on the display on which the image is displayed. The program according to (12) above, which causes the information processing device to execute the process. (14) The control unit Using images input in a time series, a subject in the image is tracked in units of tracking regions corresponding to specific parts of the subject, and the tracking results are used to track the subject. Information processing methods. [Explanation of symbols]

[0087] 1 Tracking system, 5 Input IF, 6 Processing unit, 7 Information processing unit, 11 Distance information acquisition unit, 12 Image acquisition unit, 13 IMU information acquisition unit, 14 Human body detection unit, 15 Camera motion estimation unit, 16 Human body tracking unit, 17 Time delay compensation unit, 18 Tracking area determination unit, 19 Human body identification unit, 20 Importance determination unit, 21 Situation judgment unit, 22 Human motion estimation unit, 50 Parts icon

Claims

1. A control unit that uses images input in time series via an imaging device to track a subject in the images using tracking results obtained by tracking the subject in units of tracking areas corresponding to specific parts of the subject, The control unit further extracting components of an object in the image using the image, and detecting the object in the image using the extraction result; detecting the subject in the image for each of a predetermined number of frames equal to or greater than the number of frames required for detecting the subject in the image; generating correction information including one or more frame images among the predetermined number of frame images and information for canceling out the movement of the imaging device; When tracking of the subject is started after the subject is detected, the subject is tracked using an image in which the position of the subject in the image included in the correction information is corrected using information for canceling out the movement of the imaging device, which is included in the correction information. Information processing device.

2. The control unit tracks the subject in the image in real time for each frame. The information processing device according to claim 1 .

3. The control unit performs tracking in units of one or more tracking areas, and when tracking in units of the tracking areas is performed using a plurality of tracking areas, tracks the subject in the image based on the tracking results of the tracking in each tracking area. The information processing device according to claim 1 .

4. The control unit selects a tracking area to be used for tracking in the tracking area unit from a plurality of candidates. The information processing device according to claim 1 .

5. The control unit determines a setting of a tracking area to be used for tracking in units of tracking areas according to the importance of tracking based on a predetermined index. The information processing device according to claim 1 .

6. The predetermined index includes at least one of the attribute of the subject, the background of the subject, the CPU load, or the magnitude of the motion of the subject. The information processing device according to claim 5 .

7. The control unit changes the setting of the tracking area used for tracking in units of tracking areas based on input information from a user. The information processing device according to claim 1 .

8. The control unit identifies whether a new object in the image is a predetermined object based on a state represented by a plurality of components of each object. The information processing device according to claim 1 .

9. a control unit that uses images input in a time series manner via an imaging device to track a subject in the images using a tracking result obtained by tracking the subject in units of tracking areas corresponding to specific parts of the subject; The control unit further extracting components of an object in the image using the image, and detecting the object in the image using the extraction result; detecting the subject in the image for each of a predetermined number of frames equal to or greater than the number of frames required for detecting the subject in the image; generating correction information including one or more frame images among the predetermined number of frame images and information for canceling out the movement of the imaging device; When tracking of the subject is started after the subject is detected, the subject is tracked using an image in which the position of the subject in the image included in the correction information is corrected using information for canceling out the movement of the imaging device, which is included in the correction information. A program that causes an information processing device to execute an information processing method.

10. An icon that allows a user to change the setting of the tracking area used for tracking in units of tracking areas is displayed on the display on which the image is displayed. The program according to claim 9 , which causes the information processing device to execute processing.

11. a control unit that uses images input in a time series manner via an imaging device to track a subject in the images using a tracking result obtained by tracking the subject in units of tracking areas corresponding to specific parts of the subject; The control unit further extracting components of an object in the image using the image, and detecting the object in the image using the extraction result; detecting the subject in the image for each of a predetermined number of frames equal to or greater than the number of frames required for detecting the subject in the image; generating correction information including one or more frame images among the predetermined number of frame images and information for canceling out the movement of the imaging device; When tracking of the subject is started after the subject is detected, the subject is tracked using an image in which the position of the subject in the image included in the correction information is corrected using information for canceling out the movement of the imaging device, which is included in the correction information. Information processing methods.

Images