Information processing device, control method for information processing device, imaging device, information processing system, program and storage medium

Abstract

This technology provides the ability to more reliably and continuously track the subject being tracked. [Solution] The information processing device according to the present disclosure includes an acquisition means for acquiring the positions and feature quantities of a plurality of subjects detected from a captured image, and a tracking means for associating any of the subjects with a tracking subject, which is the subject to be tracked, based on the positions and feature quantities of each of the plurality of subjects acquired by the acquisition means. The tracking means performs a first association by associating any of the subjects with the tracking subject using the similarity of feature quantities when the degree of overlap between the tracking subject and other subjects is a first value, and performs a second association by associating any of the subjects with the tracking subject using the similarity of positions when the degree of overlap between the tracking subject and other subjects is a second value indicating less overlap than the first value.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus using a tracking technique for a tracking subject, a control method for the information processing apparatus, a photographing apparatus, an information processing system, a program, and a storage medium.

Background Art

[0002] Generally, in a camera called a PTZ camera capable of adjusting pan, tilt, and zoom, a technique of detecting a subject to be tracked (hereinafter referred to as a tracking subject) specified by a user from a captured image and performing tracking photography is known. In this tracking technique, the pan, tilt, and zoom of the camera are automatically controlled so as to continuously capture the tracking subject. Also, in this tracking technique, even when the tracking subject moves, tracking photography is possible by continuously determining that the tracking subject after the movement is the same subject as the tracking subject before the movement.

[0003] Also, in Non-Patent Document 1, an algorithm has been proposed for determining whether or not it is the same subject by using feature amounts related to the appearance of a tracking subject when the tracking subject is hidden by another subject or object and appears again within the imaging angle.

Prior Art Documents

Non-Patent Documents

[0004]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the technology described in Non-Patent Document 1, if the tracked subject is partially obscured by another subject or object, the calculation of the feature quantities of the tracked subject's appearance may become unstable. As a result, a problem arises in which the tracked subject may not be determined to be the same subject before and after its movement.

[0006] This invention has been made in view of the above problems and aims to provide a technology that enables more reliable and continuous tracking of a subject. [Means for solving the problem]

[0007] The information processing device according to this disclosure comprises: an acquisition means for acquiring the position and feature quantities of each of a plurality of subjects detected from a captured image; and a tracking means for tracking one of the plurality of subjects based on the position and feature quantities of each of the plurality of subjects acquired by the acquisition means, wherein the tracking means performs a first association by associating one of the plurality of subjects with the tracking subject when the degree of overlap between the tracking subject and other subjects is a first value, and performs a second association by associating one of the plurality of subjects with the tracking subject when the degree of overlap between the tracking subject and other subjects is a second value indicating less overlap than the first value, using the similarity of positions.

[0008] Furthermore, the control method for the information processing device relating to this disclosure includes an acquisition step of acquiring the respective positions and feature quantities of multiple subjects detected from a captured image, and the acquisition step of The control method for an information processing device is characterized by comprising: a tracking step of tracking one of the multiple subjects by associating it with a tracking subject that is the target of tracking, based on the position and feature quantities of each of the multiple subjects, wherein the tracking step is characterized by performing a first association by associating one of the multiple subjects with the tracking subject using the similarity of the feature quantities when the degree of overlap between the tracking subject and other subjects is a first value, and performing a second association by associating one of the multiple subjects with the tracking subject using the similarity of the positions when the degree of overlap between the tracking subject and other subjects is a second value that indicates less overlap than the first value.

[0009] Furthermore, the present disclosure is a photographic apparatus that controls the shooting direction horizontally or vertically by a rotation mechanism, and comprises: acquisition means for acquiring the position and feature quantities of each of a plurality of subjects detected from a captured image; tracking means for tracking any of the plurality of subjects in association with a tracking subject that is the target of tracking, based on the position and feature quantities of each of the plurality of subjects acquired by the acquisition means; determination means for determining a control value for controlling the rotation mechanism in order to photograph the tracking subject; and control means for controlling the rotation mechanism based on the control value, wherein the tracking means performs a first association by using the similarity of the feature quantities to associate any of the plurality of subjects with the tracking subject when the degree of overlap between the tracking subject and other subjects is a first value, and performs a second association by using the similarity of positions to associate any of the plurality of subjects with the tracking subject when the degree of overlap between the tracking subject and other subjects is a second value indicating less overlap than the first value.

[0010] Furthermore, the information processing system relating to this disclosure is an information processing system having an imaging device and an information processing device that control the imaging direction horizontally or vertically by a rotation mechanism, wherein the information processing device includes an acquisition means for acquiring the position and feature quantities of each of a plurality of subjects detected from an image, a tracking means for tracking one of the plurality of subjects in association with a tracking subject based on the position and feature quantities of each of the plurality of subjects acquired by the acquisition means, a determination means for determining a control value for controlling the rotation mechanism in order to photograph the tracking subject, and an output of the determined control value to the imaging device. The information processing system is characterized by having an output means for recording, the imaging device having a control means for controlling the rotation mechanism based on the outputted control value, and the tracking means performing a first correspondence to associate any of the multiple subjects with the tracking subject using the similarity of the feature quantities when the degree of overlap between the tracking subject and other subjects is a first value, and performing a second correspondence to associate any of the multiple subjects with the tracking subject using the similarity of the positions when the degree of overlap between the tracking subject and other subjects is a second value indicating less overlap than the first value. [Effects of the Invention]

[0011] According to the technology disclosed herein, an information processing device can more reliably and continuously track a subject being tracked. [Brief explanation of the drawing]

[0012] [Figure 1] This diagram schematically shows an example of the configuration of the information processing system according to the first embodiment. [Figure 2] Block diagram showing an example configuration of the camera and controller according to the first embodiment. [Figure 3] Functional block diagram of the camera and controller according to the first embodiment. [Figure 4A] Flowchart of the process performed by the camera according to the first embodiment [Figure 4B]Flowchart of the process executed by the controller according to the first embodiment [Figure 5] Diagram showing an example of the display of the photographed image and the detection result according to the first embodiment [Figure 6] Flowchart of the recognition process executed by the camera according to the first embodiment [Figure 7] Flowchart of the calculation of the quality of feature amounts executed by the camera according to the first embodiment [Figure 8] Flowchart of the subject tracking process executed by the camera according to the first embodiment [Figure 9] Diagram for explaining the calculation process of the quality of feature amounts in the first embodiment [Figure 10] Diagram for explaining the subject tracking process in the first embodiment [Figure 11] Diagram for explaining the subject collation process in the first embodiment [Figure 12] Diagram for explaining the feature amount extraction process in the second embodiment

Mode for Carrying Out the Invention

[0013] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0014] [First Embodiment] Hereinafter, an information processing system according to the first embodiment will be described. As shown in FIG. 1, the information processing system 1 according to the present embodiment includes a camera 100 and a controller 200 which is a control device of the camera 100. The camera 100 and the controller 200 are connected to each other via a network 400. Thereby, the information processing system 1 according to the present embodiment is configured such that the camera 100 and the controller 200 can perform data communication with each other via the network 400. The network 400 includes networks such as a LAN (Local Area Network) and the Internet.

[0015] <Configuration of Each Device> Next, an example of the hardware configuration of each of the camera 100 and the controller 200 will be described using the block diagram of FIG. 2. Note that the configuration shown in FIG. 2 is merely an example of the hardware configuration of the camera 100 and the controller 200, and can be changed and / or modified as appropriate.

[0016] First, an example of the hardware configuration of the camera 100 will be described. The camera 100 is an information processing device having a mechanism capable of performing a pan operation and a tilt operation for changing the shooting direction by rotating the device itself, and is a shooting device. Further, the camera 100 detects a subject from the captured image and changes the shooting direction based on the detection result of the subject.

[0017] The CPU 101 executes various processes using computer programs and data stored in the RAM (Random Access Memory) 102. Thereby, the CPU (Central Processing Unit) 101 controls the overall operation of the camera 100 and controls the execution of various processes described as processes performed by the camera 100.

[0018] RAM 102 is a high-speed storage device such as DRAM (Dynamic Random-Access Memory). RAM 102 has an area for storing computer programs and data loaded from ROM (Read-Only Memory) 103 and storage device 119. RAM 102 also has an area for storing captured images output from the image processing unit 106. Furthermore, RAM 102 has an area for storing various information received from the controller 200 via the network interface 105, and an area used by the CPU 101 and inference execution unit 110 when executing various processes. In this way, RAM 102 appropriately provides areas for executing various processes in the camera 100.

[0019] ROM103 contains the camera 100's settings data and the computer related to starting up camera 100. The ROM 103 stores computer programs and data related to the basic operation of the camera 100, etc. It also stores computer programs and data for the CPU 101 and inference execution unit 110 to execute or control various processes described as being performed by the camera 100.

[0020] Network I / F 105 is an interface for connecting to the network 400 mentioned above, and is responsible for communication with external devices such as the controller 200 via a communication medium such as ETHERNET®. Note that a separate serial communication interface may also be used for communication.

[0021] The image processing unit 106 converts the video signal output from the image sensor 107 into a captured image, which is data in a predetermined format. The captured image generated by this conversion is then compressed as needed before being output to the RAM 102. The image processing unit 106 may also perform various processing on the image represented by the video signal acquired from the image sensor 107, such as color correction, exposure correction, sharpness correction, and cropping to extract only a predetermined area. These processing may also be performed according to instructions received from the controller 200 via the network interface 105.

[0022] The image sensor 107 receives light reflected from the subject, converts the brightness and color of the received light into an electric charge, and outputs a video signal based on the result of this conversion. For example, the image sensor 107 can be a photodiode, a CCD (Charged Coupled Device) sensor, a CMOS (Complementary Metal Oxide Semiconductor) sensor, etc.

[0023] The drive I / F 108 is an interface for sending and receiving instruction signals, such as control signals, to and from the drive unit 109. The drive unit 109 is a rotation mechanism for changing the shooting direction of the camera 100 and includes a mechanical drive system and a motor as a drive source. The drive unit 109 performs pan and tilt operations to change the shooting direction horizontally and vertically, and zoom operations to change the shooting angle optically, according to instructions received from the CPU 101 via the drive I / F 108.

[0024] The inference execution unit 110 performs inference processing to estimate the presence or absence of subjects in the captured image and the position (region) of each subject, and processing to extract appearance features of the captured image included in the estimated position. The inference execution unit 110 uses, for example, a GPU (Graphics Processing Unit). This is a computing device specialized for image processing and inference processing, such as an inference execution unit (GPU). Generally, the use of a GPU is effective for the inference processing performed by the inference execution unit 110. However, processing equivalent to the inference processing may be realized by a reconfigurable logic circuit such as an FPGA (Field Programmable Gate Array). Alternatively, the CPU 101 may be responsible for the processing of the inference execution unit 110.

[0025] The storage device 119 is a non-volatile storage device such as flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), or SD (Secure Digital) card. Computer programs and data, such as the OS (Operating System), are stored in the storage device 119. The storage device 119 is also used as a temporary storage area for various types of data. Furthermore, some or all of the computer programs and data stored in the ROM 103 may also be stored in the storage device 119.

[0026] The CPU 101, RAM 102, ROM 103, network I / F 105, image processing unit 106, drive I / F 108, inference execution unit 110, and storage device 119 are each system components. S111 is connected.

[0027] Next, the controller 200 will be described. The controller 200 receives captured images and detection results transmitted from the camera 100 via the network 400, and transmits the selection results of the tracked subject based on user operation to the camera 100. The user can select a tracked subject using the controller 200 of the information processing system 1, and track and photograph the tracked subject selected by the controller 200 using the camera 100.

[0028] The CPU 201 executes various processes using computer programs and data stored in the RAM 202. In doing so, the CPU 201 controls the overall operation of the controller 200 and executes or controls the various processes described below as being performed by the controller 200.

[0029] RAM202 is a high-speed memory device such as DRAM. RAM202 has areas for storing computer programs and data loaded from ROM203 and memory device 219, and areas for storing various data received from camera 100 via network I / F204. Furthermore, RAM202 has areas used by CPU201 and inference execution unit 210 when executing various processes. In this way, RAM202 appropriately provides areas for executing various processes in controller 200.

[0030] ROM203 stores configuration data for controller 200, computer programs and data related to the startup of controller 200, computer programs and data related to the basic operation of controller 200, and so on.

[0031] The inference execution unit 210 performs inference processing to estimate the presence and location of a subject from the captured image. Generally, using a GPU is effective for the inference processing performed by the inference execution unit 210. However, equivalent processing can be achieved using reconfigurable logic circuits such as FPGAs. Alternatively, the CPU 201 may handle the processing of the inference execution unit 210.

[0032] Network I / F204 is an interface for connecting to network 400 and is responsible for communication with external devices such as camera 100 via a communication medium such as ETHERNET. For example, communication with camera 100 includes sending control commands to camera 100 and receiving captured images from camera 100.

[0033] The display unit 205 is a display unit having a screen such as a liquid crystal screen or a touch panel screen, and displays the captured image and detection results received from the camera 100, the controller 200's settings screen, etc. In this embodiment, it is assumed that the display unit 205 has a touch panel screen. Note that the controller 200 does not have a display unit 205, and for example, a display device may be connected to the controller 200 and the captured image and detection results, the controller 200's settings screen, etc. may be displayed on the display device.

[0034] The user input interface 206 is an interface for receiving operations from the user to the controller 200, and includes, for example, buttons, dials, joysticks, touch panels, etc.

[0035] The storage device 219 is a non-volatile storage device such as flash memory, HDD, SSD, or SD card. The storage device 219 stores computer programs and data that cause the CPU 201 and inference execution unit 210 to execute or control various processes described as being performed by the OS and controller 200. The storage device 219 is also used as a temporary storage area for various types of data.

[0036] The CPU 201, RAM 202, ROM 203, inference execution unit 210, storage device 219, network interface 204, display unit 205, and user input interface 206 are each connected to the system bus 207. The controller 200 may also be a PC (Personal Computer) with a mouse, keyboard, etc., as the user input interface 206.

[0037] Next, the processing unit implemented by the camera 100 and controller 200 will be explained using the block diagram in Figure 3. Note that Figure 3 omits the illustration of general-purpose software such as the operating system.

[0038] The ROM 103 of the camera 100 stores software for implementing the imaging unit 301, inference unit 302, drive control unit 303, communication unit 304, and arithmetic unit 309. The CPU 101 appropriately loads this software from the ROM 103 into the RAM 102 for use. The ROM 103 is a storage medium that stores programs for causing the camera 100 to function as a means to perform the processes described below.

[0039] The imaging unit 301 has software functions for acquiring captured images including a subject by having the CPU 101 control the image processing unit 106. The inference unit 302 has software functions for detecting a subject from a captured image and for extracting appearance features of the subject included in the captured image by having the CPU 101 control the inference execution unit 110.

[0040] The drive control unit 303 has a software function to rotate the camera 100 so that its front faces the subject by causing the CPU 101 to control the drive unit 109. The communication unit 304 has a software function to cause the CPU 101 to perform data communication with the controller 200.

[0041] The arithmetic unit 309 has software functions to cause the CPU 101 to perform various arithmetic operations, such as motion prediction processing, arithmetic operations associated with control command calculation, and logical operations for branching processing.

[0042] The controller 200's storage device 219 stores software for implementing the user interface unit 305, inference unit 306, communication unit 308, and arithmetic unit 310. The CPU 201 loads this software from the storage device 219 into the RAM 202 as needed and uses it.

[0043] The user interface unit 305 has software functions to display information necessary for the user and accept user operations by having the CPU 201 control the display unit 205 and the user input interface 206.

[0044] The inference unit 306 has software functions for detecting a subject from a captured image received from the camera 100, and for extracting appearance features of the subject contained in the captured image, by causing the CPU 201 to control the inference execution unit 210. The communication unit 308 has software functions for causing the CPU 201 to perform data communication with the camera 100.

[0045] The arithmetic unit 310 has software functions to cause the CPU 201 to perform various arithmetic operations, such as motion prediction processing, arithmetic operations associated with control command calculation, and logical operations for branching processing.

[0046] Note that the software configuration shown in Figure 3 is just one example; for example, one functional unit could be configured in multiple ways depending on its function. The system may be divided into several functional units, or multiple functional units may be integrated into a single functional unit. Furthermore, one or more of the functional units shown in Figure 3 may be implemented using hardware.

[0047] <Operation of each device> Next, the operation of the camera 100 and the controller 200 in the system according to this embodiment will be described. First, the operation of the camera 100 will be described according to the flowchart in Figure 4A. The CPU 101 reads the software that implements each part described below from the ROM 103, loads it into the RAM 102, and executes each process.

[0048] In step S101, the CPU 101 executes the imaging unit 301 to acquire the captured image from the image processing unit 106 and stores the acquired image in the RAM 102.

[0049] In step S102, the CPU 101 executes the inference unit 302 and inputs the captured image stored in the RAM 102 in step S101 to the inference execution unit 110. The CPU 101 then controls the inference execution unit 110 to detect all subjects in the captured image and stores the subject detection results in the RAM 102.

[0050] At this time, the inference execution unit 110 reads a trained model created using machine learning such as deep learning from the ROM 103 and loads it into the RAM 102. The inference execution unit 110 then inputs the captured image into the trained model and performs calculations on the trained model to detect the subject in the captured image, and outputs attribute information such as the position information, size information, and orientation information of the subject as the detection result. Alternatively, the CPU 101 may reduce the size of the captured image, and the inference execution unit 110 may input the reduced size image into the trained model. This reduces the processing load on the inference execution unit 110 and makes it possible to speed up the inference process.

[0051] Here, the subject detection results in the inference execution unit 110 will be explained. When the inference execution unit 110 inputs a captured image to the trained model, the trained model outputs rectangular information (for example, the coordinates of the top-left and bottom-right vertices of the rectangle) that defines a rectangle containing the entire subject as the position information of the subject in the captured image. Note that the rectangular information is not limited to the entire subject, but may also indicate the position of a part of the subject, such as the head or face of a person. In that case, the trained model used should be changed to a trained model that provides the desired input and output. Furthermore, the position information of the subject is not limited to the coordinates of the top-left and bottom-right vertices of the rectangle containing the entire subject, but may also include information indicating the position of the subject in the captured image, such as the center coordinates, width, and height of the rectangle. The inference execution unit 110 also outputs information indicating the orientation of the subject in one of four directions: front, right, back, or left. Note that the orientation of the subject is not limited to such discontinuous directions, but may also be a continuous angle such as 0 degrees or 90 degrees.

[0052] The method used by the inference execution unit 110 to detect a subject from a captured image is not limited to a specific method. For example, the inference execution unit 110 may use a template matching method in which a template image of the subject is registered in advance, and regions in the captured image with a high degree of similarity to the template image are detected as regions of the subject.

[0053] In step S103, the CPU 101 executes the arithmetic unit 309 to identify the same subject based on motion prediction using the detection results (past detection results) stored in RAM 102 and the appearance features of the captured images within the region of each detection result. Then, the CPU 101 assigns the same identification information as the past subject to the subject detected from the captured image of the current frame that is identical to the past subject detected from the captured image of a past frame. The CPU 101 stores the identification information assigned to the subject detected from the captured image of the current frame in RAM 102. Details of the processing in step S103 will be described later.

[0054] In step S104, the CPU 101 executes the communication unit 304, reads the captured image, detection result, and identification information from the RAM 102, and transmits the read captured image, detection result, and identification information to the controller 200 via the network interface 105. In this embodiment, if the detection result and the captured image are not synchronized due to the execution time of the inference process, the CPU 101 transmits a past detection result as the current detection result.

[0055] In step S105, the CPU 101 executes the calculation unit 309 to determine whether or not it has received identification information of the tracked subject from the controller 200 via the network interface 105. If the CPU 101 has received identification information (S105: YES), it proceeds to step S106. If the CPU 101 has not received identification information (S105: NO), it proceeds to step S107.

[0056] In step S106, the CPU 101 executes the calculation unit 309 and selects a subject to be tracked based on the subject detection results and identification information stored in the RAM 102, and stores the identification information of the selected subject to be tracked in the RAM 102.

[0057] In step S107, the CPU 101 executes the arithmetic unit 309, receives identification information of the subject to be tracked specified by the user operating the controller 200 via the network interface 105, and stores the received identification information in the RAM 102.

[0058] In step S108, the CPU 101 executes the arithmetic unit 309. The CPU 101 then reads the position information of the tracked subject (the subject corresponding to the identification information stored in RAM 102 in step S106 or step S107) in the captured image of the current frame from RAM 102. The CPU 101 also reads the position information of the tracked subject in the target shooting composition. The CPU 101 then uses the read position information to calculate the difference between each position information on the captured image of the current frame. The CPU 101 then converts the calculated difference into an angle difference as seen from camera 100. For example, the CPU 101 approximates the angle per pixel of the captured image using the camera 100's shooting resolution and shooting angle information, and calculates the angle difference by multiplying the calculated angle by the above difference. The CPU 101 then calculates the angular velocity in the pan and tilt directions according to the calculated angle difference. For example, let A_1 be the angle of the tracked subject in the current frame, A_2 be the angle of the tracked subject in the target shooting composition, and G be an arbitrary velocity coefficient. In this case, CPU101 calculates the angular velocity Ω using the following equation 1. Ω = (A_2 - A_1) × G ... (Equation 1)

[0059] The value of the velocity coefficient may be determined experimentally, and the user may specify an arbitrary value through the operation of the controller 200. Furthermore, the method for calculating the angular velocity in the pan and tilt directions is not limited to the method described above. For example, the calculation may be performed such that the pan angular velocity increases if the difference in the horizontal direction is large, and the tilt angular velocity increases if the difference in the vertical direction is large. In this step, the CPU 101 stores the calculated information on the angular velocity in the pan and tilt directions in the RAM 102.

[0060] In step S109, the CPU 101 executes the drive control unit 303 to derive drive parameters for performing panning and tilting of the camera 100 in a desired direction and at a desired speed from the angular velocities in the pan and tilt directions read from the RAM 102. Here, the drive parameters are control values ​​for controlling the motors (not shown) in the pan and tilt directions included in the drive unit 109. Then, the CPU 101 controls the drive unit 109 via the drive I / F 108 based on the derived drive parameters. As the drive unit 109 rotates based on the drive parameters, the camera 100 changes its shooting direction, i.e. Performs pan and tilt movements.

[0061] In step S110, the CPU 101 executes the calculation unit 309 and stores the position information of the subject in the captured image of the current frame in the RAM 102 so that it can be used as the position information of the subject in past frames in subsequent processing.

[0062] In this embodiment, it is assumed that the motion prediction process will refer to detection results up to two frames prior. Therefore, RAM 102 will store detection results up to two frames prior. Note that the number of past frames to refer to is not limited to two frames; any number of past frames may be used for motion prediction.

[0063] In step S111, the CPU 101 executes the arithmetic unit 309 and determines whether the termination conditions for ending tracking have been met. Various conditions can be applied to the termination conditions, and are not limited to specific conditions. For example, termination conditions include "receiving a tracking termination instruction from the controller 200", "the current date and time has reached a specified date and time", and "a specified time has elapsed since tracking started". If the termination conditions are met (S111:YES), the CPU 101 terminates the process in the flowchart of Figure 4A. On the other hand, if the termination conditions are not met (S111:NO), the CPU 101 returns to step S101.

[0064] Next, the process that the controller 200 performs when transmitting identification information of the tracked subject to the camera 100 will be explained according to the flowchart in Figure 4B. The CPU 201 reads the software that implements each of the parts described below from the ROM 203, loads it into the RAM 202, and executes each process.

[0065] In step S201, the CPU 201 executes the arithmetic unit 310 to determine whether or not it has received captured images, detection results, and identification information from the camera 100 via the network interface 204. If the CPU 201 has received captured images, detection results, and identification information from the camera 100 (S201: YES), it stores the received captured images, detection results, and identification information in the RAM 202 and proceeds to step S202. On the other hand, if the CPU 201 has not received captured images, detection results, and identification information from the camera 100 (S201: NO), it repeats the process in step S201.

[0066] In step S202, the CPU 201 executes the user interface unit 305, reads the captured image and detection results from the RAM 202, and displays the read captured image and detection results on the display unit 205.

[0067] Figure 5A shows an example of the display of the captured image and detection results by the display unit 205 in step S202. As shown in Figure 5A, the display screen of the display unit 205 displays a captured image that includes three subjects 700a, 700b, and 700c. In addition, the captured image displays rectangular frames 701a, 701b, and 701c, respectively, which are defined by the rectangular information of subjects 700a, 700b, and 700c, as detection results for subjects 700a, 700b, and 700c. The user can check the captured image and detection results generated by the camera 100 on the screen of the display unit 205 shown in Figure 5A.

[0068] In step S203, the CPU 201 executes the user interface unit 305 and accepts a touch operation from the user on the display unit 205 for "selecting a subject to track". In the display example in Figure 5A, Figure 5B shows the user selecting the subject 700b in the center of the image as the subject to track. In Figure 5B, the user selects the subject 700b as the subject to track by touching the screen with their finger. Note that the method for selecting a subject to track is not limited to a specific method. For example, the user may use user input I / You can also use the F206 to select subject 700b as the tracked subject.

[0069] The CPU 201 then determines whether or not it has received a touch input for the "selection of the subject to be tracked" operation. If the CPU 201 has received a touch input for the "selection of the subject to be tracked" operation (S203: YES), it proceeds to step S204. On the other hand, if the CPU 201 has not received a touch input for the "selection of the subject to be tracked" operation (S203: NO), it returns to step S201.

[0070] In step S204, the CPU 201 executes the communication unit 308 to read the identification information of the subject selected by the user as the subject to be tracked from the identification information stored in the RAM 202. Then, the CPU 201 transmits the read identification information to the camera 100 via the network interface 204.

[0071] Next, the details of the process in step S103 described above will be explained according to the flowchart in Figure 6.

[0072] In step S601 described above, the CPU 101 executes the inference unit 302 and performs the subject re-identification task. In this embodiment, the subject re-identification task is to obtain numerical vectors as appearance features that can identify that the same subject is captured using multiple imaging devices with different shooting positions or directions.

[0073] Specifically, the CPU 101 reads the detection result DBOX of all subjects in the captured image from the RAM 102 and obtains the appearance feature quantity DSTAT from the area corresponding to the detection result DBOX of the captured image. Here, the detection result of the i-th subject (i is a positive integer) in the captured image is DBOX_i, and the appearance feature quantity of the image corresponding to DBOX_i is DSTAT_i. Also, for the i-th detected subject, DBOX_i and DSTAT_i are linked to the detected subject information OBJ_INFO_i. The CPU 101 stores the linked information in the RAM 102. After storing the detected subject information in the RAM 102 for all detected subjects, the CPU 101 proceeds to step S602.

[0074] In step S602, the CPU 101 calculates the feature quality (a value indicating the quality of a feature) QUAL for each detected subject information OBJ_INFO, and stores the calculated feature quality QUAL in RAM 102, associating it with OBJ_INFO. In this step, the i-th detected subject information OBJ_INFO_i stored in RAM 102 is information in which the detection result DBOX_i, the appearance feature DSTAT_i, and the feature quality QUAL_i are linked to each other. After storing the linked information in RAM 102, the CPU 101 proceeds to step S603.

[0075] Here, using Figure 7, we will explain how to calculate the feature quality QUAL_i of the i-th detected subject included in the captured image. The CPU 101 predicts the feature quality QUAL_i of each subject based on the degree of overlap of multiple subjects in the captured image by performing the following process.

[0076] First, in step S701, the CPU 101 selects the first detection result DBOX as the target for calculating the feature quality QUAL_i, sets index i to 1 (i=1), and stores it in RAM 102.

[0077] Next, in step S702, the CPU 101 reads index i from RAM 102 and checks whether index i is greater than the total number of detection results included in the most recent captured image. The CPU 101 makes a determination. If index i is greater than the total number of detection results (S702: YES), it considers that the calculation of feature quality QUAL_i has been completed for all detection results and terminates the calculation process of feature quality QUAL_i. On the other hand, if index i is less than or equal to the total number of detection results (S702: NO), the CPU 101 considers that there are subjects for which the calculation of feature quality QUAL_i has not been performed and proceeds to step S703.

[0078] Next, in step S703, the CPU 101 reads the detection result DBOX_i of the i-th detected subject's detection subject information OBJ_INFO_i from RAM 102 as the target for calculating the feature quality QUAL_i. In this embodiment, the CPU 101 calculates the overlap and area of ​​DBOX_i and DBOX_j in the process of calculating the feature quality QUAL_i,j to which other detection results DBOX_j (i≠j) contribute to the detection result DBOX_i. Then, the CPU 101 obtains the detection result DBOX_i as four image coordinate values: the coordinates of the top-left vertex (DTOP_i, DLEFT_i) and the coordinates of the bottom-right vertex (DBOTTOM_i, DRIGHT_i) in the planar coordinates set within the image, and stores them in RAM 102.

[0079] Next, in step S704, the CPU 101 initializes the index j for calculating the feature quality QUAL_i,j to which other detection results DBOX_j contribute to the detection result DBOX_i, in order to calculate the feature quality QUAL_i. In this embodiment, the CPU 101 sets the index j for calculating the feature quality QUAL_i,j to 1 (j=1) and stores it in RAM 102.

[0080] Next, in step S705, the CPU 101 reads index j from RAM 102 and determines whether index j is greater than the total number of detection results included in the most recent captured image. If index j is greater than the total number of detection results (S705: YES), the CPU 101 considers that the calculation of the feature quality QUAL_i,j for all detection result DBOXs has been completed for the detection result DBOX_i and proceeds to step S709. On the other hand, if index j is less than or equal to the total number of detection results (S705: NO), the CPU 101 considers that the calculation of the feature quality QUAL_i,j for all detection result DBOXs has not been completed and proceeds to step S706.

[0081] Next, in step S706, the CPU 101 reads the detection result DBOX_j from RAM 102, which is the target for calculating the feature quality QUAL_i,j for the detection result DBOX_i. In this embodiment, as described in step S703, the CPU 101 obtains the detection result DBOX_j as four image coordinate values: the top-left vertex (DTOP_j, DLEFT_j) and the bottom-right vertex (DBOTTOM_j, DRIGHT_j), and stores them in RAM 102.

[0082] Next, in step S707, the CPU 101 calculates the quality of features QUAL_i,j based on the detection results DBOX_i and DBOX_j read from RAM 102, and stores the calculated quality of features QUAL_i,j in RAM 102.

[0083] Here, the calculation process of the feature quality QUAL_i,j by CPU101 will be explained using Figures 9A and 9B. Figure 9A shows an example of the detection result of the subject detected by CPU101 in step S102 with respect to the captured image. In Figure 9A, three subjects, subject 901a, subject 902a, and subject 903a, are drawn on the captured image 910. Also in Figure 9A, the positions of subjects 901a, 902a, and 903a detected by CPU101 in step S102 are indicated by detection positions 901b, 902b, and 903b, respectively. In the following explanation, the first digit of the subscript will be used as an index to refer to the detection result. For example, detection result DBOX_1 corresponds to detection position 901b in captured image 910. Similarly, detection results DBOX_2 and DBOX_3 correspond to detection positions 902b and 903b in captured image 910, respectively.

[0084] Furthermore, using Figure 9B, we will explain the specific calculation process using the quality of features QUAL_1 and QUAL_2, which represent the contribution of subject 902a to subject 901a, calculated when index i is 1 and index j is 2, as an example.

[0085] First, the CPU 101 reads the detection result DBOX_1 of the subject 901a from RAM 102 as (DTOP_1, DLEFT_1) and (DBOTTOM_1, DRIGHT_1), and calculates the area SDBOX_1 of DBOX_1. The CPU 101 calculates SDBOX_1 using the following equation 2. SDBOX_1 = (DBOTTOM_1 - DTOP_1) × (DRIGHT_1 - DLEFT_1) ... (Equation 2)

[0086] Next, the CPU 101 reads the detection result DBOX_2 of the subject 902a from RAM 102 as (DTOP_2, DLEFT_2) and (DBOTTOM_2, DRIGHT_2). Then, the CPU 101 calculates the area of ​​the overlapping region INTERSECTION_1_2 that overlaps with the detection result DBOX_1 using the following equation 3. INTERSECTION_1_2=MAX(0,(DBOTTOM_1-DTOP_2)×(DRIGHT_1-DLEFT_2))...(Formula 3)

[0087] Finally, CPU101 calculates the quality of features QUAL_1 and QUAL_2, which represent the contribution of detection result DBOX_2 to detection result DBOX_1, using equation 4 below. CPU101 then uses the value obtained by dividing the area of ​​the region where the first and second subjects overlap in the captured image by the area of ​​the first subject in the captured image as the quality of features for the first subject. The calculated quality of features is an example of an indicator showing the degree of overlap between the tracked subject and other subjects. QUAL_1,2=INTERSECTION_1_2 / SDBOX_1 (Formula 4)

[0088] CPU 101 calculates QUAL_i,j and stores it in RAM 102, then proceeds to step S708.

[0089] Next, in step S708, CPU 101 adds 1 to the current index j in order to calculate the feature quality QUAL_i,j for the next detection result DBOX_j of detection result DBOX_i. After performing the processing in step S708, CPU 101 returns to step S705.

[0090] In step S709, the CPU 101 calculates the final feature quality QUAL_i for the detection result DBOX_i and stores it in RAM 102. In this embodiment, the CPU 101 reads out all feature quality QUAL_i,j related to the detection result DBOX_i and determines the minimum value of the read feature quality as the final feature quality QUAL_i. The CPU 101 stores the determined final feature quality QUAL_i in RAM 102.

[0091] Here, with reference to FIG. 9B, the meaning of the value of the quality QUAL_i of the final feature amount determined in step S709 will be described. For convenience of explanation, here, the detection result DBOX_i will be described using only the detection results DBOX_1 and DBOX_2 regarding the subject 901a and the subject 902a.

[0092] As shown in FIG. 9B, the relationship between the areas of the detection results DBOX_1 and DBOX_2 is SDBOX_1 < SDBOX_2. Also, the value of the overlapping region INTERSECTION_1_2 between the detection results DBOX_1 and DBOX_2 does not change. Therefore, since the quality QUAL_i of the feature amount calculated by Equation 4 has an inverse magnitude relationship with the area, the quality of the feature amount QUAL_1 > the quality of the feature amount QUAL_2.

[0093] Also, since the value of the overlapping region INTERSECTION_1_2 is at most the area SBOX_1 of the detection result DBOX_1, the quality QUAL_i of the feature amount takes a value between 0 and 1, and the closer it is to 1, the larger the overlapping area with other detection results DBOX_j. Further, for example, assume a captured image in which two persons of similar height are intersecting. In this case, considering the depth direction as viewed from the camera 100, the ratio contributing to the overlapping area of the detection result DBOX_j of the subject on the back side (the j-th detected subject) with respect to the detection result DBOX_i of the subject on the front side (the i-th detected subject) as viewed from the camera 100 is small. Therefore, the detection result DBOX_i with a large value of the quality QUAL_i of the feature amount is likely to be a detection result regarding a subject located deeper than other subjects as viewed from the camera 100.

[0094] Next, in step S710, the CPU 101 adds 1 to the index i in order to shift the target for determining the quality QUAL_i of the feature amount to the next detection result DBOX_i. After storing the incremented index i in the RAM 102, the CPU 101 returns the process to step S702.

[0095] Through the above process, CPU 101 can calculate the QUAL_i quality of each feature in the detected DBOX_i in the current frame. In the above process, CPU 101 predicts the feature quality based on the area of ​​each subject in the captured image, but CPU 101 may also predict the feature quality based on the width or height of each subject instead of, or in addition to, the area of ​​each subject.

[0096] Next, in step S603 described above, the CPU 101 tracks the subject selected as the tracking target from among multiple subjects based on the position and feature quantities of each subject. Specifically, the CPU 101 assigns the same identifier to the same subject in the captured image in accordance with the time of capture, based on the detected subject information OBJ_INFO and the quality of the feature quantities QUAL obtained by the above process. The CPU 101 links the tracking subject information TRACK_INFO and the detected subject information OBJ_INFO in past captured images and updates the tracking subject information TRACK_INFO. After the CPU 101 assigns the identifier ID to the tracking subject information TRACK_INFO and stores it in RAM 102, the process proceeds to step S105.

[0097] Here, we will explain the subject tracking process performed by the CPU 101 in step S603 using Figures 8 and 10. Figure 8 is a flowchart of the subject tracking process performed by the CPU 101. Figure 10 is a diagram illustrating an example of the subject tracking process performed by the CPU 101. Figure 10 shows subjects 1001a and 1002a drawn in the captured image 1010. Figure 10 also shows a frame 1001b showing the detection result of subject 1001a and a frame 1002b showing the detection result of subject 1002a.

[0098] Furthermore, Figure 10 shows frame 1001c, which represents the results of tracking and predicting subject 1001a with identifier ID 1. For example, frames 1001c_t-2 and 1001c_t-1 show the detection results of subject 1001a tracked over the past two frames. Frame 1001c_t shows the prediction result, which indicates the predicted position of subject 1001a in the currently captured image 1010.

[0099] Furthermore, Figure 10 shows frame 1002c, which represents the results of tracking and predicting subject 1002a with identifier ID 2. For example, frames 1002c_t-2 and 1002c_t-1 show the detection results of subject 1002a in the past two frames, and frame 1002c_t is the predicted position of subject 1002a in the currently captured image 1010. Details of the prediction processes for 01a and 1002a will be described later.

[0100] In step S801, the CPU 101 reads the tracked subject information TRACK_INFO from the RAM 102 for past captured images and predicts the position of the tracked subject in the current captured image based on the tracked subject information TRACK_INFO. In this embodiment, the tracked subject information TRACK_INFO consists of an identifier ID_i, the detection results TBOX_i_t-1 and TBOX_i_t-2 for the past two times, and the appearance feature quantity TSTAT_i_t-1 of the subject for the past one time.

[0101] The CPU 101 predicts the detection result PBOX_i_t in the captured image 1010 based on the detection result TBOX_i_t-1 and detection result TBOX_i_t-2 for all TRACK_INFO. In this embodiment, the CPU 101 predicts the detection result PBOX_i_t at the present time (time t) by assuming that the X and Y coordinates, aspect ratio, and height of the center of the detection result TBOX change linearly from time t-2 to time t-1.

[0102] The CPU 101 determines the detection result BOX_i_t for all the tracked subject information TRACK_INFO, associates the detection result BOX_i_t with the tracked subject information TRACK_INFO, stores it in RAM 102, and then proceeds to step S802.

[0103] Next, in step S802, the CPU 101 reads the detected subject information OBJ_INFO from RAM 102. Then, the CPU 101 extracts OBJ_INFO whose feature quality QUAL is higher than the threshold th, and sets the extracted OBJ_INFO as the processing target TARGET_OBJ_INFO.

[0104] Here, the threshold th is a predetermined value, and in this embodiment, it is set to 0.4 as an example, but the value of the threshold th is not limited to this and may be adaptively changed according to the information contained in OBJ_INFO. Furthermore, the threshold th is not limited to a constant value, and may be changed between 0 and 1 depending on the proportion of the area of ​​the detected result DBOX_i in the captured image 1010 for each subject.

[0105] Furthermore, the CPU 101 sets any OBJ_INFOs that were not selected as the TARGET_OBJ_INFO to be processed as NON_TARGET_OBJ_INFO. After storing the TARGET_OBJ_INFO and NON_TARGET_OBJ_INFO in RAM 102, the CPU 101 proceeds to step S803.

[0106] Next, in step S803, the CPU 101 reads TRACK_INFO and TARGET_OBJ_INFO from RAM 102 and performs the first matching process. In the first matching process, if the degree of overlap between the tracked subject and other subjects is a first value, the CPU 101 uses the similarity of the features to associate one of the detected subjects with the tracked subject. Here, the first value means a value higher than the threshold th mentioned above. The first matching process is the process of first matching subjects in the captured image. Specifically, the CPU 101 obtains the appearance feature TSTAT_i_t-1 contained in TRACK_INFO and the appearance feature DSTAT_j contained in TARGET_OBJ_INFO, and calculates the appearance cost matrix R using the obtained appearance features.

[0107] Here, if N is the number of tracked subjects in TRACK_INFO and M is the number of detected subjects in the target TARGET_OBJ_INFO, then the size of the appearance cost matrix R is N × M. Also, the value in the i-th row and j-th column of the appearance cost matrix R represents the similarity between the i-th tracked subject in TRACK_INFO and the j-th detected subject in TARGET_OBJ_INFO. This indicates the degree. In this embodiment, each value included in the appearance cost matrix R is the cosine distance calculated from the appearance feature TSTAT_i_t-1 and the appearance feature DSTAT_j. Furthermore, the value in the i-th row and j-th column of the appearance cost matrix R is calculated by the following equation 5. R_i,j=1-cos(TSTAT_i_t-1, DSTAT_j) (Equation 5)

[0108] The CPU 101 determines pairs of TRACK_INFO and TARGET_OBJ_INFO whose appearance features are similar by solving the optimal assignment problem for the calculated appearance cost matrix R. Here, optimal assignment means determining the combination that results in the lowest cost while assigning only one TARGET_OBJ_INFO to each TRACK_INFO. In this embodiment, it is assumed that the CPU 101 solves the optimal assignment problem for the appearance cost matrix R based on the Hungarian algorithm, but the method of solving the optimal assignment problem is not limited to this method. For example, the CPU 101 may employ a method of assigning the TARGET_OBJ_INFO with the lowest cost in the appearance cost matrix R in ascending order of the index of TRACK_INFO.

[0109] Alternatively, the CPU 101 may pre-filter and obtain elements with costs below a predetermined threshold from the appearance cost matrix R, and then solve the optimal assignment problem. This allows the CPU 101 to suppress misrecognition, for example, when the number of tracking subjects N is greater than the number of detection subjects M, and there are no detection subjects corresponding to the tracking subjects.

[0110] As a result of performing the optimal allocation problem, CPU101 stores the TRACK_INFO and TARGET_OBJ_INFO pairs as PAIR_INFO in RAM102. CPU101 also sets any TARGET_OBJ_INFO that could not be paired as UNMATCHED_TARGET_OBJ_INFO. Furthermore, CPU101 sets any TRACK_INFO that could not be paired as UNMATCHED_TRACK_INFO. Finally, CPU101 stores UNMATCHED_TARGET_OBJ_INFO and UNMATCHED_TRACK_INFO in RAM102.

[0111] Next, in step S804, the CPU 101 reads PAIR_INFO from RAM 102 and updates each TRACK_INFO with its corresponding TARGET_OBJ_INFO.

[0112] Specifically, the CPU 101 sets the detection result TBOX_i_t of the i-th TRACK_INFO from the read PAIR_INFO to the detection result DBOX_j contained in the j-th TARGET_OBJ_INFO that was paired in step S803. This is expected to improve the accuracy of predicting the position of the tracked subject on the next captured image when the processing of this flowchart is performed on the next captured image in step S801.

[0113] Furthermore, in this embodiment, in addition to setting the detection result TBOX_i_t included in TRACK_INFO, the CPU 101 updates the appearance feature TSTAT_i_t based on DSTAT_j included in TARGET_OBJ_INFO. The CPU 101 updates the appearance feature TSTAT_i_t using the following equation 6. Note that A in equation 6 is a value set between 0 and 1, and indicates the ratio of the appearance feature update. In this embodiment, as an example, A is assumed to be an experimentally determined value, and that value is set to 0.9. TSTAT_i_t=A×TSTAT_i_t-1+(1-A)×DSTAT_j (Formula 6)

[0114] After CPU101 has finished updating all TRACK_INFOs included in PAIR_INFO, it proceeds to step S805.

[0115] Next, in step S805, the CPU 101 sets TARGET_OBJ_INFO as the target for solving the optimal allocation problem. Specifically, the CPU 101 reads NON_TARGET_OBJ_INFO and UNMATCHED_TARGET_OBJ_INFO from RAM 102, combines them, sets them as TARGET_OBJ_INFO, and stores them in RAM 102.

[0116] Next, in step S806, the CPU 101 reads the TARGET_OBJ_INFO and UNMATCHED_TRACK_INFO from RAM 102 and performs a second matching process. That is, if the degree of overlap between the tracked subject and other subjects is less than the first value, the CPU 101 uses the positional similarity to associate one of the detected subjects with the tracked subject. The second matching process is a process of secondly associating subjects in the captured image.

[0117] Specifically, CPU101 calculates a distance-related distance cost matrix D from the predicted detection result PBOX_i_t contained in UNMATCHED_TRACK_INFO and the detection result DBOX_j contained in TARGET_OBJ_INFO.

[0118] Here, if K is the number of tracked subjects in UNMATCHED_TRACK_INFO and L is the number of detected subjects in the target TARGET_OBJ_INFO, then the size of the distance-cost matrix D is K × L. Also, the value in the kth row and lth column of the distance-cost matrix D (where k and l are positive integers) represents the similarity between the kth tracked subject in UNMATCHED_TRACK_INFO and the lth detected subject in TARGET_OBJ_INFO.

[0119] In this embodiment, the CPU 101 calculates the positional similarity by the distance between the midpoints of the top edges of the predicted detection result PBOX_t_i and the detected result DBOX_j, using the following equation 7: D_i,j=|TC(PBOX_i_t)―TC(DBOX_j)|···(Equation 7)

[0120] In Equation 7, TC represents a function that calculates the coordinates of the upper center point of the predicted detection result PBOX_t_i and the detection result DBOX_j. When the detection result is DBOX, TC(DBOX) is calculated by the following Equation 8. TC(DBOX)=(DTOP,(DLEFT+DRIGHT) / 2)...(Formula 8)

[0121] The CPU 101 determines pairs of UNMATCHED_TRACK_INFO and TARGET_OBJ_INFO based on positional similarity by solving an optimal assignment problem with the calculated distance-cost matrix D. The optimal assignment problem performed here is the same as the optimal assignment problem performed in step S803, so a detailed explanation is omitted.

[0122] The CPU 101 may also pre-filter and obtain elements with costs below a predetermined threshold from the distance-cost matrix D, and then solve the optimal assignment problem. This allows the CPU 101 to suppress misrecognition, for example, when the number of tracking subjects N is greater than the number of detection subjects M, and there are no detection subjects corresponding to the tracking subjects.

[0123] As a result of performing the optimal allocation problem, CPU 101 sets the pair of UNMATCHED_TRACK_INFO and TARGET_OBJ_INFO as PAIR_INFO and stores it in RAM 102. CPU 101 also sets the TARGET_OBJ_INFO that could not be paired as UNMATCHED_TARGET_OBJ_INFO. The CPU 101 then determines the value and stores it in RAM 102. Additionally, the CPU 101 stores any UNMATCHED_TRACK_INFOs that could not be paired into RAM 102 as UNMATCHED_TRACK_INFOs. Finally, the CPU 101 proceeds to step S807.

[0124] The effect of solving the optimal assignment problem based on the distance of the upper center point of the detected subject in this embodiment will be explained using Figure 10. Figure 10 shows a state in which subject 1001a and subject 1002a overlap in a captured image 1010 taken at time t. Figure 10 also shows a case where the detection result of subject 1001a, which is partially hidden on the far side from the camera 100, is only the upper body of subject 1001a, as shown in frame 1001b.

[0125] In this embodiment, the CPU 101 acquires appearance feature quantity STAT_i for frame 1001b, which shows the detection result of subject 1001a in the captured image 1010. However, frame 1001b does not include the lower half of subject 1001a. Also, frame 1001b overlaps with a part of subject 1002a. For this reason, for example, depending on the clothing of subject 1002a, the appearance feature quantity acquired by the CPU 101 for frame 1001b may include the appearance feature quantity of subject 1002a. As a result, the CPU 101 is more likely to fail in the first matching process using frame 1001c_t, which shows the predicted position of subject 1001a with identifier ID 1, and frame 1001b in the captured image 1010.

[0126] Next, Figure 11 shows an example of the predicted position of subject 1001a with identifier ID 1 and the detection result of subject 1001a in the captured image 1010. Figure 11 shows the upper center point 1100a of frame 1001c_t, which indicates the predicted position of subject 1001a with identifier ID 1 in the current captured image 1010. Also, Figure 11 shows the upper center point 1100b of frame 1001b, which is the detection result of subject 1001a in the captured image 1010.

[0127] As shown by the central points 1100a and 1100b in Figure 11, if the subject is the same, the upper central points will be close together even if part of the subject is hidden by other subjects. Therefore, the CPU 101 can achieve more accurate matching by performing a second matching process based on the distance between the upper central points for subjects that are judged to have low-quality features or for subjects that failed to be matched based on appearance features. Accordingly, in this embodiment, when the degree of overlap between the tracked subject and other subjects is a first value, the CPU 101 increases the contribution rate of feature similarity over the contribution rate of position similarity and performs the first matching process. Furthermore, when the degree of overlap between the tracked subject and other subjects is a second value, the CPU 101 increases the contribution rate of position similarity over the contribution rate of feature similarity and performs the second matching process. As a result, the CPU 101 can adaptively predict the quality of the features of each subject according to the degree of overlap of each subject in the captured image, and is expected to achieve more accurate subject tracking.

[0128] Next, in step S807, the CPU 101 reads PAIR_INFO from RAM 102 and updates the current position of TRACK_INFO with the detection result DBOX in the captured image. Note that the update process in this step is the same as the update process in step S804, so a detailed explanation is omitted.

[0129] CPU 101 completes the update of all TRACK_INFO contained in PAIR_INFO, stores them in RAM 102, and then proceeds to step S808.

[0130] Next, in step S808, the CPU 101 reads UNMATCHED_TARGET_OBJ_INFO and UNMATCHED_TRACK_INFO from RAM 102 and performs a third matching process.

[0131] Specifically, CPU101 uses UNMATCHED_TRACK_INFO and UNMA For TCHED_TARGET_OBJ_INFO, an overlap cost matrix O is calculated based on the overlap of detection results. Then, CPU 101 solves the optimal assignment problem using the calculated overlap cost matrix O. Here, if P is the number of tracked subjects in UNMATCHED_TRACK_INFO and Q is the number of detected subjects in the target UNMATCHED_TARGET_OBJ_INFO, then the size of the overlap cost matrix O is P × Q. The value in the p-th row, q-th column of the overlap cost matrix O (where p and q are positive integers) represents the similarity based on the overlap between the p-th tracked subject in UNMATCHED_TRACK_INFO and the q-th detected subject in UNMATCHED_TARGET_OBJ_INFO.

[0132] In this embodiment, the CPU 101 calculates the similarity between the tracked subject and the detected subject from IoU (Intersection over Union). The value of each element of the overlap cost matrix O is calculated by the following equation 9. The calculation method for IoU is a general method, so a detailed explanation is omitted. O_p,q=1-IoU(PBOX_p_t,DBOX_q) (Equation 9)

[0133] Next, CPU 101 determines the pair of UNMATCHED_TRACK_INFO and UNMATCHED_TARGET_OBJ_INFO by solving the optimal allocation problem for the overlap cost matrix O. The optimal allocation problem performed here is the same as the optimal allocation problem performed in step S803, so a detailed explanation is omitted.

[0134] The CPU 101 stores the pair of UNMATCHED_TRACK_INFO and TARGET_OBJ_INFO obtained by performing the optimal allocation problem as PAIR_INFO in RAM 102.

[0135] Furthermore, CPU 101 stores TARGET_OBJ_INFOs that could not be paired as UNMATCHED_TARGET_OBJ_INFO in RAM 102. Also, CPU 101 stores UNMATCHED_TRACK_INFOs that could not be paired as UNMATCHED_TRACK_INFO in RAM 102. Then, CPU 101 proceeds to step S809.

[0136] Next, in step S809, the CPU 101 reads PAIR_INFO from RAM 102 and updates TRACK_INFO with the detection result BOX for the currently captured image. The update process in this step is the same as the update process performed in step S804, so a detailed explanation is omitted. After the CPU 101 has finished updating all TRACK_INFO contained in PAIR_INFO and stored them in RAM 102, the process proceeds to step S810.

[0137] Next, in step S810, the CPU 101 reads UNMATCHED_TARGET_OBJ_INFO from RAM 102 and registers the subject corresponding to UNMATCHED_TARGET_OBJ_INFO as a tracked subject. Specifically, the CPU 101 obtains the maximum value of identifier IDs from the entire TRACK_INFO and determines a new identifier NID by adding 1 to the obtained identifier ID. Then, the CPU 101 generates tracked subject information TRACK_INFO for identifier NID. Furthermore, the CPU 101 sets the detection results TBOX_NID_t-2 and TBOX_NID_t-1 of TRACK_INFO_NID into the detection result DBOX included in UNMATCHED_TARGET_OBJ_INFO. Furthermore, the CPU 101 sets the appearance feature TSTAT_NID_t included in TRACK_INFO_NID into the appearance feature DSTAT included in UNMATCHED_TARGET_OBJ_INFO. Then, CPU101 stores TRACK_INFO_NID in RAM102. CPU101 then stores the tracked subject information for all subjects included in UNMATCHED_TARGET_OBJ_INFO. After the generation is complete, the process proceeds to step S811.

[0138] Next, in step S811, the CPU 101 reads UNMATCHED_TRACK_INFO from RAM 102 and performs an update process. Specifically, for each UNMATED_TRACK_INFO, the CPU 101 sets the current tracked subject's position TBOX_i_t to the predicted detection result PBOX_i_t predicted in step S801 and stores it in RAM 102. The CPU 101 updates the current tracked subject's position for all subjects included in UNMATED_TRACK_INFO, stores it in RAM 102 as TRACK_INFO, and then terminates the subject tracking process according to this flowchart.

[0139] Based on the above description, according to this embodiment, for multiple subjects detected in the captured image of the current frame, the quality of the feature quantities is determined based on the overlap between the subjects, and the matching process is adaptively switched using the quality of the feature quantities. As a result, it is expected that the accuracy of subject tracking will be improved, especially when subjects overlap within the captured image.

[0140] In this embodiment, the CPU 101 performs matching of the tracked subject and the detected subject based on the appearance cost matrix R in the first matching process and on the distance cost matrix D in the second matching process. However, each matching process is not limited to the above processes. For example, the CPU 101 may calculate the appearance cost matrix R and the distance cost matrix D in advance, and then increase the weight of the appearance cost matrix R in the first matching process and increase the weight of the distance cost matrix D in the second matching process to generate a new cost matrix.

[0141] Furthermore, in this embodiment, the distance cost matrix D is generated by the CPU 101 calculating the upper center point of the detection result frame for the tracked subject and the detected subject, but the method of calculating the distance cost matrix D is not limited to this. For example, in shooting conditions where the upper body of the subject is easily obscured, the CPU 101 can perform the matching process while reducing the effect of the upper body of the subject being obscured by calculating the distance cost matrix D from the lower center point of each detection result instead of the upper center point.

[0142] In this embodiment, the camera 100 performs the detection of the subject and the calculation of the drive amount for tracking, but the controller 200 may perform some or all of these processes. In this case, the camera 100 transmits the captured image to the controller 200. Next, the controller 200 detects the subject from the received captured image, derives drive parameters which are control values ​​that control the drive unit 109 to track the subject using the camera 100, and transmits the derived drive parameters to the camera 100. Then, the camera 100 operates each part of the camera 100 according to the received drive parameters to photograph the subject being tracked. At this time, the processing performed by the inference execution unit 110 of the camera 100 may be performed by the inference execution unit 210 of the controller 200. Furthermore, the software operation of the inference unit 302 and calculation unit 309 of the camera 100 can be replaced by the software operation of the inference unit 306 and calculation unit 310 of the controller 200, respectively.

[0143] [Second Embodiment] Next, an information processing system according to the second embodiment will be described. In the following description, the differences from the first embodiment will be explained, and unless otherwise specified, it will be assumed that it is the same as the first embodiment. In this embodiment, when detecting a subject from an image captured by the camera 100, multiple joint points of the subject are detected, and subject tracking processing is performed based on the detected joint points.

[0144] The information processing system 1 according to this embodiment, like the first embodiment (Figure 1), includes a camera 100 and a controller 200 which is a control device for the camera 100. The camera 100 and the controller 200 are connected to the network 400. As a result, the information processing system 1 according to this embodiment is configured so that the camera 100 and the controller 200 can communicate data with each other via the network 400.

[0145] <Configuration of each device> Next, examples of the hardware configurations of the camera 100 and the controller 200 will be described using the block diagram in Figure 2. Note that the configuration shown in Figure 2 is merely one example of the hardware configuration of the camera 100 and the controller 200, and can be modified and / or altered as appropriate. In this embodiment, the configuration of the inference execution unit 110 of the camera 100 and the inference execution unit 210 of the controller 200 differs from that of the first embodiment.

[0146] The inference execution unit 110 performs inference processing to estimate the presence or absence of joints and the position coordinates of joints in the captured image, and extraction processing to extract external feature quantities of each subject from the captured image. The inference execution unit 110 is, for example, a computing device specialized for image processing and inference processing, such as a GPU. While a GPU is generally effective for inference processing, equivalent functionality may be achieved with a reconfigurable logic circuit such as an FPGA. Alternatively, the processing of the inference execution unit 110 may be handled by the CPU 101.

[0147] Next, the processing unit implemented by the camera 100 and the controller 200 will be explained using the block diagram in Figure 3. Note that in Figure 3, the illustration of general-purpose software such as the operating system is omitted. In this embodiment, the contents of the inference unit 302 of the camera 100 and the inference unit 306 of the controller 200 differ from those of the first embodiment.

[0148] The inference unit 302 has a software function that detects the coordinates of the joints of a subject from a captured image and a software function that extracts the appearance features of the subject contained in the captured image, by having the CPU 101 control the inference execution unit 110.

[0149] Furthermore, the inference unit 306 has a software function that detects the coordinates of the joints of the subject from the captured image received from the camera 100, and a software function that extracts the appearance features of the subject included in the captured image, by having the CPU 201 control the inference execution unit 210.

[0150] <Operation of each device> Next, the operation of the camera 100 and controller 200 in the information processing system 1 according to this embodiment will be described. First, the operation of the camera 100 will be described according to the flowchart in Figure 4A. Note that in this embodiment, the subject detection process in step S102 differs from that of the first embodiment.

[0151] In step S102, the CPU 101 executes the inference unit 302, inputs the captured image stored in the RAM 102 in step S101 to the inference execution unit 110, and controls the inference execution unit 110 to detect the coordinates of the joints of all subjects in the captured image. Furthermore, the CPU 101 obtains the position (region) of each subject in the captured image based on the detected joint coordinates and stores the information indicating the position of each subject in the RAM 102.

[0152] Figure 12 shows an example of the results of CPU 101 detecting the coordinates of the joints of all subjects included in the captured image 1110. Using Figure 12, the process by which CPU 101 detects subjects in the captured image will be explained in this embodiment.

[0153] Figure 12 shows the captured image 1110 in which subjects 1100 and 1101 are drawn, and the coordinates of each joint detected by the CPU 101 are indicated by "KPT". For example, "KPT_1100" represents the coordinates of the joint detected relative to subject 1100, and KP T_1100_1 represents the coordinates of the first joint. In this embodiment, as an example, it is assumed that the CPU 101 detects the joints of the top of the head, neck, left shoulder, right shoulder, right hand, left hand, right foot, and left foot for each subject in the captured image 1110. For subject 1101, the joints detected by the CPU 101 are the same as those detected for subject 1100, so the following explanation will only describe subject 1100. Also, the order of the joint coordinates is not limited to the order described here.

[0154] In Figure 12, it is assumed that in the captured image 1110, the left shoulder of subject 1100 is to the left of its right shoulder. Therefore, it is inferred that subject 1100 is facing away from camera 100. Also in Figure 12, it is assumed that in the captured image 1110, the left shoulder of subject 1101 is to the right of its right shoulder. Therefore, it is inferred that subject 1101 is facing directly towards camera 100.

[0155] The CPU 101 obtains the coordinates of the joints of each subject in the captured image 1110, and then obtains the position (region) of each subject based on the obtained joint coordinates. Specifically, the CPU 101 generates a detection result DBOX for the subjects, using the maximum / minimum values ​​in the horizontal direction (left-right direction in the figure) and the maximum / minimum values ​​in the vertical direction (up-down direction in the figure) of the captured image 1110 as the values ​​of the detection result DBOX.

[0156] Next, using the subject 1100 in Figure 12 as an example, we will explain the process by which the CPU 101 determines the detection result DBOX of the subject 1100.

[0157] The CPU 101 determines four image coordinate values, including the coordinates of the top-left vertex (DTOP_1100, DLEFT_1100) and the coordinates of the bottom-right vertex (DBOTTOM_1100, DRIGHT_1100), in order to identify the detected DBOX of the subject 1100. In Figure 12, for the subject 1100, the maximum horizontal value in the captured image 1110 is indicated by KPT_1100_6. Also, for the subject 1100, the minimum horizontal value in the captured image 1110 is indicated by KPT_1100_5. Furthermore, for the subject 1100, the maximum vertical value in the captured image 1110 is indicated by KPT_1100_1. Also, for the subject 1100, the minimum vertical value in the captured image 1110 is indicated by KPT_1100_7. Based on these coordinate values, the CPU 101 determines the coordinates of the top-left and bottom-right vertices that indicate the detection result BOX for the subject 1100, and stores the determined coordinate information in the RAM 102. The CPU 101 determines the detection result DBOX for all subjects in the captured image 1110, stores it in the RAM 102, and then proceeds to step S103.

[0158] As described above, even when the camera 100 obtains the coordinates of the joints of the subject in the captured image and performs inference of the subject's position, it can estimate the region of each subject from the joint coordinate information and adaptively switch the matching process based on the quality of the features of each subject.

[0159] In the first embodiment, the method for calculating the quality of features was described in detail with reference to Figure 7, but in this embodiment as well, the CPU 101 can calculate the quality of features based on the coordinates of the joints of the subject. For example, the CPU 101 may calculate the quality of features from the difference in size between the intersection of the skeleton determined from predetermined pairs of joints and the intersecting subjects, based on the coordinates of the joints read from the RAM 102. In this case, the degree of overlap between the tracked subject and other subjects will be the degree of overlap between the tracked subject and the subjects whose skeletons intersect with each other.

[0160] For example, in Figure 12, the dotted lines connecting the coordinates of the joints correspond to the skeleton of the subject, and the skeleton shown by the dotted line connecting KPT_1100_2 and KPT_1100_4 corresponds to the right shoulder of subject 1100. Furthermore, the skeleton of the right shoulder of subject 1100 corresponds to the KPT_ of subject 1101. The dotted line connecting 1101_4 and KPT_1101_6 intersects with the skeleton of the right arm. Therefore, CPU101 calculates the quality of features of subject 1100 and subject 1101 by comparing the difference between the maximum and minimum values ​​in the vertical direction of the joint coordinates of each subject, which is used as the size of the subject. For example, when the size of subject 1100 is SIZE1100 and the size of subject 1101 is SIZE1101, CPU101 calculates the quality of features of subject 1100 and subject 1101, respectively, using equations 10 and 11 below. QUAL_1100,1101=SIZE1100 / max(SIZE1100.SIZE1101)...(Formula 10) QUAL_1101,1100=SIZE1101 / max(SIZE1100.SIZE1101)...(Formula 11)

[0161] According to equations 10 and 11, the CPU 101 can obtain the quality of the features of each subject 1100 and subject 1101 as numerical values ​​in the range of 0 to 1, and adaptive switching of the matching process becomes possible, similar to the first embodiment.

[0162] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0163] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0164] Furthermore, the embodiments described above (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0165] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0166] This embodiment includes the following configurations, methods, programs, and storage media. (Composition 1) An acquisition means for obtaining the position and feature quantities of each of multiple subjects detected from a captured image, A tracking means that tracks one of the multiple subjects in association with a tracking subject, based on the position and feature quantities of each of the multiple subjects acquired by the acquisition means, It has, The aforementioned tracking means is If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. An information processing device characterized by the following: (Configuration 2) The information processing device according to configuration 1, characterized in that the degree of overlap is the value obtained by dividing the area of ​​the region in which the first subject and the second subject overlap in the captured image by the area of ​​the region of the first subject in the captured image. (Composition 3) The information processing device according to configuration 1 or 2, characterized in that the acquisition means detects one or more joint points constituting each subject and acquires the position of each subject from the detected one or more joint points. (Composition 4) The acquisition means acquires the position of each subject based on the skeleton of each subject determined by the one or more joint points detected, The degree of overlap is the degree of overlap between the tracked subject and the subject whose skeleton intersects with the other. The information processing apparatus according to configuration 3, characterized by the features described herein. (Composition 5) The information processing device according to any one of configurations 1 to 4, characterized in that the degree of overlap is based on one or more of the area, width, and height of each subject in the captured image. (Composition 6) The information processing apparatus according to any one of configurations 1 to 5, characterized in that, when the degree of overlap between the tracked subject and the other subject is the first value, the contribution rate of the feature similarity is increased compared to the contribution rate of the position similarity, thereby performing the first correspondence. (Composition 7) The information processing apparatus according to any one of configurations 1 to 6, characterized in that, when the degree of overlap between the tracked subject and the other subject is the second value, the contribution rate of the positional similarity is made higher than the contribution rate of the feature similarity to perform the second correspondence. (Composition 8) The information processing apparatus according to configuration 7, characterized in that the tracking means calculates the similarity of the positions based on one or more points in the region of each subject and performs the second correspondence. (Composition 9 places) The information processing device according to configuration 8, characterized in that the one or more points are the upper point or the central point in the region of each subject. (Composition 10) The information processing apparatus according to any one of configurations 1 to 9, characterized in that the tracking means performs the second correspondence on subjects among the plurality of subjects that cannot be corresponded by the first correspondence. (Method 1) An acquisition step to obtain the position and feature quantities of each of the multiple subjects detected from the captured image, A tracking step in which, based on the position and feature quantities of each of the multiple subjects acquired in the acquisition step, one of the multiple subjects is associated with a tracking subject that is the target of tracking, and is tracked. It has, The aforementioned tracking step is, If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. A control method for an information processing device characterized by the following features. (Composition 11) A shooting device that controls the shooting direction horizontally or vertically by a rotation mechanism, An acquisition means for obtaining the position and feature quantities of each of multiple subjects detected from a captured image, A tracking means that tracks one of the multiple subjects in association with a tracking subject, based on the position and feature quantities of each of the multiple subjects acquired by the acquisition means, A determination means for determining control values ​​for controlling the rotation mechanism in order to photograph the subject being tracked, A control means for controlling the rotation mechanism based on the control value, It has, The aforementioned tracking means is If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. A photographic device characterized by the following features. (Composition 12) An information processing system having an imaging device and an information processing device that control the imaging direction horizontally or vertically by a rotation mechanism, The aforementioned information processing device is An acquisition means for obtaining the position and feature quantities of each of multiple subjects detected from a captured image, A tracking means that tracks one of the multiple subjects in association with a tracking subject, based on the position and feature quantities of each of the multiple subjects acquired by the acquisition means, A determination means for determining control values ​​for controlling the rotation mechanism in order to photograph the subject being tracked, Output means for outputting the determined control value to the imaging device, It has, The imaging device has control means for controlling the rotation mechanism based on the outputted control value, The aforementioned tracking means is If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. An information processing system characterized by the following: (program) A program for causing a computer to function as one of the means of an information processing device described in any of configurations 1 to 10. (storage medium) A storage medium for storing programs that cause a computer to function as one of the means of an information processing device described in any of configurations 1 to 10. [Explanation of Symbols]

[0167] 100 Camera, 101 CPU, 102 RAM, 103 ROM

Claims

1. An acquisition means for obtaining the position and feature quantities of each of multiple subjects detected from a captured image, A tracking means that tracks one of the multiple subjects in association with a tracking subject, based on the position and feature quantities of each of the multiple subjects acquired by the acquisition means, It has, The aforementioned tracking means is If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. An information processing device characterized by the following:

2. The information processing apparatus according to claim 1, characterized in that the degree of overlap is the value obtained by dividing the area of ​​the region in which the first subject and the second subject overlap in the captured image by the area of ​​the region of the first subject in the captured image.

3. The information processing apparatus according to claim 1, wherein the acquisition means detects one or more joint points constituting each subject and acquires the position of each subject from the detected one or more joint points.

4. The acquisition means acquires the position of each subject based on the skeleton of each subject determined by the one or more joint points detected, The degree of overlap is the degree of overlap between the tracked subject and the subject whose skeleton intersects with the other. The information processing apparatus according to claim 3.

5. The information processing device according to claim 1, characterized in that the degree of overlap is based on one or more of the area, width, and height of each subject in the captured image.

6. The information processing apparatus according to claim 1, characterized in that, when the degree of overlap between the tracked subject and the other subject is the first value, the tracking means increases the contribution rate of the feature similarity to the contribution rate of the position similarity, thereby performing the first correspondence.

7. The information processing apparatus according to claim 1, characterized in that, when the degree of overlap between the tracked subject and the other subject is the second value, the tracking means makes the contribution rate of the positional similarity higher than the contribution rate of the feature similarity, thereby performing the second correspondence.

8. The information processing apparatus according to claim 7, characterized in that the tracking means calculates the similarity of the positions based on one or more points in the region of each subject and performs the second correspondence.

9. The information processing apparatus according to claim 8, characterized in that the one or more points are upper points or central points in the region of each subject.

10. The tracking means performs a second association on subjects among the plurality of subjects that cannot be associated by the first association, characterized in that it is the information processing according to claim 1. equipment.

11. An acquisition step to obtain the position and feature quantities of each of the multiple subjects detected from the captured image, A tracking step in which, based on the position and feature quantities of each of the multiple subjects acquired in the acquisition step, one of the multiple subjects is associated with a tracking subject that is the target of tracking, and is tracked. It has, The aforementioned tracking step is, If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. A control method for an information processing device characterized by the following features.

12. A shooting device that controls the shooting direction horizontally or vertically by a rotation mechanism, An acquisition means for obtaining the position and feature quantities of each of multiple subjects detected from a captured image, A tracking means that tracks one of the multiple subjects in association with a tracking subject, based on the position and feature quantities of each of the multiple subjects acquired by the acquisition means, A determination means for determining control values ​​for controlling the rotation mechanism in order to photograph the subject being tracked, A control means for controlling the rotation mechanism based on the control value, It has, The aforementioned tracking means is If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. A photographic device characterized by the following features.

13. An information processing system having an imaging device and an information processing device that control the imaging direction horizontally or vertically by a rotation mechanism, The aforementioned information processing device is An acquisition means for obtaining the position and feature quantities of each of multiple subjects detected from a captured image, A tracking means that tracks one of the multiple subjects in association with a tracking subject, based on the position and feature quantities of each of the multiple subjects acquired by the acquisition means, A determination means for determining control values ​​for controlling the rotation mechanism in order to photograph the subject being tracked, Output means for outputting the determined control value to the imaging device, It has, The imaging device has control means for controlling the rotation mechanism based on the outputted control value, The aforementioned tracking means is If the degree of overlap between the tracked subject and other subjects is a first value, a first correspondence is performed using the similarity of the feature quantities to associate one of the multiple subjects with the tracked subject. If the degree of overlap between the tracked subject and other subjects is a second value indicating less overlap than the first value, a second correspondence is performed using the positional similarity to associate one of the multiple subjects with the tracked subject. An information processing system characterized by the following:

14. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 10.

15. A storage medium for storing a program that causes a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 10.

Images