Information processing apparatus, system, and method, computer program product, and storage medium

Abstract

The invention relates to an information processing apparatus, system, and method, a computer program product, and a storage medium. The information processing apparatus includes: a detection unit configured to detect a plurality of subjects from an image; a selection unit configured to select a subject as a tracking target; a tracking unit configured to track the subject selected as the tracking target by the selection unit, using information on the plurality of subjects detected by the detection unit; a counting unit configured to count a number of subjects currently being tracked by the tracking unit; and a notification unit configured to notify a tracking state of the tracking unit. The notification unit notifies the tracking state according to the number of subjects selected by the selection unit and the number of subjects counted by the counting unit.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and a tracking method for tracking a plurality of subjects. Background Art

[0002] In video image production, an automatic tracking control technique automatically performs pan, tilt, and zoom (PTZ) control on a camera device to place a specific subject at a desired position within the camera view.

[0003] Japanese Patent Application Laid-Open No. 2009-218719 discusses the following technique: showing the moving direction and moving speed of a tracked subject, and displaying a warning if a behavior that may deviate from the view is detected.

[0004] However, Japanese Patent Application Laid-Open No. 2009-218719 does not disclose control for simultaneously tracking a plurality of subjects. Summary of the Invention

[0005] According to an aspect of the present invention, there is provided an information processing apparatus including: a detection unit configured to detect a plurality of subjects from an image; a selection unit configured to select a subject as a tracking target; a tracking unit configured to track the subject selected as the tracking target by the selection unit using information related to the plurality of subjects detected by the detection unit; a counting unit configured to count the number of subjects currently being tracked by the tracking unit; and a notification unit configured to notify a tracking state of the tracking unit. The notification unit notifies the tracking state based on the number of subjects selected by the selection unit and the number of subjects counted by the counting unit.

[0006] Other features of the present invention will become apparent from the following description of embodiments with reference to the accompanying drawings. Brief Description of the Drawings

[0007] Figure 1 An example of a system configuration is shown.

[0008] Figure 2 An example of a hardware configuration of a device included in the system is shown.

[0009] Figure 3 It is a flowchart showing an example of a basic operation of a camera device.

[0010] Figure 4 It is a flowchart showing an example of a basic operation of an information device.

[0011] Figures 5A to 5C Shows an example of information display in the basic operation.

[0012] Figures 6A to 6C Shows an example of subject information in the basic operation.

[0013] Figure 7A and Figure 7B Is a flowchart showing an example of the control process according to the first embodiment.

[0014] Figures 8A to 8F Shows an example of information display according to the first embodiment.

[0015] Figures 9A to 9C Shows an example of subject information according to the first embodiment.

[0016] Figure 10 Shows an example of the management of the tracking target according to the second embodiment.

[0017] Figures 11A to 11D Shows an example of information display according to the second embodiment.

[0018] Figures 12A to 12C Shows an example of subject information according to the second embodiment.

[0019] Figures 13A to 13D Shows an example of information display considering priority according to the second embodiment.

[0020] Figures 14A to 14D Shows an example of subject information considering priority according to the second embodiment.

[0021] Figure 15 Shows an example of the lamp display mode according to the third embodiment.

[0022] Figure 16 Shows an example of the hardware configuration according to the fourth embodiment.

[0023] Figure 17 Is a flowchart showing an example of the control process according to the fourth embodiment. Detailed Description of the Invention

[0024] Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. The following embodiments do not limit the present invention to the scope of the appended claims. Although multiple features are described in the embodiments, not all features will be used in the present invention, and any combination of multiple features can be used. In the drawings, the same or similar components are assigned the same reference numerals, and their repeated description will be omitted.

[0025] <System Configuration>

[0026] Figure 1 An example configuration of an information processing system that performs processing according to a first embodiment is shown. Referring to Figure 1 , the information processing system includes a pan-tilt-zoom (PTZ) camera 100 and a personal computer (PC) 200 as information processing devices. The PTZ camera 100 and the PC 200 are connected to a network formed on a local area network (LAN) 300 to form a network in which both wired and wireless devices can communicate with each other via a communication protocol.

[0027] The PTZ camera 100 is a camera device that can capture images of a tracking target (subject) and the area around the subject and output the captured images to the PC 200 and external devices. The PTZ camera 100 according to the present embodiment includes a drive unit 109 (described below), and the drive unit 109 is provided with a mechanism for performing pan and tilt operations to change the imaging direction. The PTZ camera 100 further includes an inference unit 111 (described below), and the inference unit 111 is used to infer the position of the subject on the captured image.

[0028] The PC 200 accesses the PTZ camera 100 via the LAN 300 to obtain the images output by the PTZ camera 100, performs imaging control based on the user's operations, and sets various imaging conditions. The images according to the present embodiment include moving images and still images, and the present embodiment is applicable to both types of images.

[0029] Figure 2 The configurations of the PTZ camera 100 and the PC 200 included in the system are shown. The configuration of each device will be described.

[0030] The PTZ camera 100 according to the present embodiment includes a central processing unit (CPU) 101, a read-only memory (ROM) 102, a random access memory (RAM) 103, an image output interface (I / F) 104, and a network I / F 105. The PTZ camera 100 further includes an image processing unit 106, an image sensor 107, a drive I / F 108, a drive unit 109, an inference unit 111, and an internal bus 110 for communicably connecting the above components.

[0031] The CPU 101 generally controls the device by controlling different components of the PTZ camera 100.

[0032] The ROM 102 is a non-volatile storage device represented by a flash memory, a hard disk drive (HDD), a solid state drive (SSD), and a secure digital (SD) card. The ROM 102 serves as a permanent storage area for storing an operating system (OS), various programs, and various types of data, and also serves as a temporary storage area for storing various types of data.

[0033] The RAM 103 is a volatile high-speed storage device represented by a dynamic random access memory (DRAM) loaded with an OS, various programs, and various types of data. The RAM 103 also serves as a working area for the OS and various programs.

[0034] The image output I / F 104 is an interface for outputting an image captured by an image sensor 107 (described below) to an external device, and includes a serial digital interface (SDI) and a high-definition multimedia interface

[0035] The network I / F 105 is an interface for connecting to the LAN 300, and communicates with an external device such as a PC 200 via a communication medium such as and the like.

[0036] The image processing unit 106 connected to the image sensor 107 performs various types of image processing (e.g., defect correction, noise reduction (NR) processing, and color conversion processing) on the image data acquired from the image sensor 107 based on an instruction from the CPU 101, converts the image data into a predetermined format, and performs compression processing. The processed image data is stored in the RAM103.

[0037] The image sensor 107 including a charge-coupled device (CCD) sensor or a complementary metal oxide semiconductor (CMOS) sensor functions as an imaging unit in the PTZ camera 100. The image sensor 107 photoelectrically converts a subject image formed by an imaging optical system (not shown) to generate image data.

[0038] According to the present embodiment, although the image data is output to the image processing unit 106 as a digital signal by an analog-to-digital (A / D) conversion circuit included in the image sensor 107, the image data may also be output as an analog signal. The image sensor 107 and the image processing unit 106 may be integrated into a stacked chip configuration. According to the present embodiment, although the imaging optical system for capturing a subject image by the imaging element is also integrally formed with the image sensor 107, the imaging optical system may be configured to be attachable, detachable, and replaceable with respect to the image sensor 107. According to the present embodiment, in some cases, the imaging optical system and the image sensor 107 are collectively referred to as an imaging unit.

[0039] The drive I / F 108 is an interface for transmitting an instruction received from the CPU 101 to the drive unit 109.

[0040] The drive unit 109 is a mechanism and an optical system for changing the imaging direction of the PTZ camera 100. According to the present embodiment, the drive unit 109 changes the imaging direction by rotatably driving the image sensor 107.

[0041] The drive unit 109 is composed of a mechanical drive system and a motor as a drive source. The drive unit 109 performs rotational drives such as translation and pitching operations to change the imaging direction with respect to the horizontal and vertical directions based on instructions received from the CPU 101 via the drive I / F 108. Through an imaging optical system provided with a magnifying lens (also called a zoom lens), the drive unit 109 can perform zoom control by moving the zoom lens in the optical axis direction, thereby optically changing the imaging angle of view.

[0042] The inference unit 111 performs inference processing using a learned inference model and inference parameters according to an inference program. The inference processing of the inference unit 111 can be performed by a computational processing device dedicated to image processing and inference processing of a graphics processing unit (GPU). The GPU is a processor capable of performing a large number of sum-product calculations and capable of performing neural network matrix calculations in a short time. The inference processing of the inference unit 111 can be performed by a reconfigurable logic circuit such as a field-programmable gate array (FPGA). The inference unit 111 can perform inference processing in cooperation with the CPU 101.

[0043] The lamp 112, also called a tally lamp, is a light source such as a light-emitting diode (LED) that indicates the current control received by the PTZ camera 100. The CPU 101 receives an instruction from the outside via the network I / F 105 to change the display mode. The CPU 101 performs status display by color or periodic blinking. For example, red indicates that the video image is for broadcasting and recording, while green indicates the preview state.

[0044] The PC 200 according to the present embodiment includes a CPU 201, a ROM 202, a RAM 203, a network I / F 204, a display unit 205, a user input I / F 206, and an internal bus 207 for communicably connecting these components.

[0045] The CPU 201 controls the components of the PC 200 to perform overall control of the device.

[0046] The ROM 202 is a non-volatile storage device represented by a flash memory, an HDD, an SSD, and an SD card. The ROM 202 is used as a permanent storage device for storing an OS, various programs, and various types of data, and is also used as a temporary storage area for storing various types of data.

[0047] The RAM 203 is a volatile high-speed storage device represented by a DRAM in which an OS, various programs, and various types of data are loaded. The RAM 103 is also used as a working area for the OS and various programs.

[0048] The network I / F 204 is an interface for connecting to the LAN 300, and communicates with imaging devices such as the PTZ camera 100 and external devices such as a server via a communication medium such as Ethernet.

[0049] The display unit 205 displays the image acquired from the PTZ camera 100 and the setting screen of the PC 200. For example, the display unit 205 is a liquid crystal panel or an organic electroluminescence (EL) panel. Although the PC 200 includes the display unit 205 as an example, the PC 200 and the display unit 205 may be configured as different components, such as setting a display monitor that only displays captured images and the PC 200 as different components.

[0050] The user input I / F 206 includes input devices (operation units) such as a keyboard, a pointing device (mouse), a touch panel, and a switch, and receives instructions from the user for the PC 200. The keyboard may be a software keyboard. The CPU 201 monitors the user input I / F 206 and, when detecting an operation by the user on the user input I / F 206, performs processing in response to the detected operation.

[0051] <Description of basic operations in automatic tracking control and subject selection>

[0052] The basic control performed by the system will be described. The basic control includes automatic tracking control for controlling the PTZ camera 100 to track a subject, and subject selection for selecting a tracking target subject for the PTZ camera 100 based on a user operation received by the PC 200.

[0053] will be referred to Figure 3 to FIG. 6 to describe the automatic tracking control.

[0054] Figure 3 shows the control flow of the PTZ camera 100, and more specifically, shows a series of processes for controlling the PTZ camera 100 based on the subject position detected from the captured image.

[0055] This control flow starts when the CPU 101 of the PTZ camera 100 receives a control command for performing automatic tracking control via the network I / F 105.

[0056] In step S301, the CPU 101 determines whether a control command or an end command has been received via the network I / F 105, and stores the received control command in the RAM 103. If the CPU 101 determines that a control command has been received (the result of the confirmation operation status in step S301 is "yes"), the CPU 101 stores the received control command in the RAM 103. Then, the process proceeds to step S302. If the CPU 101 determines that an end command has been received (the result of the confirmation operation status in step S301 is "no"), the CPU 101 completes the control.

[0057] In step S302, the CPU 101 acquires the image data stored in the image processing unit 106 from the RAM 103.

[0058] In step S303, the CPU 101 determines information related to the features and positions of the subjects in each frame of the captured image data, and stores the information in the RAM 103. More specifically, the CPU 101 inputs the image data acquired from the RAM 103 into the inference unit 111. Then, the CPU 101 stores the features of the subjects inferred by the inference unit 111 and the position information about the subject images in the RAM 103. The inference unit 111 includes a learning model created using machine learning techniques such as deep learning, receives an image as input data, and outputs an identifier (ID) for identification and position information as subject information. The position information is described as information related to the upper left point, width, height, and center of gravity coordinates of the rectangle defining the subject on the image. However, the present invention is not limited thereto.

[0059] Figure 6A The table shown in shows the detected subject information. Figure 5A The information to be displayed on the screen of the display unit 205 in the operation of the PC 200 (described below) is illustrated as an example having three different subjects in the perspective. In Figure 5A the position information related to the subjects output in step S303 is superimposed as subject frames 503 to 505. The subject frame 503 corresponds to Figure 6A ID = 1 in . (x -1, (y1) indicates the upper left point of the subject frame 503, (w1, h1) indicates the size of the subject frame 503, and (gx1, gy1) indicates the coordinates of the center of gravity. When the subject frame is displayed on the PC 200, the CPU 201 converts the coordinates on the PTZ camera 100 into a coordinate system on the display. In this embodiment, the position information related to the subject is determined via the inference unit 111, and the present invention is not limited thereto as long as the position information related to the subject can be determined. For example, the CPU 101 can obtain the position information from the wireless communication terminal held by the subject and use the position information as the position information related to the subject. The CPU 101 stores the output subject information in the RAM 103. Then, the process proceeds to step S304.

[0060] In step S304, the CPU 101 determines whether it has received the subject selection information as a control command from the PC 200 via the network I / F 105. If it has received (Yes in step S304), the CPU 101 stores the subject selection information received from the PC 200 in the RAM 103. The following describes this embodiment on the premise that the subject selection information refers to the coordinates (area) in the viewing angle. In the case of selecting any subject, the process will continue to step S305 regardless of the subject selection information.

[0061] In step S305, the CPU 101 reads the subject information output in step S303 and the subject selection information received in step S304 from the RAM 103. The CPU 101 compares the position information in the subject information with the coordinate information included in the subject selection information to confirm whether the coordinate information is included in the position information related to the subject. The following will refer to Figures 5A to 5C Describe a specific example. If the coordinates specified as the subject selection information indicate Figure 5A the point 506 shown in, then the point 506 is included in the area represented by the position information (i.e., the upper left point and the frame) displayed as the subject frame 503. Therefore, the CPU 101 determines the subject corresponding to the subject frame 503 to be selected. As Figure 6B shown in, the CPU 101 stores the subject information with information indicating whether the subject is a tracking target in the RAM 103 and transmits the subject information to the PC 200. Then, the process proceeds to step S306.

[0062] In step S306, the CPU 101 calculates the control position information (control information) in the automatic tracking control and stores the control position information in the RAM 103. The control position information refers to information (imaging parameters) for controlling (moving) the image sensor 107 to any desired point, such as the pan angle, the tilt angle, and the zoom perspective. The CPU 101 calculates the pan angle, the tilt angle, the zoom perspective, and the angular velocity when moving the center-of-gravity coordinates of the tracking target subject selected in step S305 to the center position of the perspective, and stores the calculation result as the control position information in the RAM 103. The state where no subject is selected is equivalent to the state where there is a lack of control position information (the perspective is not controlled). When receiving the control position information as a control command from the outside in step S301, the CPU 101 stores the information in the RAM 103 that gives priority to the control command, enabling position control from the outside.

[0063] In step S307, the CPU 101 reads the control position information stored in the RAM 103. Based on the control position information, the CPU 101 extracts the drive parameters (control details) of the drive unit 109, enabling pan, tilt, and zoom control in a desired direction at a desired speed. More specifically, the drive parameters are used to control the motors included in the drive unit 109. The amount of the operation based on the control position information can be converted into drive parameters with reference to a conversion table pre-stored in the RAM 103.

[0064] In step S308, the CPU 101 controls the drive unit 109 via the drive I / F 108 based on the extracted drive parameters, and the drive unit 109 performs a rotation operation based on the drive parameters of the PTZ camera 100 to perform pan, tilt, and zoom operations. This control flow calculated based on the subject selection information received in step S304 enables the PTZ camera 100 to operate according to the subject selection of the user sequentially transmitted from the PC 200.

[0065] Now, the processing of controlling the PTZ camera 100 based on the user's operation on the PC 200 shown in Figure 4 will be described. The PTZ camera 100 is controlled based on the control command transmitted from the PC 200. The operation of the PTZ camera 100 is the operation when the CPU 101 receives the subject selection information in the control flow shown in Figure 3 steps S304 and S305. Therefore, the description of the operation will be omitted. This control flow starts when the CPU 201 detects via the user input I / F 206 the operation for causing the menu screen to display the image captured by the PTZ camera 100 and Figure 5AStart when operating on the information shown. When the control flow starts, the CPU 201 of the PC 200 displays on the display unit 205 Figure 5A the menu screen 500, the tracking control state 501, and the camera image 502 received from the PTZ camera 100 shown in. This embodiment will be described on the premise that no tracking target is selected (i.e., tracking control has not started).

[0066] In step S401, the CPU 201 of the PC 200 detects an operation by the user to close the menu screen via the user input I / F 206. If no operation by the user is detected (Yes in step S401), the process proceeds to step S402. If an operation by the user is detected (No in step S401), the process exits the control flow.

[0067] In step S402, the CPU 201 of the PC 200 transmits a control command for acquiring information to the PTZ camera 100 via the network I / F 204. When the CPU 101 of the PTZ camera 100 detects the received control command, the CPU 101 reads the subject information output in Figure 3 step S303 of, and then transmits the information to the PC 200 via the network I / F 105. The CPU 201 of the PC 200 calculates coordinates based on the received subject information to superimpose the subject frames 503 to 505 on the camera image 502, and updates the menu screen on the RAM 203. Although the tracking control has not started, the user can find the subject information recognized by the PTZ camera 100. Then, the process proceeds to step S403.

[0068] In step S403, the CPU 201 of the PC 200 detects via the user input I / F 206 whether the user is performing a subject selection operation. The user can input an operation by specifying a point on the camera image 502 using a mouse operation or touching the camera image 502 displayed on the touch panel. However, the method of input operation is not limited to this. Similar to Figure 3 step S304 in the above control flow in, if the user specifies the point 506, the CPU 201 converts the point 506 on the camera image 502 into coordinates in the camera view, and transmits the coordinates to the PTZ camera 100. Then, the process proceeds to step S404. If a tracking target (described below) exists, the process will continue to step S404 regardless of the user's operation.

[0069] In step S404, the CPU 201 of the PC 200 acquires the tracking state from the PTZ camera 100. The tracking state refers to Figure 3whether the ID read in step S305 of the above control flow is information related to the tracking target transmitted from the PTZ camera 100 ( Figure 6B as shown). If the information has been provided in step S402, the CPU 101 can read the information from the RAM 103. When the CPU 201 checks the tracking target ID, the PTZ camera 100 changes the display of the tracking control state 511 to "tracking". The display change can be performed by changing the string or updating the color and mode display. According to this embodiment, the string and color are updated. The CPU 101 also shows the tracking target ID to the user by changing the display format of the subject frame corresponding to the tracking target ID. More specifically, the CPU 101 updates the subject frame 503 to the subject frame 513. At the same time, as shown in the subject frames 514 and 515, the CPU 101 does not update the subject frame 504 or 505 respectively, and the CPU 101 shows the user the state that the CPU 101 has detected the subject but has not tracked the subject. The display format of the frame is an example, and the method of changing the color and mode is not limited to this embodiment. The CPU 201 updates the menu screen of the RAM 203 based on this information. Then, the process proceeds to step S405.

[0070] In step S405, the CPU 201 of the PC 200 displays the updated menu screen 510 on the RAM 203 by controlling the display unit 205. When this control flow starts, the Figure 5A screen shown appears. After the subject is selected, Figure 5A the screen in Figure 5B changes to the Figure 5C screen shown. As shown in Figure 6C , if the subject designated as the tracking target does not match the viewing angle of the PTZ camera 100, information indicating the loss of position information is transmitted from the PTZ camera 100 as shown in

[0071] Figure 6C . For example, if the coordinates of the tracking target are not found, the CPU 101 changes the string and color display to indicate the loss state as shown in the tracking control state 521. This can notify the user of the loss state of the tracking target.

[0071] If the point specified by the user in step S403 is already the tracking target subject, the CPU 101 can change the tracking target by selecting another subject, thereby canceling the tracking target. Although the tracking operation starts with subject selection, the user can change the settings to give a clear indication of starting and stopping the tracking operation.

[0072] The above basic operations enable automatic tracking control of the PTZ camera 100 according to the subject selection by the user in the control flows of the PTZ camera 100 and the PC 200.

[0073] <Description of Tracking Operation When Selecting Multiple Subjects>

[0074] Now, the tracking operation when selecting multiple subjects, which is a characteristic operation of the present invention, will be described. With reference to Figures 7A to 9C , the operation of the PTZ camera 100 that is different from the control flow shown in Figure 3 will be described, and the operation of the PC 200 that is different from the control flow shown in Figure 4 will be described.

[0075] Figure 7A The control flow shown in Figure 3 starts as shown in the control flow. The operations in steps S701 to S703, S705, and S707 to S709 are performed as the operations in steps S301 to S303, S305, and S306 to S308, respectively.

[0076] In step S705, as in step S305, the CPU 101 of the PTZ camera 100 transmits the subject information of three different subjects from the PTZ camera 100. However, the PC 200 displays the menu screen shown in Figure 8A instead of the screen shown in Figure 5A . More specifically, the CPU 101 displays the selected target number 801 as the number of currently selected targets, the tracking target number 802 for obtaining position information, and the detected number 803 as the number of currently detected subjects. Since no subject is selected, the CPU 201 determines that the selected target number 801 and the tracking target number 802 are 0 and the detected number 803 is 3, and updates the menu screen on the RAM 203.

[0077] In step S704, the CPU 101 of the PTZ camera 100 determines whether subject selection information as a control command is received from the PC 200 via the network I / F 105. As in the basic operation, if the point 804 is specified, the CPU 101 of the PTZ camera 100 transmits the information shown in Figure 6B in step S705. In this case, the CPU 101 displays the selected subject frame 813 as the state of the tracking target, and at the same time updates the menu screen so that the selected target number 811 and the tracking target number 812 are 1.

[0078] In step S706, the CPU 101 of the PTZ camera 100 compares the tracking state received from the PTZ camera 100 with the selected target number. The process in step S706 is equivalent to the control flow shown in Figure 7B . With the premise that the process enters step S710, it will be described with reference to Figure 7B . At this time, as shown in Figure 6BAs shown in [figure], the number of tracking targets corresponding to the acquired position information is 1, and the number of selected targets is also 1. When the CPU 101 of the PTZ camera 100 finishes the processing in step S706, the CPU 101 transmits the tracking control state to the PC 200 via the network I / F 105.

[0079] In step S710, the CPU 101 determines whether the number of selected targets is 0. As described above, since the number of selected targets is 1 at this time, the processing proceeds to step S711. The number of selected targets is 0 at the start of the control flow shown in [figure], and remains 0 when no subject is selected in step S704. In this case, the processing proceeds to step S713. In step S713, the CPU 101 transmits information such that the tracking control state 800 becomes "stopped" as shown in [figure], and then updates the menu screen on the RAM 203. Figure 7A In step S711, the CPU 101 determines whether the number of tracking targets is 0. Since the number of tracking targets is 1 at this time, the processing proceeds to step S712. The processing that proceeds to step S714 when the number of tracking targets becomes 0 will be described below. Figure 8A In step S712, the CPU 101 compares the number of tracking targets with the number of selected targets. If the number of selected targets matches the number of tracking targets ( "yes" in step S712), the processing proceeds to step S715. If the number of selected targets does not match the number of tracking targets ( "no" in step S712), the processing proceeds to step S716. Since both numbers are 1 at this time, the processing proceeds to step S715. In step S715, the CPU 101 transmits information such that the tracking control state 810 becomes "tracking". The CPU 201 of the PC 200 updates the menu screen on the RAM 203 as shown in [figure]. Step S716 will be described below. After completing steps S713 to S716, the processing exits the control flow shown in [figure] and enters step S706 of the control flow shown in [figure].

[0080] After the above operations are completed, the menu screen shown in [figure] appears. Then, when the specified point 814 is designated, the CPU 101 performs a similar control flow. Accordingly, the number of selected targets 821 and the number of tracking targets 822 become 2, the display of the subject frame 823 is updated, and then the menu screen shown in [figure] appears. When the specified point 824 is designated, the number of selected targets 831 and the number of tracking targets 832 become 3, the display of the subject frame 833 is updated, and the menu screen shown in [figure] appears.

[0081] In step S712, the CPU 101 compares the number of tracking targets with the number of selected targets. If the number of selected targets matches the number of tracking targets ( "yes" in step S712), the processing proceeds to step S715. If the number of selected targets does not match the number of tracking targets ( "no" in step S712), the processing proceeds to step S716. Since both numbers are 1 at this time, the processing proceeds to step S715. In step S715, the CPU 101 transmits information such that the tracking control state 810 becomes "tracking". The CPU 201 of the PC 200 updates the menu screen on the RAM 203 as shown in [figure]. Step S716 will be described below. After completing steps S713 to S716, the processing exits the control flow shown in [figure] and enters step S706 of the control flow shown in [figure]. Figure 8B After the above operations are completed, the menu screen shown in [figure] appears. Then, when the specified point 814 is designated, the CPU 101 performs a similar control flow. Accordingly, the number of selected targets 821 and the number of tracking targets 822 become 2, the display of the subject frame 823 is updated, and then the menu screen shown in [figure] appears. When the specified point 824 is designated, the number of selected targets 831 and the number of tracking targets 832 become 3, the display of the subject frame 833 is updated, and the menu screen shown in [figure] appears. Figure 7B the control flow shown in [figure] and enters Figure 7A step S706 of the control flow shown in [figure].

[0082] When the above operations are completed, the menu screen shown in [figure] appears. Figure 8B Then, when the specified point 814 is designated, the CPU 101 performs a similar control flow. Accordingly, the number of selected targets 821 and the number of tracking targets 822 become 2, the display of the subject frame 823 is updated, and then the menu screen shown in [figure] appears. When the specified point 824 is designated, the number of selected targets 831 and the number of tracking targets 832 become 3, the display of the subject frame 833 is updated, and the menu screen shown in [figure] appears. Figure 8C When the specified point 824 is designated, the number of selected targets 831 and the number of tracking targets 832 become 3, the display of the subject frame 833 is updated, and the menu screen shown in [figure] appears. Figure 8DThe menu screen shown in. When Figure 8D the menu screen shown in appears, in Figure 9A the subject information transmitted by the PTZ camera 100 and received by the PC 200 is shown.

[0083] Control the PTZ camera 100 to track the subject selected as the tracking target. Taking the state in Figure 9A as an example, change the method of calculating the control position information in step S306. Although the barycentric coordinates of the selected subject are calculated in the case of selecting one subject, the PTZ camera 100 can also be rotated to the center of the selected subject obtained by calculating the control position information by taking the barycenter of the barycentric coordinates of three subjects. For zooming, in addition to the barycenter, the width and height of all subject frames can also be considered and calculated based on the ratio to the viewing angle. However, the calculation method is not limited to this.

[0084] Assume a situation where a subject moves out of the viewing angle of the PTZ camera 100.

[0085] In Figure 8E the case of the camera image 843 shown in, the subject information transmitted in step S705 is as shown in Figure 9B . In this case, in step S712, the number of tracking targets with position information is 2 and the number of selected targets is 3. Therefore, the process proceeds to step S716. In step S716, the CPU 101 transmits information to display "Partially Lost" and updates the menu screen on the RAM 203. As shown in Figure 8E , the tracking control state 840 is displayed as "Partially Lost", and the number of tracking targets 841 and the detection number 842 become 2. The user can recognize that the specified number of targets cannot be tracked.

[0086] In addition, assume a state where all subjects move out of the viewing angle of the PTZ camera 100 and an untracked subject 854 enters the viewing angle of the PTZ camera 100. In Figure 8F the case of the camera image 853 shown in, the subject information is shown in Figure 9C . At this time, since the number of tracking targets with position information is 0 (Yes in step S711), the process proceeds to step S714. In step S714, the CPU 101 transmits information to display "Lost" and updates the menu screen on the RAM 203. As shown in Figure 8F , since the tracking control state 850 is displayed as "Lost", the number of tracking targets 852 is 0, and the detection number 851 is 1, the user can recognize that the targets corresponding to the specified number are not being tracked.

[0087] If the subject is not a tracking target, a frame corresponding to a "stop" frame is displayed as a detection result. Therefore, unless the subject moves out of the viewing angle, the subject may be in a state where it cannot be detected based on its pose or orientation. The user can explicitly specify the coordinates or can automatically determine as a tracking target a subject that has been continuously photographed within the viewing angle for a certain period of time (a predetermined period of time or a predetermined number of detections), and then can put the subject in a selected state. If the subject deviates from the viewing angle for a certain period of time (a predetermined period of time or a predetermined number of detections), the subject can be excluded from the tracking targets.

[0088] In this embodiment, an example of the PTZ camera 100 is described. However, the form of this embodiment is not limited thereto. This embodiment is also applicable to, for example, a device that can detect multiple subjects from an image and control the PTZ camera 100 based on the detection results. Specific examples of such devices include edge devices and a PC provided with an image input unit, a network communication unit, and a GPU.

[0089] According to the first embodiment described above, a camera system that tracks multiple subjects can be driven in consideration of an increase and a decrease in the number of tracking targets, and can show a change in the tracking state to the user.

[0090] A second embodiment will be described. According to the first embodiment, multiple subjects are selected by the user sequentially selecting the subjects. However, it can be imagined that pre-selecting predetermined subjects can save time and eliminate the trouble of sequential selection. Therefore, according to this embodiment, with reference to Figures 9A to 9C 、 Figure 10 、 Figures 11A to 11D 、 Figures 12A to 12C 、 Figures 13A to 13D and Figures 14A to 14D it will be described how to show to the user in a method of managing multiple subjects.

[0091] Figure 10 An example of subject management is shown. The pool 1000 indicates a state where information about features (such as faces) of subjects A to E previously registered by the user is being managed. In the registration method, the CPU 201 transmits, via the network I / F 204, information input by the user via a user interface (UI) (not shown) using the display unit 205 and the user input I / F 206 of the PC 200. The CPU 101 of the PTZ camera 100 receives the information via the network I / F 204 and stores the information in the ROM 102 or the RAM 103. The method is not limited thereto. Other methods can be implemented in the form of storing the sequentially selected subjects one by one as in the first embodiment, or can be implemented in the form of transmitting separately generated information to the PTZ camera 100. The user has registered the subjects selected by the user from the pool 1000 as the first group 1001 and the second group 1002.

[0092] The first group 1001 indicates the states of selecting subjects A to C, and the states are close to the selection state with a target number of 3 in the first embodiment.

[0093] In a system configuration similar to the first embodiment, the changes in the control flowchart will be described. Figure 3 and Figure 4 the changes in the control flowchart. As Figure 11A shown, three subjects are detected, and the selected subject number 1100 is in the "not selected" state. In this case, the CPU 201 of the PC 200 displays on the display unit 205 the pool 1000, the first group 1001, and the second group 1002 shown without displaying the target selected in step S403. Figure 10 shown, without displaying the target selected in step S403.

[0094] If the user selects the first group 1001 via the user input I / F 206, the CPU 201 displays Figure 11B the selected group 1110, and transmits the selection information to the PTZ camera 100. The CPU 101 of the PTZ camera 100 receives the subject selection information in step S304 as information related to the selection of the first group 1001 rather than as coordinates according to the first embodiment. In step S305, the CPU 101 searches for the targets corresponding to subjects A to C associated with the first group 1001 in the subject information rather than coordinates. According to this embodiment, if the target matches subjects A to C, an association with subjects A to C is established in the subject information as Figure 12A shown.

[0095] Since the selected target number and the tracking target number are 3, in step S705, as shown in the tracking control state 1111, the tracking control state is determined to be "tracking". Then, the CPU 201 of the PC 200 displays Figure 11B the menu screen shown. Similar to the first embodiment, assume the case where one subject is not detected in the viewing angle of the PTZ camera 100. In the case of the camera image 1120 shown in Figure 11C shown, the subject information to be transmitted in step S705 is shown in Figure 12B shown. In this case, since the tracking target number in the position information is 2 and the selected target number is 3 (in step S712, it is "no"), the process proceeds to step S716. In step S716, the CPU 201 transmits information so that "partially lost" is displayed in the tracking control state 1121, and updates the menu screen on the RAM 203.

[0096] The case of a subject other than the subjects in the appearance group will be described. Assume shown Figure 11DThe case of the camera image 1130 shown in. The subject 1131 corresponding to the subject D and the subject 1132 not corresponding to the pool 1000 appear in the viewing angle. The subject information is in Figure 12C shown. In step S711, since the number of tracking targets is 0, the CPU 201 transmits information so that "lost" is displayed in the tracking control state 1133, and the menu screen on the RAM 203 is updated. The group information can be set by the user storing the previously selected group information in the ROM 102 and the CPU 101 reading the group information at the start of the control flow instead of the user making a selection again.

[0097] Suppose the case where the user selects the second group 1002 via the user input I / F 206. In the second group 1002, priority settings are made for the selected subjects, and subject A is given a higher priority than subject D and subject E. By changing the above-mentioned center-of-gravity calculation in the calculation of the control position information in step S306, a difference resulting from the priority-based tracking operation can be caused.

[0098] Suppose the second group 1002 is selected and the Figure 13A menu screen shown in is displayed. The selection target 1300 is changed to the second group, and communication is performed between the PTZ camera 100 and the PC 200 Figure 14A for the subject information shown in. As Figure 14A indicated by the subject information shown in, the addition type is used as the priority, subject A is set as the main subject, and subject D and subject E are set as secondary subjects. Using this information, the CPU 201 of the PC 200 updates the detection count of the main subject 1301 to 1 and the detection count of the secondary subject 1302 to 2.

[0099] The case where no subject is detected in the viewing angle will be described. In step S716 (as described above), the CPU 101 displays a mismatch between the number of tracking targets and the number of selected targets in the form of "partial loss". However, the addition of priority causes the following changes, thereby allowing additional information to be displayed to the user. Suppose the case where the main subject A (i.e., Figure 14B the subject information shown in the result) is not detected in the viewing angle.

[0100] In step S716, the currently displayed menu screen becomes "partial loss". However, in order to emphasize the non-detection of the main subject, the CPU 101 of the PTZ camera 100 resets the tracking control state 1310 to "main subject lost". The CPU101 updates the menu screen so that the detection count of the main subject 1311 is 0. Similarly, in Figure 14CIn the case of the subject information shown, the secondary subject E is not detected. Therefore, the CPU 101 resets the tracking control state 1320 to "secondary subject lost" and updates the menu screen so that the detection count of the secondary subject 1321 is 1, thereby generating Figure 13C the display in. Figure 14D In the case of the subject information shown, the primary subject A and the secondary subject D are not detected. Therefore, the CPU 101 sets the tracking control state 1330 to "primary subject and secondary subject lost" and updates the menu screen so that the detection count of the primary subject 1331 is 0 and the detection count of the secondary subject 1332 is 1.

[0101] Similar to the case of selecting the first group, selecting a subject other than the unselected subject or group results in "lost".

[0102] Group selection can be performed because the CPU 201 determines that the subject detection result corresponds to the group information of any group, rather than a user's explicit group selection. In this case, as the priority when multiple groups match the subject detection result, for example, control can be performed to continuously set the first-found group or set a given priority to a specific group.

[0103] According to the second embodiment described above, by presenting information indicating a change in the subject selection method and subject type from the first embodiment to the user, a similar effect can be produced.

[0104] The third embodiment will be described below. According to the first and second embodiments, the menu screen is updated to present to the user. As a method for more intuitively determining the tracking control state, the display of the lamp 112 installed on the PTZ camera 100 can be changed to present to the user. Figure 15An example of replacing the tracking control state with different lighting modes of the lamp 112 is shown. When the information indicating the tracking control state is determined and transmitted in step S706, the CPU 101 changes the display by controlling the lamp 112 accordingly. The CPU 101 can continue the same display until the end of the control flow or the update of the tracking control state. In the present embodiment, as described above, regarding the configuration of the PTZ camera 100, the use of color indicates the video image. Therefore, examples of change modes (lighting up and flashing) will be described. For "stop", since there is no need to display the tracking state, the lamp 112 is turned off. For "tracking", the lamp 112 is turned on as in normal shooting to show that tracking is being performed normally. "Lost", "main subject lost", and "main and secondary subjects lost" are adverse to the video image, so it is determined that intervention for interruption or manual operation is required in the middle, so that the lamp 112 flashes at intervals of 0.5 seconds. "Partially lost" and "secondary subject lost" are not as adverse as "main and secondary subjects lost", so it is indicated that there is a problem, so that the flashing of the lamp 112 changes to an interval of one second. Switching to the display of the discrimination state is an example, but it is not limited thereto. The mode can be changed by at least one of the flashing, intensity, color, and number of light-emitting diodes (LEDs) of the lamp 112.

[0105] In the above-described third embodiment, similar effects to those of the first and second embodiments can be produced in a changed notification form.

[0106] The fourth embodiment will be described below. Meanwhile, according to the first to third embodiments, the PTZ camera 100 as the imaging device performs subject detection and tracking control, while the PC 200 as the information device or the server as the external device can be used to perform subject detection and tracking. An example in which the PC 200 is used to perform subject detection and tracking will be described below.

[0107] Figure 16 The configuration of this embodiment is shown. The CPU 101, ROM 102, RAM 103, image output I / F 104, network I / F 105, image processing unit 106, image sensor 107, drive I / F 108, drive unit 109, and internal bus 110 of the PTZ camera 100 are similar to those in other embodiments, but the inference unit 111 and the lamp 112 are removed. The CPU 201, ROM 202, RAM 203, network I / F 204, display unit 205, user input I / F 206, and internal bus 207 of the PC 200 are similar to those in other embodiments, but an inference unit is added

[0108] 208. The operations of the inference units 111 and 208 are similar to each other. The PC 200 only receives video images from the PTZ camera 100 and transmits pan and tilt control information, and processes subject information only on the PC 200.

[0109] Figure 17 The control flow of the PC 200 is shown. This control flow starts when the CPU 201 of the PC 200 receives a control command for automatic tracking control via the network I / F 204 or the user input I / F 206.

[0110] In step S1701, the CPU 101 determines whether a control command or an end command has been received via the network I / F 204, and stores the received control command in the RAM 203. When the CPU 201 receives a control command (resulting in "yes" as the confirmation operation status in step S1701), the CPU 101 stores the received control command in the RAM 203. Then, the process proceeds to step S1702. When the CPU 101 receives an end command (resulting in "no" as the confirmation operation status in step S1701), the CPU 101 completes this control.

[0111] In step S1702, the CPU 101 receives the captured image of the PTZ camera 100 via the network I / F 204 and stores the image in the RAM 203.

[0112] In step S1703, similar to step S303, the CPU 201 uses the inference unit 208 to determine information related to the characteristics and position of the subject from the captured image, and stores the information in the RAM 203.

[0113] In step S1704, similar to step S304, the CPU 201 determines the subject selection information. According to this embodiment, similar to the first embodiment, the display unit 205 can be used to display a menu screen to detect coordinates in the viewing angle via the user input I / F 206. In addition, similar to the second embodiment, group selection information can be received via the user input I / F 206. The subject information is stored in the RAM 203.

[0114] In step S1705, similar to step S705, the CPU 201 determines the tracking target in the subject information, and stores the subject information in the RAM 203 instead of transmitting the subject information to the outside.

[0115] In step S1706, similar to step S306, the CPU 201 calculates control position information. The calculated control position information is transmitted to the PTZ camera 100 via the network I / F 204. The CPU 101 of the PTZ camera 100 performs the controls in steps S307 to S308 based on the information to perform pan, tilt, and zoom operations via instructions from the PC 200.

[0116] In step S1707, similar to step S706, the CPU 201 compares the tracking state with the selected target. Similar to other embodiments, the CPU 101 determines the tracking control state based on the number of selected targets and the number of tracking targets, and stores the tracking control state in the RAM 203.

[0117] In step S1708, similar to other embodiments, the CPU 201 uses the display unit 205 to display the captured image, subject information, and tracking control state stored in the above steps.

[0118] According to the fourth embodiment, the PC 200 as the information device can produce similar effects to other embodiments.

[0119] <Other embodiments>

[0120] The present invention can also be implemented by a program supplied to a system or device via a network or a storage medium and one or more processors in a computer of the system or device, the program for implementing at least one function according to the above embodiments, the processor for reading and executing the program. In addition, the present invention can also be implemented by a circuit such as an application specific integrated circuit (ASIC) for implementing one or more functions. The present invention is not limited to the above embodiments, and can be modified and changed in different ways without departing from its scope.

[0121] The present invention enables an information processing device that tracks multiple subjects to notify a user of the tracking state.

[0122] Embodiments of the present invention can also be implemented by the method of providing software (program) that executes the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessing unit (MPU) of the system or device reads and executes the program.

[0123] Although the present invention has been described with reference to the embodiments, it should be understood that the present invention is not limited to the disclosed embodiments, but is defined by the scope of the appended claims.

Claims

1. An information processing device, comprising: a detection unit, wherein the detection unit is configured to detect a plurality of objects from an image; a selection unit configured to select a subject as a tracking target; a tracking unit configured to track the subject selected by the selection unit as the tracking target using information about the plurality of subjects detected by the detection unit; a counting unit, wherein the counting unit is configured to count the number of the objects currently being tracked by the tracking unit; as well as a notification unit configured to notify the tracking unit of a tracking state, The notification unit notifies the tracking state according to the number of the objects selected by the selection unit and the number of the objects counted by the counting unit.

2. The information processing device according to claim 1, wherein: The notification unit is configured to change notification content between a case where the number of the subjects selected by the selection unit matches the number of the subjects being tracked by the tracking unit and a case where the number of the subjects selected by the selection unit does not match the number of the subjects being tracked by the tracking unit.

3. The information processing device according to claim 1, wherein: The notification unit is configured to notify, as the tracking state, the number of the subjects selected by the selection unit and the number of the subjects counted by the counting unit.

4. The information processing device according to claim 1, wherein: The selection unit is configured to select the plurality of subjects detected by the detection unit based on the plurality of subjects continuously detected within a predetermined period of time or a predetermined number of times, and exclude any subject not detected within the predetermined period of time or the predetermined number of times from the plurality of subjects.

5. The information processing device according to claim 1, further comprising a subject management unit configured to identify and manage the subject, in, The multiple subjects selected by the selection unit based on the subject information stored by the subject management unit are compared with the subject detected by the detection unit, and when each of the multiple subjects selected by the selection unit based on the subject information stored by the subject management unit does not match the subject detected by the detection unit, the notification unit issues a notification.

6. The information processing device according to claim 5, wherein: The selection unit is configured to compare the object information stored by the object management unit with the object detected by the detection unit, and when the object information corresponds to the object, select the object as the tracking target.

7. The information processing device according to claim 1, further comprising a subject management unit configured to pre-register and manage subject information including the subject characteristics, in, The selection unit is configured to select the tracking target based on the subject information registered in the subject management unit.

8. The information processing device according to any one of claims 1 to 7, further comprising a subject management unit, wherein the subject management unit is configured to pre-register and manage subject information including the subject characteristics, in, Priorities are set for a plurality of subjects registered in the subject management unit.

9. The information processing device according to any one of claims 1 to 7, wherein: The notification unit is configured to select a main subject as the tracking target and notify a tracking state of the main subject and a tracking state of other subjects in a distinguishable manner.

10. The information processing device according to any one of claims 1 to 7, wherein: The notification unit is configured to notify, in a distinguishable manner, a region of the subject detected by the detection unit but not selected by the selection unit and a region of the subject detected by the detection unit and selected by the selection unit.

11. The information processing device according to any one of claims 1 to 7, wherein: In response to a user's instruction on the subject as the tracking target, the tracking unit is configured to delete the subject from the tracking target.

12. The information processing device according to any one of claims 1 to 7, wherein: The tracking unit is configured to delete the subject from the tracking target.

13. The information processing device according to any one of claims 1 to 7, wherein: The tracking unit is configured to change control content according to the number of the objects selected by the selection unit.

14. The information processing device according to claim 13, wherein: The tracking unit is configured to track the subject using at least one of pan control, tilt control, and zoom control.

15. The information processing device according to any one of claims 1 to 7, wherein: The notification unit is configured to notify the tracking state by changing a lighting pattern of the recording indicator lamp.

16. An information processing system, comprising: an imaging unit configured to capture an image and output the image; a detection unit, the detection unit being configured to detect a plurality of objects from the image; a selection unit configured to select a subject as a tracking target; a tracking unit configured to track the subject selected by the selection unit as the tracking target using information about the plurality of subjects detected by the detection unit; a counting unit, the counting unit being configured to count the number of the objects being tracked by the tracking unit; as well as a notification unit configured to notify the tracking unit of a tracking state, The notification unit controls notification according to the number of the objects selected by the selection unit and the number of the objects counted by the counting unit.

17. An information processing method, the information processing method comprising: A detection step for detecting multiple objects from an image; A selection step for selecting a subject as a tracking target; a tracking step for tracking the subject selected as the tracking target in the selecting step using information related to the plurality of subjects detected in the detecting step; A counting step, for counting the number of the objects being tracked in the tracking step; as well as a notification step, for notifying the tracking status in the tracking step, The notifying step is controlled according to the number of the objects selected in the selecting step and the number of the objects counted in the counting step.

18. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of claim 17.

19. A non-transitory computer-readable storage medium storing the program according to claim 18.

Images