Information processing device, information display device, information processing system

Abstract

This system aims to provide a user with a change in state that takes into account the increase or decrease in the number of subjects being tracked, in an information processing system that tracks multiple subjects. [Solution] The system comprises a detection means for detecting multiple subjects from an image, a selection means for selecting a subject to be tracked, a tracking means for tracking the subject selected by the selection means using information on the multiple subjects detected by the detection means, and a notification means for notifying the tracking status by the tracking means, wherein the notification means controls the notification according to the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means.

Description

[Technical Field]

[0001] The present invention relates to an information processing apparatus, an information processing system, and a tracking method for tracking a plurality of subjects. [Background Art]

[0002] In video production, there is an automatic tracking control technology that automatically controls the pan / tilt / zoom (PTZ) of an imaging device so as to arrange a specific subject at a desired position within the imaging angle of view.

[0003] Patent Document 1 describes a technology that presents the moving direction and moving speed of a tracking target, and displays a warning when behavior that is likely to go out of the angle of view is detected. [Prior Art Documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Unexamined Patent Application Publication No. 2009-218719 [Summary of the Invention] [Problem to be Solved by the Invention]

[0005] However, Patent Document 1 does not disclose control for simultaneously tracking a plurality of subjects, nor does it disclose a notification mode according to the states of the plurality of subjects in such a case.

[0006] The present invention has been made in view of the above problem, and an object thereof is to notify a user of a tracking state in an information processing apparatus, an information processing system, and a tracking method that track a plurality of subjects. [Means for Solving the Problem]

[0007] To solve the above problems, the information processing device of the present invention comprises: detection means for detecting multiple subjects from an image; selection means for selecting a subject to be tracked; tracking means for tracking the subject selected by the selection means using information on the multiple subjects detected by the detection means; and notification means for notifying the tracking status by the tracking means, wherein the notification means controls the notification according to the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means. [Effects of the Invention]

[0008] According to the present invention, an information processing device that tracks multiple subjects can notify the user of the tracking status. [Brief explanation of the drawing]

[0009] [Figure 1] Diagram illustrating the system configuration [Figure 2] A diagram illustrating the hardware configuration of the devices that make up the system. [Figure 3] A flowchart illustrating the basic operation of an imaging device. [Figure 4] A flowchart illustrating the basic operation of an information device. [Figure 5] A diagram illustrating information display in basic operations. [Figure 6] A diagram illustrating subject information in basic operations. [Figure 7] Flowchart illustrating the control process in Embodiment 1 [Figure 8] A diagram illustrating the information display in Embodiment 1. [Figure 9] Figure illustrating subject information in Embodiment 1 [Figure 10] A diagram illustrating the management of the tracked target in Embodiment 2. [Figure 11] A diagram illustrating the information display in Embodiment 2. [Figure 12] Figure illustrating subject information in Embodiment 2 [Figure 13]Figure illustrating an example of information display in consideration of priority in Embodiment 2 [Figure 14] Figure illustrating an example of subject information in consideration of priority in Embodiment 2 [Figure 15] Figure illustrating an example of lamp display patterns in Embodiment 3 [Figure 16] Figure illustrating an example of the hardware configuration of Embodiment 4 [Figure 17] Flowchart illustrating an example of control processing in Embodiment 4 DESCRIPTION OF EMBODIMENTS

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the claimed invention. Although a plurality of features are described in the embodiments, not all of the plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and overlapping descriptions are omitted.

[0011] [First Embodiment] <System Configuration> Figure 1 is a diagram showing an example of the configuration of an information processing system that implements processing according to the present embodiment. In Figure 1, the information processing system includes a PTZ camera serving as an information processing apparatus, and a PC (Personal Computer). The PTZ camera 100 and the PC 200 are connected to a network formed on a LAN (Local Area Network) 300, and form a network in which the devices can communicate with each other via a communication protocol. There is no limitation on whether the connection is wired or wireless.

[0012] The PTZ camera 100 is an imaging device capable of capturing an image of a tracking target (subject) and a predetermined range around the target, and outputs captured images to a PC 200 or other external devices. The PTZ camera 100 according to the present embodiment includes a driving unit 109 described later, and has a mechanism that enables pan and tilt operations for changing an imaging direction. The PTZ camera 100 also includes an inference unit 111 described later, which infers the position of the subject in the captured image.

[0013] By accessing the PTZ camera 100 via a LAN 300, the PC 200 can acquire images output by the PTZ camera 100, perform imaging control based on user operations, and set various imaging conditions. Note that the image in the present embodiment includes both moving images and still images, and the present embodiment can be applied to both.

[0014] FIG. 2 is a configuration diagram of the PTZ camera 100 and the PC 200 that constitute the system. The configuration of each device will be described below.

[0015] The PTZ camera 100 according to the present embodiment includes a CPU 101, a ROM 102, a RAM 103, a video output I / F (Interface) 104, and a network I / F 105. The PTZ camera 100 further includes an image processing unit 106, an image sensor 107, a driving I / F 108, a driving unit 109, an inference unit 111, and an internal bus 110 that communicatively connects the above components to each other.

[0016] The CPU 101 controls each component of the PTZ camera 100, thereby governing control of the entire apparatus.

[0017] The ROM 102 is a non-volatile storage device typified by flash memory, HDD (Hard Disk Drive), SSD (Solid State Drive), SD card, and the like. In addition to being used as a persistent storage area for an OS, various programs, and various data, the ROM 102 is also used as a short-term storage area for various data.

[0018] RAM103 is a volatile, high-speed memory device, such as DRAM, where the OS, various programs, and various data are loaded. It is also used as a workspace for the OS and various programs.

[0019] The video output I / F 104 is an interface for outputting images captured by the imaging sensor 107 (described later) to an external device, and consists of SDI and HDMI (registered trademark).

[0020] Network I / F105 is an interface for connecting to the aforementioned LAN300 and is responsible for communication with external devices such as PC200 via a communication medium such as Ethernet (registered trademark).

[0021] The image processing unit 106, connected to the imaging sensor 107, performs various image processing operations on the image data acquired from the imaging sensor 107 based on instructions from the CPU 101, such as defect correction, noise reduction (NR) processing, and color conversion processing, as well as processing to convert the image data to a predetermined format and compress it. The processed image data is stored in the RAM 103.

[0022] The imaging sensor 107 is an image sensor having an image sensor such as a CCD or CMOS, and functions as an imaging means in the PTZ camera 100. The imaging sensor 107 generates image data by photoelectric conversion of the subject image formed by an imaging optical system (not shown). In this embodiment, image data is output to the image processing unit 106 as a digital signal by an AD conversion circuit included in the image sensor 107, but it may also be output as an analog signal. Furthermore, the image sensor 107 and the image processing unit 106 may be integrated as a stacked chip. Also in this embodiment, the image sensor 107 has an integrated imaging optical system for receiving the subject image onto the image sensor, but the imaging optical system may be configured to be, for example, detachable or replaceable from the image sensor 107. In this embodiment, the imaging optical system and the image sensor 107 are sometimes collectively referred to as the imaging means.

[0023] The drive interface 108 is an interface for transmitting instructions received from the CPU 101 to the drive unit 109.

[0024] The drive unit 109 is a mechanical mechanism and optical system for changing the imaging direction of the PTZ camera 100. In this embodiment, the imaging direction is changed by rotating the imaging sensor 107. The drive unit 109 consists of a mechanical drive system and a motor as the drive source. Based on instructions received from the CPU 101 via the drive I / F 108, the drive unit 109 performs rotational drives such as pan and tilt operations to change the imaging direction in the horizontal and vertical directions. In addition, if the imaging optical system has a variable magnification lens (also called a zoom lens), zoom control may be performed to optically change the imaging angle of view by moving the zoom lens in the optical axis direction.

[0025] The inference unit 111 performs inference processing using a trained inference model and inference parameters according to the inference program. The inference processing in the inference unit 111 can be performed by a processing unit specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). A GPU is a processor capable of performing a large number of multiply-accumulate operations and has the processing power to perform matrix operations of neural networks in a short time. Alternatively, the inference processing in the inference unit 111 may be implemented by a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The inference processing may also be performed in cooperation with the CPU 101.

[0026] Lamp 112 is a light source, such as an LED (Light Emitting Diode), used to indicate the control status of the PTZ camera 100, and is also called a tally lamp. The CPU 101 changes the display pattern when it receives instructions from an external source via the network interface 105. For example, it can display the status using different colors, such as red to indicate that the video is being used for broadcasting or recording, or green for preview mode, or by blinking frequency.

[0027] In this embodiment, the PC200 includes a CPU201, ROM202, RAM203, network interface204, display unit205, user input interface206, and an internal bus207 that connects the above components to each other in a manner that enables communication.

[0028] CPU201 controls the entire device by controlling each component of PC200.

[0029] ROM202 is a non-volatile storage device, exemplified by flash memory, HDDs (Hard Disk Drives), SSDs (Solid State Drives), and SD cards. ROM202 is used as a persistent storage area for the OS, various programs, and data, as well as for short-term data storage.

[0030] RAM203 is a volatile, high-speed memory device, such as DRAM, where the OS, various programs, and various data are loaded. It is also used as a workspace for the OS and various programs.

[0031] Network I / F (Interface) 204 is an interface for connecting to the aforementioned LAN 300 and is responsible for communication with imaging devices such as the PTZ camera 100 and other external devices such as servers via communication media such as Ethernet (registered trademark).

[0032] The display unit 205 displays images acquired from the PTZ camera 100 and the settings screen of the PC 200. The display unit 205 is, for example, an LCD panel or an OLED panel. Although this example shows the PC 200 having a display unit, the PC 200 and the display unit may be separate components, for example, a display monitor that only displays captured images and the PC 200 may exist as separate units.

[0033] The user input interface 206 has input devices (operation units) such as a keyboard, pointing device (mouse, etc.), touch panel, and switches, and receives instructions from the user to the PC 200. The keyboard may be a software keyboard. The CPU 201 monitors the user input interface 206, and when it detects an operation by the user on the user input interface 206, it executes processing according to the detected operation.

[0034] <Explanation of basic operations for automatic tracking control and subject selection> Next, we will explain the basic control of this system, which involves automatic tracking control to control the PTZ camera 100 to track a subject, and the selection of a subject to be tracked by the PTZ camera 100 based on user operations received by the PC 200.

[0035] First, automatic tracking control will be explained using Figures 3 to 6.

[0036] Figure 3 shows the control flow of the PTZ camera 100, illustrating a series of steps to control the PTZ camera 100 according to the position of the subject detected from the captured image.

[0037] This control flow is initiated when the CPU 101 of the PTZ camera 100 receives a control command to execute automatic tracking control via the network I / F 105.

[0038] In step S301, the CPU 101 determines whether it has received a control command or a termination command via the network interface 105, and stores the received control command in RAM 103. If the CPU 101 has received a control command (operation status check result is YES), it stores the received control command in RAM 103 and proceeds to step S302. If the CPU 101 determines that it has received a termination command (operation status check result is NO), this control is terminated.

[0039] In step S302, the CPU 101 retrieves the image data stored by the image processing unit 106 from the RAM 103.

[0040] In step S303, the CPU 101 determines information regarding the features and position of the subject in each frame of the captured image data and stores it in the RAM 103. Specifically, the CPU 101 inputs the image data obtained from the RAM 103 to the inference unit 111. Next, the CPU 101 stores the features of the subject and the position information of the subject on the image, inferred by the inference unit 111, into the RAM 103. The inference unit 111 has a trained model created using machine learning methods such as deep learning, receives an image as input data, and outputs an ID for identification and position information as subject information as output data. Position information is described as, but is not limited to, the upper-left point, width and height information of the bounding rectangle touching the subject in the image, and the coordinates of the centroid.

[0041] The table shown in Figure 6(a) shows the detected subject information. Here, Figure 5(a) shows the information displayed on the display screen of the display unit during the operation of the PC200 described later, and as an example, there are three subjects within the field of view. In Figure 5(a), the subject position information output in step S303 is superimposed as subject frames 503 to 505. Focusing on subject frame 503, it corresponds to ID=1 in Figure 6(a), with the top-left point of the frame being (x-1, y1), the size of the frame being (w1, h1), and the coordinates of the centroid being (gx1, gy1). The coordinates on the PTZ camera 100 are assumed to be converted to the display coordinate system by the CPU 201 when displayed on the PC200. In this embodiment, the inference unit 111 was used to identify the subject's position information, but this is not limited to this as long as the subject's position information can be identified. For example, position information could be obtained from a wireless communication terminal held by the subject and used as the subject's position information. The CPU 101 stores the output subject information in the RAM 103 and proceeds to step S304.

[0042] In step S304, the CPU 101 determines whether it has received subject selection information as a control command from the PC 200 via the network I / F 105. If it has received it, the CPU 101 stores the subject selection information received from the PC 200 in the RAM 103. In this embodiment, subject selection information is described as indicating coordinates (area) on the field of view. If a subject has already been selected, this step will always proceed to step S305 regardless of the subject selection information. In step S305, the CPU 101 reads the subject information output in step S303 and the subject selection information received in step 304 from the RAM 103. The CPU 101 compares the position information in the subject information with the coordinate information included in the subject selection information to confirm whether the coordinate information is included in the position information of the subject. As a specific example, Figure 5 will be used for explanation. If the coordinates specified as subject selection information are point 506 as shown in Figure 5(a), then it will be included in the position information displayed as the subject frame 503, that is, the area that can be represented by the top left point and the frame. Therefore, the CPU 101 determines that the subject corresponding to the subject frame 503 has been selected. As shown in Figure 6(b), the CPU 101 adds whether or not it is a subject to be tracked to the subject information, stores it in the RAM 103, transmits it to the PC 200, and proceeds to step S306.

[0043] In step S306, the CPU 101 calculates control position information (control information) for automatic tracking control and stores this control position information in the RAM 103. Control position information refers to information such as the pan angle, tilt angle, and zoom angle of view (imaging parameters) when controlling (moving) the image sensor 107 to an arbitrary position. The CPU 101 calculates the pan angle, tilt angle, zoom angle of view, and angular velocity when the coordinates of the center of gravity of the subject to be tracked, selected in step S305, are controlled to be in the center of the angle of view, and stores the calculated results as control position information in the RAM 103. If no subject is selected, there is no control position information, which is equivalent to no control of the angle of view. If control position information is received as a control command from an external source in step S301, that information is prioritized and stored in the RAM 103, enabling external position control.

[0044] In step S307, the CPU 101 reads the control position information stored in the RAM 103. Based on the control position information, the CPU 101 derives the drive parameters (control content) for the drive unit 109 in order to pan, tilt, and zoom in the desired direction at the desired speed. Specifically, these are parameters for controlling the motor included in the drive unit 109, and the control position information may be converted into drive parameters by referring to a conversion table previously stored in the RAM 103 based on the manipulated amount included in the control position information.

[0045] In step S308, the CPU 101 controls the drive unit 109 via the drive I / F 108 based on the derived drive parameters, and the drive unit 109 rotates based on these drive parameters, causing the PTZ camera 100 to perform pan, tilt, and zoom operations. This control flow, calculated based on the subject selection information received in step S304, enables the PTZ camera 100 to operate in accordance with the subject selection sent sequentially by the user from the PC 200.

[0046] Next, we will explain the flow for controlling the PTZ camera 100 based on user operations on the PC 200 shown in Figure 4. The PTZ camera 100 is controlled based on control commands transmitted from the PC 200. The operation of the PTZ camera 100 is the operation when subject selection information is received in the control flow shown in steps S304 to S305 of Figure 3. Therefore, the explanation will be omitted here. This control flow is started when the CPU 201 detects, via the user input I / F 206, an operation to start the menu screen that displays the image and information captured by the PTZ camera 100 as illustrated in Figure 5. At the start of this control flow, the CPU 201 displays the menu screen 500, the tracking control status 501, and the camera image 502 received from the PTZ camera 100 on the display unit 205, as shown in Figure 5(a). In this embodiment, it will be explained assuming that the tracking target has not been selected, i.e., tracking control has not been started.

[0047] In step S401, the CPU201 of PC200 detects a user operation to exit the menu screen via the user input interface206. If no operation is detected, the process proceeds to S402; if an operation is detected, the control flow terminates.

[0048] In step S402, the CPU 201 of PC200 sends a control command to the PTZ camera 100 via the network interface 204 to acquire information. When the CPU 101 of the PTZ camera 100 detects that it has received the command, it reads the subject information output in step S303 of Figure 3 from the RAM 103 and sends it to PC200 via the network interface 105. Based on the received subject information, the CPU 201 of PC200 calculates the coordinates so that subject frames 503 to 505 are superimposed on the camera image 502 and updates the menu screen on RAM 203. Tracking control has not yet started, but the user can know the subject information that the PTZ camera 100 has recognized. After that, the CPU 201 proceeds to S403.

[0049] In step S403, the CPU 201 of the PC 200 detects via the user input I / F 206 whether the user is performing a subject selection action. Input methods include, but are not limited to, specifying a point on the camera image 502 by operating a mouse or performing touch operations on the camera image 502 displayed on the touch panel. Similar to step S304 in the control flow of Figure 3 described above, if the user specifies point 506, the CPU 201 converts point 506 on the camera image 502 into coordinates in the camera field of view and transmits it to the PTZ camera 100, then proceeds to step S404. Note that if there is a tracking target as described later, this step will always proceed to step S404 regardless of user operation.

[0050] In step S404, the CPU 201 of the PC 200 obtains the tracking status from the PTZ camera 100. The tracking status is the information shown in Figure 6(b) regarding whether or not the subject is being tracked, which was transmitted from the PTZ camera 100 in step S305 of the control flow in Figure 3 described above. If this information has already been assigned in step S402, it may be read from RAM 103. When the CPU 201 confirms that the subject is being tracked, it changes the display of the tracking control status 511 to indicate that the PTZ camera 100 is tracking. In addition to changing the string, the change may also be achieved by updating the display of the color or pattern, and in this embodiment, the string and color are updated. Furthermore, the display format of the subject frame corresponding to the ID that has been tracked is changed to indicate to the user whether or not the subject is being tracked. Specifically, subject frame 503 is updated to subject frame 513. On the other hand, subject frames 504 to 505 are not updated in their display, as shown in subject frames 514 to 515 respectively, and are presented to the user as being detected but not being tracked. The display format of the frames is just an example, and the method of changing patterns, colors, etc., is not limited to this embodiment. The CPU 201 updates the menu screen of RAM 203 with this information and proceeds to step S405.

[0051] In step S405, the CPU 201 of the PC 200 displays the updated menu screen 510 on the RAM 203 by controlling the display unit 205. At the start of this control flow, the screen was in the state shown in Figure 5(a), but after selecting a subject, the content was updated to the screen shown in Figure 5(b). Also, as shown in Figure 5(c), if the subject designated as the tracking target is not within the field of view of the PTZ camera 100, the PTZ camera 100 will transmit information indicating that the position information is unknown, as shown in Figure 6(c). If the coordinates of the tracking target cannot be confirmed, the CPU 201 can notify the user that the tracking target has been lost by changing the display of text and color, for example, as shown in the tracking control state 521.

[0052] Furthermore, if the point specified by the user in step S403 is already the subject being tracked, the tracking can be canceled, and the tracking target can be switched by selecting a different subject. In addition, although the tracking operation starts upon subject selection, it may be changed to explicitly instruct the start and stop of the tracking operation.

[0053] Based on the basic operation described above, the control flow of the PTZ camera 100 and PC200 makes it possible to perform automatic tracking control of the PTZ camera 100 according to the user's subject selection.

[0054] <Explanation of tracking behavior when multiple subjects are selected> Next, we will explain the tracking operation when multiple subjects are selected, which is a characteristic operation of the present invention. The operation of the PTZ camera 100 will be explained using only the differences from the control flow in Figure 3 described above, and the operation of the PC 200 will be explained using only the differences from the control flow in Figure 4 described above, with reference to Figures 7 to 9.

[0055] The control flow shown in Figure 7(a) is initiated in the same way as the control flow shown in Figure 3. Steps S701-703, 705, and 707-709 operate in the same way as steps S301-303, 305, and 306-308, respectively.

[0056] In step S705, as in step S305, it is assumed that subject information for three people is being transmitted from the PTZ camera 100. However, unlike in Figure 5(a), the PC 200 displays the menu screen shown in Figure 8(a). Specifically, it displays the number of currently selected subjects 801, the number of tracked subjects whose location information has been acquired 802, and the number of currently detected subjects 803. Since there are no subjects selected yet, the CPU 201 updates the menu screen on RAM 203, assuming that the number of selected subjects 801 and the number of tracked subjects 802 are 0, and the number of detected subjects 803 is 3.

[0057] In step S704, the CPU 101 of the PTZ camera 100 determines whether it has received subject selection information as a control command from the PC 200 via the network I / F 105. Similar to the basic operation, if point 804 is specified, the CPU 101 of the PTZ camera 100 transmits the information shown in Figure 6(b) in the subsequent step S705. At this time, the CPU 201 of the PC 200 displays an indication that the subject has been selected as a tracking target, as shown in the subject frame 813, and simultaneously updates the menu screen by setting the number of selected targets 811 to 1 and the number of tracks 812 to 1.

[0058] In step S706, the CPU 101 of the PTZ camera 100 compares the tracking status received from the PTZ camera 100 with the number of selected targets. Step S706 is equivalent to the control flow shown in Figure 7(b), and assuming a transition to step S710, we will continue the explanation using Figure 7(b). At this point, as shown in Figure 6(b), we assume that there is one tracking target whose position information has been acquired, and the number of selected targets is also 1. After completing the processing in step S706, the CPU 101 of the PTZ camera 100 transmits the tracking control status to the PC 200 via the network I / F 105.

[0059] In step S710, the CPU 101 determines whether the number of selected objects is 0. As mentioned above, the current number of selected objects is 1, so the process proceeds to S711. Note that when the control flow shown in Figure 7(a) starts, the number of selected objects is 0, and this number remains 0 as long as no object is selected in step S704. In that case, the CPU 101 proceeds to S713, transmits information so that the tracking control status 800 displays as "Stopped" as shown in Figure 8(a), and the CPU 201 updates the menu screen on RAM 203.

[0060] In step S711, the CPU 101 determines whether the number of tracks is 0. Since the number of tracks is currently 1, the CPU 101 proceeds to step S712. If the number becomes 0, the transition to step S714 will be described later.

[0061] In step S712, the CPU 101 compares the number of items being tracked with the number of items to be selected. If the number of items to be selected matches the number of items being tracked, the CPU 101 proceeds to step S715; otherwise, it proceeds to step S716. Currently, both are 1, so the CPU proceeds to step 715. In step S715, the CPU 101 transmits information so that the tracking control status 810 displays "Tracking". The CPU 201 of the PC 200 performs control to update the menu screen on RAM 203 as shown in Figure 8(b). Step S716 will be described later. After going through steps S713 to S716, the CPU 101 completes the control flow shown in Figure 7(b) and proceeds to step S706 of the control flow shown in Figure 7(a).

[0062] After the above steps, the menu screen shown in Figure 8(b) is displayed. Next, by specifying point 814, the same control flow is followed, resulting in the number of selected objects 821 becoming 2 and the number of tracked objects 822 becoming 2, the display of the subject frame 823 being updated, and the menu screen shown in Figure 8(c) being displayed. Furthermore, by specifying point 824, the number of selected objects 831 becoming 3 and the number of tracked objects 832 becoming 3, the display of the subject frame 833 being updated, and the menu screen shown in Figure 8(d) being displayed. When the menu screen shown in Figure 8(d) is displayed, the subject information transmitted by the PTZ camera 100 and received by the PC 200 is assumed to be as shown in Figure 9(a).

[0063] In the PTZ camera 100, control is required to track the subject selected as the target for tracking. Taking the state in Figure 9(a) as an example, the method of calculating the control position information calculated in step S306 is changed. If there is only one selected subject, the coordinates of the center of gravity of that subject were used, but by calculating the control position information by taking the coordinates of the center of gravity of three subjects and then taking the center of gravity of those subjects, the PTZ camera 100 can be directed towards the center of the selected subject. Note that for zoom, in addition to the center of gravity, the width and height of the entire subject frame may also be considered and calculated based on the ratio with the field of view, and the calculation method is not limited to this.

[0064] Now, let's consider the case where one of the subjects has moved out of the field of view of the PTZ camera 100. When the camera image 843 is as shown in Figure 8(e), the subject information transmitted in step S705 will be as shown in Figure 9(b). At this time, in step S712, there will be 2 tracking targets with location information and 3 selected targets. Therefore, the process will proceed to step S716, where the CPU 101 transmits information to display "Partially Lost," and the CPU 201 updates the menu screen on RAM 203. As shown in Figure 8(e), the tracking control status 840 is displayed as "Partially Lost," the number of tracking targets 841 is 2, and the number of detected targets 842 is also 2, allowing the user to recognize that the specified number of targets are not being tracked.

[0065] Furthermore, consider the case where all subjects have moved outside the field of view of the PTZ camera 100, and subject 854, which was not initially being tracked, has entered the field of view. When the camera image 853 is as shown in Figure 8(f), the subject information becomes as shown in Figure 9(c). At this time, in step S711, the number of tracked targets with position information becomes 0, so the system transitions to step S714. In step S714, the CPU 101 transmits information to display "Lost," and the CPU 201 updates the menu screen on RAM 203. As shown in Figure 8(f), the tracking control status 850 is displayed as "Lost," the number of tracked targets 852 is 0, and the number of detected targets 851 is 1, allowing the user to recognize that none of the specified targets are being tracked.

[0066] Note that if the subject is not a target for tracking, the same frame as the detection result when the subject is stationary will be displayed. In addition to moving out of the subject's field of view, the subject may also become undetectable due to its posture or orientation. Furthermore, in addition to the user explicitly specifying coordinates, the system may automatically determine and select a subject as a tracking target if it has been continuously detected within the field of view for a certain period of time (a predetermined time or number of detections). Alternatively, a subject may be excluded from tracking if it moves out of the field of view for a certain period of time (undetectable for a predetermined time or number of detections).

[0067] Although this embodiment describes an example using a PTZ camera 100, it is not limited to this form and can be applied to devices that can detect multiple subjects from an image and control the PTZ camera based on the detection results. Specifically, examples include edge devices and PCs equipped with an image input unit, a network communication unit, and a GPU.

[0068] According to the first embodiment described above, in an imaging system that tracks multiple subjects, it becomes possible to drive the imaging system while considering the increase or decrease in the number of subjects being tracked, and to present the user with changes in the tracking state.

[0069] [Second Embodiment] In the first embodiment, the user selected multiple subjects by sequentially selecting them. However, it is also conceivable that the effort of sequential selection could be eliminated by selecting predetermined subjects. Therefore, in this embodiment, the presentation to the user in a method of managing multiple subjects will be explained using Figures 9 to 12 and Figures 13 to 14.

[0070] Figure 10 shows an example of subject management. Pool 1000 manages information based on features such as faces of subjects A through E that have been registered by the user in advance. The registration method involves the user inputting information using the display unit 205 and user input I / F 206 of the PC 200 via a UI (not shown), and the CPU 201 transmitting this information via the network I / F 204. The CPU 101 of the PTZ camera 100 receives this information via the network I / F 204 and stores it in the ROM 102 and RAM 103. This can also be achieved by sequentially storing selected subjects as they are selected, as in the first embodiment, or by transmitting separately created information to the PTZ camera 100; the method is not limited to these. The first group 1001 and the second group 1002 are groups of subjects selected by the user from Pool 1000 and registered as such. In the first group 1001, subjects A through C are selected, which is similar to the state in the first embodiment where the number of subjects was selected as 3.

[0071] The following describes the changes in the control flow shown in Figures 3 and 4, using the same system configuration as the first embodiment. Assume that three subjects are detected as shown in Figure 11(a), and the selection target 1100 is unselected. Here, the CPU 201 of the PC 200 displays the pool 1000, the first group 1001, and the second group 1002 shown in Figure 10 using a UI not shown in the display unit 205, rather than selecting a target in step S403.

[0072] First, if the user selects the first group 1001 via the user input interface 206, the selected group 1110 will be displayed as shown in Figure 11(b), and the CPU 201 will transmit this selection information to the PTZ camera 100. The CPU 101 of the PTZ camera 100 receives the subject selection information in step S304, but instead of coordinates as in the first embodiment, it receives information that the first group 1001 has been selected. Subsequently, in step S305, the CPU 101 searches the subject information for objects that match subjects A to C associated with the first group 1001, rather than using coordinates. If A to C match in this embodiment, the association between A to C is made in the subject information as shown in Figure 12(a). Since the number of selected targets is 3 and the number of targets being tracked is 3, in step S705 the tracking control status is determined to be "tracking" as shown in 1111, and the CPU 201 of the PC 200 displays the menu screen shown in Figure 11(b). Also, similar to the first embodiment, consider the case where one of the subjects was not detected within the field of view of the PTZ camera 100. If the camera image 1120 is as shown in Figure 11(c), the subject information transmitted in step S705 will be as shown in Figure 12(b). At this time, in step S712, since there are 2 tracking targets with location information and the number of selected targets is 3, the system transitions to step S716, and in step S716 the CPU 101 transmits information so that the tracking control status is displayed as "partially lost" as shown in 1121, and the CPU 201 updates the menu screen on RAM 203.

[0073] Next, we will explain what happens when a subject outside the group appears. Assume that the camera image 1130 shown in Figure 11(d) is obtained, and subject 1131 corresponding to subject D and subject 1132 not belonging to pool 1000 appear within the field of view. The subject information is as shown in Figure 12(c), and in step S711, since the tracking count becomes 0, the tracking control status 1133 is sent to display "Lost," and the CPU 201 updates the menu screen on RAM 203. Alternatively, the group information previously selected by the user may be saved in ROM 102 and read by the CPU 101 at the start of the control flow, so that the group information is set without the user having to select it again.

[0074] Next, let's assume the user selects the second group 1002 via the user input interface 206. In the second group 1002, priority settings are made for the selected subjects, and subject A is given priority over subjects D through E. The difference in tracking operation based on priority can be achieved by changing the aforementioned center of gravity calculation in the calculation of control position information in step S306.

[0075] Assume that the second group 1002 is selected and the menu screen shown in Figure 13(a) is displayed. The selected target 1300 is the second group, and the subject information shown in Figure 14(a) is transmitted and received between the PTZ camera 100 and the PC 200. As shown in Figure 14(a) of the subject information, a priority type is added, with subject A set as primary and subjects D through E set as secondary. Using this information, the CPU 201 of the PC 200 updates the display accordingly, showing that the number of detected primary subjects 1301 is 1 and the number of detected secondary subjects 1302 is 2.

[0076] Here, we will explain what happens when the subject is not detected within the field of view. In step S716 mentioned above, the mismatch between the number of subjects being tracked and the number of subjects being selected was displayed as "partially lost," but with the addition of priority, further information can be displayed to the user by modifying it as described later. Let's consider the case where the main subject A is not detected within the field of view, that is, the subject information shown in Figure 14(b).

[0077] According to step S716, the displayed menu screen will show "Partially Lost," but to emphasize that the main subject has not been detected, the CPU 101 of the PTZ camera 100 resets the tracking control state 1310 to "Main Lost." Furthermore, it updates the menu screen by setting the number of detected main subjects 1311 to 0. Similarly, with the subject information shown in Figure 14(c), since the secondary subject E has not been detected, the tracking control state 1320 is reset to "Secondary Lost," and the number of detected secondary subjects 1321 is set to 1, updating the menu screen to display the information shown in Figure 13(c). Also, with the subject information shown in Figure 14(d), since the main subject A and secondary subject D have not been detected, the tracking control state 1330 is reset to "Main / Secondary Lost," the number of detected main subjects 1331 becomes 0, and the number of detected secondary subjects 1332 becomes 1. Similar to when the first group is selected, if only unselected subjects or subjects outside the group are detected, the status will be "Lost".

[0078] Furthermore, instead of the user explicitly selecting a group, the CPU201 may determine that the subject detection result matches any of the group information, and group selection may occur based on this determination. In this case, when multiple groups match, priority may be set by continuing to use the first group found, or by prioritizing a specific group.

[0079] According to the second embodiment described above, the same effect can be obtained by presenting the user with information that changes the method and type of selection of multiple subjects compared to the first embodiment.

[0080] [Third Embodiment] In the first and second embodiments, the status was presented to the user by updating the menu screen. As a more intuitive way to determine the tracking control status, the status may also be presented to the user by switching the display of a lamp 112 installed on the PTZ camera 100. Figure 15 shows an example in which the tracking control status is replaced by the lighting pattern of the lamp 112. When the CPU 101 determines and transmits the information indicating the tracking control status in S706, it changes the display by controlling the lamp 112 accordingly. The same display may be maintained until the control flow ends or the tracking control status is updated. In this embodiment, as explained in the configuration of the PTZ camera 100, the color indicates how the image is being used, so an example of this is achieved by changing the pattern of lighting or blinking. For "Stopped," there is no need to display the tracking status, so the lamp is turned off. For "Tracking," the lamp lights up as in normal imaging to indicate that tracking is working without problems. For "Lost," "Main Lost," and "Main Sub-Lost," the image is undesirable and it is determined that immediate interruption or manual intervention is necessary, so instead of lighting up, the lamp switches to blinking with a 0.5-second period. For "partial loss" and "secondary loss," the display is switched to a 1-second period to indicate that there is a problem, although it is not as undesirable as "major / secondary loss." Switching the display to distinguish the status is just one example and is not limited to this; the pattern may also be changed by modifying at least some of the LEDs on lamp 112, such as their blinking, intensity, color, and number.

[0081] According to the third embodiment described above, the same effects as those of the first and second embodiments can be obtained by changing the notification format.

[0082] [Fourth Embodiment] In the first to third embodiments, the PTZ camera 100, which is an imaging device, performed subject detection and tracking control. However, the PC 200, which is an information device, or a server, which is an external device, may also be responsible for subject detection and tracking. The following description will explain an example in which the PC 200 is responsible.

[0083] Figure 16 shows the configuration diagram of this embodiment. The CPU 101 to the internal bus 110 of the PTZ camera 100 are the same as in other embodiments, except that the inference unit 111 and the lamp 112 are omitted. The CPU 201 to the internal bus 207 of the PC 200 are the same as in other embodiments, except that the inference unit 208 is added. The operation of the inference unit 111 and the inference unit 208 is assumed to be equivalent. The PC 200 receives only video from the PTZ camera 100 and transmits pan control and tilt control information, so subject information is handled only within the PC 200.

[0084] Figure 17 shows the control flow of PC200, which is initiated when the CPU201 of PC200 receives a control command to execute automatic tracking control via the network interface 204 or user input interface 206.

[0085] In step S1701, CPU 201 determines whether it has received a control command or a termination command via the network interface 204, and stores the received control command in RAM 203. If CPU 201 has received a control command (operation status check result is YES), it stores the received control command in RAM 203 and proceeds to step S1702. If CPU 101 determines that it has received a termination command (operation status check result is NO), this control is terminated.

[0086] In step S1702, the CPU 201 receives the captured image from the PTZ camera 100 via the network I / F 204 and stores it in the RAM 203.

[0087] In step S1703, the CPU 201, similar to step S303, uses the inference unit 208 to determine information regarding the subject's features and location from the captured image and stores it in the RAM 203.

[0088] In step S1704, the CPU 201 determines the subject selection information, similar to step S304. In this embodiment, a menu screen may be displayed using the display unit 205, and coordinates on the field of view may be detected via the user input I / F 206, similar to the first embodiment. Alternatively, group selection information may be received via the user input I / F 206, similar to the second embodiment. The subject information is stored in the RAM 203.

[0089] In step S1705, the CPU 201 determines the target to track in the subject information, similar to step S705, but stores the subject information in RAM 203 without transmitting it externally.

[0090] In step S1706, the CPU 201 calculates control position information, similar to step S306. The calculated control position information is transmitted to the PTZ camera 100 via the network I / F 204. The CPU 101 of the PTZ camera 100 performs the pan, tilt, and zoom operations instructed by the PC 200 by controlling steps S307 to S308 based on this information.

[0091] In step S1707, the CPU 201 compares the tracking state with the selected target, similar to step S706. As in other embodiments, the tracking control state is determined from the selected target and the number of targets being tracked, and stored in the RAM 203.

[0092] In step S1707, the CPU 201 displays the captured image, subject information, and tracking control status stored in each of the aforementioned steps using the display unit 205, as in other embodiments.

[0093] According to the fourth embodiment described above, the same effects as in the other embodiments can be obtained in the information device PC200.

[0094] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the described program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The invention is not limited to the above embodiments, and various modifications and variations are possible without departing from the scope of the claims.

Claims

1. A detection means for detecting multiple subjects from an image, A means of selecting the subject to be tracked, A tracking means that tracks a subject to be tracked, selected by the selection means, using information on a plurality of subjects detected by the detection means, It includes a notification means for notifying the tracking state by the aforementioned tracking means, The notification means is characterized by controlling the notification according to the number of subjects selected by the selection means and the number of subjects being tracked by the tracking means.

2. The information processing apparatus according to claim 1, characterized in that the notification means changes the content of the notification depending on whether the number of subjects selected by the selection means matches the number of subjects being tracked by the tracking means or does not match.

3. The notification means, as the tracking state, has the number of subjects selected by the selection means and The information processing device according to claim 1, characterized in that it notifies the number of subjects that can be tracked by the tracking means.

4. The information processing apparatus according to claim 1, characterized in that the selection means selects a plurality of subjects detected by the detection means when they continue to be detected for a predetermined time or a predetermined number of times, and excludes them when they are not detected for a predetermined time or a predetermined number of times.

5. It further includes subject management means for identifying and managing subjects, The information processing apparatus according to claim 1, characterized in that it compares a plurality of subjects selected by the selection means from subject information held by the subject management means with subjects detected by the detection means, and notifies if they do not match.

6. The information processing apparatus according to claim 5, characterized in that the selection means compares subject information held by the subject management means with the subject detected by the detection means, and selects the subject as a tracking target if they match.

7. The information processing apparatus according to claim 5, characterized in that the subject management means sets a priority for the subject information to be held.

8. The information processing apparatus according to claim 1 or 5, characterized in that the tracking means changes the control content according to the number of subjects selected by the selection means.

9. The information processing apparatus according to claim 8, characterized in that the control content is pan control, tilt control, and zoom control.

10. The information processing device according to claim 1, characterized in that the notification means provides notification by switching the lighting pattern of a tally lamp as notification of the tracking state.

11. An imaging means that captures an image and outputs an image, A detection means for detecting multiple subjects from the aforementioned image, A means of selecting the subject to be tracked, A tracking means that tracks a subject to be tracked, selected by the selection means, using information on a plurality of subjects detected by the detection means, It includes a notification means for notifying the tracking state by the aforementioned tracking means, The notification means is characterized by controlling the notification according to the number of subjects selected by the selection means and the number of subjects being tracked by the tracking means. Information processing system

12. A detection step to detect multiple subjects from an image, A selection step to select the subject to be tracked, A tracking step in which the subject to be tracked, selected by the selection means, is tracked using information on the multiple subjects detected in the above detection step, The system includes a notification step for notifying the tracking state obtained by the aforementioned tracking step, The information processing method is characterized in that, in the notification step, the notification is controlled according to the number of subjects selected in the selection step and the number of subjects that can be tracked by the tracking means.

13. A computer-executable program describing the procedure for controlling the imaging device described in claim 12.

14. A computer-readable storage medium in which a program is stored that causes a computer to perform each step of the control method for the imaging apparatus described in claim 12.

Images