Information processing apparatus, information display apparatus, and information processing system

Abstract

To achieve an information processing system that tracks a plurality of subjects, and that can present, to a user, a change in a state considering an increase or decrease in the number of subjects to be tracked.SOLUTION: An information processing apparatus has: detection means that detects a plurality of subjects from an image; selection means that selects a subject to be tracked; tracking means that tracks the subject to be tracked selected by the selection means by using information on the plurality of subjects detected by the detection means; and notification means for notifying the state of tracking performed by the tracking means. The notification means controls the notification according to the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing system, and a tracking method for tracking a plurality of subjects.

Background Art

[0002] In video production, there is a technique of automatic tracking control for automatically controlling the pan / tilt / zoom (PTZ) of an imaging device so that a specific subject is placed at a desired position within the imaging angle.

[0003] Patent Document 1 describes a technique for presenting the moving direction and moving speed of a tracking subject and displaying a warning when detecting behavior that is likely to deviate from the imaging angle.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, Patent Document 1 does not disclose control for simultaneously tracking a plurality of subjects and notification forms according to the states of the plurality of subjects at that time.

[0006] The present invention has been made in view of the above problems, and an object thereof is to notify a user of a tracking state in an information processing apparatus, an information processing system, and a tracking method for tracking a plurality of subjects.

Means for Solving the Problems

[0007] In order to solve the above problems, the information processing apparatus of the present invention includes: detection means for detecting a plurality of subjects from an image; selection means for selecting a subject to be tracked; tracking means for tracking the subject to be tracked selected by the selection means using information on the plurality of subjects detected by the detection means; and notification means for notifying a tracking state by the tracking means, wherein the notification means controls the notification according to the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means.

Effect of the Invention

[0008] According to the present invention, it becomes possible to notify a user of a tracking state in an information processing apparatus that tracks a plurality of subjects.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

BEST MODE FOR CARRYING OUT THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential to the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant descriptions are omitted.

[0011] [First Embodiment] <System Configuration> FIG. 1 is a diagram showing an example of the configuration of an information processing system that executes the processing according to the present embodiment. In FIG. 1, the information processing system includes a PTZ camera, which is an information processing device, and a PC (Personal Computer). The PTZ camera 100 and the PC 200 are connected to a network formed on a LAN (Local Area Network) 300 and form a network in which the devices can communicate with each other according to a communication protocol. The type of wired or wireless connection does not matter.

[0012] The PTZ camera 100 is an imaging device that can image a tracking target (subject) and a determined range around it, and outputs the captured image to a PC 200 or other external devices. The PTZ camera 100 in the present embodiment includes a drive unit 109 described later, and has a mechanism capable of performing pan-tilt operations for changing the imaging direction. Further, it includes an inference unit 111 described later, and infers the position of the subject in the captured image.

[0013] The PC 200 can access the PTZ camera 100 via the LAN 300 to obtain the image output by the PTZ camera 100, and execute imaging control based on user operations and setting of various imaging conditions. Note that the image in the present embodiment includes both moving images and still images, and the present embodiment can be applied to both.

[0014] Figure 2 is a configuration diagram of the PTZ camera 100 and the PC 200 that constitute the system. The configuration of each device will be described.

[0015] The PTZ camera 100 in the present embodiment has a CPU 101, a ROM 102, a RAM 103, a video output I / F (Interface) 104, and a network I / F 105. Further, the PTZ camera 100 has an image processing unit 106, an imaging sensor 107, a drive I / F 108, a drive unit 109, an inference unit 111, and an internal bus 110 that connects the above components so as to be communicable with each other.

[0016] The CPU 101 controls each component of the PTZ camera 100 to take charge of the overall control of the device.

[0017] The ROM 102 is a non-volatile storage device typified by a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an SD card, etc. In addition to being used as a permanent storage area for an OS, various programs, and various data, the ROM 102 is also used as a storage area for various short-term data.

[0018] The RAM 103 is a high-speed volatile storage device typified by a DRAM or the like, where the OS, various programs, and various data are loaded, and it is also used as a working area for the OS and various programs.

[0019] The video output I / F 104 is an interface for outputting the image captured by the imaging sensor 107 described later to an external device, and is composed of SDI, HDMI (registered trademark), etc.

[0020] The network I / F 105 is an interface for connecting to the aforementioned LAN 300, and is responsible for communication with external devices such as the PC 200 via a communication medium such as Ethernet (registered trademark).

[0021] The image processing unit 106 is connected to the imaging sensor 107, and performs various image processes such as defect correction, NR processing, and color conversion processing on the image data acquired from the imaging sensor 107 based on instructions from the CPU 101, or performs processing such as converting to a predetermined format and compressing. The processed image data is stored in the RAM 103.

[0022] The imaging sensor 107 is an image sensor having an imaging element such as a CCD or CMOS, and functions as an imaging means in the PTZ camera 100. The imaging sensor 107 generates image data by photoelectrically converting the subject image formed by an imaging optical system (not shown). In this embodiment, the image data is output as a digital signal to the image processing unit 106 by an AD conversion circuit included in the imaging sensor 107, but it may also be output as an analog signal. Also, the imaging sensor 107 and the image processing unit 106 may be integrated in a stacked chip configuration. Further, in this embodiment, the imaging sensor 107 also has an imaging optical system for receiving the subject image by the imaging element in an integrated configuration, but the imaging optical system may be configured to be detachable or replaceable with respect to the imaging sensor 107, for example. Incidentally, in this embodiment, the imaging optical system and the imaging sensor 107 may be collectively referred to as imaging means.

[0023] The drive I / F 108 is an interface for transmitting the instructions received from the CPU 101 to the drive unit 109.

[0024] The drive unit 109 is a mechanical mechanism and an optical system for changing the imaging direction of the PTZ camera 100. In this embodiment, the imaging direction is changed by rotationally driving the imaging sensor 107. The drive unit 109 is composed of a mechanical drive system and a motor as a drive source. The drive unit 109 performs rotational driving such as pan-tilt operations for changing the imaging direction in the horizontal and vertical directions based on the instructions received from the CPU 101 via the drive I / F 108. Also, when the imaging optical system has a zoom lens (also called a varifocal lens), zoom control for optically changing the imaging angle of view may be performed by moving the zoom lens in the optical axis direction.

[0025] The inference unit 111 executes inference processing according to an inference program using a learned inference model and inference parameters. The inference processing in the inference unit 111 can be executed by an arithmetic processing unit specialized for image processing and inference processing such as a GPU (Graphics Processing Unit). The GPU is a processor capable of performing a large number of multiply-accumulate operations and has an arithmetic processing ability to perform matrix operations of neural networks and the like in a short time. Also, the inference processing in the inference unit 111 may be realized by a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). Note that the inference processing may perform calculations in cooperation with the CPU 101.

[0026] The lamp 112 is a light source such as an LED (Light Emitting Diode) for indicating under what control the PTZ camera 100 is, and is also called a tally lamp. The CPU 101 changes the display pattern by receiving an instruction from the outside via the network I / F 105. For example, state display by color such as red indicating that the video is being used for broadcasting or recording, green indicating the preview state, and state display by blinking period are performed.

[0027] In this embodiment, the PC 200 includes a CPU 201, a ROM 202, a RAM 203, a network I / F (Interface) 204, a display unit 205, a user input I / F 206, and an internal bus 207 that communicably connects the above components to each other.

[0028] The CPU 201 controls each component of the PC 200 to manage the control of the entire device.

[0029] The ROM 202 is a non-volatile storage device represented by a flash memory, an HDD (Hard Disk Drive), an SSD (Solid State Drive), an SD card, etc. In addition to being used as a permanent storage area for the OS, various programs, and various data, the ROM 202 is also used as a storage area for various short-term data.

[0030] The RAM 203 is a high-speed volatile storage device represented by a DRAM, etc., where the OS, various programs, and various data are loaded, and it is also used as a working area for the OS and various programs.

[0031] The network I / F (Interface) 204 is an interface for connecting to the aforementioned LAN 300, and is responsible for communication with imaging devices such as the PTZ camera 100 and other external devices such as servers via a communication medium such as Ethernet (registered trademark).

[0032] The display unit 205 displays the image acquired from the PTZ camera 100 and the setting screen of the PC 200. The display unit 205 is, for example, a liquid crystal panel or an organic EL panel. Here, an example where the PC 200 has a display unit is shown, but the PC 200 and the display unit may be separate, for example, a display monitor that only displays the captured image and the PC 200 exist separately.

[0033] The user input I / F 206 has input devices (operation units) such as a keyboard, a pointing device (such as a mouse), a touch panel, and a switch, and receives instructions from the user to the PC 200. Note that the keyboard may be a software keyboard. The CPU 201 monitors the user input I / F 206, and when it detects an operation by the user on the user input I / F 206, it executes processing corresponding to the detected operation.

[0034] <Explanation of the basic operations of automatic tracking control and subject selection> Subsequently, the automatic tracking control for controlling the PTZ camera 100 to track a subject, which is the basic control in this system, and the subject selection for the tracking target for the PTZ camera 100 based on the user operation received by the PC 200 will be described.

[0035] First, the automatic tracking control will be described with reference to FIGS. 3 to 6.

[0036] FIG. 3 shows the control flow of the PTZ camera 100, and shows a series of processes for controlling the PTZ camera 100 according to the subject position detected from the captured image.

[0037] This control flow starts when the CPU 101 of the PTZ camera 100 receives a control command for executing automatic tracking control via the network I / F 105.

[0038] In step S301, the CPU 101 determines whether it has received a control command or an end command via the network I / F 105, and stores the received control command in the RAM 103. If the CPU 101 has received a control command (the operation state confirmation result is YES), it stores the received control command in the RAM 103 and transitions to step S302. If the CPU 101 determines that it has received an end command (the operation state confirmation result is NO), this control is terminated.

[0039] In step S302, the CPU 101 acquires the image data stored by the image processing unit 106 from the RAM 103.

[0040] In step S303, the CPU 101 determines information regarding the feature amount and position of the subject in each frame of the captured image data, and stores it in the RAM 103. Specifically, the CPU 101 inputs the image data acquired from the RAM 103 to the inference unit 111. Next, the CPU 101 stores the feature amount of the subject inferred by the inference unit 111 and the position information of the subject on the image in the RAM 103. The inference unit 111 has a learned model created using a machine learning method such as deep learning, receives an image as input data, and outputs, as output data, an ID for identification and position information as subject information. The position information is described as, but not limited to, the information of the upper left point, width, and height of the circumscribed rectangle in contact with the subject in the image, and the coordinates of the center of gravity.

[0041] The table shown in FIG. 6(a) shows the detected subject information. Here, FIG. 5(a) shows the information displayed on the display screen of the display unit in the operation of the PC 200 to be described later. As an example, there are three subjects within the angle of view. In FIG. 5(a), the position information of the subjects output in step S303 is superimposed as subject frames 503 to 505. Focusing on the subject frame 503 here, it corresponds to ID = 1 in FIG. 6(a), the upper left point of the frame is (x-1, y1), the size of the frame is (w1, h1), and the coordinates of the center of gravity are (gx1, gy1). It is assumed that the coordinates on the PTZ camera 100 are converted to the display coordinate system by the CPU 201 when displayed on the PC 200. In the present embodiment, the position information of the subject is specified using the inference unit 111, but it is not limited to this as long as the position information of the subject can be specified. For example, it may be possible to acquire the position information from the wireless communication terminal possessed by the subject and use it as the position information of the subject. The CPU 101 stores the output subject information in the RAM 103 and transitions to step S304.

[0042] In step S304, the CPU 101 determines whether it has received subject selection information as a control command from the PC 200 via the network I / F 105. If it has received the information, the CPU 101 stores the subject selection information received from the PC 200 in the RAM 103. In the present embodiment, the subject selection information will be described as indicating the coordinates (area) on the angle of view. If there is already a selected subject, this step always transitions to step S305 regardless of the subject selection information. In step S305, the CPU 101 reads out the subject information output in step S303 and the subject selection information received in step 304 from the RAM 103. The CPU 101 compares the position information in the subject information with the coordinate information included in the subject selection information to check whether the coordinate information is included in the position information of the subject. As a specific example, this will be described with reference to FIG. 5. Suppose the coordinates specified as the subject selection information are the point 506 shown in FIG. 5(a). Then, it will be included in the position information displayed as the subject frame 503, that is, the area that can be represented by the upper left point and the frame. Therefore, the CPU 101 determines that the subject corresponding to the subject frame 503 has been selected. As shown in FIG. 6(b), the CPU 101 assigns whether it is a tracking target to the subject information, stores it in the RAM 103, transmits it to the PC 200, and transitions to step S306.

[0043] In step S306, the CPU 101 calculates control position information (control information) in the automatic tracking control and stores the control position information in the RAM 103. The control position information is information (imaging parameters) such as the pan angle, tilt angle, and zoom field angle when controlling (moving) the imaging sensor 107 to an arbitrary position. The CPU 101 calculates the pan angle, tilt angle, zoom field angle, and angular velocity when controlling the coordinates of the center of gravity of the subject of the tracking target selected in step S305 to a position where it is centered in the field angle, and stores the calculated result in the RAM 103 as the control position information. When no subject is selected, there is no control position information, which is equivalent to having no control of the field angle. In step S301, if control position information is received as an external control command, external position control can be enabled by preferentially storing it in the RAM 103.

[0044] In step S307, the CPU 101 reads out the control position information stored in the RAM 103. Based on the control position information, the CPU 101 derives drive parameters (control content) for the pan / tilt / zoom to be performed in a desired direction at a desired speed. Specifically, it may be parameters for controlling the motors included in the drive unit 109, and the drive parameters may be converted by referring to a conversion table held in the RAM 103 in advance based on the operation amount included in the control position information.

[0045] In step S308, the CPU 101 controls the drive unit 109 via the drive I / F 108 based on the derived drive parameters, and the drive unit 109 rotates based on the drive parameters, so that the PTZ camera 100 performs a pan / tilt / zoom operation. According to this control flow calculated based on the subject selection information received in step S304, the PTZ camera 100 can perform operations according to the subject selection by the user sequentially sent from the PC 200.

[0046] Next, a flow for controlling the PTZ camera 100 based on a user operation on the PC200 shown in FIG. 4 will be described. The PTZ camera 100 is controlled based on a control command transmitted from the PC200. Also, the operation of the PTZ camera 100 is the operation when subject selection information is received in the control flow shown in steps S304 to S305 of FIG. 3. Therefore, the description thereof will be omitted here. This control flow is started when the CPU 201 detects, via the user input I / F 206, an operation to start the menu screen for displaying the image captured by the PTZ camera 100 and information exemplified in FIG. 5. At the start of this control flow, the CPU 201 displays, on the display unit 205, the menu screen 500 shown in FIG. 5(a), the state 501 of the tracking control, and the camera video 502 received from the PTZ camera 100. In this embodiment, it is described on the assumption that the tracking target is not selected, that is, the tracking control has not been started.

[0047] In step S401, the CPU 201 of the PC200 detects, via the user input I / F 206, a user operation to end the menu screen. If not detected, the process proceeds to S402; if detected, this control flow ends.

[0048] In step S402, the CPU 201 of the PC200 transmits, via the network I / F 204, a control command for acquiring information from the PTZ camera 100. When the CPU 101 of the PTZ camera 100 detects receipt of the command, it reads out the subject information output in step S303 of FIG. 3 from the RAM 103 and transmits it to the PC200 via the network I / F 105. Based on the received subject information, the CPU 201 of the PC200 calculates the coordinates so as to superimpose the subject frames 503 to 505 on the camera video 502 and updates the menu screen on the RAM 203. Although the tracking control has not been started yet, the user can know the subject information recognized by the PTZ camera 100. Thereafter, the CPU 201 proceeds to S403.

[0049] In step S403, the CPU 201 of the PC 200 detects whether the user is performing an operation of selecting a subject via the user input I / F 206. As an input method, it can be realized by the user operating a mouse to specify a point on the camera image 502, or a touch operation on the camera image 502 displayed on the touch panel, etc., but the method is not limited to these. Similar to step S304 in the control flow of FIG. 3 described above, if the user designates the point 506, the CPU 201 converts the point 506 on the camera image 502 into coordinates in the camera angle of view and transmits it to the PTZ camera 100, and then proceeds to step S404. In addition, when there is a tracking target to be described later, it is assumed that this step always proceeds to step S404 regardless of the user's operation.

[0050] In step S404, the CPU 201 of the PC 200 acquires the tracking state from the PTZ camera 100. The tracking state is information on whether it is the tracking target shown in FIG. 6(b) transmitted from the PTZ camera 100 in step S305 in the control flow of FIG. 3 described above. If it has already been given in step S402, it may be read from the RAM 103. When the CPU 201 confirms the ID of the tracking target, the PTZ camera 100 changes the display of the tracking control state 511 to indicate that it is in the tracking state. In addition to changing the character string, the method of change may also be realized by updating the display of the color and pattern. In this embodiment, it is assumed that the character string and color are updated. Also, by changing the display form of the subject frame corresponding to the ID of the tracking target, it is presented to the user whether it is the tracking target. Specifically, the subject frame 503 is updated to the subject frame 513. On the other hand, the subject frames 504 to 505 are not updated in display as shown in the subject 514 to the subject frame 515 respectively, and although detection is performed, they are presented to the user as a state where tracking is not being performed. The display form of the frame is an example, and the method of change such as pattern and color is not limited to this embodiment. The CPU 201 updates the menu screen of the RAM 203 based on this information and proceeds to step S405.

[0051] In step S405, the CPU 201 of the PC 200 controls the display unit 205 to display the menu screen 510 updated on the RAM 203. At the start of this control flow, it was in the state shown in Fig. 5(a), but due to the subject selection, the content has been updated to the screen shown in Fig. 5(b). Also, as shown in Fig. 5(c), if the subject designated as the tracking target does not fit within the angle of view of the PTZ camera 100, information indicating that the position information has become unknown as shown in Fig. 6(c) will be transmitted from the PTZ camera 100. When the coordinates of the tracking target cannot be confirmed, the CPU 201 can, for example, change the display of a character string and color as if it were lost as shown in the tracking control state 521, and notify the user of the loss of the tracking target.

[0052] Note that if the point specified by the user in step S403 is already the subject of the tracking target, the operation is to cancel the tracking target, and it is possible to switch the tracking target by selecting another subject. Also, although the tracking operation has been started by subject selection, it may be changed to explicitly instruct the start and stop of the tracking operation.

[0053] According to the basic operation described above, it is possible to perform automatic tracking control of the PTZ camera 100 according to the user's subject selection by the control flows of the PTZ camera 100 and the PC 200.

[0054] <Explanation of the tracking operation when multiple subjects are selected> Subsequently, the tracking operation when multiple subjects are selected, which is a characteristic operation of the present invention, will be described. The operation of the PTZ camera 100 will be described only with the differences from the control flow of Fig. 3 described above, and the operation of the PC 200 will be described only with the differences from the control flow of Fig. 4 described above, and will be described with reference to Figs. 7 to 9.

[0055] The control flow shown in Fig. 7(a) is started in the same manner as the control flow shown in Fig. 3. Steps S701 to 703, 705, 707 to 709 operate in the same manner as steps S301 to 303, 305, 306 to 308, respectively.

[0056] In step S705, it is assumed that, similar to step S305, the PTZ camera 100 transmits subject information for three people. However, unlike in Fig. 5(a), the PC200 is displaying the menu screen shown in Fig. 8(a). That is, the currently selected number of selection targets 801, the number of tracking targets 802 for which position information can be obtained, and the number of detected subjects 803 that are currently detected are displayed together. Since there are no subjects selected yet, the CPU 201 of the PC200 updates the menu screen on the RAM 203 assuming that the number of selection targets 801 and the number of tracking targets 802 are 0, and the number of detected subjects 803 is 3.

[0057] In step S704, the CPU 101 of the PTZ camera 100 determines whether it has received subject selection information as a control command from the PC200 via the network I / F 105. Similar to the basic operation, when the point 804 is specified, the CPU 101 of the PTZ camera 100 transmits the information shown in Fig. 6(b) in a later step S705. At this time, the CPU 201 of the PC200 makes a display indicating that it has been selected as a tracking target as shown in the subject frame 813, and at the same time updates the menu screen with the number of selection targets 811 as 1 and the number of tracking targets 812 as 1.

[0058] In step S706, the CPU 101 of the PTZ camera 100 compares the tracking state received from the PTZ camera 100 with the number of selected targets. Step S706 is equivalent to the control flow shown in Fig. 7(b). Assuming that the transition to step S710 has occurred, the explanation will continue using Fig. 7(b). At the current time, as shown in the information in Fig. 6(b), it is assumed that there is one tracking target for which position information can be obtained and the number of selected targets is also 1. When the CPU 101 of the PTZ camera 100 finishes the process of step S706, it transmits the state of the tracking control to the PC200 via the network I / F 105.

[0059] In step S710, the CPU 101 determines whether the number of selection targets is 0. As described above, since the number of selection targets at the current time is 1, the process proceeds to S711. Note that when the control flow shown in FIG. 7(a) starts, the number of selection targets is 0, and it remains 0 while nothing is selected in step S704. In that case, the CPU 101 proceeds to S713, transmits information so that the tracking control state 800 shown in FIG. 8(a) is displayed as "stopped", and the CPU 201 updates the menu screen on the RAM 203.

[0060] In step S711, the CPU 101 determines whether the number of tracking objects is 0. Since the number of tracking objects is 1 at the current time, the CPU 101 proceeds to step S712. When it becomes 0, the transition to step S714 will be described later.

[0061] In step S712, the CPU 101 compares the number of tracking objects with the number of selection targets. If the number of selection targets matches the number of tracking objects, the CPU 101 proceeds to step S715; if they do not match, it proceeds to step S716. Since both are 1 at the current time, the process proceeds to step 715. In step S715, the CPU 101 transmits information so that the tracking control state 810 is displayed as "tracking". The CPU 201 of the PC 200 performs control to update the menu screen on the RAM 203 as shown in FIG. 8(b). Step S716 will be described later. The CPU 101 completes the control flow shown in FIG. 7(b) through steps S713 to S716 and proceeds to step S706 of the control flow shown in FIG. 7(a).

[0062] After the operations up to this point, the menu screen shown in Fig. 8(b) is displayed. Next, by designating point 814, as a result of going through the same control flow, the number of selection targets 821 becomes 2 and the number of tracking targets 822 becomes 2, and the display of the subject frame 823 is updated to the menu screen shown in Fig. 8(c). Furthermore, by designating point 824, the number of selection targets 831 becomes 3 and the number of tracking targets 832 becomes 3, and the display of the subject frame 833 is updated to the menu screen shown in Fig. 8(d). When it is the menu screen shown in Fig. 8(d), it is assumed that the subject information transmitted by the PTZ camera 100 and received by the PC 200 is as shown in Fig. 9(a).

[0063] In the PTZ camera 100, control is required to track the subject selected as the tracking target. Taking the state of Fig. 9(a) as an example, the calculation method of the control position information calculated in step S306 is changed. If there is one selected subject, instead of using the coordinates of the center of gravity of that subject, by calculating the control position information by further taking the center of gravity of the coordinates of the three subjects, the PTZ camera 100 can be directed at the center of the selected subject. Regarding zoom, in addition to the center of gravity, the width and height of the entire subject frame may be considered and calculated according to the ratio with the viewing angle, and the calculation method is not limited to this.

[0064] Here, consider the case where one of the subjects has moved outside the viewing angle of the PTZ camera 100. When it becomes like the camera image 843 shown in Fig. 8(e), the subject information transmitted in step S705 is as shown in Fig. 9(b). At this time, in step S712, the number of tracking targets with position information is 2 and the number of selection targets is 3. Therefore, the transition is to step S716, and the CPU 101 transmits information so as to display "partially lost" in step S716, and the CPU 201 updates the menu screen on the RAM 203. As shown in Fig. 8(e), the tracking control state 840 is displayed as "partially lost", the number of tracking targets 841 is 2, and the number of detected targets 842 is also 2, and the user can recognize that the specified number of targets cannot be tracked.

[0065] Furthermore, consider the case where all the subjects have moved outside the angle of view of the PTZ camera 100 and a subject 854 that was not initially being tracked has entered the angle of view. When the camera image 853 shown in Fig. 8(f) is obtained, the subject information will be as shown in Fig. 9(c). At this time, in step S711, since the number of tracked objects with position information becomes 0, the process transitions to step S714. The CPU 101 transmits information so as to display "lost" in step S714, and the CPU 201 updates the menu screen on the RAM 203. As shown in Fig. 8(f), the tracking control state 850 is displayed as "lost", the number of tracked objects 852 is 0, and the number of detected objects 851 is 1. The user can recognize that none of the specified target objects are being tracked.

[0066] In addition, in the case of a subject that is not a tracking target, a frame equivalent to the detection result during stoppage is displayed. In addition to moving outside the subject angle of view, there may be a state where the subject cannot be detected depending on the posture and orientation. Also, not only can the user explicitly specify the coordinates, but a subject that has been continuously detected within the angle of view for a certain period of time (detected for a predetermined time or a predetermined number of times) may be automatically determined as a tracking target and set to the selected state. Also, it may be excluded from the tracking targets due to having moved out of the angle of view for a certain period of time (not being detected for a predetermined time or a predetermined number of times).

[0067] In this embodiment, an example of an embodiment using a PTZ camera 100 has been described, but it is not limited to this form. For example, it is also applicable to devices that can detect a plurality of subjects from an image and can control a PTZ camera based on the detection results. Specifically, examples include edge devices and PCs equipped with an image input unit, a network communication unit, and a GPU.

[0068] According to the first embodiment described above, in an imaging system that tracks a plurality of subjects, it is possible to drive the imaging system considering the increase and decrease of the tracking targets and present the changes in the tracking state to the user.

[0069] [Second Embodiment] In the first embodiment, the user selected a plurality of subjects by sequentially selecting subjects. However, it is also conceivable to save the trouble of sequential selection by selecting subjects determined in advance. Therefore, in this embodiment, the presentation to the user in the method of managing a plurality of subjects will be described with reference to FIGS. 9 to 12 and FIGS. 13 to 14.

[0070] FIG. 10 shows an example of subject management. In the pool 1000, information based on feature amounts such as faces regarding subjects A to E registered in advance by the user is being managed. The registration method is such that the CPU 201 of the PC 200 transmits the information input by the user using the display unit 205 and the user input I / F 206 of the PC 200 through the network I / F 204 by a UI (not shown). The CPU 101 of the PTZ camera 100 stores it in the ROM 102 and the RAM 103 via the network I / F 204. It may be realized by storing the sequentially selected subjects each time as in the first embodiment, or by transmitting separately created information to the PTZ camera 100, and the method is not limited to this. What the user has selected from the pool 1000 as a group and registered are the first group 1001 and the second group 1002. In the first group 1001, subjects A to C are selected, which is close to the state where the number of targets is selected as 3 in the first embodiment.

[0071] In the same system configuration as the first embodiment, the changes in the control flows of FIGS. 3 and 4 will be described. As shown in FIG. 11(a), it is assumed that three subjects are detected and the selection target 1100 is in an unselected state. Here, the CPU 201 of the PC 200 displays the pool 1000, the first group 1001, and the second group 1002 shown in FIG. 10 on the display unit 205 by a UI (not shown), instead of the subject selection in step S403.

[0072] First, assume that the user selects the first group 1001 via the user input I / F 206. Then, the selected group 1110 in FIG. 11(b) is displayed, and the CPU 201 transmits the selection information to the PTZ camera 100. The CPU 101 of the PTZ camera 100 receives the subject selection information in step S304, but instead of the coordinates in the first embodiment, it receives the information that the first group 1001 has been selected. Subsequently, in step S305, the CPU 101 searches for the objects that match the subjects A to C associated with the first group 1001 from the subject information, rather than by coordinates. If A to C match in this embodiment, as shown in FIG. 12(a), the association of A to C is made in the subject information. Since the number of selection targets is 3 and the number of tracking targets is 3, subsequently, in step S705, it is determined that "tracking" as shown in the tracking control state 1111, and the CPU 201 of the PC 200 will display the menu screen in FIG. 11(b). Also, similar to the first embodiment, consider the case where one of the subjects is not detected within the viewing angle of the PTZ camera 100. When the camera image 1120 shown in FIG. 11(c) is obtained, the subject information transmitted in step S705 will be as shown in FIG. 12(b). At this time, in step S712, since there are 2 tracking targets with position information and the number of selection targets is 3, the process transitions to step S716, and the CPU 101 transmits information so as to display "partially lost" as shown in the tracking control state 1121 in step S716, and the CPU 201 updates the menu screen on the RAM 203.

[0073] Next, the case where a subject outside the group appears will be described. Assume that the camera image 1130 shown in FIG. 11(d) is obtained, and a subject 1131 corresponding to the subject D and a subject 1132 not corresponding to the pool 1000 appear within the angle of view. The subject information becomes as shown in FIG. 12(c). In step S711, since the number of tracking objects becomes 0, information is transmitted so that the tracking control state 1133 is displayed as "lost", and the CPU 201 updates the menu screen on the RAM 203. Note that the group information selected by the user in the past may be stored in the ROM 102 and read by the CPU 101 at the start of the control flow, so that the group information may be set without the user selecting it again.

[0074] Next, assume that the user selects the second group 1002 via the user input I / F 206. In the second group 1002, a priority setting for the selected subject is made, and it is assumed that the subject A has a higher priority than the subjects D to E. The difference in the tracking operation due to the priority can be realized by changing the above-mentioned center-of-gravity calculation in the calculation of the control position information in step S306.

[0075] Assume that the second group 1002 is selected and the menu screen shown in FIG. 13(a) is displayed. The selection target 1300 is the second group, and the subject information is transmitted and received between the PTZ camera 100 and the PC 200 as shown in FIG. 14(a). As shown in FIG. 14(a) of the subject information, a type as a priority is added, and the subject A is set as the main subject, and the subjects D to E are set as the secondary subjects. Using this information, the CPU 201 of the PC 200 updates the display so that the detected number 1301 of the main subjects is 1 and the detected number 1302 of the secondary subjects is 2.

[0076] Here, the case where the subject is not detected within the angle of view will be described. In step S716 described above, the mismatch between the number of tracking objects and the number of selection targets was displayed in the form of "partially lost", but by the transformation described later due to the addition of the priority, more information can be displayed to the user. Consider the case where the main subject A is not detected within the angle of view, that is, the subject information shown in FIG. 14(b).

[0077] The displayed menu screen becomes "Partially Lost" according to step S716. However, in order to emphasize that the main subject has not been detected, the CPU 101 of the PTZ camera 100 resets the tracking control state 1310 to "Main Lost". Further, the menu screen is updated with the detected number 1311 of the main subject set to 0. Similarly, for the subject information shown in FIG. 14(c), since the secondary subject E has not been detected, the tracking control state 1320 is reset to "Secondary Lost", and further, the menu screen is updated with the detected number 1321 of the secondary subject set to 1, resulting in the display of FIG. 13(c). Also, for the subject information shown in FIG. 14(d), since the main subject A and the secondary subject D have not been detected, the tracking control state 1330 is reset to "Main and Secondary Lost", the detected number 1331 of the main subject is 0, and the detected number 1332 of the secondary subject is 1. Similar to when the first group is selected, when only the unselected subjects and the subjects outside the group are detected, it is "Lost".

[0078] Note that instead of the user explicitly selecting a group, the CPU 201 may determine that the subject detection result matches any of the group information and then select the group. In that case, as the priority order when matching multiple groups, controls such as continuously setting the first found group or preferentially setting a specific group may be performed.

[0079] According to the second embodiment described above, a similar effect can be obtained by presenting the user with information in which the selection method and type of multiple subjects are changed from the first embodiment.

[0080] [Third Embodiment] In the first and second embodiments, it was presented to the user by updating the menu screen. As a method for more intuitively determining the state of the tracking control, it may be presented to the user by switching the display of the lamp 112 provided in the PTZ camera 100. An example in which the state of the tracking control is replaced with the lighting pattern of the lamp 112 is shown in FIG. 15. When the CPU 101 determines and transmits the information indicating the state of the tracking control in S706, it also controls the lamp 112 to change the display. The same display may be continued until the control flow ends or the state of the tracking control is updated. In the present embodiment, as described in the configuration of the PTZ camera 100, since the color indicates how the video is used, an example realized by changing patterns such as lighting and blinking is shown. For "stopped", since there is no need to display the state regarding tracking, the light is turned off. For "tracking", since it indicates that tracking is possible without problems, it is lit in the same way as during normal imaging. For "lost", "major lost", and "major and minor lost", since they are not preferable as videos and it can be determined that interruption or manual operation intervention is immediately required, it is switched from lighting to blinking at a cycle of 0.5 seconds. For "partial lost" and "minor lost", although they are not as preferable as "major and minor lost", since they indicate a state with problems, they are switched to blinking at a cycle of 1 second. The switching of the display for distinguishing the states is just an example and is not limited to this, and the pattern may be changed by changing at least a part of the blinking, intensity, color, number, etc. of the LEDs of the lamp 112.

[0081] According to the third embodiment described above, the same effects as those of the first and second embodiments can be obtained in a form in which the notification form is changed.

[0082] [Fourth Embodiment] In the first to third embodiments, the detection of the subject and the control of the tracking are performed by the PTZ camera 100 which is an imaging device, but the PC 200 which is an information device or further a server which is an external device may be responsible for the detection and tracking of the subject. In the following description, an example in which the PC 200 is responsible will be described.

[0083] The configuration diagram of this embodiment is shown in FIG. 16. The CPU 101 to the internal bus 110 of the PTZ camera 100 are the same as those in other embodiments, and the inference unit 111 and the lamp 112 are removed. The CPU 201 to the internal bus 207 of the PC 200 are the same as those in other embodiments, and an inference unit 208 is added. It is assumed that the operations of the inference unit 111 and the inference unit 208 are equivalent. The PC 200 receives only video from the PTZ camera 100 and transmits only information on pan control and tilt control, and the subject information is handled only within the PC 200.

[0084] FIG. 17 shows the control flow of the PC 200. This control flow is started when the CPU 201 of the PC 200 receives a control command for executing automatic tracking control via the network I / F 204 or the user input I / F 206.

[0085] In step S1701, the CPU 201 determines whether it has received a control command or an end command via the network I / F 204, and stores the received control command in the RAM 203. If the CPU 201 has received a control command (the operation state confirmation result is YES), the received control command is stored in the RAM 203, and the process proceeds to step S1702. If the CPU 101 determines that it has received an end command (the operation state confirmation result is NO), this control is terminated.

[0086] In step S1702, the CPU 201 receives the captured image of the PTZ camera 100 via the network I / F 204 and stores it in the RAM 203.

[0087] In step S1703, the CPU 201 uses the inference unit 208 to determine information on the feature amount and position of the subject from the captured image in the same manner as in step S303, and stores it in the RAM 203.

[0088] In step S1704, the CPU 201 determines the selection information of the subject in the same manner as in step S304. In the present embodiment, a menu screen may be displayed using the display unit 205, and the coordinates on the angle of view may be detected via the user input I / F 206 in the same manner as in the first embodiment. Alternatively, in the same manner as in the second embodiment, the selection information of the group may be received via the user input I / F 206. The subject information is stored in the RAM 203.

[0089] In step S1705, the CPU 201 determines the tracking target in the subject information in the same manner as in step S705, but the subject information is stored in the RAM 203 without being transmitted externally.

[0090] In step S1706, the CPU 201 calculates the control position information in the same manner as in step S306. The calculated control position information is transmitted to the PTZ camera 100 via the network I / F 204. The CPU 101 of the PTZ camera 100 executes the pan-tilt-zoom operation according to the instruction from the PC 200 by performing the control of steps S307 to S308 based on the information.

[0091] In step S1707, the CPU 201 compares the tracking state with the selected target in the same manner as in step S706. Similar to other embodiments, the state of the tracking control is determined from the selected target and the number of tracked objects, and is stored in the RAM 203.

[0092] In step S1707, the CPU 201 displays the captured image, the subject information, and the state of the tracking control stored in the above-described respective steps on the display unit 205 in the same manner as in other embodiments.

[0093] According to the fourth embodiment described above, in the PC 200 which is an information device, the same effects as in other embodiments can be obtained.

[0094] <Other Embodiments> The present invention can also be realized by a process in which a program that implements one or more functions of the above-described embodiment is supplied to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the described program. Further, it can also be realized by a circuit (for example, an ASIC) that implements one or more functions. The invention is not limited to the above-described embodiment, and various changes and modifications are possible without departing from the scope of the claims.

Claims

1. Detection means for detecting a plurality of subjects from an image, Selection means for selecting a subject to be tracked, Tracking means for tracking the subject to be tracked selected by the selection means using information on the plurality of subjects detected by the detection means, Notification means for notifying the tracking state by the tracking means, and having, The information processing apparatus, wherein the notification means controls notification according to the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means.

2. The information processing apparatus according to claim 1, wherein the notification means changes the content of notification when the number of subjects selected by the selection means matches the number of subjects that can be tracked by the tracking means and when they do not match.

3. The information processing apparatus according to claim 1, wherein the notification means notifies, as the tracking state, the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means.

4. The selection means according to claim 1, wherein the selection means selects a plurality of subjects detected by the detection means when they are continuously detected for a predetermined time or a predetermined number of times, and excludes them when they are not detected for a predetermined time or a predetermined number of times. Information processing apparatus described.

5. Further having subject management means for identifying and managing subjects, The information processing apparatus according to claim 1, wherein the subject management means compares a plurality of subjects selected by the selection means from the subject information held by the subject management means with the subjects detected by the detection means, and notifies when they do not match.

6. The information processing apparatus according to claim 5, wherein the selection means compares the subject information held by the subject management means with the subjects detected by the detection means, and selects the subject as a tracking target when they match.

7. The information processing apparatus according to claim 5, wherein the subject management means sets a priority for the subject information held.

8. The information processing apparatus according to claim 1 or claim 5, wherein the tracking means changes the control content according to the number of subjects selected by the selection means.

9. The information processing apparatus according to claim 8, wherein the control content is pan control, tilt control, and zoom control.

10. The information processing apparatus according to claim 1, wherein the notification means performs notification by switching a lighting pattern by a tally lamp as notification of the following state.

11. Imaging means for imaging and outputting an image; Detection means for detecting a plurality of subjects from the image; Selection means for selecting a subject to be tracked; Tracking means for tracking the subject to be tracked selected by the selection means using information on a plurality of subjects detected by the detection means; Notification means for notifying the tracking state by the tracking means, and The information processing system, wherein the notification means controls notification according to the number of subjects selected by the selection means and the number of subjects that can be tracked by the tracking means. Information processing system

12. A detection step of detecting a plurality of subjects from an image; A selection step of selecting a subject to be tracked; A tracking step of tracking the subject to be tracked selected by the selection means using information on a plurality of subjects detected in the detection step; A notification step for notifying the tracking state by the tracking step, and The information processing method, wherein in the notification step, notification is controlled according to the number of subjects selected in the selection step and the number of subjects that can be tracked by the tracking means.

13. A computer-executable program describing the procedure of the control method of the imaging device according to claim 12.

14. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method of the imaging device according to claim 12.

Images