System, control device, control method, and program

Abstract

This system provides the capability to track a specific subject using multiple imaging devices with different shooting positions and directions. [Solution] A system for controlling a second imaging device to track a predetermined subject based on a first image taken by a first imaging device or a second image taken by a second imaging device, with different shooting directions, wherein the first control device controls the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the first image, the second control device controls the second imaging device based on the second image, and a comparison means for comparing first control information including at least one of the pan value and tilt value of the second imaging device calculated based on the first image with second control information of the second imaging device calculated based on the second image, thereby switching between a first state in which the second imaging device is controlled from the first image and a second state in which the second imaging device is controlled from the second image.

Description

Technical Field

[0001] The present invention relates to a system for tracking a specific subject by a plurality of imaging devices having different shooting positions and shooting directions.

Background Art

[0002] There is a technique for tracking a specific subject by an imaging device capable of remotely automatically controlling pan / tilt / zoom (PTZ). In such automatic tracking control, the PTZ is automatically controlled so that the subject to be tracked is arranged at a desired position within the shooting angle of view.

[0003] Patent Document 1 describes a technique for tracking a specific subject by cooperating an imaging device with a fixed wide-angle shooting angle (fixed-angle camera) and an imaging device having a PTZ function (PTZ camera). In Patent Document 1, even when the subject to be tracked moves outside the shooting angle of the fixed-angle camera and cannot be captured, the movement of the subject to be tracked is predicted so that it can be captured by the PTZ camera.

[0004] Further, Patent Document 2 describes a technique for transmitting template data of a subject to be tracked generated by a first imaging device to a second imaging device when the subject to be tracked moves near the boundary of the shooting range of the first imaging device, and having the second imaging device take over the subject to be tracked.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0006] However, in Patent Documents 1 and 2, the subject to be tracked is identified by template matching. Therefore, when tracking a specific subject with multiple imaging devices, it is necessary to arrange the multiple imaging devices so that their shooting positions and directions are close together. Consequently, if the shooting positions and directions of the multiple imaging devices are spaced far apart, it becomes difficult to track a specific subject with the multiple imaging devices.

[0007] The present invention has been made in view of the above problems, and its objective is to realize a system that can track a specific subject using multiple imaging devices with different shooting positions and directions. [Means for solving the problem]

[0008] To solve the above problems and achieve the objective, the present invention provides a system comprising: a first imaging device and a second imaging device having different shooting directions; a first control device and a second control device for controlling the second imaging device to track a predetermined subject based on a first image captured by the first imaging device or a second image captured by the second imaging device, wherein the first control device has a first control means for controlling the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the first image; the second control device has a second control means for controlling the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the second image; and the system comprises: first control information including at least one of the pan value and tilt value of the second imaging device calculated based on the first image; and second control information including at least one of the pan value and tilt value of the second imaging device calculated based on the second image. The system has a comparison means for comparing the first and second control information, and based on the result of the comparison by the comparison means, it switches between a first state in which the first control device controls the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the first image, and a second state in which the second control device controls the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image, the first control device transmits the first control information to the second control device, the first control means switches to the second state if the difference between the first control information and the second control information satisfies a predetermined condition based on the result of the comparison by the comparison means, and switches back to the first state if the predetermined condition is not met, the predetermined condition being that the difference between the first control information and the second control information is greater than or equal to a threshold, the comparison means calculates the difference between the first control information and the second control information and outputs the result of comparing the difference with the threshold. [Effects of the Invention]

[0009] According to the present invention, it becomes possible to track a specific subject using multiple imaging devices with different shooting positions and directions. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram illustrating the system configuration of Embodiment 1. [Figure 2] A diagram illustrating the hardware configuration of the devices constituting the system of Embodiment 1. [Figure 3] A diagram illustrating the functional configuration of the devices constituting the system of Embodiment 1. [Figure 4] A flowchart illustrating the basic operation of the devices constituting the system of Embodiment 1. [Figure 5] A diagram illustrating the coordinate transformation method for captured images in Embodiment 1. [Figure 6] A diagram illustrating the subject detection method and coordinate transformation method of Embodiment 1. [Figure 7] A diagram illustrating the pan control of Embodiment 1. [Figure 8] A diagram illustrating the tilt control of Embodiment 1. [Figure 9] A flowchart illustrating the control process of Embodiment 1. [Figure 10] A diagram illustrating the method for determining the subject to be tracked in Embodiment 1. [Figure 11] A diagram illustrating the functional configuration of the devices constituting the system of Embodiment 2. [Figure 12] A flowchart illustrating the control process of Embodiment 2. [Figure 13] A flowchart illustrating the control process of Embodiment 2. [Figure 14] A diagram illustrating the system configuration of Embodiment 3. [Figure 15] A diagram illustrating the roles and functions that can be set for the imaging device of Embodiment 3. [Modes for carrying out the invention]

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although a plurality of features are described in the embodiments, not all of these plurality of features are essential for the invention, and the plurality of features may be arbitrarily combined. Further, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant explanations are omitted.

[0012] [Embodiment 1] <System Configuration> First, referring to FIG. 1, the system configuration of Embodiment 1 will be described.

[0013] The system of this embodiment includes a first control device 100, a second control device 200, a first imaging device 300, and a second imaging device 400. The system of this embodiment controls the second imaging device 400 by either the first control device 100 or the second control device 200 to track a specific subject. In this embodiment, the specific subject is, for example, a person, but may also be an animal or an object.

[0014] The first control device 100 detects a subject to be tracked from an aerial image captured by the first imaging device 300, and controls the second imaging device 400 based on the detection result. The first control device 100 is also called a workstation. The subject to be tracked is set, for example, by user operation or automatically.

[0015] The second control device

[0016] controls the second imaging device 400 based on the subject recognition result of the subject to be tracked from the aerial image captured by the first imaging device 300 and the subject recognition result of the subject to be tracked from the sub-image captured by the second imaging device 300. The second control device 200 is also called an edge box.

[0016] The first imaging device 300 has a fixed wide-angle field of view and is capable of capturing an overhead image that includes all of subjects A, B, and C. The first imaging device 300 is also called an overhead camera. The second imaging device 400 has a variable field of view and is capable of capturing at least one of subjects A, B, and C. The second imaging device 400 is called a sub-camera. The first imaging device 300 and the second imaging device 400 are positioned far apart from each other so that their shooting positions and / or shooting directions are different.

[0017] The first control device 100, the second control device 200, the first imaging device 300, and the second imaging device 400 are connected to each other via a network 600 such as a LAN (Local Area Network). In this embodiment, an example is described in which the first control device 100, the second control device 200, the first imaging device 300, and the second imaging device 400 are connected via a network 600, but they may also be connected by connection cables (not shown). In this embodiment, an example is described in which there is one second imaging device 400, but there may be two or more. If there are multiple second imaging devices 400, a second control device 200 is provided for each second imaging device 400.

[0018] Next, the basic functions of the system in this embodiment will be described.

[0019] The first imaging device 300 captures an overhead image and transmits the overhead image to the first control device 100 via the network 600.

[0020] The second imaging device 400 captures a sub-image including the subject being tracked (tracked subject) and transmits the sub-image to the second control device 200 via the network 600. The second imaging device 400 also has a PTZ function. The PTZ function is a function that can control the pan, tilt, and zoom of the imaging device. PTZ is an abbreviation of the first letters of Pan, Tilt, and Zoom. Pan is the horizontal movement of the optical axis of the imaging device. Tilt is the vertical movement of the optical axis of the imaging device. Zoom is zoom in (telephoto) and zoom out (wide-angle). Pan and tilt are functions that change the shooting direction of the imaging device. Zoom is a function that changes the shooting range (shooting angle of view) of the imaging device.

[0021] The first control device 100 determines the subject to be tracked from the subject detected in the overhead image received from the first imaging device 300, and calculates first characteristic information of the subject to be tracked from the overhead image. Based on the first characteristic information of the subject to be tracked, the first control device controls the second imaging device 400 to change the shooting direction and shooting range of the second imaging device 400 to match the shooting direction and shooting range of the subject to be tracked.

[0022] After changing the shooting direction and shooting range of the second imaging device 400 to match the shooting direction and shooting range of the tracked subject, the first control device 100 transmits the first characteristic information of the tracked subject, calculated from the overhead image, to the second control device 200.

[0023] The second control device 200 detects a subject from the sub-image received from the second imaging device 400 and calculates second characteristic information of the detected subject. The second control device 200 compares the second characteristic information of the subject detected from the sub-image with the first characteristic information of the tracked subject received from the first control device 100.

[0024] If the similarity between the first characteristic information of the tracked subject and the second characteristic information of the subject detected from the sub-image is low, the first control device 100 controls the second imaging device 400 to change the shooting direction and shooting range of the second imaging device 400 to the shooting direction and shooting range of the tracked subject based on the first characteristic information of the tracked subject.

[0025] Furthermore, if the similarity between the first characteristic information of the tracked subject and the second characteristic information of the subject detected from the sub-image is high, the second control device 200 controls the second imaging device 400 to change the shooting direction and shooting range of the second imaging device 400 to the shooting direction and shooting range of the tracked subject based on the second characteristic information of the subject detected from the sub-image which has a high similarity to the first characteristic information of the tracked subject.

[0026] Feature information is information that allows for the identification of the same subject when it is captured by multiple imaging devices with different shooting positions and / or directions. Feature information is the inference result output by performing image recognition using a trained model, with multiple images of the same subject taken by multiple imaging devices with different shooting positions and / or directions as input. If the inference result indicates that the subjects are the same, it is possible to identify that the subjects included in multiple images taken by multiple imaging devices with different shooting positions and / or directions are the same subject.

[0027] In the following explanation, the first control device 100 will be referred to as a workstation (WS), the second control device 200 as an edge box (EB), the first imaging device 300 as an overhead camera, and the second imaging device 400 as a sub-camera.

[0028] <Device configuration> Next, with reference to Figure 2, the hardware configuration of the WS100, EB200, overhead camera 300, and sub-camera 400 will be described in detail.

[0029] First, let me explain the configuration of WS100.

[0030] The WS100 comprises a control unit 101, a volatile memory 102, a non-volatile memory 103, an inference unit 104, a communication unit 105, and an operation unit 106, with each unit connected via an internal bus 110 to enable data transmission and reception.

[0031] The control unit 101 has a processor (CPU) that performs calculation and control processing for the WS100, and controls each component of the WS100 by executing a control program stored in the non-volatile memory 103.

[0032] The volatile memory 102 is a main memory such as RAM. The volatile memory 102 is loaded with constants and variables for the operation of the control unit 101, as well as control programs and inference programs read from the non-volatile memory 103. The volatile memory 102 also stores information such as image data and inference programs received from external devices by the communication unit 105. Furthermore, the volatile memory 102 stores overhead image data received from the overhead camera 300. The volatile memory 102 has sufficient storage capacity to hold this information.

[0033] The non-volatile memory 103 is an auxiliary storage device such as an EEPROM, flash memory, hard disk drive (HDD), solid state drive (SSD), or memory card. The non-volatile memory 103 stores the operating system (OS), which is the basic software executed by the control unit 101, control programs including applications that work in cooperation with the OS to realize advanced functions, and inference programs used by the inference unit 104 for inference processing.

[0034] The inference unit 104 performs inference processing using a trained inference model and inference parameters according to the inference program. The inference unit 104 performs inference processing to estimate the presence, location, and characteristic information of a specific subject from the overhead image received from the overhead camera 300. The inference processing in the inference unit 104 can be performed by a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). A GPU is a processor capable of performing a large number of multiply-accumulate operations and has the computing power to perform matrix operations of neural networks in a short time. Alternatively, the inference processing in the inference unit 104 may be implemented by a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The inference processing may be performed by the CPU and GPU of the control unit 101 working together, or by either the CPU or GPU of the control unit 101.

[0035] The communication unit 105 is an interface (I / F) compliant with wired communication standards such as Ethernet® or an interface compliant with wireless communication standards such as Wi-Fi®. The communication unit 105 can connect to external devices such as the EB200, overhead camera 300, and sub-camera 400 via a network 600 such as a wired LAN or wireless LAN, and can send and receive data with the external devices. The control unit 101 enables communication with the external devices by controlling the communication unit 105. Note that the communication method is not limited to Ethernet® or Wi-Fi®, and communication standards such as IEEE1394 may also be used.

[0036] The operation unit 106 consists of various switches, buttons, touch panels, and other operating components that receive various user operations and output operation information to the control unit 101. The operation unit 106 also provides a user interface for the user to operate the WS100.

[0037] The display unit 111 displays overhead images, subject recognition results, and a GUI (Graphical User Interface) for interactive operation. The display unit 111 is a display device such as a liquid crystal display or an organic EL display. The display unit 111 may be an integrated component of the WS100 or an external device connected to the WS100.

[0038] Next, I will explain the configuration of the EB200.

[0039] The EB200 comprises a control unit 201, a volatile memory 202, a non-volatile memory 203, an inference unit 204, and a communication unit 205, with each unit connected via an internal bus 210 to enable data transmission and reception.

[0040] The control unit 201 has a processor (CPU) that performs calculation and control processing for the EB200, and controls each component of the EB200 by executing a control program stored in the non-volatile memory 203.

[0041] The volatile memory 202 is a main memory such as RAM. The volatile memory 202 is loaded with constants and variables for the operation of the control unit 201, as well as control programs and inference programs read from the non-volatile memory 203. The volatile memory 202 also stores information such as image data and inference programs received from external devices by the communication unit 205. In addition, the volatile memory 202 stores sub-image data received from the sub-camera 400. The volatile memory 202 has sufficient storage capacity to hold this information.

[0042] The non-volatile memory 203 is an auxiliary storage device such as an EEPROM, flash memory, hard disk drive (HDD), solid state drive (SSD), or memory card. The non-volatile memory 203 stores the operating system (OS), which is the basic software executed by the control unit 201, control programs including applications that work in cooperation with the OS to realize advanced functions, and inference programs used by the inference unit 204 for inference processing.

[0043] The inference unit 204 performs inference processing using a trained inference model and inference parameters according to the inference program. The inference unit 204 performs inference processing to estimate the presence, location, and feature information of a specific subject from the sub-images received from the sub-camera 400. The inference processing in the inference unit 204 can be performed by a computing device specialized for image processing and inference processing, such as a GPU (Graphics Processing Unit). A GPU is a processor capable of performing a large number of multiply-accumulate operations and has the computational processing power to perform matrix operations of neural networks in a short time. Alternatively, the inference processing in the inference unit 204 may be implemented by a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array). The inference processing may be performed by the CPU and GPU of the control unit 201 working together, or by either the CPU or GPU of the control unit 201.

[0044] The communication unit 205 is an interface (I / F) compliant with wired communication standards such as Ethernet® or an interface compliant with wireless communication standards such as Wi-Fi®. The communication unit 205 can connect to external devices such as the WS100 and sub-camera 400 via a network 600 such as a wired LAN or wireless LAN, and can send and receive data with the external devices. The control unit 201 enables communication with the external devices by controlling the communication unit 205. Note that the communication method is not limited to Ethernet® or Wi-Fi®, and communication standards such as IEEE1394 may also be used.

[0045] Next, I will explain the configuration of the overhead camera 300.

[0046] The overhead camera 300 comprises a control unit 301, a volatile memory 302, a non-volatile memory 303, a communication unit 305, an imaging unit 306, and an image processing unit 307, with each unit connected via an internal bus 310 to enable data transmission and reception.

[0047] The control unit 301 controls the entire overhead camera 300 in accordance with the control of the WS100. The control unit 301 has a processor (CPU) that performs calculation and control processing for the overhead camera 300, and controls each component of the overhead camera 300 by executing a control program stored in the non-volatile memory 303.

[0048] The volatile memory 302 is a main memory such as RAM. The volatile memory 302 is loaded with constants and variables for the operation of the control unit 301, as well as control programs and inference programs read from the non-volatile memory 303. The volatile memory 302 also stores overhead image data captured by the imaging unit 306 and processed by the image processing unit 307. The volatile memory 302 has sufficient storage capacity to hold this information.

[0049] The non-volatile memory 303 is an auxiliary storage device such as an EEPROM, flash memory, hard disk drive (HDD), solid state drive (SSD), or memory card. The non-volatile memory 303 stores the operating system (OS), which is the basic software executed by the control unit 301, and control programs, including applications that work in cooperation with the OS to realize advanced functions.

[0050] The imaging unit 306 has an image sensor composed of a CCD (charge-coupled device), a CMOS (complementary metal-oxide-semiconductor) element, etc., which converts the optical image of the subject into an electrical signal. In this embodiment, the overhead camera 300 has a fixed shooting angle of view so that it can capture an overhead image that includes multiple subjects, including a tracked subject.

[0051] The image processing unit 307 performs various image processing operations on image data output from the imaging unit 306 or image data read from the volatile memory 302. These image processing operations include, for example, image processing operations such as noise reduction, edge enhancement, and scaling; image correction operations such as contrast correction, brightness correction, and color correction; and trimming or cropping operations to extract a portion of the image data. The image processing unit 307 converts the processed image data into an image file of a predetermined format (e.g., JPEG) and records it in the non-volatile memory 303. The image processing unit 307 also performs predetermined calculations using the image data, and the control unit 301 performs AF (autofocus) and AE (automatic exposure) operations based on the calculation results.

[0052] The communication unit 305 is an interface (I / F) compliant with wired communication standards such as Ethernet® or an interface compliant with wireless communication standards such as Wi-Fi®. The communication unit 305 can connect to external devices such as WS100 via a network 600 such as a wired LAN or wireless LAN, and can send and receive data with the external devices. The control unit 301 enables communication with the external devices by controlling the communication unit 305. Note that the communication method is not limited to Ethernet® or Wi-Fi®, and communication standards such as IEEE1394 may also be used.

[0053] Next, I will explain the configuration of sub-camera 400.

[0054] The sub-camera 400 comprises a control unit 401, a volatile memory 402, a non-volatile memory 403, a communication unit 405, an imaging unit 406, an image processing unit 407, an optical unit 408, and a PTZ drive unit 409, with each unit connected via an internal bus 410 to enable data transmission and reception.

[0055] The control unit 401 controls the entire sub-camera 400 in accordance with the control of WS100 or EB200. The control unit 401 has a processor (CPU) that performs calculation and control processing for the sub-camera 400, and controls each component of the sub-camera 400 by executing a control program stored in the non-volatile memory 403.

[0056] The volatile memory 402 is a main memory such as RAM. The volatile memory 402 is loaded with constants and variables for the operation of the control unit 401, as well as control programs and inference programs read from the non-volatile memory 403. The volatile memory 402 also stores overhead image data captured by the imaging unit 406 and processed by the image processing unit 407. The volatile memory 402 has sufficient storage capacity to hold this information.

[0057] The non-volatile memory 403 is an auxiliary storage device such as an EEPROM, flash memory, hard disk drive (HDD), solid state drive (SSD), or memory card. The non-volatile memory 403 stores the operating system (OS), which is the basic software executed by the control unit 401, and control programs, including applications that work in cooperation with the OS to realize advanced functions.

[0058] The imaging unit 406 has an image sensor composed of a CCD (charge-coupled device), a CMOS (complementary metal-oxide-semiconductor) element, etc., and converts the optical image of the subject into an electrical signal.

[0059] The image processing unit 407 performs various image processing operations on image data output from the imaging unit 406 or image data read from the volatile memory 402. These image processing operations include, for example, image processing such as noise reduction, edge enhancement, and scaling; image correction operations such as contrast correction, brightness correction, and color correction; and trimming or cropping operations to extract a portion of the image data. The image processing unit 407 converts the processed image data into an image file of a predetermined format (e.g., JPEG) and records it in the non-volatile memory 403. The image processing unit 407 also performs predetermined calculations using the image data, and the control unit 401 performs AF (autofocus) and AE (automatic exposure) operations based on the calculation results.

[0060] The communication unit 405 is an interface (I / F) compliant with wired communication standards such as Ethernet® or an interface compliant with wireless communication standards such as Wi-Fi®. The communication unit 405 can connect to external devices such as EB200 via a network 600 such as a wired LAN or wireless LAN, and can exchange data with the external devices. The control unit 401 enables communication with the external devices by controlling the communication unit 405. Note that the communication method is not limited to Ethernet® or Wi-Fi®, and communication standards such as IEEE1394 may also be used.

[0061] The optical unit 408 includes a lens group including a zoom lens and a focus lens, a shutter with an aperture function, and a mechanism for driving these optical components. The optical unit 408 drives the optical components to rotate the shooting direction of the sub-camera 400 around the pan (P) axis (horizontal direction) or the tilt (T) axis (vertical direction), or to change the shooting range (angle of view) of the sub-camera 400 along the zoom (Z) axis (magnification / reduction direction).

[0062] The PTZ drive unit 409 includes mechanical elements and actuators such as motors for driving the optical unit 408 in the PTZ direction, and drives the optical unit 408 in the PTZ direction according to the control of the control unit 401.

[0063] Furthermore, the zoom function in this embodiment is not limited to optical zoom, which changes the focal length by moving the zoom lens, but may also be digital zoom, which crops and enlarges a portion of the captured image data, or a combination of optical zoom and digital zoom may be used.

[0064] [Control Processing] Next, referring to Figures 3 to 10, we will explain the control process for tracking a subject by switching between a mode in which WS100 controls the sub-camera 400 based on an overhead image and a mode in which EB200 controls the sub-camera 400 based on a sub-image.

[0065] First, referring to Figures 3 and 4, the functional configurations of WS100 and EB200 for realizing the control processing of this embodiment will be described.

[0066] The functions of the WS100 and EB200 are implemented by hardware and / or software. If the functions shown in Figure 3 are implemented by hardware instead of software, it is sufficient to have the corresponding circuit configurations shown in Figure 3.

[0067] The WS100 includes an image recognition unit 121, a focus subject determination unit 122, a tracking target determination unit 123, a control information generation unit 124, a feature information determination unit 125, and a tracking state determination unit 126. The software that implements these functions is stored in the non-volatile memory 103, and the control unit 101 loads it into the volatile memory 102 and executes it.

[0068] The EB200 includes an image recognition unit 221, a tracking target determination unit 222, and a control information generation unit 223. This software is stored in a non-volatile memory 203, and the control unit 201 loads it into a volatile memory 202 and executes it.

[0069] Figure 4(a) is a flowchart showing the basic operation of WS100. Figure 4(b) is a flowchart showing the basic operation of EB200. Figure 4(c) is a flowchart showing the operation of overhead camera 300. Figure 4(d) is a flowchart showing the operation of sub-camera 400.

[0070] First, the functions and basic operation of the WS100 software will be explained with reference to Figures 3 and 4(a).

[0071] In step S101, the control unit 101 sends a shooting command to the overhead camera 300 using a predetermined protocol via the communication unit 105, receives an overhead image from the overhead camera 300, saves it to the volatile memory 102, and proceeds to step S102.

[0072] In step S102, the control unit 101 executes the function of the image recognition unit 121 shown in Figure 3, and proceeds to step S103.

[0073] The image recognition unit 121 controls the inference unit 104, the volatile memory 102, and the non-volatile memory 103, and performs the following subject recognition processing.

[0074] The image recognition unit 121 receives the overhead image IMG from the overhead camera 300 and the reference position information REF_POSI from the overhead camera 300, both read from the volatile memory 102. The position information REF_POSI from the overhead camera 300 includes the position of the overhead camera 300 and the marker coordinates. Based on the overhead image IMG and the reference position information REF_POSI from the overhead camera 300, the image recognition unit 121 detects the subject and calculates feature information. The image recognition unit 121 then outputs coordinate information POSITION[n] indicating the position of the detected subject, ID[n] indicating the identification information of the detected subject, and STAT[n] indicating the feature information of the detected subject. The position of the overhead camera 300 is the position in the coordinate space when viewing the shooting area of ​​the overhead camera 300 from directly above, and is known in advance by user operation or measurement using a sensor not shown. The marker coordinates are positional information of a marker placed in the coordinate space of the overhead camera 300's shooting area viewed from directly above, in order to calculate the homography transformation matrix described later. These are known values ​​measured in advance manually or using a sensor not shown. The marker is like a mark with a different color from the floor or ground, and can be anything as long as it can be measured by user operation or a sensor not shown. For example, if the sensor not shown is a camera, the marker position is obtained by extracting the color of the marker from an image taken with a marker of any color. Alternatively, the position of the overhead camera 300 and the marker coordinates may be input by the user via the operation unit 106 of the WS100 and stored in the volatile memory 102 by the control unit 101. The reference position information REF_POSI and the subject coordinate information POSITION[n] are represented in a coordinate system transformed into the coordinate space of the overhead camera 300's shooting area viewed from directly above. n is an index indicating the number of detected subjects. For example, if the inference unit 104 detects three people, the POSITION, ID, and STAT for three people are output as the inference result. The control unit 101 stores the subject recognition result from the image recognition unit 121 in the volatile memory 102. Details of the subject detection process and feature information calculation process will be described later.

[0075] Here, we will explain how the image recognition unit 121 calculates the coordinate information POSITION of the subject.

[0076] First, referring to Figure 5, we will explain the relationship between the coordinate system of the overhead image from the overhead camera 300 and the coordinate system of the shooting area of ​​the overhead camera 300 as viewed from directly above.

[0077] To calculate the pan value such that the shooting direction of the sub-camera 400 is the direction of the tracked subject, the calculation is simplified by calculating the angle in a planar coordinate space perpendicular to the axis on which the sub-camera 400 performs its panning motion. For example, if the sub-camera 400 is installed perpendicular to the contact surface (reference position) such as the floor or ground, the coordinate space perpendicular to the axis on which the sub-camera 400 performs its panning motion will be the coordinate space parallel to the reference position (the coordinate space where the sub-camera 400 and the subject are located, viewed from directly above), as shown in Figure 5(b). In this embodiment, it is assumed that the sub-camera 400 is installed perpendicular to the reference position, and the pan value is calculated in a coordinate system that views the shooting area of ​​the overhead camera 300 from directly above. That is, the subject position detected in the coordinate system of the overhead image of the overhead camera 300 shown in Figure 5(a) (hereinafter referred to as the overhead camera coordinate system) is transformed into the coordinate system that views the shooting area of ​​the overhead camera 300 from directly above, as shown in Figure 5(b) (hereinafter referred to as the planar coordinate system). The coordinate transformation is performed using the homography transformation matrix H, as shown in Equation 1 below. (Formula 1) In equation 1 of TIFF2026108787000002.tif19155, x and y are the horizontal and vertical coordinates of the overhead camera coordinate system, while X and Y are the horizontal and vertical coordinates of the planar coordinate system.

[0078] The control unit 101 reads the reference position information REF_POSI from the volatile memory 102 and calculates the homography transformation matrix H by substituting the marker coordinates Mark_A to Mark_D shown in Figures 5(a) and (b) contained in the reference position information REF_POSI into Equation 1. Note that the marker coordinates are values ​​in a planar coordinate system. By using Equation 1, any coordinate in the overhead camera coordinate system in Figure 5(a) can be mapped to any coordinate system in the planar coordinate system in Figure 5(b). In the example in Figure 5, the control unit 101 can determine the positions of subjects A, B, and C included in the overhead image IMG of the overhead camera 300 in the planar coordinate system of Figure 5(b). The control unit 101 stores the homography transformation matrix H calculated by Equation 1 in the volatile memory 102.

[0079] Next, we will explain the method for detecting the position of an object using an inference model for object detection, and the method for converting it to a planar coordinate system.

[0080] In this embodiment, subject detection is performed by performing image recognition processing using a pre-trained inference model for subject detection created by machine learning such as deep learning.

[0081] The inference model for subject detection takes an overhead image as input and outputs the image coordinate information of the subject contained in the overhead image.

[0082] The control unit 101 detects subjects by performing image recognition processing using an inference model for subject detection, taking the overhead image IMG from the overhead camera 300 as input via the inference unit 104. Figure 6(a) shows an example in which subjects detected by the inference unit 104 are displayed in a rectangular frame. As shown in Figure 6(a), the coordinates of the rectangular area circumscribing subjects A, B, and C detected from the overhead image are detected as the subject positions. The control unit 101 stores the coordinate information of the subjects detected from the overhead image in the volatile memory 102. In this embodiment, an example of subject detection using inference processing with a trained model has been described, but it is not limited to this. For example, a method called the SIFT method, which detects by matching local feature points in the image, or a template matching method, which detects by calculating the similarity with a template image, may be used.

[0083] Furthermore, the control unit 101 converts the lower end of the rectangular area of ​​the subject detected in the overhead camera coordinate system shown in Figure 6(a) to the planar coordinate system shown in Figure 6(b), using the subject detection position (the coordinates of the person's feet in the example in Figure 6). For example, the control unit 101 reads the homography transformation matrix H from the volatile memory 102 and substitutes the foot coordinates (xa, ya) of subject A in the overhead camera coordinate system into x and y in equation 1, thereby converting them to the foot coordinates (XA, YA) in the planar coordinate system. Similarly, it becomes possible to calculate the foot coordinates (XB, YB) of subject B and the foot coordinates (XC, YC) of subject C in the planar coordinate system. The control unit 101 writes the foot coordinates as the subject's position coordinates POSITION to the volatile memory 102.

[0084] Next, we will explain how the image recognition unit 121 generates the subject identification information ID and feature information STAT.

[0085] The control unit 101 uses the inference unit 104 to perform inference processing by inputting the coordinate information POSITION of the subject, which is the inference result of the subject detection inference model, and the overhead image from the overhead camera 300 into a trained inference model for subject identification, which has been created by machine learning such as deep learning. The control unit 101 outputs identification information ID and feature information STAT. The inference model for subject identification is different from the inference model for subject detection.

[0086] Here, we will explain the inference model used for identifying the subject.

[0087] The inference model for subject identification in this embodiment is a trained model that has been trained to achieve high similarity of feature information for images of the same subject, using training data that associates a set of images of a specific subject taken from multiple different shooting directions with information that can identify the specific subject, collected for each subject. By inputting an image of the subject extracted based on the coordinate position POSITION, which is the output of the inference model for subject detection, into the inference model for subject identification, feature information STAT is output. When an image of the same subject taken with a different camera is input, the output feature information has a higher similarity to the feature information STAT compared to when an image of a different subject is input. Examples of feature information include multidimensional vectors of the response of the convolutional layer of a convolutional neural network. Similarity will be described later.

[0088] The inference model for subject detection and the inference model for subject identification are stored in the non-volatile memory 103 before the control processing of this embodiment begins.

[0089] Furthermore, the image recognition unit 121 assigns an identification ID to the subject corresponding to the feature information, which is the inference result of the inference model for subject identification. In addition, the image recognition unit 121 inputs each image of the current frame and past frames as input to the inference model for subject detection, and calculates the similarity of the feature information of each subject image obtained by inputting the images of each subject into the inference model for subject identification. The similarity is calculated using cosine similarity. The cosine similarity is closer to 1 the more similar the multidimensional vectors, which are the feature information of each subject image, are, and closer to 0 the more different they are. The same ID is assigned to the subject with the closest similarity between the past frame and the current frame. Note that the method of calculating similarity is not limited to this, and any method that outputs a higher value the closer the feature information is and a lower value the farther the feature information is is acceptable. Note that in this embodiment, feature information was used to assign the ID, but this is not limited to this. Alternatively, the position and size of the rectangular information of the detected subjects may be compared using the rectangular information of the subject obtained by the inference model for subject detection between the current frame and past frames, and the same ID may be assigned to the closest subject. Alternatively, the position of the rectangular information in the current frame can be predicted using a Kalman filter or similar method based on the positional changes of the rectangular information for the same ID in the past few frames, and the same ID can be assigned to the subject closest to the predicted position of the rectangular information. Furthermore, these methods can be combined to assign IDs. Using such methods can improve the accuracy of ID assignment when visually similar subjects suddenly enter the shooting field of view.

[0090] As described above, the image recognition unit 121 takes the overhead image 300 as input and performs inference processing using an inference model for subject detection, outputting the coordinate position POSITION of the subject and storing it in the volatile memory 102. The image recognition unit 121 also inputs the coordinate information POSITION of the subject, which is the inference result of the subject detection inference model, and the overhead image from the overhead camera 300 into an inference model for subject identification and performs inference processing. As a result of the inference processing, the image recognition unit 121 outputs identification information ID and feature information STAT and stores them in the volatile memory 102.

[0091] Returning to the explanation of Figure 4, in step S103, the control unit 101 performs the function of the subject of interest determination unit 122 in Figure 3, and proceeds to step S104.

[0092] The subject of interest determination unit 122 determines the subject of interest, MAIN_SUBJECT, from the operation information input by the user via the operation unit 106 and the subject coordinate information, which is the subject recognition result from the image recognition unit 121 read from the volatile memory 102.

[0093] The control unit 101 displays the overhead image from the overhead camera 300 and the subject recognition results in the volatile memory 102 on the display unit 111 of the WS100. The control unit 101 allows the user to select a subject of interest from among the subjects displayed as subject recognition results via the operation unit 106. For example, if the operation unit 106 is a mouse, the user can select any of the subjects displayed on the display unit 111 by clicking on it. The control unit 101 stores the identification information ID corresponding to the subject of interest selected by the user as the subject of interest MAIN_SUBJECT in the volatile memory 102.

[0094] In step S104, the control unit 101 executes the function of the tracking target determination unit 123 shown in Figure 3, and proceeds to step S105.

[0095] The tracking target determination unit 123 determines the tracking subject SUBJECT_ID of the sub-camera 400 from the subject of interest MAIN_SUBJECT determined by the subject of interest determination unit 122.

[0096] Here, we will explain how the subject to be tracked by sub-camera 400 is determined.

[0097] The control unit 101 reads the subject of interest MAIN_SUBJECT determined by the subject of interest determination unit 122 from the volatile memory 102 and determines the subject of interest MAIN_SUBJECT as the tracking subject SUBJECT_ID for the subcamera 400. In this way, by setting the same subject as the subject of interest MAIN_SUBJECT selected by the user as the tracking subject SUBJECT_ID for the subcamera 400, the subcamera 400 can be controlled to track the subject selected by the user.

[0098] The method for determining the subject to be tracked is not limited to the method described above. For example, it may be determined using information on the subject of interest, MAIN_SUBJECT, and identification information ID, read from the volatile memory 102. For example, if the overhead image from the overhead camera 300 contains multiple subjects and multiple sub-cameras 400 are installed, one sub-camera may track the same subject as the subject of interest, while another sub-camera tracks a different subject. By determining the subject to be tracked in this way, each sub-camera can comprehensively track multiple subjects included in the overhead image from the overhead camera 300. Alternatively, one can read REF_POSI, which includes the subject's coordinate information POSITION, identification information ID, and sub-camera position, from the volatile memory 102, and determine the subject closest to the sub-camera among the subjects detected in the overhead image from the overhead camera 300 as the subject to be tracked. By determining the subject to be tracked in this way, the subject that is easiest to capture within the field of view from the sub-camera's position can be selected as the subject to be tracked. The control unit 101 stores the tracked subject SUBJECT_ID determined as described above in the volatile memory 102, and also stores the identification ID of the tracked subject before saving as a past tracked subject ID in the volatile memory 102.

[0099] In step S105, the control unit 101 executes the function of the feature information determination unit 125 and transmits feature information corresponding to the subject being tracked by the sub-camera 400 to the EB200. The control unit 101 also executes the function of the tracking state determination unit 126 to update the tracking state information STATE, saves it to the volatile memory 102, and proceeds to step S106.

[0100] The tracking status information STATE includes either "Tracking by WS100" or "Tracking by EB200". "Tracking by WS100" indicates that WS100 is tracking the subject by controlling sub-camera 400. "Tracking by EB200" indicates that EB200 is tracking the subject by controlling sub-camera 400. Details of the processing in step S105 will be described later.

[0101] In step S106, the control unit 101 reads the tracking status information STATE from the volatile memory 102 and determines whether the tracking is being performed by "WS100" or "EB200" based on the tracking status information STATE. If the control unit 101 determines that the tracking is being performed by "WS100", it proceeds to step S107, and if it determines that the tracking is being performed by "EB200", it returns to step S101.

[0102] In step S107, the control unit 101 executes the function of the control information generation unit 124 shown in Figure 3, and proceeds to step S108.

[0103] The control information generation unit 124 calculates the pan / tilt value PT_VALUE for the sub-camera 400 to track the subject SUBJECT_ID determined by the tracking target determination unit 123. The control unit 101 reads the coordinate information of the sub-camera 400 in a planar coordinate system and the coordinate information POSITION of the detected subject, which are included in the reference position information REF_POSI, from the volatile memory 102. Then, the control unit 101 calculates the pan / tilt value from the coordinate information of the subject corresponding to the subject SUBJECT_ID so that the shooting direction of the sub-camera 400 is the direction of the subject being tracked.

[0104] Now, with reference to Figure 7, we will explain how to calculate the bread value.

[0105] As shown in Figure 7, the angle θ between the line extending from the optical axis center of sub-camera 400 and the line connecting sub-camera 400 and the tracked subject SUBJECT_ID can be calculated using the following equation 2. (Formula 2) In TIFF2026108787000003.tif19155, px and py are the horizontal and vertical coordinates of the tracked subject's position, while subx and suby are the horizontal and vertical coordinates of the sub-camera 400's position. px and py are obtained by referencing the coordinate information corresponding to the tracked subject's SUBJECT_ID from the detected subject's coordinate information POSITION.

[0106] The control information generation unit 124 calculates the pan value of the sub-camera 400 based on the angle θ.

[0107] Next, we will explain how to calculate the tilt control value, referring to Figure 8.

[0108] As shown in Figure 8, the angle ρ between the line extending from the optical axis center of the sub-camera 400 (where h1 is the height of the optical axis of the sub-camera 400) and the line extending towards the height of a predetermined part of the tracked subject (in the case of a person, the height of the face) h2 can be calculated using the following equations 3 and 4. (Formula 3) TIFF2026108787000004.tif13155 (Formula 4) In the TIFF2026108787000005.tif19155 Equation 4, h1 is the height of the sub-camera 400 from the ground, and h2 is the height of the tracked subject from the ground to a predetermined part (face in the case of a person). h1 and h2 may be stored in advance in the volatile memory 102, or they may be measured in real time using a sensor not shown.

[0109] The control information generation unit 124 calculates the tilt control value of the sub-camera 400 based on the angle ρ.

[0110] The pan / tilt values ​​may also be used as speed values ​​for directing the sub-camera 400 toward the tracked subject. The method for calculating the pan / tilt values ​​is as follows: First, the control unit 101 obtains the current pan / tilt values ​​of the sub-camera 400 from the EB200. Next, the control unit 101 calculates the angular velocity of pan, which is proportional to the difference between the current values ​​and the pan value θ read from the volatile memory 102. The control unit 101 also calculates the angular velocity of tilt, which is proportional to the difference between the current values ​​and the tilt control value ρ read from the volatile memory 102. Finally, the control unit 101 stores the calculated control values ​​in the volatile memory 102.

[0111] In step S108, the control unit 101 reads the pan / tilt values ​​from the volatile memory 102, converts them into control commands according to a predetermined protocol for controlling the sub-camera 400, stores them in the volatile memory 102, and proceeds to step S109.

[0112] In step S109, the control unit 101 sends a control command corresponding to the pan / tilt values ​​calculated in step S108 to the sub-camera 400 via the communication unit 105, and returns the process to step S101.

[0113] The above describes the basic operation of the WS100.

[0114] Next, the functions and basic operation of the EB200 will be explained with reference to Figures 3 and 4(b).

[0115] In step S201, the control unit 201 sends a capture command to the sub-camera 400 via the communication unit 205, receives the captured sub-image from the sub-camera 400, saves it to the volatile memory 202, and proceeds to step S202.

[0116] In step S202, the control unit 201 executes the function of the image recognition unit 221 shown in Figure 3, and proceeds to step S203.

[0117] The image recognition unit 221 has the same functions as the image recognition unit 121 of the WS100. The control unit 201 inputs the sub-images from the sub-camera 400 read from the volatile memory 202 by the inference unit 204 into a trained model created by machine learning such as deep learning, and performs inference processing. The inference results include the coordinate information POSITION, feature information STAT_SUB[m], and identification information ID for each subject detected from the sub-images of the sub-camera 400, and are stored in the volatile memory 202. The trained model used for the inference processing of the image recognition unit 221 is the same model as the trained model used in the image recognition unit 121 of the WS100 (inference model for subject detection, inference model for subject identification).

[0118] In step S203, the control unit 201 receives the subject feature information STAT from the WS100 via the communication unit 205 and compares it with the feature information STAT_SUB calculated from the sub-image of the sub-camera 400 using the function of the tracking target determination unit 222 in Figure 3. If a subject with a high similarity between the feature information STAT and the feature information STAT_SUB exists within the field of view of the sub-camera 400, the identification information ID of that subject is determined as the identification information ID of the subject tracked by the sub-camera 400 = SUBJECT_ID, stored in the volatile memory 102, and the process proceeds to step S204. Details of the similarity calculation method will be described later.

[0119] In step S204, the control unit 201, via the communication unit 205, confirms the communication status for stopping tracking or continuing tracking with WS100, and performs processing according to the communication content, then proceeds to step S205. Details of the processing in step S204 will be described later.

[0120] In step S205, the control unit 201 determines whether or not information about the tracked subject SUBJECT_ID is stored in the volatile memory 202. If the control unit 201 determines that information about the tracked subject SUBJECT_ID is stored in the volatile memory 202, that is, that the identification information ID of the tracked subject of the sub-camera 400 is stored in the volatile memory 102, the process proceeds to step S206. If the control unit 201 determines that information about the tracked subject SUBJECT_ID is not stored in the volatile memory 202, that is, that the identification information ID of the tracked subject of the sub-camera 400 is not stored in the volatile memory 102, the process returns to step S201.

[0121] In step S206, the control unit 201 reads the subject identification information ID, which is the subject recognition result of step S202, from the volatile memory 202 and determines whether or not the tracked subject SUBJECT_ID exists in the sub-image of the sub-camera 400. If the control unit 201 determines that the tracked subject SUBJECT_ID exists in the sub-image, it proceeds to step S207; otherwise, it returns to step S201.

[0122] In step S207, the control unit 101 executes the function of the control information generation unit 223 shown in Figure 3, and proceeds to step S208.

[0123] The control information generation unit 223 has the function of calculating the pan value / tilt value of the sub-camera 400. The control unit 201 reads the subject coordinate information POSITION and the tracked subject SUBJECT_ID from the volatile memory 202 and identifies the current position of the tracked subject corresponding to the tracked subject SUBJECT_ID. The control unit 201 reads the past positions of the tracked subject within the shooting angle of view from the volatile memory 202 and calculates the pan angular velocity to increase if there is a large difference in the horizontal direction between the current position of the tracked subject and the past position of the tracked subject, and calculates the tilt angular velocity to increase if there is a large difference in the vertical direction. The control unit 201 stores the pan value / tilt value in the volatile memory 202.

[0124] In step S208, the control unit 201 reads the pan / tilt values ​​from the volatile memory 202, converts them into control commands according to a predetermined protocol for controlling the sub-camera 400, stores them in the volatile memory 202, and proceeds to step S209.

[0125] In step S209, the control unit 201 sends a control command corresponding to the pan / tilt values ​​calculated in step S208 to the sub-camera 400 via the communication unit 205, and returns the processing to step S101.

[0126] The above describes the basic operation of the EB200.

[0127] As described above, the WS100 performs image recognition processing on the overhead image from the overhead camera 300, and controls the pan / tilt operation of the sub-camera 400 if the tracking status information STATE is "Tracking by WS100". If it is "Tracking by EB200", it does not control the pan / tilt operation of the sub-camera 400. The EB200 performs image recognition processing on the sub-images of the sub-camera 400, and controls the pan / tilt operation of the sub-camera 400 if a tracking subject is set and detected from the sub-image. If no tracking subject is set, it does not control the pan / tilt operation of the sub-camera 400. Furthermore, by updating the tracking status information STATE and the setting of the tracking subject through the control process described later in Figure 9, it becomes possible to switch whether the sub-camera 400 is controlled by the WS100 or the EB200. Furthermore, by having only one device controlling the sub-camera 400 transmit the pan / tilt values, and not transmitting them when the other device is controlling it, the amount of communication can be reduced compared to the case where the pan / tilt values ​​are transmitted for each process shown in Figures 4(a) and (b).

[0128] Next, referring to Figure 4(c), the operation of the overhead camera 300 when it receives a shooting command from WS100 will be explained.

[0129] In step S301, the control unit 301 receives a shooting command from the WS100 via the communication unit 305 and proceeds to step S302.

[0130] In step S302, the control unit 301 starts the shooting process in response to receiving a shooting command from the communication unit 305, and proceeds to step S303. The control unit 301 captures an image with the imaging unit 306, and the image processing unit 307 performs predetermined image processing to generate the image data, which is then stored in the volatile memory 302.

[0131] In step S303, the control unit 301 reads the image data from the volatile memory 302 and transmits it to the WS100 via the communication unit 305.

[0132] The above describes the operation of the overhead camera 300.

[0133] Next, referring to Figure 4(d), the operation of the sub-camera 400 that receives a control command from WS100 or EB200 will be described.

[0134] In step S401, the control unit 401 receives a control command via the communication unit 405, stores the control command in the volatile memory 402, and proceeds to step S402.

[0135] In step S402, the control unit 401 reads the pan value / tilt value from the volatile memory 402 in response to receiving a control command from the communication unit 405, and proceeds to step S403.

[0136] In step S403, the control unit 401 calculates drive parameters for controlling the pan / tilt operation in the desired direction and at the desired speed based on the pan / tilt values ​​read from the non-volatile memory 403, and proceeds to step S404. The drive parameters are parameters for controlling each of the pan / tilt actuators included in the PTZ drive unit 409, and the pan / tilt values ​​included in the control command are converted into drive parameters by referring to a conversion table stored in the non-volatile memory 403.

[0137] In step S404, the control unit 401 controls the optical unit 408 via the PTZ drive unit 409 based on the drive parameters obtained in step S403, thereby changing the shooting direction of the sub-camera 400. The PTZ drive unit 409 changes the shooting direction of the sub-camera 400 by driving the optical unit 408 in the pan / tilt direction based on the drive parameters.

[0138] The above describes the operation of sub-camera 400.

[0139] Next, the control process of WS100 will be explained with reference to Figure 9(a).

[0140] Figure 9(a) shows the control process of WS100, and provides a detailed view of step S105 in Figure 4(a).

[0141] Part of the process shown in Figure 9(a) is realized by the control unit 101 executing the function of the tracking state determination unit 126 shown in Figure 3.

[0142] The tracking state determination unit 126 has the function of updating the tracking state information STATE stored in the volatile memory 102.

[0143] In step S110, the control unit 101 reads the SUBJECT_ID of the tracked subject of the subcamera 400, calculated in step S104 of Figure 4(a), and the identification information ID indicating past tracked subjects from the volatile memory 102. The control unit 101 then compares this with the identification information read from the volatile memory 102 to determine whether the tracked subject of the subcamera 400 has changed. If the control unit 101 determines that the tracked subject of the subcamera 400 has changed, it proceeds to step S111; otherwise, it proceeds to step S113.

[0144] In step S111, the control unit 101 sends a tracking stop command to the EB200 via the communication unit 105, and proceeds to step S112.

[0145] In step S112, the control unit 101 executes the function of the tracking state determination unit 126 in Figure 3 and changes the tracking state information STATE to "Tracking by WS100".

[0146] If the subject being tracked by sub-camera 400 changes, it is highly likely that the subject is no longer within the field of view of sub-camera 400. In this case, by performing steps S111 and S112, WS100 controls sub-camera 400 based on the overhead image from overhead camera 300, instead of sub-camera 400.

[0147] In step S113, the control unit 101 reads the tracking status information STATE from the volatile memory 102 and determines whether the tracking is being performed by "WS100" or "EB200" based on the tracking status information STATE. If the control unit 101 determines that the tracking is being performed by "WS100", it proceeds to step S117, and if it determines that the tracking is being performed by "EB200", it proceeds to step S114.

[0148] In step S114, the control unit 101 sends a tracking continuation confirmation request to the EB200 via the communication unit 105 to inquire whether the EB200 can continue tracking the subject. The response from the EB200 is either "Tracking continuation OK" or "Tracking continuation NG". If the control unit 101 receives notification of "Tracking continuation OK" from the EB200, it returns to step S101; if it receives notification of "Tracking continuation NG" from the EB200, it proceeds to step S115.

[0149] In step S115, the control unit 101 sends a tracking stop command to the EB200 via the communication unit 105, and proceeds to step S116.

[0150] In step S116, the control unit 101 executes the function of the tracking state determination unit 126 shown in Figure 3 to update the tracking state information STATE to "Tracking by WS100" and terminates the process.

[0151] By performing the processes from steps S114 to S116, even if the EB200 becomes unable to track when the tracking state is "Tracking by EB200", the WS100 can continue tracking.

[0152] In step S117, the control unit 101 determines whether or not a tracked subject exists within the shooting angle of view of the sub-camera 400. If the control unit 101 determines that a tracked subject exists within the shooting angle of view of the sub-camera 400, it proceeds to step S118. If the control unit 101 determines that a tracked subject does not exist within the shooting angle of view of the sub-camera 400, it terminates the process. Whether or not a tracked subject exists within the shooting angle of view of the sub-camera 400 can be determined by comparing the current pan / tilt value obtained from the sub-camera 400 by the control unit 101 with the new pan / tilt value calculated in step S107 of Figure 4(a). If the current pan / tilt value is sufficiently close to the new pan / tilt value, it can be determined that a tracked subject exists within the shooting angle of view of the sub-camera 400. Alternatively, if the pan / tilt speed value calculated in step S108 is sufficiently small, it can be determined that the tracking subject is within the field of view of the sub-camera 400 because the current pan / tilt value is approaching the new pan / tilt value.

[0153] In step S118, the control unit 101 executes the function of the feature information determination unit 125 shown in Figure 3, and proceeds to step S119.

[0154] The feature information determination unit 125 has the function of determining the feature information of the subject being tracked by the sub-camera 400, that is, the feature information of the subject to be transmitted to the EB200. The feature information determination unit 125 reads the feature information STAT[n] of the subject detected by the image recognition unit 121 from the overhead image of the overhead camera 300 from the volatile memory 102. The feature information determination unit 125 also reads the identification information SUBJECT_ID of the tracked subject determined by the tracked target determination unit 123 from the volatile memory 102. Then, the feature information determination unit 125 determines the feature information STAT[i] corresponding to the tracked subject from the feature information STAT[n] and stores it in the volatile memory 102. i is an index indicating the tracked subject.

[0155] In step S119, the control unit 101 sends a tracking start command and characteristic information STAT[i] of the tracked subject to the EB200 via the communication unit 105, and proceeds to step S120.

[0156] The processing from steps S117 to S119 allows the EB200 to receive a tracking start command and characteristic information of the tracked subject only if there is a high probability that the tracked subject is within the field of view of the sub-camera 400. This reduces the amount of communication compared to sending information for each process shown in Figures 4(a) and 9(a).

[0157] In step S120, the control unit 101 receives the subject matching result from the EB200 via the communication unit 105. If the control unit 101 receives matching information from the EB200 indicating that the subjects match, it proceeds to step S121; if it receives mismatch information indicating that the subjects do not match, it terminates the process.

[0158] In step S121, the control unit 101 executes the function of the tracking state determination unit 126 shown in Figure 3 to change the tracking state information STATE to "Tracking by EB200" and terminates the process.

[0159] Next, the control process of the EB200 will be explained with reference to Figures 9(b), 9(c), and 10.

[0160] Figure 9(b) shows the control process of EB200, and provides a detailed view of step S203 in Figure 4(b).

[0161] In step S210, the control unit 201 determines whether it has received a tracking start command and characteristic information STAT[i] of the tracked subject obtained from the overhead camera 300 via the communication unit 205. If the control unit 201 has received the tracking start command and characteristic information STAT[i] of the tracked subject from the WS100, it proceeds to step S211; otherwise, it terminates the process.

[0162] In steps S211 to S214, the control unit 201 executes the function of the tracking target determination unit 222 shown in Figure 3, and determines whether the feature information STAT[i] received from WS100 and the feature information STAT_SUB[m] obtained from the sub-image of the sub-camera 400 satisfy predetermined conditions.

[0163] The tracking target determination unit 222 has the function of calculating similarity between the feature information STAT[i] received from WS100 and the feature information STAT_SUB[m] obtained from the sub-image of sub-camera 400. The tracking target determination unit 222 also has the function of comparing the similarity of the feature information with a threshold stored in volatile memory 102 and storing the comparison result in volatile memory 102. For example, if there are two people in the sub-image of sub-camera 400, the tracking target determination unit 222 calculates the similarity between the feature information of the two people (STAT_SUB[1], STAT_SUB[2]) and the feature information STAT[i] received from WS100. The similarity is calculated as the cosine similarity between the feature information vectors, and a similarity value of 1 to 0 is obtained. The control unit 201 stores the similarity calculated for m subjects in volatile memory 202.

[0164] In step S211, the control unit 201 performs the function of the tracking target determination unit 222 shown in Figure 3 to perform feature information matching processing, and then proceeds to step S212.

[0165] In step S212, the control unit 201 determines whether or not there are subjects with high similarity of feature information, based on the matching result in step S211. The existence of subjects with high similarity of feature information means that the same subject is being captured by the overhead camera 300 and the sub-camera 400. If the control unit 201 determines that there are subjects with high similarity of feature information, it proceeds to step S214; if it determines that there are no subjects with high similarity of feature information, it proceeds to step S213.

[0166] The control unit 201 reads a predetermined threshold from the volatile memory 202 and determines that there is a subject with high similarity in feature information if, under predetermined conditions, the similarity is greater than or equal to the threshold, or if there is a subject with higher similarity, or if the subjects match, and stores the identification information ID of that subject in the volatile memory 202. The control unit 201 also updates the information MATCH, which indicates whether or not there is a subject with high similarity in feature information, and stores it in the volatile memory 202. In this embodiment, if the value of MATCH is 0, there is no subject with high similarity in feature information, i.e., the subjects do not match between the overhead camera 300 and the sub-camera 400. If the value of MATCH is 1, there is a subject with high similarity in feature information, i.e., the subjects match between the overhead camera 300 and the sub-camera 400. If there is a subject with high similarity in feature information, the control unit 201 stores MATCH=1 in the volatile memory 202 and proceeds to step S214. If the control unit 201 finds no subject with a high similarity in feature information, it stores MATCH=0 in the volatile memory 202 and proceeds to step S213.

[0167] Here, referring to Figure 10, we will explain the similarity of the subject feature information detected from the overhead image of the overhead camera 300 and the sub-image of the sub-camera 400.

[0168] Figure 10(a) shows the relative positions of the shooting position and direction of the overhead camera 300 and the shooting position and direction of the sub-camera 400. Figure 10(b) shows the subject detected and the tracked subject from the overhead image of the overhead camera 300.

[0169] Subjects A, B, and C are detected from the overhead image of the overhead camera 300, and it is assumed that the subject being tracked by the sub-camera 400 is subject C. The feature information of the subject being tracked by the sub-camera 400, which is transmitted from the sub-camera 400 to the WS100, corresponds to the information of subject C. Figures 10(c) and (e) show the sub-images of the sub-camera 400, and Figures 10(d) and (f) show the similarity between the feature information of the subject being tracked by the sub-camera 400 and the feature information of the subject detected from the sub-images.

[0170] As shown in Figure 10(c), when the sub-camera 400 is capturing both subject A and subject B, the similarity between the feature information of subject C in the overhead image from the overhead camera 300 and the feature information of either subject A or subject B in the sub-image from the sub-camera 400 is calculated. As shown in Figure 10(d), the similarity between the feature information of subject C in the overhead image from the overhead camera 300 and the feature information of either subject A or subject B in the sub-image from the sub-camera 400 will be low. In this case, for example, if the similarity threshold for subjects is 0.7, the result will be that both subject A and subject B do not match.

[0171] Furthermore, as shown in Figure 10(e), if the sub-camera 400 is capturing both subject B and subject C, the similarity between the feature information of subject C in the overhead image from the overhead camera 300 and the feature information of subject B or subject C in the sub-image from the sub-camera 400 is calculated. Subject C in the overhead image from the overhead camera 300 and subject C in the sub-image from the sub-camera 400 will have different forms in the images because the camera's shooting position and direction are different. For example, if subject C is facing its face or body towards the overhead camera 300, subject C will be facing forward in the overhead image from the overhead camera 300 and closer to being sideways in the sub-image from the sub-camera 400. The inference models for subject identification in the image recognition unit 121 of WS100 and the image recognition unit 221 of EB200 are models that have learned images of the same subject taken from multiple different directions. Therefore, even if the same subject is captured by multiple cameras with different shooting positions and directions, the similarity of feature information will be high, even if the morphology in each captured image differs. In other words, as shown in Figure 10(f), the similarity of feature information of subject C in the overhead image from overhead camera 300 and the similarity of feature information of subject C in the sub-image from sub-camera 400 will be high. As a result, for example, if the threshold for subject similarity is 0.7, subject B will be found to be inconsistent, while subject C will be found to be identical, and subject C can be determined to be the same subject.

[0172] Returning to the explanation of Figure 9(b), in step S213, the control unit 201 reads MATCH=0 from the volatile memory 202, transmits it to the WS100 via the communication unit 205, and terminates the process.

[0173] In step S214, the control unit 201 reads the identification information ID of the subject with the highest similarity from the volatile memory 102, saves it in the volatile memory 102 as the tracked subject SUBJECT_ID, and proceeds to step S215. By selecting the subject with the highest similarity, even if there are subjects with similar clothing, for example, the most likely subject among them can be selected as the tracking target.

[0174] In step S215, the control unit 201 reads MATCH=1 from the volatile memory 202, transmits it to the WS100 via the communication unit 205, and terminates the process.

[0175] Figure 9(c) shows the control process of EB200, and provides a detailed view of step S204 in Figure 4(b).

[0176] In step S220, the control unit 201 determines whether or not it has received a tracking stop command from WS100 via the communication unit 205. If the control unit 201 has received a tracking stop command from WS100, it proceeds to step S221; otherwise, it proceeds to step S223.

[0177] In step S221, the control unit 201 sends a control command to the sub-camera 400 via the communication unit 205 to stop the pan / tilt operation, and proceeds to step S222.

[0178] In step S222, the control unit 201 deletes the tracked subject SUBJECT_ID stored in the volatile memory 202 and proceeds to step S201.

[0179] In step S223, the control unit 201 determines whether or not it has received a tracking continuation confirmation request from WS100 via the communication unit 205. If the control unit 201 has received a tracking continuation confirmation request from WS100, it proceeds to step S224; otherwise, it terminates the process.

[0180] In step S224, the control unit 201 reads the subject recognition result from the image recognition unit 221 from the volatile memory 202 and determines whether or not the tracked subject SUBJECT_ID has been detected. If the control unit 201 determines that the tracked subject SUBJECT_ID has been detected by the image recognition unit 221, it proceeds to step S226; otherwise, it proceeds to step S225.

[0181] In step S225, the control unit 201 sends "Tracking continuation NG" to the WS100 via the communication unit 205, and the process returns to step S201.

[0182] In step S226, the control unit 201 sends "Continue tracking OK" to the WS100 via the communication unit 205, and terminates the process.

[0183] The above describes the detailed control process of the EB200.

[0184] According to Embodiment 1 described above, the same subject can be recognized by multiple cameras 300 and 400 with different shooting positions and directions. Therefore, a specific subject can be tracked by appropriately switching between the control of the sub-camera 400 by WS100 and the control of the sub-camera 400 by EB200.

[0185] If there is no tracked subject in the sub-image of sub-camera 400, WS100 controls sub-camera 400. If there is a tracked subject in the field of view of sub-camera 400, control of sub-camera 400 can be transferred from WS100 to EB200. Furthermore, if the tracked subject moves at high speed and is lost, or if the tracked subject needs to be changed, tracking can be continued by controlling sub-camera 400 with WS100.

[0186] In Embodiment 1, an example was described in which WS100 and EB200 switch whether or not to transmit pan / tilt values ​​to the sub-camera 400, but the invention is not limited to this example. For example, pan / tilt values ​​may be transmitted from WS100 and EB200 to the sub-camera 400 regardless of the tracking state, and the sub-camera 400 may control which device's pan / tilt values ​​it receives to perform pan / tilt operations. In this case, the processing of WS100 can be simplified by omitting step S106 in Figure 4(a) and adding a process where the control unit 101 transmits tracking state information STATE to the sub-camera 400 before step S107 in Figure 4(a). The processing of EB200 can be simplified by omitting steps S205 and S206 in Figure 4(b) and step S221 in Figure 9(c).

[0187] Subcamera 400 is controlled to perform pan / tilt operations in accordance with control commands received from EB200 if the tracking status information STATE received from WS100 is "Tracking by EB200". Subcamera 400 is controlled to perform pan / tilt operations in accordance with control commands received from WS100 if the tracking status information STATE received from WS100 is "Tracking by WS100".

[0188] The edge box (EB) 200 may be integrated with the sub-camera 400, or the functions of the EB200 may be built into the sub-camera 400.

[0189] [Embodiment 2] Embodiment 1 describes an example in which the sub-camera 400 is controlled by either WS100 or EB200. Embodiment 2 describes an example in which EB200 is omitted and WS100 controls the sub-camera 400 based on the overhead image from the overhead camera 300 and the sub-image from the sub-camera 400.

[0190] In Embodiment 2, the sub-camera 400 is controlled using either the pan / tilt values ​​calculated based on the overhead image from the overhead camera 300 or the pan / tilt values ​​calculated based on the sub-image from the sub-camera 400.

[0191] The system configuration of Embodiment 2 is the same as that of the system configuration in Figure 1, but without the EB200. The difference from Embodiment 1 is that the sub-image from the sub-camera 400 is input to the WS100. The operation of all other components is the same as in Embodiment 1.

[0192] In basic operation, the overhead camera 300 transmits overhead images to the WS100, and the sub-camera 400 transmits sub-images to the WS100. The sub-camera 400 also has a PTZ (Pan-Tap-Z) function.

[0193] The WS100 detects the subject from the overhead image of the overhead camera 300 and the sub-image of the sub-camera 400, and changes the imaging direction of the sub-camera 400 to the direction of the tracked subject based on the subject recognition result. Until the shooting direction of the sub-camera 400 is in the direction of the tracked subject, the WS100 controls the sub-camera 400 based on the subject recognition result of the overhead image of the overhead camera 300. After the shooting direction of the sub-camera 400 is in the direction of the tracked subject, the WS100 calculates feature information of the tracked subject from the overhead image of the overhead camera 300 and calculates feature information of the subject from the sub-image of the sub-camera 400. Then, based on this feature information, the WS100 controls the sub-camera 400. Feature information is information that allows identification of the same subject when it is captured by multiple cameras with different shooting positions and / or shooting directions.

[0194] According to Embodiment 2, the sub-camera 400 can be controlled based on the subject recognition result of either the overhead image from the overhead camera 300 or the sub-image from the sub-camera 400, thereby tracking the subject to be tracked.

[0195] The hardware configuration of WS100, overhead camera 300, and sub-camera 400 is the same as in Figure 2 of Embodiment 1.

[0196] First, with reference to Figure 11, the functional configuration of WS100 for realizing the control processing of this embodiment will be described.

[0197] The functions of the WS100 are implemented by hardware and / or software. If the functions shown in Figure 11 are implemented by hardware instead of software, it is sufficient to have the corresponding circuit configurations shown in Figure 11.

[0198] The WS100 includes an image recognition unit 121, a focus subject determination unit 122, a tracking target determination unit 123, a control information generation unit 124, a feature information determination unit 125, a tracking state determination unit 126, an image recognition unit 127, and a tracking target determination unit 128. The software that implements these functions is stored in the non-volatile memory 103, and the control unit 101 loads it into the volatile memory 102 and executes it.

[0199] The functions of the image recognition unit 121, the subject of interest determination unit 122, the tracking target determination unit 123, and the feature information determination unit 125 are the same as those shown in Figure 3 of Embodiment 1.

[0200] Refer to Figures 11 and 12 to explain the functions and basic operation of the WS100.

[0201] The processing from step S501 to step S504 is the same as the processing from step S101 to S104 in Figure 4(a) of Embodiment 1.

[0202] In step S505, the control unit 101 sends a shooting command to the sub-camera 400 via the communication unit 105, receives the captured sub-image from the sub-camera 400, saves it to the volatile memory 102, and proceeds to step S506.

[0203] In step S506, the control unit 101 executes the function of the image recognition unit 127 in Figure 11, and proceeds to step S507.

[0204] The function of the image recognition unit 127 can be described in the explanation of the image recognition unit 221 of the EB200 in Embodiment 1 by replacing the control unit 201 with the control unit 101, the volatile memory 202 with the volatile memory 102, and the non-volatile memory 203 with the non-volatile memory 103.

[0205] In step S507, the control unit 101 executes the functions of the tracking target determination unit 128 and the tracking state determination unit 126 shown in Figure 11, compares the feature information calculated in steps S502 and S506, and updates the tracking state information STATE. The control unit 101 also saves the tracking subject SELECT_ID and the tracking state information STATE to the volatile memory 102 and proceeds to step S508.

[0206] The tracking status information STATE includes either "tracking using the overhead image" or "tracking using the sub-image." "Tracking using the overhead image" indicates that the subject is being tracked by controlling the sub-camera 400 based on the subject recognition result of the overhead camera 300's overhead image. "Tracking using the sub-image" indicates that the subject is being tracked by controlling the sub-camera 400 based on the subject recognition result of the sub-image of the sub-camera 400. Details of the processing in step S507 will be described later.

[0207] The processing from steps S508 to S510 is performed by the control information generation unit 124 shown in Figure 11.

[0208] In step S508, the control unit 101 reads the tracking status information STATE from the volatile memory 102 and determines whether the system is "tracking using the overhead image" or "tracking using the sub-image" based on the tracking status information STATE. If the control unit 101 determines that the system is "tracking using the overhead image," it proceeds to step S510; if it determines that the system is "tracking using the sub-image," it proceeds to step S509.

[0209] In step S509, the control unit 101 calculates the pan / tilt values ​​of the sub-camera 400 based on the subject recognition result of the sub-image of the sub-camera 400, and proceeds to step S511. The process in step S509 can be performed by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of the control information generation unit 223 in Figure 3.

[0210] In step S510, the control unit 101 calculates the pan / tilt values ​​of the sub-camera 400 based on the subject recognition result of the overhead image from the overhead camera 300, and proceeds to step S511. The process in step S510 can be performed by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of the control information generation unit 223 in Figure 3.

[0211] In step S511, the control unit 101 executes the function of the control information generation unit 124 shown in Figure 3, and proceeds to step S512.

[0212] The processes in steps S511 and S512 are the same as those in steps S108 and S109 in Figure 4(a).

[0213] The above describes the basic operation of the WS100.

[0214] Next, the control process of WS100 will be explained with reference to Figure 13.

[0215] Figure 13 shows the control process of WS100, and provides a detailed view of step S507 in Figure 12.

[0216] The process in step S520 is the same as the process in step S110 in Figure 9(a).

[0217] In step S521, the control unit 101 performs the function of the tracking state determination unit 126 shown in Figure 11 to change the tracking state information STATE to "Tracking using overhead camera image".

[0218] The tracking state determination unit 126 has the function of updating the tracking state information STATE stored in the volatile memory 102.

[0219] In step S522, the control unit 101 reads the tracking status information STATE from the volatile memory 102 and determines whether the system is "tracking using the overhead image" or "tracking using the sub-image" based on the tracking status information STATE. If the control unit 101 determines that the system is "tracking using the overhead image," it proceeds to step S525; if it determines that the system is "tracking using the sub-image," it proceeds to step S523.

[0220] The process in step S523 can be performed by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of step S224 in Figure 9(b).

[0221] In step S524, the control unit 101 performs the function of the tracking state determination unit 126 shown in Figure 11 to change the tracking state information STATE to "Tracking using overhead image".

[0222] The processes in steps S525 and S526 are the same as those in steps S117 and S118 in Figure 9(a).

[0223] The process from steps S527 to S529 can be performed by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process from steps S211 to S214 in Figure 9(b).

[0224] In step S530, the control unit 101 executes the function of the tracking state determination unit 126 shown in Figure 11 to change the tracking state information STATE to "Tracking using sub-images" and terminates the process.

[0225] According to Embodiment 2 described above, the WS100 switches whether to control the sub-camera 400 based on the subject recognition result of the overhead camera 300 or the sub-image of the sub-camera 400. This eliminates the need for the EB200 in Embodiment 1, simplifying the system configuration and achieving the same effects as Embodiment 1.

[0226] [Embodiment 3] Embodiments 1 and 2 describe an example of a system equipped with an overhead camera 300 and a sub-camera 400.

[0227] Embodiment 3 describes an example of a system that includes a main camera 500 in addition to an overhead camera 300 and a sub-camera 400.

[0228] Figure 14 is a system configuration diagram of Embodiment 3.

[0229] Embodiment 3 differs from Embodiment 1 in that it includes a main camera 500, and the subject tracked by the sub-camera 400 is determined based on the main image captured by the main camera 500. The following will focus on the differences from Embodiment 1.

[0230] In Embodiment 3, the main camera 500 has a PTZ function. The focus subject determination unit 122 of the WS100 determines (estimates) the focus subject of the main camera 500 from the shooting range of the main camera 500, and determines the tracking subject of the sub-camera 400 based on the focus subject of the main camera 500. The tracking subject of the sub-camera 400 may be the same subject as the focus subject of the main camera 500, or it may be a different subject.

[0231] Next, we will explain an example of determining the subject to be tracked by sub-camera 400 based on the role (ROLE) set for sub-camera 400.

[0232] The role of sub-camera 400 is to indicate the subject of focus for the main camera 500, the subject tracked by sub-camera 400 in relation to the zoom operation, and the content of the zoom operation control. The role of sub-camera 400 can be set by the user via the control panel provided on WS100 or EB200. In addition, if multiple sub-cameras are installed, any of the multiple sub-cameras can be set as the main camera, and the main camera settings may be set by the user via the control panel provided on WS100 or EB200. The role of sub-camera 400 and the method of setting the main camera are not limited to the above methods and any method is acceptable.

[0233] Figure 15 illustrates the roles and functions that can be set for the sub-camera 400.

[0234] When the role is set to "Main Follow," the role of sub-camera 400 is to track the same subject that the main camera 500 is focusing on, and to control the zoom in phase with the zoom operation of the main camera 500. Based on this role, the zoom control value of sub-camera 400 is calculated. Here, "in phase" in zoom operation means that the zoom operations of the main camera 500 and sub-camera 400 are controlled in the same direction. For example, if the zoom control value of the main camera 500 is changed from wide-angle to telephoto, the zoom of sub-camera 400 will also be changed from wide-angle to telephoto.

[0235] When the role is set to "Main Counter," the role of sub-camera 400 is to track the same subject that the main camera 500 is focusing on, and to control the zoom in the opposite phase to the zoom operation of the main camera 500. Based on this role, the PTZ value of sub-camera 400 is calculated. Here, "opposite phase" in zoom operation means that the zoom operations of the main camera 500 and sub-camera 400 are controlled in opposite directions. For example, if the zoom control value of the main camera 500 is changed from wide-angle to telephoto, the zoom of sub-camera 400 will be changed from telephoto to wide-angle.

[0236] When the role is set to "Assist Follow," the sub-camera 400 tracks a subject other than the one the main camera 500 is focusing on, and controls the zoom in phase with the main camera 500's zoom operation. Based on this role (CAMERA_ROLE), the zoom control value for the sub-camera 400 is calculated.

[0237] When the role is set to "Assist Counter," the sub-camera 400 tracks a subject different from the subject that the main camera 500 is focusing on, and controls the zoom in the opposite phase to the zoom operation of the main camera 500. Based on this role (CAMERA_ROLE), the zoom control value for the sub-camera 400 is calculated. In the example in Figure 15, "Different from the main (left side)" is shown as an example of the control content of the tracked subject for "Assist Follow" and "Assist Counter," but there may also be "Assist Follow" and "Assist Counter" where the tracked subject is "Different from the main (right side)."

[0238] Furthermore, when the tracked subject is considered "separate from the main subject," it may serve the role of a subject located in positions other than left or right (such as up, down, or front and back).

[0239] If there are multiple sub-cameras, you can assign a role to each sub-camera.

[0240] Furthermore, while Embodiment 3 describes an example where the tracking subject and zoom control content are set as roles, the control content for the tracking subject alone may also be set as a role, or other items may be added.

[0241] Furthermore, in Embodiment 3, the subject to be tracked by the sub-camera 400 is set based on the main image of the main camera 500, and an example of combining Embodiment 3 with Embodiment 1 has been described. However, Embodiment 3 may also be combined with Embodiment 2.

[0242] Furthermore, in a configuration comprising an overhead camera 300 and a sub-camera 400, as in Embodiments 1 and 2, the sub-camera 400 may be controlled to track the subject based on both the overhead image and the sub-image captured by each camera.

[0243] Furthermore, in a configuration like Embodiment 3, which includes a main camera 500 in addition to the overhead camera 300 and sub-camera 400, the sub-camera 400 may be controlled to track the subject based on any two or all of the overhead images, main images, and sub-images captured by each camera.

[0244] [Other embodiments] The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0245] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention.

[0246] The disclosures herein include the following systems, control devices, control methods, and programs. [Configuration 1] A system comprising a first imaging device and a second imaging device having different shooting directions, and a first control device and a second control device that control the second imaging device to track a predetermined subject based on a first image captured by the first imaging device or a second image captured by the second imaging device, The first control device described above is A first generation means for generating first characteristic information of a predetermined subject included in the first image, The system includes a first control means for controlling the second imaging device to track the predetermined subject based on the first characteristic information, The second control device is, A second generation means for generating second characteristic information of a subject included in the second image, A comparison means for comparing the first characteristic information generated by the first control device with the second characteristic information generated by the second generation means, The system includes a second control means for controlling the second imaging device to track the predetermined subject based on the second characteristic information, The first and second characteristic information are information that allows for the identification of the same subject when the same subject is photographed by multiple imaging devices with different shooting directions. A system characterized by switching between a first state in which the first control device controls the second imaging device to track the predetermined subject based on the first characteristic information, and a second state in which the second control device controls the second imaging device to track the predetermined subject based on the second characteristic information, based on the results of the comparison by the comparison means. [Configuration 2] The first control device transmits the first characteristic information to the second control device. Based on the results of the comparison by the comparison means, the first control means switches to the second state if the first feature information and the second feature information satisfy predetermined conditions. The system according to configuration 1, characterized in that if the first characteristic information and the second characteristic information do not satisfy the predetermined conditions, the system switches to the first state. [Configuration 3] The predetermined condition is that the similarity between the first feature information and the second feature information is greater than or equal to a threshold. The system according to configuration 2, characterized in that the comparison means calculates the similarity between the first feature information and the second feature information and outputs the result of comparing the similarity with the threshold. [Structure 4] The second control means controls the second imaging device to track the predetermined subject when the predetermined subject is within the imaging range of the second imaging device. If the predetermined subject is no longer within the shooting range of the second imaging device, the first control device is notified that tracking of the predetermined subject cannot be continued. The system according to any one of configurations 1 to 3, characterized in that the first control means switches from the second state to the first state upon receiving the notification. [Composition 5] The system according to any one of configurations 1 to 4, characterized in that when the predetermined subject is changed, the first control means switches from the second state to the first state. [Composition 6] The system according to configuration 5, wherein when the predetermined subject is changed, the first control means switches from the first state to the second state if the first characteristic information and the second characteristic information satisfy predetermined conditions. [Composition 7] The first control device described above is A first tracking target determination means for determining a predetermined subject from a subject detected from the first image, A feature information determination means for determining the first feature information of the predetermined subject and transmitting it to the second control device, The system includes a first control information generation means that generates first control information for controlling the shooting direction of the second control device so as to track the predetermined subject, The second control device is, A second tracking target determination means that determines the predetermined subject from the subject detected in the second image based on the second characteristic information of the subject detected in the second image and the first characteristic information of the predetermined subject received from the first control device, The system according to any one of configurations 1 to 6, further comprising: a second control information generation means for generating second control information that controls the shooting direction of the second control device so as to track the predetermined subject. [Structure 8] The system according to configuration 7, characterized in that the second imaging device controls the shooting direction of the second control device to track a predetermined subject based on control information acquired from the first control device or the second control device. [Composition 9] The system according to configuration 7, characterized in that the second imaging device controls the shooting direction of the second control device to track a predetermined subject based on either the first control device or the control information acquired from the second control device. [Configuration 10] The system according to any one of configurations 7 to 9, characterized in that the control information includes at least one of a pan value and a tilt value. [Composition 11] The first generation means generates the first feature information by performing inference processing using a trained model with the first image as input. The system according to any one of configurations 1 to 10, characterized in that the second generation means generates the second feature information by performing inference processing using a trained model with the second image as input. [Composition 12] The trained model includes a first model for object detection and a second model for object identification. The first generation means generates first information indicating the position of an object included in the first image by performing inference processing using the first model with the first image as input. By performing inference processing using the second model with the first image and the first information as input, feature information of the subject contained in the first image is generated. The second generation means generates second information indicating the position of an object included in the second image by performing inference processing using the first model with the second image as input. The system according to configuration 11, characterized in that it generates feature information of the subject contained in the second image by performing inference processing using the second model with the second image and the second information as input. [Composition 13] The system according to configuration 12, characterized in that the second model for identifying the subject is a trained model that has been trained using images taken from multiple different shooting directions of multiple subjects as training data, so that the similarity of feature information is high for images of the same subject. [Composition 14] A control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, A generation means for generating first characteristic information of a predetermined subject included in the first image, The system includes control means for controlling the second imaging device to track the predetermined subject, The control means switches between a first state in which the control device controls the second imaging device to track a predetermined subject based on the first characteristic information, and a second state in which the external device controls the second imaging device to track a predetermined subject based on the second characteristic information, based on the result of comparing the second characteristic information of a subject included in the second image with the first characteristic information in the external device. The control device is characterized in that the first characteristic information and the second characteristic information are information that allows for the identification of the same subject when the same subject is photographed by multiple imaging devices with different shooting directions. [Composition 15] A control device that controls the second imaging device to track a predetermined subject based on a second image captured by the second imaging device, which has a different shooting direction from the first imaging device, A generation means for generating second characteristic information of a subject included in the second image, A comparison means for comparing the first characteristic information of a predetermined subject included in the first image captured by the first imaging device acquired from an external device with the second characteristic information, The system includes control means for controlling the second imaging device to track the predetermined subject based on the second characteristic information, The first and second characteristic information are information that allows for the identification of the same subject when the same subject is photographed by multiple imaging devices with different shooting directions. The control means is characterized in that, based on the results of the comparison by the comparison means, the control means controls the second imaging device to track the predetermined subject based on the second characteristic information when the first characteristic information and the second characteristic information satisfy predetermined conditions. [Composition 16] A control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, A first generation means for generating first characteristic information of a predetermined subject included in the first image, A second generation means for generating second characteristic information of a subject included in the second image, The system includes control means for controlling the second imaging device to track the predetermined subject, The first and second characteristic information are information that allows for the identification of the same subject when the same subject is photographed by multiple imaging devices with different shooting directions. The control means is characterized by switching between a first state in which the second imaging device is controlled to track a predetermined subject based on the first characteristic information, and a second state in which the second imaging device is controlled to track a predetermined subject based on the second characteristic information, based on the result of comparing the first characteristic information and the second characteristic information. [Composition 17] A control method for a control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, A generation step of generating first feature information of a predetermined subject included in the first image, The control step includes controlling the second imaging device to track the predetermined subject, In the control step, the external device switches between a first state in which the control device controls the second imaging device to track the predetermined subject based on the first characteristic information, based on the result of comparing the second characteristic information of the subject included in the second image with the first characteristic information, and a second state in which the external device controls the second imaging device to track the predetermined subject based on the second characteristic information. The method is characterized in that the first characteristic information and the second characteristic information are information that allows for the identification of the same subject when the same subject is photographed by multiple imaging devices with different shooting directions. [Composition 18] A control method for a control device that controls a second imaging device to track a predetermined subject based on a second image captured by a second imaging device having a different shooting direction from the first imaging device, A generation step of generating second feature information of the subject included in the second image, A comparison step of comparing first characteristic information of a predetermined subject included in a first image captured by the first imaging device acquired from an external device with second characteristic information, The system includes a control step of controlling the second imaging device to track the predetermined subject based on the second characteristic information, The first and second characteristic information are information that allows for the identification of the same subject when the same subject is photographed by multiple imaging devices with different shooting directions. The control step is characterized in that, based on the results of the comparison, the second imaging device is controlled to track the predetermined subject based on the second characteristic information when the first characteristic information and the second characteristic information satisfy predetermined conditions. [Composition 19] A control method for a control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, A first generation step of generating first characteristic information of a predetermined subject included in the first image, A second generation step of generating second feature information of the subject included in the second image, The control step includes controlling the second imaging device to track the predetermined subject, The first characteristic information and the second characteristic information are information that can identify that the same subject is being photographed by a plurality of imaging devices with different shooting directions. In the control step, based on the result of comparing the first characteristic information and the second characteristic information, a first state of controlling the second imaging device to track the predetermined subject based on the first characteristic information and a second state of controlling the second imaging device to track the predetermined subject based on the second characteristic information are switched. A method characterized by that. [Configuration 20] A program for causing a computer to function as the control device according to any one of Configurations 14 to 16.

Explanation of Signs

[0247] 100... First control device (workstation / WS), 200... Second control device (edge box / EB), 300... First imaging device (overhead camera), 400... Second imaging device (sub camera), 101, 201, 301, 401... Control unit

Claims

1. A system comprising a first imaging device and a second imaging device having different shooting directions, and a first control device and a second control device that control the second imaging device to track a predetermined subject based on a first image captured by the first imaging device or a second image captured by the second imaging device, The first control device is The system includes a first control means for controlling the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the first image, The second control device is The system includes a second control means for controlling the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image, The system has a comparison means for comparing first control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the first image, with second control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the second image. Based on the results of the comparison by the comparison means, the system switches between a first state in which the first control device controls the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the first image, and a second state in which the second control device controls the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image. The first control device transmits the first control information to the second control device. Based on the comparison result by the comparison means, the first control means switches to the second state if the difference between the first control information and the second control information satisfies a predetermined condition, and switches to the first state if the predetermined condition is not met. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The comparison means is characterized by calculating the difference between the first control information and the second control information and outputting the result of comparing the difference with the threshold.

2. The second control means controls the second imaging device to track the predetermined subject when the predetermined subject is within the imaging range of the second imaging device. If the predetermined subject is no longer within the shooting range of the second imaging device, the first control device is notified that tracking of the predetermined subject cannot be continued. The system according to claim 1, characterized in that the first control means switches from the second state to the first state upon receiving the notification.

3. The system according to claim 1, characterized in that when the predetermined subject is changed, the first control means switches from the second state to the first state.

4. The system according to claim 3, characterized in that the second imaging device controls the shooting direction of the second imaging device to track the predetermined subject based on control information acquired from the first control device or the second control device.

5. The system according to claim 3, characterized in that the second imaging device controls the shooting direction of the second imaging device to track a predetermined subject based on either the first control device or the control information acquired from the second control device.

6. The system according to claim 3, characterized in that the control information generated by the first control device and the second control device includes at least one of a pan value and a tilt value.

7. A control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, The system includes control means for controlling the second imaging device to track the predetermined subject, The control means, based on the result of comparing first control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the first image, with second control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the second image, switches between a first state in which the control device controls the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the first image, and a second state in which the external device controls the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the second image. The control means, in the external device, switches to the second state if the difference between the first control information and the second control information satisfies a predetermined condition based on the comparison result, and switches to the first state if the predetermined condition is not met. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The control device is characterized in that the comparison result is obtained by calculating the difference between the first control information and the second control information in the external device and comparing the difference with the threshold value.

8. A control device that controls the second imaging device to track a predetermined subject based on a second image captured by the second imaging device, which has a different shooting direction from the first imaging device, A comparison means for comparing first control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on a first image captured by the first imaging device acquired from an external device, and second control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the second image, The system includes control means for controlling the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image, Based on the results of the comparison by the comparison means, if the difference between the first control information and the second control information satisfies a predetermined condition, the control means controls the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The control device is characterized in that the comparison means calculates the difference between the first control information and the second control information and outputs the result of comparing the difference with the threshold.

9. A control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, The system includes control means for controlling the second imaging device to track the predetermined subject, The control means switches between a first state in which the second imaging device is controlled to track a predetermined subject based on information indicating the position of the predetermined subject detected from the first image, and a second state in which the second imaging device is controlled to track a predetermined subject based on information indicating the position of the predetermined subject detected from the second image, based on the result of comparing first control information including at least one of the pan value and tilt value of the second imaging device calculated based on the first image and second control information including at least one of the pan value and tilt value of the second imaging device calculated based on the second image. Based on the comparison result, the control means switches to the second state if the difference between the first control information and the second control information satisfies a predetermined condition, and switches to the first state if the predetermined condition is not met. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The control device is characterized in that the comparison result is obtained by calculating the difference between the first control information and the second control information, and comparing the difference with the threshold value.

10. A control method for a control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, The control step includes controlling the second imaging device to track the predetermined subject, In the control step, the external device compares first control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the first image, with second control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the second image, and switches between a first state in which the control device controls the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the first image, and a second state in which the external device controls the second imaging device to track a predetermined subject based on information indicating the position of the predetermined subject detected from the second image. In the control step, the external device switches to the second state if the difference between the first control information and the second control information satisfies a predetermined condition based on the comparison result, and switches to the first state if the predetermined condition is not met. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The method is characterized in that the comparison result is obtained by calculating the difference between the first control information and the second control information in the external device and comparing the difference with the threshold value.

11. A control method for a control device that controls a second imaging device to track a predetermined subject based on a second image captured by a second imaging device having a different shooting direction from the first imaging device, A comparison step of comparing first control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on a first image captured by the first imaging device acquired from an external device, with second control information, which includes at least one of the pan value and tilt value of the second imaging device calculated based on the second image, The system includes a control step of controlling the second imaging device to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image, In the control step, based on the results of the comparison, if the difference between the first control information and the second control information satisfies a predetermined condition, the second imaging device is controlled to track the predetermined subject based on information indicating the position of the predetermined subject detected from the second image. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The method is characterized in that, in the comparison step, the difference between the first control information and the second control information is calculated, and the result of comparing the difference with the threshold is output.

12. A control method for a control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a different shooting direction from the first imaging device, The control step includes controlling the second imaging device to track the predetermined subject, In the control step, based on the result of comparing first control information including at least one of the pan value and tilt value of the second imaging device calculated based on the first image and second control information including at least one of the pan value and tilt value of the second imaging device calculated based on the second image, the system switches between a first state in which the second imaging device is controlled to track a predetermined subject based on information indicating the position of the predetermined subject detected from the first image, and a second state in which the second imaging device is controlled to track a predetermined subject based on information indicating the position of the predetermined subject detected from the second image. In the control step, based on the comparison result, if the difference between the first control information and the second control information satisfies a predetermined condition, the system switches to the second state; if the predetermined condition is not met, the system switches to the first state. The predetermined condition is that the difference between the first control information and the second control information is greater than or equal to a threshold. The method is characterized in that the comparison result is obtained by calculating the difference between the first control information and the second control information, and comparing the difference with the threshold value.

13. A program for causing a computer to function as a control device according to any one of claims 7 to 9.

Images