Information processing device, system, control apparatus, control method, and program

Abstract

To reduce that image recognition processing is executed for a subject that is different from a subject of a tracking object.SOLUTION: An information processing device includes: detection means of detecting a subject contained in an image imaged by an imaging device; correction means of correcting predetermined information in regard to a region of a predetermined subject so that the other subject that is different from the predetermined subject is not contained in a region of the predetermined subject from a plurality of subjects contained in the image; and acquisition means of acquiring position information of at least one portion of the predetermined subject on the basis of the predetermined information corrected by the correction means.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a system for tracking a specific subject using a plurality of image capturing devices with different shooting positions and shooting directions. [Background technology]

[0002] There is a technology that tracks a specific subject using an imaging device that can automatically control the pan / tilt / zoom (PTZ) from a remote location. In this type of automatic tracking control, the head of the subject to be tracked is detected from the captured image using image recognition processing, and the PTZ is automatically controlled so that the head of the subject to be tracked is positioned at a desired position within the captured image field.

[0003] Furthermore, in order to perform automatic tracking control without delay, it is necessary to speed up image recognition processing. Patent Document 1 describes a technology for speeding up face detection processing by determining a mask area from image data where face detection processing can be omitted, and performing face detection processing on areas excluding the mask area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2015-201750 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, in Patent Document 1, the mask area is determined by user operation, and if the mask area deviates from the target area for face detection processing, the face detection processing of the detection target cannot be executed, and automatic tracking of the subject to be tracked may not be possible.

[0006] The present invention has been made in view of the above-mentioned problems, and has an object to realize a technology that can reduce the occurrence of image recognition processing being performed on a subject other than the subject to be tracked. [Means for solving the problem]

[0007] In order to solve the above problems and achieve the object, the information processing device of the present invention has a detection means for detecting a subject included in an image captured by an imaging device, a correction means for correcting specified information regarding an area of ​​a specified subject among multiple subjects included in the image so that the area of ​​the specified subject does not include other subjects different from the specified subject, and an acquisition means for acquiring positional information of at least one part of the specified subject based on the specified information corrected by the correction means. Effect of the Invention

[0008] According to the present invention, it is possible to reduce the occurrence of image recognition processing being performed on a subject other than the subject to be tracked. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating an example of a system configuration according to a first embodiment. [Diagram 2] FIG. 2 is a diagram illustrating an example of the hardware configuration of devices constituting the system of the first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating an example of the functional configuration of an apparatus constituting the system of the first embodiment. [Figure 4] 4 is a flowchart illustrating a basic operation of the apparatus constituting the system of the first embodiment. [Diagram 5] 4A to 4C are diagrams for explaining a coordinate conversion method for a captured image according to the first embodiment. [Figure 6] 5A to 5C are views for explaining a subject detection method and a coordinate conversion method according to the first embodiment. [Figure 7] 5A to 5C are diagrams for explaining pan control according to the first embodiment. [Figure 8] 5A to 5C are diagrams for explaining tilt control according to the first embodiment. [Figure 9] 4 is a flowchart illustrating a control process according to the first embodiment. [Figure 10] 4A to 4C are diagrams for explaining a method for determining a subject to be tracked in the first embodiment. [Figure 11] 6 is a flowchart illustrating a process of acquiring head position information of a subject from a sub-image according to the first embodiment. [Figure 12] FIG. 1 is a diagram for explaining input / output data and problems of a trained model for estimating a skeleton position according to the first embodiment. [Figure 13] 6 is a flowchart illustrating a correction process for the sub-image and the position information of the subject according to the first embodiment. [Figure 14] 5A to 5C are diagrams for explaining the degree of inclusion or overlap between subjects according to the first embodiment. [Figure 15] 5A to 5C are views for explaining a correction process for a sub-image and position information of a subject according to the first embodiment. [Figure 16] FIG. 11 is a diagram illustrating an example of the functional configuration of an apparatus constituting a system according to a second embodiment. [Figure 17] 10 is a flowchart illustrating a control process according to the second embodiment. [Figure 18] 10 is a flowchart illustrating a control process according to the second embodiment. [Figure 19] 10 is a flowchart illustrating a process for acquiring head position information of a subject from an overhead image according to the second embodiment. [Figure 20] 10 is a flowchart illustrating a process of correcting an overhead image and position information of a subject according to the second embodiment. [Figure 21] 13A to 13C are views for explaining a process of correcting an overhead image and position information of a subject according to the second embodiment. [Figure 22] FIG. 11 is a diagram illustrating an example of a system configuration according to a third embodiment. [Diagram 23] 13A to 13C are diagrams illustrating examples of roles and contents that can be set in the image capture device of the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [Embodiment 1] <System configuration> First, the system configuration of the first embodiment will be described with reference to FIG.

[0012] The system of this embodiment includes a first control device 100, a second control device 200, a first imaging device 300, and a second imaging device 400. In the system of this embodiment, either the first control device 100 or the second control device 200 controls the second imaging device 400 to track a specific subject.

[0013] The first control device 100 detects a subject from an overhead image captured by the first imaging device 300, determines a subject to be tracked from the detected subjects, and controls the second imaging device 400 based on the subject to be tracked. The first control device 100 is also called a workstation. The subject to be tracked is set, for example, by a user operation or automatically.

[0014] The second control device 200 controls the second imaging device 400 based on the result of recognizing the subject of the tracking target from the overhead image captured by the first imaging device 300 and the result of recognizing the subject of the tracking target from the sub-image captured by the second imaging device 300. The second control device 200 is also called an edge box.

[0015] The first imaging device 300 has a fixed shooting angle of view at a wide angle, and can capture an overhead image including all of subjects A, B, and C. If the subjects are people, the overhead image includes the entire bodies of subjects A, B, and C. The first imaging device 300 is also called an overhead camera. The second imaging device 400 has a variable shooting angle of view, and can capture at least one of subjects A, B, and C. The second imaging device 400 is called a sub-camera. The first imaging device 300 and the second imaging device 400 are placed at positions separated from each other so that their shooting positions and / or shooting directions are different.

[0016] The first control device 100, the second control device 200, the first imaging device 300, and the second imaging device 400 are communicatively connected by a network 600 such as a LAN (Local Area Network). In this embodiment, an example in which the first control device 100, the second control device 200, the first imaging device 300, and the second imaging device 400 are connected by the network 600 will be described, but they may be connected by a connection cable (not shown). In this embodiment, an example in which there is one second imaging device 400 will be described, but there may be two or more. When there are multiple second imaging devices 400, a second control device 200 is provided for each second imaging device 400.

[0017] Next, the basic functions of the system of this embodiment will be described.

[0018] The first imaging device 300 captures an overhead image and transmits the overhead image to the first control device 100 via the network 600 .

[0019] The second imaging device 400 captures a sub-image including a subject to be tracked (tracked subject) and transmits the sub-image to the second control device 200 via the network 600. The second imaging device 400 has a PTZ function. The PTZ function is a function that can control pan, tilt, and zoom of the imaging device. PTZ is an abbreviation of the initials of pan, tilt, and zoom. Pan is the horizontal movement of the optical axis of the imaging device. Tilt is the vertical movement of the optical axis of the imaging device. Zoom is zooming up (telephoto) and zooming out (wide angle). Pan and tilt are functions that change the shooting direction of the imaging device. Zoom is a function that changes the shooting range (shooting angle of view) of the imaging device. In this embodiment, an example will be described in which the subject and the tracking subject are people, but they are not limited to people and may be animals or other moving objects. The second control device 200 controls at least one of the panning and tilting operations of the second imaging device 400 so that a specific part of the subject to be tracked (e.g., a person's head or face) is positioned at a desired position (e.g., the center) within the shooting angle of view of the second imaging device 400.

[0020] The first control device 100 determines a tracking subject from subjects detected from the overhead image received from the first imaging device 300, and calculates first feature information of the tracking subject from the overhead image. The first control device controls the second imaging device 400 to change the shooting direction and shooting range of the second imaging device 400 to the shooting direction and shooting range of the tracking subject based on the first feature information of the tracking subject.

[0021] After changing the shooting direction and shooting range of the second imaging device 400 to the shooting direction and shooting range of the tracked subject, the first control device 100 transmits first feature information of the tracked subject calculated from the overhead image to the second control device 200.

[0022] The second control device 200 detects a subject from the sub-image received from the second imaging device 400, and calculates second feature information of the detected subject. The second control device 200 compares the second feature information of the subject detected from the sub-image with the first feature information of the tracking subject received from the first control device 100.

[0023] When the similarity between the first feature information of the tracked subject and the second feature information of the subject detected from the sub-image is low, the first control device 100 controls the second imaging device 400. The first control device 100 controls the second imaging device 400 to change the shooting direction and shooting range to those of the tracked subject based on the first feature information of the tracked subject.

[0024] Furthermore, when the similarity between the first feature information of the tracking subject and the second feature information of the subject detected from the sub-image is high, the second control device 200 controls the second imaging device 400. The second control device 200 controls the second imaging device 400 to change the shooting direction and shooting range to those of the tracking subject based on the second feature information of the subject detected from the sub-image that has a high similarity to the first feature information of the tracking subject.

[0025] The feature information is information that can identify that the same subject is captured by multiple imaging devices with different shooting positions and / or shooting directions. The feature information is an inference result that is output by performing image recognition by inference processing using a trained model on multiple images captured by multiple imaging devices with different shooting positions and / or shooting directions as input. When an inference result that the subjects are the same is obtained, it is possible to identify that the subjects included in the multiple images captured by the multiple imaging devices with different shooting positions and / or shooting directions are the same subject.

[0026] In the following description, the first control device 100 is referred to as a workstation (WS), the second control device 200 as an edge box (EB), the first imaging device 300 as an overhead camera, and the second imaging device 400 as a sub-camera.

[0027] <Device configuration> Next, the hardware configurations of the WS 100, the EB 200, the overhead camera 300, and the sub-camera 400 will be described in detail with reference to FIG.

[0028] First, the configuration of the WS 100 will be described.

[0029] The WS 100 includes a control unit 101, a volatile memory 102, a non-volatile memory 103, an inference unit 104, a communication unit 105, and an operation unit 106, and each unit is connected via an internal bus 110 so as to be able to send and receive data.

[0030] The control unit 101 has a processor (CPU) that performs arithmetic processing and control processing of the WS 100 , and controls each of the components of the WS 100 by executing a control program stored in the non-volatile memory 103 .

[0031] The volatile memory 102 is a main storage device such as a RAM. Constants and variables for the operation of the control unit 101, and control programs and inference programs read from the non-volatile memory 103 are loaded into the volatile memory 102. The volatile memory 102 also stores information such as image data and inference programs received from an external device by the communication unit 105. The volatile memory 102 also stores overhead image data received from the overhead camera 300. The volatile memory 102 has a sufficient storage capacity to hold this information.

[0032] The non-volatile memory 103 is an auxiliary storage device such as an EEPROM, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a memory card, etc. The non-volatile memory 103 stores an OS (operating system) which is basic software executed by the control unit 101, a control program including an application which cooperates with the OS to realize applied functions, etc. The non-volatile memory 103 also stores an inference program used by the inference unit 104 for inference processing, etc.

[0033] The inference unit 104 executes an inference process using a learned inference model and inference parameters according to an inference program. The inference unit 104 executes an inference process to estimate position information and feature information of a specific object from an overhead image received from the overhead camera 300. The inference process in the inference unit 104 can be executed by a calculation processing device specialized for image processing and inference process, such as a GPU (Graphics Processing Unit). The GPU is a processor capable of performing a large amount of product-sum calculations, and has a calculation processing capacity for performing matrix calculations of a neural network in a short time. The inference process in the inference unit 104 may be realized by a reconfigurable logic circuit, such as an FPGA (Field-Programmable Gate Array). The inference process may be performed by a CPU and a GPU of the control unit 101 working together, or by either the CPU or the GPU of the control unit 101.

[0034] The communication unit 105 is an interface (I / F) that complies with a wired communication standard such as Ethernet (registered trademark) or an interface that complies with a wireless communication standard such as Wi-Fi (registered trademark). The communication unit 105 is connected to external devices such as the EB 200, the overhead camera 300, and the sub-camera 400 via a network 600 such as a wired LAN or a wireless LAN, and can transmit and receive data to and from the external devices. The control unit 101 realizes communication with the external devices by controlling the communication unit 105. Note that the communication method is not limited to Ethernet (registered trademark) or Wi-Fi (registered trademark), and a communication standard such as IEEE1394 may also be used.

[0035] The operation unit 106 is an operation member such as various switches, buttons, a touch panel, etc. that receives various operations by the user and outputs operation information to the control unit 101. The operation unit 106 also provides a user interface for the user to operate the WS100.

[0036] The display unit 111 displays an overhead image, a subject recognition result, a GUI (Graphical User Interface) for interactive operations, etc. The display unit 111 is a display device such as a liquid crystal display or an organic EL display. The display unit 111 may be an integral part of the WS 100, or may be an external device connected to the WS 100.

[0037] Next, the configuration of the EB 200 will be described.

[0038] The EB 200 includes a control unit 201, a volatile memory 202, a non-volatile memory 203, an inference unit 204, and a communication unit 205, and each unit is connected via an internal bus 210 so as to be able to transmit and receive data.

[0039] The control unit 201 has a processor (CPU) that performs arithmetic processing and control processing of the EB 200, and controls each component of the EB 200 by executing a control program stored in the non-volatile memory 203.

[0040] The volatile memory 202 is a main storage device such as a RAM. Constants and variables for the operation of the control unit 201, and control programs and inference programs read from the non-volatile memory 203 are loaded into the volatile memory 202. The volatile memory 202 also stores information such as image data and inference programs received from an external device by the communication unit 205. The volatile memory 202 also stores sub-image data received from the sub-camera 400. The volatile memory 202 has a sufficient storage capacity to hold this information.

[0041] The non-volatile memory 203 is an auxiliary storage device such as an EEPROM, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a memory card, etc. The non-volatile memory 203 stores an OS (operating system) which is basic software executed by the control unit 201, a control program including an application which cooperates with the OS to realize applied functions, etc. The non-volatile memory 203 also stores an inference program used by the inference unit 204 for inference processing, etc.

[0042] The inference unit 204 executes an inference process using a learned inference model and inference parameters according to an inference program. The inference unit 204 executes an inference process to estimate position information and feature information of a specific object from a sub-image received from the sub-camera 400. The inference process in the inference unit 204 can be executed by a calculation processing device specialized for image processing and inference process, such as a GPU (Graphics Processing Unit). The GPU is a processor capable of performing a large amount of product-sum calculations, and has a calculation processing capability for performing matrix calculations of a neural network in a short time. The inference process in the inference unit 204 may be realized by a reconfigurable logic circuit, such as an FPGA (Field-Programmable Gate Array). The inference process may be performed by the CPU and GPU of the control unit 201 working together, or by either the CPU or the GPU of the control unit 201.

[0043] The communication unit 205 is an interface (I / F) that complies with a wired communication standard such as Ethernet (registered trademark) or an interface that complies with a wireless communication standard such as Wi-Fi (registered trademark). The communication unit 205 is connected to external devices such as the WS 100 and the sub-camera 400 via a network 600 such as a wired LAN or wireless LAN, and can transmit and receive data to and from the external devices. The control unit 201 realizes communication with the external devices by controlling the communication unit 205. Note that the communication method is not limited to Ethernet (registered trademark) or Wi-Fi (registered trademark), and a communication standard such as IEEE1394 may also be used.

[0044] Next, the configuration of the overhead camera 300 will be described.

[0045] The overhead camera 300 includes a control unit 301, a volatile memory 302, a non-volatile memory 303, a communication unit 305, an imaging unit 306, and an image processing unit 307, and each unit is connected via an internal bus 310 so as to be able to send and receive data.

[0046] The control unit 301 performs overall control of the overhead camera 300 under the control of the WS 100. The control unit 301 has a processor (CPU) that performs calculation processing and control processing of the overhead camera 300, and controls each component of the overhead camera 300 by executing a control program stored in the non-volatile memory 303.

[0047] Volatile memory 302 is a main storage device such as a RAM. Constants and variables for the operation of control unit 301, and control programs and inference programs read from non-volatile memory 303 are loaded into volatile memory 302. Volatile memory 302 also stores overhead image data captured by imaging unit 306 and processed by image processing unit 307. Volatile memory 302 has a sufficient storage capacity to hold this information.

[0048] The non-volatile memory 303 is an auxiliary storage device such as an EEPROM, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a memory card, etc. The non-volatile memory 303 stores an OS (operating system) which is basic software executed by the control unit 301, and a control program including an application which cooperates with the OS to realize practical functions.

[0049] The imaging unit 306 has an image sensor composed of a CCD (charge-coupled device), a CMOS (complementary metal-oxide semiconductor) element, etc., and converts an optical image of a subject into an electrical signal. In this embodiment, the overhead camera 300 has a fixed shooting angle of view so that an overhead image including a plurality of subjects including a tracking subject can be captured.

[0050] The image processing unit 307 executes various image processing on image data output from the imaging unit 306 or image data read from the volatile memory 302. The various image processing includes, for example, image processing such as noise removal, edge emphasis, and enlargement / reduction, image correction processing such as contrast correction, brightness correction, and color correction, and trimming or cropping processing for cutting out a part of the image data. The image processing unit 307 converts the image data that has been subjected to the image processing into an image file of a still image or a video in a predetermined format (for example, JPEG or MP4) and records it in the non-volatile memory 303. In addition, the image processing unit 307 executes predetermined calculation processing using the image data, and the control unit 301 executes AF (autofocus) processing and AE (autoexposure) processing based on the calculation result.

[0051] The communication unit 305 is an interface (I / F) that complies with a wired communication standard such as Ethernet (registered trademark) or an interface that complies with a wireless communication standard such as Wi-Fi (registered trademark). The communication unit 305 can connect to an external device such as WS100 via a network 600 such as a wired LAN or wireless LAN, and transmit and receive data to and from the external device. The control unit 301 realizes communication with the external device by controlling the communication unit 305. Note that the communication method is not limited to Ethernet (registered trademark) or Wi-Fi (registered trademark), and a communication standard such as IEEE1394 may also be used.

[0052] Next, the configuration of the sub-camera 400 will be described.

[0053] The sub-camera 400 comprises a control unit 401, a volatile memory 402, a non-volatile memory 403, a communication unit 405, an imaging unit 406, an image processing unit 407, an optical unit 408 and a PTZ drive unit 409, and each unit is connected via an internal bus 410 so as to be able to send and receive data.

[0054] The control unit 401 performs overall control of the sub-camera 400 under the control of the WS 100 or the EB 200. The control unit 401 has a processor (CPU) that performs arithmetic processing and control processing of the sub-camera 400, and controls each component of the sub-camera 400 by executing a control program stored in the non-volatile memory 403.

[0055] The volatile memory 402 is a main storage device such as a RAM. Constants and variables for the operation of the control unit 401, and control programs and inference programs read from the non-volatile memory 403 are loaded into the volatile memory 402. The volatile memory 402 also stores overhead image data captured by the imaging unit 406 and processed by the image processing unit 407. The volatile memory 402 has a sufficient storage capacity to hold this information.

[0056] The non-volatile memory 403 is an auxiliary storage device such as an EEPROM, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), a memory card, etc. The non-volatile memory 403 stores an OS (operating system) which is basic software executed by the control unit 401, and a control program including an application which cooperates with the OS to realize practical functions.

[0057] The imaging unit 406 has an image sensor configured with a CCD (charge-coupled device), a CMOS (complementary metal-oxide semiconductor) element, or the like, and converts an optical image of a subject into an electrical signal.

[0058] The image processing unit 407 executes various image processing on image data output from the imaging unit 406 or image data read from the volatile memory 402. The various image processing includes, for example, image processing such as noise removal, edge emphasis, and enlargement / reduction, image correction processing such as contrast correction, brightness correction, and color correction, and trimming or cropping processing for cutting out a part of the image data. The image processing unit 407 converts the image data that has been subjected to image processing into an image file in a predetermined format (for example, JPEG) and records it in the non-volatile memory 403. In addition, the image processing unit 407 executes predetermined calculation processing using the image data, and the control unit 401 executes AF (autofocus) processing and AE (autoexposure) processing based on the calculation result.

[0059] The communication unit 405 is an interface (I / F) that complies with a wired communication standard such as Ethernet (registered trademark) or an interface that complies with a wireless communication standard such as Wi-Fi (registered trademark). The communication unit 405 can connect to an external device such as the EB 200 via a network 600 such as a wired LAN or wireless LAN, and can transmit and receive data to and from the external device. The control unit 401 realizes communication with the external device by controlling the communication unit 405. Note that the communication method is not limited to Ethernet (registered trademark) or Wi-Fi (registered trademark), and a communication standard such as IEEE1394 may also be used.

[0060] The optical unit 408 includes a lens group including a zoom lens and a focus lens, a shutter with an aperture function, and a mechanism for driving these optical members. The optical unit 408 drives the optical members to at least one of rotate the shooting direction of the sub-camera 400 around the pan (P) axis (horizontal direction) or the tilt (T) axis (vertical direction), and change the shooting range (shooting angle of view) of the sub-camera 400 along the zoom (Z) axis (enlargement / reduction direction).

[0061] The PTZ driving unit 409 includes actuators such as mechanical elements and motors for driving the optical unit 408 in the PTZ direction, and drives the optical unit 408 in the PTZ direction under the control of the control unit 401 .

[0062] Note that the zoom function of this embodiment is not limited to optical zoom, which changes the focal length by moving a zoom lens, but may be digital zoom, which cuts out and enlarges a portion of the captured image data, or a combination of optical zoom and digital zoom.

[0063] [Control process] Next, with reference to Figures 3 to 10, a control process for tracking a tracking subject by switching between a mode in which WS100 controls the sub-camera 400 based on an overhead image and a mode in which EB200 controls the sub-camera 400 based on a sub-image will be described.

[0064] First, the functional configuration of the WS 100 and the EB 200 for implementing the control process of this embodiment will be described with reference to FIGS.

[0065] Each function of the WS 100 and the EB 200 is realized by hardware and / or software. If the functional units shown in Fig. 3 are configured by hardware instead of by software, it is sufficient to have a circuit configuration corresponding to each functional unit in Fig. 3.

[0066] The WS 100 includes an image recognition unit 121, an object of interest determination unit 122, a tracking target determination unit 123, a control information generation unit 124, a feature information determination unit 125, and a tracking state determination unit 126. Software for realizing these functions is stored in the non-volatile memory 103, and the control unit 101 loads it into the volatile memory 102 and executes it.

[0067] The EB 200 includes an image recognition unit 221, a tracking target determination unit 222, and a control information generation unit 223. These pieces of software are stored in the non-volatile memory 203, and the control unit 201 loads them into the volatile memory 202 and executes them.

[0068] Fig. 4(a) is a flowchart showing the basic operation of the WS 100. Fig. 4(b) is a flowchart showing the basic operation of the EB 200. Fig. 4(c) is a flowchart showing the operation of the overhead camera 300. Fig. 4(d) is a flowchart showing the operation of the sub-camera 400.

[0069] First, the software functions and basic operations of the WS 100 will be described with reference to FIG. 3 and FIG.

[0070] In step S101, the control unit 101 transmits a shooting command to the overhead camera 300 using a predetermined protocol via the communication unit 105, receives an overhead image from the overhead camera 300, stores it in the volatile memory 102, and advances the process to step S102.

[0071] In step S102, the control unit 101 executes the function of the image recognition unit 121 in FIG. 3, and advances the process to step S103.

[0072] The image recognition unit 121 controls the inference unit 104, the volatile memory 102, and the non-volatile memory 103, and performs the following object recognition process.

[0073] The image recognition unit 121 receives the overhead image IMG of the overhead camera 300 read from the volatile memory 102 and the reference position information REF_POSI of the overhead camera 300. The position information REF_POSI of the overhead camera 300 includes the coordinate information of the overhead camera 300 and the position information of the marker. The image recognition unit 121 detects the subject and calculates the position information and characteristic information of the detected subject based on the overhead image IMG of the overhead camera 300 and the reference position information REF_POSIREF_POSI. Then, the image recognition unit 121 outputs the position information POSITION[n] of the detected subject, ID[n] indicating the identification information of the detected subject, and STAT[n] indicating the characteristic information of the detected subject. The position information of the overhead camera 300 is coordinate information in a coordinate space in which the shooting area of ​​the overhead camera 300 is viewed from directly above, and is known in advance by a user operation or by measurement using a sensor (not shown). The position information of the marker is coordinate information of the marker set in a coordinate space in which the shooting area of ​​the overhead camera 300 is viewed from directly above in order to calculate a homography transformation matrix described later, and is a known value measured in advance manually or using a sensor (not shown). The marker is like a mark having a color different from the color of the floor or ground, and may be any type as long as it can be operated by a user or measured by a sensor (not shown). For example, if the sensor (not shown) is a camera, the position information of the marker is obtained by extracting the color of the marker from an image captured with the marker marked with an arbitrary color. The position information of the overhead camera 300 and the position information of the marker may be input by a user via the operation unit 106 of the WS 100, and the control unit 101 may store the position information in the volatile memory 102. The reference position information REF_POSI of the overhead camera 300 and the position information POSITION[n] of the detected subject are expressed in a coordinate system converted into a coordinate space in which the shooting area of ​​the overhead camera 300 is viewed from directly above. n is an index indicating the number of detected subjects, and for example, when the inference unit 104 detects three people, the position information POSITION, identification information ID, and feature information STAT for the three people are output as the inference result. The control unit 101 stores the subject recognition result by the image recognition unit 121 in the volatile memory 102. The subject detection process and feature information calculation process will be described in detail later.

[0074] Here, a method for calculating the subject position information POSITION by the image recognition unit 121 will be described.

[0075] First, with reference to FIG. 5, the relationship between the coordinate system of the overhead image of the overhead camera 300 and the coordinate system when the shooting area of ​​the overhead camera 300 is viewed from directly above will be described.

[0076] In order to calculate a pan value such that the shooting direction of the sub camera 400 is the direction of the tracking subject, the calculation is simplified by calculating the angle in a plane coordinate space perpendicular to the axis along which the sub camera 400 performs the panning operation. For example, when the sub camera 400 is installed perpendicular to a ground surface (reference position) such as a floor or the ground, the coordinate space perpendicular to the axis along which the sub camera 400 performs the panning operation is a coordinate space parallel to the reference position (a coordinate space in which the space in which the sub camera 400 and the subject are present is viewed from directly above) as shown in FIG. 5(b). In this embodiment, it is assumed that the sub camera 400 is installed perpendicular to the reference position, and the pan value is calculated in a coordinate system in which the shooting area of ​​the overhead camera 300 is viewed from directly above. That is, the position information of the subject detected in the coordinate system of the overhead image of the overhead camera 300 shown in FIG. 6(a) (hereinafter, overhead camera coordinate system) is converted to a coordinate system in which the shooting area of ​​the overhead camera 300 shown in FIG. 5(b) is viewed from directly above (hereinafter, plane coordinate system). The coordinate transformation is performed using the homography transformation matrix H according to the following formula 1. (Formula 1) TIFF2025060252000002.tif19155In Equation 1, x and y are the horizontal and vertical coordinates in the overhead camera coordinate system, and X and Y are the horizontal and vertical coordinates in the planar coordinate system.

[0077] The control unit 101 reads out the reference position information REF_POSI from the volatile memory 102, and substitutes the marker position information Mark_A to Mark_D shown in FIGS. 5(a) and (b) included in the reference position information REF_POSI into Equation 1 to calculate the homography transformation matrix H. The marker position information is a coordinate value in a planar coordinate system. By using Equation 1, it is possible to map any position in the overhead camera coordinate system of FIG. 5(a) to any position in the planar coordinate system of FIG. 5(b). In the example of FIG. 5, the control unit 101 can grasp the positions of the subjects A, B, and C included in the overhead image IMG of the overhead camera 300 in the planar coordinate system of FIG. 4(b). The control unit 101 stores the homography transformation matrix H calculated by the above Equation 1 in the volatile memory 102.

[0078] Next, a method for detecting a subject using an inference model for subject detection and a method for converting the position information of the subject into a planar coordinate system will be described.

[0079] In this embodiment, subject detection is performed by performing image recognition processing using a trained inference model for subject detection created by machine learning such as deep learning.

[0080] The inference model for subject detection takes an overhead image as input and outputs the position of the subject contained in the overhead image as coordinate information.

[0081] The control unit 101 detects a subject by performing image recognition processing using an inference model for subject detection using the overhead image IMG of the overhead camera 300 as input by the inference unit 104. FIG. 6(a) shows an example in which the subject detected by the inference unit 104 is displayed in a rectangular frame. As shown in FIG. 6(a), the coordinates of rectangular areas circumscribing the subjects A, B, and C detected from the overhead image are detected as position information of the subject. The control unit 101 stores the position information of the subject detected from the overhead image in the volatile memory 102. Note that, in this embodiment, an example of subject detection by inference processing using a trained model has been described, but the present invention is not limited to this. For example, a method called the SIFT method, which detects by matching local feature points in an image, or a method called a template matching method, which detects by calculating similarity with a template image, may be used.

[0082] Furthermore, the control unit 101 converts the lower end of the rectangular area of ​​the subject detected in the overhead camera coordinate system shown in FIG. 6(a) into the planar coordinate system shown in FIG. 6(b) as the position information of the subject (for example, the foot coordinates of a person). For example, the control unit 101 reads out the homography transformation matrix H from the volatile memory 102, and substitutes the foot coordinates (xa, ya) of the subject A in the overhead camera coordinate system into x and y in Equation 1, thereby making it possible to convert the foot coordinates (XA, YA) in the planar coordinate system. Similarly, for the foot coordinates (xb, yb) of the subject B and the foot coordinates (xc, yc) of the subject C, it becomes possible to calculate the foot coordinates (XB, YB) of the subject B and the foot coordinates (XC, YC) of the subject C in the planar coordinate system. The control unit 101 writes the foot coordinates of the subject into the volatile memory 102 as the position information POSITION of the subject.

[0083] Next, a method for generating subject identification information ID and feature information STAT by image recognition unit 121 will be described.

[0084] The inference unit 104 performs inference processing by inputting the position information POSITION of the detected subject, which is the inference result of the inference model for subject detection, and the overhead image of the overhead camera 300, into a trained inference model for subject identification created by machine learning such as deep learning. Then, the inference unit 104 outputs the identification information ID and feature information STAT of the detected subject. The inference model for subject identification is different from the inference model for subject detection.

[0085] Here, we will explain the inference model for subject identification.

[0086] The inference model for identifying a subject of this embodiment is a trained model that has been trained to increase the similarity of feature information for images of the same subject using training data that is a collection of data as many as the number of subjects that associate a set of images of a specific subject taken from multiple different shooting directions with information that can identify the specific subject. Feature information STAT is output by inputting an image of a subject cut out based on the position information POSITION of the subject, which is the output of the inference model for detecting a subject, into the inference model for identifying a subject. When an image of the same subject taken by a different camera is input, the output feature information has a higher similarity to the feature information STAT compared to when an image of a different subject is input. Examples of feature information include a multidimensional vector of the response of the convolution layer of a convolutional neural network. The similarity of the feature information of a subject will be described later.

[0087] The inference model for subject detection and the inference model for subject identification are stored in the non-volatile memory 103 before the control process of this embodiment is started.

[0088] Moreover, the image recognition unit 121 assigns identification information ID corresponding to the feature information of the detected subject, which is the inference result of the inference model for identifying the subject. Furthermore, the image recognition unit 121 calculates the similarity of the feature information of each image of the subject obtained by inputting the image of each subject detected by the inference model for detecting the subject, using each image of the current frame and the past frame as input, into the inference model for identifying the subject. The similarity is calculated using cosine similarity. The cosine similarity is closer to 1 as the multidimensional vector, which is the feature information of each subject image, is more similar, and closer to 0 as the multidimensional vector is more different. The same ID is assigned to the subject having the closest similarity between the past frame and the current frame. Note that the method of calculating the similarity is not limited to this, and any method may be used as long as it outputs a higher value as the feature information is closer, and outputs a lower value as the feature information is further away. Note that, in this embodiment, the feature information is used to assign the ID, but this is not limited to this. A method may be used in which the position and size of the detected subject are compared using the position information of the rectangular area of ​​the subject obtained by the inference model for detecting the subject between the current frame and the past frame, and the same ID is assigned to the closest subject. Alternatively, the position information of a rectangular area in the current frame may be predicted from the transition of the position information of the rectangular area of ​​the subject for the same ID in the past few frames using a Kalman filter or the like, and the same ID may be assigned to the subject that is closest to the position of the predicted rectangular area. IDs may also be assigned using a combination of these methods. By using such a method, the accuracy of ID assignment can be improved when a subject with a similar appearance suddenly enters the shooting angle of view.

[0089] As described above, the image recognition unit 121 performs inference processing using an inference model for subject detection with the overhead image of the overhead image 300 as input, thereby outputting position information POSITION of the subject and storing it in the volatile memory 102. The image recognition unit 121 also inputs the subject position information POSITION, which is the inference result of the inference model for subject detection, and the overhead image of the overhead camera 300 into an inference model for subject identification to perform inference processing. Then, the image recognition unit 121 outputs identification information ID and feature information STAT of the detected subject as a result of the inference processing, and stores it in the volatile memory 102.

[0090] Returning to the description of FIG. 4, in step S103, the control unit 101 executes the function of the target subject determination unit 122 in FIG. 3, and advances the process to step S104.

[0091] The target subject determining unit 122 determines a target subject MAIN_SUBJECT from operation information input by the user via the operation unit 106 and position information of the detected subject, which is the subject recognition result by the image recognition unit 121 read from the volatile memory 102 .

[0092] The control unit 101 displays the overhead image captured by the overhead camera 300 and the subject recognition results in the volatile memory 102 on the display unit 111 of the WS 100. The control unit 101 allows the user to select a subject of interest from among the subjects displayed as subject recognition results via the operation unit 106. For example, if the operation unit 106 is a mouse, the user can click and select one of the subjects displayed on the display unit 111. The control unit 101 saves the identification information ID corresponding to the subject of interest selected by the user in the volatile memory 102 as the subject of interest MAIN_SUBJECT.

[0093] In step S104, the control unit 101 executes the function of the tracking target determination unit 123 in FIG. 3, and advances the process to step S105.

[0094] The tracking target determining unit 123 determines a tracking subject SUBJECT_ID of the sub camera 400 from the target subject MAIN_SUBJECT determined by the target subject determining unit 122 .

[0095] Here, a method for determining a subject to be tracked by the sub camera 400 will be described.

[0096] The control unit 101 reads out the target subject MAIN_SUBJECT determined by the target subject determination unit 122 from the volatile memory 102, and determines the target subject MAIN_SUBJECT as the tracking subject SUBJECT_ID of the sub camera 400. In this way, by setting the same subject as the target subject MAIN_SUBJECT selected by the user as the tracking subject SUBJECT_ID of the sub camera 400, it is possible to control the sub camera 400 with the subject selected by the user as the tracking target.

[0097] The method of determining the tracking subject is not limited to the above method, and may be determined using, for example, the target subject MAIN_SUBJECT and the identification information ID read from the volatile memory 102. For example, when the overhead image of the overhead camera 300 includes multiple subjects and multiple sub cameras 400 are installed, one sub camera may track the same subject as the target subject, and another sub camera may track a different subject from the target subject. By determining the tracking subject in this way, it is possible to comprehensively track multiple subjects included in the overhead image of the overhead camera 300 for each sub camera. In addition, there is also a method of determining the subject closest to the sub camera as the tracking subject from the subjects detected from the overhead image of the overhead camera 300 based on the subject position information POSITION, identification information ID, and reference position information REF_POSI including the sub camera position. By determining the tracking subject in this way, it is possible to set the subject that is easiest to fit in the angle of view from the position of the sub camera as the tracking subject. The control unit 101 stores the tracking subject SUBJECT_ID determined as described above in the volatile memory 102, and also stores the identification information ID of the tracking subject before storage in the volatile memory 102 as the identification information ID of the tracking subject in the past.

[0098] In step S105, the control unit 101 executes the function of the feature information determination unit 125 to transmit feature information corresponding to the tracking subject of the sub camera 400 to the EB 200. In addition, the control unit 101 executes the function of the tracking state determination unit 126 to update the tracking state information STATE, store it in the volatile memory 102, and proceeds to step S106.

[0099] The tracking state information STATE includes any one of the following information: "Tracking by WS100" and "Tracking by EB200". "Tracking by WS100" indicates a state in which the WS100 is tracking the tracking subject by controlling the sub-camera 400. "Tracking by EB200" indicates a state in which the EB200 is tracking the tracking subject by controlling the sub-camera 400. The process of step S105 will be described later in detail.

[0100] In step S106, the control unit 101 reads the tracking state information STATE from the volatile memory 102, and determines whether "tracking by WS100" or "tracking by EB200" based on the tracking state information STATE. If the control unit 101 determines that "tracking by WS100" is true, the process proceeds to step S107, and if the control unit 101 determines that "tracking by EB200" is true, the process returns to step S101.

[0101] In step S107, the control unit 101 executes the function of the control information generating unit 124 in FIG. 3, and the process proceeds to step S108.

[0102] The control information generating unit 124 calculates pan / tilt values ​​PT_VALUE of the sub camera 400 for tracking the tracking subject SUBJECT_ID determined by the tracking target determining unit 123 by the sub camera 400. The control unit 101 reads out position information of the sub camera 400 in a planar coordinate system included in the reference position information REF_POSI and position information POSITION of the detected subject from the volatile memory 102. Then, the control unit 101 calculates pan / tilt values ​​such that the shooting direction of the sub camera 400 is the direction of the tracking subject from the position information of the subject corresponding to the tracking subject SUBJECT_ID.

[0103] Here, a method for calculating the pan value will be described with reference to FIG.

[0104] As shown in FIG. 7, the angle θ between a line extending from the center of the optical axis of the sub-camera 400 and a line connecting the sub-camera 400 and the tracking subject SUBJECT_ID can be calculated by the following formula 2. (Formula 2) TIFF2025060252000003.tif13155 In Equation 2, px and py are the horizontal and vertical coordinates of the tracked subject, and subx and suby are the horizontal and vertical coordinates of the sub camera 400. px and py can be obtained from the position information POSITION of the detected subject by referring to the position information of the subject corresponding to the tracked subject SUBJECT_ID.

[0105] The control information generator 124 calculates the pan value of the sub camera 400 based on the angle θ.

[0106] Next, a method for calculating the tilt value will be described with reference to FIG.

[0107] As shown in FIG. 8, the height h1 of the optical axis of the sub-camera 400 is taken as the angle ρ between a line extending from the center of the optical axis of the sub-camera 400 and a line extending toward the height of a specific part of the tracked subject (the height of the head in the case of a person) h2, and this angle ρ can be calculated using the following equations 3 and 4. (Formula 3) TIFF2025060252000004.tif13155 (Formula 4) TIFF2025060252000005.tif19155 In formula 4, h1 is the height of the sub-camera 400 from the ground surface, and h2 is the height from the ground surface to a predetermined part of the tracking subject (the head in the case of a person). h1 and h2 may be stored in the volatile memory 102 in advance, or may be measured in real time using a sensor (not shown).

[0108] The control information generator 124 calculates the tilt value of the sub camera 400 based on the angle ρ.

[0109] The pan value / tilt value may be a speed value for directing the sub camera 400 toward the tracking subject. The method of calculating the pan value / tilt value is as follows: first, the control unit 101 acquires the current pan value / tilt value of the sub camera 400 from the EB 200. Next, the control unit 101 calculates a pan angular velocity proportional to the difference with the pan value θ read from the volatile memory 102. The control unit 101 also calculates a tilt angular velocity proportional to the difference with the tilt control value ρ read from the volatile memory 102. Then, the control unit 101 stores the calculated control value in the volatile memory 102.

[0110] In step S108, the control unit 101 reads the pan value / tilt value from the volatile memory 102, converts them into control commands according to a predetermined protocol for controlling the sub camera 400, stores them in the volatile memory 102, and proceeds to step S109.

[0111] In step S109, the control unit 101 transmits a control command according to the pan value / tilt value calculated in step S108 to the sub-camera 400 via the communication unit 105, and returns the process to step S101.

[0112] The above is the basic operation of the WS100.

[0113] Next, the function and basic operation of the EB 200 will be described with reference to FIG. 3 and FIG. 4(b).

[0114] In step S201, the control unit 201 transmits a shooting command to the sub-camera 400 via the communication unit 205, receives a captured sub-image from the sub-camera 400, stores the captured sub-image in the volatile memory 202, and advances the process to step S202.

[0115] In step S202, the control unit 201 executes the function of the image recognition unit 221 in FIG. 3, and advances the process to step S203.

[0116] The image recognition unit 221 has the same function as the image recognition unit 121 of the WS 100. The control unit 201 inputs the sub-image of the sub-camera 400 read from the volatile memory 202 by the inference unit 204 into a trained model created by performing machine learning such as deep learning, and performs inference processing. The inference result includes position information POSITION, feature information STAT_SUB[m], and identification information ID for each subject detected from the sub-image of the sub-camera 400, and is stored in the volatile memory 202. The trained models used in the inference processing of the image recognition unit 221 are a first trained model (an inference model for subject detection and an inference model for subject identification) common to the trained models used in the image recognition unit 121 of the WS 100, and a second trained model (an inference model for skeletal position estimation) for estimating skeletal position information of the subject. The inference model for subject detection, the inference model for subject identification, and the inference model for skeletal position estimation are stored in the non-volatile memory 203 before starting the control processing of this embodiment.

[0117] The control unit 201 calculates the position information POS_HEAD of the subject's head from the skeletal position information of the subject, which is the image recognition result of the image recognition unit 221, and controls the sub-camera 400 based on the head position information POS_HEAD. The process of the EB200 detecting the head position information of the subject from the sub-image will be described later. The image recognition unit 221 also executes pre- and post-processing of the first and second trained models. The pre-processing of the trained model is, for example, a process of normalizing the sub-image expressed in 8 bits to match the trained model, and a process of correcting the brightness and saturation. The post-processing of the trained model is, for example, a clustering process in which, when a large number of subject position information POSITIONs are detected from the sub-image, they are grouped into a predetermined number.

[0118] In step S203, the control unit 201 receives the feature information STAT of the subject from the WS 100 via the communication unit 205, and compares it with the feature information STAT_SUB calculated from the sub-image of the sub-camera 400 using the function of the tracking target determination unit 222 in FIG. When a subject having a high similarity between the feature information STAT and the feature information STAT_SUB is present within the shooting angle of view of the sub camera 400, the control unit 201 determines the identification information ID of the subject as the identification information ID=SUBJECT_ID of the subject to be tracked by the sub camera 400. Then, the control unit 201 stores the identification information ID of the subject to be tracked in the volatile memory 102, and proceeds to step S204. The method of calculating the similarity will be described in detail later.

[0119] In step S204, the control unit 201 checks the communication state for the WS 100 to stop tracking or continue tracking via the communication unit 205, performs processing according to the communication content, and proceeds to step S205. The processing in step S204 will be described in detail later.

[0120] In step S205, the control unit 201 determines whether or not information on the tracking subject SUBJECT_ID is stored in the volatile memory 202. If the control unit 201 determines that the information of the tracking subject SUBJECT_ID is stored in the volatile memory 202, that is, the identification information ID of the tracking subject of the sub camera 400 is stored in the volatile memory 102, the process proceeds to step S206. If the control unit 201 determines that the information of the tracking subject SUBJECT_ID is not stored in the volatile memory 202, that is, the identification information ID of the tracking subject of the sub camera 400 is not stored in the volatile memory 102, the process returns to step S201.

[0121] In step S206, the control unit 201 reads out the identification information ID for each subject, which is the subject recognition result in step S202, from the volatile memory 202, and determines whether or not the tracking subject SUBJECT_ID exists in the sub-image of the sub-camera 400. If the control unit 201 determines that the tracking subject SUBJECT_ID exists in the sub-image, the process proceeds to step S207, and if it determines that the tracking subject SUBJECT_ID does not exist, the process returns to step S201.

[0122] In step S207, the control unit 101 executes the function of the control information generating unit 223 in FIG. 3, and proceeds to step S208.

[0123] The control information generating unit 223 has a function of calculating the pan value / tilt value of the sub camera 400. The control unit 201 reads out the subject position information POSITION, the subject head position information POS_HEAD, and the tracking subject SUBJECT_ID from the volatile memory 202. Then, the control unit 201 specifies the current position of the tracking subject corresponding to the tracking subject SUBJECT_ID. The control unit 201 reads out the past positions of the tracking subject within the shooting angle of view from the volatile memory 202. Then, the control unit 201 calculates the pan value so that the angular velocity of the pan increases as the difference between the current position of the tracking subject and the past position of the tracking subject increases in the horizontal direction, and calculates the tilt value so that the angular velocity of the tilt increases as the difference in the vertical direction increases. The control unit 201 stores the calculated pan value / tilt value in the volatile memory 202. In this embodiment, the EB200 controls the panning / tilting of the sub-camera 400 based on the head position information POS_HEAD of the tracking subject so that the head of the tracking subject is positioned at a desired position (for example, the center of the screen) within the shooting angle of view of the sub-camera 400. A method for calculating the head position information POS_HEAD of the subject will be described later.

[0124] In step S208, the control unit 201 converts the pan / tilt values ​​read from the volatile memory 202 into control commands in accordance with a predetermined protocol for controlling the sub camera 400, stores the control commands in the volatile memory 202, and proceeds to step S209.

[0125] In step S209, the control unit 201 transmits a control command corresponding to the pan value / tilt value calculated in step S208 to the sub-camera 400 via the communication unit 205, and returns the process to step S101.

[0126] The above is the basic operation of the EB200.

[0127] As described above, WS100 performs image recognition processing on the overhead image of overhead camera 300, and controls the panning / tilting operations of sub camera 400 when tracking state information STATE is "tracking by WS100". When tracking state information STATE is "tracking by EB200", WS100 does not control the panning / tilting operations of sub camera 400. EB200 performs image recognition processing on the sub image of sub camera 400, and controls the panning / tilting operations of sub camera 400 when a tracking subject is set and detected from the sub image. When a tracking subject is not set, EB200 does not control the panning / tilting operations of sub camera 400. In addition, by updating the tracking state information STATE and the setting of the tracking subject through a control process described later in FIG. 9, it becomes possible to switch between WS100 and EB200 to control sub camera 400. In addition, the pan / tilt values ​​are transmitted only by one of the devices controlling the sub-camera 400, and are not transmitted when the other device is in control, thereby making it possible to reduce the amount of communication compared to the case where the pan / tilt values ​​are transmitted for each process in Figures 4(a) and (b).

[0128] Next, the operation of the overhead camera 300 when a shooting command is received from the WS 100 will be described with reference to FIG.

[0129] In step S301, the control unit 301 receives a shooting command from the WS 100 via the communication unit 305, and the process proceeds to step S302.

[0130] In step S302, the control unit 301 starts an image capturing process in response to receiving an image capturing command via the communication unit 305, and proceeds to step S303. The control unit 301 captures an image using the imaging unit 306, and stores the generated image data in the volatile memory 302 after performing predetermined image processing using the image processing unit 307.

[0131] In step S 303 , the control unit 301 reads the image data from the volatile memory 302 , and transmits it to the WS 100 via the communication unit 305 .

[0132] The operation of the overhead camera 300 has been described above.

[0133] Next, the operation of the sub-camera 400 that receives a control command from the WS 100 or the EB 200 will be described with reference to FIG.

[0134] In step S401, the control unit 401 receives a control command via the communication unit 405, stores the control command in the volatile memory 402, and advances the process to step S402.

[0135] In step S402, the control unit 401 reads out the pan / tilt values ​​from the volatile memory 402 in response to receiving a control command from the communication unit 405, and advances the process to step S403.

[0136] In step S403, the control unit 401 calculates drive parameters for controlling the pan operation / tilt operation at a desired speed in a desired direction based on the pan value / tilt value read from the nonvolatile memory 403, and proceeds to step S404. The drive parameters are values ​​for controlling the actuators for the pan / tilt directions included in the PTZ drive unit 409. The pan value / tilt value included in the control command is converted into the drive parameters by referring to a conversion table stored in the nonvolatile memory 403.

[0137] In step S404, the control unit 401 controls the optical unit 408 by the PTZ driving unit 409 based on the driving parameters obtained in step S403, and changes the shooting direction of the sub camera 400. The PTZ driving unit 409 changes the shooting direction of the sub camera 400 by driving the optical unit 408 in the pan / tilt direction based on the driving parameters.

[0138] The operation of the sub-camera 400 has been described above.

[0139] Next, the control process of the WS 100 will be described with reference to FIG.

[0140] FIG. 9(a) shows the control process of the WS 100, and shows the detailed process of step S105 in FIG. 4(a).

[0141] Part of the processing in FIG. 9(a) is realized by the control unit 101 executing the function of the tracking state determination unit 126 in FIG.

[0142] The tracking state determination unit 126 has a function of updating the tracking state information STATE stored in the volatile memory 102 .

[0143] In step S110, the control unit 101 reads out from the volatile memory 102 the SUBJECT_ID of the tracking subject of the sub camera 400 calculated in step S104 in Fig. 4A and the identification information ID indicating the previous tracking subject. Then, the control unit 101 compares the identification information read out from the volatile memory 102 to determine whether or not the tracking subject of the sub camera 400 has changed. If the control unit 101 determines that the tracking subject of the sub camera 400 has changed, the process proceeds to step S111, and if the control unit 101 determines that the tracking subject has not changed, the process proceeds to step S113.

[0144] In step S111, the control unit 101 transmits a tracking stop command to the EB 200 via the communication unit 105, and the process proceeds to step S112.

[0145] In step S112, the control unit 101 executes the function of the tracking state determination unit 126 in FIG. 3, and changes the tracking state information STATE to "tracking by WS 100".

[0146] When the tracking subject of the sub camera 400 is changed, there is a high possibility that the tracking subject is not present within the shooting angle of view of the sub camera 400. In this case, by performing the processes of steps S111 and S112, the WS 100 controls the sub camera 400 based on the overhead image of the overhead camera 300 instead of the sub camera 400.

[0147] In step S113, the control unit 101 reads the tracking state information STATE from the volatile memory 102, and determines whether "tracking by WS100" or "tracking by EB200" based on the tracking state information STATE. If the control unit 101 determines that "tracking by WS100" is true, the process proceeds to step S117, and if the control unit 101 determines that "tracking by EB200" is true, the process proceeds to step S114.

[0148] In step S114, the control unit 101 transmits a tracking continuation confirmation request to the EB200 via the communication unit 105, and inquires whether the EB200 can continue tracking the tracked subject. The response from the EB200 is "Continue tracking OK" or "Continue tracking NG." If the control unit 101 receives a notification of "Continue tracking OK" from the EB200, the process returns to step S101, and if the control unit 101 receives a notification of "Continue tracking NG" from the EB200, the process proceeds to step S115.

[0149] In step S115, the control unit 101 transmits a tracking stop command to the EB 200 via the communication unit 105, and the process proceeds to step S116.

[0150] In step S116, the control unit 101 executes the function of the tracking state determination unit 126 in FIG. 3, updates the tracking state information STATE to "tracking by WS 100", and ends the process.

[0151] By performing the processes from steps S114 to S116, when the tracking state is "tracking by EB200", tracking can be continued by WS100 even if the EB200 becomes unable to perform tracking.

[0152] In step S117, the control unit 101 determines whether or not the tracking subject is present within the shooting angle of view of the sub camera 400. If the control unit 101 determines that the tracking subject is present within the shooting angle of view of the sub camera 400, the control unit 101 advances the process to step S118. If the control unit 101 determines that the tracking subject is not present within the shooting angle of view of the sub camera 400, the control unit 101 ends the process. Whether or not the tracking subject is present within the shooting angle of view of the sub camera 400 can be determined by comparing the current pan value / tilt value acquired by the control unit 101 from the sub camera 400 with the new pan value / tilt value calculated in step S107 of FIG. 4(a). If the current pan value / tilt value is sufficiently close to the new pan value / tilt value, it can be determined that the tracking subject is present within the shooting angle of view of the sub camera 400. Alternatively, if the pan / tilt speed value calculated in step S108 is sufficiently small, it can be determined that the tracked subject is present within the shooting angle of view of the sub camera 400 because the current pan / tilt value is approaching the new pan / tilt value.

[0153] In step S118, the control unit 101 executes the function of the feature information determining unit 125 in FIG. 3, and advances the process to step S119.

[0154] The feature information determination unit 125 has a function of determining feature information of a subject being tracked by the sub camera 400, that is, feature information of the subject to be transmitted to the EB 200. The feature information determination unit 125 reads out feature information STAT[n] of the subject detected by the image recognition unit 121 from the overhead image of the overhead camera 300 from the volatile memory 102. The feature information determination unit 125 also reads out identification information SUBJECT_ID of the subject being tracked determined by the tracking target determination unit 123 from the volatile memory 102. Then, the feature information determination unit 125 determines feature information STAT[i] corresponding to the subject being tracked from the feature information STAT[n], and stores it in the volatile memory 102. i is an index indicating the subject being tracked.

[0155] In step S119, the control unit 101 transmits a tracking start command and feature information STAT[i] of the tracked subject to the EB 200 via the communication unit 105, and the process proceeds to step S120.

[0156] By the processing of steps S117 to S119, a tracking start command and feature information of the tracking subject can be transmitted to the EB200 only when there is a high possibility that the tracking subject is present within the shooting angle of view of the sub-camera 400. This makes it possible to reduce the amount of communication compared to the case where information is transmitted for each process of Fig. 4(a) and Fig. 9(a).

[0157] In step S120, control unit 101 receives the comparison result of the subject from EB 200 via communication unit 105. If control unit 101 receives match information from EB 200 indicating that the subjects match, the process proceeds to step S121, and if control unit 101 receives mismatch information indicating that the subjects do not match, the process ends.

[0158] In step S121, the control unit 101 executes the function of the tracking state determination unit 126 in FIG. 3, changes the tracking state information STATE to "Tracking by EB200", and ends the process.

[0159] Next, the control process of the EB 200 will be described with reference to FIGS.

[0160] FIG. 9B shows the control process of the EB 200, and shows the detailed process of step S203 in FIG.

[0161] In step S210, control unit 201 determines whether or not it has received from WS 100, via communication unit 205, a tracking start command and feature information STAT[i] of the tracking subject obtained from the overhead image of overhead camera 300. If control unit 201 has received the tracking start command and feature information STAT[i] of the tracking subject from WS 100, it proceeds to step S211;

[0162] In steps S211 to S214, the control unit 201 executes the function of the tracking target determination unit 222 in Fig. 3. The control unit 201 determines whether or not the feature information STAT[i] received from the WS 100 and the feature information STAT_SUB[m] obtained from the sub-image of the sub-camera 400 satisfy a predetermined condition.

[0163] The tracking target determination unit 222 has a function of calculating a similarity between the feature information STAT[i] received from the WS 100 and the feature information STAT_SUB[m] obtained from the sub-image of the sub-camera 400. The tracking target determination unit 222 also has a function of comparing the similarity of the feature information with a threshold value stored in the volatile memory 102, and storing the comparison result in the volatile memory 102. For example, when two people are present in the sub-image of the sub-camera 400, the tracking target determination unit 222 calculates a similarity between the feature information (STAT_SUB[1], STAT_SUB[2]) of the two people and the feature information STAT[i] received from the WS 100. The similarity is calculated as a cosine similarity between the feature information vectors, and a value between 1 and 0 is obtained as the similarity. The control unit 201 stores the similarities calculated for the m subjects in the volatile memory 202.

[0164] In step S211, the control unit 201 executes the function of the tracking target determination unit 222 in FIG. 3 to perform a comparison process of the feature information, and proceeds to step S212.

[0165] In step S212, control unit 201 determines whether or not a subject with highly similar feature information exists, based on the comparison result in step S211. The existence of a subject with highly similar feature information means that the same subject is photographed by overhead camera 300 and sub camera 400. If control unit 201 determines that a subject with highly similar feature information exists, control unit 201 proceeds to step S214, and if control unit 201 determines that a subject with highly similar feature information does not exist, control unit 201 proceeds to step S213.

[0166] The control unit 201 reads out a predetermined threshold value from the volatile memory 202, and when the similarity is equal to or greater than the threshold value as a predetermined condition, it determines that a subject with high similarity in feature information exists, and stores the identification information ID of the subject in the volatile memory 202. The predetermined condition is not limited to when the similarity is equal to or greater than the threshold value, but may be when a subject with higher similarity exists, or when the subjects match. The control unit 201 also updates information MATCH indicating whether or not a subject with high similarity in feature information exists, and stores the information in the volatile memory 202. In this embodiment, if the value of MATCH is 0, there is no subject with high similarity in feature information, that is, the subjects do not match between the overhead camera 300 and the sub camera 400. If the value of MATCH is 1, there is a subject with high similarity in feature information, that is, the subjects match between the overhead camera 300 and the sub camera 400. If there is a subject with high similarity in feature information, the control unit 201 stores MATCH=1 in the volatile memory 202, and proceeds to step S214. If there is no subject with a high similarity in feature information, the control unit 201 stores MATCH=0 in the volatile memory 202, and the process proceeds to step S213.

[0167] Here, the similarity of feature information of a subject detected from the overhead image captured by the overhead camera 300 and the sub-image captured by the sub camera 400 will be described with reference to FIG.

[0168] Fig. 10(a) illustrates an example of the positional relationship between the shooting position and shooting direction of the overhead camera 300 and the shooting position and shooting direction of the sub camera 400. Fig. 10(b) illustrates an example of a subject detected from an overhead image of the overhead camera 300 and a tracked subject.

[0169] Assume that subjects A, B, and C are detected from the overhead image of overhead camera 300, and the subject being tracked by sub camera 400 is subject C. The feature information of the subject being tracked by sub camera 400 transmitted from sub camera 400 to WS 100 is information corresponding to subject C. Figures 10(c) and (e) show the sub-images of sub camera 400, and Figures 10(d) and (f) show the similarity between the feature information of the subject being tracked by sub camera 400 and the feature information of the subject detected from the sub-image.

[0170] As shown in Fig. 10(c), when the sub camera 400 is photographing subjects A and B, the similarity is calculated between the feature information of subject C in the overhead image of the overhead camera 300 and each feature information of subject A or subject B in the sub image of the sub camera 400. As shown in Fig. 10(d), the similarity is low between the feature information of subject C in the overhead image of the overhead camera 300 and the feature information of subject A or subject B in the sub image of the sub camera 400. In this case, for example, when the threshold value of the subject similarity is 0.7, the result is that both subjects A and B do not match.

[0171] Also, as shown in FIG. 10(e), when the sub camera 400 captures subjects B and C, the similarity between the feature information of subject C in the overhead image of the overhead camera 300 and the feature information of subject B or subject C in the sub image of the sub camera 400 is calculated. The subject C in the overhead image of the overhead camera 300 and the subject C in the sub image of the sub camera 400 have different forms in the images because the camera shooting positions and shooting directions are different. For example, when the subject C faces or faces the body toward the overhead camera 300, the subject C faces forward in the overhead image of the overhead camera 300 and is closer to sideways in the sub image of the sub camera 400. The inference model for identifying the subject in the image recognition unit 121 of the WS100 and the image recognition unit 221 of the EB200 is a model that learns images of the same subject captured from multiple different directions. Therefore, even if the same subject is photographed by multiple cameras with different photographing positions and directions and has a different form in each photographed image, the similarity of the feature information is high. That is, as shown in Fig. 10(f), the similarity of the feature information of subject C in the overhead image of overhead camera 300 and the feature information of subject C in the sub-image of sub camera 400 is high. As a result, for example, when the threshold value of the subject similarity is 0.7, subject B will be determined to be a mismatch and subject C will be determined to be a match, and subject C can be determined to be the same subject.

[0172] Returning to the explanation of FIG. 9B, in step S213, the control unit 201 reads MATCH=0 from the volatile memory 202, transmits it to the WS 100 via the communication unit 205, and ends the process.

[0173] In step S214, the control unit 201 reads out the identification information ID of the subject for which the highest similarity has been calculated from the volatile memory 102, stores it in the volatile memory 102 as the tracking subject SUBJECT_ID, and proceeds to step S215. By selecting the subject for which the highest similarity has been calculated, even if there are subjects wearing similar clothes, for example, it is possible to set the most likely subject among them as the tracking target.

[0174] In step S215, the control unit 201 reads MATCH=1 from the volatile memory 202, transmits it to the WS 100 via the communication unit 205, and ends the process.

[0175] FIG. 9C shows the control process of the EB 200, and shows the detailed process of step S204 in FIG.

[0176] In step S220, the control unit 201 determines whether or not a tracking stop command has been received from the WS 100 via the communication unit 205. If a tracking stop command has been received from the WS 100, the control unit 201 proceeds to step S221, and if not, the control unit 201 proceeds to step S223.

[0177] In step S221, the control unit 201 transmits a control command to stop the panning / tilting operation to the sub-camera 400 via the communication unit 205, and the process proceeds to step S222.

[0178] In step S222, the control unit 201 deletes the tracking subject SUBJECT_ID stored in the volatile memory 202, and the process proceeds to step S201.

[0179] In step S223, the control unit 201 determines whether or not a tracking continuation confirmation request has been received from the WS 100 via the communication unit 205. If a tracking continuation confirmation request has been received from the WS 100, the control unit 201 proceeds to step S224, and if not, the control unit 201 ends the process.

[0180] In step S224, the control unit 201 reads out the subject recognition result by the image recognition unit 221 from the volatile memory 202, and determines whether or not the tracking subject SUBJECT_ID is detected. If the control unit 201 determines that the tracking subject SUBJECT_ID is detected by the image recognition unit 221, the process proceeds to step S226, and if not, the process proceeds to step S225.

[0181] In step S225, the control unit 201 transmits "Continue tracking NG" to the WS 100 via the communication unit 205, and the process returns to step S201.

[0182] In step S226, the control unit 201 transmits "Continue tracking OK" to the WS 100 via the communication unit 205, and ends the process.

[0183] The above is the detailed control process of the EB200.

[0184] According to the above-described control process, the same subject can be recognized by the multiple cameras 300 and 400 with different shooting positions and shooting directions. Therefore, a specific subject can be tracked by appropriately switching between the control of the sub-camera 400 by the WS 100 and the control of the sub-camera 400 by the EB 200.

[0185] When the subject to be tracked does not exist in the sub-image of the sub-camera 400, the WS 100 controls the sub-camera 400, and when the subject to be tracked exists in the shooting angle of view of the sub-camera 400, the control of the sub-camera 400 can be handed over from the WS 100 to the EB 200. Also, when the subject to be tracked moves at high speed and is lost, or when the subject to be tracked is changed, tracking can be continued by controlling the sub-camera 400 with the WS 100.

[0186] In the above-mentioned control process, an example has been described in which WS100 and EB200 switch whether or not to transmit pan / tilt values ​​to the sub-camera 400, but the present invention is not limited to this example. For example, the pan / tilt values ​​may be transmitted from WS100 and EB200 to the sub-camera 400 regardless of the tracking state, and the sub-camera 400 may control which device the pan / tilt values ​​received from to perform the pan / tilt operation. In this case, the process of WS100 may omit the process of step S106 in FIG. 4(a), and before the process of step S107 in FIG. 4(a), the control unit 101 may add a process of transmitting tracking state information STATE to the sub-camera 400. The process of EB200 may omit the processes of steps S205 and S206 in FIG. 4(b) and step S221 in FIG. 9(c).

[0187] When the tracking state information STATE received from WS100 is "tracking by EB200", the sub-camera 400 controls to perform a panning operation / tilting operation in response to a control command received from EB200. When the tracking state information STATE received from WS100 is "tracking by WS100", the sub-camera 400 controls to perform a panning operation / tilting operation in response to a control command received from WS100.

[0188] Next, a process in which the EB 200 acquires position information of a predetermined part of a tracking subject from a sub-image will be described with reference to Fig. 11. The process of acquiring head position information of a person will be described below.

[0189] FIG. 11 shows the image recognition process of the EB 200, and shows detailed processing of step S202 in FIG.

[0190] In step S301, the control unit 201 executes image recognition processing by the image recognition unit 221 based on the sub-image read from the volatile memory 202 and the first trained model read from the non-volatile memory 203. The control unit 201 acquires position information POSITION and identification information ID of the subject in the sub-image as the image recognition result, stores them in the volatile memory 202, and proceeds to the process of step S302.

[0191] In step S302, the control unit 201 corrects at least one of the position information POSITION of the subject corresponding to the tracking subject SUBJECT_ID read from the volatile memory 202 and the sub-image of the sub camera 400. The method of correcting the subject position information POSITION and the sub-image will be described later. The control unit 201 saves the corrected subject position information POSITION_CORR and the corrected sub-image in the volatile memory 202, and proceeds to the process of S303.

[0192] In step S303, the control unit 201 acquires the subject's skeletal position information KP based on the second learned model for skeletal position estimation read from the non-volatile memory 203, the sub-image or the corrected sub-image read from the volatile memory 202, and the subject's position information POSITION or the corrected subject's position information POSITION_CORR.

[0193] Here, with reference to FIG. 12, a trained model for estimating a skeleton position according to this embodiment will be described.

[0194] 12(a) and (b) show input and output data of the trained model for estimating a skeleton position in this embodiment.

[0195] Fig. 12(a) shows a schematic diagram of the sub-image and subject position information input to the trained model for skeletal position estimation, and the subject included in the sub-image. Fig. 12(b) shows an example of the skeleton position information KP output from the trained model for skeletal position estimation.

[0196] In this embodiment, the sub-image shown in Fig. 12(a) and position information POSITION of the subject for which skeletal position estimation is performed are input to the trained model for skeletal position estimation. In the example of Fig. 12(a), to estimate the skeletal position of subject B, the sub-image and position information POSITION_B of subject B are input to the trained model for skeletal position estimation.

[0197] FIG. 12(b) illustrates an example of skeleton position information KP output from a trained model for estimating a skeleton position using the sub-image in FIG. 12(a) and position information POSITION_B of subject B as input.

[0198] In the example of FIG. 12(b), the position information POSITION_B of the subject B is input and the skeletal position information KP of the subject B is output from the trained model for skeletal position estimation. The skeletal position information KP includes the top of the head KP_HEAD, the neck KP_NECK, the right shoulder KP_RSHOULDER, the left shoulder KP_LSHOULDER, the right hand KP_RHAND, the left hand KP_LHAND, the right base of the foot KP_RLEG, and the left base of the foot KP_LREG. Note that the types and number of the skeletal position information KP are not limited to these, and skeletal position information of more detailed parts of the subject may be output. For example, the trained model for skeletal position estimation may output more detailed skeletal position information of the head, such as both eyes, nose, and both ears. This makes it possible to estimate the horizontal center of gravity position of the position information of both ears as the head position of the subject, even if the top of the head KP_HEAD of the subject is not located in the image. Conversely, if the tracking subject can be identified, the number of skeletal position information may be reduced. For example, if the skeletal position information of the top of the head and the neck can be estimated, the face of the tracking subject can be identified. By reducing the amount of skeletal position information in this way, it is expected that the load required for calculation processing can be reduced. In addition, in the present embodiment, an example in which a trained model for skeletal position estimation is used has been described, but a trained model capable of estimating the position information of the subject's area may also be used. For example, a model that estimates the head position information of the subject in an image by approximating it with a circumscribing rectangle, or a segmentation model that can estimate the head position information at the pixel level may be used.

[0199] Next, with reference to Figures 12(c) and (d), we will explain the issues with image recognition processing using a trained model for skeletal position estimation.

[0200] Fig. 12(c) shows a schematic diagram of a sub-image and subject position information POSITION input to a trained model for skeletal position estimation, and the subjects included in the sub-image. In the example of Fig. 12(c), a scene is shown in which two subjects A and B are close to each other in the sub-image, and the subject whose skeletal position is to be estimated is subject B.

[0201] Fig. 12(d) illustrates an example of skeletal position information KP output from a trained model for skeletal position estimation using as input the sub-image in Fig. 12(c) and position information POSITION_B of subject B. For simplicity of explanation, Fig. 12(d) illustrates only the skeletal position information of the top of the head KP_HEAD and the neck KP_NECK related to the head of the tracked subject, which are referenced when controlling the panning / tilting operations of the sub-camera 400.

[0202] Since subjects A and B are included within the rectangular area of ​​position marker information POSITION_B of subject B, even though position information POSITION_B of subject B is input to the trained model for skeletal position estimation, skeletal position information of subject A may be output. Therefore, in this embodiment, at least one of the sub-image and position information POSITION of subject B is corrected in step S302 in Fig. 11. Furthermore, at step S303 in Fig. 11, skeletal position information KP of the tracked subject is estimated based on at least one of the corrected sub-image and the corrected position information POSITION_CORR of the subject.

[0203] In step S303, control unit 201 inputs the corrected sub-image into a trained model for skeletal position estimation, and executes image recognition processing by image recognition unit 221 to obtain skeletal position information KP of the subject corresponding to the identification information ID of the tracked subject. Control unit 201 stores skeletal position information KP of the subject corresponding to the identification information ID of the tracked subject in volatile memory 202, and proceeds to step S304.

[0204] In step S304, control unit 201 reads out skeletal position information KP of the subject's head corresponding to the identification information ID of the tracking subject from non-volatile memory 203, and estimates the position of the subject's head. In this embodiment, control unit 201 reads out the top of the head KP_HEAD and the neck KP_NECK as skeletal position information of the subject corresponding to the identification information ID of the tracking subject from volatile memory 202. Then, control unit 201 calculates the center of gravity positions of the top of the head and the neck to calculate the head position POS_HEAD of the subject, and stores it in volatile memory 202.

[0205] Next, the process of correcting the sub-image and the position information of the subject in step S302 in FIG. 11 will be described with reference to FIGS.

[0206] FIG. 13(a) illustrates the correction process of the position information POSITON of the sub-image and the subject, and FIG. 13(b) illustrates the correction process of the area of ​​the subject to be corrected in step S409 of FIG. 13(a). Note that the example of FIG. 13 is a process that is executed after identifying the area of ​​the subject to be corrected, so the area corresponding to the position information POSITION of the subject to be corrected is BOXtarget, area information corresponding to the corrected position information POSITION_CORR of the subject to be corrected is BOXtarget_CORR, and the area corresponding to the position information POSITION of other subjects other than the subject to be corrected is BOXi. The subscript i in BOXi represents the subject position number, and for example, when the positions of two subjects other than the subject to be corrected can be obtained, BOXi is assigned as BOX1 and BOX2. In addition, in this embodiment, the subscript i is assumed to be smaller as the subject is located vertically higher.

[0207] FIG. 13(a) shows a process of correcting the position information of the sub-image and the subject to be corrected.

[0208] In step S401, the control unit 201 reads the area BOXtarget of the subject to be corrected and the area BOXi of the other subject from the volatile memory 202, and calculates an intersection over Target-box between them. The intersection over Target-box (IoT) is an index indicating the degree to which the area BOXtarget of the subject to be corrected encompasses the area BOXi of the other subject.

[0209] Here, a method for calculating the inclusion degree IoT will be described with reference to FIG.

[0210] 14(a) shows an example of a state where the areas of two subjects detected from a sub-image are partially overlapping. The inclusion degree IoT of the area BOXtarget of the subject to be corrected in the area BOXi of another subject can be calculated from the following formula 5, where the overlapping area between the two is Intersection and the area of ​​the area of ​​the other subject is AREA(BOXi). (Formula 5) In the example of FIG. 14(a) in TIFF2025060252000006.tif13155, the overlapping area Intersection is the area shaded diagonally downward to the right, and the area of ​​the other subject is the area indicated by the sum of the area shaded diagonally upward to the right and the area shaded diagonally downward to the right. The inclusion degree IoT is calculated as a larger value the greater the degree of overlap of the multiple subject areas, and is calculated as 1.0 when the area BOXi of the other subject is completely included in the area BOXtarget of the subject to be corrected. Also, when there is no overlap at all between the area BOXi of the other subject and the area BOXtarget of the subject to be corrected, the value of IoT is calculated as 0.

[0211] The control unit 201 calculates the degree of inclusion IoT of the area BOXi of the other object in the area BOXtarget of the object to be corrected, stores it in the volatile memory 202, and advances the process to step S402.

[0212] In step S402, the control unit 201 reads the area BOXtarget of the subject to be corrected and the area BOXi of the other subject from the volatile memory 202, and calculates the degree of overlap IoU between them. The degree of overlap IoU (Intersection over Union) is an index indicating the degree to which the area BOXi of the other subject overlaps with the area BOXtarget of the subject to be corrected.

[0213] Here, a method for calculating the degree of overlap IoU will be described with reference to FIG.

[0214] 14(b) illustrates an example of a state in which two objects detected from a sub-image are partially overlapping. The degree of overlap IoU of the area BOXi of another object with respect to the area BOXtarget of the object to be corrected can be calculated from the following formula 6, where the union (OR area) of the two is BOXi∪BOXtarget. (Formula 6) In the example of FIG. 14(b), the OR area Intersection is the area shaded diagonally downward to the right, and the area of ​​the other subject is the area indicated by the sum of the area shaded diagonally upward to the right and the area shaded diagonally downward to the right. When the area BOXi of the other subject completely overlaps with the area BOXtarget of the subject to be corrected, the IoU is calculated as 1.0. When the area BOXi of the other subject does not overlap with the area BOXtarget of the subject to be corrected at all, the IoU value is calculated as 0.

[0215] The control unit 201 calculates the degree of overlap IoU of the area BOXi of another object relative to the area BOXtarget of the object to be corrected, stores it in the volatile memory 202, and the process proceeds to step S403.

[0216] In step S403, the control unit 201 reads the inclusion degree IoT from the volatile memory 202 and compares it with a predetermined threshold value THiot. In this embodiment, the threshold value THiot of the inclusion degree is set to 0.95, but it is not limited to this, and other values may be experimentally obtained. When the inclusion degree IoT is greater than the threshold value THiot (IoT>THiot), the control unit 201 determines that the region BOXtarget of the subject to be corrected includes the region BOXi of another subject. Then, the control unit 201 advances the process to step S406. Also, when the inclusion degree IoT is less than or equal to the threshold value THiot (IoT≦THiot), the control unit 201 determines that the region BOXtarget of the subject to be corrected does not include the region BOXi of another subject, and advances the process to step S404.

[0217] In step S404, the control unit 201 reads the overlapping degree IoU read from the volatile memory 202 and compares it with a predetermined threshold value THiou. In this embodiment, the threshold value THiou of the overlapping degree is set to 0.4, but it is not limited to this, and other values may be experimentally obtained. When the overlapping degree IoU is less than the threshold value THiou (IoU<THiou), the control unit 201 determines that the region BOXtarget of the subject to be corrected includes the region BOXi of another subject and the size of the other subject is smaller than the size of the subject to be corrected, and advances the process to step S405. Also, when the overlapping degree IoU is greater than or equal to the threshold value THiou (IoU≧THiou), the control unit 201 determines that the region BOXtarget of the subject to be corrected includes the region BOXi of another subject and the size of the other subject is larger than the size of the subject to be corrected, and advances the process to step S407.

[0218] In step S405, the control unit 201 performs a masking process on the region BOXi of the other subject in the sub-image to correct the sub-image, saves the corrected sub-image in the volatile memory 202, and advances the process to step S410.

[0219] Here, with reference to FIGS. 15(a) and (b), the masking process of the other subject region will be described.

[0220] Fig. 15(a) shows two subjects A and B in a sub-image, and the area BOXtarget of subject A and the area BOXi of subject B. The area BOXtarget of subject A to be corrected and the area BOXi of another subject B correspond to position information POSITION of the subjects acquired by image recognition unit 221 in step S301. Also, in Figs. 15(a) and 15(b), the position information POSITION of subject B is shown as the area BOXtarget of the subject to be corrected. Also, the position information POSITION of subject A is shown as areas BOX1 and BOX2 of the other subjects separated above and below by the arm of subject B.

[0221] In this embodiment, the inference model for subject detection assumes that the sub-image is divided into regions at multiple scales and subject detection results are output for each region, and an example is described in which a small-sized region of subject A in the sub-image is output as BOX1 and BOX2.

[0222] The control unit 201 applies mask processing to areas BOX1 and BOX2 of subject A that are included in area BOXtarget of subject B to be corrected and that are smaller in size in the sub-image than subject B. FIG. 15(b) shows the corrected sub-image that has been subjected to mask processing, and skeletal position information KP that is output from a trained model for skeletal position estimation using the corrected sub-image as input. Note that, for simplicity of explanation, FIG. 15(b) shows only skeletal position information of the top of the head KP_HEAD and neck KP_NECK related to the head of subject B to be corrected.

[0223] Next, the control unit 201 performs mask processing on the areas BOX1 and BOX2 of the subject A shown in FIG. 15(b). The mask processing is, for example, processing to fill the areas BOX1 and BOX2 of the subject A in the sub-image with gray gradations. The mask processing is not limited to this, and other colors and methods may be used as long as they can mask the characteristic information of the subject to be corrected and other subjects in the sub-image. For example, the area to be filled may be filled with the average pixel value for each RGB of the area to be filled, or may be filled with a predetermined color. In the example of FIG. 12(d), if the mask processing is not performed on the subject A, the skeletal position information KP of the subject A may be estimated, but by performing the mask processing on the subject A as in FIG. 15(b), it becomes possible to appropriately estimate the skeletal position information KP of the subject B.

[0224] In this embodiment, an example has been described in which the area of ​​subject A is detected as areas BOX1 and BOX2 separated vertically by the arms of subject B, but the present invention is also applicable to cases in which areas BOX1 and BOX2 of subject A are detected in a connected state. This is because it is possible to calculate the corrected area information BOXtarget_CORR of the subject to be corrected in accordance with the skeletal position information of the top of the head KP_HEAD and neck KP_NECK to be obtained from subject B.

[0225] The control unit 201 stores the masked and corrected sub-image in the volatile memory 202, and the process proceeds to step S410.

[0226] In step S406, control unit 201 reads out the overlap degree IoU from volatile memory 202, and determines whether it is greater than 0. If the overlap degree IoU is not 0 (IOU>0), control unit 201 determines that area BOXi of another subject partially overlaps area BOXtarget of the subject to be corrected. Then control unit 201 advances the process to step S407. If the overlap degree IoU is 0 (IOU=0), control unit 201 determines that area BOXi of another subject does not overlap area BOXtarget of the subject to be corrected at all, and advances the process to step S410.

[0227] In step S407, the control unit 201 calculates the area Area(BOXtarget) of the region BOXtarget of the subject to be corrected and the area Area(BOXtarget) of the region BOX i of other subjects, stores them in the volatile memory 202, and advances the process to step S408.

[0228] In step S408, the control unit 201 compares the area Area(BOXtarget) of the region of the subject to be corrected read from the volatile memory 202 with the area Area(BOXtarget) of the region of other subjects. When the area Area(BOX) of the region of other subjects is smaller than the area Area(BOXtarget) of the region of the subject to be corrected (Area(BOX) < Area(BOXtarget)), the control unit 201 determines that the subject to be corrected is located in front of other subjects (closer to the sub-camera 400), and advances the process to step S409. Further, when the area Area(BOX) of the region of other subjects is equal to or larger than the area Area(BOXtarget) of the region of the subject to be corrected (Area(BOX) ≧ Area(BOXtarget)), the control unit 201 determines that the subject to be corrected is located behind other subjects (farther from the sub-camera 400), and advances the process to step S410. In the present embodiment, the positional relationship between the subjects A and B is estimated based on the area Area(BOX) of the region of other subjects and the area Area(BOXtarget) of the region of the subject to be corrected, but it is not limited thereto. For example, the sub-camera 400 includes a focus detection unit, and transmits a distance map, which is a focus detection (distance measurement) result corresponding to each pixel of the sub-image, to the EB200, and the EB200 stores the distance map in the volatile memory 202. Then, the control unit 201 can grasp the positional relationship between the subjects A and B by reading the distance map from the volatile memory 202 and referring to the focus detection results corresponding to the region BOXtarget of the subject to be corrected and the region BOX i of other subjects.

[0229] In step S409, the control unit 201 executes a correction process for the region BOXtarget of the object to be corrected that has been read from the volatile memory 202. The control unit 201 stores the corrected region information BOXtarget_CORR of the object to be corrected in the volatile memory 202, and the process proceeds to step S410.

[0230] FIG. 13B shows the correction process for the area BOXtarget of the subject to be corrected in step S409 in FIG. 13A.

[0231] In this embodiment, at least one of the sub-image and the area BOXtarget of the subject to be corrected is corrected so that the area BOXtarget of the subject to be corrected does not include the area BOXi of the other subject. In more detail, a mask process is performed on the area BOXi of the other subject included in the area BOXtarget of the subject to be corrected in the sub-image. Also, the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected is moved in the opposite direction to the direction in which the center of gravity GRAVITYi of the area BOXi of the other subject is located relative to the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected.

[0232] In step S421, the control unit 201 calculates the center of gravity GRAVITYtarget of the region BOXtarget of the subject to be corrected and the center of gravity GRAVITYi of the region BOXi of the other subject, and stores them in the volatile memory 202. The control unit 201 calculates the center of gravity GRAVITYtarget of the region BOXtarget of the subject to be corrected by the average of the horizontal positions of the right and left sides of the region BOXtarget of the subject to be corrected and the average of the vertical positions of the upper and lower sides. Similarly, the control unit 201 calculates the center of gravity GRAVITYi of the region BOXi of the other subject by the average of the horizontal positions of the right and left sides of the region BOXi of the other subject and the average of the vertical positions of the upper and lower sides. The control unit 201 stores the center of gravity GRAVITYtarget of the region BOXtarget of the subject to be corrected and the center of gravity GRAVITYi of the region BOXi of the other subject in the volatile memory 202, and proceeds to step S422.

[0233] In step S422, the control unit 201 compares the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected read from the volatile memory 202 with the center of gravity GRAVITYi of the area BOXi of the other subject. If the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected is larger than the center of gravity GRAVITYi of the area BOXi of the other subject (GRAVITYtarget>GRAVITYi), the control unit 201 performs a correction process of the position information so that the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected is shifted to the right, and the process proceeds to step S423. If the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected is equal to or smaller than the center of gravity GRAVITYi of the area BOXi of the other subject (GRAVITYtarget≦GRAVITYi), the control unit 201 performs a correction process of the position information so that the center of gravity GRAVITYtarget of the area BOXtarget of the subject to be corrected is shifted to the left, and the process proceeds to step S424.

[0234] In step S423, the control unit 201 reads out the area BOXtarget of the object to be corrected and the areas BOXi of the other objects from the volatile memory 202. Then, the control unit 201 calculates post-correction area information BOXtarget_CORR of the object to be corrected, stores it in the volatile memory 202, and ends the process.

[0235] Here, the correction process for the area BOXtarget of the subject to be corrected will be described with reference to FIGS.

[0236] 15(c) shows two subjects A and B in a sub-image, an area BOXtarget of subject A, and an area BOXi of subject B. The control unit 201 performs a correction process for the area BOXtarget of subject B based on an area BOX1 of subject A that partially overlaps with the area BOXtarget of subject B to be corrected.

[0237] FIG. 15D is a diagram illustrating the result of correcting the area BOXtarget of the subject to be corrected.

[0238] Step S423 is a case where the center of gravity GRAVITYtarget of the region BOXtarget of the subject to be corrected is larger than the center of gravity GRAVITYi of the region BOXi of another subject (GRAVITYtarget>GRAVITYi). In this case, the control unit 201 determines that the tracking target is located on the right side of the region BOXtarget of the subject to be corrected, and calculates the corrected region information BOXtarget_CORR of the subject to be corrected. In the example of Figs. 15(c) and (d), the control unit 201 calculates the corrected region information BOXtarget_CORR of the subject B by replacing the position of the left side of the region BOXtarget of the subject B with the position of the right side of the region BOX1 of the subject A, and stores it in the volatile memory 202. As a result, as shown in Fig. 15(d), it is possible to calculate the corrected region information BOXtarget_CORR of the subject B in which the head and torso of the subject B are arranged at the center. Then, by inputting the corrected region information BOXtarget_CORR of the subject B to the trained model for skeletal position estimation, it is possible to acquire the skeletal position information KP of the subject B so that the head of another subject A is not included.

[0239] In step S423, the control unit 201 reads out the area BOXtarget of the object to be corrected and the areas BOXi of the other objects from the volatile memory 202. Then, the control unit 201 calculates post-correction area information BOXtarget_CORR of the object to be corrected, stores it in the volatile memory 202, and ends the process.

[0240] Here, the correction process for the area BOXtarget of the subject to be corrected will be described with reference to FIGS.

[0241] The processing of step S424 is performed with the positional relationship between subjects A and B in Figure 15(c) reversed horizontally, so Figures 15(e) and (f) are like Figures 15(c) and (d) reversed horizontally.

[0242] 15(e) shows two subjects A and B in a sub-image, an area BOXtarget of subject A, and an area BOXi of subject B. The control unit 201 performs a correction process for the area BOXtarget of subject B based on an area BOX1 of subject A that partially overlaps with the area BOXtarget of subject B to be corrected.

[0243] FIG. 15F is a diagram illustrating the result of correcting the area BOXtarget of the subject to be corrected.

[0244] Step S424 is a case where the center of gravity GRAVITYtarget of the region BOXtarget of the subject to be corrected is equal to or smaller than the center of gravity GRAVITYi of the region BOXi of another subject (GRAVITYtarget≦GRAVITYi). In this case, the control unit 201 determines that the tracking target is located to the left of the region BOXtarget of the subject to be corrected, and calculates region information BOXtarget_CORR obtained by correcting the region BOXtarget of the subject to be corrected. In the example of FIG. 15(e) and (f), the control unit 201 calculates the corrected region information BOXtarget_CORR of the subject B by replacing the position of the right side of the region BOXtarget of the subject B with the position of the left side of the region BOX1 of the subject A, and stores it in the volatile memory 202. As a result, as shown in FIG. 15(f), it is possible to calculate the corrected region information BOXtarget_CORR of the subject B in which the head and torso of the subject B are arranged at the center. Then, by inputting the corrected region information BOXtarget_CORR of the subject B to the trained model for skeletal position estimation, it is possible to acquire the skeletal position information KP of the subject B so that the head of another subject A is not included.

[0245] In step S424, the control unit 201 reads out the area BOXtarget of the object to be corrected and the areas BOXi of the other objects from the volatile memory 202. Then, the control unit 201 calculates post-correction area information BOXtarget_CORR of the object to be corrected, stores it in the volatile memory 202, and ends the process.

[0246] In this embodiment, the method of replacing the position of the right side or the left side of the area BOXtarget of the subject s to be corrected with the position of the left side or the right side of the area BOXi of another subject in steps S423 and S424 has been described, but the present invention is not limited to this. For example, in step S424, the right side of the area BOXtarget of the subject to be corrected is shifted leftward by a predetermined ratio of the horizontal size of the area BOXtarget of the subject to be corrected. This makes it possible to correct the center of gravity of the tracking subject with respect to the area BOXtarget of the subject to be corrected.

[0247] In the present embodiment, an example has been described in which the control unit 201 calculates the corrected region information BOXtarget_CORR of the subject to be corrected, but the present invention is not limited to this. For example, when the input of the trained model for skeletal position estimation is a sub-image and center of gravity information of the subject whose skeletal position is to be estimated, the center of gravity information of the corrected region information BOXtarget_CORR of the subject to be corrected may be calculated. In this case, the control unit 201 inputs the sub-image and center of gravity information of the subject read from the volatile memory 202 to the trained model for skeletal position estimation, and it becomes possible to acquire the skeletal position information of the tracked subject.

[0248] In step S410, the control unit 201 determines whether or not the correction process shown in Fig. 13(a) has been performed on all of the areas BOXi of the other subjects read from the volatile memory 202. If the control unit 201 determines that the correction process has been performed on all of the areas BOXi of the other subjects, the control unit 201 ends the process. If the control unit 201 determines that the correction process has not been performed on all of the areas BOXi of the other subjects, the control unit 201 returns the process to step S401, reads the next BOXi from the volatile memory 202, and performs the same correction process.

[0249] According to the above-mentioned embodiment 1, at least one of the sub-image and the position information of the tracking subject is corrected so that among the multiple subjects included in the sub-image, other subjects are not included in the area of ​​the tracking subject. This makes it possible to obtain position information of a predetermined part (e.g., head) of the tracking subject based on the corrected sub-image and the corrected position information of the tracking subject. As a result, it becomes possible to obtain skeletal position information of the tracking subject such that the predetermined part of the tracking subject is located at a desired position (e.g., center of the screen) within the shooting angle of view of the sub-camera 400.

[0250] In this embodiment, the control unit 201 uses a trained model for estimating a skeletal position that acquires skeletal position information KP of the upper body of the subject to estimate the head position of the tracking subject, but is not limited to this. For example, a trained model that detects the head and face of the tracking subject by approximating them with a circumscribing rectangle may be used, or a trained model that acquires the positions of facial parts such as ears, nose, and eyes in more detail may be used.

[0251] [Embodiment 2] In the first embodiment, an example has been described in which either the WS 100 or the EB 200 controls the sub-camera 400. In the second embodiment, an example will be described in which the EB 200 is omitted and the WS 100 controls the sub-camera 400 based on the overhead image of the overhead camera 300 and the sub-image of the sub-camera 400.

[0252] In the second embodiment, the sub camera 400 is controlled using either the pan value / tilt value calculated based on the overhead image of the overhead camera 300 or the pan value / tilt value calculated based on the sub image of the sub camera 400.

[0253] 1 without the EB 200, and differs from the first embodiment in that a sub-image from the sub-camera 400 is input to the WS 100. Operations other than those of the WS 100 are the same as those of the first embodiment.

[0254] In basic operation, the overhead camera 300 transmits an overhead image to the WS 100. The sub-camera 400 transmits a sub-image to the WS 100. The sub-camera 400 also has a PTZ function.

[0255] The WS100 detects a subject from the overhead image of the overhead camera 300 and the sub-image of the sub-camera 400, and changes the imaging direction of the sub-camera 400 to the direction of the tracked subject based on the subject recognition result. Until the shooting direction of the sub-camera 400 becomes the direction of the tracked subject, the WS100 controls the sub-camera 400 based on the subject recognition result of the overhead image of the overhead camera 300. After the shooting direction of the sub-camera 400 becomes the direction of the tracked subject, the WS100 calculates feature information of the tracked subject from the overhead image of the overhead camera 300, and calculates feature information of the subject from the sub-image of the sub-camera 400. Then, the WS100 controls the sub-camera 400 based on these feature information. The feature information is information that can identify that the same subject is the same when the same subject is shot by multiple cameras with different shooting positions and / or shooting directions.

[0256] According to the second embodiment, the sub camera 400 can be controlled based on the subject recognition result of either the overhead image of the overhead camera 300 or the sub image of the sub camera 400, to track the tracking subject.

[0257] The hardware configurations of the WS 100, the overhead camera 300, and the sub-camera 400 are the same as those in FIG. 2 of the first embodiment.

[0258] First, the functional configuration of the WS 100 for implementing the control process of this embodiment will be described with reference to FIG.

[0259] The functions of the WS 100 are realized by hardware and / or software. If the functional units shown in Fig. 16 are configured by hardware instead of by software, it is sufficient to have a circuit configuration corresponding to each of the functional units in Fig. 16.

[0260] WS100 includes an image recognition unit 121, an object of interest determination unit 122, a tracking target determination unit 123, a control information generation unit 124, a feature information determination unit 125, a tracking state determination unit 126, an image recognition unit 127, and a tracking target determination unit 128. Software for realizing these functions is stored in non-volatile memory 103, and the control unit 101 loads it into volatile memory 102 and executes it.

[0261] The functions of the target subject determining unit 122, the tracking target determining unit 123, and the feature information determining unit 125 are similar to those in the first embodiment shown in Fig. 3. The functions of the recognition units 121 and 127 are similar to those of the image recognition unit 221 described in the first embodiment.

[0262] The functions and basic operations of the WS 100 will be described with reference to FIGS.

[0263] The process of step S501 is similar to the process of step S101 in FIG.

[0264] In step S502, the control unit 101 executes the function of the image recognition unit 121 in Fig. 3. The process executed by the control unit 101 through the image recognition unit 121 will be described later.

[0265] The processing in steps S503 to S504 is similar to that in steps S103 to S104 in FIG.

[0266] In step S505, the control unit 101 transmits a shooting command to the sub-camera 400 via the communication unit 105, receives the captured sub-image from the sub-camera 400, stores it in the volatile memory 102, and proceeds to step S506.

[0267] In step S506, the control unit 101 executes the function of the image recognition unit 127 in FIG. 16, and advances the process to step S507.

[0268] The functions of the image recognition unit 127 can be achieved by replacing the control unit 201 with the control unit 101, the volatile memory 202 with the volatile memory 102, and the non-volatile memory 203 with the non-volatile memory 103 in the description of the image recognition unit 221 of the EB 200 of the first embodiment.

[0269] 16, compares the feature information calculated in steps S502 and S506, and updates the tracking state information STATE. Furthermore, the control unit 101 stores the tracking subject SELECT_ID and the tracking state information STATE in the volatile memory 102, and the process proceeds to step S508.

[0270] The tracking state information STATE includes information on either "tracking using overhead image" or "tracking using sub-image." "Tracking using overhead image" indicates a state in which the tracking subject is tracked by controlling sub-camera 400 based on the subject recognition result of the overhead image of overhead camera 300. "Tracking using sub-image" indicates a state in which the tracking subject is tracked by controlling sub-camera 400 based on the subject recognition result of the sub-image of sub-camera 400. The process of step S507 will be described in detail later.

[0271] The processes in steps S508 to S510 are executed by the function of the control information generating unit 124 in FIG.

[0272] In step S508, the control unit 101 reads the tracking state information STATE from the volatile memory 102, and determines whether "tracking using an overhead image" or "tracking using a sub-image" based on the tracking state information STATE. If the control unit 101 determines that "tracking using an overhead image" is in progress, the process proceeds to step S510, and if the control unit 101 determines that "tracking using a sub-image is in progress", the process proceeds to step S509.

[0273] In step S509, the control unit 101 calculates the pan value / tilt value of the sub camera 400 based on the subject recognition result of the sub image of the sub camera 400, and proceeds to step S511. The process of step S509 can be performed by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of the control information generation unit 223 in FIG.

[0274] In step S510, the control unit 101 calculates the pan value / tilt value of the sub camera 400 based on the subject recognition result of the overhead image of the overhead camera 300, and proceeds to step S511. The process of step S510 can be performed by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of the control information generation unit 223 in FIG.

[0275] In step S511, the control unit 101 executes the function of the control information generating unit 124 in FIG. 3, and advances the process to step S512.

[0276] The processes in steps S511 and S512 are similar to those in steps S108 and S109 in FIG.

[0277] The above is the basic operation of the WS100.

[0278] Next, the control process of the WS 100 will be described with reference to FIG.

[0279] FIG. 18 shows the control process of the WS 100, and shows the detailed process of step S507 in FIG.

[0280] The process of step S520 is similar to that of step S110 in FIG.

[0281] In step S521, the control unit 101 executes the function of the tracking state determination unit 126 in FIG. 16, and changes the tracking state information STATE to "tracking using an image from an overhead camera".

[0282] The tracking state determination unit 126 has a function of updating the tracking state information STATE stored in the volatile memory 102 .

[0283] In step S522, the control unit 101 reads the tracking state information STATE from the volatile memory 102, and determines whether "tracking using an overhead image" or "tracking using a sub-image" is in progress based on the tracking state information STATE. If the control unit 101 determines that "tracking using an overhead image" is in progress, the process proceeds to step S525, and if the control unit 101 determines that "tracking using a sub-image is in progress", the process proceeds to step S523.

[0284] The process of step S523 can be achieved by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of step S224 in FIG. 9B.

[0285] In step S524, the control unit 101 executes the function of the tracking state determination unit 126 in FIG. 16, and changes the tracking state information STATE to "tracking using overhead image".

[0286] The processes in steps S525 and S526 are similar to those in steps S117 and S118 in FIG.

[0287] The process of steps S527 to S529 can be achieved by replacing the control unit 201 with the control unit 101 and the volatile memory 202 with the volatile memory 102 in the process of steps S211 to S214 in FIG. 9B.

[0288] In step S530, the control unit 101 executes the function of the tracking state determination unit 126 in FIG. 16 to change the tracking state information STATE to "tracking using sub-image", and ends the process.

[0289] According to the above-described process, WS 100 switches whether to control sub-camera 400 based on the subject recognition result of the overhead image of overhead camera 300 or the sub-image of sub-camera 400. This makes EB 200 of the first embodiment unnecessary, and the system configuration can be simplified to obtain the same effect as in the first embodiment.

[0290] Next, a process in which the WS 100 acquires position information of a predetermined part of a tracking subject from an overhead image will be described with reference to Fig. 19. The process for acquiring head position information of a person will be described below.

[0291] FIG. 19 shows the image recognition process of the WS 100, and shows the detailed process of step S502 in FIG.

[0292] In the second embodiment, the image recognition process that is applied to the sub-image in the first embodiment is applied to the overhead image.

[0293] In step S601, the control unit 101 executes image recognition processing by the image recognition unit 121 based on the overhead image read from the volatile memory 102 and the first trained model read from the non-volatile memory 103. Then, the control unit 101 acquires position information POSITION and identification information ID of the subject in the overhead image acquired by the image recognition processing, stores them in the volatile memory 102, and proceeds to step S602.

[0294] In step S602, the control unit 101 corrects the position information POSITION of the subject corresponding to the tracking subject SUBJECT_ID read from the volatile memory 102 and the overhead image of the overhead camera 300. A method of calculating the corrected subject position information POSITION_CORR for skeletal position estimation by the image recognition unit 121 and a method of correcting the overhead image will be described later. The control unit 101 saves the corrected subject position information POSITION_CORR and the sub-image in the volatile memory 102 and proceeds to S603.

[0295] The process of step S603 is the same as the process of step S303 in Fig. 11. The control unit 101 estimates skeletal position information KP of the subject based on the second trained model for skeletal position estimation read from the non-volatile memory 103 and the overhead image read from the volatile memory 102.

[0296] The process of step S604 is similar to step S304 in Fig. 11. Control unit 101 reads skeletal position information KP of the head of the subject corresponding to the identification information ID of the tracking subject from non-volatile memory 103, and estimates the head position of the subject.

[0297] Next, the process of correcting the overhead image and the subject area in step S602 in FIG. 19 will be described with reference to FIGS. 20 and 21. FIG.

[0298] Fig. 20 is a flowchart illustrating a correction process for an overhead image and position information POSITON of a subject. The processes in steps S701 to S707 and S709 to S710 in Fig. 20 are similar to steps S401 to S407 and S409 to S410 in Fig. 13(a), and the process in step S708 is different from Fig. 13(a). The following mainly describes the differences from the first embodiment.

[0299] Since an overhead image includes the entire subject (if the subject is a person, the entire body of the person), the area BOXtarget of the subject to be corrected and the areas BOXi of other subjects become position information of the entire subject.

[0300] In step S708, the control unit 101 calculates the foot coordinates from the area BOXtarget of the subject to be corrected read from the volatile memory 102 and the areas BOXi of the other subjects, and compares the two.

[0301] Here, a method for calculating the subject's feet coordinates and the determination process in step S708 will be described with reference to FIG.

[0302] FIG. 21(a) illustrates the subjects A, B, and C in the bird's-eye view image, the rectangular regions BOXtarget, BOX1, and BOX2 of the subjects A, B, and C, and the foot coordinates (xa, ya), (xb, yb), and (xc, yc) of the subjects A, B, and C in the bird's-eye camera coordinate system.

[0303] In FIG. 21, the position information POSITION of the subject B is shown as the region BOXtarget of the subject to be corrected, and the position information POSITION of the subjects A and C is shown as the regions BOX1 and BOX2 of the other subjects.

[0304] In step S708, the control unit 101 compares the foot coordinate in the vertical direction of the region BOXtarget of the subject to be corrected with the foot coordinate in the vertical direction of the region BOXi of the other subject. When the foot coordinate of the region BOXi of the other subject is smaller than the foot coordinate of the region BOXtarget of the subject to be corrected (foot coordinate of BOXi < foot coordinate of BOXtarget), the control unit 101 determines that the subject to be corrected is located in front (closer to the sub-camera 400) of the other subject, and advances the process to step S709. Also, when the foot coordinate of the region BOxi of the other subject is greater than or equal to the foot coordinate of the region BOXtarget of the subject to be corrected (foot coordinate of BOXi ≧ foot coordinate of BOXtarget), the control unit 101 determines that the subject to be corrected is located behind (farther from the sub-camera 400) the other subject, and advances the process to step S710.

[0305] FIG. 21(b) illustrates the result of calculating the corrected region information BOXtarget_CORR by executing the correction process of step S602 on the rectangular region BOXtarget of the subject B to be corrected shown in FIG. 21(a).

[0306] The control unit 101 generates the corrected region information BOXtarget_CORR of the subject B to be corrected based on the result of determining the front-back positional relationship between the subjects according to the foot coordinates of the subjects, and acquires the skeleton position information KP regarding the head of the tracking subject in step S603.

[0307] The above-described correction process enables the control unit 101 to accurately calculate the head position (height h2 in FIG. 8) of the tracking subject based on the vertical difference between the foot coordinates and head position of the tracking subject. This enables tracking control such that the head of the tracking subject is positioned at a desired position (for example, the center of the screen) within the shooting angle of view of the sub-camera 400, as in the first embodiment. As a result, it becomes possible to smoothly switch between control of the sub-camera 400 based on the image recognition result of the overhead image by the image recognition unit 121 and control of the sub-camera 400 based on the image recognition result of the sub-image by the image recognition unit 127.

[0308] As in the first embodiment, overhead camera 300 includes a focus detection unit, and transmits a distance map, which is a focus detection (distance measurement) result corresponding to each pixel of the overhead image, to WS 100. Then, control unit 201 of WS 100 reads out the distance map from volatile memory 102, and can grasp the positional relationship between subjects A, B, and C by referring to the focus detection results corresponding to the area BOXtarget of the subject to be corrected and the areas BOXi of the other subjects.

[0309] According to the above-mentioned second embodiment, the WS 100 switches whether to control the sub camera 400 based on the subject recognition result of the overhead image of the overhead camera 300 or the sub image of the sub camera 400. In this case, at least one of the overhead image of the overhead camera 300 and the position information of the tracked subject is corrected. This makes it possible to obtain position information of a predetermined part of the tracked subject (e.g., the head) based on the corrected overhead image and the corrected position information of the tracked subject. As a result, tracking control is possible such that a predetermined part of the tracked subject is located at a desired position (e.g., the center of the screen) within the shooting angle of view of the sub camera 400.

[0310] It is also possible to add the second embodiment, which does not use the EB200, to the first embodiment, which uses the EB200. In this case, the control of the sub-camera 400 by the WS100 and the control of the sub-camera 400 by the EB200 can commonly capture a predetermined part of the tracking subject (a person's face) with high accuracy, and the control of the sub-camera 400 can be smoothly switched.

[0311] [Embodiment 3] In the first and second embodiments, an example of a system including the overhead camera 300 and the sub-camera 400 has been described.

[0312] In the third embodiment, an example of a system including a main camera 500 in addition to an overhead camera 300 and a sub-camera 400 will be described.

[0313] FIG. 22 is a system configuration diagram of the third embodiment.

[0314] The third embodiment differs from the first embodiment in that a main camera 500 is provided, and a subject to be tracked by the sub camera 400 is determined based on a main image captured by the main camera 500. The following description will focus on the differences from the first embodiment.

[0315] In the third embodiment, the main camera 500 has a PTZ function. The target subject determination unit 122 of the WS 100 determines (estimates) a target subject of the main camera 500 from the shooting range of the main camera 500, and determines a tracking subject of the sub camera 400 based on the target subject of the main camera 500. The tracking subject of the sub camera 400 may be the same subject as the target subject of the main camera 500, or may be a different subject.

[0316] Next, an example of determining a tracking subject of the sub camera 400 based on a role (ROLE) set to the sub camera 400 will be described.

[0317] The role of the sub camera 400 indicates the subject of interest in the main camera 500, the subject to be tracked by the sub camera 400 associated with the zoom operation, and the control content of the zoom operation. The role of the sub camera 400 can be set by the user via an operation unit provided in the WS100 or EB200. In addition, when multiple sub cameras are installed, any of the multiple sub cameras can be set as the main camera, and the setting of the main camera may be set by the user via an operation unit provided in the WS100 or EB200. The role of the sub camera 400 and the setting method of the main camera are not limited to the above method, and any method may be used.

[0318] FIG. 23 shows an example of the roles and contents that can be set for the sub-camera 400.

[0319] When the role (ROLE) is "main follow", the role (CAMERA_ROLE) of the sub camera 400 is to track the same subject as the subject that the main camera 500 is focusing on, and to perform zoom control in the same phase as the zoom operation of the main camera 500. Based on this role (CAMERA_ROLE), the zoom control value of the sub camera 400 is calculated. Here, the same phase in the zoom operation means that the zoom operations of the main camera 500 and the sub camera 400 are controlled in the same direction. For example, when the zoom control value of the main camera 500 is changed from the wide-angle side to the telephoto side, the zoom of the sub camera 400 is also changed from the wide-angle side to the telephoto side.

[0320] When the role (ROLE) is "main counter", the role (CAMERA_ROLE) of the sub camera 400 is to track the same subject as the subject that the main camera 500 is focusing on, and to perform zoom control in the opposite phase to the zoom operation of the main camera 500. The PTZ value of the sub camera 400 is calculated based on this role (CAMERA_ROLE). Here, the opposite phase in the zoom operation means that the zoom operations of the main camera 500 and the sub camera 400 are controlled in the opposite directions. For example, when the zoom control value of the main camera 500 is changed from the wide-angle side to the telephoto side, the zoom of the sub camera 400 is changed from the telephoto side to the wide-angle side.

[0321] When the role (ROLE) is "assist follow", the sub camera 400 tracks a subject other than the subject that the main camera 500 is focusing on, and performs zoom control in the same phase as the zoom operation of the main camera 500. Based on this role (CAMERA_ROLE), the zoom control value of the sub camera 400 is calculated.

[0322] When the role (ROLE) is "assist counter", the sub camera 400 tracks a subject other than the subject that the main camera 500 is focusing on, and performs zoom control in the opposite phase to the zoom operation of the main camera 500. Based on this role (CAMERA_ROLE), the zoom control value of the sub camera 400 is calculated. In the example of FIG. 23, "separate from main (left side)" is exemplified as the control content of the tracking subject for "assist follow" and "assist counter", but there may be "assist follow" and "assist counter" in which the tracking subject is "separate from main (right side)".

[0323] In addition, when the tracking subject is "separate from the main subject", it may also serve the role of tracking a subject in a position other than left and right (up and down, or front and back).

[0324] When there are multiple sub-cameras, a role may be set for each sub-camera.

[0325] Furthermore, in the third embodiment, an example has been described in which the control contents for the tracking subject and zoom are set as roles, but the control contents for only the tracking subject may be set as a role, or other items may be added.

[0326] In the third embodiment, a tracking subject of the sub camera 400 is set based on the main image of the main camera 500, and an example is described in which the first embodiment is combined with the third embodiment. However, the second embodiment may be combined with the third embodiment.

[0327] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0328] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention.

[0329] The disclosure of this specification includes the following information processing device, system, control device, control method, and program. [Configuration 1] A detection means for detecting a subject included in an image captured by the imaging device; a correction means for correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the image so that the region of the predetermined subject does not include other subjects different from the predetermined subject; and acquiring means for acquiring position information of at least one part of the predetermined subject based on the predetermined information corrected by the correction means. [Configuration 2] an input means for inputting an image captured by the imaging device; 2. The information processing device according to configuration 1, further comprising: a determining unit that determines the predetermined subject from a plurality of subjects included in the image. [Configuration 3] The information processing device according to configuration 1 or 2, characterized in that the predetermined information includes at least one of an image in which the other subject is included in the area of ​​the predetermined subject and position information indicating the area of ​​the predetermined subject or center of gravity information of the area of ​​the predetermined subject. [Configuration 4] 4. The information processing apparatus according to configuration 3, wherein the correction means performs a mask process on the area of ​​the other subject in the image when the other subject is included in the area of ​​the predetermined subject. [Configuration 5] when a size of the other subject is smaller than a size of the predetermined subject, the correction means performs the mask processing on an area of ​​the other subject in the image; 5. The information processing device according to configuration 4, wherein, when a size of the other subject is larger than a size of the predetermined subject, the area of ​​the predetermined subject is corrected so as not to include the other subject. [Configuration 6] when the other subject partially overlaps an area of ​​the predetermined subject and the area of ​​the area of ​​the predetermined subject is larger than an area of ​​the area of ​​the other subject, it is determined that the predetermined subject is located closer to the imaging device than the other subject; The information processing device according to any one of configurations 1 to 3, further comprising a determination means for determining that the specified subject is located farther from the imaging device than the other subjects when the area of ​​the specified subject partially overlaps with the other subjects and the area of ​​the specified subject is smaller than the area of ​​the other subjects. [Configuration 7] The information processing device according to configuration 6, wherein the correction means corrects the area of ​​the specified subject so that the other subject is not included when the other subject partially overlaps the area of ​​the specified subject and the specified subject is located closer to the imaging device than the other subject. [Configuration 8] The information processing device described in configuration 7, wherein the correction means corrects the area of ​​the specified subject so that the other subject is not included by moving the center of gravity of the area of ​​the specified subject in a direction opposite to a direction in which the center of gravity of the area of ​​the other subject is located relative to the center of gravity of the area of ​​the specified subject. [Configuration 9] The information processing device according to configuration 6, characterized in that the correction means does not correct the specified information when the area of ​​the specified subject partially overlaps with the other subject and the specified subject is located farther from the imaging device than the other subject. [Configuration 10] 10. The information processing device according to any one of configurations 1 to 9, wherein the region of the subject includes coordinate information of a rectangle that circumscribes the subject in the image. [Configuration 11] the subject is a person, 11. The information processing device according to any one of configurations 1 to 10, wherein the part includes at least one of a human skeleton, a head, a face, and facial organs. [Configuration 12] a control unit that controls a photographing direction of the imaging device so as to track the predetermined subject based on the position information of the part; The information processing device according to any one of configurations 1 to 11, characterized in that the control means controls at least one of a pan operation and a tilt operation of the imaging device so that a predetermined portion of the predetermined subject is positioned at a desired position within a shooting angle of view of the imaging device. [Configuration 13] the acquisition means acquires position information of the predetermined subject by performing inference processing using a first trained model with the image as an input; The information processing device according to any one of configurations 1 to 12, characterized in that the position information of the part is obtained by performing inference processing using a second trained model with the image and position information of the specified subject as input. [Configuration 14] A system including a first imaging device and a second imaging device having different shooting directions, and a first control device and a second control device that control the second imaging device so as to track a predetermined subject based on a first image captured by the first imaging device or a second image captured by the second imaging device, the system switches between a first state in which the first control device controls the second imaging device to track the predetermined subject based on the first image and a second state in which the second control device controls the second imaging device to track the predetermined subject based on the second image; The second control device includes: a detection means for detecting a subject included in an image captured by the second imaging device; a correction means for correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the image so that a subject other than the predetermined subject is not included in the region of the predetermined subject; an acquisition means for acquiring position information of at least one part of the predetermined subject based on the predetermined information corrected by the correction means; and a control means for controlling an imaging direction of the second imaging device so as to track the specified subject based on the position information of the part. [Configuration 15] A control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a shooting direction different from that of the first imaging device, A detection means for detecting a subject included in the first image; a correction means for correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the first image such that a subject other than the predetermined subject is not included in the region of the predetermined subject; an acquisition means for acquiring position information of at least one part of the predetermined subject based on the predetermined information corrected by the correction means; a control unit that switches between a first state in which the second imaging device is controlled so as to track the predetermined subject based on the first image and a second state in which the second imaging device is controlled so as to track the predetermined subject based on the second image, The control device is characterized in that the control means controls an imaging direction of the second imaging device so as to track the specified subject based on position information of the part in the first state. [Configuration 16] The control device described in configuration 15, characterized in that when the subject is a person and the first image includes the entire body of the person, the correction means compares the foot coordinates of a specific person as the specific subject with another person different from the specific person, and when the specific person is located closer to the first imaging device than the other person, corrects the position information of the specific person so that the different person is not included in the area of ​​the specific person. [Configuration 17] A method for controlling an information processing device, comprising: Detecting a subject included in a captured image; correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the image such that the region of the predetermined subject does not include other subjects different from the predetermined subject; and acquiring position information of at least one part of the specified subject based on the corrected specified information. [Configuration 18] A method for controlling a second imaging device to track a predetermined object based on a first image captured by the first imaging device or a second image captured by a second imaging device having a shooting direction different from that of the first imaging device, comprising: detecting an object included in the first image; correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the first image such that a subject other than the predetermined subject is not included in the region of the predetermined subject; acquiring position information of at least one part of the predetermined subject based on the corrected predetermined information; and switching between a first state in which the second imaging device is controlled so as to track the predetermined subject based on the first image and a second state in which the second imaging device is controlled so as to track the predetermined subject based on the second image, A control method comprising: controlling an imaging direction of the second imaging device so as to track the specified subject based on position information of the part in the first state. [Configuration 19] A program for causing a computer to function as an information processing device according to any one of configurations 1 to 13 or as a control device according to configuration 15 or 16. [Explanation of symbols]

[0330] 100...first control device (workstation / WS), 200...second control device (edge ​​box / EB), 300...first imaging device (overhead camera), 400...second imaging device (sub-camera), 101, 201, 301, 401...control unit

Claims

1. A detection means for detecting a subject included in an image captured by the imaging device; a correction means for correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the image so that the region of the predetermined subject does not include other subjects different from the predetermined subject; and acquiring means for acquiring position information of at least one part of the predetermined subject based on the predetermined information corrected by the correction means.

2. an input means for inputting an image captured by the imaging device; 2. The information processing apparatus according to claim 1, further comprising: a determining unit that determines the predetermined subject from a plurality of subjects included in the image.

3. 2. The information processing device according to claim 1, wherein the predetermined information includes at least one of an image in which the other subject is included in the area of ​​the predetermined subject and position information indicating the area of ​​the predetermined subject or center of gravity information of the area of ​​the predetermined subject.

4. 4. The information processing apparatus according to claim 3, wherein the correction means, when the other subject is included in the area of ​​the predetermined subject, performs a mask process on the area of ​​the other subject in the image.

5. when a size of the other subject is smaller than a size of the predetermined subject, the correction means performs the mask processing on an area of ​​the other subject in the image; 5 . The information processing apparatus according to claim 4 , wherein, when a size of the other subject is larger than a size of the predetermined subject, the area of ​​the predetermined subject is corrected so as not to include the other subject.

6. when the other subject partially overlaps an area of ​​the predetermined subject and the area of ​​the area of ​​the predetermined subject is larger than an area of ​​the area of ​​the other subject, it is determined that the predetermined subject is located closer to the imaging device than the other subject; 2. The information processing device according to claim 1, further comprising a determination means for determining that the specified subject is located farther from the imaging device than the other subject when the area of ​​the specified subject partially overlaps with the area of ​​the other subject and the area of ​​the specified subject is smaller than the area of ​​the other subject.

7. The information processing device according to claim 6, characterized in that the correction means corrects the area of ​​the specified subject so that the other subject is not included when the other subject partially overlaps the area of ​​the specified subject and the specified subject is located closer to the imaging device than the other subject.

8. The information processing device according to claim 7, characterized in that the correction means corrects the area of ​​the specified subject so that the other subject is not included by moving the center of gravity of the area of ​​the specified subject in a direction opposite to a direction in which the center of gravity of the area of ​​the other subject is located relative to the center of gravity of the area of ​​the specified subject.

9. The information processing device according to claim 6, characterized in that the correction means does not correct the specified information when the area of ​​the specified subject partially overlaps with the other subject and the specified subject is located farther from the imaging device than the other subject.

10. The information processing apparatus according to claim 1 , wherein the region of the subject includes coordinate information of a rectangle circumscribing the subject in the image.

11. the subject is a person, 2. The information processing apparatus according to claim 1, wherein the part includes at least one of a skeleton, a head, a face, and facial organs of a person.

12. a control unit that controls a photographing direction of the imaging device so as to track the predetermined subject based on the position information of the part; 2 . The information processing apparatus according to claim 1 , wherein the control means controls at least one of a pan operation and a tilt operation of the imaging device so that a predetermined portion of the predetermined subject is positioned at a desired position within a shooting angle of view of the imaging device.

13. the acquisition means acquires position information of the predetermined subject by performing inference processing using a first trained model with the image as an input; The information processing device according to claim 1, characterized in that the position information of the part is obtained by performing inference processing using a second trained model with the image and position information of the specified subject as input.

14. A system including a first imaging device and a second imaging device having different shooting directions, and a first control device and a second control device that control the second imaging device so as to track a predetermined subject based on a first image captured by the first imaging device or a second image captured by the second imaging device, the system switches between a first state in which the first control device controls the second imaging device to track the predetermined subject based on the first image and a second state in which the second control device controls the second imaging device to track the predetermined subject based on the second image; The second control device includes: a detection means for detecting a subject included in an image captured by the second imaging device; a correction means for correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the image so that a subject other than the predetermined subject is not included in the region of the predetermined subject; an acquisition means for acquiring position information of at least one part of the predetermined subject based on the predetermined information corrected by the correction means; and a control means for controlling a shooting direction of the second imaging device so as to track the specified subject based on the position information of the part.

15. A control device that controls a second imaging device to track a predetermined subject based on a first image captured by a first imaging device or a second image captured by a second imaging device having a shooting direction different from that of the first imaging device, A detection means for detecting a subject included in the first image; a correction means for correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the first image such that a subject other than the predetermined subject is not included in the region of the predetermined subject; an acquisition means for acquiring position information of at least one part of the predetermined subject based on the predetermined information corrected by the correction means; a control unit that switches between a first state in which the second imaging device is controlled so as to track the predetermined subject based on the first image and a second state in which the second imaging device is controlled so as to track the predetermined subject based on the second image, The control device according to claim 1, wherein the control means controls a shooting direction of the second imaging device so as to track the specified subject based on position information of the part in the first state.

16. The control device described in claim 15, characterized in that when the subject is a person and the first image includes the entire body of the person, the correction means compares the foot coordinates of a specific person as the specific subject with another person different from the specific person, and when the specific person is located closer to the first imaging device than the other person, corrects the position information of the specific person so that the different person is not included in the area of ​​the specific person.

17. A method for controlling an information processing device, comprising: Detecting a subject included in a captured image; correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the image such that the region of the predetermined subject does not include other subjects different from the predetermined subject; and acquiring position information of at least one part of the predetermined subject based on the corrected predetermined information.

18. A method for controlling a second imaging device to track a predetermined subject based on a first image captured by the first imaging device or a second image captured by a second imaging device having a shooting direction different from that of the first imaging device, comprising: detecting an object included in the first image; correcting predetermined information relating to a region of a predetermined subject among a plurality of subjects included in the first image such that a subject other than the predetermined subject is not included in the region of the predetermined subject; acquiring position information of at least one part of the predetermined subject based on the corrected predetermined information; and switching between a first state in which the second imaging device is controlled so as to track the predetermined subject based on the first image and a second state in which the second imaging device is controlled so as to track the predetermined subject based on the second image, a control method for controlling a shooting direction of the second imaging device so as to track the specified subject based on position information of the part in the first state.

19. A program for causing a computer to function as the information processing device according to any one of claims 1 to 13 or the control device according to claim 15 or 16.

Images