Image processing apparatus, image processing system, control method of image processing apparatus, and program for identifying a subject in a captured image

Abstract

[Problem] An image processing device, etc., capable of accurately identifying a predetermined action performed by an object in a captured image regardless of its position in the camera's viewpoint. [Solution] The present invention is characterized by comprising: a distortion information generation unit that generates distortion information indicating the distortion of the shape of a first object in the captured image input by an input unit relative to a known shape; a calculation unit (step S705) that calculates attention position coordinates as the position of a second object in the captured image; a determination unit that determines an attention region based on the distortion information and the attention position coordinates; an image generation unit (step S402) that generates an image signal (601) of the attention region where the attention object exists from the captured image; a scaling unit (step S404) that scales the attention region image to a predetermined size; and a recognition unit (step S406) that performs predetermined recognition based on the scaled attention region image. The determination unit determines the size of the attention region corresponding to the actual size based on the distortion information and the attention position coordinates.

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing system for identifying objects in captured images, and a control method and program for the image processing apparatus. Background Technology

[0002] Currently, there is a need for automatic video recording of exciting moments during sports and other video scenarios. As an example of an exciting moment, a player shooting a basket in basketball would be such a scene. To automatically capture images of a shooting scene, one method involves determining whether a player is shooting based on the positional relationship between the ball and the player derived from camera data, as well as the player's posture, and then controlling the camera to record the shot for the player making the shot.

[0003] Known methods for estimating player pose from camera data involve detecting "player joint coordinates." In this case, a method using a trained pose estimation model (pose estimation model) can be used to detect the "player joint coordinates." Specifically, this method involves inputting an image of the player into the pose estimation model, which then outputs the coordinates of joints such as the top of the head, neck, wrists, elbows, shoulders, and waist, thereby obtaining the "joint coordinates." In the case of estimating player pose using basketball as an example, inputting camera data covering the entire basketball court into the "pose estimation model" results in long processing times due to the high input resolution, which is unsuitable for real-time processing. To address this, known methods involve reducing the input resolution and shortening the processing time by scaling down the camera data before inputting it into the "pose estimation model."

[0004] However, when the camera data is scaled down, the resolution of the players appearing in the camera data also decreases accordingly, which may hinder the accurate detection of the player's joint coordinates. In other words, this may hinder the determination of whether a player is shooting. To address this, Patent Document 1 discloses a method for performing predetermined recognition at high speed while improving the resolution of the input to a trained model including a pose estimation model. Patent Document 1 discloses a technique for detecting the coordinates of a ball from camera data and determining the player's action based on an image cropped to the vicinity of the ball. Since players are highly likely to perform predetermined actions near the ball in certain types of sports, this technique can be used to determine the player's action based on the ball's coordinates and the player's joint coordinates. Existing technical documents Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2020-054748 Summary of the Invention The problem the invention aims to solve

[0006] However, when the camera captures an image covering the entire basketball court, players farther from the camera appear smaller in the image, while those closer to the camera appear larger. In the conventional technique disclosed in Patent Document 1, the cropping size is always the same. Therefore, there is a possibility that when cropping is based on players farther from the camera, players closer to the camera will be cropped out. Since the "cropped portion" is not used for detecting the player's joint coordinates, it is difficult to estimate the player's posture with high accuracy. On the other hand, when cropping is based on players closer to the camera, there is a possibility that the resolution of players farther from the camera will become even lower. Low resolution makes it difficult to identify players with high accuracy because the resolution is further reduced due to the downscaling process performed before the data is input into the trained model. When using data obtained by capturing images of a wide area such as a basketball court for predetermined recognition, cropping to a fixed size without considering the distance between the camera and the subject (players, etc.) leads to cropping that is unsuitable for recognition. As a result, recognition accuracy decreases, which becomes a problem.

[0007] The present invention provides an image processing apparatus, an image processing system, a control method, and a program that enable highly accurate recognition of predetermined actions performed by an object in an image, regardless of the position in the camera's viewpoint. Solution for solving the problem

[0008] The image processing apparatus according to the present invention is an image processing apparatus capable of recognizing a predetermined scene in a captured image, and includes: an input unit for inputting a captured image obtained by capturing an image of a first object; a distortion information generation unit for generating distortion information, the distortion information indicating the distortion of the shape of the first object in the captured image input by the input unit relative to a known shape; a calculation unit for calculating a point of interest position in the captured image; a determination unit for determining a point of interest region based on the distortion information generated by the distortion information generation unit and the point of interest position calculated by the calculation unit; an image generation unit for generating a point of interest region image from the captured image based on the point of interest region determined by the determination unit; a scaling unit for scaling the point of interest region image generated by the image generation unit to a predetermined size; and a recognition unit for performing a predetermined recognition based on the scaled point of interest region image. The determination unit determines the size of the point of interest region based on the distortion information generated by the distortion information generation unit and the point of interest position calculated by the calculation unit, the size of the point of interest region being the size of the point of interest region corresponding to the actual size. Advantages of the invention

[0009] According to the present invention, an image processing device, image processing system, control method, and program can be provided that enables highly accurate recognition of predetermined actions performed by an object in an image regardless of its position in the camera's viewpoint. Attached Figure Description

[0010] Figure 1 This is a configuration diagram of the image processing system according to the first to third embodiments. Figure 2 This is a hardware configuration diagram of the image processing system according to the first to third embodiments. Figure 3 This is a functional configuration diagram of the image processing system according to the first to third embodiments. Figure 4A This is a flowchart illustrating the processes to be performed by the overhead camera according to the first to third embodiments. Figure 4B This is a flowchart illustrating the processing to be performed by the image processing device according to the first to third embodiments. Figure 4C This is a flowchart illustrating the processes to be performed by the attention camera according to the first to third embodiments. Figure 5A These are explanatory diagrams of overhead image signals according to the first to third embodiments. Figure 5B These are explanatory diagrams of the attention area image signals according to the first to third embodiments. Figure 5C These are explanatory diagrams of reduced image signals according to the first to third embodiments. Figure 6 This is a flowchart illustrating the attention area image generation process according to the first embodiment. Figure 7 This is an explanatory diagram of the attention area image generation process according to the first embodiment. Figure 8 This is a flowchart illustrating the attention area image generation process according to the second embodiment. Figure 9A This is an explanatory diagram (side perspective view) of the attention area image generation process according to the second embodiment. Figure 9B This is an explanatory diagram (top perspective view) of the attention area image generation process according to the second embodiment. Figure 10 This is a flowchart illustrating the attention area image generation process according to the third embodiment. Figure 11A This is an explanatory diagram (stereoscopic view) of the attention area image generation process according to the third embodiment. Figure 11B This is an explanatory diagram (plan view) of the attention area image generation process according to the third embodiment. Detailed Implementation

[0011] Embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the following embodiments are provided for illustrative purposes only and are not intended to limit the scope of the invention. First, a first embodiment of the invention will be described. It should be noted that in the drawings, the same or similar components are represented by the same reference numerals, and repeated descriptions will be omitted.

[0012] First, a first embodiment of the present invention will be described. In this first embodiment, basketball is used as an example, and the following process is described: determining whether a shooting scene has occurred based on camera data including the entire basketball court, and capturing video images including the shooting scene. It should be noted that in this first embodiment, using basketball as the camera target is merely for illustration and is not intended to limit the invention to any particular camera target. In the following description, a player is represented by 109, and a ball by 110. In descriptions involving multiple objects, only numbers are used to represent the entire object, while when individual objects are to be identified in the description, numbers with letters added, such as "109a," are used.

[0013] ( Figure 1 (Image processing system configuration diagram) Figure 1 This is a configuration diagram of an image processing system according to the first embodiment, including an image processing device 103 for performing shooting scene judgment processing. The image processing system includes an Internet 100, a local network 101, an overhead camera 102, an image processing device 103, a focus camera 104, a training server 105, a data collection server 106, and a client terminal 107. The image processing system also suitably includes a position sensor receiving device 115.

[0014] (Internet 100 / Local Network 101) The Internet 100 is a network used for communicating necessary information with training servers 105 and data collection servers 106. Furthermore, the Internet 100 can communicate with electronic devices connected to local network 101 via local network 101. Local network 101 is a network used for communicating necessary information with overhead camera 102, image processing equipment 103, observation camera 104, and client terminal 107. Furthermore, local network 101 can communicate necessary information with electronic devices connected to the Internet 100 via the Internet 100.

[0015] (Aerial view camera 102) The overhead camera 102 is a camera used to capture images of the entire basketball court 108. Furthermore, the overhead camera 102 transmits the captured images to the image processing device 103 via video cable 113. In this embodiment, it is assumed that the overhead camera 102 will capture images at 30 frames per second (30 fps), but this is not a limitation on the frame rate. Additionally, the imaging optical axis 114 of the overhead camera 102 is oriented parallel to the half-court line of the basketball court 108 and captures images at a "viewpoint θ," within which the nearest sideline of the basketball court 108 falls.

[0016] (Image processing device 103 / Attention camera 104) When the image processing device 103 performs object detection on the ball 110, posture estimation on the player 109, and determination of the presence / absence of the shooting scene on the captured image received from the overhead camera 102 in order to capture the shooting scene, it controls the focus camera 104. The focus camera 104 includes a pan / tilt mechanism and a zoom mechanism, and is a camera used to capture the shooting scene under the control of the image processing device 103. In this embodiment, it is assumed that the focus camera 104 will capture images at 60 frames per second (60 fps), but this is not a limitation on the frame rate.

[0017] (Training Server 105) Training server 105 is a server used to generate "trained models" to be used in object detection and pose estimation at image processing device 103. Training server 105 includes two trained models: a trained object detection model (hereinafter referred to as the "object detection model") for detecting ball 110 from camera data from overhead camera 102; and a trained pose estimation model (hereinafter referred to as the "pose estimation model") for estimating joint coordinates of player 109 from image signal 601 of the region of attention.

[0018] (Data collection server 106 / Client terminal 107 / Position sensor receiver 115) Data collection server 106 is a server that collects and accumulates supervised data for machine learning used by the trained model generated by training server 105. Client terminal 107 is a terminal that operates an electronic device connected to Internet 100 and local network 101, or issues instructions for sending and receiving data. The type of client terminal 107 is not limited to any particular type, and client terminal 107 can, for example, be formed from a general-purpose PC. Position sensor receiving device 115 reads ball position information from a position sensor attached to ball 110 and measures the position of ball 110 on basketball court 108.

[0019] The above is a description of an example configuration of the system for performing shooting scene judgment processing according to the first embodiment. The system extracts the position of the ball 110 and the posture of the player 109 from a basketball game, detects shooting scenes based on the conditions of the ball 110 and the player 109, and automatically captures images of the shooting scenes.

[0020] ( Figure 2 (Hardware configuration diagram of the image processing system) (Aerial view of the hardware configuration of camera 102) Figure 2 It constitutes Figure 1 The hardware configuration diagram of the system is shown below. The overhead camera 102 includes a CPU 201, RAM 202, ROM 203, video engine 204, NIC 205, video I / F 206, image sensor 207, zoom drive unit 208, and tilt sensor 227. These components are connected to the system bus 200 and are designed to send and receive necessary information relative to each other.

[0021] CPU 201 controls the overhead camera 102. CPU 201 controls the various units described later and performs control operations such as capturing images using image sensor 207 and transmitting captured data using video I / F 206. RAM 202 is a volatile memory whose stored information can be rewritten, and a working area is formed in RAM 202 for writing the control content of the overhead camera 102. RAM 202 is formed, for example, by a volatile memory (DRAM, etc.) using semiconductor elements. ROM 203 is a non-volatile memory and stores the program containing the control content of the overhead camera 102, as well as various parameters, in a non-volatile manner. ROM 203 is formed, for example, by flash memory. In response to the user turning on the power to the overhead camera 102, CPU 201 reads the program from ROM 203, loads the program into RAM 202, and executes the program, thus initiating control of the overhead camera 102.

[0022] Video engine 204 is an image processing engine that converts the electrical charge obtained from image sensor 207 into image signals. NIC 205 is a network interface card (NIC) used to enable the overhead camera 102 to communicate with other devices via local network 101. For example, it enables communication based on communication schemes standardized by Ethernet (registered trademark) or the IEEE 802.3 series. Video I / F 206 is an interface used to connect the overhead camera 102 and image processing device 103 using video cable 113.

[0023] The overhead camera 102 transmits video data to the image processing device 103 via the video I / F 206. It should be noted that the video I / F 206 is not limited to any particular type, as long as it is an interface capable of communicating with the image processing device 103. For example, a High-Definition Multimedia Interface (HDMI, registered trademark) can be used. Furthermore, the overhead camera 102 can be a webcam, or it can transmit video data to the image processing device 103 via the NIC 205.

[0024] Image sensor 207 is a sensor that receives light emitted from the photographed target and converts the brightness and color of the light into electrical charges. This sensor can be, for example, a photodiode, a charge-coupled device (CCD) sensor, or a complementary metal-oxide-semiconductor (CMOS) sensor. Zoom drive unit 208 is a unit used to change the "zoom" degree of the zoom lens included in the overhead camera 102. Tilt sensor 227 is a sensor that detects the angle of tilt of the main body of the overhead camera 102 relative to the horizontal direction.

[0025] (Hardware configuration of image processing device 103) The image processing device 103 includes a CPU 210, RAM 211, ROM 212, GPU 213, video I / F 214, NIC 215, and HDD 216. These components are connected to the system bus 209 and are designed to send and receive necessary information relative to each other.

[0026] The CPU 210 controls the image processing device 103. The CPU 210 controls the various units described later and performs operations corresponding to camera data received from the video I / F 206 or communication data received from the NIC 215. The RAM 211 is a volatile memory whose stored information can be rewritten, and a working area is formed in the RAM 211 for the program containing the control content of the image processing device 103. The RAM 211 is, for example, formed of a volatile memory (DRAM, etc.) using semiconductor elements. The ROM 212 is a non-volatile memory and stores the program containing the control content of the image processing device 103, as well as various parameters, in a non-volatile manner. In response to the user turning on the power to the image processing device 103, the CPU 210 reads the program from the ROM 212, loads the program into the RAM 211, and executes the program. Thus, control of the image processing device 103 begins. The ROM 212 is, for example, formed of a flash memory.

[0027] The GPU 213 is a unit for parallel computation of data. In inference processing where a large number of product-sum operations are required, the GPU 213 is faster and more efficient than the CPU 210 in execution. Although the GPU 213 is typically referred to as an LSI called a "Graphics Processing Unit," equivalent functionality can be achieved using reconfigurable logic circuitry hardware called an FPGA.

[0028] Video I / F 214 is an interface used to connect the overhead camera 102 and the image processing device 103 via video cable 113. The image processing device 103 receives video data from the overhead camera 102 via video I / F 214. Video I / F 214 is not limited to any particular type, as long as it is an interface capable of communicating with the overhead camera 102. For example, a Universal Serial Bus (USB) interface can be used. NIC 215 is a network interface card and is used for the image processing device 103 to communicate with other devices via local network 101. For example, communication based on communication schemes standardized by Ethernet (registered trademark) or the IEEE 802.3 series.

[0029] HDD 216 is a storage device that stores video data received from the overhead camera 102 and a "trained model" received from the training server 105. In this embodiment, HDD 216 is a hard disk drive (HDD) using a magnetic storage system, but HDD 216 is not limited to this type. For example, external storage devices such as solid-state drives (SSDs) using semiconductor elements can be used as HDD 216.

[0030] (Pay attention to the hardware configuration of the camera 104) The focus camera 104 includes a CPU 218, RAM 219, ROM 220, video engine 221, NIC 222, video I / F 223, image sensor 224, zoom drive unit 225, and pan / tilt drive unit 226. These components are connected to a system bus 217 and are designed to send and receive necessary information relative to each other.

[0031] The CPU 218 controls the camera 104. The CPU 218 controls the various units described later, and performs operations such as image capture using the image sensor 224 and communication data reception using the NIC 222. The RAM 219 is a rewritable memory, and a working area is formed in the RAM 219 containing the program that writes the control content of the camera 104. The RAM 219 is, for example, formed of a volatile memory using semiconductor elements. The ROM 220 is a non-volatile memory, and stores the program that writes the control content of the camera 104, as well as various parameters, in a non-volatile manner. In response to the user turning on the power to the camera 104, the CPU 218 reads the program from the ROM 220, loads the program into the RAM 219, and executes the program. Thus, control of the camera 104 begins. For example, the ROM 220 is formed of flash memory.

[0032] Video engine 221 is an image processing engine that converts the electrical charge obtained from image sensor 224 into image signals. NIC 222 is a network interface card used by the focus camera 104 to communicate with other devices via local network 101. For example, it can perform communication based on communication schemes standardized by Ethernet (registered trademark) or the IEEE 802.3 series. Video I / F 223 is an interface for transmitting video data from the focus camera 104. In the case of capturing video images of a basketball game using this system, the video data transmitted from video I / F 223 is a video image of the desired highlight scene. Therefore, video I / F 206 is used to connect an external storage device (not shown) for storing video images of the highlight scene. It should be noted that video I / F 206 is not limited to any particular type, as long as it is an interface that can communicate with image processing device 103. For example, an HDMI interface can be used.

[0033] Image sensor 224 is a sensor that receives light emitted from the photographed target and converts the brightness and color of the light into electrical charge. This sensor can be, for example, a photodiode, a CCD sensor, or a CMOS sensor. Zoom drive unit 225 is a unit for changing the "zoom" degree of the zoom lens included in the focusing camera 104. Pan / tilt drive unit 226 is a unit for changing the rotation degree of the pan / tilt rotation mechanism and the tilt rotation mechanism included in the focusing camera 104.

[0034] (Hardware configuration of training server 105) Training server 105 includes CPU 229, RAM 230, ROM 231, GPU 232, HDD 233, and NIC 234, and these components are designed to send and receive necessary information relative to each other via system bus 228. CPU 229 controls training server 105. CPU 229 controls the units described later and performs operations corresponding to the communication data received from NIC 234. RAM 230 is a rewritable memory, and a working area is formed in RAM 230 for the program that writes the control content of training server 105. RAM 230 is formed, for example, by volatile memory using semiconductor elements. ROM 231 is a non-volatile memory and stores the program that writes the control content of training server 105, as well as various parameters, in a non-volatile manner. In response to the user powering on the training server 105, the CPU 229 reads a program from the ROM 231, loads the program into the RAM 230, and executes the program, thus initiating control of the training server 105. For example, the ROM 231 is formed from flash memory.

[0035] GPU 232 is a unit that performs parallel computation of data and performs efficient computation by processing large amounts of data in parallel. Therefore, GPU 232 is effective when training a model for deep learning or similar purposes multiple times. In this embodiment, in the training process using training server 105, GPU 232 is used in addition to CPU 229. Specifically, when executing a training program including the training model, CPU 229 and GPU 232 cooperate to perform computation for training. It should be noted that training processing can be performed by only one of CPU 229 and GPU 232. Although GPU 232 is formed, for example, by an LSI called a "graphics processing unit", functions equivalent to those of an LSI can be formed by reconfigurable logic circuit hardware called an FPGA.

[0036] HDD 233 is a storage device that stores the supervision data received from the data collection server 106 and the trained model learned from the supervision data in a non-volatile manner. In this embodiment, HDD 233 is a hard disk drive using a magnetic storage system, but HDD 233 is not limited to this type. For example, an external storage device such as a solid-state drive using semiconductor elements can be used as HDD 233. NIC 234 is a network interface card and is used for the training server 105 to communicate with other devices via the Internet 100. For example, communication is performed based on communication schemes standardized by Ethernet (registered trademark) or the IEEE 802.3 series.

[0037] (Hardware configuration of data collection server 106) The data collection server 106 includes a CPU 236, RAM 237, ROM 238, GPU 239, HDD 240, and NIC 241, and these components are designed to send and receive necessary information relative to each other via system bus 235. The CPU 236 controls the data collection server 106. The CPU 236 controls the units described later and performs operations corresponding to the communication data received from the NIC 241. The RAM 237 is a rewritable memory, and a working area is formed within the RAM 237 for the program that writes the control contents of the data collection server 106. The RAM 237 is, for example, formed of volatile memory using semiconductor elements.

[0038] ROM 238 is a non-volatile memory that stores, in a non-volatile manner, the program containing the control content of the data collection server 106, as well as various parameters. In response to the user powering on the data collection server 106, CPU 236 reads the program from ROM 238, loads the program into RAM 237, and executes the program, thus initiating control of the data collection server 106. For example, ROM 238 may be formed from flash memory. GPU 239 is a unit that performs parallel computation of data. Although GPU 239 is typically formed from an LSI called a "graphics processing unit," functionally equivalent to an LSI can be formed from reconfigurable logic circuit hardware called an FPGA.

[0039] HDD 240 is a storage device that stores the supervision data requested from training server 105. Although HDD 240 is a hard disk drive using a magnetic storage system in this embodiment, HDD 240 is not limited to any particular type, and storage devices such as solid-state drives using semiconductor elements can be used as HDD 240. NIC 241 is a network interface card and is used by data collection server 106 to communicate with other devices via Internet 100. For example, communication can be conducted based on communication schemes standardized by Ethernet (registered trademark) or the IEEE 802.3 series.

[0040] (Hardware configuration of client terminal 107) The client terminal 107 includes a CPU 243, RAM 244, ROM 245, GPU 246, HDD 247, NIC 248, display unit 249, and input unit 250. These components are designed to send and receive necessary information relative to each other via system bus 242. The CPU 243 controls the client terminal 107. The CPU 243 controls the units described later and performs operations corresponding to data input from input unit 250 or communication data received from NIC 241. RAM 244 is a rewritable memory, and a working area is formed in RAM 244 for the program that writes the control content of the client terminal 107. RAM 244 is formed, for example, by a volatile memory using semiconductor elements.

[0041] ROM 245 is a non-volatile memory that stores the program containing the control content of client terminal 107, as well as various parameters, in a non-volatile manner. In response to the user powering on client terminal 107, CPU 243 reads the program from ROM 245, loads the program into RAM 244, and executes the program. Thus, control of client terminal 107 begins. For example, ROM 245 may be formed from flash memory. GPU 246 is a unit that performs parallel computation of data and is used to display image data received from image processing device 103 via NIC 248 on display unit 249. Although GPU 246 is typically formed from an LSI called a "graphics processing unit," equivalent functionality can be formed from reconfigurable logic circuit hardware called an FPGA.

[0042] HDD 247 is a storage device that stores image data received from image processing device 103 via NIC 248 in a non-volatile manner. In this embodiment, HDD 247 is a hard disk drive using a magnetic storage system, but HDD 247 is not limited to this type. Storage devices such as solid-state drives using semiconductor elements can be used as HDD 247. NIC 248 is a network interface card and is used by client terminal 107 to communicate with other devices via local network 101. For example, communication based on communication schemes standardized by Ethernet (registered trademark) or the IEEE 802.3 series. Display unit 249 is a display device that displays image data received from image processing device 103 via NIC 248. Display unit 249 is, for example, formed of a liquid crystal display (LCD). Input unit 250 is an input device that receives operations and instructions from users using the system. Input unit 250 is, for example, formed of a pointing device such as a mouse and an input device such as a keyboard.

[0043] ( Figure 3(Functional configuration diagram of the image processing system) Figure 3 Through Figure 2 The hardware configuration diagram illustrates the functional configuration of the software achieved through the collaboration between hardware and programs. It should be noted that general-purpose software such as an operating system is not illustrated in this software configuration.

[0044] <Functional Configuration of the Overhead Camera 102> The overhead camera 102 includes a camera unit 800, an operation unit 801, and a communication unit 802. These functional units are implemented by a CPU 201 loading a program stored in ROM 203 into RAM 202 and executing that program. In the camera unit 800, the CPU 201 controls the video engine 204, image sensor 207, and zoom drive unit 208 to acquire camera data covering a viewing angle θ across the entire basketball court 108. In the operation unit 801, the CPU 201 controls the NIC 205 to perform operations such as starting or stopping camera recording based on instructions received from the client terminal 107. In the communication unit 802, the CPU 201 controls the NIC 205 and video I / F 206 to send camera data and camera information to the image processing device 103.

[0045] <Functional Configuration of Image Processing Device 103> The image processing device 103 includes a communication unit 803, a processing / computing unit 804, an inference unit 805, and a data storage unit 806. These functional units are implemented by a CPU 210 loading a program stored in ROM 212 into RAM 211 and executing that program. In the communication unit 803, the CPU 210 controls the video I / F 214 and NIC 215 to receive data from the overhead camera 102 and the training server 105, or to send data to the observation camera 104. In the processing / computing unit 804, the CPU 210 performs image processing on the camera data received from the overhead camera 102, such as downscaling and Hough transform. Furthermore, in the processing / computing unit 804, the CPU 210 extracts an image from the camera data based on the result of inference performed by the inference unit 805, which will be described later. Additionally, the processing / computing unit 804 determines whether a shooting scene exists in the extracted image, or calculates control values ​​for the observation camera 104, based on the extracted image.

[0046] (Object detection model inference and pose estimation model inference) In the inference unit 805, the CPU 210 controls the GPU 213 to perform inference processing for object detection and pose estimation on the "camera data" received from the overhead camera 102 using the "trained model" trained by the training server 105. It should be noted that in the "object detection model inference," image data including the ball 110 is set as input data, and the "rectangular coordinates" of the circumscribed ball 110 in the aforementioned "camera data" are set as output data. On the other hand, in the "pose estimation model inference," image data including the player 109 is set as input data, and the "joint coordinates" of the player 109 in the aforementioned "camera data" are set as output data. In the data storage unit 806, the CPU 210 controls the HDD 216 to store and manage the trained model received from the training server 105.

[0047] <Functional Configuration of Camera 104> The focus camera 104 includes an image capture unit 807, a drive control unit 808, and a communication unit 809. These functional units are implemented by a CPU 218 loading a program stored in ROM 220 into RAM 219 and executing that program. In the image capture unit 807, the CPU 218 controls the video engine 221, the image sensor 224, and the zoom drive unit 225 to capture images with a view covering the player 109 and the ball 110. In the drive control unit 808, the CPU 218 controls the pan / tilt drive unit 226 and the zoom drive unit 225 to adjust the rotation of the pan / tilt mechanism and the zoom of the zoom lens according to instructions received from the image processing device 103. In the communication unit 809, the CPU 218 controls the NIC 205 and the video I / F 206 to receive various control values ​​for the pan / tilt drive unit 226 and the zoom drive unit 225 from the image processing device 103.

[0048] <Functional Configuration of Data Collection Server 106> The data collection server 106 includes a data collection / providing unit 813 and a data storage unit 814. These functional units are implemented by the CPU 236 loading the program stored in the ROM 238 into the RAM 237 and executing the program.

[0049] (Supervised data used for "object detection" and "pose estimation") When the CPU 236 controls the NIC 241, the data collection / providing unit 813 has the function of collecting "supervisory data" requested by the training server 105 from the Internet 100, and sending the collected "supervisory data" to the training server 105. It should be noted that in this embodiment, an image including the ball 110 and the "position coordinates" of the ball 110 in the image are used as "supervisory data" for object detection. Furthermore, as "supervisory data" for pose estimation, a "training dataset for pose estimation" is used, which includes a combination of an image of a person's entire body and the position coordinates of each joint of the person appearing in the image. In the data storage unit 814, the CPU 236 controls the HDD 240 to store and manage the supervisory data collected from the Internet 100.

[0050] <Functional Configuration of Client Terminal 107> The client terminal 107 includes a display control unit 815, an operation unit 816, and a communication unit 817. These functional units are implemented by the CPU 243 loading the program stored in the ROM 245 into the RAM 244 and executing the program.

[0051] In the display control unit 815, the CPU 243 controls the display unit 249 to display image data received from the image processing device 103 on the display unit 249. In the operation unit 816, the CPU 243 controls the input unit 250 to receive operations or instructions from the user using the image processing system. In the communication unit 817, the CPU 243 controls the NIC 248 to send control commands to devices connected to the Internet 100 and the local network 101.

[0052] <Functional Configuration of Training Server 105> The training server 105 includes a training data generation unit 810, a training unit 811, and a data storage unit 812. These functional units are implemented by the CPU 229 loading the program stored in the ROM 231 into the RAM 230 and executing the program.

[0053] In the training data generation unit 810, the CPU 229 generates training data by performing data augmentation on the "supervised data" received from the data collection server 106. In the "data augmentation," firstly, image transformation processing, such as affine transformation, is performed on the image of the supervised data and the image of the corresponding ground truth data. As a result, an image different from the original image is generated. Furthermore, information related to the ground truth data is transformed using the image transformation, and the amount of training data is increased.

[0054] In the training unit 811, the CPU 229 controls the GPU 232 to train both the "trained model for object detection" and the "trained model for pose estimation," and generates individual trained models for object detection and pose estimation. It should be noted that examples of specific algorithms for machine learning using the training models include nearest neighbor methods, Naive Bayes methods, decision trees, and support vector machines. Furthermore, deep learning, which automatically generates feature quantities and connection weighting coefficients used for learning using neural networks, is also considered an example of a training model. From the above algorithms, suitable algorithms can be selected, and these suitable algorithms can be applied to this embodiment.

[0055] <The "Error Detection Department" and "Update Department" included in Training Department 811> It should be noted that the training unit 811 may include an "error detection unit" and an "update unit". The "error detection unit" obtains the error between the supervision data and the output data output from the output layer of the neural network in response to the input data input to the input layer. Furthermore, the "error detection unit" can use a loss function to calculate the error between the supervision data and the output data from the neural network. The "update unit" updates the connection weighting coefficients between the nodes of the neural network based on the error calculated by the error detection unit in a way that reduces the error. The "update unit" may, for example, use an error backpropagation method to update the connection weighting coefficients. Here, the error backpropagation method is a method used to adjust the connection weighting coefficients between the nodes of each neural network to reduce the error.

[0056] In the data storage unit 812, the CPU 229 controls the HDD 233 to store and manage the supervisory data received from the data collection server 106 and the trained model trained by the training unit 811.

[0057] <Prerequisites for the Process> Next, before describing the specific processing flow in this image processing system, the conditions and operations that constitute the processing flow will be described. The processing flow in the image processing system is executed assuming that the "object detection model" and "pose estimation model" trained by the training server 105 are already stored in the HDD 216 of the image processing device 103. Therefore, the image processing device 103 does not need to request the trained model each time it performs processing according to this embodiment, and can use the "trained model" stored in the HDD 216 to perform each inference process.

[0058] Here, using an "object detection model" as an example, the method for preparing a trained model is described. The basic operations of the client terminal 107, data collection server 106, and image processing device 103 are also the same when preparing the trained model.

[0059] Before commencing processing according to this embodiment, the CPU 243 of the client terminal 107 requests the following from the CPU 229 of the training server 105 via the NIC 248 of the client terminal 107. Specifically, the CPU 243 of the client terminal 107 requests the generation of an "object detection model" that detects the sphere 110 from the zoomed-out image signal 604 of the overhead image signal 600.

[0060] Next, the CPU 229 of training server 105 requests the CPU 236 of data collection server 106 to collect "supervision data" including ball 110 via the NIC 234 of training server 105. Then, the CPU 236 of data collection server 106 collects the "supervision data" including ball 110 via Internet 100.

[0061] Next, the CPU 236 of the data collection server 106 sends the collected supervision data to the training server. The CPU 229 of the training server 105 generates training data by performing data augmentation on the supervision data received from the data collection server 106. Then, the CPU 229 of the training server 105 controls the GPU 232 of the training server 105 to generate an "object detection model" based on the generated training data. The CPU 229 of the training server 105 then sends the generated trained model to the image processing device 103. The CPU 210 of the image processing device 103 stores the "trained model" received from the training server 105 into the HDD 216.

[0062] The above describes the method for preparing the "object detection model" for detecting ball 110. It should be noted that the method for creating the "pose estimation model" can be implemented as follows: prepare "image data" and "joint coordinate data" of player 109 as "supervised data" using the method described above, and then train the network model for pose estimation.

[0063] ( Figures 4A to 4C : A flowchart illustrating the processing procedures performed by an overhead camera, image processing equipment, and a focus camera. Figures 5A to 5C (Illustrative diagram of overhead image signal, focus area image signal, and zoomed-out image signal) Next, refer to Figures 4A to 5C Describe the specific processing to be performed in this system. It should be noted that... Figure 5A , Figure 5B and Figure 5C This is an illustrative example for reference. Figure 4A , Figure 4B and Figure 4C An explanatory diagram of the content to be described. Figure 4A This is a flowchart illustrating the processing to be performed by the overhead camera 102, and also illustrating the processing to be performed by... Figure 3 The processing is performed by the camera unit 800, the operation unit 801, and the communication unit 802. See below for reference. Figure 4A The description is of the processing to be performed by the overhead camera 102.

[0064] ( Figure 4A (Processing to be performed by the overhead camera 102) First, CPU 201 reads the software for camera unit 800 from ROM 203, loads the software into RAM 202, and executes step S300. Specifically, in step S300, CPU 201 controls video engine 204 and image sensor 207, and generates camera data with a viewing angle θ covering the entire basketball court 108 (this data will be referred to below as "overlooking image signal 600"). It should be noted that... Figure 5A This is an illustration of the overhead view signal 600. Afterwards, the CPU 201 stores the generated overhead view signal 600 in the RAM 202 and proceeds to step S301.

[0065] Next, CPU 201 reads the software for communication unit 802 from ROM 203, loads the software into RAM 202, and executes step S301. Specifically, in step S301, CPU 201 sends the overhead image signal 600 stored in RAM 202 in step S300 to image processing device 103 via video I / F 206. Then, CPU 201 proceeds to step S302.

[0066] Next, CPU 201 reads the software for operation unit 801 from ROM 203, loads the software into RAM 202, and executes step S302. Specifically, in step S302, CPU 201 determines whether there is an operation from the user interface (UI) (not shown) of the overhead camera 102, or whether there is an instruction to end recording from the client terminal 107 via NIC 205. If it is determined that there is an instruction to end recording ("Yes"), CPU 201 ends recording. On the other hand, if it is determined that there is no instruction to end recording ("No"), CPU 201 returns the process to step S300 and repeats the processes in steps S300 and S301.

[0067] ( Figure 4B (Processing to be performed by image processing device 103) Figure 4B This is a flowchart illustrating the processing to be performed by the image processing device 103, and it is also an example of the processing to be performed by... Figure 3 The diagram illustrates the processes performed by the communication unit 803, processing / computing unit 804, inference unit 805, and data storage unit 806. See below for reference. Figure 4B The process to be performed by the image processing device 103 is described. First, the CPU 210 reads the software for the communication unit 803 from the ROM 212, loads the software into the RAM 211, and executes step S400. Specifically, in step S400, the CPU 210 determines whether the overhead image signal 600 sent from the overhead camera 102 in step S301 has been received via the video I / F 214 (see [link to video I / F]). Figure 5A If it is determined that the overhead view signal 600 has been received ("Yes"), the CPU 210 stores the overhead view signal 600 in RAM 211 and proceeds to step S401. On the other hand, if it is determined that the overhead view signal 600 has not been received ("No"), the CPU 210 waits for the reception of the overhead view signal 600 in step S400.

[0068] Next, the CPU 210 reads the software for the processing / computing unit 804 from the ROM 212, loads the software into the RAM 211, and executes step S401. Specifically, in step S401, the CPU 210 reads the overhead view signal 600 stored in the RAM 211 in step S400, and generates a reduced image signal 604 obtained by reducing the overhead view signal 600 (see...). Figure 5C It should be noted that the factor used for downscaling (hereinafter referred to as the "downscaling factor") is a factor suitable for inputting the trained model to be used in the inference processing of subsequent stages, but it can be any factor specified by the user. The CPU 210 stores the generated downsized image signal 604 and the downscaling factor in RAM 211 and proceeds to step S402.

[0069] Next, CPU 210 reads the software for inference unit 805 from ROM 212, loads the software into RAM 211, and executes step S402. Specifically, in step S402, CPU 210 reads the "object detection model" from HDD 216 and stores the read model into RAM 211. Then, CPU 210 controls GPU 213 to read the reduced image signal 604 stored in RAM 211 in step S401, and inputs the reduced image signal 604 into the "object detection model" stored in RAM 211 to obtain "spherical coordinates 602c". Then, CPU 210 stores the detected spherical coordinates 602c into RAM 211 and proceeds to step S403. It should be noted that if no spherical coordinates are detected, the most recently detected spherical coordinates can be used as the next spherical coordinates. Alternatively, in the absence of detected spherical coordinates, the trajectory can be predicted based on the time series of spherical coordinates detected to date, and the predicted coordinates calculated from the prediction results can be used as spherical coordinates.

[0070] Next, CPU 210 reads the software for the processing / computing unit 804 from ROM 212, loads the software into RAM 211, and executes step S403. Specifically, in step S403, CPU 210 reads the spherical coordinates 602c and the scaling factor already stored in RAM 211 in step S402, and uses the scaling factor to transform the spherical coordinates 602c in the scaled image signal 604 into spherical coordinates 602a in the overhead image signal 600. Then, CPU 210 stores the transformed spherical coordinates 602a into RAM 211. Afterwards, based on the spherical coordinates 602a stored in RAM 211, CPU 210 crops the area including the sphere 110 and the area near the sphere from the overhead image signal 600, and generates a focus area image signal 601 (see See...). Figure 5B It should be noted that the process of determining the cutting size in step S403 is a characteristic of the present invention, and will be referred to later. Figure 6 and Figure 7 A detailed description is provided. Afterwards, the CPU 210 stores the generated attention area image signal 601 in RAM 211 and proceeds to step S404.

[0071] Here, the attention area image signal 601 is an image corresponding to the "attention area size" (which is the size of the "attention area"). The "attention area" is a region in the captured image corresponding to the location (basketball court 108) being captured by the overhead camera 102. The CPU 210 (determining unit) determines the "attention area" based on distortion information and attention position (spherical coordinates 602). Furthermore, the CPU 210 determines the "attention area size" corresponding to the actual size based on the "distortion information" and the "attention position". Moreover, as will be apparent in the following description, the "distortion information" is information used to indicate the distortion of the shape of the first object (basketball court 108) in the captured image taken by the overhead camera 102 compared to a known shape, and the CPU 210 (distortion information generation unit) generates the distortion information. Attention objects (a second object and a third object) are present in the attention area image signal 601.

[0072] In other words, in order to accommodate the horizontally elongated rectangular basketball court 108 within the viewing angle θ of the overhead camera 102, such as Figure 5A and Figure 11A As illustrated, the captured image is distorted into a quadrilateral shape (trapezoidal shape, etc.). Therefore, as an example (third embodiment), the CPU 210 generates "distortion information" based on the projective transformation matrix between the coordinates of the vertices indicating the first object (basketball court 108) and the coordinates of the vertices indicating the known shape of the first object. Here, the CPU 210 calculates the "attention position (spherical coordinates 602)" in the captured image. Then, the CPU 210 determines the "attention area" based on the "distortion information" and the calculated "attention position". Next, the CPU 210 determines the "attention area size" (which is the size of the attention area). The image corresponding to the "attention area size" is the "attention area image (attention area image signal 601)". In embodiments of the present invention, the "attention area" is specifically the camera area used to capture a shooting scene or a ball, etc. It should be noted that the CPU 210 can also determine the “area of ​​attention” based on positional information (joint coordinates) related to a second object (player 109), wherein the position of the second object is estimated in an image captured by the overhead camera 102.

[0073] Then, after executing step S403, CPU 210 reads the software for the processing / computing unit 804 from ROM 212, loads the software into RAM 211, and executes step S404. Specifically, in step S404, CPU 210 reads the region of attention image signal 601 stored in RAM 211 in step S403 and scales the region of attention image signal 601 to match the size of the input to the pose estimation model. It should be noted that the scaling factor is suitable for the input to the trained model to be used in the inference processing, but can be any factor specified by the user. Afterwards, CPU 210 stores the scaled region of attention image signal 601 into RAM 211 and proceeds to step S405. As the region of attention image signal 601 is scaled (enlarged or reduced), the resolution of the region of attention image signal 601 changes.

[0074] Next, CPU 210 reads the software for inference unit 805 from ROM 212, loads the software into RAM 211, and executes step S405. Specifically, in step S405, CPU 210 reads the "pose estimation model" from HDD 216 and stores the read model into RAM 211. Subsequently, in step S404, CPU 210 controls GPU 213 to read the attention area image signal 601 stored in RAM 211 and uses the "pose estimation model" stored in RAM 211 to detect the "joint coordinates 603" of player 109a (see [link to relevant documentation]). Figure 5B It should be noted that when multiple players 109 appear in the attention area image signal 601, joint coordinates 603 are detected for each player 109. The CPU 210 stores the detected joint coordinates 603 in RAM 211 and proceeds to step S406.

[0075] Next, the CPU 210 reads the software for the processing / computing unit 804 from the ROM 212, loads the software into the RAM 211, and executes step S406. Specifically, in step S406, the CPU 210 determines whether the player 109a is shooting based on the ball coordinates 602a stored in RAM 211 in step S403 and the joint coordinates 603 stored in RAM 211 in step S405. Specifically, the CPU 210 transforms the coordinate system of the ball coordinates 602a from the coordinate system in the overhead image signal 600 to the coordinate system in the attention area image signal 601, and stores the transformed ball coordinates 602b into RAM 211.

[0076] Then, the CPU 210 reads the spherical coordinates 602b and joint coordinates 603 stored in RAM 211 and compares the positional relationships between the read coordinates. For example, if the CPU 210 detects that the coordinate value of the elbow joint is higher than the coordinate value of the shoulder joint of player 109a, and the coordinate value of the wrist joint is higher than the coordinate value of the elbow joint, it determines that player 109a is in a state of extending his arm in the upward direction. Here, a higher "coordinate value" means a larger coordinate value on the vertical coordinate axis.

[0077] Furthermore, if the coordinate value of ball coordinate 602b is higher than the coordinate value of player 109a's wrist joint, CPU 210 determines that player 109a is shooting. These determinations are performed on the number of players 109 whose joint coordinates 603 have been detected, and the joint coordinate 603 that makes the center of gravity of ball coordinate 602b closest to the coordinate value of player 109's wrist joint is selected. In this way, CPU 210 can determine whether a shot is in progress.

[0078] It should be noted that in this embodiment, the determination of whether a player has taken a shot is made by comparing the ball coordinates and the player's joint coordinates, but the determination method is not limited to this. For example, a "decision tree model" can be prepared (an image of player 109a in motion (such as attention area image signal 601, etc.) is input into the "decision tree model," and the "decision tree model" outputs a determination result regarding whether a shot has been taken), and the determination result made by the decision tree model can be used to determine whether a shot has been taken. Furthermore, in this embodiment, the type of action to be determined for the purpose of capturing a scene of a player taking a shot is set to "taking a shot," and the action is not limited to any specific type. For example, the present invention can be applied to the determination of any action that can be identified from the player's posture (such as "passing" or "dribbling," etc.). If it is determined that player 109a is taking a shot ("yes"), the CPU 210 proceeds to step S407. On the other hand, if it is determined that player 109a is not taking a shot ("no"), the CPU 210 proceeds to step S408.

[0079] Next, based on the result of the processing in step S406, the CPU 210 reads the software for the processing / computation unit 804 from the ROM 212, loads the software into the RAM 211, and executes step S407. Specifically, in step S407, the CPU 210 calculates the control value for the pan / tilt drive unit 226 of the focus camera 104 to set the camera direction of the focus camera 104 toward the shooting scene. Here, in this embodiment, the "control value" is a combination of "control direction" and "control speed". For example, the control value for the pan drive is expressed as "a speed of 10 (degrees / second) in the right direction", etc., and the pan control speed is determined based on the difference between the current position for panning and the focus position. It should be noted that the focus position for panning can be directly specified as the control value. In this way, in step S407, the CPU 210 generates a camera control signal for pointing the focus camera 104 toward the shooting scene.

[0080] When the shooting direction and position of the overhead camera 102 are significantly different from those of the observation camera 104, the positions of the overhead camera 102 and the observation camera 104 can be pre-calibrated. In this case, the spherical coordinates 602a and the player's joint coordinates 603 as seen from the overhead camera 102 can be transformed into the coordinate system of the observation camera 104. Therefore, the present invention can be applied even when the shooting direction and position of the overhead camera 102 are significantly different from those of the observation camera 104. It should be noted that an example of the position calibration method described herein is a method in which a calibration pattern, such as a grid pattern, is simultaneously captured by cameras located at two points, thereby calculating camera parameters that include information related to the relative position and orientation of the cameras. Furthermore, the corresponding points between the shooting coordinates of the overhead camera 102 and the observation camera 104 can be calculated using the calculated camera parameters based on epipolar geometry.

[0081] Next, the CPU 210 performs the following calculations based on the spherical coordinates 602a stored in RAM 211 and the size of the attention area (hereinafter referred to as the size of the attention area) of the attention area image signal 601. That is, the CPU 210 calculates the control value for the zoom drive unit 225 of the attention camera 104 to keep the ball 110 and the player 109 located nearby within the field of view θ.

[0082] Here, a specific process for calculating control values ​​for lens drive is described. First, the maximum and minimum focal lengths of the focus camera 104 in the camera are defined. Specifically, before the processing according to this embodiment begins, the user measures the focal length T (maximum value), where, with this focal length T (maximum value), the entire body of the ball 110 and player 109 falls within the viewing angle θ when the ball 110 is at its deepest position in the basketball court 108. Furthermore, the user measures the focal length T (minimum value), where, with this focal length T (minimum value), the entire body of the ball 110 and player 109 falls within the viewing angle θ when the ball 110 is at its foremost position in the basketball court 108. Here, at the "deepest position" and "foremost position," when the focus camera 104 is mounted near the overhead camera 102, which has an optical axis perpendicular to the sideline of the basketball court 108, the image captured by the focus camera 104 can be approximated by the overhead image signal 600. In this case, the upper edge of the overhead image signal 600 is in the "deepest position," and the lower edge of the overhead image signal 600 is in the "foremost position." Furthermore, in situations such as... Figure 1 In the illustrated case where the optical axis of the focus camera 104 is set perpendicular to the end lines of the basketball court 108, when viewed from the front side overlooking the camera 102, the left end line is in its "deepest position" and the right end line is in its "foremost position." It should be noted that although the focal lengths T and W are set such that the entire body of the ball 110 and the player 109 falls within the viewing angle θ, the focal length is not necessarily limited to these lengths and can be set by the user to any focal length.

[0083] The measured focal lengths T and W are input to the client terminal 107, then sent to the image processing device 103, and stored in RAM 211. Next, the CPU 210 transforms the mounting position, deepest position, and foremost position of the observation camera 104 into the coordinate system of the overhead image signal 600 based on the position calibration results. The CPU 210 then defines the distance from the observation camera 104 to the deepest position in the overhead image signal 600 as "LL1", the distance from the observation camera 104 to the foremost position as "LL2", and the distance from the observation camera 104 to the spherical coordinate system 602a as "LL3". The CPU 210 then calculates the target zoom position Z associated with the distance from focal length T to focal length W according to the following expression 1.

[0084] [Mathematical Expression 1] …expression 1

[0085] The lens drive control direction is determined based on the current zoom position and the target zoom position Z. For example, if the current zoom position is "30 (mm)" and the attention position is "100 (mm)", the control direction is "telephoto (TELE) direction". When the zoom drive unit 225 operates to capture a large image of the subject, telephoto is referred to as "telephoto (tele) direction". On the other hand, the lens drive control speed is determined based on the magnitude of the difference between the current zoom position and the target zoom position Z. For example, if the difference DD between the current zoom position and the target zoom position is less than "10 (mm)", the control speed is set to "0 (mm / sec)". If the difference DD is "10 mm or greater", the control speed is set to "(DD×10)(mm / sec)". Each of these values ​​is an example, and the control is not limited. For example, an upper limit can be set for the control speed.

[0086] It should be noted that in this embodiment, the control value for zoom is calculated based on the mounting position of the focus camera 104, but the calculation process is not limited to this. For example, a new path can be established to feed the image captured by the focus camera 104 to the image processing unit 103, and object detection and pose estimation can be performed on the image captured by the focus camera 104 in the same manner as the overhead image signal 600. Then, the positions of the player 109 and the ball 110 within the viewing angle θ of the focus camera 104 can be estimated, and zoom control can be performed so that the ball 110 and the entire body of the player 109 located nearby fall within the viewing angle θ. Furthermore, in this embodiment, the control value for the zoom drive unit 225 is the control value required to keep the ball 110 and the entire body of the player 109 located nearby within the viewing angle θ. However, the control value for the zoom drive unit 225 is not limited to this. For example, in the case of shooting a player in an image with a composition focused on the upper body, the control value can be calculated as described below. In other words, the joint coordinates 603 of player 109 can be transformed from the coordinate system in the attention area image signal 601 to the coordinate system in the overhead image signal 600, and can be set to the control values ​​required to keep the joint coordinates of the body parts above the waist joint of player 109 within the viewpoint θ.

[0087] Furthermore, after executing step S407, CPU 210 stores the control values ​​(hereinafter referred to as "camera control signals") of the pan / tilt drive unit 226 and zoom drive unit 225 calculated in step S407 into RAM 211, and proceeds to step S409. Based on the results of the processing in step S406, CPU 210 reads the software for the processing / computing unit 804 from ROM 212, loads the software into RAM 211, and executes step S408. Specifically, in step S408, CPU 210 calculates the control values ​​of the pan / tilt drive unit 226 for the focus camera 104 based on the spherical coordinates 602a stored in RAM 211 in step S402, so as to set the imaging direction of the focus camera 104 toward the spherical coordinates 602a. In addition, the CPU 210 calculates control values ​​for the zoom drive unit 225 of the focusing camera 104 based on the spherical coordinates 602a stored in RAM 211, so as to keep the ball 110 and the "predetermined area" around the ball 110 within the viewing angle θ.

[0088] Here, the "predetermined area" can be the region that includes all players whose joint coordinates 603 have been detected in the attention area image signal 601. Alternatively, the "predetermined area" can be the region that includes only the player closest to the ball coordinate 602a. It should be noted that the processing for calculating the control values ​​of the pan / tilt drive unit 226 and the zoom drive unit 225 is the same as the processing in step S407.

[0089] It should be noted that even if the scene is determined not to be a shooting scene in step S406, in step S408, the camera direction of the focus camera 104 is still set to face the ball coordinate 602a because the shooting occurs from the position of the ball used as the starting point. As a result, when a shooting scene occurs, the drive amount of the focus camera 104 can be minimized, and the time delay until the focus camera 104 is fully facing the direction of the shooting scene can be reduced. Afterwards, the CPU 210 stores the camera control signal calculated in step S407 into RAM 211 and proceeds to step S409. That is, in both steps S407 and S408, the CPU 210 generates camera control signals based on the ball coordinate 602a and joint coordinate 603 in step S406. These camera control signals are used to point the focus camera 104 in the direction corresponding to the recognition result regarding whether the player 109a has made a shot. Since a camera control signal for pointing the attention camera 104 at the shooting scene is generated in step S407, and a camera control signal for pointing the attention camera 104 at spherical coordinates is generated in step S408, a camera control signal for setting the shooting direction of the attention camera 104 to face the attention area is generated in either case.

[0090] Next, the CPU 210 reads the software for the communication unit 803 from the ROM 212, loads the software into the RAM 211, and executes step S409. Specifically, in step S409, the CPU 210 sends the camera control signal stored in the RAM 211 in step S407 to the focus camera 104 via the NIC 215.

[0091] ( Figure 4C (Processing to be performed by the attention camera 104) Figure 4C This is a flowchart illustrating the processing to be performed by the attention camera 104, and it is to be performed by... Figure 3 An explanatory diagram illustrating the processing performed by the camera unit 807, drive control unit 808, and communication unit 809. See below for reference. Figure 4C The description is of the processing to be performed by the attention camera 104.

[0092] First, the CPU 218 reads the software for the communication unit 809 from the ROM 220, loads the software into the RAM 219, and executes step S500. Specifically, in step S500, the CPU 218 determines whether the "camera control signal" sent from the image processing device 103 in step S409 has been received via the NIC 222. If it is determined that the camera control signal has been received ("yes"), the CPU 218 stores the received camera control signal into the RAM 219 and proceeds to step S501. On the other hand, if it is determined that the camera control signal has not been received ("no"), the CPU 218 waits to receive it in step S500.

[0093] Next, the CPU 218 reads the software for the drive control unit 808 from the ROM 220, loads the software into the RAM 219, and executes step S501. Specifically, in step S501, the CPU 218 controls the pan / tilt drive unit 226 and the zoom drive unit 225 based on the camera control signals stored in the RAM 219 in step S500.

[0094] Next, the CPU 218 reads the software for the camera unit 807 from the ROM 220, loads the software into the RAM 219, and executes step S502. Specifically, in step S502, the CPU 218 controls the video engine 221 and the image sensor 224, and generates camera data having a viewing angle θ including the player 109 and the ball 110 located in the camera direction. That is, in step S502, the camera unit 807 captures a focus area that includes the shooting scene being performed by the player 109 located in the camera direction and the position of the ball located in the camera direction.

[0095] ( Figure 6 Example flowchart of image generation process for the area of ​​focus / Figure 7 (Illustration diagram) Next, refer to Figure 6 and Figure 7 The processing of generating the image signal 601 of the region of interest, which is a processing characteristic of this system, is described. It should be noted that... Figure 7 This is an illustrative example for reference. Figure 6 An explanatory diagram of the content to be described. Figure 6 This is a flowchart illustrating the detailed processing in step S403 (subroutine) of the flowchart illustrating the processing to be performed by the image processing device 103. See below for reference. Figure 6 The description is of the processing to be performed by the image processing device 103.

[0096] First, in step S700, the CPU 210 performs a calculation based on Hough transform processing on the overhead view image signal 600 stored in RAM 211. Here, the quadrilaterals formed by the resulting straight lines are considered as the court lines of the basketball court 108. It should be noted that if there are court lines in the overhead view image signal 600 with a color different from the court lines of the basketball court 108, the calculation can be performed after specifying the color of the court lines of the basketball court 108. Furthermore, if the floor color is different between the inside and outside of the basketball court 108, the boundary line in the floor color can be used as the court line. The CPU 210 stores the determined court lines of the basketball court 108 in RAM 211 and proceeds to step S701. It should be noted that the court line calculation processing is not limited to this, and for example, the court lines can be determined by performing edge extraction processing on the overhead view image signal 600 with a SOBEL filter or the CANNY method to calculate the boundaries of the quadrilaterals. In this way, in step S700, CPU210 performs a Hough transform on the overhead image signal 600 to extract the court lines of basketball court 108.

[0097] Next, in step S701, CPU 210 obtains the quadrilateral formed by the court lines of the basketball court 108 stored in RAM 211 in step S700, and calculates the coordinate values ​​of the four vertices of the quadrilateral (these coordinate values ​​will be referred to as "court coordinates" below). Here, as Figure 7 As illustrated, the four vertices of the quadrilateral are the top-left vertex "A", the bottom-left vertex "B", the bottom-right vertex "C", and the top-right vertex "D". That is, the quadrilateral formed based on the field lines is quadrilateral ABCD. The CPU 210 stores the calculated field coordinates in RAM 211 and proceeds to step S702.

[0098] Next, in step S702, the CPU 210 calculates the "upper base L1" formed by line segment AD of quadrilateral ABCD and the "lower base L2" formed by line segment BC, based on the site coordinates stored in RAM 211 in step S701 (see...). Figure 7 Then, the CPU 210 stores the calculated positions and lengths of the upper base L1 and the lower base L2 into the RAM 211 and proceeds to step S703.

[0099] Next, in step S703, based on the spherical coordinates 602 stored in RAM 211 and the positions of the upper base L1 and lower base L2 stored in RAM 211 in step S701, CPU 210 calculates a straight line parallel to the upper base L1 and lower base L2 and passing through the centroid of the spherical coordinates 602. It should be noted that the case where the upper base L1 and lower base L2 in the overhead image signal 600 are not parallel will be described in the third embodiment. Next, CPU 210 calculates line segment PQ, whose intersection point with line segment AB is "P" and whose intersection point with line segment CD is "Q", as "line segment L3" (see...). Figure 7 The CPU 210 stores the calculated length of line segment L3 in RAM 211 and proceeds to step S704.

[0100] Next, in step S704, based on the length of the lower base L2 stored in RAM 211 in step S702 and the length of the line segment L3 stored in RAM 211 in step S703, the CPU 210 calculates the "attention area size" (which is the image size of the attention area image signal 601) according to the ratio between the lengths of the various line segments. Specifically, before the processing according to this embodiment begins, the user takes a picture of the player 109 existing in Figure 1The CPU 210 takes an image of the player 109 from the top-down view signal 600 at the bottom edge of the image and measures the full-body size of the player 109 in that image. Then, the CPU 210 stores the full-body size input by the user via the client terminal 107 into RAM 211 and generates a "reference attention area size" based on this full-body size. Here, the "reference attention area size" is the length of one side of the quadrilateral forming the "attention area," with a vertical length twice the height of the player 109 in the top-down view signal 600, and a ratio of 1:1 between the horizontal and vertical lengths. It should be noted that the "reference attention area size" is not limited to this and can be any size that allows for cropping of the player taking a shot without cutting them out.

[0101] Then, CPU 210 stores the generated "reference attention area size" in RAM 211. CPU 210 calculates the attention area size based on the reference attention area size stored in RAM 211 and the ratio between the lower base L2 and the line segment L3. With the reference attention area size defined as "SB", the length of the lower base L2 defined as "L2", and the length of the line segment L3 defined as "L3", the attention area size S is calculated according to the following expression 2.

[0102] [Mathematical Expression 2] …expression2

[0103] Next, CPU 210 stores the size of the attention area in RAM 211 and proceeds to step S705. Through the above processing, as the distance from the overhead camera 102 to the sphere 110 increases, the size of the attention area decreases. Conversely, as the distance from the overhead camera 102 to the sphere 110 decreases, the size of the attention area increases.

[0104] Next, in step S705, CPU 210 calculates the "attention area coordinates 701" based on the spherical coordinates 602 stored in RAM 211 and the attention area size stored in RAM 211 in step S704 (see...). Figure 7 Here, "attention area coordinates 701" refers to the coordinates of the upper left and lower right vertices of the following region (or a rectangular region obtained by magnifying the rectangular region by a preset magnification value in four directions), wherein the centroid of the spherical coordinates 602 is used as the center, and one side of the rectangle has the size of the attention area. The CPU 210 stores the calculated attention area coordinates 701 in RAM 211 and causes the processing to proceed to step S706.

[0105] Next, in step S706, CPU 210 crops the "attention region" from the overhead image signal 600 based on the attention region coordinates 701 stored in RAM 211 in step S705, and generates the attention region image signal 601. Afterwards, CPU 210 stores the generated attention region image signal 601 in RAM 211. The above is a description of step S403 of the processing to be performed by image processing device 103. Subsequent processing continues to step S404 to be performed by image processing device 103 (see...). Figure 4B ).

[0106] The above is about utilizing Figure 1 The illustrated configuration is based on the description of the processing to be performed according to the first embodiment. As described above, the player (second object) in the captured image can be cropped based on the size of the position of the ball (third object) in the basketball court 108 (first object). Therefore, the possibility that the player is cut off due to the cropping size being too small relative to the player's size, or that the subject is flattened due to the cropping size being too large, thus preventing the intended recognition, can be reduced. It should be noted that the scope of application of the present invention is not limited to the above, and the present invention can be applied to various scenarios other than basketball, such as live music performances and lectures in auditoriums. For example, the present invention is suitable for sports such as football and tennis, in which actions such as shooting, serving, and spiking by a player near the ball are recognized.

[0107] It should be noted that although an example of automatic video recording based on video data from a single overhead camera 102 has been described in the first embodiment, the invention is not limited thereto. The invention can also be applied to systems using multiple overhead cameras. In this case, the overhead cameras are positionally calibrated, and the individual video data are integrated based on the calibration results to generate a set of video data. The invention can then be applied to the integrated video data. Therefore, the invention can be readily applied to sports and venues involving large playing fields such as football and rugby.

[0108] Furthermore, in scenarios where the present invention is applied to subjects performing predetermined actions (such as singing or pointing at a blackboard, as in live music performances and lectures in auditoriums), it can sometimes be difficult to detect the site coordinates from the overhead image signal 600. In such cases, the site lines can be calculated by setting target straight lines to enclose the stage, thus making the present invention applicable. Moreover, instead of site lines, markers of specific colors and shapes can be attached to the positions of the vertices of a quadrilateral stage in the real world, and the markers can be calculated based on the image data from the overhead camera 102 to calculate the quadrilateral ABCD forming the stage. That is, the CPU 210 can be designed to detect the markers set at the positions corresponding to the vertices of the basketball court 108 in the real world, and to acquire the area with each marker as a vertex as the object shape of the basketball court 108.

[0109] (Assembly of the position sensor) Furthermore, in the first embodiment, in step S402 of the processing flow to be executed by the image processing device 103, the CPU 210 uses an object detection model to detect the position coordinates of the ball 110. However, the ball detection processing is not limited to this. For example, a position sensor can be attached to the ball 110, and can be... Figure 1 The system configuration includes a position sensor receiving device 115 for receiving signals from a position sensor. The coordinate system of the position sensor receiving device 115 and the coordinate system in the overhead image signal 731 to be captured by the overhead camera 102 should be pre-calibrated. The position sensor receiving device 115 receives the position of a position sensor attached to the sphere 110. The position sensor receiving device 115 sends the position obtained from the position sensor to the image processing device 103. The image processing device 103 transforms the position received from the position sensor by the position sensor receiving device 115 into the coordinate system of the overhead image signal 600 and uses the transformed position of the position sensor as spherical coordinates 602. In this way, a sensor for detecting the position of the sphere 110 can be used. As described above, the CPU 210 can acquire position information related to the sensor attached to the sphere 110 and detecting the position of the sphere 110 (which is a real-world object corresponding to the sphere object (a third object)), and calculate the attention position based on the position information acquired by the sensor.

[0110] Furthermore, in this embodiment, assuming an exciting scene occurs near the ball 110, the position coordinates of the ball 110 are detected from the overhead image signal 600 as the point of interest. However, the invention is not limited to this. For example, instead of the ball 110, the position coordinates of a specific player 109 can be detected as the point of interest. In this case, before the processing according to this embodiment begins, an object detection model can be used that takes image data including player 109 as input and uses the "rectangular coordinates" of the player 109 in the image data as output for inference. Specifically, firstly, in Figure 4B In step S402, the CPU 210 of the image processing device 103 controls the GPU 213 to use a trained model to detect the position coordinates of the player 109 from the reduced image signal 604. Next, in Figure 4B In step S403, the CPU 210 of the image processing device 103 can also perform the processing according to the invention by generating a focus area image signal 601 from the overhead image signal 600 based on the detected position coordinates of the player 109. Therefore, exciting scenes focusing only on the player can be effectively obtained.

[0111] Furthermore, in this embodiment, in step S700 to be performed by the image processing device 103, the CPU 210 uses the Hough transform to calculate the court lines of the basketball court 108. However, the method for calculating the court lines is not limited to this. The present invention can be applied to any calculation mode capable of calculating court lines. For example, a user interface (UI) that allows a user to specify the vertices of the basketball court 108 can be used. Specifically, before the processing according to this embodiment begins, the CPU 243 of the client terminal 107 operates the overhead camera 102 via the NIC 248 of the client terminal 107 to receive the overhead image signal 600. Next, the CPU 243 displays the received overhead image signal 600 on the display unit 249 of the client terminal 107. Next, the CPU 243 receives the operation of the input unit 250 of the client terminal. The user operating the client terminal 107 specifies the four vertices of the basketball court 108 in the displayed overhead image signal 600. Then, the CPU 243 sends the four coordinate sets in the overhead image signal 600 specified via the NIC 248 to the image processing device 103.

[0112] Next, the CPU 210 defines the quadrilateral formed by the four coordinate sets in the overhead image signal 600 received from the client terminal 107 as the court line, making the present invention applicable thereto. Therefore, even when the court line cannot be calculated from the overhead image signal 600, the effect of applying the present invention can still be achieved. Furthermore, for example, the distortion shape in the overhead image signal 600 can be estimated based on the known shape of the basketball court 108 to detect the court line. Specifically, first, the CPU 210 of the image processing device 103 estimates the distortion shape of the basketball court 108 based on the shooting angle of the overhead camera 102 relative to the basketball court 108. The CPU 210 of the image processing device 103 obtains the distorted quadrilateral (which is...) based on the estimation result. Figure 7 The example is a trapezoidal basketball court (108).

[0113] Next, the CPU 210 of the image processing device 103 detects the position and shape of the object, wherein the calculation result obtained by edge calculation processing of the overhead image signal 600 and the distorted quadrilateral has a similarity relationship with the position and shape of the object. Then, the CPU 210 of the image processing device 103 can obtain the court lines based on the detected position and shape of the object, and apply the present invention in the same way. In this way, even if the court lines cannot be calculated based on the overhead image signal 600, the present invention can be applied by calculating the court lines of the basketball court 108. Furthermore, as described above, the "distortion information" is generated by the CPU 210 estimating the distorted shape and performing predetermined detection, etc.

[0114] Next, a second embodiment of the present invention will be described. In the second embodiment, a process for obtaining the shape of the basketball court 108 and generating a focus area image signal 601 by using camera information acquired from the overhead camera 102 is described. Since the basic configuration of the image processing system is the same as that of the first embodiment, repeated descriptions are avoided, and different processes to be performed by the image processing device 103 are described. Furthermore, in the second embodiment, it is assumed that... Figure 1 As illustrated, the imaging optical axis 114 of the overhead camera 102 is parallel to the half-court line of the basketball court 108, and the overhead camera 102 is mounted such that the near sideline of the basketball court 108 falls within the viewing angle θ. That is, the horizontal length of the overhead image signal 600 is the same as or substantially equal to the length of the near sideline of the basketball court 108 in the overhead image signal 600.

[0115] ( Figure 8 (Figure 9: Explanatory Figure) This illustrates a flowchart of the attention region image generation process according to the second embodiment. Figure 8This diagram illustrates the processing of generating a region-of-interest image signal 601 as an example of the processing characteristics of the image processing apparatus 103 according to the second embodiment. Figure 9A and Figure 9B yes Figure 8 The illustrated diagram illustrates the generation process. The process from when the CPU 210 receives the overhead image signal 600 from the overhead camera 102 and zooms out on the overhead image signal 600 until the position of the ball is detected from the zoomed-out image signal 604 is similar to the process up to step S402 in the first embodiment (see [link to diagram]). Figure 4B The same applies. The following description pertains to the processing according to the second embodiment in step S403.

[0116] First, in step S900, the CPU 210 of the image processing device 103 requests the CPU 201 of the overhead camera 102 to send information via NIC 215 indicating the focal length of the lens (not shown) of the overhead camera 102 and the tilt angle of the overhead camera 102. In response, the CPU 201 obtains the focal length and "tilt angle k" of the overhead camera 102 from the zoom drive unit 208 and the tilt sensor 227. Then, the CPU 201 sends the focal length and "tilt angle k" obtained via NIC 205 to the image processing device 103. In response, the CPU 210 obtains the focal length and "tilt angle k" received from the overhead camera 102, stores the focal length and "tilt angle k" in RAM 211, and proceeds to step S901.

[0117] Next, in step S901, CPU 210 calculates the viewing angle θ of the overhead camera 102 based on the focal length stored in RAM 211 in step S900. Then, CPU 210 calculates the "distance d1" from the overhead camera 102 to the nearest sideline based on the viewing angle θ and the length (SL) of the nearest sideline of the basketball court 108 in the real world. Specifically, in step S901, when the length of the nearest sideline of the basketball court 108 in the real world is defined as "SL", the "distance d1" from the overhead camera 102 to the sideline SL is calculated according to the following expression 3.

[0118] [Mathematical Expression 3] …expression 3

[0119] The CPU 210 stores the calculated "distance d1" in RAM 211 and proceeds to step S902. In step S902, the CPU 210 calculates the height h from the plane where the basketball court 108 exists in the real world BR>E up to the overhead camera 102, based on the "distance d1" stored in RAM 211 and the "tilt k" stored in RAM 211 in step S900. Specifically, when the tilt of the overhead camera 102 relative to the horizontal direction is defined as "k" and the distance from the overhead camera 102 to the sideline SL is defined as "d1", the CPU 210 calculates the "height h" of the overhead camera 102 according to the following expression 4.

[0120] [Mathematical Expression 4] …expression4

[0121] CPU 210 stores the calculated "height h" in RAM 211. Based on the "height h" stored in RAM 211, the "distance d1" stored in RAM 211 in step S901, and the length "EL" of the end lines of the basketball court 108 in the real world, CPU 210 calculates the "distance d2" from the overhead camera 102 to the farthest sideline. Specifically, first, CPU 210 defines the distance from the overhead camera 102 to the nearest sideline SL as "d1", the height of the overhead camera 102 as "h", and the length of the end lines of the basketball court 108 in the real world as "EL". Then, CPU 210 calculates the "distance d2" from the overhead camera 102 to the farthest sideline according to the following expression 5. CPU 210 stores the calculated "distance d2" in RAM 211 and proceeds to step S903.

[0122] [Mathematical Expression 5] …expression 5

[0123] Next, in step S903, the CPU 210 calculates the "distance d3" described below based on the "tilt angle k" and "height h", "distance d1" and "distance d2" of the overhead camera 102 stored in RAM 211. That is, the "distance d3" is calculated for a line perpendicular to the line of "distance d1", and the nearest sideline SL of the basketball court 108 in the real world is connected to the intersection with the line of "distance d2". Specifically, the CPU 210 defines the tilt angle of the overhead camera 102 relative to the horizontal direction as "k", the height of the overhead camera 102 as "h", and the distance from the overhead camera 102 to the nearest sideline SL as "d1". Furthermore, when the distance from the overhead camera 102 to the farthest sideline is defined as "d2", the CPU 210 calculates the "distance d3" from the sideline SL to "distance d2" according to the following expression 6. Then, the CPU 210 stores the calculated "distance d3" in RAM 211 and proceeds to step S904.

[0124] [Mathematical Expression 6] …expression 6

[0125] Next, in step S904, based on the "distance d1", "distance d2", and "distance d3" stored in RAM 211, CPU 210 identifies a trapezoid whose upper base, lower base, and height correspond to "distance d1", "distance d2", and "distance d3", respectively. Specifically, the distance from the overhead camera 102 to the closer edge SL is defined as "d1", and the distance from the overhead camera 102 to the farther edge is defined as "d2". At this point in time, the upper and lower bases of the trapezoid calculated from the overhead image signal 600 satisfy the following relational expression 7.

[0126] [Mathematical Expression 7] d1:d2=upper base:lower base… relational expression 7

[0127] Subsequently, the distance from the edge SL to the line segment "distance d2" is defined as "d3". At this point in time, the height of the trapezoid calculated from the overhead image signal 600 satisfies the following relational expression 8.

[0128] [Mathematical Expression 8] SL:d3=lower base L2:height…relational expression 8

[0129] Based on the identified trapezoid, CPU 210 then calculates the coordinates of each vertex of the identified trapezoid in the overhead image signal 600 (hereinafter referred to as "trapezoidal coordinates"). CPU 210 stores the calculated trapezoidal coordinates in RAM 211. CPU 210 is based on the method described in the first embodiment. Figure 6 In step S703, the processing calculates line segment L3 based on the spherical coordinates 602 stored in RAM 211 and the positions of the upper and lower bases of the trapezoid calculated according to the trapezoidal coordinates. CPU 210 stores the calculated length of line segment L3 in RAM 211 and proceeds to step S905. Figure 7 As illustrated, line segment L3 is a line segment PQ that is parallel to the upper and lower bases of the identified trapezoid and extends through the centroid of the sphere's coordinates.

[0130] Next, in step S905, CPU 210 performs the same processing as in step S704 of the first embodiment to calculate the "attention area size". That is, CPU 210 calculates the attention area size based on the ratio between the length of the lower base L2 and the length of the line segment L3. Here, the lower base L2 is... Figure 7 The illustrated lower base. The CPU 210 stores the calculated attention area size in RAM 211 and proceeds the processing to step S906.

[0131] Next, in step S906, CPU 210 performs the same processing as in step S705 of the first embodiment to calculate the coordinates 701 of the attention region. That is, CPU 210 calculates the coordinates 701 of the attention region based on spherical coordinates 602 and the size of the attention region. CPU 210 stores the calculated coordinates 701 of the attention region in RAM 211 and proceeds to step S907.

[0132] Next, in step S907, the CPU 210 performs the same processing as in step S706 of the first embodiment to generate a region of attention image signal 601. That is, the CPU 210 calculates the region of attention image signal 601 based on the region of attention coordinates 701. Then, the CPU 210 stores the generated region of attention image signal 601 in RAM 211. The above is a description of the processing of generating the region of attention image signal 601 as a processing feature of the image processing device 103 in the second embodiment. The subsequent processing is the same as the processing to be performed by the image processing device 103 of the first embodiment (see...). Figure 4B After step S404 in )

[0133] The above is about Figure 1The illustrated image processing system configuration describes the processing to be performed according to the second embodiment of the present invention. As described above, the object shape of the basketball court 108 can be calculated based on camera information obtained from the overhead camera 102. Therefore, even if the object shape of the basketball court 108 cannot be detected from the overhead image signal 600 by the processing performed by the image processing device 103, the present invention can be applied without requiring any additional operation by the user.

[0134] Next, a third embodiment of the present invention will be described. In the third embodiment, the process of generating a focus area image signal 601 when the imaging optical axis 114 of the overhead camera 102 is not parallel to the half-court line of the basketball court 108 will be described. Since the basic system configuration is the same as that of the first embodiment, repeated descriptions are avoided, and the processing to be performed by the image processing device 103 as a different aspect will be described.

[0135] ( Figure 10 : A flowchart illustrating the attention region image generation process according to the third embodiment / Figure 11A and Figure 11B (Illustration diagram) Figure 10 This diagram illustrates the processing of generating a region-of-interest image signal 601 as an example of the processing characteristics of the image processing apparatus 103 according to the third embodiment. Figure 11A and Figure 11B yes Figure 10 The illustrated diagram shows the processing from the moment the overhead camera 102 receives the overhead image signal 600 and zooms out on it until the position of the ball is detected from the zoomed-out image signal 604, compared to the processing to be performed by the image processing device 103 in the first embodiment (see [reference]). Figure 4B The process up to step S402 is the same as that in the first embodiment. The following describes the process in step S403 in the first embodiment.

[0136] First, in step S1100, CPU 210 performs the same processing as in step S700 of the first embodiment, and performs calculations based on Hough transform processing on the overhead image signal 600. Here, the quadrilateral formed by the straight lines obtained as a result of the calculation is regarded as the court lines of basketball court 108. CPU 210 stores the court lines of basketball court 108 obtained by calculation in RAM 211 and proceeds to step S1101.

[0137] Next, in step S1101, CPU 210 performs the same processing as in step S701 of the first embodiment to calculate the court coordinates. That is, CPU 210 calculates the vertex coordinates of the four vertices A, B, C, and D of the quadrilateral formed by the court lines. Figure 11 illustrates these four vertices A, B, C, and D. It should be noted that in the third embodiment, the camera optical axis 114 of the overhead camera 102 is not parallel to the half-court line of the basketball court 108, and therefore, the quadrilateral of the basketball court 108 appearing in the overhead image signal 600 has a distorted shape (see...). Figure 11A Furthermore, in conjunction with this, the calculated site coordinates differ from the site coordinates of the first embodiment, and are the coordinates of each vertex of the distorted quadrilateral. The CPU 210 stores the calculated site coordinates in RAM 211 and proceeds to step S1102.

[0138] Next, in step S1102, CPU 210 performs a projective transformation between the coordinate system of the overhead view signal 600 and the basketball court coordinate system defined based on the known shape of the basketball court 108. Specifically, CPU 210 generates a "projective transformation matrix" between the field coordinates in the coordinate system of the overhead view signal 600 and the vertex coordinates of the four vertices of the basketball court in the basketball court coordinate system, which was stored in RAM 211 in step S1101. CPU 210 stores the calculated projective transformation matrix in RAM 211 and proceeds to step S1103.

[0139] Next, in step S1103, based on the projective transformation matrix stored in RAM 211 in step S1102, CPU 210 performs a projective transformation on spherical coordinates 602a from the coordinate system in the overhead image signal 600 to the basketball court coordinate system. The result is as follows: Figure 11B As illustrated, the spherical coordinates are indicated by 602d. The CPU 210 stores the spherical coordinates 602d of the basketball court coordinate system into RAM 211 and causes the processing to proceed to step S1104.

[0140] Next, in step S1104, based on the spherical coordinates 602d stored in RAM 211 in step S1103, CPU 210 calculates a straight line parallel to the end line of the basketball court 108 and passing through the centroid of spherical coordinates 602d. On the calculated straight line, CPU 210 calculates the following line segment RT as "line segment L4", in which the intersection point with line segment AD of the quadrilateral ABCD formed by the basketball court coordinate system is defined as "R", and the intersection point with line segment BC is defined as "T". CPU 210 stores the length of the calculated "line segment L4" in RAM 211 and proceeds to step S1105.

[0141] Next, in step S1105, based on the length of line segment L4 stored in RAM 211 in step S1104, CPU 210 calculates the "attention area size" (which is the image size of the attention area image signal 601) according to the ratio of the inward division of the length of line segment L4 by the centroid of spherical coordinates 602d. Specifically, before the processing according to this embodiment begins, the full-body size of player 109 in the overhead image signal 600 when player 109 is located on the near and far sidelines of the basketball court 108 is stored. Then, the "near reference attention area size" and the "far reference attention area size" are generated based on the above image sizes.

[0142] Here, the "reference attention area size" in the vertical direction is twice the size of the player 109 in the overhead image signal 600, and the size in the horizontal direction is equal to the size in the vertical direction at a "1:1" ratio. It should be noted that the "reference attention area size" can be any size, as long as it can be cropped without cutting out the player taking the shot. The CPU 210 stores the generated near and far "reference attention area sizes" in RAM 211. The CPU 210 calculates the line segment on line segment L4 from the intersection point R to the centroid of spherical coordinate 602d as "line segment L5". The CPU 210 stores the length of the calculated "line segment L5" in RAM 211.

[0143] Then, CPU 210 scales the reference attention area size according to the ratio of the length of line segment L5 to the length of line segment L4 to generate a new attention area size. Specifically, when the closer reference attention area size is defined as "SF", the farther reference attention area size is defined as "SB", the length of line segment L4 is defined as "L4", and the length of line segment L5 is defined as "L5", CPU 210 calculates the new "attention area size S" according to the following expression 9. CPU 210 stores the length of the new attention area size in RAM 211 and proceeds to step S1106.

[0144] [Mathematical Expression 9] …expression 9

[0145] Next, in step S1106, CPU 210 calculates the "attention area coordinates 701" based on the spherical coordinates 602d stored in RAM 211 in step S705 according to the first embodiment and the attention area size stored in RAM 211 in step S1105. CPU 210 stores the calculated attention area coordinates 701 in RAM 211 and proceeds to step S1107.

[0146] Next, in step S1107, the CPU 210, according to the processing in step S706 of the first embodiment, crops the "attention area" from the overhead image signal 600 based on the attention area coordinates 701 stored in RAM 211 in step S1106, and generates the attention area image signal 601. The CPU 210 stores the generated attention area image signal 601 in RAM 211.

[0147] The above describes the processing of generating the attention region image signal 601, which is a processing characteristic of the image processing device 103 in this embodiment. As described above, the CPU 210 can generate "distortion information" based on the projective transformation matrix between the coordinates of the vertices indicating the first object (basketball court 108) and the coordinates of the vertices indicating the known shape of the first object. Then, the CPU 210 can determine the "attention region" based on the distortion information and the calculated spherical coordinates 602 (attention position), and determine the "attention region size" (which is the size of the "attention region"). The image that matches the "attention region size" is the "attention region image".

[0148] The subsequent processing follows the processing flow to be performed by the image processing device 103 of the first embodiment (see...). Figure 4B After step S404 in ), the above is a summary of the process. Figure 1 The illustrated configuration describes the processing to be performed according to a third embodiment of the present invention. As described above, the court coordinates in the coordinate system of the overhead image signal 600 can be projectively transformed into a basketball court coordinate system of a known shape. Therefore, the present invention can be applied even when the imaging optical axis 114 of the overhead camera 102 is not parallel to the half-court line of the basketball court 108.

[0149] Furthermore, CPU 210 (or CPU 243) may include a user interface (UI unit) that acquires the shape of an object (basketball court 108) received through user operation. Alternatively, CPU 210 may acquire optical information (such as the optical axis direction and focal length of the lens of the overhead camera 102 that captures the overhead image) and generate distortion information based on the acquired optical information. Additionally, HDD 216 or similar devices may store shape information related to a group of objects including objects of multiple shapes. In this case, CPU 210 may perform the following processing: That is, if any shape information among the multiple types of shape information stored in HDD 216 or similar devices is correlated with an object in an image captured by the overhead camera 102, CPU 210 acquires the object corresponding to that shape information as the object of basketball court 108.

[0150] While preferred embodiments of the invention have been described above, the invention is not limited to the embodiments described herein, and various modifications and changes can be made to these embodiments within the scope of the invention. For example, the invention can be embodied by supplying a program embodying one or more of the functions of the above embodiments to a system or device via a network or recording medium, and having a processor in a computer of the system or device read and execute the program. The invention can also be embodied by circuitry (e.g., an ASIC) for performing one or more of these functions.

[0151] This application claims priority to Japanese Patent Application 2023-210167, filed on December 13, 2023, the entire contents of which are incorporated herein by reference. Explanation of reference numerals in the attached figures

[0152] 100 Internet 101 Local Network 102. Overhead camera view. 103 Image processing equipment 104 cameras 105 Training Servers 106 Data Collection Server 107 Client Terminal 108 basketball courts 109 players 110 balls 113 Video Cable 114 camera optical axes 115 Position sensor receiving device

Claims

1. An image processing device capable of recognizing a predetermined scene in a captured image, the image processing device comprising: An input unit is configured to input a captured image obtained by capturing an image of a first object; A distortion information generation unit is configured to generate distortion information, which is used to indicate the distortion of the shape of the first object in the captured image input by the input unit relative to a known shape; A computing unit configured to calculate the location of attention in the captured image; The determining unit is configured to determine the attention area based on the distortion information generated by the distortion information generation unit and the attention position calculated by the calculation unit. An image generation unit is configured to generate an image of the region of attention from the captured image based on the region of attention determined by the determining unit; A scaling unit is configured to scale the attention area image generated by the image generation unit to a predetermined size; as well as The recognition unit is configured to perform predetermined recognition based on the image of the attention region scaled by the scaling unit. The determining unit determines the size of the attention area based on the distortion information generated by the distortion information generation unit and the attention position calculated by the calculation unit. The size of the attention area is the size of the attention area corresponding to the actual size.

2. The image processing apparatus according to claim 1, wherein, The distortion information generation unit generates the distortion information based on one or more of the following: the degree of distortion, size, and shape of the object shape relative to a known shape, in information related to the first object in the captured image.

3. The image processing apparatus according to claim 1 or 2, further comprising: A shape acquisition unit is configured to acquire the shape of the first object in the captured image. The distortion information generation unit generates the distortion information based on the shape of the first object obtained by the shape acquisition unit.

4. The image processing apparatus according to claim 3, wherein, The shape acquisition unit further includes: The UI unit is configured to acquire the shape of an object received through user interaction as the shape of the first object.

5. The image processing apparatus according to claim 3, further comprising: A shape storage unit, configured to store shape information of each object in an object group comprising objects of multiple shape types. Wherein, if the shape information among the multiple types of shape information stored in the shape storage unit is related to an object in the captured image, the shape acquisition unit also acquires an object related to the shape information for the shape of the first object.

6. The image processing apparatus according to claim 3, wherein, The shape acquisition unit also detects markers set at points in the real world corresponding to the vertices of the first object, and acquires regions with the markers as vertices for the shape of the first object.

7. The image processing apparatus according to claim 3, wherein, The distortion information generation unit generates the distortion information based on a projective transformation matrix between the coordinates of the vertices of the first object obtained by the shape acquisition unit and the coordinates of the vertices of the known shape of the first object.

8. The image processing apparatus according to claim 1 or 2, wherein, The determining unit determines the area of ​​attention based on location information related to the second object, wherein the location of the second object has already been estimated in the captured image.

9. The image processing apparatus according to claim 1 or 2, further comprising: A location information acquisition unit is configured to acquire location information from a sensor, wherein the sensor is attached to a real-world object corresponding to a third object and is configured to perform location detection. The calculation unit calculates the attention position based on the position information obtained from the sensor.

10. The image processing apparatus according to claim 1 or 2, wherein, The recognition unit determines whether the second object in the image of the area of ​​attention has performed a predetermined action.

11. The image processing apparatus according to claim 10, wherein, The identification unit determines whether the second object has performed the predetermined action based on the position of the third object and the position of the second object.

12. The image processing apparatus of claim 11, further comprising a first trained model configured to estimate the position of the second object from the image of the region of attention, wherein, The first trained model is used to estimate the location of the second object from the image of the region of interest.

13. The image processing apparatus of claim 11, further comprising a second trained model configured to detect the location of the third object from the image of the region of attention, wherein, The second trained model is used to detect the location of the third object from the reduced image of the region of interest.

14. The image processing apparatus according to claim 11, wherein, The position of the second object is the position of the joint of a player playing a match in the real world, corresponding to the position of the first object. The position of the third object is the position of the object used in the competition.

15. The image processing apparatus according to claim 1 or 2, further comprising: An optical information acquisition unit is configured to acquire optical information, including the optical axis direction and focal length of the lens of a camera used to capture the captured image. The distortion information generation unit generates the distortion information based on the optical information acquired by the optical information acquisition unit.

16. An image processing system, comprising: A first camera is configured to capture an image of a first object; A second camera is configured to capture images in a specified shooting direction; And an image processing device configured to control the shooting direction of the second camera. The first camera includes: A first camera unit is configured to acquire a captured image of the first object; and The first transmitting unit is configured to transmit the captured image obtained by the first camera unit to the image processing device. The image processing device includes: The first receiving unit is configured to receive the captured image sent by the first sending unit; A distortion information generation unit is configured to generate distortion information, which is used to indicate the distortion of the shape of the first object in the captured image received by the first receiving unit relative to a known shape. A computing unit configured to calculate the location of attention in the captured image; The determining unit is configured to determine the attention area based on the distortion information generated by the distortion information generation unit and the attention position calculated by the calculation unit. An image generation unit is configured to generate an image of the region of attention from the captured image based on the region of attention determined by the determining unit; A scaling unit is configured to scale the attention area image generated by the image generation unit to a predetermined size; A recognition unit is configured to perform a predetermined recognition based on the image of the attention region scaled by the scaling unit; A control signal generation unit is configured to generate a control signal for directing the second camera to an imaging direction corresponding to the recognition result performed by the recognition unit; and The second transmitting unit is configured to transmit the control signal generated by the control signal generating unit to the second camera. The determining unit has the function of determining the size of the attention region based on the distortion information generated by the distortion information generation unit and the attention position calculated by the calculation unit. The size of the attention region is the size of the attention region corresponding to the actual size. The second camera includes: The second camera unit is configured to perform video recording; The second receiving unit is configured to receive control signals sent by the second transmitting unit; and The control unit is configured to control the camera direction of the second camera unit based on the control signal received by the second receiving unit.

17. A control method for an image processing device, the image processing device being capable of recognizing a predetermined scene in a captured image, the control method comprising: The input step is used to input the captured image obtained by capturing an image of the first object; A distortion information generation step is used to generate distortion information, which is used to indicate the distortion of the shape of the first object in the captured image input in the input step relative to a known shape; The calculation step is used to calculate the attention position in the captured image; The determination step is used to determine the attention area based on the distortion information generated in the distortion information generation step and the attention position calculated in the calculation step; An image generation step is used to generate an image of the region of attention from the captured image based on the region of attention determined in the determining step; A scaling step is used to scale the image of the attention region generated in the image generation step to a predetermined size; as well as The identification step is used to perform predetermined identification based on the image of the attention region scaled up in the scaling step. The determination step is a step of determining the size of the attention area based on the distortion information generated by the distortion information generation step and the attention position calculated by the calculation step. The size of the attention area is the size of the attention area corresponding to the actual size.

18. A program for enabling a computer to implement a control method for an image processing device, the image processing device being able to identify a predetermined scene in a captured image, the control method comprising: The input step is used to input the captured image obtained by capturing an image of the first object; A distortion information generation step is used to generate distortion information, which is used to indicate the distortion of the shape of the first object in the captured image input in the input step relative to a known shape; The calculation step is used to calculate the attention position in the captured image; The determination step is used to determine the attention area based on the distortion information generated in the distortion information generation step and the attention position calculated in the calculation step; An image generation step is used to generate an image of the region of attention from the captured image based on the region of attention determined in the determining step; A scaling step is used to scale the image of the attention region generated in the image generation step to a predetermined size; as well as The identification step is used to perform predetermined identification based on the image of the attention region scaled up in the scaling step. The determination step is a step of determining the size of the attention area based on the distortion information generated by the distortion information generation step and the attention position calculated by the calculation step. The size of the attention area is the size of the attention area corresponding to the actual size.

Images