Image processing apparatus, control method thereof, and program

Abstract

To clip a natural partial image that is not weird, in processing an image which shows a panoramic view of a playing field.SOLUTION: An image processing apparatus for clipping a partial image in an input image includes: a detection unit which detects positions and attributes of objects included in the input image; a selection unit which selects an object to be referred in clipping the partial image, based on the attributes detected by the detection unit; a determination unit which determines a clip region in the input image, on the basis of a distribution of the positions of the object selected by the selection unit; and an output unit which clips a partial image of the region determined in the determination unit, in the input image, and outputs the partial image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, a control method and a program therefor, and in particular to a technique for extracting a specific region from an image. [Background technology]

[0002] In recent years, automatic generation of video footage of sports matches has become common. For example, because it is difficult to control an imaging device in accordance with the movement of a subject, one method involves acquiring an image captured with an angle of view that includes the entire playing field on which the match is being played and then cropping out the required portion. However, simply cropping a rectangle from the acquired image results in an angle of view that differs from the angle of view obtained when the imaging device is directed toward the subject. Therefore, the cropped region image is transformed to obtain an angle of view that is as if the imaging device were directed toward the subject. Because it is necessary to determine each time which part of the acquired image should be cropped depending on the scene of the match, a system has been proposed that determines the angle of view by determining where in the acquired image what is located.

[0003] Patent Document 1 divides a stadium into multiple areas, determines the positions of the players, and determines the area that contains the most players to determine the area to broadcast. Patent Document 1 also takes the game scenario into consideration and adjusts the pan, tilt, and zoom of the imaging device. Patent Document 2 tracks the position information of a specific target and changes the zoom magnification based on that position information to control the specific target to fit within the angle of view. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Special Publication No. 2017-531979 [Patent Document 2] Japanese Patent Publication No. 2020-77952 Summary of the Invention [Problem to be solved by the invention]

[0005] However, while Patent Document 1 applies when only players are present on the competition field, there are still issues with determining the area when referees and other people are present. Patent Document 2 has the issue that, although it is possible to adjust the angle of view to widen it in order to include specific objects, it is not possible to adjust the angle of view to narrow it within the competition field.

[0006] In view of the above problems, the present invention aims to provide a technique for cutting out a partial image that is natural and does not look unnatural, particularly when targeting an image overlooking a competition field. [Means for solving the problem]

[0007] In order to solve this problem, for example, an image processing device of the present invention has the following arrangement: An image processing device that cuts out a partial image from an input image, a detection means for detecting the position and attributes of an object included in the input image; a selection means for selecting an object to be referenced when cutting out the object based on the attribute detected by the detection means; a determining means for determining an area to be cut out in the input image based on the distribution of positions of the objects selected by the selecting means; The image processing device further comprises an output unit for cutting out a partial image of the region determined by the determination unit from the input image and outputting the cutout partial image. [Effects of the Invention]

[0008] According to the present invention, it is possible to extract a natural and natural partial image, particularly when processing an image overlooking a competition field. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of an entire system. [Figure 2]FIG. 2 is a diagram showing the hardware configuration of an image processing apparatus that constitutes part of the system. [Figure 3] FIG. 2 is a diagram showing the software configuration of the system. [Figure 4] Conceptual diagram of the learning model. [Figure 5] FIG. 1 is a diagram showing the operation of a system to which the present invention can be applied, using a learning model. [Figure 6] 10 is a flowchart showing a learning process in the embodiment. [Figure 7] 5 is a flowchart showing a video clipping process according to the first embodiment. [Figure 8] 3A to 3C are diagrams showing examples of images obtained by an imaging device in the first embodiment. [Figure 9] FIG. 4 is a schematic diagram in which results obtained by inference are superimposed in the first embodiment. [Figure 10] FIG. 10A is a diagram showing superimposed cut-out regions in the first embodiment, and FIG. 10B is a diagram showing an example of an image that has been subjected to conversion after cut-out. [Figure 11] FIG. 3 is a diagram for explaining a conversion process according to the first embodiment. [Figure 12] FIG. 10 is a diagram in which an inference result and a clipped region are superimposed in the second embodiment. [Figure 13] 10 is a flowchart showing a cutout process according to the second embodiment. [Figure 14] 10 is a flowchart showing a process relating to a competition status in the second embodiment. [Figure 15] 10 is a flowchart relating to weighting when calculating the angle of view in the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] The following embodiments will be described in detail with reference to the accompanying drawings. The following embodiments do not limit the scope of the claimed invention. Although the embodiments describe multiple features, not all of these features are necessarily essential to the invention, and multiple features may be combined in any desired manner. Furthermore, in the accompanying drawings, identical or similar components are denoted by the same reference numerals, and redundant description will be omitted. While the following description will be given using sports as an example to facilitate understanding, the present invention can also be applied to capturing images of events, concerts, and lectures. Furthermore, although the following embodiments describe the imaging device and image processing device as being housed in separate housings, they may also be integrated into a single device.

[0011] [First embodiment] Fig. 1 is a diagram showing an example of the configuration of a system including an image processing device 103 that performs cropping processing according to the first embodiment. In Fig. 1, the system includes a network 101 formed on a LAN (Local Area Network), an imaging device 102, an image processing device 103, and a client terminal 104. The client terminal 104 is an information processing device typified by a PC, a tablet, etc. Although the figure shows an example of one client terminal 104, there is no particular limit to the number of client terminals.

[0012] The network 101 is a network to which an imaging device 102, an image processing device 103, and a client terminal 104 are connected, and they can communicate with each other using various communication protocols. The network 101 may be wired or wireless.

[0013] The imaging device 102 captures an image of a predetermined range (in this embodiment, a range overlooking the entire stadium) and outputs the acquired image to the image processing device 103. The imaging device 102 in this embodiment is described as having a network interface, but video transfer from the imaging device 102 to the image processing device 103 may be performed via a video transmission cable such as an SDI (Serial Digital Interface) or an HDMI (High-Definition Multimedia Interface).

[0014] The image processing device 103 receives an image captured by the imaging device 102 and infers the position and attributes of people in the received image using a learning model. Based on the inference results, the image processing device 103 then determines a cutout area in the image and performs conversion processing on the partial image cut out from the determined cutout area. The image obtained by this conversion is stored internally and output (transmitted) in response to a request from the client terminal 104.

[0015] The client terminal 104 can use a web browser to check and set input / output in the imaging device 102 and image processing device 103. In this embodiment, the client terminal 104 will be described as acquiring images from the image processing device 103 via a web browser using the HTTP protocol, but other communication protocols may also be used. Note that the image in this embodiment refers to either a still image or each frame of a moving image, and the present invention can be applied to both.

[0016] 2 is a diagram showing the hardware configuration of an image processing device 103, which constitutes part of the system. The image processing device 103 in this embodiment includes a CPU 201, a ROM 202, a RAM 203, a NIC (Network Interface Card) 204, an input unit 205, a display unit 206, a GPU 207, an external storage device 209, and an internal bus 208 that connects the above components to each other so that they can communicate with each other.

[0017] The CPU 201 controls each component of the image processing device 103, thereby controlling the entire device. The ROM 202 is a non-volatile memory for storing a control program that describes the processing procedures of the CPU 201. It also stores the learning model shown in Fig. 4 and data referenced by the control program during operation. The RAM 203 serves as a work memory for temporarily storing the control program and data.

[0018] The NIC 204 is used to communicate with external devices via the network 101. The NIC 204 in this embodiment is used to transmit a web application for operation to the client terminal 104, receive user instructions from the client terminal 104, send and receive images from the imaging device 102, and so on.

[0019] The input unit 205 receives instructions from a user and transmits instruction signals to the CPU 201. Specific examples of the input unit 205 include a mouse, a keyboard, and a touch panel. The display unit 206 displays image data. For example, it may be a liquid crystal panel or an organic EL panel. The input unit 205 and the display unit 206 form an interface (user interface) between the user and the image processing device 103.

[0020] The GPU (Graphics Processing Unit) 207 is a unit used for parallel data processing. When performing multiple learning rounds using a learning model such as deep learning, it is effective to use the GPU 207 for processing. While a GPU is generally effective for use in learning processing, equivalent functionality may also be achieved using a reconfigurable logic circuit such as an FPGA (Field-Programmable Gate Array).

[0021] The external storage device 209 is a writable storage device such as a hard disk, and in this embodiment is used to store images received from the imaging device 102, cropped images, etc. Information regarding the installation position of the imaging device 102, the imaging line of sight direction, and the imaging angle of view is also stored in advance in this external storage device 209.

[0022] 3 is a diagram showing the software configuration of each of the imaging device 102, image processing device 103, and client terminal 104 that make up the system in this embodiment. This software configuration is realized by using the hardware resources and programs described with reference to FIG. 2. Note that this software configuration does not include general-purpose software configurations such as an operating system.

[0023] The software of the imaging device 102 includes a communication unit 321, an imaging unit 322, and a UI display unit 323. The communication unit 321 has a software function that can be incorporated so that, from among the imaging data held by the imaging device 102, data selected by a UI display unit 323 (described later) or data received from the imaging unit 322 by performing a setting in advance is sequentially transmitted to a communication unit 331 of the image processing device. The imaging unit 322 has a software function for controlling recording-related hardware such as optical elements such as an image sensor and a lens, and a sound-collecting microphone (not shown), which the imaging device 102 has. The UI display unit 323 has a software function for providing a user interface that displays arbitrary image data from among the image data held by the imaging device 102 in a user-selectable manner.

[0024] The software of the image processing device 103 is composed of a communication unit 331, an image processing unit 332, a data storage unit 333, a learning unit 334, and an inference unit 335. In the hardware configuration shown in Fig. 2, the software of each processing unit is realized by the CPU 201 of the image processing device 103 reading out a program stored in the ROM 202, expanding it in the RAM 203, and then executing it sequentially, and the CPU 201 switches its operation based on commands received via the NIC 204 or the input unit 205.

[0025] The communication unit 331 has a function for transmitting and receiving data to and from the imaging device 102 and the client terminal 104. For example, the CPU 201 receives image data from the imaging device 102 via the NIC 204, transmits a web application to the client terminal 104, receives commands from a user, and transmits information in response to the commands.

[0026] The image processing unit 332 has a function of generating an image to be presented to the user using the output of an inference unit 335 (described later) for image data read by the CPU 201 from the ROM 202 or RAM 203. The processed data is stored in the external storage device 209 or RAM 203 and is passed to the communication unit 331 in response to a request from the client terminal 104.

[0027] The data storage unit 333 has a function of storing learning data in the external storage device 209 and manipulating the stored data. Under the control of the CPU 201, the data storage unit 333 operates the data stored in the RAM 203 of the external storage device 209 based on commands received via the communication unit 331.

[0028] Here, the learning model will be described with reference to FIGS. 4(a) and 4(b). Specific examples of machine learning algorithms include nearest neighbor algorithms, naive Bayes algorithms, decision trees, and support vector machines. Also included is deep learning, which uses a neural network to generate features and connection weighting coefficients for learning. Any of the above algorithms that can be used can be used as appropriate and applied to this embodiment. In this embodiment, a model configured by a neural network will be described.

[0029] The learning model 403 of this embodiment uses the learning data 404 shown in FIG. 1B during learning, and outputs an image from the imaging device 103 as input data 401 and the position and attributes of a person in the image as output data 402 during inference. In this embodiment, information representing the person's position includes the horizontal coordinate x and vertical coordinate y of the upper left corner of a circumscribing rectangle of the area in the image where the person exists, as well as the width w and height h of the circumscribing rectangle. However, this is not limited to this, as long as it indicates the position and size. While the description here uses a circumscribing rectangle of the object, the circumscribing rectangle may be an area obtained by scaling the circumscribing rectangle at a preset ratio, or may be a polygon with more sides than a rectangle. Furthermore, the attributes of people are of two types: player and referee, but these may be changed as appropriate depending on the sport or event to which the person is applied. Specific processes for learning and inference will be described later.

[0030] The learning unit 334 has a function of learning the learning model 403 based on the learning data. The CPU 201 generates the learning model 403 using data received from the data storage unit 333 and stores it in the external storage device 209 or RAM 203. The learning unit 334 may include an error detection unit and an update unit. The error detection unit obtains the error between the training data and output data output from the output layer of the neural network in response to input data input to the input layer. The error detection unit may use a loss function to calculate the error between the output data from the neural network and the training data. The update unit updates the connection weighting coefficients between the nodes of the neural network based on the error obtained by the error detection unit so as to reduce the error. This update unit updates the connection weighting coefficients using, for example, an error backpropagation method. The error backpropagation method is a method of adjusting the connection weighting coefficients between the nodes of each neural network so as to reduce the error.

[0031] The inference unit 335 has a function of outputting an inference result using the learning model 403. Specifically, the inference unit 335 performs inference using the learning model 403 based on image data read by the CPU 201 from the external storage device 209 or RAM 203 or image data and instructions received via the communication unit 331, and stores the output in the external storage device 209 or RAM 203.

[0032] In this embodiment, the GPU 207 is used in addition to the CPU 201 for processing by the learning unit 334. Specifically, when a learning program including a learning model is executed, the CPU 201 and GPU 207 cooperate to perform calculations to perform learning. Note that the processing of the learning unit 334 may be performed by only the CPU 201 or the GPU 207. Furthermore, the inference unit 335 may also use the GPU 207, just like the learning unit 334. The use of the GPU 207 also applies to the explanation of the control flows in FIGS. 6 and 7, which will be described later.

[0033] The software of the client terminal 104 comprises a web browser 341 . The web browser 341 has a function for shaping and displaying data acquired from the communication unit 331 of the image processing device 103 so as to present it to the user. Specifically, it receives a web application from the communication unit 331 and develops it on the web browser 341. By operating the web browser 341, the user can issue commands and input data to the communication unit 331 via the web application.

[0034] FIG. 5 is an explanatory diagram of the operation of a system to which this embodiment is applied, using the structure of the learning model shown in FIG. 4. The communication unit 321 of the imaging device 102 transmits image data obtained by imaging using the imaging unit 322 to the image processing device 103. The communication unit 331 of the image processing device 103 receives this image data as input image data. The inference unit 335 of the image processing device 103 then inputs the input image data to the learning model 403, thereby outputting the inference result. The image processing unit 332 determines (or determines) the position and size of a crop area in the received image based on the inference result, and crops the received image according to the determined position and size to generate a cropped image. The communication unit 331 of the image processing device 103 then transmits information about a series of processes and the cropped image to the web browser 341 of the client terminal 104 in response to a user instruction or request from the client terminal 104. The user of the client terminal 104 checks the image displayed on a display unit (not shown) of the client terminal 104 via the web browser 341.

[0035] Next, a detailed flow of learning by the learning unit 334 of the image processing device 103 in this embodiment will be described with reference to the flowchart in FIG.

[0036] Prior to the explanation, it is assumed that the client terminal 104 holds images previously captured by the imaging device 102. Then, it is assumed that a user of the client terminal 104 has displayed the images on the display unit of the client terminal 104, and then performed an operation to surround a person in the image with a rectangular frame and an operation to select the person's attributes (athlete, referee), thereby creating the displayed image at that time and learning data that associates the displayed image with the rectangular frame (position and size) and attributes. It is assumed that this created learning data is held in a storage device (not shown) of the client terminal 104.

[0037] 6, a web page provided by the image processing device 103 is displayed on a web application deployed on the client terminal 104, and an instruction to execute learning is input on the displayed page. Then, the user of the client terminal 104 performs an operation on the web application to upload the learning data to the image processing device 103.

[0038] 6 is triggered when the CPU 201 of the image processing device 103 receives an instruction input from the user of the client terminal 104 as an instruction command to execute learning via the NIC 204. After the flow ends, the CPU 201 enters a standby state again, waiting for an instruction from the user.

[0039] In S601, the CPU 201 receives the information shown in the learning data 404 in FIG. 4 from the client terminal 104 via the communication unit 331 and stores it in, for example, the external storage device 209 using the data storage unit 333. If the CPU 201 finds any loss in the received learning data, it waits until learning data is sent again. If the CPU 201 has successfully stored the learning data, it transitions to S602. Note that if the user specifies learning data that has already been stored in the external storage device 209, the CPU 201 transitions to S602 regardless of whether learning data has been received or not.

[0040] In S602, the CPU 201 passes the learning data read by the data storage unit 333 to the learning unit 334. As a result, the learning unit 334 inputs the passed learning data to the learning model 403. As shown in the learning data 404, one piece of learning data is a set having an image as input data and a person's position and person's attributes as training data, and is input to the learning model 403 one by one as indicated by the learning data ID. When the CPU 201 has finished inputting, it transitions to S603.

[0041] In S603, the CPU 201 performs learning using the learning unit 334. After obtaining the output result, the CPU 201 proceeds to S604. As described above, the GPU 207 may be used for learning by the learning unit 334 in this step.

[0042] In S604, the CPU 201 checks whether all the learning data has been input, and if so, stores the learning model 403 in the ROM 202 or RAM 203, and ends this control flow.

[0043] Next, a detailed flow of inference in the inference phase will be described with reference to the control flow shown in FIG. 7(a). Similar to the control flow shown in FIG. 6, this control flow is woken up when the CPU 201 of the image processing device 103 receives, via the NIC 204, a command instructing execution of the web browser 341 of the client terminal 104. After the flow ends, the CPU 201 enters a standby state again, waiting for a command from the user. In this embodiment, inference is performed as pre-processing of the segmentation process, and therefore a command from the user is equivalent to the start of the segmentation process. Details of the segmentation process in S704 will be described using the control flow shown in FIG. 7(b). That is, inference in this embodiment refers to the processes shown in S701 to S703 in FIG. 7(a).

[0044] In S701, the CPU 201 of the image processing device 103 waits for reception of image data from the imaging device 102. When the CPU 201 confirms reception of the image data, it stores the received image data in the RAM 203 and transitions the process to S702.

[0045] In S702, the CPU 201 inputs the image data stored in S701 into the learning model of the inference unit 335. When the CPU 201 has finished inputting the image data, the CPU 201 transitions the process to S703.

[0046] In S703, the CPU 201 causes the inference unit 335 to perform inference and stores the output result in the RAM 203. Here, the output result is a pair indicating the position of a person and the attribute of the person, as shown in the learning data 404.

[0047] Then, in S704, the CPU 201, functioning as the image processing unit 332, executes a process of cutting out the received image based on the position and attributes of each person appearing in the received image.

[0048] The cutout process in step S707 is shown in detail in Fig. 7(b). Below, the characteristic processes of this embodiment will be described with reference to Fig. 7(b) and Figs. 8 to 11.

[0049] FIG. 8 shows image data 801 obtained by the imaging device 102. As shown in the figure, the imaging device 102 captures an overhead image encompassing the entire competition field 804 and transmits the captured image as image data 801 to the image processing device 103. The image data 801 includes images of multiple players 802 and a referee 803 on the competition field 804. The CPU 201 of the image processing device 103 uses the image data 801 as an input image and performs inference according to steps S701 to S703 of the control flow shown in FIG. 7(a) to obtain the positions and attributes of people as output results in the image data 801. Here, as a method of presenting the output results to the user, when the received image is displayed on the display unit 206, additional processing may be performed to surround the people obtained by inference with a rectangular frame. That is, a mark such as a rectangle 901 may be superimposed on the image data 801, as shown in FIG. 9. Furthermore, to distinguish between players and referees, a different mark such as a rectangle 902 may be used for the referee. The additional processing may be generated by the CPU 201 using the image processing unit 332 when the output result is obtained in step S703, and stored in the external storage device 209 or RAM 203, and presented in response to a user request.

[0050] In S704, that is, in S711, the CPU 201 of the image processing apparatus 103 reads the output result stored in S703 from the RAM 203. Then, the CPU 201 transitions the process to S712.

[0051] In S712, the CPU 201 checks the output results of the inference process for the cutout process. That is, the CPU 201 reads one set of a person's position and person's attribute, transitions to S713, and determines whether all of the sets have been read. If the CPU 201 determines that all of the sets have been read, it transitions the process to S715. If the CPU 201 determines that there are output results that have not been read, it transitions the process to S713.

[0052] In S713, the CPU 201 determines whether the person attribute read in S711 indicates a player or a referee. If the CPU 201 determines that the person attribute indicates a player, the process proceeds to S714, and if the CPU 201 determines that the person attribute indicates a referee, the process returns to S712.

[0053] In S714, the CPU 201 stores (adds) the read position information of the person to a list prepared in advance on the RAM 203, in order to use the position information as a target for calculating the angle of view for cropping. After this storage process, the CPU 201 returns the process to S712.

[0054] When the process proceeds to S715, the players (position information) excluding the referees are selected from among the people appearing in the received image and stored in a list in RAM 203. Then, in S714, the CPU 201 of the image processing device 103 reads the list stored in RAM 203 and calculates the center position of the cutout area. The cutout area here refers to a location on the playing field that deserves attention, such as the area indicated by the frame 1001 in FIG. 10(a).

[0055] Here, the conversion process related to cropping will be described with reference to Figures 11(a) to 11(c). Figure 11(a) shows the relationship between a reference image at a certain position captured by the imaging device 102 and spherical coordinates with the position O of the imaging device as the origin. If the center of the reference image is R, and the pixel positions on the reference image are shown with the upward direction from the center as the positive direction of the z coordinate axis and the right as the positive direction of the x coordinate axis, they can be mapped one-to-one onto a spherical surface as seen from position O. Specifically, point Q(x, y, z) on the reference image and coordinate P(xp, yp, zp) on a spherical surface of radius r with position O as the origin are expressed by the following relationship: The spherical coordinates (r, θ, φ) are defined as shown in Figure 11(b).

[0056] Point Q in spherical coordinates:

[0057]

number

[0058] If we express the point P in spherical coordinates:

[0059]

number

[0060] Next, the conversion of the cropped area will be described with reference to Figure 11(c). The central coordinates of the cropped area found on the reference image are found and converted to point U (θc, φc) on the spherical coordinate system using equations (1) and (2). With point U as the center, the horizontal angle of view of the size to be cropped is set to 2Δθ and the vertical angle of view is set to 2Δφ, and four vertices (F1, F2, F3, F4) are obtained on the spherical surface.

[0061]

number

[0062] The four vertices of the cropped area obtained on the spherical surface are converted again into coordinates on the reference image using (Equation (1)), and these are obtained as the four vertices of the cropped area. Note that the above θ and φ may be changed by the user of the client terminal by operating buttons or the like provided on the UI of the web browser.

[0063] Returning to the explanation of S715, in S715, the CPU 201 of the image processing device 103 determines the center position of the cutout region based on the distribution of player positions in the list stored in the RAM 203. Specifically, the CPU 201 determines the center of gravity position from the positions of the players shown in the list, and sets this center of gravity position as the center of the cutout region. In this embodiment, the weight of each player is assumed to be equivalent, and only position information contributes to the calculation (the position of the referee is excluded). Here, there are multiple possible methods for determining the center of the cutout region. This is an example of the content of this proposal, and does not limit the content of this proposal. After the center position is determined, the process proceeds to S716.

[0064] In S716, the CPU 201 acquires the four vertices of the crop area, crops it, and performs the conversion based on the conversion process described with reference to Fig. 11. The processed image is stored in the RAM 203, and this control flow ends. Projective conversion may be used to map the cropped image onto a rectangular area.

[0065] When the user operates a UI (not shown) on the client terminal 104, the CPU 201 can transmit the processed image via the communication unit 331. As a result, the user of the client terminal can obtain processed image data 1002 for the captured image data 801.

[0066] In this embodiment, since the referee is not included in the calculation of the angle of view, it is possible to extract an area in which players are gathered within the angle of view. Depending on the sport or event, attributes other than those of the players and the referee may be added and used for judgment.

[0067] As described above, according to the embodiment, when there are people with multiple roles, such as players and referees, on the competition field, it is possible to extract a partial image with a field of view that looks as if it were taken by a photographer from an image that overlooks the entire field.

[0068] [Second embodiment] In the first embodiment, whether or not to use the attribute obtained by inference in calculating the angle of view was determined, but by modifying this, effects appropriate for the competition scene can be obtained. In the second embodiment, the same configuration and control as in the first embodiment are assumed, and differences from the control flow shown in FIG. 7(b) will be described. In the second embodiment, the imaging device 102 is assumed to have a built-in microphone (assumed to be omnidirectional), and to output audio data collected by the microphone along with the captured video.

[0069] 13, steps S1301 to S1304 and S1306 are equivalent to steps S711 to S714 and S716 in Fig. 7(b), and therefore descriptions thereof will be omitted. In addition, in the second embodiment, the image data 1201 in Fig. 12 will be described as being received from the imaging device 102.

[0070] In S1307, the CPU 201 of the image processing device 103 determines whether the position information of the person, which is the output result of the inference, is within the competition field 1203. If the CPU 201 determines that the position information of the person being processed, which is the output result of the inference, is within the competition field 1203, it transitions to S1308, and if it is not within (outside the competition field), it returns the process to S1302.

[0071] Note that the boundary information for the competition field 1203 may be detected by the image processing unit 332 when the CPU 201 acquires the image data 1201. Alternatively, when a user of the client terminal 104 uploads learning data, the user may set the boundary position of the competition field by operating the operation unit, and upload this set information (boundaries of the competition field) as well. Alternatively, the user of the client terminal 104 may specify this using the UI for the image displayed by the web application on the client terminal 104, or an administrator may specify this using the input unit 205 and store it in RAM 203.

[0072] Now, if in S1307 the CPU 201 determines that the target person is located outside the competition field, the process returns to S712 (the process of S714 is not performed), and therefore, regardless of whether the target person is a player or a referee, the position of that person is not included in the list for determining the cut-out position.

[0073] On the other hand, when the process transitions to S1308, the CPU 201 reads information indicating the current competition status from the RAM 203 and makes a determination. The competition status here is information indicating whether the competition is in progress or suspended, and is updated by waking up the control flow shown in FIG. 14 at preset time intervals. If the CPU 201 determines that the competition status is suspended, the process transitions to S1304. In other words, when the game is suspended, the position of any person on the competition field, regardless of whether they are a player or a referee, is included in the list.

[0074] On the other hand, if the CPU 201 determines in S1308 that the competition status is in play, it transitions the process to S1303. In S1303, the CPU 201 determines whether the target person is a player, and if so, transitions the process to S1304, and if it is determined that the target person is a referee, it returns the process to S1302.

[0075] As a result of the above, when the competition status is in play, the positions of only the players are stored in the list created in S1304. When the competition status is suspended, the positions of not only the players but also the referees are registered in the list.

[0076] Here, the competition status determination process executed by the CPU 201 of the image processing device 103 will be described with reference to the flowchart of Fig. 14. As explained above, this process is initiated at appropriate time intervals.

[0077] In S1401, the CPU 201 of the image processing device 103 waits for audio data to be sent from the imaging device 102. When the CPU 201 confirms reception of the audio data, it stores it in the RAM 203 and transitions the process to S1402.

[0078] In S1402, the CPU 201 reads the competition status (previously determined competition status) from the RAM 203 and checks whether it indicates that the competition is in progress. If the CPU 201 determines that the read competition status indicates that the competition is in progress, the process proceeds to S1403, and if it determines that the competition status indicates that the competition is suspended, the process proceeds to S1405.

[0079] In S1403, the CPU 201 compares the read competition status with the audio data stored in the RAM 203 to determine whether or not it is an interruption sound (a whistle sound indicating an interruption). The audio data stored in the RAM 202 may be registered in advance, or may be configured to query another information processing device via the communication unit 331. If the CPU 201 determines that it is an interruption sound, it transitions the process to S1404, and if not, it transitions to S1401.

[0080] In S1404, the CPU 201 rewrites the competition status read in S1402 to "suspended", stores (updates) it in the RAM 203, and ends this control flow.

[0081] In S1405, the CPU 201 compares the read competition status with the audio data stored in the RAM 203 to determine whether it is a start sound (a whistle sound indicating the start). The audio data to be compared may have the same configuration as in S1403. If the CPU 201 determines that it is a start sound, it transitions the process to S1406, and if it determines that it is not, it returns the process to S1401.

[0082] In S1406, the CPU 201 rewrites the competition status read in S1402 to "in progress", stores (updates) it in the RAM 203, and ends this control flow.

[0083] The description will continue by returning to the control flow in Fig. 13. When the determination process for all people appearing in the image is completed, the CPU 201 transitions the process from S1302 to S1305.

[0084] In step S1305, the CPU 201 assigns a weight to each attribute when calculating the center of gravity, unlike in the first embodiment. In other words, in the first embodiment, the center of gravity was calculated assuming that each player had the same weight. In the second embodiment, for example, the weight of person 1202, who has the attribute of a referee, in FIG. 12 is increased. By assigning a value sufficiently larger than the total number of players, it is possible to obtain an angle of view centered on the referee (this applies, of course, when the game is suspended). This results in a referee-centered clipping, such as clipping area 1206, and the control flow in FIG. 13 makes it possible to obtain a clipping area focusing on the referee when the game is suspended. While the second embodiment uses two states, namely, the progress of the game and the suspension of the game, as examples, the weighting rate may be changed as appropriate depending on the sport or event.

[0085] [Third embodiment] In the first and second embodiments, whether or not to use attributes obtained by inference in angle-of-view calculation is determined. However, in the third embodiment, a case will be described in which a uniform angle-of-view calculation is used, but a similar effect can be obtained by changing the weight.

[0086] Fig. 15 is a flowchart of the control process in the third embodiment. Assuming that the same configuration and control as in the second embodiment are used, the differences from the control flow shown in Fig. 13 will be explained.

[0087] In the control flow shown in Fig. 15, S1501 to S1504 and S1506 are equivalent to S1301, S1302, S1307, S1304 and S1306 in the control flow of Fig. 13, and therefore a description thereof will be omitted. In other words, the position of a person determined to be within the competition field is added to a list that determines the cut-out position, regardless of the attributes of that person.

[0088] The characteristic processing portion of the third embodiment is concentrated in the processing of S1505. When implementing a configuration similar to that of the first embodiment, in S1505, the CPU 201 of the image processing device 103 always sets the position weight of objects having the attribute of a referee to 0. This effectively eliminates the need for use in angle-of-view calculation. On the other hand, when implementing a configuration similar to that of the second embodiment, in S1505, the CPU 201 of the image processing device 103 reads the competition status information shown in FIG. 14 from the RAM 203, and sets the referee weight to "0" (zero) if the match is in progress, or to a weight corresponding to the total number of players if the match is suspended. This achieves the same effect corresponding to the competition status for objects having the attribute of a referee. As with the other embodiments, the weighting rate may be changed as appropriate depending on the sport or event.

[0089] The first to third embodiments have been described above. Of the above-described processing units, the inference unit 335 executes processing using a machine-learned model. However, rule-based processing such as a lookup table (LUT) may also be performed. In this case, for example, the relationship between input data and output data is created in advance as an LUT. The created LUT may then be stored in the memory of the image processing device 103. When performing processing by the inference unit 335, output data can be acquired by referencing the stored LUT. In other words, the LUT operates in cooperation with a CPU or GPU as a program for performing processing equivalent to that of the processing unit, thereby performing the processing of the processing unit. The present invention can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or storage medium, and having the computer of the system or device read and execute the program. The computer may have one or more processors or circuits, and may include multiple separate computers or a network of multiple separate processors or circuits to read and execute computer-executable instructions.

[0090] The processor or circuitry may include a central processing unit (CPU), a microprocessing unit (MPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gateway (FPGA), a digital signal processor (DSP), a data flow processor (DFP), or a neural processing unit (NPU).

[0091] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0092] The disclosure of this specification includes the following image processing device, method, and program. (Item 1) An image processing device that cuts out a partial image from an input image, a detection means for detecting the position and attributes of an object included in the input image; a selection means for selecting an object to be referenced when cutting out the object based on the attribute detected by the detection means; a determining means for determining an area to be cut out in the input image based on the distribution of positions of the objects selected by the selecting means; an output means for cutting out a partial image of the region determined by the determination means from the input image and outputting the cutout; 1. An image processing device comprising: (Item 2) The input image is an overhead image of a competition field, The detection means detects the position of a person and whether the person has the attribute of a player or a referee. 2. The image processing device according to item 1, (Item 3) the selection means selects an object having a player attribute; The determining means determines an area having a preset angle of view centered on the center of gravity of the positions of the objects selected by the selecting means as the area to be cut out. 3. The image processing device according to item 2, (Item 4) Further, a game status determination means is provided for determining whether the game is in a playing state or a paused state, The selection means If the determination by the competition status determination means indicates that the game is in progress, an object having a player attribute is selected; If the determination by the competition status determination means indicates that the competition status is being suspended, an object having an attribute of a player or a referee is selected; The determining means determines an area having a preset angle of view centered on the center of gravity of the positions of the objects selected by the selecting means as the area to be cut out. 3. The image processing device according to item 2, (Item 5) 5. The image processing device according to item 3 or 4, wherein the selection means does not select people outside the competition field. (Item 6) Further, a game status determination means is provided for determining whether the game is in a playing state or a paused state, the selection means selects an object located within the competition field and does not select an object outside the competition field; the determining means determines the region to be cut out by calculating a weighted center of gravity position based on the attributes of the object selected by the selecting means; The determination means When the result of the determination by the competition status determination means indicates that the game is in play, the weight of the position having the attribute of the referee is set to zero; If the result of the determination by the competition status determination means indicates that the game is suspended, the weight of the position having the attribute of the referee is set to a value greater than the weight indicated by the total number of players. The center of gravity is determined to determine the region to be cut out. 3. The image processing device according to item 2, (Item 7) A control method for an image processing device that cuts out a partial image from an input image, comprising: a detection step of detecting the position and attributes of an object included in the input image; a selection step of selecting an object to be referenced when cutting out the object based on the attributes detected in the detection step; a determination step of determining an area to be cut out in the input image based on the distribution of positions of the objects selected in the selection step; an output step of cutting out a partial image of the region determined in the determination step from the input image and outputting the partial image; 1. A method for controlling an image processing apparatus, comprising: (Item 8) A program that, when read and executed by a computer, causes the computer to function as each of the means possessed by the device described in any one of items 1 to 6.

[0093] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0094] 101... network, 102... imaging device, 103... image processing device, 104... client terminal

Claims

1. An image processing device that cuts out a partial image from an input image, a detection means for detecting a position and an attribute of an object included in the input image; a selection means for selecting an object to be referenced when cutting out the object based on the attribute detected by the detection means; a determining means for determining an area to be cut out in the input image based on a distribution of positions of the objects selected by the selecting means; an output means for cutting out and outputting a partial image of the region determined by the determination means from the input image; 13. An image processing device comprising:

2. The input image is an overhead image of a competition field, The detection means detects the position of a person and whether the person has an attribute of a player or a referee.

2. The image processing device according to claim 1,

3. the selection means selects an object having an attribute of a player; The determining means determines, as the region to be cut out, a region having a preset angle of view centered on a center of gravity of the positions of the objects selected by the selecting means.

3. The image processing device according to claim 2.

4. Further, a game status determination means is provided for determining whether the game is in a playing state or a paused state, The selection means is When the determination by the athletic status determination means indicates that the player is in play, an object having an attribute of a player is selected; When the judgment by the competition status judgment means indicates that the competition status is being suspended, an object having an attribute of a player or a referee is selected; The determining means determines, as the region to be cut out, a region having a preset angle of view centered on a center of gravity of the positions of the objects selected by the selecting means.

3. The image processing device according to claim 2.

5. 5. The image processing apparatus according to claim 4, wherein said selection means does not select people outside the playing field.

6. Further, a game status determination means is provided for determining whether the game is in a playing state or a paused state, said selection means selecting an object located within the competition field and not selecting an object outside the competition field; the determining means determines the region to be cut out by calculating a weighted center of gravity position based on the attributes of the object selected by the selecting means; The determination means is When the result of the judgment by the competition status judgment means indicates that the game is being played, the weight of the position of the object having the attribute of a referee is set to zero; When the result of the judgment by the competition status judgment means indicates that the competition is suspended, the weight of the position of the object having the attribute of a referee is set to a value greater than the weight indicated by the total number of players. The center of gravity is obtained to determine the region to be cut out.

3. The image processing device according to claim 2.

7. A method for controlling an image processing device that cuts out a partial image from an input image, comprising the steps of: a detection step of detecting a position and an attribute of an object included in the input image; a selection step of selecting an object to be referenced when cutting out the object based on the attributes detected in the detection step; a determination step of determining an area to be cut out in the input image based on a distribution of positions of the objects selected in the selection step; an output step of cutting out a partial image of the region determined in the determination step from the input image and outputting the partial image; 13. A method for controlling an image processing apparatus comprising the steps of:

8. A program for causing a computer to execute each step of the control method according to claim 7 when the program is read and executed by the computer.

Images