Image processing device, image processing method and program

Abstract

To provide an image processing device capable of appropriate behavior recognition of a competitor even in using a captured image captured so that a coat falls within a field angle.SOLUTION: An image processing device according to the present disclosure includes acquirement means that acquires an image captured so that a coat of a sport competition falls within a field angle, detection means that detects a plurality of objects in the image, trimming means that trims a range specified based on positions of the plurality of objects in the image, and recognition means that recognizes a behavior of a specific player among the plurality of objects on the basis of the trimmed image. Here, the trimming means trims the range including the specific player specified using a position of the first object used in the sport competition as a reference among the plurality of objects.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an image processing method, and a program. [Background technology]

[0002] Conventionally, there are known techniques for recognizing the behavior of athletes in images captured during sporting events. Patent Document 1 proposes a technique for detecting players in video footage of a volleyball match from features of facial features such as the eyes and nose, and recognizing the player serving based on the player's position and posture on the court and whether or not they are holding the ball. Patent Document 2 also proposes a technique for recognizing a climbing course using machine learning from images captured during a sport climbing competition, and analyzing the behavior of the athletes by estimating the athletes on the course and their skeletal structure. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2019 / 235350 [Patent Document 2] Patent Publication No. 2021-26292 Summary of the Invention [Problem to be solved by the invention]

[0004] In order to detect each player moving around on the court from an image, it is desirable to capture a bird's-eye view of the entire court. Meanwhile, in machine learning inference processing, low-resolution images are generally input to prevent an increase in processing load. Therefore, when applying machine learning to an image of an entire court, such as a volleyball court, the image may not contain enough pixel information to recognize faces and gestures, resulting in inappropriate recognition results. Furthermore, in some sports competitions, it is often impossible to identify a player whose behavior should be recognized by predetermining his or her position on the court. In other words, it is also necessary to determine, among multiple players in an image of the entire court, a player whose behavior needs to be recognized, regardless of his or her position in the image. The aforementioned Patent Documents 1 and 2 did not take such issues into consideration.

[0005] The present invention has been made in consideration of the above-mentioned problems, and its purpose is to realize a technology that can appropriately recognize the behavior of players even when using photographic images that include the court in the field of view. [Means for solving the problem]

[0006] In order to solve this problem, for example, an image processing device of the present invention has the following configuration: namely, it has an acquisition means for acquiring an image captured so that a sports court is included within an angle of view, a detection means for detecting a plurality of objects within the image, a cropping means for cropping an area identified based on the positions of the plurality of objects within the image, and a recognition means for recognizing the action of a specific player among the plurality of objects based on the cropped image, wherein the cropping means crops an area including the specific player identified based on the position of a first object used in the sports game among the plurality of objects. [Effects of the Invention]

[0007] According to the present invention, it is possible to appropriately recognize the actions of players even when using photographed images that include the court within the angle of view. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a system configuration according to a first embodiment. [Figure 2A] FIG. 2 is a block diagram showing an example of the hardware configuration of a learning server and an image processing device according to the first embodiment. [Figure 2B] FIG. 2 is a diagram showing a sequence of data transmission and reception and processing between devices in the image processing system according to the first embodiment. [Figure 2C] 4 is a flowchart showing the operation of the device in the image processing system of the first embodiment. [Figure 2D] FIG. 1 is a conceptual diagram showing the input / output structure using the learning model of the first embodiment. [Figure 3] 1 is a block diagram showing an example of the configuration of an image processing device according to a first embodiment. [Figure 4] FIG. 2 is a diagram illustrating a subject to be photographed in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating an example of an overhead image according to the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining an object detection unit according to the first embodiment. [Figure 7] FIG. 1 is a diagram (1) for explaining a specific player detection unit according to the first embodiment. [Figure 8] FIG. 10 is a diagram (2) for explaining the specific player detection unit of the first embodiment. [Figure 9] FIG. 1 is a diagram (1) for explaining a trimming coordinate determination unit according to the first embodiment. [Figure 10] FIG. 10 is a diagram (2) for explaining the trimming coordinate determination unit of the first embodiment. [Figure 11] 10A and 10B are diagrams illustrating the application of this embodiment to other sports. [Figure 12] FIG. 10 is a block diagram showing an example of the configuration of an image processing device according to a second embodiment. [Figure 13] FIG. 10 is a diagram for explaining a duplicate player detection unit according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (Embodiment 1) Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] <Image processing system configuration> An example of an image processing system according to this embodiment will be described with reference to Fig. 1. The image processing system includes, for example, the Internet 100, a local network 101, a learning server 102, a data collection server 103, a client terminal 104, an image processing device 105, and an overhead camera 106.

[0011] The Internet 100 and the local network 101 are networks connecting the devices in the image processing system. Either one may be used as long as the devices are connected via a network. The learning server 102 is, for example, a server computer as an example of an information processing device, and executes the learning stage processing of machine learning described below to determine parameters of a trained model. The data collection server 103 is, for example, a server computer as an example of an information processing device. The data collection server 103 accumulates training data used in the learning stage processing and provides the training data to the learning server 102. The client terminal 104 is, for example, a communication device, and initiates data transmission and reception between devices in the system. The overhead camera 106 is, for example, an imaging device such as a digital camera, and outputs overhead images described below. The image processing device 105 is, for example, a personal computer, and executes machine learning inference processing described below on video images captured by the overhead camera 106.

[0012] <Configuration of the learning server and image processing device> FIG. 2A shows an example of the hardware configuration of the learning server 102 and the image processing device 105 in the image processing system according to this embodiment.

[0013] The learning server 102 includes, for example, a CPU 202, a ROM 203, a RAM 204, a HDD 205, a NIC 206, an input unit 207, a display unit 208, and a GPU 209. The CPU 202 is a computation circuit such as a CPU (Central Processing Unit) and realizes each function of the learning server 102 by loading and executing a program stored in the ROM 203 or the HDD 205 into the RAM 204. The ROM 203 includes a non-volatile storage medium such as a semiconductor memory and stores, for example, programs executed by the CPU 202 and necessary data. The RAM 204 includes a volatile storage medium such as a semiconductor memory and temporarily stores, for example, calculation results of the CPU 202. The HDD 205 includes a hard disk drive and stores, for example, programs executed by the CPU 202 and training data of this embodiment. The GPU (Graphics Processing Unit) 209 includes a computation circuit and can execute some or all of the calculations, for example, to train a learning model. The NIC 206 includes a network interface for communication via a network (e.g., the Internet 100, the local network 101). The input unit 207 includes, for example, a keyboard or an interface for connecting a keyboard, etc., that accepts operational input from the administrator of the learning server 102, but it does not necessarily have to be included in the learning server 102. The display unit 208 includes, for example, a display, and displays a user interface that allows the administrator of the learning server 102 to check the operating status of the learning server 102 or operate the learning server 102, but it does not necessarily have to be included in the learning server 102.

[0014] For example, the CPU 202 loads the learning program stored in the HDD 205 and the ROM 203 and the training data stored in the HDD 205 into the RAM 204. Next, the CPU 202 executes the program loaded into the RAM 204 and trains a learning model using the training data. The process of training the learning model may be executed by the GPU 209 in response to an instruction from the CPU 202.

[0015] The image processing device 105 includes, for example, a CPU 212, a ROM 213, a RAM 214, a HDD 215, a NIC 216, an input unit 217, a display unit 218, and an image processing engine 219. The CPU 212 is an arithmetic circuit such as a CPU (Central Processing Unit), and realizes each function of the image processing device 105 by loading a program stored in the ROM 213 or the HDD 215 into the RAM 214 and executing it. The ROM 213 includes a non-volatile storage medium such as a semiconductor memory, and stores, for example, programs executed by the CPU 212 and necessary data. The RAM 214 includes a volatile storage medium such as a semiconductor memory, and temporarily stores, for example, calculation results of the CPU 212. The HDD 215 includes a hard disk drive, and stores, for example, processing results of programs executed by the CPU 212. The NIC 216 includes a network interface for communication via a network (for example, the Internet 100 or the local network 101). The input unit 217 includes, for example, a keyboard or an interface for connecting a keyboard, which accepts operation input to the image processing device 105. The display unit 218 includes, for example, a display, and displays, for example, a user interface for checking the operating status of the image processing device 105 or for operating the image processing device 105. The image processing engine 219 is, for example, an image processing circuit that executes predetermined processing (such as reduction processing) on ​​an input image.

[0016] The image processing device 105 may be connected directly or via a network to a second camera separate from the overhead camera (not shown), and the CPU 212 may control the capture of the second camera, for example. The second camera is a camera that captures images with a narrower angle of view than the overhead camera and captures a portion of the court. For example, the second camera may capture a close-up image of a player whose behavior is recognized by the behavior recognition process described below. For example, when the image processing device 105 recognizes that a player is performing a specific behavior based on the behavior recognition results described below, it may control the pan and zoom of the second camera to capture the player's behavior. In this way, it becomes possible to capture a specific player as the main subject with the second camera in accordance with the player's behavior.

[0017] <Player behavior recognition processing in image processing device> Next, a process of recognizing a player's behavior from an overhead image of a sporting event in the image processing device (referred to as a player behavior recognition process) will be described. The player behavior recognition process is realized by, for example, the CPU 212 of the image processing device 105 executing a program.

[0018] In the following description, an example of player behavior recognition processing will be described in which the behavior of a specific basketball player is recognized from an overhead image of a basketball court. However, the player behavior recognition processing can also be applied to behavior recognition in other sports in which multiple players compete on a sports field. For example, the processing can also be applied to behavior recognition of players in other sports such as soccer, rugby, and volleyball. In this case, the sports field is a soccer court, a rugby court, or a volleyball court, respectively.

[0019] The player action recognition process according to this embodiment uses a learning model in the object detection and action recognition processes described below. Each learning model undergoes learning stage processing in the learning server 102, and then undergoes inference stage processing using learned parameters in the image processing device 105. First, the learning stage processing of the learning server 102 in the image processing system will be described, and then the configuration for realizing the player action recognition process in the image processing device 105 will be described.

[0020] <Operation of each device in the image processing system> The sequence of data transmission and reception and processing between devices in the image processing system will be described with reference to FIG. 2B.

[0021] In the following description of the image processing system, an example of training a learning model for object detection used in the player behavior recognition process will be described. In this case, in the description of Figures 2B, 2C, and 2D, an image that will be simply described as a "bird's-eye view image" for the sake of simplicity represents a bird's-eye view image that has been reduced to have the same number of pixels as reduced image signal 351 (reduced bird's-eye view image) shown in Figure 3, which will be described later.

[0022] In S201, the client terminal 104 instructs the learning server 102 to acquire training data. Note that the training data for object detection in this embodiment may be, for example, a set of data including an overhead image, a basketball player in the overhead image, and ball coordinate values. In S202, the learning server 102 requests training data from the data collection server 103. The learning server 102 may request training data by specifying, for example, information indicating the type of training data. In S203, the data collection server 103 extracts the requested training data from its storage unit and transmits the extracted training data to the learning server. In S205, after receiving the training data, the learning server 102 performs a machine learning learning stage process to obtain (by calculation) parameters of a trained model. In S206, the learning server 102 transmits the obtained trained model parameters to the image processing device 105. In S207, the image processing device 105 uses the parameters of the trained model received from the training server 102 to perform processing at the inference stage of the training model (for example, object detection for a newly captured overhead image).

[0023] <Data collection server operation> Next, with reference to Fig. 2C(b), the operation of the data collection server 103 will be described. Note that in the explanation shown in Fig. 2C(b), the operation is performed by the data collection server, but each operation is realized by the CPU of the data collection server (not shown) executing a program.

[0024] In S221, the data collection server 103 receives a request for teacher data from the learning server 102. Next, in S222, the data collection server 103 identifies the type of teacher data requested. In this embodiment, the type of teacher data is the coordinate values ​​of the overhead image, the player, and the basketball. In S223, the data collection server 103 transmits to the learning server 102, from among the stored teacher data, the teacher data to be used by the learning server 102.

[0025] <Operation on the learning server> Next, the operation of the learning server 102 will be described with reference to Fig. 2C(c). In learning in the learning server 102, training data (e.g., an overhead image) is input to a learning model 503 configured with a neural network, as schematically shown in Fig. 2D. In the case of a learning model for object detection, for example, the learning model 503 outputs the coordinates of the player and basketball in the overhead image as a result of calculations on the overhead image.

[0026] In this learning process, the GPU 209 is used in addition to the CPU 202 of the learning server 102. That is, when a learning program including a learning model is executed, the CPU 202 and GPU 209 work together to perform calculations, thereby performing learning. The GPU 209 can perform efficient calculations by processing a larger amount of data in parallel, so it is effective to use the GPU 209 for deep learning, which involves repeated calculations using a learning model. Note that calculations may be performed by only the CPU 202 or the GPU 209 during the learning stage. Therefore, in the explanation shown in FIG. 2C(c), the operations are described as being performed by the learning server, but each operation is realized by at least one of the CPU 202 and the GPU 209 executing a program.

[0027] In S230, the learning server 102 requests the teacher data specified by the client terminal from the data collection server 103. In S231, the learning server 102 determines whether it has received the teacher data from the data collection server 103. If the learning server 102 determines that it has received the teacher data from the data collection server 103, it proceeds to S232; otherwise, it returns to S231 and repeats the process.

[0028] In S232, the learning server 102 inputs the training data received from the data collection server and the training setting values ​​corresponding to the training data into the training model. Here, the training model is the aforementioned training model 503. In this embodiment, the training setting values ​​are, for example, parameter values ​​of data augmentation applied to the input signal of the training model 503.

[0029] In S233, the learning server 102 executes a process to train the learning model 503. In S234, the learning server 102 determines whether all the training data has been input. If all the training data has been input, the process ends. If not, the process returns to S232 and repeats. Note that the determination of the end of learning in S234 is just an example; the input of all the training data may be repeated a predetermined number of times, or the process may end when the value of the loss function satisfies a predetermined condition. By completing the learning, the learning server 102 obtains parameters of the trained model (such as the connection weighting coefficients after training of the neural network).

[0030] The learning process in S234 includes an error detection process and an update process. In the error detection process, the learning server calculates the error between the output data (the coordinates of the player and the basketball) output from the output layer of the neural network in response to the overhead image input to the input layer and the coordinates of the player and the basketball included in the training data. Here, the coordinates of the player and the basketball included in the training data are previously assigned to the overhead image and are so-called correct labels. In the error detection process, a loss function may be used to calculate the difference between the output data from the neural network and the training data. In the update process, the learning server updates the connection weighting coefficients between the nodes of the neural network based on the error obtained in the error detection process to reduce the error. This update process updates the connection weighting coefficients, for example, using backpropagation. Backpropagation is a technique for adjusting the connection weighting coefficients between the nodes of each neural network to reduce the error.

[0031] <Inference processing in image processing devices> Next, with reference to FIG. 2C(a), the operation of the inference processing in the image processing device 105 will be described. The image processing device 105 performs the inference stage of machine learning processing using a program stored in the HDD 215 or the ROM 213 and parameters of a trained model stored in the HDD 215 (received from the learning server). That is, the CPU 212 of the image processing device 105 performs the inference processing on a newly captured overhead image using the parameters of the trained model and the program. As described above, the image simply described as a "bird's-eye view image" in FIG. 2C(a) for the sake of simplicity corresponds to the reduced image signal 351 obtained by reducing the overhead view image in the configuration shown in FIG. 3. Furthermore, although the operations will be described as being performed by the image processing device, each operation is realized by the CPU 212 executing a program.

[0032] In S211, the image processing device 105 determines whether parameters of the trained model have been received from the learning server 102. If parameters of the trained model have not been received, the process returns to S211; otherwise, the process proceeds to S212. In S212, the image processing device 105 determines whether an overhead image has been acquired. If not, the process returns to S212; otherwise, the process proceeds to S213. In S213, the image processing device 105 determines whether an instruction to start the inference process has been received from the user. If the instruction to start the inference process has not been received, the process returns to S213; otherwise, the process proceeds to S214. In S214, the image processing device 105 inputs the acquired overhead image into the learning model and executes the inference process. In S215, the image processing device 105 stores the coordinate positions of the player and the ball, which are the inference results, in the HDD 215. The image processing device 105 then terminates this process.

[0033] <Configuration for player behavior recognition> Next, the configuration for player behavior recognition will be described with reference to Fig. 3. The configuration shown in Fig. 3 is a software configuration (also referred to as a player behavior recognition module) realized by, for example, the CPU 212 of the image processing device 105 executing a program. The player behavior recognition module includes, for example, an image reduction unit 301, an object detection unit 302, an image cropping unit 303, a specific player detection unit 304, a cropping coordinate determination unit 305, a cropped image reduction unit 306, and a behavior recognition unit 307.

[0034] The image input to the player behavior recognition module is, for example, an image (bird's-eye view image 100) of an entire basketball court as shown in Fig. 4. For example, basketball court 400 includes a player 401, a basketball 402, and a goal hoop 403. Bird's-eye view camera 106 takes an image so that the basketball court fits within the angle of view (so that no part of the court is cut off), as shown in Fig. 5, for example, and outputs the captured bird's-eye view image 100. Note that bird's-eye view camera 106 outputs the image of the entire basketball court as a video or a still image.

[0035] The captured image may include, for example, 3840 pixels horizontally and 2160 pixels vertically, but the number of pixels is not limited to this and may be an image with a different number of pixels. The image is output from the overhead camera 106 in a format conforming to HDMI (High-Definition Multimedia Interface) (registered trademark) or SDI (Serial Digital Interface). Note that the overhead image 100 may be an image that is temporarily recorded on a recording medium (not shown) in the overhead camera and then read out (exported).

[0036] The image reducing unit 301 reduces the overhead image 100 to an image suitable for processing by the object detection unit 302 at the subsequent stage. As described above, the overhead image 100 is composed of, for example, 3840 pixels horizontally and 2160 pixels vertically. If this image is input directly to the object detection unit 302, the large number of pixels would increase the processing load on the object detection unit 302. Therefore, the image reducing unit 301 reduces the number of pixels of the overhead image 100 from 3840 pixels horizontally and 2160 pixels vertically to an image with 400 pixels horizontally and 400 pixels vertically, and outputs the reduced image signal 351. The number of pixels of the reduced image signal 351 is not limited to the above and may be set appropriately depending on the processing capacity of the object detection unit 302.

[0037] As shown in FIG. 6, the object detection unit 302 detects the player and the basketball on the basketball court from the reduced image signal 351. The object detection unit 302 executes the inference stage process using, for example, a deep neural network trained by the learning server 102 described above, to detect the player and the basketball (outputting player coordinates 352 and basketball ball coordinates 353 within the image). This deep neural network is trained to detect the player as a whole, rather than specific parts of the player's body. In other words, the player is detected based on the overall appearance of the player's body.

[0038] A plurality of player coordinate values ​​are detected and output from the object detection unit 302 as multiple player coordinates 352. The ball coordinate values ​​are output from the object detection unit 302 as ball coordinates 353. The player and ball coordinate values ​​may be, for example, the upper left, lower left, upper right, and lower right coordinate values ​​of a rectangle. Note that this embodiment will be described taking as an example a case where a basketball and a player are detected in a basketball game, but in the case of ice hockey, a puck may be detected instead of the ball.

[0039] In this embodiment, an example is described in which deep learning is used, which uses a neural network to automatically adjust the feature values ​​and connection weighting coefficients for learning. However, as a specific machine learning algorithm, any suitable algorithm may be applied to this embodiment, such as nearest neighbor algorithm, naive Bayes algorithm, decision tree, or support vector machine. The detection results of the player, etc., obtained by the inference stage processing may be represented by rectangular coordinate values, as shown in FIG. 6.

[0040] As is clear from the above explanation, the object detection unit 302 detects the ball and the player through deep neural network processing that inputs an image with fewer pixels than the overhead image 100. Therefore, the amount of calculation is less than that of a deep neural network that inputs the overhead image 100, and the detection process can be performed faster or with less power consumption.

[0041] The specific player detection unit 304 outputs specific player coordinates 354 from the multiple player coordinates 352 and the ball coordinates 353. The specific player coordinates 354 are determined according to the positional relationship between the ball coordinates 353 and the multiple player coordinates 352. For example, the specific player detection unit 304 first determines the center position of each coordinate from the ball coordinates 353 and the multiple player coordinates 352. For example, if the upper left coordinate value is (100, 100), the lower left coordinate value is (100, 300), the upper right coordinate value is (300, 100), and the lower right coordinate value is (300, 300), the center position of the coordinates is (200, 200).

[0042] Next, the specific player detection unit 304 detects the center positions of the coordinates of the closest players using the center position of the ball coordinates as a reference. Here, "closest" means that the distance between the centers is closest. In the example shown in FIG. 7, the multiple players closest to the center position of the ball coordinates 353 are the specific player coordinates 354. Note that the method for detecting the specific player coordinates 354 is not limited to the above, and multiple player coordinates that are close (within a predetermined distance) from the center position of the ball coordinates 353 may also be detected. Specifically, as shown in FIG. 8, the coordinates of the multiple players that are within a predetermined distance from the center position of the ball coordinates 353 are the specific player coordinates 354.

[0043] The trimming coordinate determination unit 305 determines image trimming coordinates from the specific player coordinates 354 and outputs them as trimming coordinates 355. When there is only one specific player coordinates 354 as shown in Fig. 7, the same coordinate values ​​as the specific player coordinates 354 are output as trimming coordinates 355 as shown in Fig. 9.

[0044] 7, the trimming coordinate determination unit 305 determines rectangular coordinates that include the multiple specific player coordinates 354, as shown in Fig. 10, and outputs them as trimming coordinates 355. The trimming coordinates 355 may be the coordinate values ​​of the top left, bottom left, top right, and bottom right of the rectangle (i.e., they represent the range in which the image is trimmed).

[0045] The image trimming unit 303 determines a trimmed image 356 from the overhead image 100 and the trimming coordinates 355. The image of the overhead image 100 is trimmed at the coordinate values ​​corresponding to the trimming coordinates 355.

[0046] The trimmed image reducing unit 306 reduces the trimmed image 356 to an image suitable for processing by the behavior recognition unit 307 at a subsequent stage. The number of pixels in the trimmed image 356 changes depending on the trimming coordinates 355. For example, as shown in FIG. 10 , when there are multiple specific player coordinates 354, the rectangle of the trimming coordinates 355 may become large, in which case the number of pixels in the trimmed image 356 increases. Here, if the number of pixels in the trimmed image 356 increases, the processing load on the behavior recognition unit 307 increases, so the trimmed image reducing unit 306 reduces the image to be input to the behavior recognition unit 307.

[0047] For example, if the trimmed image 356 is composed of 500 pixels horizontally and 300 pixels vertically, the trimmed image reduction unit 306 reduces the image to 200 pixels horizontally and 200 pixels vertically, and outputs it as trimmed reduced image 357. Note that the image size after reduction is not limited to the above, and can be determined depending on the processing capacity of the behavior recognition unit 307.

[0048] The action recognition unit 307 recognizes the player's action from the trimmed reduced image 357 and outputs it as an action recognition result 358. The actions recognized by the action recognition unit 307 include, for example, actions performed by a player in a basketball game, such as shooting, passing, and dribbling. The action recognition by the action recognition unit 307 may be detected, for example, by deep learning. In this embodiment, for example, the learning server 102 trains a learning model to recognize shooting, passing, and dribbling of a basketball player. The action recognition unit 307 executes inference processing of the learning model using, for example, parameters of the trained model received from the learning server 102. The action recognition unit 307 recognizes the action of a specific player by inputting the trimmed reduced image 357.

[0049] The behavior recognition unit 307 may recognize the player's behavior based on spatial features of one trimmed reduced image 357. In this case, the behavior recognition unit 307 recognizes the player's behavior using, for example, a deep neural network configured to recognize behavior from spatial features. The behavior recognition unit 307 may also be configured to recognize the player's behavior based on time-series features using time-series trimmed reduced images 357 corresponding to each frame of the video. In this case, the behavior recognition unit 307 may recognize the player's behavior using a deep neural network configured to recognize behavior from time-series features. The behavior recognition unit 307 outputs the result of recognizing the player's behavior as the behavior recognition result 358.

[0050] As is clear from the above explanation, the behavior recognition unit 307 recognizes the behavior of a specific player through processing by a deep neural network that inputs an image with fewer pixels than an overhead image. Therefore, the amount of calculation required is less than that required for a deep neural network that inputs the overhead image 100, and behavior recognition processing can be performed faster or with less power consumption.

[0051] As described above, the player behavior recognition process of this embodiment can be applied to sports other than basketball. For example, consider the case where the above-described player behavior recognition process is applied to soccer. When determining multiple players who are close to the center position of the ball at the specific player coordinates 354, players who are closer than in the case of basketball are detected.

[0052] When the entire soccer court 1100 is captured in the reduced image signal 351 as shown in FIG. 11 , the soccer court is larger than the basketball court, so the players and the soccer ball appear relatively smaller. In other words, the actual distance per pixel in the reduced image signal 351 is greater than that in the basketball court, so it is necessary to detect players closer to the ball than in the basketball court. Therefore, for example, the distance between the player and the ball in the image when identifying a specific player may be set to a different value depending on the ratio between the size of the court and the size of the player. Furthermore, when the above embodiment is applied to soccer, the behavior recognition unit 307 recognizes actions of soccer players, such as shooting, passing, and heading.

[0053] Also, for example, if the object detection unit 302 fails to detect the player and the ball (for example, if it fails in a particular frame), it may use the coordinate values ​​in the last frame in which detection was successful (i.e., the coordinate values ​​of the most recent successful detection). This is because detection of both the player and the ball may fail if two players overlap or if the ball is hidden behind a player.

[0054] As described above, in this embodiment, an image is acquired that is captured so that the court of a sports competition is included within the angle of view, multiple objects (players, ball, etc.) are detected within the image, and an area identified based on the positions of these objects within the image is trimmed. When trimming, an area including a specific player is trimmed based on the positions of objects used in the game (ball, puck, etc.). The behavior of the specific player is then recognized based on the trimmed image. In this way, it is possible to appropriately recognize the behavior of a player even when using a captured image that includes the court within the angle of view.

[0055] (Embodiment 2) In the second embodiment, a method for detecting overlaps between the coordinates of multiple players and the coordinates of the ball and determining a specific player will be described. In this embodiment, a part of the configuration of the player behavior recognition module (overlapping player detection unit) is different from that of the first embodiment, but the other configurations are substantially similar to those of the first embodiment. Therefore, the substantially same configurations are assigned the same reference numerals, and redundant explanations will be omitted, and the explanation will focus on the differences.

[0056] (Configuration for player behavior recognition) The configuration for player behavior recognition in the second embodiment will be described with reference to Fig. 12. As with the first embodiment, the configuration shown in Fig. 12 is a software configuration realized by the CPU 212 of the image processing device 105 executing a program. The configuration shown in Fig. 12 includes an image reduction unit 301, an object detection unit 302, an image cropping unit 303, a specific player detection unit 304, a cropping coordinate determination unit 305, a cropped image reduction unit 306, a behavior recognition unit 307, and an overlapping player detection unit 1201. Of these, the configuration other than the overlapping player detection unit 1201 is substantially the same as that of the first embodiment.

[0057] The overlapping player detection unit 1201 outputs overlapping coordinates 1202 from the multiple player coordinates 352 and ball coordinates 353 output from the object detection unit 302 .

[0058] First, the overlapping player detection unit 1201 detects whether the rectangles of the multiple player coordinates 352 and the ball coordinates 353 overlap. For example, FIG. 13 shows a case where the rectangles of the multiple player coordinates 352 and the ball coordinates 353 overlap. If the overlapping player detection unit 1201 determines that the rectangles of the multiple player coordinates 352 and the ball coordinates 353 overlap, it outputs the coordinate values ​​of the overlapping players as overlapping coordinates 1202. Here, if there are multiple rectangles of the player coordinates 352 that overlap with the rectangle of the ball coordinates 353, the overlapping player detection unit 1201 outputs the coordinate value of the player that overlaps the rectangle of the ball coordinates 353 to the greatest extent as the overlapping coordinates 1202. On the other hand, if there is no rectangle of the player coordinates 352 that overlaps with the rectangle of the ball coordinates 353, information indicating no coordinate value is output from the overlapping coordinates 1202.

[0059] The specific player detection unit 304 determines specific player coordinates 354 from the multiple player coordinates 352, the ball coordinates 353, and the overlap coordinates 1202. If player coordinate values ​​are input to the overlap coordinates 1202, the specific player detection unit 304 outputs only the player coordinate values ​​indicated by the overlap coordinates 1202 as the specific player coordinates 354. Furthermore, if information without coordinate values ​​is input to the overlap coordinates 1202, the specific player detection unit 304 performs the same operation as in the first embodiment. That is, the specific player detection unit 304 outputs the coordinate values ​​of the player closest (or within a predetermined distance) to the center position of the ball coordinates 353 as the specific player coordinates 354. This makes it possible to identify the player closest to the ball even when players are clustered in random locations on the court, thereby enabling appropriate trimming of the player whose behavior is to be recognized. Ultimately, it becomes possible to appropriately recognize the behavior of players even when using a photographed image that includes the court within the field of view.

[0060] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0061] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0062] 301 image reduction unit, 302 object detection unit, 303 image trimming unit, 304 specific player detection unit, 305 trimming coordinate determination unit, 306 trimmed image reduction unit, 307 behavior recognition unit

Claims

1. an acquisition means for acquiring an image captured so that the sports court is included within the angle of view; a detection means for detecting a plurality of objects in the image; a trimming means for trimming a range specified based on the positions of the plurality of objects in the image; recognition means for recognizing an action of a specific player among the plurality of objects based on the trimmed image; The image processing device is characterized in that the trimming means trims an area including the specific player identified based on the position of a first object used in the sporting event among the plurality of objects.

2. 2. The image processing device according to claim 1, wherein the trimming means trims a range including the specific player, which is a player located within a predetermined distance from the position of the first object.

3. 3. The image processing apparatus according to claim 2, wherein the trimming means trims a range including the specific player, which is the player located closest to the position of the first object.

4. 4. The image processing device according to claim 1, wherein when a rectangle detected as a player overlaps with a rectangle detected as the first object, the player is identified as the specific player, and the trimming means trims a range including the specific player.

5. 3. The image processing device according to claim 2, wherein the predetermined distance varies depending on the ratio between the size of the court of the sporting event and the size of the player.

6. a first reduction means for reducing the image acquired by the acquisition means; 6. The image processing apparatus according to claim 1, wherein the detecting means detects the plurality of objects in the image by using the reduced image.

7. a second reduction means for reducing the image trimmed by the trimming means; 7. The image processing device according to claim 1, wherein said recognition means recognizes the action of said specific player by using a trimmed and reduced image.

8. The image processing device according to any one of claims 1 to 7, characterized in that the recognition means recognizes the behavior of the specific player by processing a learning model that inputs an image with a smaller number of pixels than the image acquired by the acquisition means.

9. 9. The image processing device according to claim 1, wherein the detection means detects the plurality of objects by processing a learning model that inputs an image having a smaller number of pixels than the image acquired by the acquisition means.

10. a control means for controlling zooming of a second camera that captures an image of a part of the court, the second camera being different from a first camera that captures an image of the court within its angle of view; 10. The image processing device according to claim 1, wherein the control means controls the second camera to take a close-up image of the specific player performing the predetermined action in response to the recognition means recognizing the predetermined action.

11. 11. The image processing device according to claim 1, wherein the trimming means trims an image including the specific player, which is one or more players.

12. A control method for an image processing device, comprising: an acquisition step of acquiring an image captured so that the sports court is included within an angle of view; a detection step of detecting a plurality of objects in the image; a trimming step of trimming a range identified based on the positions of the plurality of objects within the image; a recognition step of recognizing an action of a specific player among the plurality of objects based on the trimmed image, A control method for an image processing device, characterized in that the trimming process trims an area including the specific player identified based on the position of a first object used in the sports competition among the multiple objects.

13. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images