OBJECT POSITION DETECTION SYSTEM, OBJECT POSITION DETECTION DEVICE, AND OBJECT POSITION DETECTION METHOD

Abstract

To easily detect the position of a moving object with high accuracy. [Solution] The object position detection system includes a ground contact position detection unit that detects a ground contact position where an object present on a specific surface touches the specific surface in an captured image captured by a single monocular camera, an object position estimation unit that estimates the position of the object in the captured image based on the ground contact position detected by the ground contact position detection unit, and an object position detection unit that detects the position of the object on the specific surface by converting the position of the object in the captured image estimated by the object position estimation unit into a position on the specific surface.

Description

[Technical field]

[0001] The present invention relates to an object position detection system, an object position detection device, and Object position detection method By law Regarding. [Background technology]

[0002] Techniques for detecting the position of a moving object (e.g., a person) have been disclosed. For example, Patent Document 1 discloses a technique for an object position estimation device that estimates the position of a moving object or person in space from an image captured using one or more cameras. The technique disclosed in Patent Document 1 positions the XY plane as the bottom edge (floor, ground, etc.) of the monitored area, and estimates the position of a person or object by assuming that the person or object moves on this plane. Patent Document 2 also discloses a technique for acquiring three-dimensional information of a moving object using three or more cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4675368 [Patent Document 2] Patent No. 3631266 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the technology disclosed in Patent Document 1 assumes that moving objects and people move on a plane, and does not assume three-dimensional movement. Therefore, when there is three-dimensional movement accompanied by jumping movements in sports, etc., the position of a moving object or person cannot be detected. In addition, the technology disclosed in Patent Document 1 can obtain highly redundant information about moving objects using three or more cameras, but since it is necessary to install many cameras (three or more) in advance, there is no freedom in the location of the detection target, and the cost burden and the burden of installing the cameras are large and it is not easy.

[0005] The present invention has been made in consideration of the above circumstances, and provides an object position detection system and an object position detection device that can easily and accurately detect the position of a moving object. and Object position detection method Law One of the aims is to provide [Means for solving the problem]

[0006] The present invention has been made to solve the above-mentioned problems, and one aspect of the present invention is an object position detection system comprising: a ground position detection unit that detects a ground position where an object present on a specific surface touches the specific surface in an captured image captured by a single monocular camera; an object position estimation unit that estimates a position of the object in the captured image based on the ground position detected by the ground position detection unit; and an object position detection unit that detects the position of the object on the specific surface by converting the position of the object in the captured image estimated by the object position estimation unit into a position on the specific surface.

[0007] Furthermore, one aspect of the present invention is an object position detection system comprising an object grounding determination unit that determines whether the object captured in the captured image is in contact with the specific surface, and the object position estimation unit estimates the position of the object in the captured image based on the grounding position detected by the grounding position detection unit when the object is in contact with the specific surface.

[0008] Also, one aspect of the present invention is that, in the above-mentioned object position detection system, the object position estimation unit estimates the position of the object in the captured image when the object is not in contact with the specific surface based on the contact position detected when the object is in contact with the specific surface.

[0009] Moreover, one aspect of the present invention is that, in the above-mentioned object position detection system, the ground contact position detection unit and the object ground contact determination unit each detect the ground contact position and determine whether or not the object is in contact with the specific surface at a synchronized predetermined time interval, and the object position estimation unit estimates the position of the object in the captured image when the object is not in contact with the specific surface by interpolating based on the ground contact position detected when the object is in contact with the specific surface based on the ground contact position detected by the ground contact position detection unit at the time interval and the determination result determined by the object ground contact determination unit at the time interval.

[0010] Moreover, one aspect of the present invention is that, in the above-mentioned object position detection system, the object position estimation unit tracks the position of the object in the captured image based on the grounding position detected by the grounding position detection unit at the time interval and the judgment result judged by the object grounding judgment unit at the time interval, and the object position detection unit tracks the position of the object on the specific surface by converting the position of the object in the captured image estimated by the object position estimation unit into a position on the specific surface.

[0011] In one aspect of the present invention, in the object position detection system, the object position detection unit detects a moving speed, a moving amount, or an acceleration of the object based on the tracked position of the object on the specific surface.

[0012] In addition, one aspect of the present invention is that the above-mentioned object position detection system further includes a position information output unit that outputs display information that visualizes position information indicating the position of the object on the specific surface detected by the object position detection unit.

[0013] In addition, one aspect of the present invention is that in the above-mentioned object position detection system, when the grounding position detection unit detects multiple grounding positions for one object, the object position estimation unit estimates a predetermined position between the multiple grounding positions as the position of the object.

[0014] In addition, one aspect of the present invention is an object position detection system that includes a learning unit that generates a trained model for detecting the grounding position where the object touches the specific surface in the captured image by machine learning using a dataset that includes a plurality of training images in which an object present on the specific surface is captured and grounding position information indicating the grounding position of the object captured in each of the plurality of training images where the object touches the specific surface, and the grounding position detection unit detects the grounding position where the object touches the specific surface in the captured image using the trained model generated by the learning unit.

[0015] In addition, one aspect of the present invention is a system for detecting object position, comprising a learning unit that performs machine learning using a dataset including a plurality of training images of an object present on the specific surface and information indicating whether the object captured in each of the plurality of training images is in contact with the specific surface, thereby generating a trained model for determining whether the object captured in the captured image is in contact with the specific surface, and the object contact determination unit uses the trained model generated by the learning unit to determine whether the object captured in the captured image is in contact with the specific surface.

[0016] Moreover, one aspect of the present invention is an object position detection device that includes a ground contact position detection unit that detects a ground contact position where an object present on a specific surface touches the specific surface in an image captured by a single monocular camera, an object position estimation unit that estimates a position of the object in the captured image based on the ground contact position detected by the ground contact position detection unit, and an object position detection unit that detects the position of the object on the specific surface by converting the position of the object in the captured image estimated by the object position estimation unit into a position on the specific surface.

[0017] In addition, one aspect of the present invention is an object position detection device comprising an object grounding determination unit that determines whether the object captured in the captured image is in contact with the specific surface, and the object position estimation unit estimates the position of the object in the captured image based on the grounding position detected by the grounding position detection unit when the object is in contact with the specific surface.

[0018] Another aspect of the present invention is an object position detection method in an object position detection system, the object position detection method including: a ground position detection step in which a ground position detection unit detects a ground position at which an object present on a specific surface is in contact with the specific surface in an captured image captured by a single monocular camera; an object position estimation step in which an object position estimation unit estimates a position of the object in the captured image based on the ground position detected by the ground position detection step; and an object position detection step in which the object position detection unit detects the position of the object on the specific surface by converting the position of the object in the captured image estimated by the object position estimation step into a position on the specific surface.

[0019] Another aspect of the present invention is a learning device that includes a learning unit that generates a trained model for detecting a grounding position where an object exists on a specific surface in an image in which the object exists on the specific surface, by performing machine learning using a dataset that includes a plurality of training images in which an object existing on the specific surface is captured, and grounding position information indicating a grounding position where the object, which is captured in each of the plurality of training images, is in contact with the specific surface.

[0020] Another aspect of the present invention is a learning device that includes a learning unit that performs machine learning using a dataset that includes a plurality of training images of an object existing on a specific surface and information indicating whether the object captured in the training images is in contact with the specific surface, thereby generating a trained model for determining whether the object captured in an image of the object existing on the specific surface is in contact with the specific surface. Effect of the Invention

[0021] According to the present invention, the position of a moving object can be easily detected with high accuracy. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is an explanatory diagram showing an overview of an object position detection system according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing a first example of a position estimation method according to the embodiment. [Diagram 3] FIG. 11 is a diagram showing a second example of a position estimation method according to the embodiment. [Figure 4] FIG. 4 is a diagram showing an example of a flow of a learning stage according to the embodiment. [Diagram 5] FIG. 4 is a diagram showing an example of a flow of a usage stage according to the embodiment. [Figure 6] FIG. 1 is a block diagram showing an example of the configuration of an object position detection system according to an embodiment. [Figure 7] FIG. 4 is a diagram showing a first example of detection result data according to the embodiment. [Figure 8] FIG. 11 is a diagram showing a second example of detection result data according to the embodiment. [Figure 9] 11A and 11B are diagrams showing examples of visualization of detection results of player positions according to an embodiment; [Figure 10] 6 is a flowchart showing an example of a position detection process according to the embodiment. [Figure 11] An explanatory diagram explaining the steps involved in machine learning using CNN. [Figure 12] An explanatory diagram explaining the learning procedure of machine learning using CNN. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] Hereinafter, one embodiment of the present invention will be described with reference to the drawings. The object position detection system according to this embodiment is a system that detects the position of a moving object and tracks the moving object using one camera (monocular camera). For example, the moving object is a person.

[0024] 1 is an explanatory diagram showing an overview of an object position detection system according to this embodiment. Here, an example will be described in which the detection target is a person playing a sport (here, a player playing a badminton match). In the object position detection system SYS according to this embodiment, a single camera 10 is used to capture images of players playing a match on a court, an information processing device 20 detects the positions of the players from the captured images, and the detection results of the positions of the players on the court surface are visualized and output on a display unit 30.

[0025] The camera 10 is a monocular camera that captures captured images such as video, and outputs the captured images to the information processing device 20. This captured image (video) is composed of a plurality of frame images captured at a predetermined frame rate. Each frame image is associated with time information indicating the timing at which it was captured.

[0026] FIG. 1(A) shows an example of an image captured by a camera 10. For example, the camera 10 is installed at the side of a court (e.g., in the spectator seats) at a match venue. The image captured by the camera 10 captures the players on the court as seen from the side of the court (e.g., in the spectator seats). The image capture range is a range that includes an area in which the positions of the players are detected, for example, a range that includes the court and its surroundings. The object position detection system SYS detects the players from the image captured by the camera 10. In the example shown in this figure, four players (player 1, player 2, player 3, player 4) are detected on the court.

[0027] The object position detection system SYS estimates the position coordinates of the player in the captured image based on the contact position where the detected player touches the court surface. Here, "position coordinates in the captured image" refers to position coordinates (pixel coordinates) in the pixel coordinate system in the captured image. The object position detection system SYS then converts the position coordinates (pixel coordinates) of the player in the captured image into position coordinates on the court surface, visualizes them, and outputs them on the display unit 30. "Position coordinates on the court surface" refers to position coordinates (court coordinates) in the coordinate system when the court is viewed from above.

[0028] FIG. 1(B) shows an example of a display that visualizes the position coordinates of players on the court. Here, the position coordinates P1, P2, P3, and P4 on the court of each of the four players (player 1, player 2, player 3, and player 4) captured in the captured image of FIG. 1(A) are indicated by marks. The object position detection system SYS detects and tracks the positions of the players at a predetermined time interval. The predetermined time interval is, for example, a time interval according to the frame rate at which the camera 10 captures images. Note that the position detection may target all images of the frames captured by the camera 10, or may target images of some frames, such as every two or three frames.

[0029] Here, when detecting the positions of players on the court, the object position detection system SYS does not use an image captured from above (the top surface) of the court, but uses an image captured by one camera 10 from the side of the court (for example, the spectator seats). This eliminates the need to install the camera 10 on the ceiling or the like in advance, and position detection can be easily performed in a general location. On the other hand, since the image is captured from the side of the court (for example, the spectator seats), there is a concern that the detection accuracy may decrease depending on the camera angle, the posture of the player, and the like, when detecting a person's circumscribing rectangle (bounding box) using a monocular camera. In addition, the method of calculating a two-dimensional position from the three-dimensional coordinates of a person's head, waist, and the like used in position detection using a compound-eye camera cannot be used.

[0030] Therefore, the object position detection system SYS detects the contact position between the player and the court surface, and estimates the player's position based on the contact position. For example, an AI (Artificial Intelligence) model for contact position detection is used to detect the contact position between the player and the court surface. In the following, the captured images used for learning are referred to as "learning images" to distinguish them from captured images for detecting the player's position.

[0031] For example, the object position detection system SYS performs machine learning using a plurality of learning images of players on the court surface and position coordinates (pixel coordinates) in the learning images of the positions where the players, who are captured in each of the plurality of learning images, touch the court surface as a data set, to generate a trained model for detecting the contact position between the player and the court surface from the captured images. Hereinafter, the position coordinates (pixel coordinates) of the positions where the player touches the court surface are referred to as "contact position coordinates." Also, the trained model for detecting the contact position is referred to as "contact position detection model." The object position detection system SYS uses the contact position detection model to detect the contact position coordinates (pixel coordinates) where the player touches the court surface in the captured images captured by the camera 10.

[0032] In addition, when multiple ground contact positions are detected for a single player (e.g., when both feet are on the court surface), the object position detection system SYS estimates a predetermined position (e.g., the midpoint) between the multiple ground contact positions as the player's position.

[0033] FIG. 2 is a diagram showing a first example of a position estimation method according to this embodiment. This diagram shows a player at the position P2 in FIG. 1. In FIG. 2(A), the player on the court steps forward with his right foot in relation to his left foot, and both feet on the court surface are open in front and behind. The position of the player at this time is the position PA on the court surface in the vertical direction from the center of the body (center of gravity). However, if the midpoint PB of the lower side of the bounding box of the detected player is simply estimated as the position of the player, it will be estimated to be in front of the actual position PA of the player, resulting in a large error. If the midpoint PB of the lower side of the bounding box is simply estimated as the position of the player in this way, the position of the player will change depending on the posture of the detection target.

[0034] As shown in Fig. 2(B), the object position detection system SYS estimates the player's position P2 based on a predetermined position (e.g., the midpoint) between the left foot ground contact position GP1 and the right foot ground contact position GP2, and therefore can accurately detect the actual position PA of the player. Note that the ground contact position between the player and the court surface (i.e., the ground contact position when the detection target is a person) can use, for example, a ground contact position detection model trained as the position of the person's shoes.

[0035] In addition, there are cases where both feet are not in contact with the court surface, such as when a player is jumping. Next, a method for detecting the position when both feet are not in contact with the court surface will be described with reference to FIG.

[0036] FIG. 3 is a diagram showing a second example of a position estimation method according to this embodiment. In FIG. 3(A), a player on the court is jumping, and both feet are not in contact with the court surface. The position of the player at this time is the position PA on the court surface in the vertical direction from the center of the body (center of gravity). However, if the midpoint PB of the lower side of the bounding box of the detected player is simply estimated as the position of the player, it will be estimated to be on the back side of the actual position PA of the player, resulting in a large error. In addition, if the position of the player's shoes is taken as the contact position between the player and the court surface, it will be estimated to be on the back side of the actual position PA of the player, since the player is not actually in contact with the court surface, resulting in a large error.

[0037] Therefore, the object position detection system SYS determines whether the target player is in contact with the court surface, and when the player is not in contact with the court surface, the system interpolates the position of the player before and after (before taking off and after landing) without using the detection result of the touchdown position. To determine whether the player is in contact with the court surface, for example, an AI (Artificial Intelligence) model for touchdown / takeoff determination is used. For example, the object position detection system SYS performs machine learning using a dataset of multiple learning images in which players on the court surface are captured, and information indicating whether the players captured in the multiple learning images are in contact with the court surface, thereby generating a trained model for determining whether the player captured in the captured image is in contact with the court surface. Hereinafter, the trained model for determining whether the player is in contact with the court surface is referred to as a "touchdown / takeoff determination model." The object position detection system SYS uses the touchdown / takeoff determination model to determine whether the player is in contact with the court surface in the captured image captured by the camera 10.

[0038] For example, as shown in Fig. 3(B), the object position detection system SYS detects the ground contact position at a predetermined time interval, and detects the player's positions as Pa, Pb, Pc, Pd, Pe, and Pf over time. The object position detection system SYS also determines whether the player is in contact with the court surface at a predetermined time interval, and determines the player's positions Pa, Pb, Pe, and Pf as being in a ground contact state, and the player's positions Pc and Pd as being not in a ground contact state (off the ground). In this case, the object position detection system SYS does not use the detected positions Pc and Pd when the player is not in contact with the court surface, but interpolates from the player's positions Pa, Pb, Pe, and Pf before and after (before taking off and after landing), and assumes that the player's positions change (move) to Pa, Pb, Pc', Pd', Pe, and Pf over time.

[0039] In this way, the object position detection system SYS uses AI to detect the contact position between the player and the court surface and to determine whether the player is in contact with the court surface, thereby enabling accurate detection and tracking of the position of a moving player (an example of a moving object) from images captured by a single camera 10 (monocular camera).

[0040] Next, with reference to Figures 4 and 5, an overview of the processes in the learning stage for generating trained models (landing position detection model and ground-off determination model) and the utilization stage for utilizing the generated trained models will be described.

[0041] 4 is a diagram showing an example of a flow of a learning stage according to the present embodiment. The learning stage is further divided into two stages: creation of a learning dataset and generation of a trained model using the learning dataset.

[0042] In the stage of creating a learning dataset, a plurality of learning images (raw data) are prepared (step S101), and a learning dataset is created from the plurality of learning images (step S102). The plurality of learning images are raw data of captured images captured by an imaging device (e.g., a monocular camera such as camera 10) of a scene where position detection is to be performed (e.g., a badminton match). The learning images include images in which players overlap, surrounding people (e.g., referees and spectators), and objects overlap.

[0043] For example, a learning data set for generating a ground contact position detection model includes a plurality of learning images of players in contact with the court surface during a match, and ground contact position coordinates (pixel coordinates) of the players captured in the learning images and the court surface. The ground contact position coordinates (pixel coordinates) are generated based on, for example, the positions of the players' shoes in the captured images.

[0044] The learning dataset for generating the ground contact position detection model may include not only learning images of players in contact with the court surface, but also learning images of players not in contact with the court surface. The learning dataset of learning images of players not in contact with the court surface does not include ground contact position coordinates (pixel coordinates), and is machine-learned as an error image, for example.

[0045] The learning dataset for generating the contact / take-off judgment model includes a plurality of learning images of players in contact with the court surface during a match and a plurality of learning images of players not in contact with the court surface, and information indicating whether the player captured in each learning image is in contact with the court surface or not. That is, information indicating that the player is in contact with the court surface (contact) is associated as a dataset with the learning images of players in contact with the court surface, and information indicating that the player is not in contact with the court surface (take-off) is associated as a dataset with the learning images of players not in contact with the court surface.

[0046] In the stage of generating the trained model, the object position detection system SYS inputs the generated training data set into a training program to perform machine learning (step S103), and outputs the trained model (step S104). As the training program, for example, a convolutional neural network (CNN) or the like is used. More specifically, for example, a training program based on Mask R-CNN, YOLO, or the like may be applied.

[0047] FIG. 5 is a diagram showing an example of a flow of a usage stage using the trained model according to this embodiment. The object position detection system SYS executes a position detection process to detect the position of a player on the court using the trained model (landing position detection model and landing / taking off judgment model). First, the object position detection system SYS detects a person from an image captured by the camera 10 and detects the position of the player in the captured image (step S201). For example, the object position detection system SYS inputs the image captured by the camera 10 to the landing position detection model and the landing / taking off judgment model, thereby detecting the landing position coordinates (pixel coordinates) of the player only from the captured image in which the player is on the court surface, and estimating the position coordinates (pixel coordinates) of the player based on the landing position coordinates (see FIG. 2).

[0048] The object position detection system SYS detects the ground contact position coordinates (pixel coordinates) of the player from the captured images at a predetermined time interval and tracks the player's position (step S202). In addition, the object position detection system SYS corrects the player's position when the player is not on the court surface by interpolating (see FIG. 3) the player's positions before and after (before taking off and after landing) (step S203).

[0049] Then, the object position detection system SYS converts the position coordinates (pixel coordinates) of the player in the captured image into position coordinates (court coordinates) on the court surface (step S204).

[0050] [Configuration of object position detection system] Next, the configuration of the object position detection system SYS will be described in detail. Fig. 6 is a block diagram showing an example of the configuration of an object position detection system SYS according to this embodiment. In this figure, the same reference numerals are used for components corresponding to those shown in Fig. 1. The object position detection system SYS shown in the figure includes a camera 10 (imaging device), an information processing device 20 (an example of an object position detection device and learning device), and a display unit 30.

[0051] The camera 10 is a video camera or a digital camera capable of capturing moving images. For example, as shown in Fig. 1, the camera 10 captures images of players playing a game on a court and outputs the captured images to the information processing device 20. This captured image (moving image) includes a plurality of frame images captured at a predetermined frame rate as described above. Each frame image is associated with time information indicating the timing at which it was captured. The camera 10 is, for example, a monocular camera.

[0052] The information processing device 20 is, for example, a computer device such as a PC (personal computer) including a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a memory, etc. The information processing device 20 executes an object position detection process for detecting the positions of players (an example of a moving object) from an image captured by the camera 10, as described with reference to Figs. 1 to 5.

[0053] For example, the information processing device 20 includes a storage unit 21 and a control unit 22 as functional components for executing an object position detection process. The storage unit 21 is a storage device that stores information used for various processes of the information processing device 20. For example, the storage unit 21 includes an image information storage unit 210, an object coordinate storage unit 211, a ground contact position coordinate storage unit 212, a ground contact / take-off frame storage unit 213, an estimated position coordinate storage unit 214, a detection result storage unit 215, a learning dataset storage unit 216, a learning program storage unit 217, and a learned model storage unit 218.

[0054] The image information storage unit 210 stores data of captured images captured by the camera 10. For example, the image information storage unit 210 stores data of captured images (moving images) acquired from the camera 10 by the image acquisition unit 220 described below. Note that the captured images (moving images) stored in the image information storage unit 210 are associated with a plurality of frame images and time information.

[0055] The object coordinate storage unit 211 stores coordinate information of a player (an example of a moving object) detected from a captured image. For example, the object coordinate storage unit 211 stores bounding box information as the coordinates of a player (image area of ​​a player) detected for each frame image of a captured image.

[0056] The ground contact position coordinate storage unit 212 stores the ground contact position coordinates (pixel coordinates) of the player detected from the captured image and the court surface. For example, the ground contact position coordinate storage unit 212 stores the ground contact position coordinates (pixel coordinates) of the player touching the court surface in association with each frame image of the captured image.

[0057] The touchdown / takeoff frame storage unit 213 stores the determination result of whether or not the player detected from the captured image is in contact with the court surface. For example, the touchdown / takeoff frame storage unit 213 stores information on whether the player is in contact with or has taken off the court surface in association with each frame image of the captured image.

[0058] The estimated position coordinate storage unit 214 stores the position coordinates (pixel coordinates) of the player estimated based on the ground contact coordinates of the player touching the court surface. For example, the estimated position coordinate storage unit 214 stores the position coordinates (pixel coordinates) of the player estimated based on the ground contact coordinates of the player for each frame image of the captured image in association with each other.

[0059] The detection result storage unit 215 stores, as detection results, position data of the position coordinates (court coordinates) of the players on the court surface detected for each frame image. The detection result storage unit 215 also stores, as detection results, data such as the movement speed, movement amount, and acceleration calculated based on the position coordinates of the players.

[0060] The learning dataset storage unit 216 stores a learning dataset for generating a touchdown position detection model and a touchdown / ground-off judgment model. As described above, for example, a learning dataset for generating a touchdown position detection model is associated with a plurality of learning images of a player in a state of touching the court surface during a match, and the touchdown position coordinates (pixel coordinates) of the player captured in the learning images and the court surface. In addition, a learning dataset for generating a touchdown / ground-off judgment model is associated with a plurality of learning images of a player in a state of touching the court surface during a match and a plurality of learning images of a player not in a state of touching the court surface, and information indicating whether or not the player captured in each learning image is in contact with the court surface.

[0061] The learning program storage unit 217 stores, as a learning program, for example, a learning program based on a convolutional neural network (CNN), more specifically, Mask R-CNN, YOLO, or the like.

[0062] The trained model memory unit 218 stores a touchdown position detection model and a touchdown / takeoff judgment model that are generated by machine learning by inputting the learning dataset stored in the training dataset memory unit 216 into the training program stored in the training program memory unit 217.

[0063] Next, the configuration of the control unit 22 will be described. The control unit 22 is a functional configuration realized by, for example, a processor such as a CPU executing a program stored in a memory, and controls the information processing device 20 in an integrated manner. For example, the control unit 22 includes an image acquisition unit 220, an object detection unit 221, a ground position detection unit 222, an object grounding determination unit 223, an object position estimation unit 224, a coordinate conversion unit 225, a position information generation unit 226, a position information output unit 227, and a trained model generation unit 228.

[0064] The image acquisition unit 220 acquires a captured image captured by the camera 10. For example, as shown in Fig. 1(A), the image acquisition unit 220 acquires a captured image (video) of a player (an example of a moving object) present on the court whose position is to be detected from the camera 10, and stores the image in the image information storage unit 210. Note that the image acquisition unit 220 (an example of an acquisition unit) may acquire the captured image captured by the camera 10 from a computer-readable recording medium, and store the image in the image information storage unit 210.

[0065] The object detection unit 221 detects an image area including the player to be detected from the captured image acquired by the image acquisition unit 220, and stores information on a bounding box corresponding to the image area in the object coordinate storage unit 211.

[0066] The ground contact position detection unit 222 detects the ground contact position where the player on the court touches the court surface in the captured image of the player on the court. For example, the ground contact position detection unit 222 detects the ground contact position coordinates (pixel coordinates) where the player touches the court surface for each frame image from the captured image acquired by the image acquisition unit 220 using a ground contact position detection model, and stores the detected ground contact position coordinates in the ground contact position coordinate storage unit 212.

[0067] The object grounding determination unit 223 determines whether or not the player captured in the captured image acquired by the image acquisition unit 220 is in contact with the court surface. For example, the object grounding determination unit 223 uses a grounding / take-off determination model to determine whether or not the player captured in the captured image is in contact with the court surface for each frame image, and stores information on whether the player is in contact with or off the court surface in the grounding / take-off frame storage unit 213 in association with the information.

[0068] The object position estimation unit 224 estimates the position of the player in the captured image based on the ground contact position detected by the ground contact position detection unit 222. For example, the object position estimation unit 224 estimates position coordinates (pixel coordinates) indicating the position of the player in the captured image based on the ground contact position coordinates detected by the ground contact position detection unit 222 when the player is in contact with the court surface. As an example: When the ground contact position detection unit 222 detects multiple ground contact positions (e.g., ground contact points of both feet) for one player, the object position estimation unit 224 estimates the position coordinates of a predetermined position (e.g., a midpoint) between the multiple ground contact position coordinates as the player's position coordinates (pixel coordinates).

[0069] Furthermore, the object position estimation unit 224 estimates the position of the player in the captured image when the player is not in contact with the court surface based on the contact position detected when the player is in contact with the court surface. For example, the object position estimation unit 224 estimates the position coordinates (pixel coordinates) of the player in the captured image when the player is not in contact with the court surface by interpolating from the position coordinates (pixel coordinates) of the player when the player is in contact with the court surface. As an example, the object position estimation unit 224 estimates the position coordinates (pixel coordinates) of the player in the captured image when the player is not in contact with the court surface by interpolating from the position coordinates of the player before and after (before taking off and after landing) (see FIG. 3).

[0070] The object position estimation unit 224 associates the estimated position coordinates (pixel coordinates) of the player with each frame image of the captured image and stores them in the estimated position coordinate storage unit 214. The object position estimation unit 224 also tracks the position coordinates of the player in the captured image based on the ground contact position coordinates (pixel coordinates) detected by the ground contact position detection unit 222 at predetermined time intervals and the determination results determined by the object ground contact determination unit 223 at predetermined time intervals.

[0071] The coordinate conversion unit 225 functions as an object position detection unit that detects the position coordinates (court coordinates) of the player on the court surface by converting the position coordinates (pixel coordinates) of the player in the captured images estimated by the object position estimation unit 224 into position coordinates (court coordinates) on the court surface. For example, the coordinate conversion unit 225 tracks the position coordinates (court coordinates) of the player on the court surface by converting the position coordinates (pixel coordinates) of the player in the captured images trunked at a predetermined time interval by the object position estimation unit 224 into position coordinates (court coordinates) on the court surface.

[0072] The position information generating unit 226 generates, as a detection result, position data in which the position coordinates (court coordinates) of the player after conversion by the coordinate conversion unit 225 are associated with each frame image. The position information generating unit 226 may also calculate, as a detection result, the movement speed, movement amount, or acceleration of the player based on the tracked position coordinates (court coordinates) of the player on the court surface. The position information generating unit 226 stores, as a detection result, the generated position data, or data such as the movement speed, movement amount, or acceleration calculated based on the position data in the detection result storage unit 215.

[0073] 7 is a diagram showing a first example of detection result data according to this embodiment. In the illustrated data example, the time information (time) of a frame image, the position coordinates (x, y) of a player on the court surface, the movement speed (speed), and the movement distance (total distance) are associated with each other. When multiple players are detected from one frame image, the detection results such as the position coordinates (x, y), movement speed (speed), and movement distance (total distance) of the multiple players are associated with the time information (time) of one frame image.

[0074] 8 is a diagram showing a second example of detection result data according to this embodiment. The illustrated data example is an example in which the total movement distance of a player is calculated as the detection result. For example, the position information generating unit 226 may calculate the total movement distance for each set, the total movement distance for the entire game, etc. for each player and generate the results as the detection result.

[0075] In this way, the position information generating section 226 may generate any data as the detection result, as long as the data can be calculated based on the tracked position coordinates (court coordinates) of the player on the court surface.

[0076] Returning to Fig. 6, the position information output unit 227 outputs display information (see Fig. 1(B) for example) that visualizes the position coordinates (court coordinates) of the player on the court surface converted by the coordinate conversion unit 225. The position information output unit 227 outputs display information that visualizes tracking information of the position of a moving player (position coordinates that change over time). As an example, the position information output unit 227 may output a trajectory according to the movement as display information, based on the tracking information of the position of the moving player.

[0077] Furthermore, the position information output unit 227 may output a heat map of the position of each player for each set as display information based on tracking information of the position of the moving player. Fig. 9 is a diagram showing a display example in which the detection result of the position of the player according to this embodiment is visualized as a heat map.

[0078] For example, the position information output unit 227 outputs the visualized display information to the display unit 30. Note that the position information output unit 227 may output the visualized display information as a file that can be displayed by an application or the like or a file that can be printed by a printer.

[0079] As explained with reference to the learning stage flow in Figure 4, the trained model generation unit 228 (an example of a learning unit) inputs the learning data stored in the learning dataset memory unit 216 into the learning program stored in the learning program memory unit 217, performs machine learning, and generates a touchdown position detection model and a touchdown / takeoff judgment model.

[0080] For example, the trained model generating unit 228 performs machine learning using a dataset of a plurality of learning images of players on the court surface and position coordinates (pixel coordinates) of positions where the players captured in each of the plurality of learning images are in contact with the court surface, thereby generating a contact position detection model for detecting contact positions between the players and the court surface from the captured images. The trained model generating unit 228 also performs machine learning using a dataset of a plurality of learning images of players on the court surface and information indicating whether the players captured in the plurality of learning images are in contact with the court surface, thereby generating a contact / take-off determination model for determining whether the players captured in the captured images are in contact with the court surface. The trained model generating unit 228 then stores the generated contact position detection model and contact / take-off determination model in the trained model storage unit 218.

[0081] Next, details of a position detection process (flow of a utilization stage) in which the information processing device 20 detects the position of a moving object (e.g., a player) by using a trained model will be described with reference to Fig. 10. Fig. 10 is a flowchart showing an example of the position detection process executed by the information processing device 20 according to the present embodiment.

[0082] (Step S211) The information processing device 20 acquires a captured image captured by the camera 10. For example, as shown in Fig. 1(A), the information processing device 20 acquires a captured image (video) of a player (an example of a moving object) present on the court whose position is to be detected, from the camera 10. Then, the process proceeds to step S215.

[0083] (Step S215) The information processing device 20 detects an image area including the player to be detected from the unanalyzed frame image. For example, the information processing device 20 detects an image area (bounding box) including the player from the frame image. Then, the process proceeds to step S217.

[0084] (Step S217) The information processing device 20 detects whether the player captured in the frame image is in contact with the court surface. For example, the information processing device 20 uses a touchdown / takeoff determination model to detect whether the frame image shows the player in contact with the court surface (touch frame) or in contact with the court surface (takeoff frame). Then, the process proceeds to step S219.

[0085] (Step S219) The information processing device 20 determines whether or not the frame image shows the player in contact with the court surface (ground frame). If the information processing device 20 determines that the frame image shows the player in contact with the court surface (ground frame) (YES), the information processing device 20 proceeds to step S221. On the other hand, if the information processing device 20 determines that the frame image shows the player off the court surface (ground frame) (NO), the information processing device 20 proceeds to step S225.

[0086] (Step S221) The information processing device 20 detects the ground contact position where the player touches the court surface in the frame image. For example, the ground contact position detection unit 222 detects the ground contact position coordinates (pixel coordinates) where the player touches the court surface from the frame image using a ground contact position detection model. Then, the process proceeds to step S223.

[0087] (Step S223) The information processing device 20 estimates position coordinates (pixel coordinates) indicating the position of the player in the frame image based on the ground contact position coordinates (pixel coordinates) detected in step S221. As an example, when multiple ground contact positions (e.g., ground contact points of both feet) are detected for one player, the information processing device 20 estimates the position coordinates of a predetermined position (e.g., the midpoint) between the multiple ground contact position coordinates (pixel coordinates) as the player's position coordinates (pixel coordinates). Then, the process proceeds to step S227.

[0088] (Step S225) The information processing device 20 sets the position coordinates (pixel coordinates) of the player at the time of the takeoff frame to be empty, and then proceeds to step S227.

[0089] (Step S227) The information processing device 20 determines whether there is an unanalyzed frame image (a frame image for which position detection has not been performed) in the acquired captured image (video). If the information processing device 20 determines that there is an unanalyzed frame image (NO), the process returns to step S215. On the other hand, if the information processing device 20 determines that there is no unanalyzed frame image (YES), the process proceeds to step S229.

[0090] Note that, after determining in step S219 that a frame is a takeoff frame and performing the process of step S225, if N seconds' worth of touchdown frames (for example, 1 second's worth) have been detected and the player's position coordinates have been estimated, the information processing device 20 may proceed to step S229 even before determining in step S227 that there are no unanalyzed frame images. Here, N seconds (for example, 1 second's worth) is set as the time required to interpolate the player's position coordinates in the takeoff frames from the player's position coordinates in the touchdown frames.

[0091] (Step S229) The information processing device 20 interpolates the position coordinates (pixel coordinates) of the player in the takeoff frame from the position coordinates (pixel coordinates) of the player in the contact frame. For example, the information processing device 20 estimates the position coordinates (pixel coordinates) of the player in the takeoff frame by interpolating from the position coordinates (pixel coordinates) of the player before and after (before takeoff and after landing). Then, the process proceeds to step S231.

[0092] (Step S231) The information processing device 20 converts the position coordinates (pixel coordinates) of the player in the frame image into position coordinates on the court surface (court coordinates), and then proceeds to step S233.

[0093] (Step S233) The information processing device 20 visualizes and outputs the tracking information of the player's position coordinates (court coordinates) converted in step S231 (i.e., the detection result of the player's position) (for example, see FIG. 1(B)). Note that the information processing device 20 may output a trajectory according to movement as display information based on the tracking information of the player's position coordinates (court coordinates). Also, the information processing device 20 may output a heat map of the position of each player as display information based on the tracking information of the player's position coordinates (court coordinates).

[0094] [Examples of learning programs] Here, a specific example of a learning program will be described. As described above, the learning program may be, for example, a convolutional neural network (CNN), more specifically, Mask R-CNN, etc. Below, the convolutional neural network (CNN) will be described as an example of machine learning.

[0095] About CNN FIG. 11 is an explanatory diagram illustrating the procedure for executing machine learning using CNN. In this diagram, the CNN is composed of I+1 layers L0 to LI. Layer L0 is also called the input layer, layers L1 to L(I-1) are called intermediate layers or hidden layers, and layer LI is called the output layer. I is determined by the structure of the CNN, for example, I=3 or 4.

[0096] In the CNN, an input image is input to the input layer L0. The input image is represented by a pixel matrix D11 in which the vertical and horizontal positions of the input image are the positions in the matrix. Each element of the pixel matrix D11 is input with the R (red), G (green), and B (blue) subpixel values ​​as the subpixel values ​​of the pixel corresponding to the position in the matrix. The first hidden layer, L1, is the layer where convolutional processing (also called filtering) and pooling processing are performed.

[0097] (Convolution processing) An example of the convolution process in the intermediate layer L1 will be described below. The convolution process is a process of applying a filter to an original image and outputting a feature map. Specifically, the input pixel values ​​are divided into an R subpixel matrix D121, a B subpixel matrix D122, and a G subpixel matrix D123. In each of the subpixel matrices D121, D122, and D123 (each also referred to as a "subpixel matrix D12"), the elements of the submatrix with s rows and t columns are multiplied by the elements of a convolution matrix CM1 (also called a kernel) with s rows and t columns, and then added together to calculate a first pixel value. The first pixel values ​​calculated in each of the subpixel matrices D12 are multiplied by a weighting coefficient and then added together to calculate a second pixel value. The second pixel value is set in each element of the convoluted image matrix D131 as a matrix element corresponding to the position of the submatrix. The position of the submatrix in each of the subpixel matrices D12 is shifted for each element (subpixel), and thus the second pixel value at each position is calculated, and all the matrix elements of the convoluted image matrix D131 are calculated.

[0098] 11 shows an example of a convolution matrix CM1 with 3 rows and 3 columns, where the first pixel value of the convolved pixel value D1311 is calculated for the submatrix with 3 rows and 3 columns from the second row to the fourth row and the second column to the fourth column of each subpixel matrix D12. A weighting coefficient is calculated and added to the first pixel value of each subpixel matrix D121, D122, and D123, whereby a second pixel value is calculated as the matrix element in the second row and the second column of the convolved image matrix D131. Similarly, the second pixel value of the matrix element in the third row and the second column of the convolved image matrix D131 is calculated from the submatrix with the third row to the fifth row and the second column to the fourth column. Similarly, convolved image matrices D132, . . . are calculated using other weighting coefficients or other convolution matrices.

[0099] (Pooling process) An example of the pooling process in the hidden layer L1 will be described below. The pooling process is a process for reducing an image while retaining the features of the image. Specifically, for the convoluted image matrix D131, a representative value of matrix elements in each region PM of u rows and v columns is calculated. The representative value is, for example, a maximum value. The representative value is set for each element of the CNN image matrix D141 as a matrix element corresponding to the position of the region. By shifting the regions in the convoluted image matrix D131 for each region PM, a representative value at each position is calculated, and all matrix elements of the convoluted image matrix D131 are calculated.

[0100] For example, FIG. 11 shows an example of a 2-row, 2-column region PM, where the maximum value of the second pixel values ​​in the 2-row, 2-column region from the 3rd row to the 4th row and the 3rd column to the 4th column of the convolution image matrix D131 is calculated as a representative value. This representative value is set to the matrix element in the 2nd row, 2nd column of the CNN image matrix D141. Similarly, a representative value of the matrix element in the 3rd row, 2nd column of the CNN image matrix D141 is calculated from the submatrix in the 5th row to the 6th row and the 2nd column to the 4th column. Similarly, the CNN image matrix D142, is calculated from the convolution image matrix D132,.

[0101] Each matrix element (N elements) of the CNN image matrices D141, D142, ... is arranged in a predetermined order to generate a vector x. In FIG. 7, the element x of the vector x is n (n=1, 2, 3, ...N) is represented by N nodes.

[0102] The hidden layer Li represents the i-th hidden layer (i = 2 to I-1). From the i-th hidden layer node, a vector u (i) is the function f(u (i) ) as the input value of the vector z (i) The vector u (i) is the vector z output from the i-1th hidden layer node. (i-1) Then, the weight matrix W (i) Multiply it from the left and give vector b (i) The function f(u (i) ) is the activation function, and the vector b (i) is the bias. Also, the vector u (0) is the vector x.

[0103] The node of the output layer L4 is z (I-1) and the output is M y m (m=1, 2, . . . M). In other words, the output layer LI of the CNN gives y m Vector y(=(y1, y2, y3, y M )) is output. As a result, when the pixel values ​​of the input image are input as input variables, the CNN outputs the vector y as an output variable. The vector y represents an evaluation value.

[0104] FIG. 12 is an explanatory diagram illustrating a learning procedure of machine learning using CNN. This figure is an explanatory diagram of the case where the CNN in FIG. 11 performs machine learning. For the pixel values ​​of the images in the training dataset, the vector x output from the first hidden layer is defined as vector X. The vector representing the determined class of the training dataset is defined as vector Y.

[0105] Weight matrix W (i) An initial value is set for . When an input image is input to the input layer, and as a result, a vector X is input to the second hidden layer, a vector y(X) corresponding to vector X is output from the output layer. The error E between vector y(X) and vector Y is calculated using a loss function. The gradient ΔE of the i-th layer i is the output from each layer z i and the error signal δ i The error signal δ is calculated using i is the error signal δ i-1 This process of transmitting the error signal from the output layer to the input layer is also called backpropagation. Weight matrix W (i) is the gradient ΔE i Similarly, in the first hidden layer, the convolution matrix CM or the weighting coefficients are updated.

[0106] (Setting the trained model) The trained model generation unit 228 sets the number of layers, the number of nodes in each layer, the node connection method between each layer, the activation function, the error function, the gradient descent algorithm, the pooling region, the kernel, the weighting coefficient, and the weighting matrix for the CNN. The trained model generation unit 228 sets, for example, three layers (I=3) as the number of layers. As the number of nodes in each layer (also referred to as the "node number"), the trained model generation unit 228 sets the number of elements of vector x (node ​​number N) to 800, the number of nodes in the second intermediate layer (i=2) to 500, and the number of nodes in the output layer (i=3) to 10. However, the present invention is not limited to this, and the total number may be four or more layers, and a different value may be set for the number of nodes.

[0107] The trained model generation unit 228 sets 20 convolution matrices CM of 5 rows and 5 columns and regions PM of 2 rows and 2 columns. However, the present invention is not limited to this, and a different number of matrices or a different number of convolution matrices CM may be set. Also, a different number of matrices or regions PM may be set. The trained model generation unit 228 may perform more convolution processing or pooling processing.

[0108] The trained model generation unit 228 sets full connections as connections for each layer of the neural network. However, the present invention is not limited to this, and connections for some or all layers may be set to non-full connections. The trained model generation unit 228 sets a sigmoid function as an activation function for all layers. However, the present invention is not limited to this, and the activation function for each layer may be other activation functions such as a step function, linear combination, soft sine, soft plus, ramp function, truncated power function, polynomial, absolute value, radial basis function, wavelet, maxout, etc. Also, the activation function of a certain layer may be a different type from that of other layers.

[0109] The trained model generation unit 228 sets squared loss (mean squared error) as the error function. However, the present invention is not limited to this, and the error function may be cross entropy, τ-quantile loss, Huber loss, or ε sensitivity loss (ε tolerance error function). In addition, the trained model generation unit 228 sets SGD (stochastic gradient descent) as an algorithm (gradient descent algorithm) for calculating the gradient. However, the present invention is not limited to this, and Momentum (inertia term) SDG, AdaGrad, RMSprop, AdaDelta, Adam (adaptive moment estimation), etc. may be used as the gradient descent algorithm.

[0110] (Variations in learning) As the learning program, R(Region)-CNN, Fast R-CNN, Faster R-CNN, YOLO, etc. may be used. As the learning program, Stacked Hourglass Networks, PoseResnet, HRNet, Cascaded Pyramid Network, etc. may be used. Other neural networks such as Perceptron neural networks, Recurrent Neural Networks (RNN), Residual Networks (ResNet), Transformers, etc. may be set. Also, a trained model of supervised learning such as decision tree, regression tree, random forest, gradient boosting tree, linear regression, logistic regression, or SVM (support vector machine) may be used.

[0111] In addition, the input image for each input unit input to the learning program is not limited to a single image, but may be multiple images (video) over time. For example, a video may be divided into a certain number of frames and used as input images. This allows an AI that takes spatiotemporal information into account, eliminating ambiguity and improving temporal consistency. For example, posture estimation can be performed with high accuracy even when a part of the body is hidden (when there is occlusion) or when the image is blurred due to a sudden movement (when there is motion blur). As a learning program, 3D CNN (3D Convolutional Neural Networks), RNN (Recurrent Neural Network), Video Transformer, etc., which combine spatial information (2D) and time information (1D) for the input video and perform 3D convolution, may be used.

[0112] As described above, the information processing device 20 (an example of an object position detection device) in the object position detection system SYS according to this embodiment detects a contact position (e.g., contact position coordinates) where a player (an example of an object) present on the court surface (an example of a specific surface) touches the court surface in an image captured by the camera 10 (an example of a single monocular camera), and estimates the position (e.g., position coordinates) of the player in the captured image based on the detected contact position. The information processing device 20 then detects the position (e.g., position coordinates) of the player on the court surface by converting the estimated position (e.g., position coordinates) of the player in the captured image into a position (e.g., position coordinates) on the court surface.

[0113] As a result, the object position detection system SYS (information processing device 20) detects the contact position of a player on the court surface by capturing an image of the player with a single monocular camera, and detects the position of the player based on the contact position, so that the position of a moving player can be detected easily and accurately.

[0114] The information processing device 20 also determines whether or not a player (an example of an object) captured in a captured image is in contact with the court surface (an example of a specific surface). The information processing device 20 then estimates the position (e.g., position coordinates) of the player in the captured image based on the contact position (contact position coordinates) detected when the player is in contact with the court surface.

[0115] This allows the object position detection system SYS (information processing device 20) to easily and accurately detect the positions of the players who are in contact with the court surface.

[0116] In addition, the information processing device 20 estimates the position (e.g., position coordinates) of a player (an example of an object) in an captured image when the player is not in contact with the court surface (an example of a specific surface) based on the player's contact position (e.g., contact position coordinates) detected when the player is in contact with the court surface.

[0117] This allows the object position detection system SYS (information processing device 20) to accurately detect the positions of players who are not in contact with the court surface, based on the positions of players who are in contact with the court surface.

[0118] Furthermore, the information processing device 20 detects the contact position (e.g., contact position coordinates) where a player (an example of an object) touches the court surface (an example of a specific surface) at a synchronized predetermined time interval and determines whether or not the player is in contact with the court surface. Then, based on the contact position (e.g., contact position coordinates) of the player detected at the predetermined time interval and the determination result of whether or not the player is in contact with the court surface determined at the predetermined time interval, the information processing device 20 estimates the position (e.g., position coordinates) of the player in the captured image when the player is not in contact with the court surface by interpolating based on the contact position (e.g., contact position coordinates) detected when the player is in contact with the court surface.

[0119] This allows the object position detection system SYS (information processing device 20) to accurately detect the position of a player who is not in contact with the court surface, based on the position of the player detected when the player is in contact with the court surface.

[0120] Furthermore, the information processing device 20 tracks the position (e.g., position coordinates) of the player (an example of an object) in the captured image based on the ground contact position (e.g., ground contact position coordinates) of the player (an example of an object) detected at a predetermined time interval and the determination result of whether or not the player is on the ground determined at a predetermined time interval. Then, the information processing device 20 tracks the position (e.g., position coordinates) of the player on the court surface by converting the estimated position (e.g., position coordinates) of the player in the captured image into a position (e.g., position coordinates) on the court surface (an example of a specific surface).

[0121] This enables the object position detection system SYS (information processing device 20) to detect the positions of players on the court surface in terms of position coordinates (court coordinates) in a coordinate system when the court is viewed from above, from an image captured by a single camera 10 from the side of the court (e.g., the spectator seats).

[0122] Furthermore, the information processing device 20 detects the moving speed, moving amount, or acceleration of a player based on the position (for example, position coordinates) of the player (an example of an object) on the tracked court surface (an example of a specific surface).

[0123] As a result, the object position detection system SYS (information processing device 20) can easily and accurately detect the movement speed, movement amount, or acceleration of a player by capturing an image of the player on the court surface with one monocular camera.

[0124] Furthermore, the information processing device 20 outputs display information that visualizes position information indicating the positions (for example, position coordinates) of players (an example of an object) on the detected court surface (an example of a specific surface).

[0125] This allows the object position detection system SYS (information processing device 20) to easily display the positions of players on the court surface.

[0126] Furthermore, when multiple ground contact positions (e.g., ground contact position coordinates) are detected for one player (an example of one object), the information processing device 20 estimates a predetermined position (e.g., a midpoint) between the multiple ground contact positions (e.g., ground contact position coordinates) as the position of the player. Note that multiple ground contact positions are, for example, ground contact positions where both feet (right foot and left foot) of the player are in contact with the court surface (an example of a specific surface).

[0127] This allows the object position detection system SYS (information processing device 20) to accurately detect the position of the player regardless of the player's posture.

[0128] When only one foot of a player is in contact with the court surface, the information processing device 20 may estimate the contact position of the one foot as the position of the player. When only one foot of a player is in contact with the court surface, the information processing device 20 may determine that the player is not in contact with the court surface.

[0129] The information processing device 20 also has a function of a learning unit that generates a ground contact position detection model (an example of a trained model) for detecting the ground contact position (e.g., ground contact position coordinates) where the player touches the court surface in the captured image by machine learning using a dataset including a plurality of learning images in which a player (an example of an object) present on the court surface (an example of a specific surface) is captured, and ground contact position information indicating the ground contact position (e.g., ground contact position coordinates) of the player captured in each of the plurality of learning images.The information processing device 20 then uses the generated ground contact position detection model to detect the ground contact position (e.g., ground contact position coordinates) where the player touches the court surface in the captured image.

[0130] As a result, the object position detection system SYS (information processing device 20) can capture an image of a player on the court surface with a single monocular camera, and accurately detect the contact position of the player with the court surface using a trained model that has been machine-learned.

[0131] The information processing device 20 also has a function of a learning unit that generates a touchdown / takeoff determination model (an example of a trained model) for determining whether or not a player captured in a captured image is in contact with the court surface by machine learning using a dataset including a plurality of learning images in which a player (an example of an object) residing on the court surface (an example of a specific surface) is captured and information indicating whether or not the player captured in each of the plurality of learning images is in contact with the court surface. Then, the information processing device 20 uses the generated touchdown / takeoff determination model to determine whether or not a player captured in a captured image is in contact with the court surface.

[0132] As a result, the object position detection system SYS (information processing device 20) can accurately determine whether a player is in contact with the court surface by capturing an image of the player with a single monocular camera, using a trained model that has been machine-learned.

[0133] Furthermore, the object position detection method in the object position detection system SYS of this embodiment includes a ground contact position detection step in which the information processing device 20 detects a ground contact position (e.g., ground contact position coordinates) where a player (an example of an object) present on the court surface (an example of a specific surface) touches the court surface in an image captured by the camera 10 (an example of a single monocular camera), an object position estimation step in which the position (e.g., position coordinates) of the player in the image captured based on the ground contact position (e.g., ground contact position coordinates) detected by the ground contact position detection step, and an object position detection step in which the position (e.g., position coordinates) of the player in the image captured estimated by the object position estimation step is converted into a position (e.g., position coordinates) on the court surface to detect the position (e.g., position coordinates) of the player on the court surface.

[0134] As a result, the object position detection method in the object position detection system SYS detects the contact position of a player with the court surface by capturing an image of the player with a single monocular camera, and detects the position of the player based on the contact position, so that the position of a moving player can be easily and accurately detected.

[0135] Furthermore, the information processing device 20 according to this embodiment may function as a learning device that generates a ground contact position detection model (an example of a trained model) for detecting the ground contact position (e.g., ground contact position coordinates) at which a player touches the court surface in an image in which the player is imaged while standing on the court surface, by performing machine learning using a dataset including a plurality of learning images in which a player (an example of an object) standing on the court surface (an example of a specific surface) is captured, and ground contact position information indicating the ground contact position (e.g., ground contact position coordinates) at which the player, who is captured in each of the plurality of learning images, touches the court surface.

[0136] As a result, the object position detection system SYS (information processing device 20) can capture an image of a player on the court surface with a single monocular camera, and accurately detect the contact position of the player with the court surface using a trained model that has been machine-learned.

[0137] Furthermore, the information processing device 20 according to this embodiment may function as a learning device that generates a contact / ground departure determination model (an example of a trained model) for determining whether or not a player captured in an image capturing a player on the court surface (an example of a specific surface) is in contact with the court surface, by performing machine learning using a dataset including a plurality of learning images capturing a player (an example of an object) present on the court surface (an example of a specific surface) and information indicating whether or not the player captured in each of the plurality of learning images is in contact with the court surface.

[0138] As a result, the object position detection system SYS (information processing device 20) can accurately determine whether a player is in contact with the court surface by capturing an image of the player with a single monocular camera, using a trained model that has been machine-learned.

[0139] Although one embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.

[0140] In the above embodiment, an example of detecting the position of a player on a court where a badminton match is being played has been described, but the present invention can also be applied to detecting the position of an arbitrary object. For example, the present invention may be applied to a player playing a sport other than badminton, and may be applied not only to a match but also to practice. The present invention may also be applied to detect the position of an arbitrary person or object other than a person, not limited to a player playing a sport. The person or object other than a person may be present on a specific surface that is a target of ground contact when determining whether or not the object is on the ground. The specific surface is not limited to a court surface, but may be any surface such as the ground or floor that can specify a range for detecting the position of the object. The specific surface is not limited to a flat surface, but may be a surface including a curved surface or a surface including unevenness. The specific surface is not limited to a horizontal surface, but may be a surface that is inclined relative to the horizontal, or may be a surface that is close to vertical. An object present on a specific surface (for example, a person such as a player) refers to a person present on the surface side of the specific surface, and is not limited to a surface in the opposite direction to the vertical direction relative to the specific surface.

[0141] Furthermore, the camera 10 in the above embodiment is not limited to a dedicated camera such as a video camera or digital camera, but may be an electronic device having a camera function as part of its functions, such as a smartphone. The camera 10 and the information processing device 20 may be communicatively connected by wire or wirelessly, or, instead of being communicatively connected, an image captured by the camera 10 may be transferred to the information processing device 20 via a storage medium such as an optical disk or a memory card.

[0142] In the above embodiment, the camera 10 is not included in the information processing device 20 but is configured as a separate device. However, the camera 10 and the information processing device 20 may be configured as a single integrated device. For example, a smartphone or tablet computer having a camera function may include the camera 10 and the units included in the information processing device 20. The object position detection system SYS may be configured to include a plurality of devices in which the camera 10 and the information processing device 20 are integrated. The captured image processed by the information processing device 20 may be a video captured in the past by a camera of another system (an environment not connected to the information processing device 20) and recorded in the cloud or physical media, or a video provided by a video sharing service or broadcast.

[0143] In addition, the information processing device 20 in the above embodiment is described as being one device, but is not limited thereto, and may be composed of a plurality of devices. For example, in the above embodiment, a configuration example has been described in which the information processing device 20 has both a configuration as a learning device that generates a touchdown position detection model and a touchdown / takeoff judgment model, and a configuration as a position detection device that detects the position of a moving object using the touchdown position detection model and the touchdown / takeoff judgment model, but the configuration as the learning device and the configuration as the position detection device may be configured separately in different devices. For example, among the configurations of the information processing device 20, the learning dataset storage unit 216, the learning program storage unit 217, the trained model storage unit 218, and the trained model generation unit 228 may be included in a computer device as a learning device other than the information processing device 20.

[0144] The information processing device 20 may be configured so that a part or the whole of the storage unit 21 is provided in an external storage device. In this case, the external storage device may be a storage device connectable to the information processing device 20 via a network. The information processing device 20 may be configured so that a part of the functions of the control unit 22 is provided in an external device.

[0145] The camera 10 or the information processing device 20 included in the object position detection system SYS described above has a computer system therein. A program for implementing the functions of each component included in the camera 10 or the information processing device 20 described above may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to perform processing in each component included in the camera 10 or the information processing device 20 described above. Here, "reading a program recorded in a recording medium into a computer system and executing it" includes installing the program into a computer system. The "computer system" here includes hardware such as an OS and peripheral devices. The "computer system" may also include multiple computer devices connected via a network including a communication line such as the Internet, a WAN, a LAN, or a dedicated line. The "computer-readable recording medium" refers to a portable medium such as a flexible disk, a magneto-optical disk, a ROM, or a CD-ROM, or a storage device such as a hard disk built into a computer system. In this way, the recording medium storing the program may be a non-transient recording medium such as a CD-ROM.

[0146] The recording medium also includes a recording medium installed inside or outside and accessible from a distribution server to distribute the program. The program may be divided into a plurality of parts, downloaded at different times, and then combined by each component of the camera 10 or the information processing device 20, or each divided program may be distributed by a different distribution server. Furthermore, the "computer-readable recording medium" includes a recording medium that holds a program for a certain period of time, such as a volatile memory (RAM) in a computer system that becomes a server or a client when a program is transmitted via a network. The above program may also be a recording medium for realizing part of the above-mentioned functions. Furthermore, the above-mentioned functions may be realized in combination with a program already recorded in the computer system, that is, a so-called difference file (difference program).

[0147] Also, the camera 10 or the information processing device 20 may be realized in part or in whole as an integrated circuit such as LSI (Large Scale Integration). Also, each component in the camera 10 or the information processing device 20 of the present embodiment may be individually made into a processor, or may be integrated in part or in whole into a processor. Also, the method of integration is not limited to LSI, and may be realized by a dedicated circuit or a general-purpose processor. Also, when an integrated circuit technology that replaces LSI appears due to the advancement of semiconductor technology, an integrated circuit based on that technology may be used. [Explanation of symbols]

[0148] SYS... object position detection system, 10... camera, 20... information processing device, 21... memory unit, 22... control unit, 210... image information memory unit, 211... object coordinate memory unit, 212... ground contact position coordinate memory unit, 213... ground contact / take-off frame memory unit, 214... estimated position coordinate memory unit, 215... detection result memory unit, 216... learning data set memory unit, 217... learning program memory unit, 218... trained model memory unit, 220... image acquisition unit, 221... object detection unit, 222... ground contact position detection unit, 223... object ground contact determination unit, 224... object position estimation unit, 225... coordinate conversion unit, 226... position information generation unit, 227... position information output unit, 228... trained model generation unit

Claims

1. a touchdown position detection unit that detects a touchdown position where an object present on a specific surface is in contact with the specific surface in a captured image captured by a single monocular camera; an object grounding determination unit that determines whether the object captured in the captured image is grounded on the specific surface; an object position estimation unit that estimates a position of the object in the captured image based on the ground contact position detected by the ground contact position detection unit when the object is in contact with the specific surface; an object position detection unit that detects a position of the object on the specific plane by converting the position of the object in the captured image estimated by the object position estimation unit into a position on the specific plane; An object location detection system comprising:

2. The object position estimation unit is estimating a position of the object in the captured image when the object is not in contact with the specific surface based on the contact position detected when the object is in contact with the specific surface; The object location detection system according to claim 1 .

3. the ground contact position detection unit and the object ground contact determination unit each detect the ground contact position and determine whether or not the object is in contact with the specific surface at a synchronous predetermined time interval, The object position estimation unit is based on the ground contact position detected by the ground contact position detection unit at the time interval and the determination result determined by the object ground contact determination unit at the time interval, a position of the object in the captured image when the object is not in contact with the specific surface is estimated by interpolation based on the ground contact position detected when the object is in contact with the specific surface. The object location detection system according to claim 2 .

4. The object position estimation unit is tracking a position of the object in the captured image based on the ground contact position detected by the ground contact position detection unit at the time interval and the determination result determined by the object ground contact determination unit at the time interval; The object position detection unit a position of the object in the captured image estimated by the object position estimation unit is converted into a position on the specific plane, thereby tracking the position of the object on the specific plane; The object location detection system according to claim 3 .

5. The object position detection unit detecting a moving speed, a moving amount, or an acceleration of the object based on the tracked position of the object on the specific surface; The object location detection system according to claim 4 .

6. a position information output unit that outputs display information that visualizes position information indicating the position of the object on the specific surface detected by the object position detection unit; The object location detection system according to claim 1 .

7. The object position estimation unit is When a plurality of the ground contact positions are detected for one of the objects by the ground contact position detection unit, a predetermined position between the plurality of the ground contact positions is estimated as the position of the object. The object location detection system according to claim 1 .

8. a learning unit that performs machine learning using a dataset including a plurality of learning images in which an object existing on the specific surface is captured, and ground contact position information indicating a ground contact position of the object captured in each of the plurality of learning images and the specific surface, thereby generating a trained model for detecting the ground contact position of the object in the captured image at which the object is in contact with the specific surface; Equipped with The ground contact position detection unit is detecting the contact position where the object is in contact with the specific surface in the captured image by using the trained model generated by the training unit; The object location detection system according to claim 1 .

9. a learning unit that performs machine learning using a dataset including a plurality of learning images in which an object existing on the specific surface is captured, and information indicating whether the object captured in each of the plurality of learning images is in contact with the specific surface, thereby generating a trained model for determining whether the object captured in the captured image is in contact with the specific surface; Equipped with The object grounding determination unit Using the trained model generated by the training unit, it is determined whether or not the object captured in the captured image is in contact with the specific surface. The object location detection system according to claim 1 .

10. a touchdown position detection unit that detects a touchdown position where an object present on a specific surface is in contact with the specific surface in a captured image captured by a single monocular camera; an object grounding determination unit that determines whether the object captured in the captured image is grounded on the specific surface; an object position estimation unit that estimates a position of the object in the captured image based on the ground contact position detected by the ground contact position detection unit when the object is in contact with the specific surface; an object position detection unit that detects a position of the object on the specific plane by converting the position of the object in the captured image estimated by the object position estimation unit into a position on the specific plane; An object position detection device comprising:

11. 1. An object position detection method in an object position detection system, comprising: an object grounding determination step in which an object grounding determination unit determines whether an object present on a specific surface, which is captured in an image captured by a single monocular camera, is grounded on the specific surface; a grounding position detection step in which a grounding position detection unit detects a grounding position where the object is in contact with the specific surface in the captured image; an object position estimating step in which an object position estimating unit estimates a position of the object in the captured image based on the grounding position detected in the grounding position detecting step when the object is in contact with the specific surface; an object position detection step in which an object position detection unit detects a position of the object on the specific plane by converting the position of the object in the captured image estimated in the object position estimation step into a position on the specific plane; 1. An object position detection method comprising:

Images