Coaching system

Abstract

To present information useful for training from a video.SOLUTION: A coaching system includes: a camera for acquiring a captured video; and an information processing device including a video generation unit for acquiring a region including feet of a subject and a skateboard on which the subject rides from the captured video and generating a region video having the region as an angle of view, a position calculation unit for calculating an intermediate position of the skateboard on the basis of the skateboard in the region video, and an output control unit for generating an output video including the region video and a marker indicating the calculated intermediate position.SELECTED DRAWING: Figure 12

Description

[Technical Field]

[0001] The present invention relates to a coaching system. [Background technology]

[0002] In sports training, there is known a technology for capturing video of an athlete and checking the athlete's form, etc. from the captured video. For example, Patent Document 1 describes a technology for detecting human body movement during filming and automatically starting slow-motion video recording. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Special Publication No. 2022-545800 Summary of the Invention [Problem to be solved by the invention]

[0004] In skateboarding training, users need to interpret information from the video that will help them improve their skills (called tricks). It is desirable to be able to present useful information for training from the video.

[0005] An object of the present invention is to provide a coaching system that can present information useful for training from video. [Means for solving the problem]

[0006] The coaching system of the present invention comprises an information processing device having a camera that acquires a captured image, an image generation unit that acquires an area from the captured image that includes the subject's feet and the skateboard that the subject is riding and generates an area image with the area as its angle of view, a position calculation unit that calculates the intermediate position of the skateboard based on the skateboard in the area image, and an output control unit that generates an output image that includes the area image and a marker that indicates the calculated intermediate position.

[0007] The coaching system of the present invention comprises an information processing device having a camera that acquires captured footage, an image generation unit that acquires an area from the captured footage that includes the subject's feet and the skateboard that the subject is riding and generates an area image with the area as its angle of view, a rotation calculation unit that calculates the number of rotations of the skateboard based on changes in the skateboard in the area image, and an output control unit that generates an output image that includes the area image and the calculated number of rotations. [Effects of the Invention]

[0008] According to the present invention, it is possible to present information useful for training from video. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of a coaching system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing an example of the configuration of a camera according to the first embodiment. [Figure 3] FIG. 3 is a schematic diagram showing an example of a skateboard. [Figure 4] FIG. 4 is a block diagram showing an example of the configuration of the information processing device according to the first embodiment. [Figure 5] FIG. 5 is a schematic diagram showing a captured image and an area image. [Figure 6] FIG. 6 is a schematic diagram showing the intermediate position of the skateboard in the area image. [Figure 7]FIG. 7 is a schematic diagram showing a matching image. [Figure 8] FIG. 8 is a schematic diagram showing the rotation direction of a skateboard. [Figure 9] FIG. 9 is a diagram illustrating the change in the image when the skateboard rotates around its major axis. [Figure 10] FIG. 10 is a diagram for explaining the matching process between the area image and each frame data. [Figure 11] FIG. 11 is a diagram illustrating the change in the image when the skateboard rotates around the vertical axis. [Figure 12] FIG. 12 is a schematic diagram showing an example of an output image that shows the situation during skating before the start of a trick. [Figure 13] FIG. 13 is a schematic diagram showing an example of an output image that shows the situation when a trick is being performed. [Figure 14] FIG. 14 is a schematic diagram showing an example of a region image displayed in the output image. [Figure 15] FIG. 15 is a flowchart showing the flow of the output video generation process by the coaching system according to the first embodiment. [Figure 16] FIG. 16 is a diagram illustrating the process of calculating the number of rotations of a skateboard according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to this embodiment, and in the following embodiments, the same components are designated by the same reference numerals, and redundant description will be omitted.

[0011] [First embodiment] (Coaching System) The coaching system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of a coaching system 1 according to the first embodiment.

[0012] 1, the coaching system 1 includes a camera 2 that captures a captured image 10, and an information processing device 3. The camera 2 and the information processing device 3 are connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0013] The coaching system 1 is a system used for coaching in skateboarding competitions. The coaching system 1 is installed in facilities for skateboarding practice and competitions. The coaching system 1 takes video of the skateboarder practicing or competing, with the skateboarder as the subject SB. The skater performs various tricks while gliding on a skateboard 4. The coaching system 1 analyzes the video using an information processing device 3 and presents various information to improve trick techniques.

[0014] The coaching system 1 is provided with one or more cameras 2. FIG. 1 shows an example in which the coaching system 1 is provided with one camera 2. The camera 2 continuously captures images of the subject SB while he or she is skating. The camera 2 acquires a video 10 that captures a series of movements from before the start of a trick to the end. The camera 2 has a function to automatically track and capture images of the subject SB, for example.

[0015] When the coaching system 1 includes multiple cameras 2, the cameras 2 are arranged, for example, along the area where the subject SB skates. In this case, the cameras 2 may not have the function of tracking the subject SB, and may have a fixed field of view. The fields of view of the cameras 2 may partially overlap each other.

[0016] In this embodiment, the coaching system 1 acquires and processes video footage of a subject SB skating on a skateboard 4, captured from the side. At least one camera 2 captures the image of the subject SB from a direction intersecting the direction in which the subject SB is skating.

[0017] (camera) FIG. 2 is a block diagram showing an example of the configuration of the camera 2 according to the first embodiment.

[0018] Camera 2 includes an optical element 20, an imaging element 21, a drive unit 22, a communication unit 23, a storage unit 24, and a control unit 25. Camera 2 is a PTZ (Pan-Tilt-Zoom) camera that can be controlled in the pan, tilt, and zoom directions. Camera 2 may also be a wide-angle camera.

[0019] The optical element 20 is, for example, an element of an optical system such as a lens. There may be one or more optical elements 20. The optical element 20 includes a zoom mechanism that changes the imaging magnification.

[0020] The imaging element 21 is an element that converts light incident through the optical element 20 into an image signal, which is an electrical signal. The imaging element 21 is, for example, a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor.

[0021] The drive unit 22 drives the imaging system including the optical element 20 and the image sensor 21. The drive unit 22 includes, for example, a first motor for moving the imaging system in the pan direction and a second motor for moving the imaging system in the tilt direction.

[0022] The communication unit 23 executes communication between the camera 2 and an external device. For example, the communication unit 23 executes communication between the camera 2 and the information processing device 3. The communication unit 23 is realized by a communication module that executes communication using a method such as a wired LAN (Local Area Network) or a wireless LAN.

[0023] The storage unit 24 stores various types of information. The storage unit 24 stores information such as the contents of calculations performed by the control unit 25 and programs. The storage unit 24 includes at least one of a main storage device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), and an external storage device such as an HDD (Hard Disk Drive).

[0024] The control unit 25 controls each unit of the camera 2. The control unit 25 has, for example, an information processing device such as a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or a GPU (Graphics Processing Unit), and a storage device such as a RAM or a ROM. The control unit 25 may be realized by, for example, an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 25 may also be realized by a combination of hardware and software.

[0025] The control unit 25 includes an imaging control unit 25A, an image processing unit 25B, an image conversion unit 25C, and an object detection unit 25D.

[0026] The imaging control unit 25A controls imaging by the camera 2. The imaging control unit 25A controls, for example, the zoom mechanism of the optical element 20 and the first motor and second motor of the drive unit 22 to perform PTZ control of the camera 2. The imaging control unit 25A controls, for example, the imaging element 21 to acquire an image signal. The imaging control unit 25A acquires, for example, from the imaging element 21, the image signal generated by the imaging element 21.

[0027] The image processing unit 25B generates image data for each frame from the image signal acquired by the imaging control unit 25A. Specifically, the image processing unit 25B performs processes such as defect correction for the image sensor 21, noise reduction (NR), debayer processing, gamma processing, color reproduction, resizing, and video signal timing conversion on the image signal acquired by the imaging control unit 25A to generate image data.

[0028] The image conversion unit 25C encodes the image data generated by the image processing unit 25B into image data in a file format encoded by any method, such as H.264, H.265, JPEG (Joint Photographic Experts Group), etc. The image conversion unit 25C stores the encoded image data in the storage unit 24.

[0029] The object detection unit 25D detects the subject SB included in the image data generated by the image processing unit 25B. In this embodiment, the object detection unit 25D includes a skeletal information generation unit 26 and a feature detection unit 27.

[0030] The skeletal information generation unit 26 detects the subject SB by performing a skeletal detection process on the image data generated by the image processing unit 25B. The skeletal information generation unit 26 generates skeletal position information 51 (see FIG. 5) of the subject SB from the captured video 10. The skeletal position information 51 includes position coordinates of each point (skeletal point) of a human skeletal model in the image. The skeletal position information 51 includes, for example, position coordinates of each point of the subject SB's head, shoulders, elbows, hands (wrists), waist, knees, and feet (ankles).

[0031] The feature detection unit 27 detects the skateboard 4 on which the subject SB is riding from the captured video 10. The feature detection unit 27 detects each part of the skateboard 4. For example, the feature detection unit 27 detects the wheels 41, the deck 42, and the trucks 43 of the skateboard 4.

[0032] FIG. 3 is a schematic diagram showing an example of a skateboard 4. FIG. 3(A) is a side view of the skateboard 4, and FIG. 3(B) is a bottom view of the skateboard 4. The skateboard 4 includes wheels 41, a deck 42, trucks 43, and bearings 44. The deck 42 is a long, thin, plate-like member on which a skater rides. Trucks 43 are fixed to the underside of the deck 42. One truck 43 is provided at one end of the deck 42 and one at the other end. The trucks 43 are inverted T-shaped holding members that hold the wheels 41. The trucks 43 extend to the left and right in the width direction of the deck 42 on the underside of the deck 42. The wheels 41 are wheels of the skateboard 4. One wheel 41 is rotatably attached to each end of the truck 43 in the left-right direction via bearings 44. A total of four wheels 41 are provided, two on each of the front and rear trucks 43.

[0033] The feature detection unit 27 performs feature detection processing on the image data to detect wheels 41, deck 42, and trucks 43 located near the feet (feet LG) of the subject SB obtained from the skeletal position information 51. The feature detection unit 27 detects wheels 41 from a roughly circular image portion located near the feet LG in the photographed video 10, and acquires the position coordinates of each wheel 41 appearing in the photographed video 10. The feature detection unit 27 detects deck 42 from a long, thin, linear image portion located near the feet LG in the photographed video 10, and acquires the position coordinates of the deck 42 appearing in the photographed video 10. The feature detection unit 27 detects trucks 43 from an inverted-T-shaped image portion located near the feet LG in the image data, and acquires the position coordinates of the trucks 43 appearing in the photographed video 10. Note that the trucks 43 are barely visible in the image data during normal skating, when the side of the skateboard 4 is visible. When one end or the other end of the skateboard 4 faces the camera 2 during a trick, which will be described later, the truck 43 appears as an inverted T-shape in the captured image 10 and is detected by the feature detection unit 27.

[0034] The object detection unit 25D transmits information about the subject SB detected from the image converted by the image conversion unit 25C to the information processing device 3 via the communication unit 23. The object detection unit 25D transmits the detected skeletal position information of the subject SB and the position information of the skateboard 4 (position information of the wheels 41 and the deck 42) to the information processing device 3 via the communication unit 23.

[0035] Furthermore, the control unit 25 reads out, for example, image data encoded by the image conversion unit 25C from the storage unit 24, and transmits it as the captured video 10 via the communication unit 23 to the information processing device 3 or an external server device.

[0036] (Information processing device) FIG. 4 is a block diagram showing an example of the configuration of the information processing device 3 according to the first embodiment.

[0037] The information processing device 3 includes an input unit 30, a display unit 31, a communication unit 32, a storage unit 33, and a control unit 34. The information processing device 3 is realized, for example, by a general-purpose PC (Personal Computer) or a server device.

[0038] The input unit 30 accepts various input operations to the information processing device 3. The input unit 30 includes, for example, a keyboard, a mouse, a touch panel, buttons, and switches.

[0039] The display unit 31 displays various images. The display unit 31 is, for example, a display including a liquid crystal display (LCD) or an organic electroluminescence (EL) display.

[0040] The communication unit 32 executes communication between the information processing device 3 and an external device. For example, the communication unit 32 executes communication between the information processing device 3 and the camera 2. The communication unit 32 is realized by a communication module that executes communication using a method such as a wired LAN or a wireless LAN.

[0041] The storage unit 33 stores various types of information, such as the captured video 10 received from the camera 2, skeletal position information 51 of the subject SB, position information of the skateboard 4, the calculation contents of the control unit 34, the output video 13, and programs. The storage unit 33 includes at least one of a main storage device such as a RAM or a ROM, and an external storage device such as an HDD.

[0042] The control unit 34 controls each unit of the information processing device 3. The control unit 34 includes, for example, an information processing device such as a CPU, MPU, or GPU, and a storage device such as a RAM or ROM. The control unit 34 may be realized by, for example, an integrated circuit such as an ASIC or FPGA. The control unit 34 may also be realized by a combination of hardware and software.

[0043] The control unit 34 includes a video generation unit 34A, a position calculation unit 34B, a matching image generation unit 34C, a rotation calculation unit 34D, a speed calculation unit 34E, and an output control unit 34F.

[0044] The image generating unit 34A acquires an area including the feet (feet LG) of the subject SB and the skateboard 4 on which the subject SB is riding from the captured image 10, and generates an area image 11 having the area as the angle of view.

[0045] FIG. 5 is a schematic diagram showing a captured image 10 and an area image 11. The captured image 10 of the subject SB performing a trick includes multiple frame images F that capture the movements of the subject SB from before the start of the trick to after the trick is completed. FIG. 5 shows a representative portion of the multiple frame images F, extracted frame by frame. Specifically, the multiple frame images F include a frame image F1 of the subject SB skating before the start of the trick, a frame image F2 of the subject SB jumping and ascending immediately after the start of the trick, a frame image F3 of the subject SB near the highest point during the jump, a frame image F4 of the subject SB descending during the jump, and a frame image F5 of the subject SB landing after the trick is completed. The position and posture of the subject SB in each frame image F are obtained from skeletal position information 51.

[0046] The video generation unit 34A acquires the area of ​​the feet of subject SB in the frame image F based on the skeleton position information 51. The area of ​​the feet of subject SB is acquired from the position information of the feet LG of subject SB in the skeleton position information 51. The video generation unit 34A acquires the area of ​​the skateboard 4 in the frame image F based on the position information of the wheels 41 and the deck 42 of the skateboard 4. The video generation unit 34A cuts out a continuous area that includes both the area of ​​the feet LG of subject SB and the entire area of ​​the skateboard 4 from the frame image F by cropping. The video generation unit 34A records the cut-out image portion in the storage unit 33 as frame data FR of the area video 11 in that frame image F. As a result, the video generation unit 34A generates, from each frame image F, an area video 11 whose angle of view is an area that includes the feet of subject SB and the skateboard 4 on which subject SB is riding. Therefore, the area video 11 is a moving image including a collection of frame data FR obtained by cropping the area around the feet of the subject SB and the skateboard 4 from the photographed video 10. The video generation unit 34A associates each frame data FR of the area video 11 with each frame image F of the photographed video 10. The video generation unit 34A records the generated area video 11 in the storage unit 33.

[0047] The position calculation unit 34B calculates the middle position CP of the skateboard 4 based on the skateboard 4 in the area image 11.

[0048] FIG. 6 is a schematic diagram showing the middle position CP of the skateboard 4 in the area image 11. The skateboard 4 and the feet LG of the subject SB are captured in the area image 11. The position calculation unit 34B acquires the position coordinates of each wheel 41 in each frame data FR of the area image 11 based on the detection results of the object detection unit 25D. The position calculation unit 34B acquires the middle position between the front and rear wheels 41 as the middle position CP of the skateboard 4. For example, the middle position CP is the midpoint of a line segment connecting the wheel 41 on one end side of the skateboard 4 to the wheel 41 on the other end side. The position calculation unit 34B calculates information about the middle position CP for each frame data FR of the area image 11. The position calculation unit 34B associates the calculated information about the middle position CP (position coordinates) with the frame data FR of the area image 11 and stores it in the storage unit 33.

[0049] The matching image generation unit 34C generates a matching image for calculation processing by the rotation calculation unit 34D. The matching image is a template image used in the matching processing for detecting the attitude of the skateboard 4.

[0050] 7 is a schematic diagram showing a matching image. The matching image generation unit 34C generates a matching image 12A from frame data FR of the area image 11. The matching image generation unit 34C acquires, from the area image 11, frame data FR corresponding to a frame image F1 during skating before the start of a trick, as the matching image 12A. In the matching image 12A, the feet LG of the subject SB are in contact with the upper surface of the deck 42, and the wheels 41 are in contact with the floor surface B.

[0051] The matching image generation unit 34C also generates a matching image 12B by flipping the skateboard 4 upside down from the matching image 12A. The matching image generation unit 34C generates the matching image 12B as an image that includes only the area where the skateboard 4 appears, and does not include image portions of the feet LG of the subject SB or the floor B. The matching image generation unit 34C removes the image portions of the feet LG of the subject SB and the floor B, for example, using a trained model that has previously been machine-learned to remove image elements other than the object from the input image and output an image of the object. The matching image generation unit 34C may also extract the area where the skateboard 4 appears using a known image processing algorithm (such as edge detection) other than the trained model. The matching image generation unit 34C generates the matching image 12B as an inverted image by flipping the image portion of the skateboard 4 upside down. The matching image 12B corresponds to a scene of the skateboard 4 spinning when a trick is performed on the skateboard 4, such as frame image F3 in FIG. 5.

[0052] The matching image generating unit 34C records the generated matching image 12A and matching image 12B in the storage unit 33 in association with the area video 11.

[0053] The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 based on the change in the skateboard 4 in the area image 11.

[0054] FIG. 8 is a schematic diagram showing the rotation direction of the skateboard 4. There are various tricks for rotating the skateboard 4, but here, we will explain a case where the skateboard 4 is rotated around the long axis A1 of the skateboard 4 and a case where the skateboard 4 is rotated around the up-down axis A2 of the skateboard 4. The long axis A1 of the skateboard 4 is an axis along the long side of the skateboard 4, and is an axis that extends in the direction of travel of the skateboard 4 while gliding. The up-down axis A2 of the skateboard 4 is an axis that extends in the thickness direction of the deck 42.

[0055] First, rotation around the major axis A1 will be described. The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 around the major axis A1 based on the number of times the skateboard 4 is turned upside down in the area image 11.

[0056] FIG. 9 is a diagram illustrating changes in an image when the skateboard 4 rotates around the long axis A1. For convenience, FIG. 9 shows only the skateboard 4, omitting the subject SB. In a rotation trick around the long axis A1, the skateboard 4 rolls in the air, causing the skateboard 4 to flip upside down, and then the skateboard 4 returns to its original position and lands. In FIG. 9, frame image F11 of the captured video 10 is an image taken while the skateboard 4 is skating before the trick begins. Frame image F12 is an image taken after the trick begins, with the skateboard 4 having rotated 90 degrees in the air. Frame image F13 is an image taken after the skateboard 4 has rotated 180 degrees in the air. Frame image F14 is an image taken after the skateboard 4 has rotated 270 degrees in the air. Frame image F15 is an image taken after the skateboard 4 has rotated 360 degrees and landed.

[0057] In the first embodiment, attention is focused on a state where the skateboard 4 has made 0 or 1 rotation (360-degree rotation) and a state where the skateboard 4 has made a 1 / 2 rotation (180-degree rotation). Specifically, the rotation calculation unit 34D performs a matching process between the matching image 12A and the matching image 12B generated by the matching image generation unit 34C and each frame data FR of the area video 11.

[0058] FIG. 10 is a diagram illustrating the matching process with each frame data FR of the area image 11. FIG. 10 shows each frame data FR corresponding to the frame image F11, frame image F13, and frame image F15 shown in FIG. 9. First, the matching image generation unit 34C generates a matching image 12A and a matching image 12B, which is an inverted image, from the frame data FR corresponding to the frame image F11 during skating before the start of a trick. The rotation calculation unit 34D acquires the matching image 12A and the matching image 12B, and performs a matching process with each of the subsequent frame data FR of the area image 11, with the matching image 12A and the matching image 12B, respectively. After the start of a trick, the skateboard 4 in frame image F13, which shows the skateboard 4 rotated 180 degrees in the air, is shown upside down compared to the skateboard 4 in frame image F11. Since the skateboard 4 during the trick (in the air) is away from the feet of the subject SB and the floor B, the skateboard 4 in frame image F13 matches the matching image 12B from which the image portions of the feet LG and the floor B have been removed. The rotation calculation unit 34D counts the number of matches with the matching image 12B. The skateboard 4 in frame image F15, after the trick has finished and the skateboard 4 has landed on the floor B, appears in the same state as the skateboard 4 in frame image F11 and therefore matches the matching image 12A.

[0059] The number of matches with the matching image 12B indicates the number of times the skateboard 4 flips (1 / 2 rotation). The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 based on the number of times the skateboard 4 shown in the area image 11 matches the matching image 12B from the start of the matching process until it lands. The rotation calculation unit 34D detects landing based on the fact that the frame data FR of the area image 11 matches the matching image 12A. The matching image 12A is an image in which the feet LG of the subject SB are in contact with the upper surface of the deck 42 and the wheels 41 are in contact with the floor surface B, so it is matched when the skateboard 4 lands. In the case of FIG. 10, the skateboard 4 in the area image 11 matches the matching image 12B once, and then matches the matching image 12A, so the number of rotations is "1." The rotation calculation unit 34D records the calculated number of rotations information in the storage unit 33 in association with the area image 11.

[0060] The matching method used by the rotation calculation unit 34D is not particularly limited. In one example, the matching process performed by the rotation calculation unit 34D is a contour search. In the contour search, the rotation calculation unit 34D extracts contour features from the matching images 12A and 12B and acquires a feature vector for each matching image. Then, in the matching process, the rotation calculation unit 34D similarly extracts contour features from each frame data FR of the area image 11 and acquires a feature vector. While performing an affine transformation, the rotation calculation unit 34D counts the number of matching points between the feature vectors of each matching image and the feature vectors of the frame data FR and calculates a matching score based on the number of matching points. If the calculated matching score is equal to or greater than a threshold, the rotation calculation unit 34D detects that the frame data FR matches one of the matching images. The contour search matches vectors representing the contours of image elements, enabling a matching process that is robust against deviations in the rotation angle due to the affine transformation.

[0061] Next, the rotation around the vertical axis A2 will be described. The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 around the vertical axis A2 based on the change in the wheel spacing W of the skateboard 4 in the area image 11.

[0062] FIG. 11 is a diagram illustrating changes in an image when the skateboard 4 rotates around the vertical axis A2. For convenience, FIG. 11 shows only the skateboard 4, omitting the subject SB. In a rotation trick around the vertical axis A2, the skateboard 4 yaws in the air, causing the skateboard 4 to rotate horizontally and then land in the same orientation as its original traveling direction. In FIG. 11, frame image F21 of the captured video 10 is an image taken while the skateboard 4 is gliding before the start of the trick. Frame image F22 is an image taken after the start of the trick, in which the skateboard 4 has rotated 90 degrees (a quarter rotation) in the air. Frame image F23 is an image taken after the skateboard 4 has rotated 180 degrees (a half rotation) in the air and landed on floor B. The example shown in FIG. 11 is a trick in which the skateboard 4 rotates a half rotation, and the positional relationship between one end (nose) and the other end (tail) of the skateboard 4 is reversed between frame image F21 and frame image F23.

[0063] In the first embodiment, attention is paid to the spacing W1 between the wheels 41 when the skateboard 4 is aligned with the traveling direction (zero rotation or half rotation), and the spacing W2 between the wheels 41 when the skateboard 4 is oriented in a direction perpendicular to the traveling direction (quarter rotation). Specifically, the rotation calculation unit 34D calculates the number of rotations of the skateboard 4 around the vertical axis A2 based on the fact that the wheel spacing W in each frame data FR of the area image 11 changes between the spacing W1 and the spacing W2.

[0064] Specifically, the rotation calculation unit 34D acquires the position coordinates of the multiple wheels 41 from the detection results of the feature detection unit 27. The rotation calculation unit 34D acquires the wheel spacing W, which is the distance between two wheels 41. In the frame image F21 before the start of the trick and the frame image F23 after landing, the side of the skateboard 4 is captured in the area image 11. The wheel spacing W acquired in the frame image F21 and the frame image F23 is the spacing W1 between the front and rear wheels 41.

[0065] In the frame image F22 at the time of a quarter rotation during a trick, one end or the other end of the skateboard 4 is captured in the area image 11. The wheel spacing W acquired in the frame image F22 is the spacing W2 between the left and right wheels 41 in the width direction. The spacing W2 is the spacing between a pair of wheels 41 held on the same truck 43, and can be considered to be equal to the width of the truck 43. There is a clear difference between the spacing W1 and the spacing W2 due to the structure of the skateboard 4. Therefore, the spacing W1 and the spacing W2 can be distinguished by threshold processing or the like.

[0066] The rotation calculation unit 34D detects the rotation of the skateboard 4 around the vertical axis A2 based on the change in the wheel spacing W from W1 to W2. The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 based on the number of times the wheel spacing W changes between W1 and W2 before the skateboard 4 lands. The rotation calculation unit 34D detects landing based on the frame data FR matching the matching image 12A. The rotation calculation unit 34D may also detect landing based on the length of time (number of frames) during which the wheel spacing W matches the spacing W1 exceeding a predetermined value. In the case of FIG. 11 , the wheel spacing W of the skateboard 4 in the area image 11 changes from W1 to W2, and then changes from W2 to W1 before landing, so the number of rotations is “1 / 2.” The rotation calculation unit 34D records the calculated number of rotations information in the storage unit 33 in association with the area image 11. When the number of rotations becomes "1", a time series change in the wheel spacing W, i.e., spacing W1, spacing W2, spacing W1, is detected once more before landing.

[0067] Note that, whether the trick is a spinning trick about the major axis A1 or a spinning trick about the up-down axis A2, if the trick fails, it is highly likely that the subject SB will fall or the skateboard 4 will separate from the subject SB, making it impossible to detect landing. If no landing is detected within a predetermined upper limit time from the generation of the matching image 12A, the rotation calculation unit 34D records the number of rotations detected within the predetermined upper limit time from the generation of the matching image 12A as the number of rotations in the captured video 10 in the storage unit 33. The upper limit time is not particularly limited, but may be, for example, 3 seconds or 4 seconds, and may be set via the input unit 30. If no landing is detected, the rotation calculation unit 34D may record information indicating that the number of rotations cannot be determined in the storage unit 33.

[0068] The speed calculation unit 34E calculates the moving speed of the skateboard 4 in the traveling direction based on the skateboard 4 in the photographed video 10 (area video 11). The speed calculation unit 34E acquires the jump time of the subject SB in the photographed video 10. In the case of a jump around the major axis A1 in FIG. 10, the speed calculation unit 34E acquires the number of frames from the frame image that matches the matching image 12B in the air to the frame image that matches the matching image 12A at the time of landing. The speed calculation unit 34E considers the number of frames (T) from matching the matching image 12B to landing to be twice (i.e., 2T) as the total number of frames during the jump. The speed calculation unit 34E calculates the jump time based on the total number of frames during the jump and the frame rate of the photographed video 10. Similarly, in the case of a jump around the vertical axis A2 in Figure 11, the speed calculation unit 34E considers the total number of frames during the jump to be twice the number of frames from when the wheel spacing W matches spacing W2 until the wheel lands, and calculates the jump time based on the total number of frames during the jump and the frame rate of the captured video 10.

[0069] The speed calculation unit 34E calculates the distance traveled in the forward direction of the skateboard 4 during a jump based on environmental information about the shooting location and the amount of movement of the skateboard 4 in the photographed video 10. The environmental information is information that associates coordinates (pixels) in the photographed video 10 with the actual distance (meters) at the shooting location. The environmental information is acquired in advance before using the coaching system 1 and recorded in the memory unit 33. For example, lines indicating unit lengths may be added at equal intervals to the floor B of the track along which the subject SB will perform tricks, so that the lines appear in the photographed video 10 along with the subject SB and the skateboard 4. The speed calculation unit 34E can obtain the distance per pixel based on the spacing (number of pixels) between the lines in the photographed video 10 and the actual spacing (meters) between the lines recorded in the memory unit 33. The speed calculation unit 34E calculates the actual distance traveled by the skateboard 4 from the distance (number of pixels) traveled by the skateboard 4 during the jump time in the photographed video 10. The speed calculation unit 34E calculates the moving speed of the skateboard 4 from the actual moving distance of the skateboard 4 and the jump time.

[0070] The environmental information may be information about the total length of the skateboard 4 (the length of the deck 42), for example. In this case, the speed calculation unit 34E can calculate the actual travel distance by associating the total length of the skateboard 4 with coordinates (pixels) in the photographed video 10 and turning off the tracking function of the camera 2. Specifically, if the number of horizontal pixels in the photographed video 10 is X and the number of pixels in the photographed video 10 relative to the total length Y of the skateboard 4 is Z, the actual length D, which corresponds to the width of the entire photographed angle of view, is expressed as D = X / Z × Y. For example, if the photographed angle of view is 1920 × 1080 pixels, the total length Y of the skateboard 4 is 80 cm, and the number of pixels Z in the photographed video 10 relative to the total length Y of the skateboard 4 is 120 pixels, the actual length D can be calculated as 1920 / 120 × 0.8 = 16 (m). Therefore, the actual travel distance can be calculated based on the actual length D and the coordinate position of the skateboard 4 in the photographed video 10.

[0071] The speed calculation unit 34E associates the calculated moving speed with the captured image 10 (area image 11) and records it in the storage unit 33. If the speed calculation unit 34E cannot detect landing, it records in the storage unit 33 information indicating that the jump time or moving speed cannot be measured.

[0072] The output control unit 34F generates an output video 13 that the coaching system 1 presents to the user. In the first embodiment, the output control unit 34F generates the output video 13 including the area video 11 and a marker 14 indicating the calculated intermediate position CP. The output control unit 34F also generates the output video 13 including the area video 11 and the calculated number of rotations. The output video 13 is a moving image created based on the filmed video 10 from before the start to the end of the trick.

[0073] Fig. 12 is a schematic diagram showing an example of output video 13 representing the situation while skating before the start of a trick. Fig. 13 is a schematic diagram showing an example of output video 13 representing the situation while a trick is being performed. Output control unit 34F generates output video 13 including captured video 10 and area video 11 taken at the same time.

[0074] In the examples of FIGS. 12 and 13, the output control unit 34F generates the output image 13 including the regional image 11, the marker 14 indicating the calculated intermediate position CP, and the skeleton position information 51.

[0075] That is, the output control unit 34F superimposes a marker 14 representing the intermediate position CP on the captured image 10 and the area image 11 based on the intermediate position CP calculated by the position calculation unit 34B. The display form of the marker 14 is not particularly limited, but in the examples of FIGS. 12 and 13, the marker 14 is a dotted line extending vertically. The marker 14 extends vertically from the intermediate position CP in the captured image 10 and the area image 11 to the upper and lower ends of the angle of view. The marker 14 provides a clue to the position of the center of the skateboard 4.

[0076] The output control unit 34F superimposes and displays the skeleton position information 51 generated by the skeleton information generation unit 26 on the captured image 10 and the area image 11. While the display mode of the skeleton position information 51 is not particularly limited, in the examples of FIGS. 12 and 13, the output control unit 34F superimposes circular dots 15 indicating the skeleton positions as the skeleton position information 51. The dots 15 may be connected by lines indicating the connection relationships between the skeleton positions. It is preferable that the output control unit 34F superimposes multiple skeleton positions, including a skeletal position Q1 of the subject SB's feet (feet LG), a skeletal position Q2 of the subject SB's waist, and a skeletal position Q3 of the subject SB's head, on the image portion of the subject SB. The skeletal position Q1 of the feet is the position of the part of the skateboard 4 that is used to operate the skateboard 4, and is therefore important for improving tricks. The skeletal position Q2 of the waist and the skeletal position Q3 of the head serve as indicators for understanding the position of the center of gravity and posture of the subject SB during a trick.

[0077] The output control unit 34F displays a rotation number display area 16, which indicates the rotation number of the skateboard 4 calculated by the rotation calculation unit 34D, together with the photographed image 10 and the area image 11. The rotation number display area 16 includes numerical information on the rotation number. The rotation number display area 16 may be superimposed on at least one of the photographed image 10 and the area image 11, or may be displayed outside the photographed image 10 and the area image 11 so as not to overlap.

[0078] The output control unit 34F displays a speed display area 17 indicating the speed of the skateboard 4 calculated by the speed calculation unit 34E, together with the photographed video 10 and the area video 11. In the example of FIGS. 11 and 12, the output control unit 34F displays a jump time display area 18 indicating the jump time calculated by the speed calculation unit 34E, together with the photographed video 10 and the area video 11. The speed display area 17 includes numerical information on the moving speed of the skateboard 4. The jump time display area 18 includes numerical information on the jump time. The speed display area 17 and the jump time display area 18 may be superimposed on at least one of the photographed video 10 and the area video 11, or may be displayed outside the photographed video 10 and the area video 11 so as not to overlap.

[0079] In the examples of Figures 12 and 13, the output control unit 34F generates output frame images in chronological order, including frame images F of the captured video 10 on which markers 14 and dots 15 are superimposed, frame data FR of the area video 11, and images of the rotation speed display area 16, speed display area 17, and jump time display area 18, and generates an output video 13 from the series of output frame images.

[0080] The output control unit 34F displays the generated output video 13 on the display unit 31. The output control unit 34F may transmit the generated output video 13 to a pre-specified destination via the communication unit 32. The destination may be, for example, an information processing terminal owned by the user or a cloud server.

[0081] The output control unit 34F may generate an output video 13 that is a still image of a specific scene (specific frame image F) in the photographed video 10. In this case, the information processing device 3 receives, via the input unit 30, a designation of a scene from the photographed video 10 for which the output video 13 is to be generated, and generates the output video 13 for the received scene.

[0082] FIG. 14 is a schematic diagram showing an example of the area image 11 displayed in the output image 13. FIG. 14(A) shows an example of the output image 13 when a trick is successful, and FIG. 14(B) shows an example of the output image 13 when a trick is unsuccessful. In FIG. 14(A), it can be seen that the skeletal position Q1 of the feet is located at approximately equal distances on both the left and right sides of the marker 14 indicating the center position CP of the skateboard 4. In FIG. 14(B), it can be seen that the skeletal position Q1 of the feet is biased to the left side of the image relative to the marker 14 indicating the center position CP of the skateboard 4. By superimposing the marker 14 and the dot 15, the user can easily understand the difference in the position of the feet relative to the skateboard 4 between when a trick is successful and when it is unsuccessful. Furthermore, using the marker 14 as a clue, the user can easily understand whether, for example, the skeletal position Q2 of the waist and the skeletal position Q3 of the head in FIGS. 12 and 13 are appropriate.

[0083] (Coaching system processing) FIG. 15 is a flowchart showing the flow of the process of generating the output video 13 by the coaching system 1 according to the first embodiment.

[0084] The camera 2 acquires the photographed video 10 (step S10). The subject SB attempts a trick on the skateboard 4. The photographed video 10 is generated as a moving image that includes everything from the time before the start of the trick to the time when the trick ends (when the subject lands). The skeleton information generation unit 26 performs skeleton position detection for each frame image F of the photographed video 10 and generates skeleton position information 51 (step S11). The feature detection unit 27 performs feature detection for each frame image F of the photographed video 10 and detects each part of the skateboard 4 (step S12). The feature detection unit 27 generates position coordinates of the wheels 41, deck 42, and trucks 43 that appear in the photographed video 10. The camera 2 transmits the photographed video 10, the skeleton position information 51, and the position information of each part of the skateboard 4 to the information processing device 3 via the communication unit 23.

[0085] The information processing device 3 receives information from the camera 2 via the communication unit 32. The image generation unit 34A generates an area image 11 from the captured image 10 based on the skeleton position information 51 and the position information of each part of the skateboard 4 (step S13). The position calculation unit 34B calculates the middle position CP of the skateboard 4 based on the position information of each part of the skateboard 4 in the area image 11 (step S14). The matching image generation unit 34C generates a matching image 12A and a matching image 12B from the area image 11 (step S15). The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 based on changes in the skateboard 4 in the area image 11 (step S16). The rotation calculation unit 34D obtains the number of times the skateboard 4 has been turned upside down by matching the matching image 12A and the matching image 12B with the area image 11, and calculates the number of rotations of the skateboard 4 about the major axis A1 from the number of times the skateboard 4 has been turned upside down. The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 around the vertical axis A2 based on the change in the wheel spacing W between the spacing W1 and the spacing W2.

[0086] The speed calculation unit 34E calculates the jump time based on the total number of frames during the jump and the frame rate of the captured video 10, and calculates the movement speed of the skateboard 4 from the jump time and the movement distance of the skateboard 4 (step S17).

[0087] The output control unit 34F generates an output video 13 including the captured video 10 and area video 11 on which the marker 14 indicating the intermediate position CP and the dot 15 indicating the skeleton position information 51 are superimposed, as well as the rotation speed display area 16, the speed display area 17, and the jump time display area 18 (step S18). The output control unit 34F outputs the generated output video 13 (step S19). For example, the output control unit 34F displays the output video 13 on the display unit 31 or transmits it to a pre-specified destination via the communication unit 32. This completes the process of generating the output video 13 of the coaching system 1.

[0088] As described above, the first embodiment generates the output image 13 including the area image 11 and the marker 14 indicating the intermediate position CP of the skateboard 4. This allows the user to easily understand from the output image 13 the positional relationship between the skateboard 4 and the posture and center of gravity position of the user while attempting a trick, and therefore provides information useful for training from the image.

[0089] Furthermore, the first embodiment generates an output image 13 including the area image 11, a marker 14 indicating the calculated intermediate position CP, and skeleton position information 51. This allows the user to easily grasp the position of the center of gravity and posture of the body relative to the center of the skateboard 4 by comparing the marker 14 with the skeleton position information 51, thereby providing even more useful information.

[0090] Furthermore, the first embodiment generates an output image 13 including the area image 11 and the calculated number of rotations. This allows the user to understand the number of rotations of the skateboard 4 from the output image 13 while watching the video of the trick being attempted without having to check it himself, so that information useful for training can be presented from the video.

[0091] In addition, in the first embodiment, the number of rotations of the skateboard 4 about the major axis A1 is calculated based on the number of times the skateboard 4 is turned upside down in the area image 11, so it is possible to present to the user the number of rotations of the skateboard 4 about the major axis A1. In addition, in the first embodiment, the number of rotations of the skateboard 4 about the up-down axis A2 is calculated based on a change in the wheel spacing W of the skateboard 4 in the area image 11, so it is possible to present to the user the number of rotations of the skateboard 4 about the up-down axis A2.

[0092] [Second embodiment] A second embodiment of the present invention will be described. In the first embodiment, an example was described in which the number of rotations around the major axis A1 of the skateboard 4 is calculated using a matching image 12A and a matching image 12B obtained by flipping the matching image 12A upside down. In the second embodiment, an example is described in which the number of rotations around the major axis A1 of the skateboard 4 is calculated without using the matching image 12B. In the second embodiment, the process is the same as in the first embodiment except for the process of calculating the number of rotations of the skateboard 4 in the rotation calculation unit 34D, and therefore description thereof will be omitted.

[0093] FIG. 16 is a diagram illustrating the rotation speed calculation process for the skateboard 4 according to the second embodiment.

[0094] In the second embodiment as well, the rotation calculation unit 34D calculates the number of rotations of the skateboard 4 around the long axis A1 based on the number of times the skateboard 4 is turned upside down in the area image 11. However, in the second embodiment, the rotation calculation unit 34D counts the number of times the skateboard 4 is turned upside down based on changes in the positional relationship between the various parts of the skateboard 4 detected by the feature detection unit 27.

[0095] 16 shows each frame data FR of the regional video 11 corresponding to the frame image F11, frame image F13, and frame image F15 shown in FIG. 9. As described above, the skeletal information generation unit 26 generates skeletal position information 51 in the captured video 10. The rotation calculation unit 34D acquires position information of the feet (feet LG) of the subject SB in the frame images. In addition, the feature detection unit 27 detects the deck 42 and the wheels 41, and generates the position coordinates of the deck 42 and the wheels 41 in the frame image F. The rotation calculation unit 34D acquires the position coordinates of the deck 42 and the wheels 41 in the frame data FR.

[0096] The rotation calculation unit 34D determines the top and bottom of the skateboard 4 based on the relationship between the positions of the feet (feet LG) of the subject SB, the deck 42, and the wheels 41 in the frame data FR. In FIG. 16, in frame images F11 and F15, the feet (feet LG), the deck 42, and the wheels 41 are arranged in this order from the top of the image. In particular, the deck 42 is interposed between the wheels 41 and the feet LG. This positional relationship is referred to as a normal positional relationship. On the other hand, in frame image F13, which shows a state in which the skateboard 4 has made a half rotation (a state in which the top and bottom are inverted), the feet (feet LG), the wheels 41, and the deck 42 are arranged in this order from the top of the image. In particular, the wheels 41 and the feet LG are arranged in positions close to each other without the deck 42 in between. This positional relationship is referred to as an inverted positional relationship.

[0097] The rotation calculation unit 34D counts the number of times the region image 11 switches between the normal positional relationship and the inverted positional relationship (i.e., the number of times the position is turned upside down). The rotation calculation unit 34D calculates the number of rotations of the skateboard 4 based on the number of counts from when the normal positional relationship is first detected until the skateboard 4 lands. A switch between the normal positional relationship and the inverted positional relationship corresponds to a half rotation. In the case of FIG. 10, after frame data FR (frame image F11) in which the normal positional relationship has been achieved is detected, frame data FR (frame image F13) in which the inverted positional relationship has been achieved is detected, thereby counting one switch between the normal positional relationship and the inverted positional relationship (a half rotation). Subsequently, after the frame image F13, frame data FR (frame image F15) in which the normal positional relationship has been achieved is detected, thereby counting one switch between the normal positional relationship and the inverted positional relationship (one rotation). A landing is detected when the frame data FR matches the matching image 12A. As a result, in the case of FIG. 10, the rotation calculation unit 34D calculates the number of rotations as "1" because the normal positional relationship and the inverted positional relationship are switched twice.

[0098] Other configurations of the second embodiment are the same as those of the first embodiment. The effects of the coaching system 1 according to the second embodiment are the same as those of the first embodiment.

[0099] The components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads and usage conditions. This distribution and integration configuration may also be performed dynamically.

[0100] For example, in the above embodiment, an example was shown in which the coaching system 1 was provided with the camera 2 and the information processing device 3 separately, but instead of providing the information processing device 3, the camera 2 may be equipped with the functions of the information processing device 3 so that the output image 13 is generated solely by the camera 2. In other words, the control unit 25 of the camera 2 may be provided with an image generation unit 34A, a position calculation unit 34B, a matching image generation unit 34C, a rotation calculation unit 34D, and an output control unit 34F.

[0101] Although the embodiments of the present invention have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the so-called equivalent range. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments. [Explanation of symbols]

[0102] 1. Coaching System 2 Cameras 4. Skateboarding 10 Footage 11 Area Images 13 Output video 14 Markers 26 Skeleton information generation unit 34A Image generation unit 34B Position calculation section 34D Rotation Calculation Unit 34F Output control section 51 Skeletal position information CP intermediate position A1 long axis A2 Vertical axis

Claims

1. A camera for acquiring the captured image; an image generating unit that acquires an area including the subject's feet and the skateboard on which the subject is riding from the captured image and generates an area image having an angle of view of the area; a position calculation unit that calculates an intermediate position of the skateboard based on the skateboard in the area image; an output control unit that generates an output image including the region image and a marker that indicates the calculated intermediate position; Coaching system.

2. the camera further includes a skeleton information generating unit that generates skeleton position information of a subject from the captured video; the output control unit generates the output image including the region image, a marker indicating the calculated intermediate position, and the skeleton position information. The coaching system of claim 1 .

3. A camera for acquiring the captured image; an image generating unit that acquires an area including the subject's feet and the skateboard on which the subject is riding from the captured image and generates an area image having an angle of view of the area; a rotation calculation unit that calculates the number of rotations of the skateboard based on a change in the skateboard in the area image; and an output control unit that generates an output image including the region image and the calculated number of rotations. Coaching system.

4. the rotation calculation unit calculates the number of rotations of the skateboard around the major axis based on the number of times the skateboard is turned upside down in the region image. The coaching system according to claim 3 .

5. the rotation calculation unit calculates the number of rotations of the skateboard around the vertical axis based on a change in the wheel spacing of the skateboard in the area image.

5. The coaching system according to claim 3 or 4.

Images