Information processing device, information processing method, and program

Abstract

The present technology relates to an information processing device, an information processing method, and a program which make it easy to set the frame, in a moving image of ball game play, from which prescribed processing is to start. This information processing device is provided with: an analyzing unit for detecting the trajectory of a ball in a first moving image; and a starting frame setting unit for setting a starting frame from which prescribed processing is to start, from among the frames in the first moving image, on the basis of the detection result of the trajectory of the ball in the first moving image. The present technology can be applied to a system for analyzing serve revolution characteristics in table tennis, for example.

Description

[Technical field]

[0001] The present technology relates to an information processing device, an information processing method, and a program, and in particular to an information processing device, an information processing method, and a program that process video of plays in ball games. [Background technology]

[0002] For example, when analyzing the rotation characteristics of a ball immediately after it is hit in a ball game such as table tennis, in order to improve the accuracy of the analysis, it is desirable to detect a frame that captures the moment of impact of the ball from a video of the ball game being played, and to perform detection processing based on that frame.

[0003] Conventionally, however, a person would search for the corresponding frame by visual inspection. However, for example, when a video has a high frame rate, the number of frames becomes enormous, making it difficult to detect the corresponding frame.

[0004] In response to this, a technology has been proposed in the past that detects a motion pattern based on sensor data from a sensor attached to the user or an object that comes into contact with the user, and selects an image of the moment of ball impact based on the motion pattern (see, for example, Patent Document 1).

[0005] Further, a technique has been proposed in the past in which the positional relationship between a golf club head and a ball is calculated based on the movement vector of the club head, and the moment of impact is predicted based on the calculated positional relationship (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2015-82817 A [Patent Document 2] JP 2010-239169 A Summary of the Invention [Problem to be solved by the invention]

[0007] However, the technology described in Patent Document 1 requires a sensor to be attached to the user or to an object that comes into contact with the user.

[0008] Furthermore, the technology described in Patent Document 2 does not specifically consider a method for predicting the moment of impact in ball games such as table tennis, in which a moving ball is hit.

[0009] The present technology has been made in consideration of such circumstances, and makes it possible to easily set a frame at which to start a predetermined process in a video of a ball game play. [Means for solving the problem]

[0010] According to one aspect of the present technology, there is provided an information processing device comprising: an extraction range setting unit that sets an extraction range for extracting a first moving image from a second moving image based on a third moving image having a frame rate lower than that of a second moving image generated by capturing images of a ball game; The device includes an analysis unit that detects the trajectory of the ball in a first video, and a start frame setting unit that sets a start frame from among the frames of the first video to start a predetermined process based on the detection result of the trajectory of the ball in the first video.

[0011] An information processing method according to one aspect of the present technology includes: setting an extraction range for extracting a first moving image from a second moving image based on a third moving image having a frame rate lower than that of a second moving image generated by capturing images of a ball game; Detecting the ball's trajectory in a video To do , the above 1. Based on the detection result of the trajectory of the ball in the video, 1. Set the start frame from which a certain process will start among the frames of the video This includes the following.

[0012] A program according to one aspect of the present technology includes: setting an extraction range for extracting a first moving image from a second moving image based on a third moving image having a frame rate lower than that of a second moving image generated by capturing images of a ball game; Detecting the ball's trajectory in a video To do , the above 1. Based on the detection result of the trajectory of the ball in the video, 1. Set the start frame from which a certain process will start among the frames of the video Including The processing is executed by a computer.

[0013] In one aspect of the present technology, An extraction range for extracting the first moving image from the second moving image is set based on a third moving image having a frame rate lower than that of the second moving image generated by capturing images of a ball game, and the first The trajectory of the ball in the video is detected, 1. Based on the detection result of the trajectory of the ball in the video, 1. A start frame from which a predetermined process is to be started is set from among the frames of the video. [Brief description of the drawings]

[0014] [Figure 1] 1 is a block diagram showing a configuration example of an information processing system to which the present technology is applied. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of a server. [Diagram 3] FIG. 2 is a block diagram showing an example of a functional configuration of an information processing device. [Figure 4] 11 is a flowchart illustrating a video editing process executed by the information processing system. [Diagram 5] FIG. 13 is a diagram showing an example of a trajectory of a ball at the moment of serving. [Figure 6] 13 is a flowchart illustrating details of a start frame setting process. [Figure 7] 13 is a flowchart illustrating details of a start frame setting process. [Figure 8] 13 is a flowchart illustrating details of a start frame setting process. [Figure 9] 10A and 10B are diagrams for explaining a method of detecting a change point in the trajectory of a ball. [Figure 10] FIG. 11 is a diagram for explaining a method for setting a start frame. [Figure 11] FIG. 11 is a diagram for explaining a method for setting a start frame. [Figure 12] 10 is a flowchart for explaining details of a rotation characteristic analysis process. [Figure 13] 11 is a graph showing an example of a calculation result of the similarity between logo extraction images of each frame. [Figure 14] FIG. 1 illustrates an example of the configuration of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] Hereinafter, an embodiment of the present technology will be described in the following order. 1. Embodiment 2. Variations 3.Other

[0016] <<1. Preferred embodiment>> An embodiment of the present technology will be described with reference to FIGS.

[0017] <Example of information processing system configuration> FIG. 1 is a block diagram showing an embodiment of an information processing system 1 to which the present technology is applied.

[0018] The information processing system 1 is, for example, a system that captures images of ball games being played and performs processes such as analysis and editing of captured moving images generated from the captured images.

[0019] In this specification, ball games refer to sports that use balls in a broad sense, including a shuttlecock in badminton and a puck in ice hockey.

[0020] The information processing system 1 includes a camera 11, a server 12, and an information processing device 13. The server 12 and the information processing device 13 are connected to each other via a network 21.

[0021] Camera 11 captures images of ball games played at a high frame rate (e.g., 960p) and records audio. Camera 11 generates captured video including audio recorded at the same time as the images are captured, that is, audio corresponding to the video (hereinafter referred to as recorded audio), and transmits the captured video to server 12.

[0022] The server 12 accumulates the captured video supplied from the camera 11. The server 12 also generates a sub-clip, which is a video obtained by extracting a part of the captured video. Furthermore, the server 12 converts the frame rate of the captured video to generate a video with a lower frame rate (for example, 60i) (hereinafter referred to as LFR video). The server 12 transmits various types of video to the information processing device 13 via the network 21.

[0023] The information processing device 13 generates information requesting transmission of various moving images, transmits the information to the server 12 via the network 21, and receives the requested moving images from the server 12. The information processing device 13 also performs processes such as analysis and editing of the received moving images.

[0024] <Example of Server 12 function configuration> FIG. 2 shows an example of the functional configuration of the server 12.

[0025] The server 12 includes a communication unit 51, an information processing unit 52, and a storage unit 53.

[0026] The communication unit 51 communicates with the camera 11 and with the information processing device 13 via the network 21. For example, the communication unit 51 receives captured moving images from the camera 11 and supplies the captured moving images to the information processing unit 52. Also, for example, the communication unit 51 transmits various moving images acquired from the information processing unit 52 to the information processing device 13 via the network 21.

[0027] Any communication method may be used by the communication unit 51. The communication unit 51 may also be compatible with a plurality of communication methods.

[0028] The information processing unit 52 controls each unit of the server 12 and performs various information processes. The information processing unit 52 includes an image processing unit 61.

[0029] The image processing unit 61 generates an LFR moving image by converting the frame rate of the captured moving image. The image processing unit 61 supplies the generated LFR moving image to the communication unit 51 and stores it in the storage unit 53.

[0030] The image processing unit 61 also generates a sub-clip by extracting a part of the captured video. The image processing unit 61 supplies the sub-clip to the communication unit 51 and stores it in the storage unit 53.

[0031] The storage unit 53 includes, for example, non-volatile and volatile memory, and stores various programs and data used in the processing of the server 12.

[0032] <Example of functional configuration of information processing device 13> FIG. 3 shows an example of the functional configuration of the information processing device 13.

[0033] The information processing device 13 includes an input unit 101 , a communication unit 102 , an information processing unit 103 , an output unit 104 , and a storage unit 105 .

[0034] The input unit 101 includes input devices such as a keyboard, a mouse, and a microphone, and supplies input data to the information processing unit 103 .

[0035] The communication unit 102 communicates with the server 12 via the network 21. For example, the communication unit 102 transmits information requesting transmission of various moving images to the server 12 and receives the requested moving images from the server 12. In addition, for example, the communication unit 102 can also communicate with other devices not included in the information processing system 1.

[0036] Any communication method may be used by the communication unit 102. The communication unit 102 may also be compatible with a plurality of communication methods.

[0037] The information processing unit 103 controls each unit of the information processing device 13 and performs various information processes. The information processing unit 103 includes an analysis unit 121, an extraction range setting unit 122, a start frame setting unit 123, and an image processing unit .

[0038] The analysis unit 121 performs various analytical processes on various moving images. For example, the analysis unit 121 analyzes plays in ball games. Also, for example, the analysis unit 121 analyzes recorded audio included in the moving images. The analysis unit 121 supplies information indicating the analysis results to the extraction range setting unit 122, the start frame setting unit 123, the image processing unit 124, and the output unit 104 as necessary, or stores the information in the storage unit 105.

[0039] The extraction range setting unit 122 sets an extraction range for extracting a sub-clip based on the analysis result of the LFR video, etc. The extraction range setting unit 122 supplies information including the extraction range and requesting transmission of the sub-clip to the communication unit 102.

[0040] The start frame setting unit 123 sets a start frame for starting analysis of a play in a ball game based on the analysis result of the sub-clip. The start frame setting unit 123 supplies the analysis unit 121 with information indicating the start frame.

[0041] The image processing unit 124 edits the moving image based on instructions from the user input to the input unit 101. The image processing unit 124 supplies the edited moving image to the communication unit 102 and the output unit 104, or stores the edited moving image in the storage unit 105, as necessary.

[0042] The output unit 104 includes output devices such as a display and a speaker, and displays moving images and outputs audio associated with the moving images.

[0043] The storage unit 105 includes, for example, non-volatile and volatile memories, and stores various programs and data used in the processing of the information processing device 13 .

[0044] <Video editing processing> Next, the video editing process executed by the information processing system 1 will be described with reference to the flowchart of FIG.

[0045] Hereinafter, a specific example will be described in which the rotation characteristics of a serve are analyzed based on a video captured at a high frame rate by capturing a table tennis play with the camera 11, and rotation information indicating the rotation characteristics of the serve is synthesized with the video. It is assumed that the captured video has already been captured and stored in the storage unit 53 of the server 12.

[0046] In step S1, the information processing system 1 sets a range for extracting a sub-clip based on a low frame rate video (LFR video).

[0047] Specifically, the analysis unit 121 of the information processing device 13 generates information requesting transmission of the LFR video, and transmits the information to the server 12 via the communication unit 102 and the network 21.

[0048] In response to this, the image processing unit 61 of the server 12 receives information requesting transmission of the LFR moving image via the communication unit 51. The image processing unit 61 converts the captured moving image with a high frame rate stored in the storage unit 53 into an LFR moving image with a low frame rate. The image processing unit 61 also transmits the LFR moving image to the information processing device 13 via the communication unit 51 and the network 21.

[0049] The LFR video also includes recorded audio corresponding to the captured video. The image processor 61 may also process the recorded audio, such as by lowering the sampling rate.

[0050] In response to this, the information processing unit 103 of the information processing device 13 receives the LFR moving image via the communication unit 102. The extraction range setting unit 122 sets a range for extracting a sub-clip from the captured moving image, based on the LFR moving image.

[0051] Here, an example of a method for setting the extraction range of a sub-clip will be described.

[0052] For example, the analysis unit 121 analyzes at least one of the LFR video and the recorded audio, and the extraction range setting unit 122 sets the extraction range of the sub-clip based on the analysis result.

[0053] For example, the analysis unit 121 detects the trajectory of the ball by tracking the ball in the LFR video. Based on the change in the trajectory of the ball, the analysis unit 121 detects a frame of the LFR video at the moment of or immediately after the impact of the serve.

[0054] FIG. 5 shows an example of the trajectory of a table tennis ball at the moment of serving.

[0055] When a table tennis player serves, he or she throws the ball up in the air and then hits the ball as it falls. Therefore, the ball falls vertically before hitting the racket, as shown by positions P1 to P4 in Figure 5. If the serve is hit when the ball reaches position P4, the direction of travel of the ball changes and it moves in the direction shown by positions P4 to P7.

[0056] For example, the analysis unit 121 detects a change point in the LFR video where the ball's traveling direction changes by a predetermined angle or more, and detects a frame of the LFR video at the moment when the ball is hit at position P4 (moment of impact) or immediately thereafter.

[0057] However, since an LFR video has a low frame rate, it does not necessarily include a frame that captures the moment of impact, and therefore the analysis unit 121 may only be able to detect an approximate moment of impact.

[0058] The analysis unit 121 supplies information indicating the detected frame of the LFR video to the extraction range setting unit 122. The extraction range setting unit 122 sets, as the extraction range, a range of a predetermined number of frames before and after the frame of the HFR video corresponding to the detected frame of the LFR video.

[0059] For example, the user may detect a frame at or immediately after the impact of the serve while watching the LFR video.

[0060] Also, for example, the venue usually becomes quiet just before a serve. Therefore, for example, the analysis unit 121 detects the time when the venue becomes quiet by analyzing the recorded sound, and supplies information indicating the detection result to the extraction range setting unit 122. The time when the venue becomes quiet is assumed to be a little before the impact of the serve. The extraction range setting unit 122 sets the extraction range to a predetermined range before and after the frame of the HFR video a predetermined time after the detected time.

[0061] Furthermore, for example, if a rally continues after a serve, hitting sounds and rebound sounds occur with a certain degree of regularity from the first hitting sound. Therefore, for example, the analysis unit 121 analyzes the recorded sound, and when it detects hitting sounds and rebound sounds occurring with a certain degree of regularity, it detects the time when the first hitting sound occurred and supplies information indicating the detection result to the extraction range setting unit 122. It is assumed that the time when this first hitting sound occurred is approximately the same as the moment of impact of the serve. The extraction range setting unit 122 sets a predetermined range before and after the frame of the HFR video at the detected time as the extraction range.

[0062] Also, for example, the analysis unit 121 may recognize at least one of the player's movements and postures by analyzing the LFR video, and the extraction range setting unit 122 may set the extraction range based on the recognition result of at least one of the player's movements and postures.

[0063] For example, many table tennis players perform a set action (routine) immediately before serving. Therefore, for example, when the analysis unit 121 analyzes the LFR video and detects a player's routine, it detects the time when the routine ends and supplies information indicating the detection result to the extraction range setting unit 122. The time when the routine ends is assumed to be shortly before the impact of the serve. The extraction range setting unit 122 sets, as the extraction range, a predetermined range before and after the frame of the HFR video a predetermined time after the detection time.

[0064] Also, for example, the analysis unit 121 recognizes bones of a player based on the LFR video and detects the movement of the player's bones. Then, based on the movement of the player's bones, the analysis unit 121 detects a frame of the LFR video at the moment of or immediately after the impact of the serve. The analysis unit 121 supplies information indicating the detected frame of the LFR video to the extraction range setting unit 122. The extraction range setting unit 122 sets a predetermined range before and after a frame corresponding to the detected LFR video frame in the HFR video as an extraction range.

[0065] Note that the extraction range of the sub-clip may be set by combining some of the above-mentioned methods.

[0066] In the following description, the number of frames in a sub-clip (the number of frames in the extraction range) is Fmax, the frame number of the first frame of a sub-clip is 1, and the frame number of the last frame is Fmax.

[0067] The extraction range setting unit 122 transmits, via the communication unit 102 and the network 21, information including the extraction range of the sub-clip and requesting transmission of the sub-clip.

[0068] In response to this, the image processing unit 61 of the server 12 receives, via the communication unit 51, information requesting transmission of the sub-clip.

[0069] In step S2, the server 12 extracts a sub-clip from the captured video.

[0070] Specifically, the image processing unit 61 extracts, from the captured moving image, a moving image in the extraction range set by the extraction range setting unit 122 as a sub-clip. The image processing unit 61 transmits the extracted sub-clip to the information processing device 13 via the communication unit 51 and the network 21.

[0071] In response to this, the information processing unit 103 of the information processing device 13 receives the sub-clip via the communication unit 102 .

[0072] In step S3, the information processing section 103 executes a start frame setting process.

[0073] The start frame setting process will now be described in detail with reference to the flowcharts of FIGS.

[0074] In step S51, the analysis unit 121 sets 1 to a variable i.

[0075] In step S52, the analysis unit 121 determines whether or not a ball is detected in the i-th frame (hereinafter referred to as frame i) of the sub clip. The analysis unit 121 searches for the ball in frame i of the sub clip. As a result, if the analysis unit 121 determines that the ball cannot be detected in frame i of the sub clip, the process proceeds to step S53.

[0076] In step S53, the analysis unit 121 increments the value of the variable i by one.

[0077] In step S54, the analysis unit 121 determines whether or not the variable i is equal to or less than the number of frames Fmax. If it is determined that the variable i is equal to or less than the number of frames Fmax, the process returns to step S52.

[0078] Thereafter, the processes of steps S52 to S54 are repeatedly executed until it is determined in step S52 that a ball has been detected in frame i of the sub-clip, or until it is determined in step S54 that the variable i has exceeded the number of frames Fmax.

[0079] On the other hand, if it is determined in step S54 that the variable i exceeds the number of frames Fmax, the start frame setting process ends. This assumes that the ball has not been detected in any of the frames of the sub-clip, and no start frame is set.

[0080] Also, if it is determined in step S52 that a ball has been detected in frame i of the sub-clip, the process proceeds to step S55.

[0081] In step S55, the analysis unit 121 sets the value of the variable i to the variable Fs. That is, the frame number in which the ball first appears in the sub-clip is set to the variable Fs.

[0082] In step S56, the analysis unit 121 starts tracking the ball.

[0083] In step S57, the analysis unit 121 increments the value of the variable i by one.

[0084] In step S58, similarly to the process in step S54, it is determined whether or not the variable i is equal to or less than the number of frames Fmax. If it is determined that the variable i is equal to or less than the number of frames Fmax, the process proceeds to step S59.

[0085] In step S59, similar to the process in step S52, it is determined whether or not a ball is detected in frame i of the sub clip. If it is determined that a ball is detected in frame i of the sub clip, the process proceeds to step S60.

[0086] In step S60, the analysis unit 121 determines whether or not a change point has been detected.

[0087] Now, with reference to FIG. 9, an example of a method for detecting a change point in the trajectory of the ball will be described.

[0088] The analysis unit 121 calculates the vector V(i) of the ball in frame i by the following equation (1).

[0089] V(i) = P(i) - P(ic) (1)

[0090] c is a constant. P(i) indicates the position of the ball in frame i. P(ic) indicates the position of the ball in frame ic, which is c frames before frame i.

[0091] Next, the analysis unit 121 calculates the angle θ(i) of the ball's traveling direction in frame i using the following equation (2).

[0092] θ(i)=tan -1 (V(i).y / V(i).x)×(180 / π) (2)

[0093] V(i).x is the x-component of vector V(i), and V(i).y is the y-component of vector V(i).

[0094] Next, the analysis unit 121 calculates the amount of change Δθ(i) in the traveling direction between frame i and frame i−1 by the following equation (3).

[0095] Δθ(i)=|θ(i)-θ(i-1)| ···(3)

[0096] According to the above-mentioned formulas (1) to (3), the ball's turning point is usually detected in a frame after the moment of impact of the ball.

[0097] In addition, the smaller the constant c in formula (1), the closer the change point can be detected to the position of impact of the ball. On the other hand, the larger the constant c, the more reliably it is possible to prevent erroneous detection of the change point.

[0098] If the amount of change Δθ(i) is equal to or greater than a predetermined threshold, i.e., if the traveling direction of the ball has changed by a predetermined angle or more, the analysis unit 121 determines that the position P(i) of the ball in frame i of the sub clip is a change point. On the other hand, if the amount of change Δθ(i) is less than the predetermined threshold, i.e., if the traveling direction of the ball has not changed by the threshold or more, the analysis unit 121 determines that the position P(i) of the ball in frame i of the sub clip is not a change point.

[0099] If the amount of change Δθ(i) is less than the predetermined threshold, the analysis unit 121 determines that a change point has not been detected, and the process proceeds to step S61.

[0100] In step S61, the analysis unit 121 determines whether or not the ball has been detected for a predetermined number of consecutive frames. If it is determined that the ball has not yet been detected for the predetermined number of consecutive frames, the process returns to step S57.

[0101] The predetermined number of frames is set, for example, to be equal to or greater than the minimum number of frames required to analyze the rotation characteristics of the ball.

[0102] Thereafter, steps S57 to S61 are repeatedly executed until it is determined in step S58 that the variable i has exceeded the number of frames Fmax, or in step S59 that the ball has not been detected in frame i of the sub-clip, or in step S60 that a change point has been detected, or in step S61 that the ball has been detected for a predetermined number of consecutive frames.

[0103] On the other hand, if it is determined in step S61 that the ball has been detected continuously for the predetermined number of frames, the process proceeds to step S62.

[0104] This is assumed to be the case where, after the ball is first detected in the sub-clip, the ball is detected for a predetermined number of consecutive frames without detecting a change point. For example, it is assumed that, in the sub-clip, the ball does not appear before the serve or at the moment of impact because it is hidden by the shadow of a player, and then appears continuously after the serve is hit. Specifically, for example, as shown in FIG. 10, it is assumed that, in the sub-clip, the ball does not appear until it reaches position P5, and then appears continuously after it reaches position P5.

[0105] In step S62, the start frame setting unit 123 sets a start frame based on the Fs-th frame (frame Fs) of the sub-clip.

[0106] Specifically, the analysis unit 121 notifies the start frame setting unit 123 that the ball has been detected in a predetermined number of consecutive frames starting from frame Fs of the sub-clip.

[0107] The start frame setting unit 123 sets, for example, frame Fs of the sub-clip as the start frame. That is, for example, the frame in which the ball appears for the first time after the serve is finished is set as the start frame. As a result, among the frames in which the ball appears in the sub-clip, the frame that is after the moment of impact of the serve and closest to the moment of impact of the serve (the frame in the vicinity of the moment of impact) is set as the start frame.

[0108] The start frame setting unit 123 supplies information indicating the set start frame to the analysis unit 121 .

[0109] After that, the start frame setting process ends.

[0110] On the other hand, in step S60, if the amount of change Δθ(i) is equal to or greater than the predetermined threshold, the analysis unit 121 determines that a change point has been detected, and the process proceeds to step S63.

[0111] In step S63, the analysis unit 121 sets the value of the variable i to the variable Fs. That is, the frame number in which the ball's changing point is detected in the sub-clip is set to the variable Fs.

[0112] In step S64, the analysis unit 121 increments the value of the variable i by one.

[0113] In step S65, similarly to the process in step S54, it is determined whether or not the variable i is equal to or less than the number of frames Fmax. If it is determined that the variable i is equal to or less than the number of frames Fmax, the process proceeds to step S66.

[0114] In step S66, similar to the process in step S52, it is determined whether or not a ball is detected in frame i of the sub clip. If it is determined that a ball is detected in frame i of the sub clip, the process proceeds to step S67.

[0115] In step S67, the analysis unit 121 determines whether or not the ball has been detected for a predetermined number of consecutive frames since the change point was detected. If it is determined that the ball has not been detected for the predetermined number of consecutive frames since the change point was detected, the process returns to step S64.

[0116] The predetermined number of frames is set, for example, to be equal to or greater than the minimum number of frames required to analyze the rotation characteristics of the ball.

[0117] Thereafter, steps S64 to S67 are repeatedly executed until it is determined in step S65 that the variable i has exceeded the number of frames Fmax, or it is determined in step S66 that the ball has not been detected in frame i of the sub-clip, or it is determined in step S67 that the ball has been detected for a predetermined number of consecutive frames after the detection of a change point.

[0118] On the other hand, if it is determined in step S67 that the ball has been detected for a predetermined number of consecutive frames since the change point was detected, the process proceeds to step S68.

[0119] This may be the case, for example, as shown in Figure 5 above, in a sub-clip where the ball appears continuously from before the serve is hit until after it is finished, and the ball is detected for a predetermined number of consecutive frames after the impact of the serve.

[0120] In step S68, the start frame setting unit 123 sets a start frame based on the Fs-th frame (frame Fs) of the sub-clip.

[0121] Specifically, the analysis unit 121 detects a change point in frame Fs of the sub-clip, and then notifies the start frame setting unit 123 that the ball has been detected for a predetermined number of consecutive frames.

[0122] The start frame setting unit 123 sets, for example, a frame a predetermined frame before (for example, one frame before) frame Fs of the sub-clip as the start frame. In this way, for example, a frame capturing the moment or immediately after the impact of the serve is set as the start frame.

[0123] The start frame setting unit 123 supplies information indicating the set start frame to the analysis unit 121 .

[0124] After that, the start frame setting process ends.

[0125] On the other hand, if it is determined in step S66 that the ball cannot be detected in frame i of the sub-clip, the start frame setting process ends. This is assumed to be the case, for example, when the ball cannot be detected for a predetermined number of consecutive frames after a transition point is detected, and a start frame is not set.

[0126] Furthermore, if it is determined in step S65 that the variable i has exceeded the number of frames Fmax, the start frame setting process ends. This assumes that the sub-clip ends before the ball is detected for a predetermined number of consecutive frames after the change point is detected, and no start frame is set.

[0127] Furthermore, if it is determined in step S59 that the ball cannot be detected in frame i of the sub-clip, the process proceeds to step S69, which assumes that the ball once appeared in the sub-clip and then disappeared.

[0128] In step S69, the analysis unit 121 increments the value of the variable i by one.

[0129] In step S70, similarly to the process in step S54, it is determined whether or not the variable i is equal to or less than the number of frames Fmax. If it is determined that the variable i is equal to or less than the number of frames Fmax, the process proceeds to step S71.

[0130] In step S71, similar to the process in step S52, it is determined whether or not a ball is detected in frame i of the sub-clip. If it is determined that a ball is not detected in frame i of the sub-clip, the process returns to step S69.

[0131] Thereafter, the processes of steps S69 to S71 are repeatedly executed until it is determined in step S70 that the variable i has exceeded the number of frames Fmax, or until it is determined in step S71 that a ball has been detected in frame i of the sub-clip.

[0132] On the other hand, if it is determined in step S71 that the ball has been detected in frame i of the sub-clip, the process proceeds to step S72. This is assumed to be the case when the ball disappears once and then reappears in the sub-clip.

[0133] In step S72, the analysis unit 121 sets the value of the variable i to the variable Fs. That is, the frame number in which the ball reappears in the sub-clip is set to the variable Fs.

[0134] In step S73, ball tracking is started, similar to the process in step S56.

[0135] In step S74, the analysis unit 121 increments the value of the variable i by one.

[0136] In step S75, similarly to the process in step S54, it is determined whether or not the variable i is equal to or less than the number of frames Fmax. If it is determined that the variable i is equal to or less than the number of frames Fmax, the process proceeds to step S76.

[0137] In step S76, similar to the process in step S52, it is determined whether or not a ball is detected in frame i of the sub clip. If it is determined that a ball is detected in frame i of the sub clip, the process proceeds to step S77.

[0138] In step S77, analysis unit 121 determines whether or not the ball has been detected for a predetermined number of consecutive frames since the ball was redetected. If it is determined that the ball has not been detected for the predetermined number of consecutive frames since the ball was redetected, processing returns to step S74.

[0139] The predetermined number of frames is set, for example, to be equal to or greater than the minimum number of frames required to analyze the rotation characteristics of the ball.

[0140] Thereafter, steps S74 through S77 are repeated until it is determined in step S75 that the variable i has exceeded the number of frames Fmax, or it is determined in step S76 that the ball has not been detected in frame i of the sub-clip, or it is determined in step S77 that the ball has been detected for a predetermined number of consecutive frames since the ball was redetected.

[0141] On the other hand, if it is determined in step S77 that the ball has been detected consecutively for the predetermined number of frames since the ball was detected again, the process proceeds to step S78.

[0142] This may be the case, for example, where in a sub-clip, a ball appears before a serve is hit, then disappears temporarily by being hidden by a player's shadow, and then reappears after the serve is hit. Specifically, for example, as shown in FIG. 11, a sub-clip may have a case where a ball appears at position P1, then disappears at position P3, and then reappears at position P6.

[0143] In step S78, the start frame setting unit 123 sets a start frame based on the Fs-th frame (frame Fs) of the sub-clip.

[0144] Specifically, the analysis unit 121 notifies the start frame setting unit 123 that the ball has been detected for a predetermined number of consecutive frames since detecting the ball in frame Fs of the sub-clip.

[0145] The start frame setting unit 123 sets, for example, frame Fs of the sub-clip as the start frame. That is, for example, the frame in which the ball first appears after the serve is finished is set as the start frame. As a result, among the frames in which the ball appears in the sub-clip, the frame that is after the moment of impact of the serve and closest to the moment of impact of the serve is set as the start frame.

[0146] The start frame setting unit 123 supplies information indicating the set start frame to the analysis unit 121 .

[0147] After that, the start frame setting process ends.

[0148] On the other hand, if it is determined in step S76 that the ball cannot be detected in frame i of the sub-clip, the start frame setting process ends. This is assumed to be the case, for example, when the ball is not detected for a predetermined number of consecutive frames in the sub-clip after it has been redetected, and a start frame is not set.

[0149] Furthermore, if it is determined in step S75 that the variable i has exceeded the number of frames Fmax, the start frame setting process ends. This assumes that the sub-clip ends before the ball is detected for a predetermined number of consecutive frames after the ball is detected again in the sub-clip, and a start frame is not set.

[0150] Furthermore, if it is determined in step S70 that the variable i has exceeded the number of frames Fmax, the start frame setting process ends. This is assumed to be the case in which the ball has disappeared in the sub-clip and has not been detected again, and the start frame is not set.

[0151] Furthermore, if it is determined in step S58 that the variable i has exceeded the number of frames Fmax, the start frame setting process ends. This assumes that the sub-clip ends before the ball is detected for a predetermined number of consecutive frames after the ball is detected in the sub-clip, and no start frame is set.

[0152] 4, in step S4, analysis unit 121 determines whether or not a start frame has been set based on the result of the process in step S3. If it is determined that a start frame has been set, the process proceeds to step S5.

[0153] In step S5, the analysis unit 121 executes a rotation characteristic analysis process.

[0154] Here, the rotation characteristic analysis process will be described in detail with reference to the flowchart of FIG.

[0155] In step S101, the analysis unit 121 extracts a ball image from each frame. Specifically, the analysis unit 121 detects an area in which the ball is captured and a center point of the ball in each frame from the start frame of the sub-clip to a predetermined number of frames later.

[0156] The predetermined number of frames is set, for example, to be equal to or greater than the minimum number of frames required to analyze the rotation characteristics of the ball.

[0157] The analysis unit 121 extracts an image of the area in which the ball appears (hereinafter referred to as a ball image) from each frame, with the detected center point of the ball as the center. If necessary, the analysis unit 121 adjusts the size of the ball image so that the size of the ball matches between each of the extracted ball images.

[0158] In step S102, the analysis unit 121 extracts the ball logo from the ball image of each frame. For example, the analysis unit 121 performs moving average difference, noise removal, etc. on each ball image to generate a logo extraction image in which the ball logo is extracted from each ball image. The logo extraction image is, for example, a binary image in which the logo part is displayed in white and the other parts are displayed in black.

[0159] In step S103, analysis unit 121 calculates the similarity between the logo extraction images of each frame. For example, analysis unit 121 calculates the sum of the similarities between the logo extraction images of each frame while changing the frame interval n. The similarity is calculated using, for example, ZNCC (Zero means Normalized Cross Correlation).

[0160] In step S104, analysis unit 121 determines the frame interval that maximizes the sum of the similarities between the logo extraction images.

[0161] 13 is a graph showing an example of the calculation results when ZNCC is used to calculate the similarity between logo-extracted images of each frame. The horizontal axis of Fig. 13 represents the frame interval, and the vertical axis represents the sum of the similarity.

[0162] In this example, the frame interval Imax at which the sum of the similarities between the logo extracted images of each frame becomes maximum is 26.5.

[0163] In step S105, the analysis unit 121 calculates the number of rotations of the ball. Specifically, the analysis unit 121 calculates the number of rotations of the ball by dividing the frame rate of the sub-clip (picked up video) by the frame interval obtained in the process of step S104.

[0164] For example, in the example of FIG. 13, when the frame rate of the sub-clip is 960 fps, the number of rotations of the ball is 36 rps (≈960 / 26.5).

[0165] In step S106, the analysis unit 121 detects the rotation axis of the ball using the gradient method.

[0166] For example, the analysis unit 121 generates an exploration rotated image by rotating the ball image of each frame around the rotation axis r(θ, φ) for one frame. The amount of rotation for one frame is calculated based on the number of rotations calculated in the process of step S105.

[0167] Then, the analysis unit 121 calculates the similarity between the exploration rotated image of each frame and the logo extraction image one frame later. For example, the sum of the similarity between the exploration rotated image of the first frame and the logo extraction image of the second frame, the similarity between the exploration rotated image of the second frame and the logo extraction image of the third frame, ..., the similarity between the exploration rotated image of the (n-1)th frame and the logo extraction image of the nth frame is calculated. The analysis unit 121 calculates the sum of the similarity for each rotation axis r(θ, φ) while changing the rotation axis r(θ, φ).

[0168] Next, the analysis unit 121 searches for the rotation axis r(θ,φ) that maximizes the sum of the similarities using a gradient method, and estimates that the rotation axis r(θ,φ) that maximizes the sum of the similarities is the rotation axis of the ball.

[0169] Furthermore, analysis unit 121 estimates the rotation direction of the ball based on the movement direction of the logo in the logo extraction image.

[0170] The analysis unit 121 supplies the image processing unit 124 with information indicating the number of rotations, the axis of rotation, and the direction of rotation of the ball, as well as the starting frame.

[0171] Then, the rotation characteristic analysis process ends.

[0172] Returning to FIG. 4, in step S6, the image processing unit 124 combines the rotation characteristic information with the moving image.

[0173] For example, the image processing unit 124 generates information requesting transmission of a captured video, and transmits the information to the server 12 via the communication unit 102 and the network 21.

[0174] In response to this, the image processing unit 61 of the server 12 receives information requesting transmission of the captured video via the communication unit 51. The image processing unit 61 reads out the captured image from the storage unit 53, and transmits the captured image to the information processing device 13 via the communication unit 51 and the network 21.

[0175] In response to this, the image processing unit 124 of the information processing device 13 receives the captured video via the communication unit 102. The image processing unit 124 detects a frame corresponding to the start frame in the captured video. Then, the image processing unit 124 superimposes rotation characteristic information indicating the rotation characteristic of the serve near the detected frame.

[0176] The spin characteristic information includes, for example, display effects and numerical values ​​that indicate the spin characteristics of the serve.

[0177] For example, it is also possible to superimpose the rotation characteristic information on the sub-clip. Also, for example, the image processing unit 124 of the information processing device 13 may obtain a captured video of a predetermined range before and after the start frame from the server 12, and superimpose the rotation characteristic information.

[0178] After that, the video editing process ends.

[0179] On the other hand, if it is determined in step S4 that the start frame has not been set, the processes in steps S5 and S6 are skipped and the video editing process ends. In other words, due to the setting error of the start frame, analysis of the ball's spin characteristics and composition of the spin characteristics information into the video are not performed.

[0180] In this manner, the rotation characteristics of the ball can be easily analyzed.

[0181] Specifically, the extraction range of the sub-clip is set using a low frame rate LFR video, and the start frame is set using the sub-clip. Therefore, compared to setting the start frame using a high frame rate captured video, the processing is reduced and the start frame can be set more quickly.

[0182] Furthermore, even without attaching sensors to the player or the racket, the moment of impact of the serve can be easily detected in a video (sub-clip) captured at a high frame rate.

[0183] Furthermore, even if the ball disappears from the video at the moment of impact of the serve due to being covered by a player's shadow, the frame closest to the moment of impact of the serve is set as the start frame, thereby improving the accuracy of analysis of the rotation characteristics of the serve.

[0184] <<2. Modifications>> Below, a modification of the above-described embodiment of the present technology will be described.

[0185] Although the above description has been given as an example of analyzing the spin characteristics of a table tennis serve, the present technology can also be applied to analyzing the spin characteristics of a table tennis smash, drive, cut, etc. In other words, the present technology can also be applied to analyzing the spin characteristics immediately after impact of a ball other than a table tennis serve.

[0186] The present technology can also be applied to analyzing the rotation characteristics immediately after impact of a ball in ball games other than table tennis. For example, the present technology can be applied to analyzing the rotation characteristics immediately after impact of a ball in tennis, badminton, baseball, ice hockey, etc. Also, the present technology can be applied to analyzing the rotation characteristics immediately after impact of a ball in ball games in which the ball is hit (including kicking) by the human body without using any equipment, such as volleyball and soccer.

[0187] Furthermore, the present technology can also be applied to detecting the rotation characteristics of a ball immediately after impact in ball games such as golf and billiards in which a stationary ball is hit.

[0188] The present technology can also be applied to analyzing the rotation characteristics of a thrown ball in ball games such as bowling and darts, in which a player throws a ball.

[0189] Furthermore, the present technology can be applied to cases where sub-clips are used for image analysis other than analysis of ball rotation characteristics. For example, it is assumed that the highlight of a ball game scene is the scene before and after the impact of the ball. Therefore, for example, the present technology can be applied to cases where sub-clips are extracted as highlight scenes of sports.

[0190] <<3.Other>> <Computer configuration example> The above-mentioned series of processes can be executed by hardware or software. When the series of processes is executed by software, the programs constituting the software are installed in a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, capable of executing various functions by installing various programs.

[0191] FIG. 14 is a block diagram showing an example of the hardware configuration of a computer that executes the above-mentioned series of processes by a program.

[0192] In a computer 1000, a central processing unit (CPU) 1001, a read only memory (ROM) 1002, and a random access memory (RAM) 1003 are interconnected by a bus 1004.

[0193] An input / output interface 1005 is further connected to the bus 1004. An input unit 1006, an output unit 1007, a recording unit 1008, a communication unit 1009, and a drive 1010 are connected to the input / output interface 1005.

[0194] The input unit 1006 includes an input switch, a button, a microphone, an image sensor, etc. The output unit 1007 includes a display, a speaker, etc. The recording unit 1008 includes a hard disk, a non-volatile memory, etc. The communication unit 1009 includes a network interface, etc. The drive 1010 drives removable media 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0195] In the computer 1000 configured as described above, the CPU 1001 performs the above-mentioned series of processes by, for example, loading a program recorded in the recording unit 1008 into the RAM 1003 via the input / output interface 1005 and the bus 1004 and executing it.

[0196] The program executed by the computer 1000 (CPU 1001) can be provided by being recorded on a removable medium 1011 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0197] In the computer 1000, the program can be installed in the recording unit 1008 via the input / output interface 1005 by mounting the removable medium 1011 in the drive 1010. The program can also be received by the communication unit 1009 via a wired or wireless transmission medium and installed in the recording unit 1008. Alternatively, the program can be installed in the ROM 1002 or the recording unit 1008 in advance.

[0198] In addition, the program executed by the computer may be a program in which processing is performed chronologically in the order described in this specification, or it may be a program in which processing is performed in parallel or at the required timing, such as when called.

[0199] In this specification, a system refers to a collection of multiple components (devices, modules (parts), etc.), regardless of whether all the components are in the same case. Therefore, multiple devices housed in separate cases and connected via a network, and a single device in which multiple modules are housed in a single case, are both systems.

[0200] Furthermore, the embodiments of the present technology are not limited to the above-described embodiments, and various modifications are possible without departing from the gist of the present technology.

[0201] For example, the present technology can be configured as cloud computing, in which a single function is shared and processed collaboratively by multiple devices via a network.

[0202] Furthermore, each step described in the above flow chart can be executed by one device, or can be shared and executed by a plurality of devices.

[0203] Furthermore, when a single step includes multiple processes, the multiple processes included in the single step can be executed by a single device, or can be shared and executed by multiple devices.

[0204] <Examples of configuration combinations> The present technology can also be configured as follows.

[0205] (1) an analysis unit that detects a trajectory of the ball in the first video; a start frame setting unit that sets a start frame for starting a predetermined process from among frames of the first moving image based on a detection result of the trajectory of the ball in the first moving image; An information processing device comprising: (2) the analysis unit detects a change point in the first video at which the traveling direction of the ball changes by a predetermined angle or more; The start frame setting unit sets the start frame based on a frame in which the transition point is detected. The information processing device according to (1). (3) The start frame is a frame that captures the moment of impact of the ball or a frame in the vicinity of the moment of impact of the ball. The information processing device according to (2). (4) When the change point is not detected in the first video, the start frame setting unit sets the start frame based on a first frame in a range in which the ball is detected consecutively in the first video for a predetermined number of frames or more. The information processing device according to (2) or (3). (5) The start frame setting unit does not set the start frame if the ball is not detected continuously for a predetermined number of frames or more in the first video. The information processing device according to any one of (2) to (4). (6) The first video is extracted from a second video generated by capturing images of a ball game. The information processing device according to any one of (1) to (5). (7) an extraction range setting unit that sets an extraction range for extracting the first moving image from the second moving image based on a third moving image having a frame rate lower than that of the first moving image; The information processing device according to (6) further comprises: (8) The analysis unit analyzes the third moving image, The extraction range setting unit sets the extraction range based on an analysis result of the third moving image. The information processing device according to (7). (9) The analysis unit detects a trajectory of the ball in the third video; The extraction range setting unit sets the extraction range based on a detection result of the trajectory of the ball in the third video. The information processing device according to (8). (10) the analysis unit detects a change point at which the traveling direction of the ball changes by a predetermined angle or more in the third video; The extraction range setting unit sets the extraction range based on a frame in which the transition point is detected. The information processing device according to (9). (11) The analysis unit recognizes at least one of a motion and a posture of a player based on the third video; The extraction range setting unit sets the extraction range based on at least one of the recognition results of the player's movement and posture. The information processing device according to any one of (8) to (10). (12) The analysis unit analyzes audio corresponding to the third video, The extraction range setting unit sets the extraction range based on a result of the analysis of the voice. The information processing device according to any one of (7) to (11). (13) The third moving image is generated based on the second moving image. The information processing device according to any one of (7) to (12). (14) The predetermined processing is image analysis. The information processing device according to any one of (1) to (13). (15) The image analysis is an analysis of the rotation characteristics of the ball. The information processing device according to (14). (16) The image analysis is an analysis of the rotation characteristics of a table tennis serve. The information processing device according to (15). (17) The analysis unit analyzes the rotation characteristics of the ball. The information processing device according to (15) or (16). (18) The ball is a table tennis ball 18. The information processing device according to any one of (1) to (17). (19) An information processing device, Detect the ball's trajectory in the video, A start frame for starting a predetermined process is set from among frames of the video based on a detection result of the trajectory of the ball in the video. Information processing methods. (20) Detect the ball's trajectory in the video, A start frame for starting a predetermined process is set from among frames of the video based on a detection result of the trajectory of the ball in the video. A program that causes a computer to execute a process.

[0206] It should be noted that the effects described in this specification are merely examples and are not limiting, and other effects may also be obtained. [Explanation of symbols]

[0207] 1 Information processing system, 11 Camera, 12 Server, 13 Information processing device, 52 Information processing section, 61 Image processing section, 103 Information processing section, 121 Analysis section, 122 Extraction range setting section, 123 Start frame setting section, 124 Image processing section

Claims

1. An extraction range setting unit that sets an extraction range for extracting a first video from a second video based on a third video generated by an image captured by capturing an image of a ball game, the second video having a lower frame rate than the third video; an analysis unit that detects a trajectory of a ball in the first video; a start frame setting unit that sets a start frame for starting a predetermined process from among frames of the first moving image based on a detection result of the trajectory of the ball in the first moving image; An information processing device comprising:

2. the analysis unit detects a change point at which the traveling direction of the ball changes by a predetermined angle or more in the first video; The start frame setting unit sets the start frame based on a frame in which the transition point is detected. The information processing device according to claim 1 .

3. The start frame is a frame that captures the moment of impact of the ball or a frame in the vicinity of the moment of impact of the ball. The information processing device according to claim 2 .

4. When the change point is not detected in the first video, the start frame setting unit sets the start frame based on a first frame in a range in which the ball is detected consecutively in the first video for a predetermined number of frames or more. The information processing device according to claim 2 .

5. The start frame setting unit does not set the start frame if the ball is not detected continuously for a predetermined number of frames or more in the first moving image. The information processing device according to claim 2 .

6. The analysis unit analyzes the third moving image, The extraction range setting unit sets the extraction range based on an analysis result of the third moving image. The information processing device according to claim 1 .

7. The analysis unit detects a trajectory of the ball in the third video; The extraction range setting unit sets the extraction range based on a detection result of the trajectory of the ball in the third video. The information processing device according to claim 6.

8. the analysis unit detects a change point at which the traveling direction of the ball changes by a predetermined angle or more in the third video; The extraction range setting unit sets the extraction range based on a frame in which the transition point is detected. The information processing device according to claim 7.

9. The analysis unit recognizes at least one of a movement and a posture of a player based on the third video; The extraction range setting unit sets the extraction range based on at least one of the recognition results of the player's movement and posture. The information processing device according to claim 6.

10. The analysis unit analyzes audio corresponding to the third video, The extraction range setting unit sets the extraction range based on a result of the analysis of the voice. The information processing device according to claim 1 .

11. The third moving image is generated based on the second moving image. The information processing device according to claim 1 .

12. The predetermined processing is image analysis. The information processing device according to claim 1 .

13. The image analysis is an analysis of the rotation characteristics of the ball. The information processing device according to claim 12.

14. The image analysis is an analysis of the rotation characteristics of a table tennis serve. The information processing device according to claim 13.

15. The analysis unit analyzes the rotation characteristics of the ball. The information processing device according to claim 13.

16. The ball is a table tennis ball The information processing device according to claim 1 .

17. An information processing device, setting an extraction range for extracting the first moving image from the second moving image based on a third moving image having a frame rate lower than that of the second moving image generated by capturing images of a ball game; Detecting a trajectory of a ball in the first video; setting a start frame for starting a predetermined process from among frames of the first video based on a detection result of the trajectory of the ball in the first video; An information processing method comprising:

18. Setting an extraction range for extracting a first video from a second video based on a third video having a lower frame rate than a second video generated by capturing an image of a ball game; and Detecting a trajectory of a ball in the first video; setting a start frame for starting a predetermined process from among frames of the first video based on a detection result of the trajectory of the ball in the first video; A program for causing a computer to execute a process including the steps of:

Images