Moving image reproduction device, program, and moving image reproduction method

Abstract

An information processing device (1) is provided with an acquisition unit (101), a first specification unit (102), a second specification unit (103), and a display control unit (104). The acquisition unit (101) acquires designation information indicating a predetermined posture that is included in a first motion of a user and a second motion to be compared and that is designated by the user. A first specifying unit (102) specifies a first frame corresponding to the specifying information from among a plurality of frames obtained by imaging a first operation. A second specifying unit (103) specifies a second frame corresponding to the specifying information from among a plurality of frames obtained by imaging a second operation. The display control unit (104) reproduces the moving image data corresponding to the first operation and the second operation in parallel so that the reproduction timings of the first frame and the second frame coincide with each other.

Description

Technical Field

[0001] The present disclosure relates to a moving image reproducing device, a program, and a moving image reproducing method. Background Art

[0002] In sports, accurately performing a series of movements can sometimes lead to improved performance. As an example of a series of movements in sports, there can be mentioned the swing of a golf club in golf, the swing of a bat in baseball, or pitching in baseball. In recent years, various technologies have been proposed for filming a series of movements in sports and using the filmed motion images to improve the movements. As an example of a technology used for movement improvement, a disclosed technology is listed in non-patent document 1. Non-patent document 1 discloses an application for smartphones that allows users to compare a motion image of their own swing with a motion image of an ideal swing. In this application, the timing of starting the reproduction of the motion image of the user's swing can be made consistent with the timing of starting the reproduction of the motion image of the ideal swing.

[0003] Prior art literature

[0004] Non-patent literature

[0005] Non-patent document 1: "Golf Vision|プロSupervision-AI ゴルフスイングANALYSIS アプリ", [online], [検検検検検検氪, March 3, 5], インターネット<URL: https: / / www.golf-vision.com / ja / > Summary of the Invention

[0006] Problems to be solved by the invention

[0007] In a golf swing, a player's posture changes continuously from the takeaway, through the apex and impact, to the finish. The takeaway refers to the timing of starting the backswing of the golf club. The apex refers to the timing of starting the downswing of the golf club after the upswing. The impact refers to the timing of the club head striking the golf ball after the downswing. The finish refers to the timing of fully swinging the golf club, completing the swing. When trying to improve a golf swing, it would be convenient to compare the timing of the user's desired posture during the takeaway, apex, impact, and finish with that of an ideal swing, such as that of a professional golfer. However, the time from the apex to the finish, the time from the apex to the impact, and the time from the impact to the finish typically vary from player to player. Therefore, in conventional technology, even if a moving image of a user's swing and a moving image of an ideal swing are reproduced in parallel, it is difficult to compare the user's posture before and after the timing at which the user desires to improve with the posture during the ideal swing.

[0008] In consideration of such circumstances, an object of one aspect of the present disclosure is to provide a technique capable of comparing a user-specified gesture included in a series of actions with a gesture included in an action to be compared.

[0009] Means for solving problems

[0010] A motion image reproduction device according to one embodiment of the present disclosure includes an acquisition unit, a first determination unit, a second determination unit, and a display control unit. The acquisition unit acquires designated information indicating a prescribed posture specified by the user and included in a first action of the user and a second action to be compared with the first action. The first determination unit determines a first frame corresponding to the designated information among a plurality of frames represented by first motion image data obtained by photographing the first action. The second determination unit determines a second frame corresponding to the designated information among a plurality of frames represented by second motion image data obtained by photographing the second action. The reproduction control unit reproduces the first motion image data and the second motion image data in parallel in such a manner that the reproduction timing of the first frame coincides with the reproduction timing of the second frame, thereby displaying a motion image corresponding to the first action and a motion image corresponding to the second action on a display device.

[0011] Further, a program according to an aspect of the present disclosure causes a computer to function as an acquisition unit, a first determination unit, a second determination unit, and a display control unit. The acquisition unit acquires specification information indicating a prescribed posture included in a first action of a user and a second action that becomes a comparison target with the first action and specified by the user. The first determination unit determines a first frame corresponding to the specification information among a plurality of frames indicated by first motion image data obtained by capturing the first action. The second determination unit determines a second frame corresponding to the specification information among a plurality of frames indicated by second motion image data obtained by capturing the second action. The reproduction control unit reproduates the first motion image data and the second motion image data in parallel in a manner that a reproduction timing of the first frame coincides with a reproduction timing of the second frame, and thereby displays a motion image corresponding to the first action and a motion image corresponding to the second action on a display device.

[0012] A motion image reproduction method according to an aspect of the present disclosure includes: acquiring specification information indicating a prescribed posture included in a first action of a user and a second action that becomes a comparison target with the first action and specified by the user; determining a first frame corresponding to the specification information among a plurality of frames indicated by first motion image data obtained by capturing the first action; determining a second frame corresponding to the specification information among a plurality of frames indicated by second motion image data obtained by capturing the second action; and reproduating the first motion image data and the second motion image data in parallel in a manner that a reproduction timing of the first frame coincides with a reproduction timing of the second frame, and thereby displaying a motion image corresponding to the first action and a motion image corresponding to the second action on a display device. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 is a diagram showing an example of a usage situation of an information processing apparatus according to an embodiment.

[0014] Figure 2 is a diagram showing a structure of hardware of an information processing apparatus.

[0015] Figure 3 is a diagram showing an example of user data.

[0016] Figure 4 is a diagram showing an example of target data.

[0017] Figure 5 is a diagram showing an example of a posture table.

[0018] Figure 6 is a functional block diagram of an information processing apparatus.

[0019] Figure 7 This is a diagram showing an example of a UI screen displayed on the display device of the information processing device.

[0020] Figure 8 This is a diagram showing an example of an image indicating the progress status of moving image reproduction.

[0021] Figure 9 This is a diagram showing an example of the first relationship information.

[0022] Figure 10 This is a diagram showing an example of the second relationship information.

[0023] Figure 11 It is a diagram for explaining the operation of the display control unit.

[0024] Figure 12 This is a diagram showing an example of a UI screen displayed on the display device of the information processing device.

[0025] Figure 13 1 is a flowchart showing the flow of processing in the moving image reproducing method executed by the processing device 10 of the moving image reproducing device in accordance with the control program PR1.

[0026] Figure 14 This is a diagram for explaining the operation of the display control unit in the modification (2). DETAILED DESCRIPTION

[0027] Hereinafter, a moving image reproducing apparatus according to an embodiment of the present disclosure will be described with reference to the drawings.

[0028] In addition, in each figure, the size and scale of each part are appropriately different from the actual size and scale. In addition, the embodiment described below is a preferred specific example, and therefore various preferred limitations are added technically. However, as long as there is no special description of the gist of the present invention in the following description, the scope of the present invention is not limited to these embodiments.

[0029] (A: Implementation method)

[0030] Figure 1 This diagram illustrates an example of how an information processing device 1 according to one embodiment of the present disclosure is used. Information processing device 1 has a photography function. For example, information processing device 1 is a smartphone. Information processing device 1 has the function of capturing a golf swing performed by user U and reproducing the captured moving image in response to user instructions. Information processing device 1 is not limited to a smartphone; any device with photography and moving image reproduction functions may also be a tablet terminal or personal computer.

[0031] Figure 2is a diagram showing the structure of hardware in the information processing apparatus 1. As shown in Figure 2 , the information processing apparatus 1 has a processing apparatus 10, a photographing apparatus 11, an operation input apparatus 12, a display apparatus 13, a storage apparatus 14, and a communication apparatus 15.

[0032] The processing apparatus 10 is constituted by a single or a plurality of arithmetic processing circuits such as a CPU (Central Processing Unit), for example, and comprehensively controls each element of the information processing apparatus 1. In addition to the CPU, the processing apparatus 10 can also be constituted by a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), or the like.

[0033] The photographing apparatus 11 is a video camera equipped with a lens, an imaging element, a microphone, and a processing circuit. The imaging element converts light incident via the lens into an electric signal and outputs an image signal to the processing circuit. The microphone converts sound into an electric signal and outputs a sound signal to the processing circuit. The processing circuit generates moving image data based on the image signal and the sound signal. The moving image data contains image data and sound data. In a case where the photographing apparatus 11 is used to photograph a swing of the user U, the moving image data contains image data of the swing and sound data of the swing. In the following description, the moving image data obtained by photographing the swing of the user U is referred to as user data D1. The swing of the user U is an example of the first action in the present disclosure. The user data D1 is an example of the first moving image data in the present disclosure.

[0034] Figure 3 is a diagram showing an example of the user data D1. The user data contains image data composed of a plurality of frames Fr and sound data synchronized with the image data although omitted in Figure 3 . The image data records an image of the swing of the user U per frame Fr. The sound data records sound collected at the time of the swing. In addition, each frame Fr of the user data is associated with a frame number as a consecutive number from a start frame.

[0035] The frame number in the present embodiment is a consecutive number in which the frame number of the start frame is set to 1. Figure 3 (A), (B), (C), and (D) in show poses of each timing of the take-off, the apex, the impact, and the follow-through in a plurality of poses representing characteristics of the swing of golf. In Figure 3 the user data D1 shown in, the frame number of the frame corresponding to the timing (A) of the take-off is 30. In the frame of the frame number 30, a pose of a time point of the take-off in the swing of the user U is photographed. In Figure 3In the user data D1 shown, the frame number of the frame corresponding to the timing (B) of the apex is 50. The posture at the time point of the apex in the swing of the user U is captured in the frame with the frame number of 50. In the user data D1 shown, Figure 3 In the user data D1 shown, the frame number of the frame corresponding to the timing (C) of the hit is 70. The posture at the time point of the hit in the swing of the user U is captured in the frame with the frame number of 70. In the user data D1 shown, Figure 3 In the user data D1 shown, the frame number of the frame corresponding to the timing (D) of the takeback is 90. The posture at the time point of the takeback in the swing of the user U is captured in the frame with the frame number of 90.

[0036] The sound collected at the time of the swing contains the hit sound generated at the time of the hit, that is, the sound generated when the head of the golf club hits the golf ball. The form of the user data D1 in the present embodiment can be a non-compressed form, but can also be a compressed form. The compressed form can be, for example, an MPEG form. The form of the user data D1 can be any form as long as it contains image data continuous in a plurality of frames and sound data synchronized with the image data.

[0037] The operation input device 12 accepts the operation of the user. The operation input device 12 is, for example, a plurality of operation pieces pressed by the user, or a touch panel that detects the contact of the user. The touch panel is overlaid on the image display surface of the display device 13. In addition, the user who operates the operation input device 12 can be the user himself / herself who performed the swing, or another user. The display device 13 contains, for example, a liquid crystal panel or an organic EL panel. The display device 13 displays the moving image of the captured swing, the moving image of the ideal swing that becomes the target of the user U, and the moving image reproduced synchronously with both.

[0038] The communication device 15 communicates with a server or the like via a network under the control of the processing device 10.

[0039] The storage device 14 is, for example, a single or a plurality of memories constituted by a publicly known recording medium such as a magnetic recording medium or a semiconductor recording medium. In addition, the storage device 14 can be constituted by a combination of a plurality of recording media. Furthermore, a removable type recording medium that can be attached to and detached from the information processing device 1, or an external recording medium (for example, an online storage) that the information processing device 1 can communicate with via a network can be utilized as a part of the storage device 14.

[0040] The storage device 14 stores a control program PR1 executed by the processing device 10, various data used by the processing device 10, user data D1, target data D2, and a posture table TBL. Target data D2 is motion image data representing an ideal swing. The ideal swing is the swing that the user U aims for. An example of an ideal swing is the swing of a professional golfer. Like the user data D1, target data D2 includes image data consisting of multiple frames and sound data synchronized with the image data. The image data records an image of the ideal swing for each frame. The sound data records the sound collected during the swing. Each frame in the target data D2 is associated with a frame number as a consecutive number starting from the start frame. In the target data D2, the frame number of the start frame is also 1.

[0041] Figure 4 is a diagram showing an example of target data D2. Figure 3 Likewise, Figure 4 (A), (B), (C) and (D) show the postures at each timing of the start, top, impact and finish. Figure 4 In the target data D2 shown, the frame number corresponding to the timing of the backswing (A) is 25. In the frame number 25, the posture of the subject (hereinafter referred to as the competitor) of the ideal swing at the timing of the backswing is captured. Figure 4 In the target data D2 shown, the frame number corresponding to the timing of the top (B) is 45. The frame number 45 captures the athlete's posture at the time of the top in the ideal swing. Figure 4 In the target data D2 shown, the frame number corresponding to the timing of impact (C) is 45. The frame number 45 captures the posture of the competitor at the time of impact in the ideal swing. Figure 4 In the target data D2 shown, the frame number corresponding to the finish timing (D) is 55. The frame number 55 captures the athlete's posture at the finish timing of an ideal swing.

[0042] Target data D2 can be obtained by downloading from a server connected to a network, or by photographing a professional golfer's swing using the camera 11. An ideal swing is an example of a second motion to be compared with the first motion, and target data D2 is an example of second moving image data in the present disclosure.

[0043] Figure 5 : is a diagram showing an example of the posture table TBL. Figure 5As shown, in the posture table TBL, identifiers uniquely indicating each of a plurality of postures representing characteristics of a golf swing are associated with each other, and feature data representing the characteristics of the skeleton of the action subject M in the posture represented by the identifier is pre-stored. Figure 5 In the example shown, the posture table TBL stores sets of identifiers and feature data for postures at four timings of takeback, top, downswing, impact, follow-through, and finish.

[0044] The feature data associated with the identifier and stored in the posture table TBL is data representing the three-dimensional coordinates of each joint J of the action subject M in the posture indicated by the corresponding identifier. The three-dimensional coordinate system in this embodiment is a coordinate system with the middle position of the two feet of the action subject M as the origin and the distance between the two feet as 1. As examples of joints J, there are 8 joints in total, namely, the left knee, right knee, left hip joint, right hip joint, left shoulder joint, right shoulder joint, left elbow and right elbow. Figure 6 In FIG, each joint J is represented by a black circle when the right-handed action subject M is viewed from the front (east when the hitting direction is set to north). Figure 5 In FIG, the positions of the joints J at the respective timings of the take-off, backswing, top, downswing, impact, follow-through, and finish are sequentially exemplified as (1) to (7).

[0045] Figure 5 The feature data A in is data representing the three-dimensional coordinates of each joint J in the posture of the moving subject M at the time of lifting the pole, that is, a collection of eight coordinate data. Figure 5 The feature data T in is data representing the three-dimensional coordinates of each joint J in the posture of the action subject M at the time of the vertex. Figure 5 The feature data I in is data indicating the three-dimensional coordinates of each joint J in the posture of the actor M at the time of the strike. Figure 5 The feature data F in is data indicating the three-dimensional coordinates of each joint J in the posture of the moving subject M at the time of finishing the club.

[0046] Generally speaking, a golf swing differs depending on the user U's gender, dominant hand, generation (age), and golf proficiency (professional, semi-professional, amateur), among other attributes. The feature data stored in the posture table TBL can also be generated by statistical processing, such as averaging, across multiple feature data sets representing the same posture with different attributes. Furthermore, multiple types of posture tables TBL can be prepared based on these attributes. In this embodiment, the posture table TBL is stored in the storage device 14. However, the posture table TBL may also be stored in a cloud server accessible to the information processing device 1 via telecommunication lines such as the Internet.

[0047] Figure 6 This is a block diagram illustrating the functional configuration of the processing device 10. When a user instructs execution of the control program PR1 by operating the operation input device 12, the processing device 10 reads the control program PR1 from the storage device 14. By executing the read control program PR1, the processing device 10 functions as an acquisition unit 101, a first determination unit 102, a second determination unit 103, and a display control unit 104. By executing the control program PR1, the processing device 10 synchronizes and reproduces the moving image represented by the user data D1 and the moving image represented by the target data D2 at the timing specified by the user.

[0048] The acquisition unit 101 acquires information indicating a predetermined posture designated by the user U from among multiple postures commonly included in the user U's swing and the swing being compared (in this embodiment, the ideal swing). The multiple postures in this embodiment are postures of the main movement at each timing of the takeaway, top, impact, and finish. In this embodiment, the acquisition unit 101 displays a UI screen on the display device 13 for concurrently reproducing the motion image represented by the user data D1 and the motion image represented by the target data D2.

[0049] Figure 7 FIG is a diagram showing an example of the UI screen. Figure 7 As shown, the UI screen includes areas A1 to A4 and virtual operating elements B1 to B9. Area A1 is a display area for displaying the moving image represented by user data D1. Area A2 is a display area for displaying the moving image represented by target data D2. Area A3 displays an image showing the progress of playback of the moving image represented by user data D1. Area A4 displays an image showing the progress of playback of the moving image represented by target data D2.

[0050] Figure 8 This is a diagram showing an example of an image G1 indicating the progress status of the playback of a moving image. Figure 8 The line segment L1 in represents the length of time from the start frame S to the end frame E of the moving image. Figure 8Each time a frame of a moving image is reproduced, the right end of the line segment L2 in FIG. extends to the right, indicating the current reproduction position in the moving image. Figure 11 In the user data D1, frames corresponding to the respective timings of the drive, the top, the impact, and the finish are represented by characters A, T, I, and F, respectively. Figure 8 The four black circle marks in correspond to the timings of the take-off, apex, strike, and finish from left to right. The time Tat from the take-off to the apex is displayed between the black circle corresponding to the take-off and the black circle corresponding to the apex. The time Tti from the apex to the strike is displayed between the black circle corresponding to the apex and the black circle corresponding to the strike. The time Tif from the strike to the finish is displayed between the black circle corresponding to the strike and the black circle corresponding to the finish. These times Tat, Tti, and Tif are calculated based on the number of frames between the take-off and the apex, the number of frames between the apex and the strike, and the number of frames between the strike and the finish. The same applies to the target data D2. Among these four black circle marks, the black circle mark corresponding to the timing of the posture specified by the user U is displayed larger than the other three black circle marks. In Figure 8 , a case where the hitting gesture is designated by the user U is illustrated.

[0051] Figure 7 The virtual operating elements B1 to B4 shown are virtual buttons that allow the user U to specify a desired posture for each timing of the take-off, top, impact, and finish. Virtual operating element B6 is a virtual button that allows the user to instruct the start of motion image playback. Virtual operating element B5 is a virtual button that allows the user to instruct the reverse playback of the motion image. Virtual operating element B7 is a virtual button that allows the user to instruct the fast-forward playback of the motion image. Virtual operating element B8 is a virtual button that allows the user to instruct the playback of the motion image at standard speed, and virtual operating element B9 is a virtual button that allows the user to instruct the slow playback of the motion image. Playing back the motion image at standard speed means, for example, playing back each frame at 1 / 60 second intervals for motion image data with a frame interval of 1 / 60 second. Playing back the motion image data with a frame interval of 1 / 60 second at slow speed means playing back each frame at 1 / 10 second intervals.

[0052] For example, if the desired gesture of the user U is a gesture at the time of striking, the user U can designate the gesture at the time of striking by performing an input operation of clicking the virtual operating element B3 on the operation input device 12. The acquisition unit 101 acquires designation information based on the operation performed on the operation input device 12. When the input operation of clicking the virtual operating element B3 is performed on the operation input device 12, the acquisition unit 101 acquires designation information indicating a strike.

[0053] The first identification unit 102 identifies the first frame corresponding to the designated information from among the multiple frames represented by the user data D1. More specifically, the first identification unit 102 first performs skeleton estimation processing on the subject (in this embodiment, the user U) for each of the multiple frames represented by the user data D1. This process determines the three-dimensional coordinates of each of the subject's eight joints J for each frame. The skeleton estimation process can be performed using conventional techniques as appropriate.

[0054] Next, for each of the multiple frames represented by the user data D1, the first identification unit 102 compares the three-dimensional coordinates of the subject's eight joints J with the three-dimensional coordinates of the eight joints J represented by the feature data stored in the posture table TBL, for each joint J. This identifies the frame corresponding to each timing of the take-off, top, impact, and finish. For example, if the target frame is a frame corresponding to the impact, the first identification unit 102 calculates the distance between the three-dimensional coordinates represented by the feature data I and the three-dimensional coordinates determined by the skeleton estimation process for each of the eight joints J, and identifies the frame with the smallest sum of these distances for the eight joints J as the frame corresponding to the impact.

[0055] Then, if Figure 9 As shown, the first determination unit 102 generates first relationship information D3 by associating the identifiers representing each of the take-off, top, impact, and finish with the frame numbers of the corresponding frames. The first relationship information D3 is an example of the first information. The first determination unit 102 determines the first frame based on the designated information and the first relationship information D3. For example, when the designated information indicates impact, Figure 9 In the example shown, the frame with frame number 70 is determined as the first frame.

[0056] The second determination unit 103 determines the second frame corresponding to the designated information among the plurality of frames represented by the target data D2. The second determination unit 103, like the first determination unit 102, compares the three-dimensional coordinates of each joint J of the subject captured in each of the plurality of frames represented by the target data D2 with the three-dimensional position of each joint J represented by each feature data stored in the posture table TBL, thereby determining the second frame corresponding to the designated information. Figure 10 As shown, the second relationship information D4 is generated. The second relationship information D4 is data that associates the identifiers representing each of the take-off, top, impact, and finish with the frame number of the corresponding target data D2 frame. Then, the second determination unit 103 determines the second frame based on the designated information and the second relationship information D4. For example, in the case where the designated information indicates impact, Figure 10 In the example shown, the frame with the frame number 45 is determined as the second frame. The second relationship information D4 is an example of the second information.

[0057] When an input operation is performed on the virtual operating element B6 on the UI screen, the display control unit 104 reproduces the user data D1 and the target data D2 in parallel so that the reproduction timing of the first frame coincides with the reproduction timing of the second frame, thereby displaying a motion image of the swing of the user U and a motion image of the ideal swing on the display device 13. In this embodiment, the display control unit 104 delays the timing of starting to read the motion image data of either the user data D1 or the target data D2 from the storage device 14 relative to the timing of starting to read the other motion image data from the storage device 14, thereby aligning the reproduction timing of the first frame with the reproduction timing of the second frame.

[0058] Figure 11 1 is a diagram showing an example of the operation of the display control unit 104. Figure 11 In, with Figure 8 Similarly, the frames corresponding to the timings of the take-off, top, impact, and finish are represented by the letters A, T, I, and F. For example, suppose that the user U specifies the posture during impact, and the frame number of the first frame is "70" and the frame number of the second frame is "45". In this case, Figure 11 As shown, the display control unit 104 delays the start timing of the reproduction of the target data D2 by a time ΔT corresponding to the difference in frame numbers (70-45) relative to the start timing of the reproduction of the user data D1, and reproduces the two data in parallel. Figure 12 As shown, the timing of the impact is consistent in the moving image of the swing of the user U displayed in the area A1 and the moving image of the ideal swing displayed in the area A2.

[0059] Furthermore, in area A1, 0.7 seconds is displayed as the time from the start to the top, Tat; 0.6 seconds is displayed as the time from the top to the impact, Tti; and 0.5 seconds is displayed as the time from the impact to the finish, Tif. Meanwhile, in area A2, 0.6 seconds is displayed as the time from the start to the top, Tat; 0.5 seconds is displayed as the time from the top to the impact, Tti; and 0.7 seconds is displayed as the time from the impact to the finish, Tif. Thus, the various times in the user U's swing and the various times in the ideal swing are displayed side by side. This allows the user U to compare their swing with the ideal swing and at a glance identify which parts are shorter and which parts are longer.

[0060] Furthermore, the processing device 10 operating in accordance with the control program PR1 executes a moving image reproduction method that significantly illustrates the characteristics of the present disclosure. Figure 13 FIG. 1 is a flowchart showing the flow of processing in the moving image reproduction method. Figure 13As shown, this moving image reproduction method includes acquisition processing SA110, first determination processing SA120, second determination processing SA130, and display control processing SA140. The processing contents of each of the acquisition processing SA110, first determination processing SA120, second determination processing SA130, and display control processing SA140 are as follows.

[0061] In acquisition process SA110, processing device 10 functions as acquisition unit 101. In acquisition process SA110, processing device 10 acquires designation information indicating a predetermined posture designated by user U from among a plurality of postures commonly included in the user U's swing and the ideal swing.

[0062] In the first identification process SA120, the processing device 10 functions as the first identification unit 102. In the first identification process SA120, the processing device 10 identifies the first frame corresponding to the designated information among the multiple frames represented by the user data D1 based on the skeleton of the user U captured in the multiple frames. The first frame is the frame of the user data D1 corresponding to the predetermined posture represented by the designated information.

[0063] In the second determination process SA130, the processing device 10 functions as the second determination unit 103. In the second determination process SA130, the processing device 10 determines the second frame corresponding to the designated information among the multiple frames represented by the target data D2 based on the competitor's skeleton captured in the multiple frames. The second frame is the frame of the target data D2 corresponding to the predetermined posture represented by the designated information. In this embodiment, the first determination process SA120 is executed before the second determination process SA130. However, the order of executing the first determination process SA120 and the second determination process SA130 may be reversed.

[0064] In display control process SA140, processing device 10 functions as display control unit 104. In display control process SA140, processing device 10 reproduces user data D1 and target data D2 in parallel so that the reproduction timing of the first frame coincides with the reproduction timing of the second frame, thereby displaying a motion image of the user U's swing and a motion image of an ideal swing on display device 13.

[0065] As described above, according to this embodiment, the user U synchronizes the motion image of their own swing with the motion image of an ideal swing at the timing specified by the user U during the takeaway, top, impact, and finish. Therefore, the user U can compare their own posture with that of the ideal swing before and after the specified timing, correct their own posture, and improve their swing.

[0066] Further, the display control section 140 in the present embodiment starts reproduction from the start frame (i.e., the frame of frame number 1) for each of the user data Dl and the target data D2. However, the frame at which the reproduction of each of the user data Dl and the target data D2 is started can be any frame before the frame corresponding to the posture indicated by the specification information, and there is no limitation on the start frame. Further, the frame at which the reproduction of each of the data Dl and the target data D2 is started can be specified by the user U. For example, it is assumed that the frame corresponding to the timing of the apex is specified by the user U as the frame at which the reproduction of each of the user data Dl and the target data D2 is started in the case where the specification information indicates a punch. In this case, the display control section 140 calculates the difference between the time difference between the apex and the punch in the moving image indicated by the user data Dl (hereinafter, first time difference) and the time difference between the apex and the punch in the moving image indicated by the target data D2 (hereinafter, second time difference). Then, the display control section 140 delays the timing at which the moving image data of one of the user data Dl and the target data D2 is read out from the storage device 14 with respect to the timing at which the moving image data of the other is read out from the storage device 14, by an amount corresponding to the time indicated by the calculated difference (first time difference - second time difference). For example, if the calculated difference is a positive value, the display control section 140 delays the timing at which the target data D2 is read out from the storage device 14 with respect to the timing at which the user data Dl is read out from the storage device 14, by an amount corresponding to the time indicated by the difference. If the calculated difference is a negative value, the display control section 140 delays the timing at which the user data D2 is read out from the storage device 14 with respect to the timing at which the target data Dl is read out from the storage device 14, by an amount corresponding to the time indicated by the absolute value of the difference.

[0067] (B: Modification)

[0068] The present disclosure is not limited to the above-described embodiments. Specific modification modes are as follows. Any two or more modes specified below can be combined. (B-1: Modification Example 1)

[0069] Although the moving subject in the moving image indicated by the first moving image data and the moving subject in the moving image indicated by the second moving image data are different from each other in the above-described embodiments, they can be the same. For example, the first moving image data can be moving image data obtained by capturing the swing of the user U before the improvement of the swing, and the second moving image data can be moving image data obtained by capturing the swing of the user U after the improvement of the swing. According to this mode, the user U can visually recognize the improvement result of the swing.

[0070] (B-2: Modification Example 2)

[0071] In the above embodiment, user data D1 and target data D2 are both motion image data obtained by filming a golf swing. However, user data D1 and target data D2 can also be motion image data obtained by filming a baseball bat swing or a baseball pitch, that is, motion image data obtained by filming a series of movements in a sport other than golf. Furthermore, user data D1 and target data D2 are not limited to motion image data obtained by filming a series of movements in a sport; any motion image data obtained by filming movements involving multiple poses, such as yoga, dance, or pantomime, can be used. Furthermore, the present disclosure is not limited to sports. For example, it can also be applied to assembly work in a factory or to knife handling in cooking.

[0072] (B-3: Modification 3)

[0073] In the above-described embodiment, the information processing device 1 includes a photography function for generating user data D1. However, user data D1 can also be obtained by downloading via a communication line, in which case the information processing device 1 does not need to include a photography function. In other words, the photography device 11 in the above-described embodiment is not an essential component of the moving image reproduction device of the present disclosure and can be omitted.

[0074] (B-4: Modification 4)

[0075] The display control unit 104 reproduces the motion image represented by the user data D1 and the motion image represented by the target data D2 in synchronization with each other at the timing specified by the user by making the position at which the motion image data of one of the user data D1 and the target data D2 is started to be read from the storage device 14 different from the position at which the motion image data of the other is started to be read from the storage device 14. For example, it is assumed that both the user data D1 and the target data D2 are motion image data obtained by photographing a golf swing, and the user specifies the timing of the hit. Furthermore, it is assumed that the frame number of the frame corresponding to the hit in the motion image represented by the user data D1 is 70, and the frame number of the frame corresponding to the hit in the motion image represented by the target data D2 is 45. In this case, as Figure 14 As shown, the display control unit 104 may reproduce the moving image represented by the target data D2 from the start frame S, and may reproduce the moving image represented by the user data D1 from the frame S1 at a time point corresponding to ΔT having elapsed from the start frame S.

[0076] (B-5: Modification 5)

[0077] In the above embodiment, frames corresponding to the timings of the backswing, top, impact, and finish are determined based on the skeletal features of the subject in the moving image. However, if the swinging subject makes a sound "1" at the backswing, "2" at the top, "3" at impact, and "4" at the finish, the frames corresponding to the timings can also be determined based on the sound of the swinging subject.

[0078] Furthermore, in a golf swing, a hitting sound is produced at the moment of impact. Thus, if the user data D1 and target data D2 are moving image data obtained by capturing a series of movements during exercise that produce a predetermined sound at a timing corresponding to a predetermined posture, and if the posture at the timing of producing the sound is designated as the predetermined posture, the first and second frames can also be identified using the sound.

[0079] For example, assume that user data D1 is moving image data obtained by filming a golf swing of user U, and the timing of a golf swing is specified by user U. In this case, if the height of user U is pre-input into information processing device 1, the first frame can be more accurately identified by utilizing the hitting sound produced during the golf swing compared to the above-described embodiment.

[0080] Specifically, the first determination unit 102 calculates the distance DL between the imaging device 11 and the user U based on the height input by the user U, the number of pixels representing the height of the user U captured in each frame represented by the user data D1, and the internal parameters (focal length and magnification) of the imaging device 11. Alternatively, instead of using the internal parameters, a table may be used that associates the height of the subject with the subject's height (number of pixels) in the image captured by the imaging device 11 for each distance between the subject and the imaging device 11.

[0081] Then, the first determination unit 102 determines the frame corresponding to the moment DL / VC is traced back from the frame corresponding to the moment the striking sound was recorded as the first frame. In addition, VC is the speed of sound. Similarly, the second determination unit 103 determines the second frame based on the height of the competitor performing the ideal swing input by user U, the number of pixels representing the competitor's height captured in the motion image represented by target data D2, and the striking sound in target data D2. According to this method, by using the sound generated at the timing of the gesture specified by user U, the first and second frames can be accurately determined.

[0082] In Modification 5, although the height of the user U is input, the height of a typical user may be stored in advance in the storage device 14, and the distance from the information processing device 1 to the golf finish line may be calculated using the typical height. Furthermore, the information processing device 1 may estimate the distance between joints based on internal parameters and the user data D1, and calculate the height of the user U based on the estimated distance between joints.

[0083] (B-6: Modification 6)

[0084] The moving image reproduction device disclosed herein may also include: a generation unit that applies a predetermined effect to user data D1 based on the first frame in user data D1, thereby generating new moving image data; and an output unit that outputs the new moving image data to an external device. When user data D1 is moving image data obtained by capturing a golf swing of user U, and the user specifies the timing of the impact, an example of the predetermined effect may include replacing the impact sound with a gunshot. Other examples of effects applied based on the first frame include applying cheers from the audience to frames several frames after the first frame, superimposing an image of fireworks launching into the sky, an image imitating the trajectory of a shot, or an image representing a numerical value of the projected flight distance. A specific example of an external device may be a server device that provides a moving image distribution service. According to this embodiment, a user can generate a new moving image with a predetermined effect applied based on the timing of a specified posture, and can output moving image data representing the new moving image to an external device. Furthermore, the generation unit may crop a portion of the new moving image generated by applying the effect, and the output unit may output (upload) the image to a social media server device.

[0085] Furthermore, the application of effects is not limited to the first frame; new motion image data can also be generated by manually editing user data D1 to change the posture of the subject in user data D1. Furthermore, the generation unit can also generate new motion image data by deforming the image of each part of the user's body, such as by enlarging or reducing it, and pasting it as a texture on each part of the 3D model of the human body corresponding to the skeleton extracted from each frame in target data D2. New motion image data generated by applying predetermined effects, etc., is an example of the third motion image data in this disclosure.

[0086] (B-7: Modification 7)

[0087] In the above embodiment, the acquisition unit 101, first determination unit 102, second determination unit 103, and display control unit 104 are provided in a smartphone. However, any one, any two, any three, or all of the first determination unit 102, second determination unit 103, and display control unit 104 may be provided in a server device that communicates with the smartphone via a telecommunications line such as the Internet.

[0088] (B-8: Modification 8)

[0089] In the above embodiment, the acquisition unit 101, first determination unit 102, second determination unit 103, and display control unit 104 are all software modules. However, any one, any two, any three, any four, or all of the acquisition unit 101, first determination unit 102, second determination unit 103, and display control unit 104 may be hardware modules. Specific examples of hardware modules include DSPs (Digital Signal Processors), ASICs (Application Specific Integrated Circuits), PLDs (Programmable Logic Devices), and FPGAs (Field Programmable Gate Arrays). Even if at least one of the acquisition unit 101, first determination unit 102, second determination unit 103, and display control unit 104 is a hardware module, the same effects as those of the above embodiment can be achieved.

[0090] (B-9: Modification 9)

[0091] The control program PR1 may be manufactured or sold as a single unit. When selling the control program PR1, the control program PR1 may be provided to the purchase destination by writing the control program PR1 on a computer-readable recording medium such as a flash ROM and distributing it, or by downloading the control program PR1 via a telecommunications line.

[0092] (C: A method according to the above method or a modified example)

[0093] From the above description, for example, the following preferred embodiments of the present disclosure can be understood.

[0094] A motion image reproduction device according to one embodiment (first embodiment) of the present disclosure includes an acquisition unit, a first determination unit, a second determination unit, and a display control unit. The acquisition unit acquires designation information indicating a predetermined posture specified by the user and included in a first action of the user and a second action to be compared with the first action. The first determination unit determines a first frame corresponding to the designation information from among a plurality of frames represented by first motion image data obtained by capturing the first action. The second determination unit determines a second frame corresponding to the designation information from among a plurality of frames represented by second motion image data obtained by capturing the second action. The display control unit reproduces the first motion image data and the second motion image data in parallel, aligning the reproduction timing of the first frame with the reproduction timing of the second frame, thereby displaying a motion image corresponding to the first action and a motion image corresponding to the second action on a display device. According to the first embodiment, the reproduction timing of the gesture specified by the user can be synchronized to reproduce the motion image of the first action and the motion image of the second action.

[0095] In a motion image reproduction device according to a second embodiment of the first embodiment, the first motion image data and the second motion image data are stored in a storage device. A display control unit in the motion image reproduction device according to the second embodiment delays the timing of starting to read the motion image data of one of the first motion image data and the second motion image data from the storage device relative to the timing of starting to read the other motion image data from the storage device. According to the second embodiment, by offsetting the timing of starting to reproduce the first motion image data and the timing of starting to reproduce the second motion image data, it is possible to synchronize the reproduction timing of the gesture specified by the user and reproduce the motion image of the first action and the motion image of the second action.

[0096] In a moving image reproduction device according to a third aspect specifically related to the first aspect, the first moving image data and the second moving image data are stored in a storage device. In the moving image reproduction device according to the second aspect, the display control unit causes the position at which one of the first moving image data and the second moving image data is read from the storage device to be different from the position at which the other moving image data is read from the storage device. According to the third aspect, by offsetting the position at which the first moving image data is reproduced from the position at which the second moving image data is reproduced, the timing of the reproduction of the gesture specified by the user can be synchronized, thereby reproducing the moving image of the first action and the moving image of the second action.

[0097] In a moving image reproduction device according to a fourth aspect, which is a specific embodiment of the first aspect, an acquisition unit receives an instruction from the user specifying any one of a plurality of postures characteristic of the first and second actions, and acquires the designation information based on the received instruction. According to the fourth aspect, the timing of reproducing the user-specified posture among the plurality of postures included in the first and second actions can be synchronized to reproduce a moving image of the first and second actions.

[0098] In a moving image reproduction device according to a fifth embodiment, a first determination unit generates, based on the first moving image data, first information indicating a correspondence between a plurality of frames represented by the first moving image data and a plurality of postures included in the first action, and determines the first frame based on the generated first information. In a moving image reproduction device according to the fifth embodiment, a second determination unit generates, based on the second moving image data, second information indicating a correspondence between a plurality of frames represented by the second moving image data and a plurality of postures included in the second action, and determines the second frame based on the generated second information. According to the fifth embodiment, the first and second frames can be determined based on the information indicating the correspondence between the frames and the postures.

[0099] In a specific sixth aspect of the fifth aspect, the first action and the second action are a series of actions in a predetermined sport. The first determination unit in the motion image reproduction device according to the sixth aspect generates the first information based on the skeleton of the user estimated from each of the plurality of frames represented by the first motion image data. The second determination unit in the motion image reproduction device according to the sixth aspect generates the second information based on the skeleton of the competitor in the sport estimated from each of the plurality of frames represented by the second motion image data. According to the sixth aspect, the reproduction timing of the postures specified by the user can be synchronized, thereby reproducing a motion image representing a series of actions of the user in the predetermined sport and a motion image representing a series of actions of the competitor to be compared.

[0100] In the specific seventh aspect of the sixth aspect, in the prescribed motion, a prescribed sound is generated at the timing of the prescribed posture. The first motion image data and the second motion image data in the seventh aspect include the prescribed sound. The first determination unit in the motion image reproduction device involved in the seventh aspect determines the first frame based on the skeleton of the user and the prescribed sound in the first motion image data. The second determination unit in the motion image reproduction device involved in the seventh aspect determines the second frame based on the skeleton of the competitor and the prescribed sound in the second motion image data. According to the seventh aspect, the first frame and the second frame can be accurately determined by using the sounds generated at the timing of the user-specified posture.

[0101] A moving image reproduction device according to an eighth aspect, specifically the first aspect, further includes a generation unit and an output unit. The generation unit applies a predetermined effect to the first moving image data based on the first frame in the first moving image data, thereby generating third moving image data. The output unit outputs the third moving image data to an external device. According to the eighth aspect, a moving image can be generated that has the predetermined effect applied at a timing corresponding to a user-specified gesture.

[0102] A program according to one aspect (ninth aspect) of the present disclosure causes a computer to function as an acquisition unit, a first determination unit, a second determination unit, and a display control unit. The acquisition unit acquires designation information indicating a predetermined user-specified gesture included in a first user action and a second action to be compared with the first action. The first determination unit determines a first frame corresponding to the designation information from among a plurality of frames represented by first moving image data obtained by capturing the first action. The second determination unit determines a second frame corresponding to the designation information from among a plurality of frames represented by second moving image data obtained by capturing the second action. The display control unit displays a moving image corresponding to the first action and a moving image corresponding to the second action on a display device by reproducing the first and second moving image data in parallel such that the reproduction timing of the first frame coincides with the reproduction timing of the second frame. According to the ninth aspect, similar to the first aspect, the reproduction timing of the user-specified gesture can be synchronized to reproduce the moving image of the first and second actions.

[0103] A moving image reproduction method according to one aspect (a tenth aspect) of the present disclosure includes: obtaining designation information indicating a predetermined gesture specified by the user and included in a first user action and a second action to be compared with the first action; identifying a first frame corresponding to the designation information among a plurality of frames represented by first moving image data obtained by capturing the first action; identifying a second frame corresponding to the designation information among a plurality of frames represented by second moving image data obtained by capturing the second action; and displaying a moving image corresponding to the first action and a moving image corresponding to the second action on a display device by reproducing the first and second moving image data in parallel such that the reproduction timing of the first frame coincides with the reproduction timing of the second frame. According to the tenth aspect, similar to the first and ninth aspects, the reproduction timing of the gesture specified by the user can be synchronized to reproduce the moving image of the first and second actions.

[0104] Description of Reference Numerals

[0105] 1: information processing apparatus, 10: processing apparatus, 11: imaging device, 12: operation input device, 13: display device, 14: storage device, 15: communication device, 101: acquisition section, 102: first determination section, 103: second determination section, 104: display control section, PR1: control program, D1: user data, D2: target data, D3: first relationship information, D4: second relationship information, TBL: posture table.

Claims

1. A moving image reproduction device comprising: an acquisition unit that acquires designation information indicating a predetermined gesture specified by the user and included in a first motion of the user and a second motion to be compared with the first motion; a first specifying unit for specifying a first frame corresponding to the designated information among a plurality of frames represented by first moving image data obtained by capturing the first action; a second determining unit for determining a second frame corresponding to the designated information among a plurality of frames represented by second moving image data obtained by capturing the second action; as well as The display control unit displays a motion image corresponding to the first action and a motion image corresponding to the second action on a display device by reproducing the first motion image data and the second motion image data in parallel in a manner such that the reproduction timing of the first frame coincides with the reproduction timing of the second frame.

2. The moving image reproduction device according to claim 1, wherein: The first motion image data and the second motion image data are stored in a storage device, The display control unit delays a timing of starting to read out one of the first moving image data and the second moving image data from the storage device relative to a timing of starting to read out the other moving image data from the storage device.

3. The moving image reproducing apparatus according to claim 1, wherein: The first motion image data and the second motion image data are stored in a storage device, The display control unit makes a position at which one of the first moving image data and the second moving image data is started to be read from the storage device different from a position at which the other moving image data is started to be read from the storage device.

4. The moving image reproducing apparatus according to claim 1, wherein: The acquisition unit receives an instruction from the user specifying any one of a plurality of gestures that characterize the first and second motions, and acquires the specified information based on the received instruction.

5. The moving image reproducing apparatus according to claim 1, wherein: The first determination unit generates first information based on the first moving image data, and determines the first frame based on the generated first information, the first information indicating correspondence between a plurality of frames represented by the first moving image data and the plurality of postures included in the first action. The second specifying unit generates second information based on the second moving image data, indicating correspondence between a plurality of frames represented by the second moving image data and the plurality of gestures included in the second action, and specifies the second frame based on the generated second information.

6. The moving image reproducing apparatus according to claim 5, wherein: The first action and the second action are a series of actions in a predetermined movement. The first specifying unit generates the first information based on the skeleton of the user estimated from each of a plurality of frames represented by the first moving image data. The second specifying unit generates the second information based on a skeleton of the competitor involved in the sport estimated from each of a plurality of frames represented by the second moving image data.

7. The moving image reproducing apparatus according to claim 6, wherein: During the predetermined movement, a predetermined sound is generated at the timing of the predetermined posture. The first moving image data and the second moving image data include the predetermined sound. The first specifying unit specifies the first frame based on the skeleton of the user and the predetermined sound in the first moving image data. The second identification unit identifies the second frame based on the skeleton of the competitor and the predetermined sound in the second moving image data.

8. The moving image reproduction device according to claim 1, comprising: a generating unit that generates third moving image data by applying a predetermined effect to the first moving image data based on the first frame in the first moving image data; and The output unit outputs the third moving image data to an external device.

9. A program that causes a computer to function as the following functional unit: an acquisition unit that acquires designation information indicating a predetermined gesture specified by the user and included in a first motion of the user and a second motion to be compared with the first motion; a first specifying unit for specifying a first frame corresponding to the designated information among a plurality of frames represented by first moving image data obtained by capturing the first action; a second determining unit for determining a second frame corresponding to the designated information among a plurality of frames represented by second moving image data obtained by capturing the second action; as well as The display control unit displays a motion image corresponding to the first action and a motion image corresponding to the second action on a display device by reproducing the first motion image data and the second motion image data in parallel in a manner such that the reproduction timing of the first frame coincides with the reproduction timing of the second frame.

10. A method for reproducing a moving image, comprising: acquiring designation information indicating a predetermined gesture included in a first motion of the user and a second motion to be compared with the first motion and designated by the user; determining a first frame corresponding to the designated information among a plurality of frames represented by first moving image data obtained by capturing the first action; determining a second frame corresponding to the designated information among a plurality of frames represented by second moving image data obtained by capturing the second action; as well as By reproducing the first and second moving image data in parallel so that the reproduction timing of the first frame coincides with the reproduction timing of the second frame, a moving image corresponding to the first action and a moving image corresponding to the second action are displayed on a display device.

Images