A reference point switching type dynamic synchronous playback system and method for time-series media data of repetitive motion.

Abstract

When comparing and analyzing multiple recordings of repetitive movements, there is no means to dynamically switch the reference point for synchronized playback. Therefore, it was difficult to recognize the discrepancies between movements that appeared different depending on the selection of the reference point, and to identify their causes. [Solution] The system accepts multiple time-series media data, each recording multiple repetitive actions, with multiple reference point candidates assigned to each recording. It displays these data in parallel or superimposed, and presents the reference point candidates on a timeline. For each type of reference point candidate, the system calculates the median time position across multiple recordings as the representative position, and sets intervals on the timeline using the midpoint between adjacent representative positions as boundaries. The system designates the reference point candidate corresponding to the interval to which the user's input location belongs as the reference point, and realigns all recordings to the time position of that reference point to update synchronized playback.

Description

Technical Field

[0003]

[0001] The present invention relates to a system and method for comparative analysis of time-series media data of a plurality of repetitive operations by synchronous playback. More specifically, in time-series media data of repetitive operations recorded multiple times, a plurality of reference point candidates corresponding to characteristic events occurring during the operation are assigned to each record, and from among them, the user designates a reference point by an operation input on the timeline, and the present invention relates to a technique for dynamically realigning and synchronously playing back all records at the time position of the reference point.

Background Art

[0002] In the fields of sports science and biomechanics, it is widely practiced to take a plurality of shots of repetitive operations such as pitching, hitting, and swinging, and perform comparative analysis by arranging and synchronously playing back those videos. In this comparative analysis, when synchronously playing back a plurality of videos in time alignment, some reference point is determined, and each video is made to coincide (aligned) at the time position of the reference point. Here, in one repetitive operation, there usually exist a plurality of points that can be characteristic reference points. For example, in the case of a baseball pitching motion, the start point of the form (set position), the ground contact point of the front foot (foot plant), the ball release point, etc. can each be a reference point for alignment. Similarly, in the case of a golf swing, the address (stance), the apex of the takeback (top), the moment of impact from the downswing, the end of the follow-through, etc. are candidates for reference points. Depending on which reference point is used to align the time positions for synchronous playback, the operation characteristics that emerge from the comparison are fundamentally different.

[0003] For example, in baseball pitching, synchronized playback aligned at the release point visualizes differences in take-back timing, while synchronized playback aligned at the foot plant visualizes differences in trunk rotation and arm swing from that point onward. In golf swings, aligning at impact reveals differences in downswing trajectory, while aligning at the top reveals differences in backswing tempo. Non-patent document 1 (Honert & Pataky, 2021) compares 40 different gait events as alignment reference points in gait analysis and demonstrates that the selection of reference points has a significant impact on the statistical analysis results.

[0004] Applying the findings from Non-Patent Document 1, for example, the following situation may occur. Consider a scenario in which two types of pitches, fastball A and fastball B, are compared and analyzed. When the data is aligned and synchronized playback is performed using the footplant (the point at which the lead foot makes contact with the ground) as the reference point, it is observed that at the same time, the arm has already been fully extended in fastball A, while in fastball B, the arm has not yet been fully extended. From this synchronized playback, the conclusion is drawn that "fastball B has a slow arm swing and a problem with its form." However, when the data is realigned using the release point (the moment the ball leaves the hand) as the reference point, it can be seen that the position of the arm and the use of the torso just before release are almost the same for both fastball A and fastball B. In other words, there is no difference in the form itself; only the time taken from the footplant to the release (tempo) is different. Thus, by simply changing the reference point for the same data, the conclusion obtained from the analysis can fundamentally change from "a problem with form" to "a problem with tempo."

[0005] While some conventional repetitive motion analysis systems partially offer multiple synchronization methods and display configurations, they do not disclose a function that allows the user to dynamically switch reference points and synchronize playback of multiple recordings of a repetitive motion. Patent Document 1 (Blast Motion, US9349049B2) discloses a motion analysis system centered on motion capture using an IMU sensor and a mobile device, and has multi-format display functions such as 3D overlay, stroboscopic effect, and Bullet Time display, as well as multiple reference frames (world coordinate system, equipment part, virtual spine, bone landmark, etc.). However, it does not disclose a function to dynamically realign multiple recordings of a repetitive motion at different reference points and play them back in sync.

[0006] Patent Document 2 (K-Motion Interactive, US7264554B2 and its continuation family) discloses a motion analysis system using inertial sensors, with synchronized display of live video, 3D animation, and time-series data graphs as its core functions. The patent describes functions such as selection of display configurations and performance parameters (multiple, selectable configurations), Kinetic Index scoring, a motion prescription engine, and real-time biofeedback (color changes and sound). However, although display configurations and parameters can be selected, the patent does not disclose a function to dynamically resynchronize and play back multiple recordings of repetitive movements at different alignment reference points.

[0007] Patent document 3 (Nike, US9389057B2) discloses a comprehensive time-based exercise measurement and display system using shoe sensors, GPS, heart rate monitors, etc. The patent describes multiple synchronization methods, including automatic event detection (such as when the foot leaves the ground), stored time indicators, on-the-fly image analysis, and automatic detection of multiple types of events such as pitch release and slam dunks. However, it does not disclose a function that allows the user to dynamically switch reference points for multiple repetition recordings and play them back in a synchronized manner.

[0008] Patent document 4 (Dartfish SA, WO2001039130A1 / US8675021B2) discloses a composite display technology called SimulCam, which involves spatial and temporal normalization of multiple videos. The patent describes multiple synchronization criteria, such as synchronization at four control points at 90, 180, 270, and 360 degrees in a golf swing, and synchronization at the start of a race. Furthermore, it describes interactive changes to spatial and temporal normalization parameters (mouse clicks, keyboard input, etc.), interactive selection of transparency by the user, semi-automatic determination of control points through user intervention, real-time feedback (comparison of the form of a performer and instructor in aerobics), and an alignment method that does not require control points through dynamic programming. However, these interactive functions relate to the adjustment of spatial and temporal normalization parameters, and a UI for dynamically switching the time synchronization reference point during playback is not disclosed.

[0009] Dartfish's product (Analyzer) implements a synchronization function called Key Position (Non-Patent Literature 2). This function allows users to manually place key positions (markers) on up to four comparison videos and resynchronize all videos to that point in time by double-clicking on any key position. However, key positions must be placed manually by the user while visually checking the videos, and there is no suggestion of reference point candidates based on automatic event detection from videos or sensor data.

[0010] Patent document 5 (Adobe, US9201580B2) discloses a sound alignment UI that displays the spectrograms (time-frequency representations) of two audio signals in parallel, allowing the user to manually specify corresponding feature points. The manually specified points act as anchors, and automatic alignment is performed within each section using DTW (Dynamic Time Stretching), a hybrid method. While it is related in that the user interactively specifies corresponding points, this patent is limited to alignment between two audio tracks and does not disclose the concept of synchronously playing back videos of multiple repetitions at a single reference point, nor does it disclose searching by switching between multiple candidate reference points.

[0011] In the field of electroencephalography (EEG) measurement, EEGLAB's erpimage function (Non-Patent Literature 3) provides the ability to realign multiple EEG trials using multiple types of events (stimulus presentation, button responses, etc.) that are pre-recorded (logged) in the experimental protocol. This function enables selection from multiple events, dynamic realignment, and display updates. However, this tool uses a batch processing workflow involving parameter changes and replotting in a dialog window, which is fundamentally different from immediate interaction through operation input on a timeline. Furthermore, it deals with static waveform plots and does not handle synchronized playback of videos.

[0012] In manufacturing process analysis, Seeq's Capsule Time View (Non-Patent Literature 4) provides a web-based interactive visualization tool that overlays and displays time-series signal waveforms of multiple batch processes based on the capsule start point. This tool has automatic condition generation functions from signals, such as Value Search (automatic detection of when a signal value exceeds a specified threshold) and Profile Search (automatic search for similar signal patterns using pattern matching and machine learning), and features advanced interactive visualization functions such as pan, zoom, capsule selection / deselection, and real-time updates. However, it deals with signal waveforms of manufacturing processes and does not handle synchronized playback of repetitive motion videos. Furthermore, it does not disclose a function that allows the user to switch between multiple alignment reference point candidates displayed on the timeline.

[0013] As described above, in the prior art, (1) Patent documents 1 to 3 describe display configurations, parameter selections, and multiple synchronization methods, but do not disclose a function for the user to dynamically switch reference points for multiple repetitive operation recordings and perform synchronized playback in a batch. (2) Patent document 4 (Dartfish patent) describes interactive changes to spatial and temporal normalization parameters, but does not disclose a UI for dynamically switching the reference point for time synchronization during playback. Furthermore, while Dartfish products (non-patent document 2) allow for switching of reference points, they lack automatic detection of reference point candidates and require manual placement. (3) EEGLAB (non-patent document 3) and Seeq (non-patent document 4) have related functions such as signal data realignment and automatic condition generation, but do not target synchronized playback of video, and do not disclose a UI for switching reference points by operation input on the timeline. In other words, there is no system that allows a user to specify a reference point from among multiple reference point candidates by operation input, dynamically realign time-series media data including video, and perform synchronized playback. [Prior art documents] [Patent Documents]

[0014] [Patent Document 1] US9349049B2(Blast Motion Inc.)

Patent document 2

Patent document 3

Patent document 4

Patent document 5

Non-licensed literature

[0015] [Non-licensed document 1] Honert, EC & Pataky, TC, "Timing of gait events affects whole trajectory analyses", Journal of Biomechanics, Vol.119, 110329, 2021

Non-licensed Document 2

Non-licensed Document 4

Non-licensed Document 5

[0016] Conventional technologies do not offer a function for users to dynamically switch reference points and synchronize playback of multiple repetitive motion recordings in a single batch during comparative analysis of time-series media data of repetitive motions. Therefore, it has been difficult to interactively explore the differences in the results of synchronized playback using different alignment reference points and to gain a multifaceted understanding of the motion.

[0017] The present invention has been made in view of the above problems, and aims to provide a system and method for dynamically realigning and synchronously playing back all recordings of multiple time-series media data of repetitive operations, by allowing the user to specify a reference point from among multiple assigned reference point candidates by inputting an operation on the timeline. [Means for solving the problem]

[0018] To solve the above problems, a repetitive operation comparison analysis system according to an aspect of the present invention receives a plurality of time-series media data each recording a plurality of repetitive operations, and each of the plurality of time-series media data is provided with a plurality of reference point candidates corresponding to characteristic events occurring during the operation for each recording. A display unit that displays the plurality of time-series media data in parallel or in an overlapping manner and presents the plurality of reference point candidates on a timeline; an input reception unit that designates one of the reference point candidates as a reference point according to a user's operation input on the timeline; and a synchronization processing unit that realigns all of the plurality of time-series media data at the time position of the designated reference point according to the designation and synchronously reproduces the plurality of time-series media data in the realigned state.

[0019] A repetitive operation comparison analysis method according to another aspect of the present invention is a method executed by a computer, the method comprising: receiving a plurality of time-series media data each recording a plurality of repetitive operations, and each of the plurality of time-series media data is provided with a plurality of reference point candidates corresponding to characteristic events occurring during the operation for each recording; displaying the plurality of time-series media data in parallel or in an overlapping manner and presenting the plurality of reference point candidates on a timeline; designating one of the reference point candidates as a reference point according to a user's operation input on the timeline; and realigning all of the plurality of time-series media data at the time position of the designated reference point according to the designation and synchronously reproducing the plurality of time-series media data in the realigned state.

[0020] A program according to another aspect of the present invention causes a computer to receive a plurality of time-series media data each recording a plurality of repetitive operations, and having a plurality of reference point candidates respectively corresponding to characteristic events occurring during the operations for each recording, display the plurality of time-series media data in parallel or in an overlapping manner and present the plurality of reference point candidates on a timeline, receive an input for designating one of the reference point candidates as a reference point in response to a user's operation input on the timeline, and function as synchronization processing means for realigning all of the plurality of time-series media data at the time position of the designated reference point and synchronously reproducing the plurality of time-series media data in the realigned state.

[0021] In the above repetitive operation comparison and analysis system, the input reception unit may set an interval for each type of the reference point candidates on the timeline, and designate the reference point candidate of the type corresponding to the interval to which the position on the timeline where the operation input is made belongs, as the reference point. In this case, the input reception unit may calculate a representative position based on the time positions in the plurality of time-series media data for each type of the reference point candidates, and set an intermediate point between adjacent representative positions as a boundary of the interval. The representative position may be calculated as a robust statistic of the time positions in the plurality of time-series media data for each type of the reference point candidates, and the robust statistic may be a median. Intervals before the earliest representative position and after the latest representative position on the timeline may be respectively assigned to the reference point candidates of the type corresponding to the nearest representative position.

Advantages of the Invention

[0022] According to the present invention, by specifying a reference point from among a plurality of assigned reference point candidates through user input, synchronized playback based on different perspectives can be obtained from the same time-series media data set. Exploratory operation comparison analysis is possible through timeline input without the need for dialog operations or batch processing. Furthermore, by setting intervals for each type of reference point candidate on the timeline and specifying the reference point candidate corresponding to the interval to which the input location belongs, users can select the intended type of reference point candidate by manipulating the approximate position on the timeline without needing to precisely tap individual markers. Moreover, by using robust statistics (e.g., median) to calculate the representative position, even if the time positions of reference point candidates in some records deviate significantly as outliers, the interval boundaries are stably maintained, ensuring operability even when analyzing datasets containing outliers. [Brief explanation of the drawing]

[0023] [Figure 1] A block diagram showing the functional configuration of a repetitive motion comparison analysis system according to one embodiment of the present invention.

[0024] [Figure 2] This is a screenshot showing the initial state where multiple reference point candidates are presented on the timeline UI, and playback is synchronized at reference point A.

[0025] [Figure 3] This is a screen diagram showing the timeline with intervals set for each type of reference point candidate. The boundary of each interval is set as the midpoint between the representative positions (medians) of adjacent reference point candidates.

[0026] [Figure 4] This is a screenshot showing an example of synchronized playback after switching to reference point D.

[0027] [Figure 5] This is an enlarged view of the timeline showing the distribution of the time positions of candidate reference points A to C, the calculation of representative positions xa to xc, and the setting of interval boundaries.

[0028] [Figure 6] This is a flowchart showing the processing flow related to one embodiment of the present invention. [Modes for carrying out the invention]

[0029] Embodiments of the present invention will be described below with reference to the drawings. Note that the following embodiments are merely examples of the present invention and do not limit the technical scope of the present invention. In the following description, video will be primarily used as an example of time-series media data, but the present invention is not limited to video. It is broadly applicable to any time-series media data that has a time axis, includes characteristic events occurring during operation, and is recorded repeatedly, including sensor signal waveforms, audio signals, etc. (Input data)

[0030] The repetitive motion comparison and analysis system 100 receives multiple time-series media data, each recording of multiple repetitive motions, with each recording assigned multiple reference point candidates corresponding to characteristic events occurring during the motion. In this embodiment, the time-series media data is primarily video data. The means of acquiring the video data are not limited, and videos captured with any shooting means, such as a smartphone camera, tablet device camera, digital video camera, or camera linked to a motion capture system, may be used. In addition to or instead of video data, time-series media data such as signal waveform data and audio data acquired from wearable sensors (accelerometers, gyroscopes, etc.) may also be used.

[0031] The assignment of reference point candidates is performed in advance outside of this system. The means of assigning reference point candidates are not limited and may include, but are not limited to, one or more of the following methods: (a) Pose estimation-based detection: Estimating the human skeleton (joint positions) from video frames and detecting extreme values ​​of joint angles and points of velocity change. (b) Optical flow analysis: Detecting points in time when the amount of pixel movement between consecutive frames changes abruptly. (c) Object detection-based: Detecting specific events (release, impact, etc.) from changes in the position of tools such as balls and club heads. (d) Audio signal-linked detection: Detecting impact sounds, etc., from audio channels associated with the video. (e) Associated sensor data-linked detection: Detecting events from signals of wearable sensors recorded in synchronization with the video. (f) Machine learning-based detection: Detecting motion events using models trained by supervised or unsupervised learning. (g) Manual assignment: The user manually sets reference point candidates while visually checking the video.

[0032] For example, in the pitching motion of baseball, candidate reference points can be assigned to correspond to the following events: set position (the end of the period in which the pitcher's posture is stationary), leg lift start (the point at which the knee joint angle of the front leg begins to rise), foot plant (the point at which the front foot touches the ground), maximum external rotation (the extreme value of the shoulder joint external rotation angle), release point (the moment the ball leaves the hand), and end of follow-through (the point at which the movement of the entire body converges). (System Configuration)

[0033] Figure 1 is a block diagram showing the functional configuration of a repetitive motion comparison and analysis system 100 according to one embodiment of the present invention. The repetitive motion comparison and analysis system 100 includes a display unit 110, an input receiving unit 120, and a synchronization processing unit 130. (Display)

[0034] The display unit 110 receives multiple time-series media data to which reference point candidates have been assigned, and displays the multiple time-series media data in parallel or superimposed. In the case of parallel display, multiple videos of video data are displayed simultaneously on the screen side by side (for example, in a grid arrangement of 2 columns horizontally and 3 rows vertically). The layout of the parallel display may include, but is not limited to, horizontal arrangement, vertical arrangement, grid, or a main video and a group of small thumbnails. In the case of superimposed display, multiple videos are displayed superimposed within the same display area by alpha blending (semi-transparent overlay). This makes the differences in the forms of multiple actions synchronized at the same reference point directly visible on a single screen. The transparency of each video in the superimposed display may be adjustable by the user. The display unit 110 may be configured to switch between parallel display and superimposed display.

[0035] The display unit 110 presents multiple reference point candidates assigned to each record as visual markers on the timeline. The timeline is displayed below each video or at the bottom of the screen and includes a playhead (displayed as a vertical line traversing the timeline of all records) indicating the current playback position, and markers (shape symbols, labeled icons, colored dots, etc.) indicating the time positions of each reference point candidate. If a reference point is specified, the playhead is positioned at the time position of that reference point. The marker for the currently specified reference point is distinguished from other markers by visual emphasis (thick line, accent color, fill, etc.). (Input reception section)

[0036] The input receiving unit 120 designates one of several candidate reference points as the reference point in response to user input on the timeline. User input may include double-tap or long-press operations on the touchscreen, mouse clicks, keyboard input, or a combination thereof.

[0037] In one embodiment, when the user double-tap one of the reference point markers on the timeline, that reference point is newly designated, and realignment and synchronized playback are triggered by the synchronization processing unit 130. In another embodiment, a group of labeled buttons for reference point candidates is displayed above or below the timeline, and the user can specify a reference point by tapping one of the buttons.

[0038] In one embodiment, the input receiving unit 120 determines which position on the timeline corresponds to which type of reference point candidate using the following procedure. First, for each type of reference point candidate, the median of the time positions of reference point candidates of that type in multiple records is calculated as the representative position. Next, the midpoint between adjacent representative positions on the timeline is set as the boundary. As a result, the timeline is divided into the same number of adjacent intervals as the number of types of reference point candidates. Both ends of the timeline (the interval before the earliest representative position and the interval after the latest representative position) are assigned to the type corresponding to the nearest representative position. When a user input is made at a position on the timeline, the input receiving unit 120 determines which interval the position belongs to and designates the corresponding type of reference point candidate as the reference point.

[0039] By using the median as the representative position, the representative position and boundary positions are stably maintained even when the time positions of certain types of reference point candidates deviate significantly from other records in some records. For example, if the release point time positions in five pitching records are 5.2 seconds, 5.3 seconds, 4.6 seconds, 5.1 seconds, and 5.2 seconds, the average is 5.08 seconds, affected by the outlier (4.6 seconds), while the median is 5.2 seconds, virtually unaffected by the outlier. This ensures that the type of reference point candidate intended by the user is stably selected, regardless of the variability in the time positions of reference point candidates in individual records. In particular, in this system, records with time positions that deviate significantly from other records often represent important data for analysis, indicating abnormalities in movement or breakdowns in form. In datasets containing such outliers, using the average to calculate the representative position leads to the problem that the outlier itself distorts the interval boundaries, making it difficult to select the type of reference point candidate intended by the user. Using the median ensures that interval boundaries remain stable regardless of the presence of outliers, maintaining usability even when analyzing data containing outliers. Note that the statistic used to calculate the representative position is not limited to the median; other robust statistics such as the trimmed mean may also be used.

[0040] Refer to Figure 5 to explain a specific example of specifying reference points based on interval settings and operation input. Figure 5 shows the time positions of three types of reference point candidates A (leg lift, ●), B (foot plant, □), and C (maximum external rotation, △) in three pitching records (pitch 1 to pitch 3). The time positions of each type of reference point candidate vary from record to record. In particular, the time position of reference point candidate B in pitch 2 and the time position of reference point candidate C in pitch 3 deviate significantly from the time positions of the same type of reference point candidate in other records.

[0041] The input receiving unit 120 calculates a representative position for each type of reference point candidate using the robust statistic (median in this example) of the time position of that type in multiple records. Let xa be the representative position of reference point candidate A, xb be the representative position of reference point candidate B, and xc be the representative position of reference point candidate C (dashed lines in Figure 5). Although the time position of reference point candidate B in pitch 2 deviates significantly from the other records, xb, which is the median of the three records, is a stable value close to the time position of reference point candidate B in pitch 1 and pitch 3. Similarly, although the time position of reference point candidate C in pitch 3 deviates significantly from the other records, xc is a stable value close to the time position of reference point candidate C in pitch 1 and pitch 2.

[0042] The input receiving unit 120 sets the midpoint between adjacent representative positions as the boundary of the interval. That is, (xa+xb) / 2 becomes the boundary between interval A and interval B, and (xb+xc) / 2 becomes the boundary between interval B and interval C (solid line in Figure 5). Since the representative positions xa, xb, and xc are not affected by outliers, the interval boundaries set as their midpoints are also stably maintained. When a user makes an operation input at position x on the timeline, the input receiving unit 120 determines which interval that position x belongs to. Specifically, it determines that if x < (xa+xb) / 2, it is interval A; if (xa+xb) / 2 ≤ x < (xb+xc) / 2, it is interval B; and if (xb+xc) / 2 ≤ x, it is interval C, and designates the corresponding type of reference point candidate as the reference point. As a result, the synchronization processing unit 130 realigns all records at the time position of the corresponding type of reference point candidate in each record and updates the synchronized playback. Thus, even when the time positions of potential reference points diverge significantly in some recordings, users can reliably select the intended type of reference point candidate by inputting a rough position on the timeline. (Synchronization Computing)

[0043] The synchronization processing unit 130, in response to the reference point specified by the input receiving unit 120, realigns all of the multiple time-series media data to the time position of the specified reference point. Specifically, it obtains the time ti corresponding to the reference point assigned to each record i and realigns the playback timeline of each record i so that the time ti of the reference point in that record is aligned to the same position on the playback timeline. As a result, all time-series media data are played back in synchronous manner so that they reach the moment of the specified reference point simultaneously.

[0044] The synchronization processing unit 130 synchronizes and plays back multiple time-series media data in a realigned state. That is, multiple time-series media data displayed in parallel or superimposed via the display unit 110 are played back simultaneously in a state where they are temporally aligned at a specified reference point. Each time the user changes the reference point, the playback timing offset of each time-series media data is recalculated, and playback is updated to the new synchronized state. (Playback speed control)

[0045] The synchronization processing unit 130 provides various playback speed controls while maintaining the realigned synchronization state. Specifically, in addition to synchronized playback at normal speed, it can play all time-series media data in sync at slow playback (e.g., 0.25x speed, 0.5x speed) and double-speed playback (e.g., 2x speed, 4x speed). At any playback speed, the synchronization relationship based on the reference point between each record is maintained. (Frame-by-frame synchronous forward / backward)

[0046] The synchronization processing unit 130 provides a function to advance or rewind all time-series media data synchronously on a frame-by-frame basis while maintaining the realigned synchronization state. That is, when the user advances one frame, all displayed time-series media data advances synchronously by one frame. This makes it possible to compare subtle differences in movement before and after a reference point with frame-level precision. (Example of operation)

[0047] Refer to Figures 2 to 4 to explain an example of the operation of System 100. Suppose a baseball coach films multiple pitches thrown by a pitcher, and inputs these multiple pitching videos, each with a reference point candidate attached, into System 100.

[0048] Each pitching video is assigned five types of reference point candidates: "leg lift," "foot plant," "maximum external rotation," "release point," and "follow-through." The display unit 110 displays multiple pitching videos side-by-side or superimposed on the screen, and also presents the five types of reference point candidates as labeled markers on the timeline (Figure 2).

[0049] Initially, "leg lift" is designated as the reference point, and multiple videos are synchronized to reach the moment of the leg lift simultaneously (Figure 2). When the coach presses the play button, all videos play simultaneously, all aligned at the moment the legs begin to be lifted. This synchronized playback allows the coach to visually confirm the consistency of the movement tempo from the leg lift onward.

[0050] When the coach double-tap the area on the timeline where the candidate reference points for the "release point" are located (Figure 3), the input reception unit 120 accepts this input, and the synchronization processing unit 130 realigns all the videos so that they are aligned at the moment of the release point, and the synchronized playback is updated (Figure 4). In this synchronized playback, the moment the ball leaves the hand is displayed simultaneously in all videos. The coach switches to slow-motion playback to closely observe the arm swing and follow-through before and after the release. In a particular throw, it is possible to visually discover a motion pattern where the timing of the foot plant just before the release is earlier than in other throws, and the release occurs before the torso rotation is fully completed.

[0051] Furthermore, the coach uses the frame-advancing function to synchronize and rewind all videos one frame at a time, starting from a few frames before the release point. Because the synchronization is based on the release point, the moment of foot planting appears at a different frame position in each pitching video. For example, in one pitch, the foot planting is observed 12 frames before the release, while in another, it is observed 8 frames before. This means that the time required from foot planting to release differs from pitch to pitch, and by rewinding frame by frame in synchronized playback based on the release point, this time difference is directly visualized as a discrepancy in the timing of the foot planting in each pitch. Starting from this discrepancy, the coach can compare the degree of trunk rotation after foot planting and the start timing of the arm swing between pitches, and identify with frame-level accuracy at which stage of the motion chain the discrepancy is escalating. This allows the coach to go beyond simply observing that "the timing of the foot planting is different" and to understand specifically which phase of the motion needs improvement, enabling them to provide instruction accordingly.

[0052] In this way, by switching the reference point, different operational characteristics emerge from the same time-series media data set. In this invention, by allowing the user to specify a reference point from among multiple assigned reference point candidates through user input, it becomes possible to perform multifaceted comparative analysis interactively. (Application to other repetitive actions)

[0053] The present invention is not limited to the baseball pitching motion described above. It is applicable to any repetitive motion recorded as time-series media data, such as golf swings, tennis serves, dart throws, bowling, swimming strokes, and other repetitive sports movements, as well as gait cycle analysis in rehabilitation exercises and repetitive assembly work in factories. (modified version)

[0054] In the above embodiments, video data was described as the primary time-series media data, but the present invention is not limited to this. Sensor signal waveform data (acceleration, angular velocity, electromyography, etc.), audio data, or combinations thereof can also be used as time-series media data. For example, video and sensor signal waveforms may be displayed simultaneously in parallel or superimposed, and the realignment of the signal waveform display may be performed in conjunction with the synchronized playback of the video in response to the switching of the reference point.

[0055] Furthermore, the synchronization processing unit 130 is not limited to alignment by simple time shifting. Other alignment methods may be applied, such as alignment by dynamic time stretching (DTW) or piecewise linear alignment using multiple reference points.

[0056] Furthermore, the input receiving unit 120 may accept the specification of a reference point using input means other than touchscreen operations, such as voice commands, keyboard shortcuts, and physical buttons. [Industrial applicability]

[0057] This invention can be used in a wide range of industrial fields where time-series media data of repetitive movements are compared and analyzed through synchronized playback, such as sports motion analysis, biomechanics research, gait analysis in rehabilitation medicine, skill instruction in martial arts and dance, and quality control of repetitive tasks in manufacturing processes. [Explanation of Symbols]

[0058] 100 Repetitive Motion Comparison and Analysis System 110 Display section 120 Input reception section 130 Synchronization Processing Unit

Claims

1. A repetitive motion comparison and analysis system, A display unit that receives multiple time-series media data, each recording multiple repetitive operations, and each recording is assigned multiple reference point candidates corresponding to characteristic events that occur during the operation; displays the multiple time-series media data in parallel or superimposed on each other, and presents the multiple reference point candidates on a timeline; An input receiving unit that, in response to user input on the timeline, designates one of the candidate reference points as the reference point, A synchronization processing unit realigns all of the multiple time-series media data so that they coincide with each other at the time position of the specified reference point, and plays back the multiple time-series media data in a synchronized state in accordance with the above specification. A repetitive motion comparison and analysis system equipped with the following features.

2. A computer-based method for comparing and analyzing repeated actions, The process involves receiving multiple time-series media data, each recording multiple repetitive actions, and assigning multiple reference point candidates to each recording, corresponding to characteristic events that occur during the action; displaying the multiple time-series media data in parallel or superimposed; and presenting the multiple reference point candidates on a timeline. The steps include: designating one of the candidate reference points as the reference point in response to user input on the timeline; The steps include: realigning all of the multiple time-series media data so that they coincide with each other at the time position of the specified reference point, and synchronously playing back the multiple time-series media data in the realigned state; A method for comparative analysis of repetitive actions, including the following:

3. Computers, A display means that receives multiple time-series media data, each recording multiple repetitive actions, and each recording is assigned multiple reference point candidates corresponding to characteristic events that occur during the action; displays the multiple time-series media data in parallel or superimposed on each other; and presents the multiple reference point candidates on a timeline. An input receiving means that, in response to user input on the timeline, designates one of the candidate reference points as the reference point, and, in response to the designation, realigns all of the multiple time-series media data so that they coincide with each other at the time position of the designated reference point. A program for functioning as a synchronization processing means for synchronously playing back the multiple time-series media data in a realigned state.

4. A repetitive motion comparison analysis system according to claim 1, The input receiving unit sets intervals on the timeline for each type of reference point candidate, and designates the reference point candidate of the type corresponding to the interval to which the position on the timeline where the operation input was made belongs as the reference point. A repetitive motion comparison and analysis system characterized by the following:

5. A repetitive motion comparison analysis system according to claim 4, The input receiving unit calculates a representative position for each type of reference point candidate based on the time position in the plurality of time-series media data, and sets the midpoint between adjacent representative positions as the boundary of the interval. A repetitive motion comparison and analysis system characterized by the following:

6. A repetitive motion comparison analysis system according to claim 5, The representative position is calculated as a robust statistic of the time position of each type of reference point candidate in the plurality of time-series media data. A repetitive motion comparison and analysis system characterized by the following:

7. A repetitive motion comparison analysis system according to claim 6, The robust statistic mentioned above is the median. A repetitive motion comparison and analysis system characterized by the following:

8. A repetitive motion comparison analysis system according to any one of claims 5 to 7, The intervals prior to the earliest representative position and the intervals following the latest representative position on the timeline are each assigned to the type of reference point candidate corresponding to the closest representative position. A repetitive motion comparison and analysis system characterized by the following:

9. A repetitive motion comparison analysis system according to claim 1, The synchronization processing unit can change the playback speed while maintaining the synchronization state between the realigned time-series media data. A repetitive motion comparison and analysis system characterized by the following:

10. A repetitive motion comparison analysis system according to claim 1, The synchronization processing unit maintains the synchronization state between the realigned time-series media data and synchronizes sending or returning the multiple time-series media data on a frame-by-frame basis. A repetitive motion comparison and analysis system characterized by the following:

11. A method for comparing and analyzing repeated motions according to claim 2, In the step of specifying the reference point, intervals for each type of reference point candidate are set on the timeline, and the reference point candidate of the type corresponding to the interval to which the position on the timeline where the operation input was made belongs is designated as the reference point. A method for comparative analysis of repetitive movements, characterized by the following features.

12. A method for comparing and analyzing repeated motions according to claim 11, In setting the interval, a representative position is calculated for each type of reference point candidate based on the time position in the multiple time-series media data, and the midpoint between adjacent representative positions is set as the boundary of the interval. A method for comparative analysis of repetitive movements, characterized by the following features.

13. A method for comparing and analyzing repeated motions according to claim 12, The representative position is calculated as a robust statistic of the time position of each type of reference point candidate in the plurality of time-series media data. A method for comparative analysis of repetitive movements, characterized by the following features.

14. A method for comparing and analyzing repeated motions according to claim 13, The robust statistic mentioned above is the median. A method for comparative analysis of repetitive movements, characterized by the following features.

15. A method for comparing and analyzing repeated motions according to any one of claims 12 to 14, The intervals prior to the earliest representative position and the intervals following the latest representative position on the timeline are each assigned to the type of reference point candidate corresponding to the closest representative position. A method for comparative analysis of repetitive movements, characterized by the following features.

16. A method for comparing and analyzing repeated motions according to claim 2, In the synchronized playback step, the playback speed can be changed while maintaining the synchronization state between the realigned time-series media data. A method for comparative analysis of repetitive movements, characterized by the following features.

17. A method for comparing and analyzing repeated motions according to claim 2, In the synchronized playback step, the synchronized state between the realigned time-series media data is maintained while the multiple time-series media data are advanced or advanced in a frame-by-frame manner. A method for comparative analysis of repetitive movements, characterized by the following features.

18. The program according to claim 3, The input receiving means sets intervals on the timeline for each type of reference point candidate, and designates the reference point candidate of the type corresponding to the interval to which the position on the timeline where the operation input was made belongs as the reference point. A program characterized by the following features.

19. The program according to claim 18, The input receiving means calculates a representative position for each type of reference point candidate based on the time position in the plurality of time-series media data, and sets the midpoint between adjacent representative positions as the boundary of the interval. A program characterized by the following features.

20. The program according to claim 19, The representative position is calculated as a robust statistic of the time position of each type of reference point candidate in the plurality of time-series media data. A program characterized by the following features.

21. The program according to claim 20, The robust statistic mentioned above is the median. A program characterized by the following features.

22. A program according to any one of claims 19 to 21, The intervals prior to the earliest representative position and the intervals following the latest representative position on the timeline are each assigned to the type of reference point candidate corresponding to the closest representative position. A program characterized by the following features.

23. The program according to claim 3, The synchronization processing means can change the playback speed while maintaining the synchronization state between the realigned plurality of time-series media data. A program characterized by the following features.

24. The program according to claim 3, The synchronization processing means maintains the synchronization state between the realigned time-series media data and synchronizes sending or returning the plurality of time-series media data on a frame-by-frame basis. A program characterized by the following features.

Images