Information processing apparatus, information processing method, and program

Abstract

To provide an information processing apparatus that can make a playback start position of time-series data correspond to an instruction position of image data simply with sufficient accuracy.SOLUTION: An information processing apparatus (21) has: a time-series data acquisition unit (22) that acquires a playback start time of time-series data; an image data acquisition unit (23) that acquires image data related to the time-series data; an instruction information acquisition unit (24) that acquires, during playback of the time-series data, instruction information related to an instruction time and an instruction position in the image data; and a data processing unit (25) that associates, by using the instruction information, the instruction position with an elapsed time representing the time elapsed from the playback start time of the time-series data corresponding to the instruction time.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Conventionally, there are known methods for controlling the playback of audio and video in accordance with musical scores. Patent Document 1 discloses a method for creating a "bar time conversion table" in advance, which is referenced when associating the playback time of a video with the bar number of a musical score. Patent Document 2 discloses a method for identifying the playback time of a recording by converting the recording and the musical score into data representing pitches and comparing them to associate the playback time of a recording with the position information of the musical score. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-113727 A [Patent Document 2] International Publication No. 2017 / 057318 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the method disclosed in Patent Document 1 requires that the score and the video be associated in advance, which makes the process complicated and time-consuming.The method disclosed in Patent Document 2 requires that the recorded sound data be analyzed or that the pitches written on the score be image-recognized, which makes the process difficult.

[0005] SUMMARY OF THE PRESENT DISCLOSURE An object of the present invention is to provide an information processing device that can easily and accurately match the playback start position of time-series data to a designated position of image data. [Means for solving the problem]

[0006] An information processing device as one aspect of the present invention has a time series data acquisition unit that acquires a playback start time of time series data, an image data acquisition unit that acquires image data related to the time series data, an instruction information acquisition unit that acquires instruction information related to a specified time and a specified position in the image data during playback of the time series data, and a data processing unit that uses the instruction information to associate the specified position with an elapsed time that represents the time that has elapsed from the playback start time of the time series data that corresponds to the specified time.

[0007] Other objects and features of the present invention are illustrated in the following examples. Effect of the Invention

[0008] According to the present invention, it is possible to provide an information processing device that can easily and sufficiently accurately make the playback start position of time-series data correspond to the designated position of image data. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a block diagram of a terminal device in each embodiment. [Diagram 2] FIG. 2 is a block diagram of an information processing device in each embodiment. [Figure 3(a)] 4 is a flowchart showing a process of the information processing device in the first embodiment. [Figure 3(b)] 4 is a flowchart showing a process of the information processing device in the first embodiment. [Figure 3(c)] 4 is a flowchart showing a process of the information processing device in the first embodiment. [Figure 4] FIG. 2 is an explanatory diagram showing the process of the information processing device in the first embodiment on a time axis. [Diagram 5] FIG. 4 is an explanatory diagram of data recorded in the information processing device in the first embodiment. [Figure 6] FIG. 4 is an explanatory diagram of an input standby mode in the first embodiment. [Figure 7] FIG. 2 is an explanatory diagram of a viewing mode in the first embodiment. [Figure 8]13 is a flowchart showing a process of an information processing device in a second embodiment. [Figure 9] FIG. 11 is an explanatory diagram showing the process of the information processing device in the second embodiment on a time axis. [Figure 10] FIG. 11 is an explanatory diagram of data recorded in an information processing device in the second embodiment. [Figure 11] FIG. 11 is an explanatory diagram of an input standby mode in the second embodiment. [Figure 12] FIG. 11 is an explanatory diagram of data recorded in an information processing device in the third embodiment. [Figure 13] FIG. 13 is an explanatory diagram of an input standby mode in the third embodiment. [Figure 14] FIG. 13 is an explanatory diagram of a viewing mode in the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. In each drawing, the same reference numerals are used for the same components, and duplicated explanations will be omitted. In the following, the user is assumed to be an instructor, a student who checks his / her own performance, etc. EXAMPLES

[0011] First, a first embodiment of the present invention will be described. In this embodiment, a case will be described in which an instructor gives guidance to a learner's performance when playing a keyboard instrument as an example of musical instrument performance. However, this embodiment can also be applied to similar situations with other instruments such as stringed instruments and drums. This embodiment can also be applied to a case in which a learner checks his or her own performance. It is assumed that the performance is not a free improvisation, but a piece of music written on a musical score.

[0012] 1 is a block diagram of a terminal device 100. The terminal device 100 is a device such as a personal computer, a notebook computer, a tablet terminal, or a smartphone that allows a user to obtain, input, view, and the like information. The terminal device 100 has a control / calculation unit 11, a storage unit 12, a communication unit 13, a video capture unit 14, an input unit 15, and a display unit 16.

[0013] The control and calculation unit 11 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), etc. The ROM stores programs and the like that are read and executed by the CPU. The RAM is used as a working memory for the CPU. The CPU executes various processes and issues commands according to the programs stored in the ROM, thereby controlling the entire terminal device 100.

[0014] The storage unit 12 is a large-capacity storage medium using a hard disk, a semiconductor memory, etc. The storage unit 12 can store still images captured by a video imaging unit 14, video and audio, and the like.

[0015] The communication unit 13 is a communication module or a communication connector for communicating with other devices or the Internet. The communication by the communication unit 13 is wired communication such as USB communication, wireless LAN such as Wi-Fi, Bluetooth (registered trademark), ZigBee (registered trademark), 4G (fourth generation mobile communication system), 5G (fifth generation mobile communication system), broadband, and other wireless communication. The communication unit 13 is not an essential component of the terminal device 100. The communication unit 13 is not necessary for a user who simply processes, inputs, and displays information on the terminal. In this embodiment, it is basically assumed that data is stored on the terminal and inputs, displays, and processes information. However, other than that, a data storage device is prepared on the server to store data, and the communication unit 13 is constantly communicating to perform input and display operations, or an information processing device is prepared on the server to perform information processing.

[0016] The video imaging unit 14 includes an imaging element, an image processing LSI, etc., and has a camera function capable of capturing still images and video images. The video imaging unit 14 also has a microphone function capable of recording audio.

[0017] The input unit 15 is, for example, a touch panel integrated with the display unit 16, a pointing device operated by tracing a finger over a flat plate-like sensor not integrated with the display unit 16 called a track pad or touch pad, a keyboard, a mouse, etc. The input unit 15 accepts operation input from a user to the terminal device 100. When the user makes an input to the input unit 15, an input signal corresponding to the input is generated and output to the control and calculation unit 11. The control and calculation unit 11 performs calculation processing corresponding to the acquired input signal, and also controls the terminal device 100.

[0018] The display unit 16 is a display device configured by, for example, an LCD (Liquid Crystal Display), a PDP (Plasma Display Panel), an organic EL (Electro Luminescence) panel, etc. On the display unit 16, a user interface of the terminal device 100, etc. are displayed.

[0019] Next, the terminal device 100 will be described in detail. In this embodiment, the time series data is described assuming that it is recorded data (video data including sound or actions, etc.). However, this embodiment is not limited to this, and the time series data may be audio data (recorded data). The acquired recorded data is assumed to be data that records the sound or physical actions (dance, etc.) of a performer playing an instrument. In addition, the description assumes that it is an image of a musical score as an image from which time series information can be read. Note that an image of the musical score can be prepared and used by capturing an image of the musical score. In this case, it is not necessary for the image to recognize the positions of each measure of the musical score or the notes, etc., within the information processing device.

[0020] 2 is a block diagram showing the connection relationship between the information processing device in the terminal device 100 and devices other than the information processing device. The information processing device 21 has a time series data acquisition unit 22, an image data acquisition unit 23, an instruction information acquisition unit 24, and a data processing unit 25. The terminal device 100 has a time series data recording unit 26, a display unit 27, and an instruction input unit 28, and is connected to the information processing device 21.

[0021] The time series data acquisition unit 22 acquires the video recording data recorded in the time series data recording unit 26 described later. The image data acquisition unit 23 acquires images of musical scores prepared in advance (image data related to time series data, or image data from which time series information can be read). The instruction information acquisition unit 24 acquires instruction information (instruction time and instruction position) inputted by the instruction input unit 28 described later.

[0022] As described later, the data processing unit 25 uses the designated time and the designated position to associate the designated position with an elapsed time indicating the time elapsed from the playback start time of the time-series data corresponding to the designated time. That is, the data processing unit 25 calculates the elapsed time from the playback start time of the time-series data corresponding to the designated time based on the playback start time of the time-series data and the designated time, and associates it with the designated position on the image.

[0023] The time-series data recording unit 26 corresponds to the video imaging unit 14, and is a recording unit that records the recorded data as time-series data. The recorded data does not have to be photographed (recorded) by the time-series data recording unit 26, and may be data photographed by a general video imaging device. The display unit 27 corresponds to the display unit 16, and displays the image data of the musical score and the recorded data. The instruction input unit 28 corresponds to the input unit 15 such as a touch panel or a mouse, and inputs various instructions such as instructions for the image data of the musical score and instructions for playing the recorded data.

[0024] 3(a), 3(b), and 3(c) are flowcharts showing the processing of the information processing device 21. In the following, it is assumed that the user is an instructor or a learner, and that the instructor inputs instructions and the learner plays back the recording according to the input points. Note that the instructor may input instructions and play back the recording according to the input points, or the instructor may perform the operations by himself to check his own performance.

[0025] 3(a) is a flowchart showing the instructor's instruction input and the corresponding processing of the information processing device 21, assuming that the user is an instructor. First, in step SA301, the user issues an instruction to display the musical score and enter an input standby mode through the instruction input unit 28. In step SA302, the information processing device 21 displays an image of the musical score (an image related to time-series data) on the display unit 27. The information processing device 21 also starts an input standby mode for the musical score.

[0026] Next, in step SA303, the user issues an instruction to play the recorded video (time series data) through instruction input unit 28. Note that in time series data acquisition unit 22, the playback of the time series data may be performed simultaneously with the recording of the time series data in time series data recording unit 26. In other words, it is possible not only to play back the time series data previously recorded in time series data recording unit 26, but also to play back the time series data while recording it in time series data recording unit 26.

[0027] In this embodiment, recorded data is assumed as time-series data, and for example, while recording is being performed in the video imaging unit 14 of the terminal device 100, the data is displayed on the display unit 27 at approximately the same time. This creates a state in which pre-recorded data is being played back and viewed on the display unit 27 at the same time as the recording of the recorded data, and makes it possible to accept operations on the instruction input unit 28. In step SA304, the information processing device 21 obtains the playback start time T_st of the recorded data by the time-series data acquisition unit 22.

[0028] Next, in step SA305, the user judges the state of the performance at each time (for example, whether it is a part that requires attention or a part that is well performed, etc.) by viewing the recorded video being played back and listening to the audio contained in the recorded video. Then, based on that judgment, the user recognizes the part on the score image that corresponds to the playback time and gives an input instruction using instruction input unit 28. This can be done, for example, by using a touch sensor on a tablet to press the relevant part on the image with a finger or with a special pen. It is assumed that the operation of step SA305 will be performed relatively immediately in response to the performance.

[0029] In step SA306, the information processing device 21 acquires, by the instruction information acquisition unit 24, position information on the image where an instruction was given on the score image and time T_ins. The instruction information acquired by the instruction information acquisition unit 24 may include not only one piece of information but also multiple types of selectable instruction information (instruction contents). For example, multiple types of instruction contents such as "Caution" and "Good" are prepared, and one of the instruction contents can be selected when giving an instruction and given on the score image. Other types of instruction contents such as "Faster" and "Slower" can also be assumed. The position information on the image is acquired as a position on the image data taking into account the enlargement or reduction of the image. If the score data spans multiple pages, the page number is also acquired. By acquiring this instruction, the information is saved and displayed at the location where the instruction was given on the score image. Each of steps SA305 and SA306 may be performed multiple times.

[0030] Next, in step SA307, the user ends the playback of the recording by using the instruction input unit 28. Note that the playback may be automatically ended when the recording is played to the end. In step SA308, the information processing device 21 ends the input standby state for the musical score.

[0031] 3(b) is a flowchart showing the processing of the information processing device 21 based on the instructor's input. First, in step SB301, the information processing device 21 calculates, by the data processing unit 25, the elapsed time t_ins from the playback start time of the recorded image, which corresponds to the position information on the image where the instruction was made on the score image. At that time, the calculation is performed according to the following formula (1) using the playback start time T_st of the recorded image and the time T_ins when the instruction was made on the score image.

[0032] t_ins = T_ins - T_st ... (1) As mentioned above, the user judges the state of the performance at each time by viewing the playback video or listening to the audio included in the video. In this case, it is assumed that the judgment takes some time, and there is a possibility that there will be a delay in pointing to the position on the score image that corresponds to the playback part that the user wants to point out. To account for this time, a predetermined time width d_t is shifted to calculate the time that is estimated to correspond to the time of the playback part that the user wants to point out.

[0033] Next, in step SB302, the data processing unit 25 calculates an estimated playback time t_est using a predetermined time span d_t. The data processing unit 25 sets the predetermined time span d_t, and can calculate the estimated playback time t_est based on the elapsed time t_ins from the start of playback of the recorded video, as shown in the following formula (2).

[0034] t_est=t_ins-d_t (2) Other calculation methods that meet this intention may be used. With the above, the processing for the video and the score is completed. It is possible, but not necessarily desirable, for the user to pause the playback of the video and associate the position on the score image with the playback time of the video. The reason for this is that pausing the playback of the video every time significantly reduces the user's work efficiency. Also, when working by recording the video in the time-series data recording unit 26 and playing back the video at the same time, it is almost impossible to pause the playback of the video.

[0035] 3(c) is a flowchart showing a learner's instruction to record and play back and the corresponding processing of information processing device 21, assuming that the user is a learner. First, in step SC301, the user starts displaying the musical score using instruction input unit 28. At the same time, in step SC302, information processing device 21 starts a musical score viewing mode.

[0036] Next, in step SC303, the user uses the instruction input unit 28 to press the position on the score image where the instruction was given. At this time, it is assumed that the display unit 27 and the instruction input unit 28 are the same terminal device 100, and the operation is performed on the terminal device 100. In other words, it is assumed that the display unit 27 displays the position on the image where the instruction was given on the score image, and the instruction input unit 28 receives information on the specified position among the positions on the image. One method of determining the position of the press is to obtain information on the position pressed with a finger or a special pen using a touch sensor on the tablet, and calculate the distance on the image from the information on the position of the instruction given on the score image that has already been saved and displayed. If the distance on the image is smaller than a certain value, it can be determined that the indicated position on the score image has been pressed.

[0037] In step SC304, the information processing device 21 plays back the recording from a predetermined time (playback time t_est) using the instruction information acquisition unit 24 and the data processing unit 25. This allows the user to play back the recording from the time corresponding to the position on the score image where the instruction was input during playback of the recording. Note that the display unit 27 used when playing back the recording does not need to be the same as the display unit 27 that displays the score.

[0038] In this way, the instruction entered by the instructor on the score image not only enables the recording to be played back from that point on the score image, but also makes it possible to point out points that the learner should pay attention to when reviewing, which can improve the effectiveness of the learner's review.

[0039] Next, in step SC305, the user ends the display of the score using the instruction input unit 28. At the same time, in step SC306, the information processing device 21 ends the score viewing mode.

[0040] FIG. 4 is an explanatory diagram showing the processing of the information processing device 21 on a time axis. There is a time axis 41 and an axis 42 of elapsed time from the start of playback of the sound recording data. Time is represented by an uppercase T, and elapsed time from the start of playback is represented by a lowercase t. The start (0 seconds) 44 of the sound recording data corresponds to the playback start time T_st 43. The time T_ins 45 when an instruction is made on the score image is acquired by the instruction information acquisition unit 24. Since an instruction is made on the score image while playing the recorded data, the corresponding playback time t_ins 46 can be acquired. The estimated playback time is t_est 47. Usually, the estimated playback time t_est 47 is a time before the corresponding playback time t_ins 46. The time span between the playback time t_ins 46 and the estimated playback time t_est 47 is d_t 48. The end time of the recorded data is 49.

[0041] FIG. 5 is an explanatory diagram of the data recorded in the information processing device 21. Explaining the columns in order, "No." is the data number. "Position information (page number)" is the page number of the sheet music image. "Position information (position on image (x, y))" is information on the position on the sheet music image. This information is expressed as a position on the sheet music that remains unchanged even if the image is enlarged or reduced.

[0042] "Time (No. 1 is the playback start time T_st. No. 2 and onwards are the time when the instruction to the image was given T_ins)" is the time when the input instruction was given on the music score image. The playback start time of the recording is recorded first, and then the time when the input instruction was given on the music score image is recorded in order. "Elapsed time from playback start (corresponding playback time t_ins)" is the elapsed time from the start of playback when the input instruction was given on the music score image. The playback start time of the recording is set to "0:00" and the elapsed time after that is calculated. "Elapsed time from playback start (estimated playback time t_est)" is the time obtained by subtracting a predetermined time width d_t from "Elapsed time from playback start (corresponding playback time t_ins)". In this example, the predetermined time width d_t is calculated as 2 seconds. When a position on the music score image that corresponds to "Position information (page number)" and "Position information (position on the image (x, y))" is pressed, the recorded data is played from "Elapsed time from playback start (estimated playback time t_est)".

[0043] Fig. 6 is an explanatory diagram of the input standby mode, showing an image of the display unit 27 and the instruction input unit 28 of the terminal device 100. Fig. 6 shows a state in which, after a specific application is started on the terminal device 100, the sheet music image and the recorded data are read, and the input standby mode for the sheet music is entered. Also, the state in which some of the input described below has been performed is shown.

[0044] Reference numeral 61 denotes a display indicating that the screen is in standby mode for inputting the musical score (image data related to time-series data). Reference numeral 62 denotes a display area for the musical score image. The page number is displayed in the upper left, and a scroll bar for page forwarding is located on the right edge. Stamps and markers, which will be described later, can be inputted into the musical score. Reference numeral 63 denotes the stamp type (type of instruction information). The type of stamp can be selected by pressing the "Caution" or "Good" mark on the screen.

[0045] 64 is the vertical width of the marker. As will be described later, it is assumed that the vertical width of the marker is specified and painted horizontally to the right on the music score screen. In this case, the vertical width of the marker can be changed to make input easier, depending on the original notation method of the music score image, etc. In the example of Figure 6, it is assumed that the number of pixels will be input, so it is set to "50". The actual vertical width is displayed to the right of that number. Other possible methods include allowing options for the vertical width of the marker, such as "thin", "medium", and "thick".

[0046] Reference numeral 65 denotes the shift time after touching, which corresponds to a predetermined time span d_t. In Fig. 6, it is set to 2 seconds. Reference numeral 66 denotes the recorded data display (display of time-series data). You can also listen to the audio. At the bottom, from the left, there are a button for starting playback, a bar showing the current playback time, and a number showing the total recording time. It is assumed that playback of the recorded video will not be stopped at any point during playback.

[0047] In the display field 62 for the score image, 67 is a stamp and 68 is a marker. The stamp 67 and the marker 68 are instruction information (instruction time and instruction position) in image data such as the score image. The marker 68 is assumed to be applied horizontally to the right on the score screen by tracing the screen with a finger or a special pen. The stamp 67 is automatically applied to the leftmost position where the marker 68 begins to be applied. The time when the marker 68 begins to be applied (the time corresponding to the position of the stamp 67) corresponds to the time T_ins (instruction time) when the instruction was given to the score image. Regarding the type of stamp 67, the type of marker 68 can be applied by selecting a type such as "Caution" or "Good" before applying the marker 68. The stamp 67 and the marker 68 can be added to the review points on the screen.

[0048] The stamp 67 and the marker 68 can be applied anywhere on the screen. For example, they may be applied not only to musical notes but also to various kinds of facial expressions written on musical scores. The time when the marker 68 is applied is recorded as the time T_ins. The stamp 67 and the marker 68 can also be erased after the instruction is input.

[0049] 7(a) and (b) are explanatory diagrams of the viewing mode (music score viewing mode), showing an image of the display unit 27 and the instruction input unit 28 operated by the user in the music score viewing mode. Consider what happens after inputting instructions to the score while playing the recorded image is finished. FIG. 7(a) shows an image of the display unit 27 and the instruction input unit 28 for the score image in the music score viewing mode, and compared to FIG. 6, A71 is in the music score viewing mode. To play the recorded image from the point where the instruction was input, simply press the area near the stamp on the score image in FIG. 7(a). This causes the recorded image to be played from the time corresponding to that point on the score image.

[0050] 7(b) shows an image of the display unit 27 and the instruction input unit 28 when playing back a recording from a specified time. B71 displays the playback screen of the recording. B72 displays, from the left, a button for starting playback, a bar displaying the current playback time, and a value indicating the total recording time.

[0051] B73 displays a "return to playback start position" button. Pressing this button returns to the time when playback begins when a specific stamp is pressed on the score screen, and playback can begin. In addition to the functions described here, B72 and B73 may also be equipped with basic functions for playing back recorded images, such as stop, pause, fast forward, slow playback, and frame forward. Playback of a recorded image may be stopped after a certain period of playback, for example 10 seconds. In addition, it may be configured so that only those 10 seconds can be played repeatedly. Alternatively, playback may continue until the end of the recording and then stop.

[0052] B74 displays a "Save current playback time" button. By pressing this button at any playback time, you can overwrite and save the time that will be played when you press a specific stamp on the score image. By incorporating this function, you can manually correct the playback start position. Pressing B75 "Return to score screen" returns you to the screen in Figure 7(a).

[0053] According to this embodiment, it is possible to provide an information processing device that can simply and accurately match time series data such as sheet music with image data related to the time series data, and control the playback of audio and video recording. EXAMPLES

[0054] Next, a second embodiment of the present invention will be described. The basic configuration of the information processing device of this embodiment is the same as the information processing device 21 of the first embodiment. Therefore, in this embodiment, only the configuration different from the first embodiment will be described, and the description common to the first embodiment will be omitted. In this embodiment, as in the first embodiment, a scene will be described in which an instructor gives guidance to a learner's performance when playing a keyboard instrument as an example of musical instrument performance. Figures 1 and 2 are also common to this embodiment.

[0055] 8(a) and (b) are flowcharts showing the processing of the information processing device in this embodiment, and correspond to FIG. 3(a), FIG. 3(b), and FIG. 3(c) in the first embodiment. The flowchart corresponding to FIG. 3(a) is the same in this embodiment. Therefore, the processing thereafter will be described. FIG. 8(a) corresponds to FIG. 3(b), and FIG. 8(b) corresponds to FIG. 3(c).

[0056] 8(a) is a flowchart showing the processing of the information processing device 21 based on the instructor's input. First, in step SA81, the information processing device 21 calculates, by the data processing unit 25, the elapsed time t_ins from the start of playback of the recorded image, which corresponds to the position information on the image where the instruction was made on the sheet music image. This step is the same as step SB301 in the first embodiment.

[0057] Next, in step SA82, the information processing device 21 uses the data processing unit 25 to calculate an estimated playback time t_est using a predetermined time width d_t. This step is the same as step SB302 in the first embodiment. Although not essential in this embodiment, the inclusion of step SA82 allows the initial value of matching, which determines the playback time of a recording, to be described later, to be determined more accurately.

[0058] Next, in step SA83, the information processing device 21 performs image recognition processing for the positions of each measure, note, etc. on the musical score. Note that instead of performing image recognition for each measure, note, etc. on the musical score, an electronic musical score in which the positions of each measure and note are read in advance may be prepared.

[0059] Next, in step SA84, the information processing device 21 acquires note arrangement data for the measures before and after the position (indicated position) of the stamp pressed on the score image. If the position of each measure on the score is recognized in the previous step, the position of the measure before and after the stamp can be determined by comparing it with the position of the stamp pressed on the score image. Within the range of the measure, each note on the score recognized in the previous step can be stored in order from the left as note data that also takes into account the time length of the note. Note that even if the position of the stamp pressed on the score image is out of the position that constitutes a measure on the score, it can be recognized as belonging to a nearby measure somewhere on the image.

[0060] Next, in step SA85, the information processing device 21 acquires musical note data by performing frequency analysis of sounds contained in the recorded video for a time period before and after the estimated playback time (t_est). Although t_est has been acquired in step SA82, the information processing device 21 analyzes sounds contained in the recorded video for a time period of, for example, 3 seconds before and after the t_est using frequency analysis, chroma vector analysis, machine learning, and other methods. This recognizes the pitch of the musical tones, their sounding timing, and the like. The musical tones recognized in this way can be stored as musical note data in chronological order, including the sounding timing.

[0061] Next, in step SA86, the information processing device 21 determines the playback time (t_mat) of the video recording corresponding to the position on the score where the stamp was pressed by matching. For matching, the note data acquired from the score in step SA84 is matched with the note data acquired from the video recording in step SA85, and the position where each data is most accurately associated is determined. A method such as DTW (dynamic time warping) can be used for the matching process. If the association can be determined, the playback time (t_mat) of the video recording corresponding to the position on the score can be determined.

[0062] It should be noted that various existing known techniques can be used for the image recognition, video recording analysis, and matching processing methods described in relation to FIG. 8(a). The above-mentioned methods are merely examples, and various other appropriate techniques can be used. With regard to the analysis of video recording and matching with image recognition results, the above-mentioned processing is performed only in the time period before and after t_est described in step SA82 (a predetermined period based on the designated time). As a result, the processing load can be significantly reduced compared to processing the entire score or video recording, and the actual effect is to play a role in fine-tuning the playback time of the video recording.

[0063] FIG. 8(b) is a flowchart showing a recording / playback instruction from the learner and the corresponding processing of the information processing device 21, assuming that the user is a learner, and is basically the same as FIG. 3(c).

[0064] First, in step SB81, the user starts displaying the musical score. At the same time, in step SB82, information processing device 21 starts the musical score viewing mode. Next, in step SB83, the user presses the position on the musical score image where the user made an instruction. In step SB84, information processing device 21 plays back the recording from a predetermined time t_mat using instruction information acquisition unit 24 and data processing unit 25. Next, in step SB85, the user ends displaying the musical score. At the same time, in step SB86, information processing device 21 ends the musical score viewing mode.

[0065] With regard to the process flow in this embodiment, which corresponds to Fig. 3(a), it is assumed that a stamp is pressed onto the score image by a human user using the instruction input unit 28, and the instruction information is acquired by the instruction information acquisition unit 24. However, this embodiment is not limited to this, and for example, artificial intelligence (AI) mounted on the instruction information acquisition unit 24 may be utilized. In other words, an automatic determination may be made based on image analysis of the recorded data or analysis of the sounds contained in the recorded data, and a stamp may be applied to the corresponding location on the score image.

[0066] FIG. 9 is an explanatory diagram showing the processing of the information processing device 21 in this embodiment on a time axis, and corresponds to FIG. 4 in the first embodiment. The contents other than 909 are the same as those in FIG. 4. 909 indicates the playback time t_mat obtained by matching. After the estimated playback time t_est 907 is obtained using the corresponding playback time t_ins 906, a predetermined time width d_t 908, and the like, the playback time t_mat obtained by matching is obtained as described with reference to FIGS. 8(a) and (b). This makes it possible to obtain the playback time of the recorded data corresponding to the position of the instruction on the musical score image more accurately than the estimated playback time t_est.

[0067] FIG. 10 is an explanatory diagram of data recorded by the information processing device 21 in this embodiment, and corresponds to FIG. 5 in the first embodiment. The columns from "No." to "Elapsed time from start of playback (estimated playback time t_est)" are the same as in FIG. 5. "Elapsed time from start of playback (playback time t_mat found by matching)" is the playback time of the recorded video corresponding to the position on the score where the stamp was pressed, determined by matching. This is as explained with reference to FIGS. 8(a) and (b). When a position on the score image that corresponds to the "position information (page number)" and "position information (position on the score (x, y))" is pressed, the recorded video data is played back from "Elapsed time from start of playback (playback time t_mat found by matching)".

[0068] Fig. 11 is an explanatory diagram of the input standby mode, showing an image of the display unit 27 and the instruction input unit 28 operated by the user in the input standby mode for the musical score, and corresponds to Fig. 6 in the first embodiment. Items other than 119 are the same as the corresponding contents in Fig. 6. 119 indicates the positions of the bars before and after the stamp pressed on the musical score image, as described with reference to Figs. 8(a) and (b).

[0069] First, it is determined which bar range the position of the stamp 117 falls within. It is also determined which note the position of the stamp 117 is closest to by the distance on the musical score image using the position of the stamp 117 and the position of each note. Then, for example, within a range of three bars in total, including one bar before and one bar after (a predetermined range), each note on the musical score contained therein is recognized and stored as note data. Furthermore, note data is obtained for the time period before and after the estimated playback time (t_est) of the video recording, and the playback time (t_mat) of the video recording corresponding to the position on the musical score where the stamp was pressed is determined by matching. This process has been described with reference to Figs. 8(a) and (b).

[0070] In this embodiment, the end position of the right end of the marker 118 may be recognized with respect to the range filled with the marker 118. Using that position, the same process as described above may be performed to determine the playback end time when playing back the recorded video. In this embodiment, the viewing mode is as described with reference to Figures 7(a) and (b) in the first embodiment.

[0071] As described above, the image data acquisition unit 23 acquires the position information of each measure or note of the musical score. The data processing unit 25 analyzes the sounds included in the time series data for a predetermined period based on the designated time, thereby associating the position information with the elapsed time from the playback start time of the time series data. In this way, the position adjustment is performed by analyzing only the predetermined period based on the designated time (the time before and after the designated position), so that the time can be changed according to the analysis result so that the deviation between the instruction and the timing in the time series data is reduced. Therefore, according to this embodiment, it is possible to provide an information processing device that can simply and sufficiently accurately control the playback of time series data in association with an image related to the time series data. EXAMPLES

[0072] Next, a third embodiment of the present invention will be described. The basic configuration of the information processing device of this embodiment is similar to that of the information processing device 21 of the first embodiment. Therefore, in this embodiment, only the configuration different from that of the first embodiment will be described, and the description of the same as that of the first embodiment will be omitted. In this embodiment, a scene will be described in which an instructor gives guidance to the performance of a student when performing a play based on a script.

[0073] 1 and 2 in the first embodiment are common to this embodiment. Figures 3(a), 3(b), 3(c), and 4 in the first embodiment are essentially common if the musical score is replaced with a script, so their explanation will be omitted.

[0074] Fig. 12 is an explanatory diagram of data recorded in the information processing device 21. The format is the same as Fig. 5 in the first embodiment, but specific numerical values ​​are shown for this embodiment in which a script is used instead of a musical score.

[0075] Fig. 13 is an explanatory diagram of the input standby mode, and shows an image of the display unit 27 and the instruction input unit 28 operated by the user in the script input standby mode. Fig. 13 shows the contents corresponding to this embodiment using a script instead of a musical score, in contrast to Fig. 6 in the first embodiment.

[0076] Reference numeral 131 denotes a display indicating that the screen is in a standby mode for inputting to the script. Reference numeral 132 denotes a display area for the script image. The page number is displayed at the bottom, and a scroll bar for page forwarding is located on the right edge. Stamps and markers, which will be described later, can be input to the script. Reference numeral 133 denotes the type of stamp. The type of stamp 133 can be selected by pressing the "Caution" or "Good" mark on the screen.

[0077] 134 is the horizontal width of the marker. As will be described later, it is assumed that the marker will be painted downwards on a vertically written script screen with the horizontal width specified. The horizontal width of the marker can be changed to make input easier depending on the original notation method of the script image. In this example, it is assumed that the number of pixels will be used for input, so it is set to "50". The actual horizontal width is displayed to the right of the number. Other options for the horizontal width of the marker may be "thin", "medium", "thick", etc.

[0078] Reference numeral 135 denotes the shift time after touching, which corresponds to a predetermined time span d_t. In this diagram, it is set to 2 seconds. Reference numeral 136 denotes the display of recorded data. Audio can also be heard. At the bottom, from the left, there are displayed a button for starting playback, a bar displaying the current playback time, and a number indicating the total recording time. It is assumed that playback of the recorded video will not be stopped at any point during playback.

[0079] In the display field 132, 137 is a stamp and 138 is a marker. The stamp 137 and the marker 138 are instruction information related to the instruction time and the instruction position in the image data. The marker 138 is assumed to be painted downward on the vertically written script screen by tracing the screen with a finger or a special pen. The stamp 137 is automatically attached to the topmost part where the marker 138 starts to be painted. The time when the marker 138 starts to be painted corresponds to the time T_ins (instruction time) when the instruction to the script image is given. Regarding the type of the stamp 137, the type of the marker 138 can be painted by selecting a type such as "Caution" or "Good" before painting the marker 138. The stamp 137 and the marker 138 can be added to the part to be reviewed on the screen. The stamp 137 and the marker 138 can be painted at any position on the screen.

[0080] 14(a) and (b) are explanatory diagrams of the viewing mode (script viewing mode), showing an image of the display unit 27 and the instruction input unit 28 operated by the user. In contrast to FIGS. 7(a) and (b) in the first embodiment, FIGS. 14(a) and (b) show a case where a script image is used instead of a musical score image as image data related to time-series data.

[0081] Let us consider what happens after inputting instructions to the script while playing back the recording. Fig. 14(a) shows an image of the script image display unit 27 and the instruction input unit 28 in script viewing mode, where A141 is in script viewing mode compared to Fig. 13. To play back the recording from the point where the instruction was input, simply press the area near the stamp on the script image in Fig. 14(a). This causes the recording to be played back from the time corresponding to that point on the script image.

[0082] Fig. 14(b) shows an image of the display unit 27 and the instruction input unit 28 when playing back a recording from a specified time. B141 displays the playback screen of the recording. B142 displays, from the left, a button for starting playback, a bar displaying the current playback time, and a value indicating the total recording time. B143 displays a "return to playback start position" button. Pressing this button allows you to return to the time that would have been played back if you had pressed a specified stamp on the script screen, and start playback.

[0083] B144 displays a "Save current playback time" button. By pressing this button at any playback time, the time that will be played when a specific stamp on the script image is pressed can be overwritten and saved as desired. By incorporating this function, it may be possible to configure the device so that the playback start position can be manually corrected. Pressing B145 "Return to script screen" returns to the screen in FIG. 14(a).

[0084] According to this embodiment, it is possible to provide an information processing device capable of simply and sufficiently accurately controlling the playback of audio and video in accordance with image data such as a script related to time-series data.

[0085] (Other Examples) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0086] According to each embodiment, it is possible to provide an information processing device, an information processing method, and a program that can easily and sufficiently accurately match the playback start position of time-series data to the designated position of image data.

[0087] The disclosure of each embodiment includes the following configurations and methods. (Configuration 1) A time series data acquisition unit that acquires a playback start time of the time series data; an image data acquisition unit that acquires image data related to the time series data; an instruction information acquisition unit that acquires instruction information relating to an instruction time and an instruction position in the image data during playback of the time series data; an information processing device comprising: a data processing unit that uses the instruction information to associate the instruction position with an elapsed time that indicates the time that has elapsed since the playback start time of the time series data that corresponds to the instruction time; (Configuration 2) 2. The information processing device according to configuration 1, wherein the time series data is audio data or video data. (Configuration 3) A recording unit that records the time series data, 3. The information processing device according to configuration 1 or 2, wherein the instruction information acquisition unit acquires the instruction information during playback of the time series data pre-recorded in the recording unit. (Configuration 4) A recording unit that records the time series data, 3. The information processing device according to configuration 1 or 2, wherein the instruction information acquisition unit acquires the instruction information while the time series data is being recorded in the recording unit. (Configuration 5) 5. The information processing device according to any one of configurations 1 to 4, wherein the image data is an image from which time-series information can be read. (Configuration 6) 6. The information processing device according to any one of configurations 1 to 5, wherein the image data is an image of a musical score or a script. (Configuration 7) 7. The information processing device according to any one of configurations 1 to 6, wherein the instruction information acquisition unit acquires one type of the instruction information from a plurality of selectable types. (Configuration 8) When the playback start time of the time series data is T_st, the elapsed time is t_ins, and the designated time is T_ins, the elapsed time t_ins is expressed as follows: t_ins = T_ins - T_st 8. The information processing device according to any one of configurations 1 to 7, wherein the calculation is performed by the following formula: (Configuration 9) 9. The information processing device according to any one of configurations 1 to 8, wherein the data processing unit sets a playback time of the time-series data corresponding to the designated position based on the elapsed time. (Configuration 10) When a predetermined time span based on the elapsed time is d_t, the elapsed time is t_ins, and the playback time of the time series data is t_est, the playback time t_est is expressed as follows: t_est=t_ins-d_t The information processing device according to configuration 9, wherein the calculation is performed by the following method. (Configuration 11) 11. The information processing device according to any one of configurations 1 to 10, further comprising a display unit that displays the image data. (Configuration 12) 11. The information processing device according to any one of configurations 1 to 10, further comprising an instruction input unit for receiving an instruction input for the image data. (Configuration 13) 13. The information processing apparatus according to configuration 12, wherein the data processing unit, when receiving the instruction input at the instruction position, reproduces the time series data from a time corresponding to the elapsed time. (Configuration 14) the image data is an image of a musical score, the image data acquisition unit acquires position information of each measure or note of the musical score, The information processing device according to any one of configurations 1 to 13, characterized in that the data processing unit correlates the position information with the elapsed time by analyzing sounds contained in the time series data for a predetermined period based on the specified time. (Method 1) Obtaining a playback start time of the time series data; acquiring image data relating to the time series data; acquiring instruction information relating to an instruction time and an instruction position in the image data during playback of the time series data; An information processing method comprising a step of using the instruction information to associate the instruction position with an elapsed time representing the time elapsed from the playback start time of the time series data corresponding to the instruction time. (Configuration 15) A program for causing a computer to execute the information processing method according to Method 1.

[0088] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0089] The time-series data in each embodiment may be, for example, data related to various musical performances, entertainment, or sports based on musical scores, scripts, choreography, or the like that describe a scenario in time series. [Explanation of symbols]

[0090] 21 Information processing equipment 22 Time series data acquisition section 23 Image data acquisition unit 24 Instruction information acquisition section 25 Data Processing Section

Claims

1. A time series data acquisition unit that acquires a playback start time of the time series data; an image data acquisition unit that acquires image data related to the time series data; an instruction information acquisition unit that acquires instruction information relating to an instruction time and an instruction position in the image data during playback of the time series data; an information processing device comprising: a data processing unit that uses the instruction information to associate the instruction position with an elapsed time that indicates the time that has elapsed since the playback start time of the time series data that corresponds to the instruction time;

2. 2. The information processing apparatus according to claim 1, wherein the time series data is audio data or video data.

3. A recording unit that records the time series data, 2 . The information processing apparatus according to claim 1 , wherein the instruction information acquisition unit acquires the instruction information during playback of the time-series data pre-recorded in the recording unit.

4. A recording unit that records the time series data, The information processing apparatus according to claim 1 , wherein the instruction information acquisition unit acquires the instruction information while the time series data is being recorded in the recording unit.

5. 2. The information processing apparatus according to claim 1, wherein the image data is an image from which time-series information can be read.

6. 2. The information processing apparatus according to claim 1, wherein the image data is an image of a musical score or a script.

7. The information processing apparatus according to claim 1 , wherein the instruction information acquisition unit acquires one type of the instruction information from a plurality of selectable types.

8. When the playback start time of the time series data is T_st, the elapsed time is t_ins, and the designated time is T_ins, the elapsed time t_ins is expressed as follows: t_ins=T_ins−T_st 2. The information processing apparatus according to claim 1, wherein the calculation is performed by the following formula:

9. The information processing apparatus according to claim 1 , wherein the data processing unit sets a playback time of the time-series data corresponding to the designated position based on the elapsed time.

10. When a predetermined time span based on the elapsed time is d_t, the elapsed time is t_ins, and the playback time of the time series data is t_est, the playback time t_est is expressed as follows: t_est=t_ins-d_t 10. The information processing apparatus according to claim 9, wherein the calculation is performed by the following formula:

11. 11. The information processing apparatus according to claim 1, further comprising a display unit that displays the image data.

12. 11. The information processing apparatus according to claim 1, further comprising an instruction input unit for receiving an instruction input for the image data.

13. 13 . The information processing apparatus according to claim 12 , wherein the data processing unit, when receiving the instruction input at the instruction position, reproduces the time-series data from a time corresponding to the elapsed time.

14. the image data is an image of a musical score, the image data acquisition unit acquires position information of each measure or note of the musical score, The information processing device according to any one of claims 1 to 10, characterized in that the data processing unit corresponds the position information to the elapsed time by analyzing sounds contained in the time series data for a predetermined period based on the specified time.

15. Obtaining a playback start time of the time series data; acquiring image data relating to the time series data; acquiring instruction information relating to an instruction time and an instruction position in the image data during playback of the time series data; An information processing method comprising a step of using the instruction information to associate the instruction position with an elapsed time representing the time elapsed from the playback start time of the time series data corresponding to the instruction time.

16. A program causing a computer to execute the information processing method according to claim 15.

Images