Information processing apparatus, information processing method, computer-readable medium and computer program product

Abstract

This invention provides an information processing apparatus, an information processing method, and a computer-readable medium. The information processing apparatus includes a processor that acquires video data capable of reproducing a video in which speech, images, and subtitles are synchronized in a time sequence, receives a deletion interval in the playback time of the video, and deletes a portion of the subtitles corresponding to the speech within the received interval and which constitutes at least a part of the subtitles from the images within the interval.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing methods, and computer-readable media. Background Technology

[0002] Japanese Patent No. 5676087 discloses a technology related to the reproduction of improved special effects modes, and particularly related to the display of subtitle information during the reproduction of special effects modes in video demonstrations. Summary of the Invention

[0003] Here, existing video editing devices have, for example, the following structure: even if a specific interval in the video's playback time is specified for deletion, only the images within that interval are deleted from the edited video, without deleting the subtitles displayed on those images. Therefore, this existing device requires not only deleting the images within the specified interval in the video's playback time but also deleting the subtitles displayed on those images.

[0004] Therefore, the purpose of this disclosure is to reduce the number of user jobs required for subtitle removal in video editing, compared to a structure that requires subtitle removal in addition to image removal in the video.

[0005] According to a first aspect of this disclosure, an information processing apparatus is provided, comprising a processor that acquires video data capable of reproducing a video in which speech, images, and subtitles are synchronized in a time sequence, receives a deletion interval in the playback time of the video, and deletes a portion of the subtitles corresponding to the speech within the received interval and being at least part of the subtitles from the images within the interval.

[0006] According to the second aspect of this disclosure, the processor provides a prompt indicating when the portion of the subtitles has been deleted before deleting it.

[0007] According to the third aspect of this disclosure, the processor displays the subtitles without deleting the portion of the subtitles, together with the subtitles when the portion of the subtitles has been deleted.

[0008] According to the fourth aspect of this disclosure, the processor prompts the partial subtitles in a form that can be determined relative to other subtitles.

[0009] According to the fifth aspect of this disclosure, the processor receives an adjustment of the deletion range in the subtitles disclosed before deleting the partial subtitles, and uses the received deletion range as the new partial subtitles.

[0010] According to the sixth aspect of this disclosure, the processor, together with the subtitles when the partial subtitles are deleted, prompts at least a portion of the image within the interval when the partial subtitles are deleted.

[0011] According to the seventh aspect of this disclosure, when the languages ​​of the speech and the subtitles are different, the processor translates the language of one of the speech and the language of the subtitles into the language of the other to determine the partial subtitles.

[0012] According to the eighth aspect of this disclosure, a computer-readable medium is provided, which stores a program that causes a computer to perform processing, in which video data capable of reproducing a video in which speech, images, and subtitles are synchronized in a time sequence is acquired, a deletion interval in the playback time of the video is received, and a partial subtitle corresponding to the speech in the received interval and being at least part of the subtitle is deleted from the image in the interval.

[0013] According to the ninth aspect of this disclosure, an information processing method is provided, wherein the following processing is performed in the information processing method: acquiring video data capable of reproducing a video in which speech, images, and subtitles are synchronized in a time sequence; receiving a deletion interval in the playback time of the video; and deleting a portion of the subtitles that corresponds to the speech in the received interval and is at least part of the subtitles from the images in the interval.

[0014] (Effect)

[0015] According to the first or ninth scheme, in video editing, compared to a structure that requires a subtitle deletion operation in addition to the image deletion operation in the video, the number of user operations required for subtitle deletion is reduced.

[0016] According to the second scheme, it is possible to confirm the content of the subtitles when some subtitles have been deleted before deleting them.

[0017] According to the third scheme, it is possible to compare the subtitle content before and after deletion.

[0018] According to the fourth scheme, compared with the structure where all the text in the suggested subtitles has the same form, it is easier to identify some subtitles.

[0019] According to the fifth scheme, it is possible to confirm the revealed subtitle content while adjusting the range to be deleted from the subtitle.

[0020] According to the sixth scheme, the image structure of the video after deletion can be understood together with the subtitle content when some subtitles have been deleted.

[0021] According to the seventh scheme, even if the languages ​​of the audio and the subtitles are different, it is possible to determine some of the subtitles.

[0022] According to the eighth scheme, in video editing, compared to a structure that requires a subtitle deletion task in addition to the image deletion task in the video, the number of user tasks required for subtitle deletion tasks is reduced. Attached Figure Description

[0023] Figure 1 A block diagram illustrating the hardware structure of an information processing device.

[0024] Figure 2 This is a flowchart illustrating the subtitle processing workflow.

[0025] Figure 3 This is an example of recreating what the video looked like before editing.

[0026] Figure 4 This is an example of editing a screen.

[0027] Figure 5 Compared to Figure 4 The editing screen shown illustrates the area to be deleted.

[0028] Figure 6 This is a flowchart illustrating the process of determining partial subtitles.

[0029] Figure 7 Compared to Figure 5 The editing screen shown illustrates the voice text.

[0030] Figure 8 Compared to Figure 7 The editing screen shown illustrates the finalized portion of the subtitles.

[0031] Figure 9 This is a flowchart illustrating the process of generating the confirmation screen.

[0032] Figure 10 Compared to Figure 8 The edit screen shown illustrates the text after deletion.

[0033] Figure 11 This is an example of a confirmation screen in the first embodiment.

[0034] Figure 12 This is an example of an editing screen following a button.

[0035] Figure 13 This is an example of the structure of an edited video.

[0036] Figure 14 This is an example of a confirmation screen in the second embodiment.

[0037] Figure 15 This is an example of a confirmation screen in the third embodiment.

[0038] Figure 16 This is an example of a confirmation screen in the third embodiment.

[0039] Figure 17 This is an example of a confirmation screen in the fourth embodiment. Detailed Implementation

[0040] The information processing apparatus 10 according to this embodiment will be described below.

[0041] (First Embodiment)

[0042] The information processing device 10 is a computer capable of performing video editing processing that synchronizes voice, images, and subtitles in a time sequence. As an example, the information processing device 10 may include general-purpose computer devices such as server computers or personal computers (PCs), or portable computer devices such as smartphones or tablet terminals.

[0043] Furthermore, the "deletion" of audio, images, or subtitles described below refers to processing where the user cannot identify the audio, images, or subtitles to be deleted from the reproduced video. Therefore, the aforementioned "deletion" includes the following processing: processing to eliminate the audio, images, or subtitles to be deleted; and processing to stop the output of the audio or the prompt of the image or subtitles without eliminating the audio, images, or subtitles to be deleted.

[0044] Next, the hardware structure of the information processing device 10 will be described.

[0045] Figure 1 This is a block diagram showing the hardware structure of the information processing device 10.

[0046] like Figure 1 As shown, the information processing device 10 has a control unit 20 that controls the operation of the information processing device 10. The control unit 20 connects the CPU 22 (Central Processing Unit), ROM 24 (Read Only Memory), RAM 26 (Random Access Memory), and input / output interface (I / O) 28 via a bus so that they can communicate with each other.

[0047] CPU 22 is the central processing unit, executing various programs or controlling various parts. Specifically, CPU 22 reads programs from ROM 24 and uses RAM 26 as the working area to execute the programs. CPU 22 performs control of the aforementioned structures and various arithmetic operations according to the programs recorded in ROM 24.

[0048] ROM 24 stores various programs and various data. Among these programs is an information processing program for enabling the information processing device 10 to perform video editing processes. Furthermore, the information processing program can be pre-installed on the information processing device 10, or it can be appropriately installed on the information processing device 10 by storing it on a non-volatile storage medium or by distributing it via a network. Examples of non-volatile storage media include CD-ROMs, optical disks, HDDs (Hard Disk Drives), DVD-ROMs, flash memory, memory cards, etc.

[0049] RAM26 is used as a temporary storage area for programs or data in the job area.

[0050] The I / O28 is connected to a storage unit 30, a display unit 32, and an operation unit 34.

[0051] As a storage unit 30, for example, HDD, SSD (=Solid State Drive), flash memory, etc. are used.

[0052] The storage unit 30 stores video data capable of reproducing videos in a time-series synchronized manner of audio, images, and subtitles. The video data is a file in video format, such as MP4 or AVI.

[0053] Then, the video data includes audio data for outputting the aforementioned speech, image data for prompting the aforementioned images, and subtitle data for prompting the aforementioned subtitles. The audio data is a file in audio format, such as AAC or MP3. The image data is a file in image format, such as JPEG or PNG. The subtitle data is described, for example, in TTML or WebVTT format.

[0054] Furthermore, the video data includes time data storing the playback time of the videos. This time data, in order to synchronize the audio, images, and subtitles of each video in a time sequence, includes data at the following positions: the start and end positions of each audio output; the start and end positions of each image prompt; and the start and end positions of each subtitle prompt.

[0055] The display unit 32 uses, for example, a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display. The display unit 32 has a touch panel integrated into it.

[0056] The operation unit 34 receives various instructions from the user of the information processing device 10. Then, the display unit 32 displays various information such as the result of the processing performed according to the instructions received from the user or notifications regarding the processing.

[0057] Next, using Figures 2 to 13 The video editing process performed by the information processing device 10 is explained.

[0058] Figure 2 This is a flowchart illustrating the subtitle processing procedure for acquiring or deleting subtitle data.

[0059] exist Figure 2 In step S10, after the CPU 22 retrieves the video data of the video to be edited (hereinafter referred to as the "pre-edit video") from the storage unit 30 according to the user's operation on the operation unit 34, it determines the deletion interval within the playback time of the pre-edit video. For example, the CPU 22 retrieves the playback time of the pre-edit video from the time data and receives input of the start and end positions of the deletion interval within the retrieved playback time. Then, it proceeds to step S11. Furthermore, the reception of the start and end positions can be achieved as follows: For example, the CPU 22 displays a line graph on the display unit 32 representing the playback time from the beginning to the end as line segments, and receives input of the start and end positions of the deletion interval within the total length of the line graph from the user.

[0060] In step S11, CPU 22 refers to the time data of the video before editing to obtain the number of subtitle intervals as the number of subtitles N (N is a variable). The subtitle interval is the interval between the start position of the start prompt subtitle and the end position of the end prompt subtitle. Furthermore, CPU 22 stores the value of variable N in Cn (n is a variable) representing the nth subtitle, and when the value of variable N is 2 or higher, arranges the subtitle data corresponding to Cn in ascending order of the start position of the subtitle interval. Then, proceed to step S12.

[0061] Next, CPU22 begins to perform a loop process of repeatedly executing steps S12 to S22.

[0062] In the first step S12, CPU22 sets the initial value of variable n, the end condition of the loop processing, and the increment value of variable n. For example, CPU22 sets the initial value of variable n to "1", sets the end condition of the loop processing to "n > N", and sets the increment value of variable n to "1".

[0063] Furthermore, in step S12 after the second iteration, after adding the increment value to the variable n, CPU22 determines whether the loop termination condition is met. Then, if CPU22 determines that the loop termination condition is met, it exits the loop and ends the process; if CPU22 determines that the loop termination condition is not met, it continues the loop and proceeds to step S13.

[0064] In step S13, CPU22 obtains the subtitle text as the subtitle text from the subtitle data corresponding to Cn, and obtains the subtitle range indicated by the subtitle data from the time data. Then, it proceeds to step S14.

[0065] In step S14, CPU22 determines whether the start position of the subtitle interval is after the deleted interval. If it is determined to be after the deleted interval (step S14: Yes), the loop process is exited and the process ends. On the other hand, if CPU22 determines that it is not after the deleted interval (step S14: No), the process proceeds to step S15.

[0066] In step S15, CPU22 determines whether the end position of the subtitle interval is before the deleted interval. If it is determined to be before the deleted interval (step S15: Yes), it proceeds to step S22. On the other hand, if CPU22 determines that it is not before the deleted interval (step S15: No), it proceeds to step S16.

[0067] In step S16, CPU22 determines whether the subtitle range is included within the deletion range. If it is determined to be included within the deletion range (step S16: Yes), the process proceeds to step S21. On the other hand, if CPU22 determines that the subtitle range is not included within the deletion range (step S16: No), the process proceeds to step S17.

[0068] In step S17, CPU22 determines whether the end position of the subtitle interval is included within the deletion interval. If it is determined to be included within the deletion interval (step S17: Yes), the process proceeds to step S20. On the other hand, if CPU22 determines that the subtitle interval is not included within the deletion interval (step S17: No), the process proceeds to step S18.

[0069] In step S18, CPU 22 determines whether the start position of the subtitle interval is included within the deletion interval. If it is determined to be included within the deletion interval (step S18: Yes), the process proceeds to step S20. On the other hand, if CPU 22 determines that the subtitle interval is not included within the deletion interval (step S18: No), the process proceeds to step S19.

[0070] In step S19, CPU22 determines whether the deleted section is included within the subtitle section. If it is determined to be included within the subtitle section (step S19: Yes), the process proceeds to step S20. On the other hand, if CPU22 determines that the deleted section is not included within the subtitle section (step S19: No), the process proceeds to step S22.

[0071] In step S20, CPU22 retrieves the subtitle data corresponding to Cn from the video data. Then, it proceeds to step S22.

[0072] In step S21, CPU22 deletes the subtitles indicated by the subtitle data corresponding to Cn. Then, it proceeds to step S22.

[0073] In step S22, CPU22 performs loop termination processing and proceeds to step S12, which is the start processing of the loop.

[0074] Next, using Figures 3 to 5 Specific examples of subtitle processing will be explained.

[0075] Figure 3 The scene shown is a recreation of the video before editing. The recreation time of the pre-edited video is 30 seconds. Furthermore, the pre-edited video consists of the following parts: including... Figure 3 (A) and Figure 3 Image 44 shown in (B) and Figure 3 The images of the second image 46 shown in (C); containing Figure 3 (A) shows the first subtitle 48. Figure 3 The second subtitle 50 shown in (B) and Figure 3 The subtitles of the third subtitle 52 shown in (C); and the audio expressing the content of each subtitle. The video is formed by synchronously reproducing each image 44, 46, each subtitle 48, 50, 52 and the audio in a time sequence.

[0076] Figure 4 This is an example of an editing screen displayed on the display unit 32 when editing a video before editing.

[0077] The user operates the operation unit 34 and inputs instructions for editing the video before editing. If the CPU 22 receives the instructions, it will prompt... Figure 4 The editing screen shown.

[0078] exist Figure 4 The editing screen shown includes: an image bar 80 displaying information related to each image in the video before editing; and a subtitle bar 82 displaying information related to each subtitle in the video before editing.

[0079] The image bar 80 displays the images that are shown during the playback of the pre-edited video, specifically the first image 44 and the second image 46. Furthermore, the upper part of the image bar 80 displays the display time for each image during the playback time of the pre-edited video. For example, during the playback time of the pre-edited video, the display time for the first image 44 is in the range of "00:00 to 00:20 (seconds)," and the display time for the second image 46 is in the range of "00:20 to 00:30 (seconds)."

[0080] The subtitle bar 82 displays the subtitles that were displayed during the playback of the video before editing, specifically subtitle 1 (48), subtitle 2 (50), and subtitle 3 (52). Furthermore, the lower part of subtitle bar 82 displays the subtitle range of each subtitle during the playback time of the video before editing.

[0081] The interval between "00:01 to 00:08 (seconds)" in the playback time of the pre-edited video of subtitle 48 is the subtitle interval. Within this interval, along with image 44, the text "Hello everyone, I am Reid, CEO of ABC Company" is displayed (see reference). Figure 3 (A)). The interval between "00:11 to 00:18 (seconds)" in the playback time of the pre-edited video of subtitle 50 is the subtitle interval. Within this interval, together with image 44 of image 1, the text "Explanation of the performance in the fourth quarter of 2020" is displayed (see reference). Figure 3 (B)). The interval between "00:21 to 00:28 (seconds)" in the playback time of the pre-edited video of subtitle 52 is the subtitle interval. Within this interval, together with image 46 of the second section, the text "Sales in the fourth quarter reached US$11.4 million, a significant increase" is displayed (see reference). Figure 3 (C)).

[0082] Figure 5 Compared to Figure 4 The editing screen shown illustrates the area to be deleted.

[0083] exist Figure 5 When specifying a deletion range in the editing screen, such as Figure 5 As shown, the deleted intervals are indicated by dashed lines. Figure 5In the example shown, the deletion interval is the interval between the playback time "00:05 to 00:20 (seconds)". Alternatively, the start and end positions of the deletion interval can be set as follows: a time bar (not shown) representing the playback time of the video as line segments from the beginning to the end is displayed on the editing screen on the display unit 32, and the user can specify any time on that time bar.

[0084] Alternatively, it may not be necessary to display the image bar 80 and the subtitle bar 82 in the editing screen. For example, the start and end positions of the deletion range can be specified by the user directly inputting values, and input from the user can be received.

[0085] Here, the end position of the subtitle interval of subtitle 48 is included within the deletion interval, the subtitle interval of subtitle 50 is included within the deletion interval, the subtitle interval of subtitle 52 is not included within the deletion interval, and the beginning position of the subtitle interval of subtitle 52 is after the deletion interval. In this case, as processing subtitle 48, CPU 22 in... Figure 2 In step S20, the subtitle data of the first subtitle 48 is obtained from the video data, and partial subtitle determination processing and confirmation screen generation processing are performed as described later. Furthermore, as part of the processing of the second subtitle 50, the CPU 22... Figure 2 The second subtitle 50 is deleted in step S21 shown. Furthermore, as processing the third subtitle 52, the CPU 22... Figure 2 The flowchart shown does not involve acquiring or deleting the subtitle data for subtitle 52; the process ends there.

[0086] As a result of the above, subtitle 2 50 was deleted, and subtitle 3 52 remained unchanged. Therefore, in the following... Figure 6 In the following figures, the deleted object image 54, which is shown together with the first image 44 within the deletion range and is indicated by the first caption 48 (see reference). Figure 3 The process of deleting some subtitles in (A) will be explained.

[0087] Figure 6 This is a flowchart illustrating the process of determining partial subtitles that correspond to the speech output within the deletion interval and serve as at least a part of the subtitles displayed within the deletion interval.

[0088] exist Figure 6 In step S30, as shown, CPU 22 obtains the audio data of the audio output within the deletion interval from the video data, and obtains the audio interval from the time data, wherein the audio interval is the interval between the start position of the audio output and the end position of the audio output. Then, it proceeds to step S31.

[0089] In step S31, CPU22 will Figure 2 The number of subtitle data obtained in step S20 is stored in the number of subtitles M (M is a variable) that stores the number of subtitle data. When the value of variable M is 2 or higher, the subtitle data corresponding to Cm (m is a variable) representing the m-th subtitle are arranged in order from earliest to latest according to the starting position of the subtitle interval. Then, proceed to step S32.

[0090] Next, CPU22 begins to perform a loop process of repeatedly executing steps S32 to S36.

[0091] In the first step S32, CPU22 sets the initial value of variable m, the end condition of the loop processing, and the increment value of variable m. For example, CPU22 sets the initial value of variable m to "1", sets the end condition of the loop processing to "m > M", and sets the increment value of variable m to "1".

[0092] Furthermore, in step S32 after the second iteration, after adding the increment value to the variable m, CPU22 determines whether the loop termination condition is met. Then, if CPU22 determines that the loop termination condition is met, it exits the loop and ends the process; if CPU22 determines that the loop termination condition is not met, it continues the loop process and proceeds to step S33.

[0093] In step S33, CPU22 obtains the subtitle text from the subtitle data corresponding to Cm, and obtains the subtitle range indicated by the subtitle data from the time data. Then, it proceeds to step S34.

[0094] In step S34, CPU22 acquires the speech-text of the overlapping portion of the deleted section and the subtitle section, converted into text. The aforementioned "subtitle section" refers to the subtitle section indicated by subtitle data corresponding to Cm. Here, CPU22 uses known speech recognition technology to perform speech recognition processing, converting the speech data into text. Then, it proceeds to step S35.

[0095] In addition, the range of voice text obtained in step S34 includes the following three types.

[0096] (1) When the deletion interval is contained within the subtitle interval, the range from the start position to the end position of the deletion interval.

[0097] (2) The range from the beginning of the subtitle interval to the end of the deletion interval, where only the beginning of the subtitle interval is contained within the deletion interval and the end of the subtitle interval is outside the deletion interval.

[0098] (3) The range from the beginning of the deletion interval to the end of the subtitle interval when only the end position of the subtitle interval is contained within the deletion interval and the beginning position is outside the deletion interval.

[0099] In step S35, CPU22 determines a partial subtitle that corresponds to the speech output within the deletion interval and is at least a part of the subtitle displayed within the deletion interval.

[0100] Here, the situation of "partial correspondence between subtitles and audio" includes two cases: the audio text and the subtitle text are identical; and the audio text and the subtitle text are considered identical based on a predefined similarity score. For example, the aforementioned similarity score can be categorized as language similarity. When comparing the languages ​​of the audio and the subtitle, if the comparison exceeds a predefined threshold, they are considered similar. For example, it can be configured such that if the meaning associated with the languages ​​of the audio and the subtitle exceeds a predefined threshold, they are considered similar. Then, if the CPU 22 determines that the audio and subtitle text are similar based on the aforementioned similarity score, they are considered identical based on that similarity score.

[0101] As an example of "partial subtitle-voice correspondence", consider the following scenarios: The voice text and the subtitle text are both "I am Reid, CEO of ABC Company"; the voice text is "Hello everyone" and the subtitle text is also "Hello everyone"; the voice text is "I am Reid, CEO of ABC Company" and the subtitle text is "I am Reid, Chief Executive Officer of ABC Company"; and so on.

[0102] The partial subtitles in step S35 are the text portion of the subtitles displayed along with the image within the deletion interval. The image within the deletion interval is determined by comparing the subtitle text obtained in step S33 with the audio text obtained in step S34. Details of the process for determining the partial subtitles will be described later. Then, proceed to step S36.

[0103] In step S36, CPU22 performs loop end processing and enters step S32, which is the start processing of loop processing.

[0104] Next, using Figure 7 as well as Figure 8 Specific examples of the processing of certain subtitles are explained.

[0105] Figure 7 Compared to Figure 5 The editing screen shown illustrates the conversion of speech data of the overlapping portion of the deleted section and the subtitle section of subtitle 48 into speech text.

[0106] exist Figure 7 The editing screen shown now displays a voice bar 84 that provides information related to the audio in the video before editing.

[0107] Here, the audio in the pre-edited video includes audio displaying the content of subtitle 1 (48), subtitle 2 (50), and subtitle 3 (52). Then, in Figure 7 In one example shown, the deletion interval is "00:05~00:20 (seconds)", and the subtitle interval for subtitle 48 is "00:01~00:08 (seconds)". In this case, CPU22 in Figure 6 In step S34, the audio text ranging from the start of the deletion interval to the end of the subtitle interval is obtained. Therefore, the audio text ranging from the start of the deletion interval to the end of the subtitle interval is... Figure 7 The voice bar 84 shown contains the message "I am Reid, the CEO of ABC".

[0108] Figure 8 Compared to Figure 7 The editing screen shown illustrates the finalized portion of the subtitles.

[0109] exist Figure 8 In the process, by comparing the audio text of the first subtitle 48 with the overlapping part of the deleted section and the subtitle section of the first subtitle 48 by the CPU22, the underlined text "I am Reid, CEO of ABC Company" in a part of the text of the first subtitle 48 in the subtitle bar 82 is identified as a partial subtitle.

[0110] Figure 9 This is a flowchart illustrating the process of generating a confirmation screen for confirming the deletion of subtitles.

[0111] exist Figure 9 In step S40, CPU 22 obtains the audio data of the audio output within the deletion interval from the video data, and obtains the audio interval of the audio from the time data. Then, it proceeds to step S41.

[0112] In step S41, CPU22 will Figure 2 The number of subtitle data obtained in step S20 is stored in the number of subtitles M (M is a variable) that stores the number of subtitle data. When the value of variable M is 2 or higher, the subtitle data corresponding to Cm (m is a variable) representing the m-th subtitle are arranged in order from earliest to latest according to the starting position of the subtitle interval. Then, proceed to step S42.

[0113] Next, CPU22 begins to perform a loop process of repeatedly executing steps S42 to S48.

[0114] In the first step S42, CPU22 sets the initial value of variable m, the end condition of the loop processing, and the increment value of variable m. For example, CPU22 sets the initial value of variable m to "1", sets the end condition of the loop processing to "m > M", and sets the increment value of variable m to "1".

[0115] Furthermore, in step S42 after the second iteration, after adding the increment value to the variable m, CPU22 determines whether the loop termination condition is met. Then, if CPU22 determines that the loop termination condition is met, it exits the loop and ends the process; if CPU22 determines that the loop termination condition is not met, it continues the loop process and proceeds to step S43.

[0116] In step S43, CPU22 obtains the subtitle text from the subtitle data corresponding to Cm, and obtains the subtitle range indicated by the subtitle data from the time data. Then, it proceeds to step S44.

[0117] In step S44, CPU22 determines whether the subtitle text obtained in step S33 and the partial subtitles determined in step S35 meet the consistency condition. If CPU22 determines that the consistency condition is met (step S44: Yes), proceed to step S47. On the other hand, if CPU22 determines that the consistency condition is not met (step S44: No), proceed to step S45. As an example, if the subtitle text and the partial subtitle text data are consistent, CPU22 determines that the consistency condition is met. For example, if the subtitle text and the partial subtitle are "I am Reid, CEO of ABC Company", CPU22 determines that the consistency condition is met. However, if the subtitle text is "Hello everyone, I am Reid, CEO of ABC Company" and the partial subtitle is "I am Reid, CEO of ABC Company", then the consistency condition is not met.

[0118] In step S45, CPU22 generates a subtitle for the deleted portion of the subtitle. For example, if the subtitle displayed along with the image in the deleted area is "Hello everyone, I am Reid, CEO of ABC Company," and the partial subtitle is "I am Reid, CEO of ABC Company," CPU22 generates the subtitle "Hello everyone" as the subtitle for the deleted portion. Then, proceed to step S46.

[0119] In step S46, CPU22 updates at least one of the start and end positions of the subtitle interval of the subtitle to be deleted. Then, proceed to step S47.

[0120] In addition, there are three types of update modes for the subtitle range in step S46.

[0121] (1) When the deletion interval is contained within the subtitle interval, the start position of the subtitle interval remains unchanged before and after the update, and the end position after the update becomes the position corresponding to the deletion interval, which is earlier than the end position before the update.

[0122] (2) In the case where only the start position of the subtitle interval is contained within the deletion interval and the end position is outside the deletion interval, the updated start position of the subtitle interval becomes the position of the time before the start position before the update, which is the time of the overlapping part of the deletion interval and the subtitle interval. The end position remains unchanged before and after the update.

[0123] (3) If only the end position of the subtitle interval is contained within the deletion interval and the start position is outside the deletion interval, the start position of the subtitle interval does not change before and after the update. The updated end position becomes the position that advances the time of the overlapping part of the deletion interval and the subtitle interval from the end position before the update.

[0124] In step S47, CPU22 generates a confirmation screen for verifying the deletion of the subtitle. Then, it proceeds to step S48. Details of the confirmation screen will be described later.

[0125] In step S48, CPU22 performs loop end processing and enters step S42, which is the start processing of loop processing.

[0126] Next, using Figure 10 as well as Figure 11 A specific example of the confirmation screen generation process is explained.

[0127] Figure 10 Compared to Figure 8 The editing screen shown illustrates the deleted subtitle 56 after part of the subtitle was removed from the deleted object image 54.

[0128] exist Figure 10 From Figure 8 The text displayed in the subtitle bar 82 was changed. The part of the subtitle "Hello, I am Reid, CEO of ABC Company" that was displayed together with the first subtitle 48 "Hello, I am Reid, CEO of ABC Company" was deleted as part of it, and the "Hello" after "I am Reid, CEO of ABC Company" became the deleted subtitle 56.

[0129] Figure 11 This is an example of the confirmation screen in the first embodiment.

[0130] like Figure 11As shown, the confirmation screen on the display unit 32 displays the following information: before editing 60, after editing 62, Yes button 64, No button 66, and Cancel button 68. The before editing information 60 includes the text portion and subtitle range of the first subtitle 48, and the after editing information 62 includes the text portion and subtitle range of the deleted subtitle 56.

[0131] Here, if the operation is button 64, a portion of the subtitles are deleted from the image to be deleted 54, the subtitle displayed along with the first image 44 is changed to the deleted subtitle 56, and the image and audio are changed to the deleted content of the deleted range. In the example above, the CPU 22 updates the time data of the video before editing to the deleted content of the deleted range by operating button 64, changes the prompt time of the first image 44 and the second image 46 by deleting images within the deleted range, changes the subtitle displayed along with the first image 44 to the deleted subtitle 56 by deleting a portion of the subtitles, and stops outputting audio by deleting audio within the deleted range.

[0132] As described above, in the first embodiment, before deleting a portion of the subtitles from the deletion target image 54, the CPU 22 displays a confirmation screen indicating the subtitles 56 after deletion. Therefore, according to the first embodiment, it is possible to confirm the subtitle content when a portion of the subtitles has been deleted before deleting the subtitles.

[0133] Furthermore, in the first embodiment, the CPU 22 displays the subtitles (the first subtitle 48 in the example above) on the confirmation screen along with the deleted subtitles 56, showing the subtitles when the deleted portion of the subtitles is not removed. Therefore, according to the first embodiment, it is possible to compare the subtitle content before and after deletion.

[0134] In addition, Figure 11 In the confirmation screen, the original information 60 is displayed along with the edited information 62, but it can also be displayed relative to the original information 60, reflecting the editing resume. For example, it can be displayed by annotating the "I am Reid, CEO of ABC Company" section of "Hello everyone, I am Reid, CEO of ABC Company" as described below. Figure 15 as well as Figure 16 The double strikethrough shown, along with the information 60 before editing reflecting the editing process, allows users to understand the original and edited subtitles, as well as the edited sections.

[0135] Figure 12 This is an example of an editing screen following button 64.

[0136] like Figure 12As shown, after pressing the "Yes" button 64, the prompt time for the first image 44 changes to the interval of "00:00 to 00:05 (seconds)," and the prompt time for the second image 46 changes to the interval of "00:05 to 00:15 (seconds)." Furthermore, after pressing the "Yes" button 64, "00:01 to 00:05 (seconds)" becomes the subtitle interval for the deleted subtitle 56 and the audio interval for the content of the deleted subtitle 56, and "00:06 to 00:13 (seconds)" becomes the subtitle interval for the third subtitle 52 and the audio interval for the content of the third subtitle 52.

[0137] In addition, Figure 11 If the "No" button 66 is pressed on the confirmation screen, some subtitles will not be deleted from the image 54 to be deleted. The subtitles displayed together with the first image 44 will remain in the state of the first subtitle 48. The image and audio will be changed to the content after deletion of the deletion range. If the "Cancel" button 68 is pressed, a series of video editing processes will be canceled.

[0138] Figure 13 The scene depicts the reproduction of an edited video (hereinafter referred to as "edited video"). The reproduction time of the edited video decreases to 15 seconds as the deleted interval is removed. Furthermore, the edited video's playback time decreases from [previous time] as the deleted interval is removed. Figure 3 The deleted portion of the caption in image 54 shown in (A) is as follows: Figure 13 As shown in (A), the caption displayed along with the first image 44 is changed to the deleted caption 56. Furthermore, the edited video is removed from the video's playback time as the deleted section is deleted. Figure 3 The subtitle range of the second subtitle 50 shown in (B), and as Figure 13 As shown in (B), the subtitle range of subtitle 52 starts from... Figure 3 The video shown in (C) has been modified before editing.

[0139] Here, existing video editing devices have, for example, the following structure: even if a specific interval in the video's playback time is specified for deletion, only the images within that interval are deleted from the edited video, without deleting the subtitles displayed on those images. Therefore, this existing device requires not only deleting the images within the specified interval in the video's playback time but also deleting the subtitles displayed on those images.

[0140] In contrast, in the first embodiment, the CPU 22 acquires video data, receives a deletion interval from the playback time of the video based on the acquired video data, and deletes a portion of the subtitles within the received deletion interval from the deletion target image 54. Through these structures, in the first embodiment, the deletion of the image and the subtitles within the deletion interval are performed by receiving the deletion interval.

[0141] In summary, according to the first embodiment, in video editing, compared to a structure that requires a subtitle deletion operation in addition to the image deletion operation in the video, the number of user operations required for subtitle deletion is reduced.

[0142] In the first embodiment, the CPU 22 displays the deleted subtitle 56 on the confirmation screen before deleting part of the subtitle from the object to be deleted image 54. However, it is not limited to this. It is also possible to delete part of the subtitle from the object to be deleted image 54 without displaying the deleted subtitle 56, based on the fact that the deletion range has been received.

[0143] In the first embodiment, the CPU22 displays the subtitles without deleting the subtitles along with the deleted subtitles 56 on the confirmation screen. However, it is not limited to this. It may also display the deleted subtitles 56 on the confirmation screen, but not the subtitles without deleting the subtitles.

[0144] The file format of the video data, audio data, image data, and subtitle data described in the first embodiment is an example, but other file formats may also be used.

[0145] In the first embodiment, in Figure 6 In step S34, the CPU 22 acquires the speech text obtained by converting the speech of the overlapping portion of the deletion interval and the subtitle interval of the subtitle displayed by the subtitle data corresponding to Cm into text. However, it is not limited to this and can also be configured as follows: by storing the speech text obtained by converting the speech of the video before editing into text in the storage unit 30, the CPU 22 does not need to convert the speech as the object into text, but acquires the speech text of the speech as the object from the storage unit 30. In this case, the "subtitle that corresponds to the speech output in the deletion interval and is displayed in the deletion interval" that becomes part of the subtitle in step S35 is determined by comparing the following two texts: the speech text obtained by the CPU 22 by converting the speech as the object into text and the speech text of the speech as the object acquired by the CPU 22 from the storage unit 30; and the subtitle text acquired in step S33.

[0146] In the first embodiment, the data used to prompt images in the video and the data used to prompt subtitles in the video are respectively treated as image data and subtitle data, and are thus considered as separate data. However, this is not a limitation; the data used to prompt images in the video and the data used to prompt subtitles in the video can also be treated as a single set of data, and then separated into separate data during video editing.

[0147] In the first embodiment, the subtitles in the video are displayed in units of sentences, but it is not limited to this; the subtitles can also be displayed slowly in units of one character.

[0148] The first embodiment is described Figure 2 , Figure 6 as well as Figure 9 The processing order shown in the flowchart is an example. As long as the processing result remains unchanged, the processing order of the flowchart can be appropriately modified. The first embodiment is... Figure 6 In the flowchart shown, after CPU 22 obtains the subtitle text from the subtitle data corresponding to Cm in step S33, it obtains the speech text of the overlapping portion of the deleted section and the subtitle section indicated by the subtitle data corresponding to Cm in step S34. For example, it can also be configured as follows: after CPU 22 determines the speech of the deleted section and obtains the speech text of that speech, it obtains the subtitle text corresponding to that speech text.

[0149] (Second Implementation)

[0150] Next, regarding the second embodiment, the parts that are repeated in other embodiments will be omitted or simplified in the description.

[0151] Figure 14 This is an example of a confirmation screen in the second embodiment.

[0152] like Figure 14 As shown, CPU 22 displays a portion of the subtitle in the confirmation screen in a form that can be determined in the first subtitle 48, which is displayed as pre-editing information 60. For example, in Figure 14 In the confirmation screen shown, the portion of the subtitle "I am Reid, CEO of ABC Company" in subtitle 48, "Hello everyone, I am Reid, CEO of ABC Company," is circled with a dotted line.

[0153] With the above structure, according to the second embodiment, compared to a structure where all the text in the subtitles displayed on the confirmation screen has the same form, it is easier to identify some subtitles.

[0154] In the second embodiment, the CPU 22 prompts a portion of the subtitles in the first subtitle 48, which is prompted as pre-editing information 60, in a form that can be determined relative to other subtitles. However, it is not limited to this; it may also prompt a portion of the subtitles in the deleted subtitle 56, which is prompted as post-editing information 62, in a form that can be determined relative to other subtitles.

[0155] (Third implementation)

[0156] Next, regarding the third embodiment, the parts that are repeated in other embodiments will be omitted or simplified in the description.

[0157] The third embodiment is configured such that: before deleting a portion of the subtitle, the CPU22 receives the adjustment of the deletion range as edited information 62 in the deleted subtitle 56 in the confirmation screen, and uses the received deletion range as the new portion of the subtitle.

[0158] Figure 15 This is an example of a confirmation screen in the third embodiment.

[0159] exist Figure 15 In the confirmation screen shown, a portion of the text in the deleted subtitle 56, which is displayed as edited information 62, is marked with a double strikethrough. Furthermore, in this confirmation screen, an adjustment mark 70, indicated by an arrow, is displayed below the "A" which is the first character marked with the double strikethrough. In this confirmation screen, the area to be deleted as part of the subtitle can be adjusted by moving the adjustment mark 70 left or right.

[0160] Figure 16 The adjustment mark 70 is shown from Figure 15 The confirmation screen shows the state after it has been moved.

[0161] exist Figure 16 In the confirmation screen shown, adjust marker 70 from Figure 15 The confirmation screen moves to the left, and the first character marked with a double strikethrough becomes "OK". Therefore, in this confirmation screen, a double strikethrough is added to "OK, I am Reid, CEO of ABC Company" in the deleted subtitle 56, which is displayed as edit information 62. In this case, CPU 22 determines "OK, I am Reid, CEO of ABC Company" as the received deletion range as the new partial subtitle. Furthermore, as the range of the partial subtitle changes, CPU 22 changes the subtitle interval of the deleted subtitle 56, which is displayed as edit information 62, to "00:01~00:02 (seconds)".

[0162] With the above structure, according to the third embodiment, it is possible to check the disclosed subtitle content while adjusting the range to be deleted from the subtitle.

[0163] like Figure 15 as well as Figure 16 As shown, although there is a difference in the third embodiment where a portion of the deleted subtitle 56 is marked with a double strikethrough, the confirmation screen displays the same text, "Hello everyone, I am Reid, CEO of ABC," in both the first subtitle 48 and the deleted subtitle 56. Furthermore, in the first and second embodiments described above, the confirmation screen displays different text in the first subtitle 48 and the deleted subtitle 56; specifically, the deleted subtitle 56 displays the text indicating that a portion of the subtitle was deleted from the first subtitle 48 (see [reference]). Figure 11 as well as Figure 14 As mentioned above, the deleted subtitle 56 displayed on the confirmation screen can be the text indicating that part of the subtitle has been deleted from subtitle 48, or it can be displayed in a form that can be determined relative to other subtitles.

[0164] In the third embodiment, the CPU22 receives the adjustment of the deletion range in the deleted subtitle 56 in the confirmation screen as post-editing information 62 before deleting part of the subtitle. However, it is not limited to this. It may also receive the adjustment of the deletion range in the first subtitle 48 in the confirmation screen as pre-editing information 60 before deleting part of the subtitle.

[0165] (Fourth implementation)

[0166] Next, regarding the fourth embodiment, the parts that are repeated in other embodiments will be omitted or simplified in the description.

[0167] Figure 17 This is an example of a confirmation screen in the fourth embodiment.

[0168] like Figure 17 As shown, in this confirmation screen, CPU 22 displays the deleted image 72 along with the content of the deleted subtitle 56 and the subtitle range as edited information 62 when a portion of the subtitle was deleted from the deleted object image 54. For example, in Figure 17 The confirmation screen shown includes a message along with image 44 indicating that the deleted text 56 “Hello everyone” appears as the deleted image 72.

[0169] With the above structure, according to the fourth embodiment, the image structure of the video after deletion can be understood together with the subtitle content when deleting some subtitles.

[0170] In the fourth embodiment, the CPU22, together with the content of the deleted subtitle 56 and the subtitle range, indicates that there is one image as the deleted image 72. However, it is not limited to this and may also indicate multiple images as the deleted image 72.

[0171] (Fifth implementation)

[0172] Next, regarding the fifth embodiment, the parts that overlap with other embodiments will be omitted or simplified in the description.

[0173] Unlike other embodiments, the fifth embodiment is an embodiment of an information processing apparatus 10 that performs editing processing of videos with different languages ​​for audio and subtitles.

[0174] The fifth embodiment is configured such that the CPU22 translates the language of one side of the audio in the video and the language of the subtitles into the language of the other side to determine the subtitles.

[0175] The following example illustrates how to set the audio in a video to English, the subtitles to Japanese, and the English audio output within the deleted section to "I'm ABC company CEO Red. I will explain the financial results for the fourth quarter of 2020."

[0176] When the audio and subtitles in the video are in different languages, CPU22 performs... Figure 6 In step S34, the English speech of the deleted interval and the overlapping part of the subtitle interval of the subtitle prompted by the subtitle data corresponding to Cm are converted into English text, and then the English text is converted into Japanese speech text using known translation technology.

[0177] For example, after converting the English speech into English text "I'm ABC company CEO Red. I will explain the financial results for the fourth quarter of 2020." in step S34, CPU22 converts the English text into Japanese speech text "I am ABC company CEO Red, and I will explain the financial results for the fourth quarter of 2020."

[0178] Next, CPU22 compares the aforementioned Japanese audio text with the Japanese subtitle text obtained in step S33, and identifies "I am Reid, CEO of ABC Company" as a partial subtitle. Furthermore, since the process after identifying the partial subtitle is the same as in other implementation methods, it is omitted.

[0179] With the above structure, according to the fifth embodiment, even if the languages ​​of the audio and subtitles in the video are different, some subtitles can be identified.

[0180] In the fifth embodiment, the audio in the video is set to English and the subtitles are set to Japanese, but it is not limited to this. The audio in the video can also be set to Japanese and the subtitles to English. The combination of the languages ​​of the audio and subtitles in the video is not limited to Japanese and English.

[0181] In the fifth embodiment, the languages ​​of the audio and subtitles in the video are preset, but it is not limited to this. Alternatively, the languages ​​of the audio and subtitles in the video can be obtained and set each time the video is edited.

[0182] In addition, the above-described embodiments can be combined appropriately.

[0183] In the above embodiments, the CPU 22 of the information processing device 10 is an example of a processor. Moreover, the processor refers to a processor in a broad sense, including general-purpose processors (e.g., CPU:=Central Processing Unit, etc.) or special-purpose processors (e.g., GPU:=Graphics Processing Unit, ASIC:=Application Specific Integrated Circuit, FPGA:=Field Programmable Gate Array, programmable logic device, etc.).

[0184] Furthermore, the actions of the aforementioned processor can be performed not only by a single processor, but also by multiple processors located in physically separate positions working together. Moreover, the order of the actions of the aforementioned processors is not limited to the order described in the above embodiments, and can be appropriately changed.

Claims

1. An information processing device comprising a processor, The processor acquires video data capable of reproducing video in which speech, images, and subtitles are synchronized in time. During video editing, the deletion intervals within the video's playback time are received. Obtain the speech-text by converting the repeated portions of the deleted interval and the subtitle interval into text. The portion of the subtitle that corresponds to the received speech text within the deletion interval and is at least part of the subtitle is deleted from the image within the deletion interval. By receiving the deletion range, the image and subtitles within the deletion range are deleted. Before deleting the portion of the subtitle corresponding to the received audio text within the deletion range from the image within the deletion range, the processor displays the deleted subtitle in a confirmation screen for confirming the deletion of the subtitle. Before deleting the portion of the subtitle, the processor receives adjustments to the deletion range in the deleted subtitle displayed on the confirmation screen as post-editing information. The received deletion range will be used as the new partial subtitle.

2. The information processing apparatus according to claim 1, wherein, The processor displays the subtitles without deleting the deleting portion of the subtitles, along with the subtitles when the deleting portion of the subtitles has been removed.

3. The information processing apparatus according to claim 2, wherein, The processor prompts the partial subtitles in a form that can be determined relative to other subtitles.

4. The information processing apparatus according to any one of claims 1 to 3, wherein, The processor, along with the subtitles indicating when the partial subtitles were deleted, displays a message indicating that at least a portion of the image within the specified interval was deleted.

5. The information processing apparatus according to claim 1, wherein, If the languages ​​of the audio and the subtitles are different, the processor translates one of the languages ​​of the audio and the subtitles into the other language to determine the partial subtitles.

6. A computer-readable medium storing a program that causes a computer to perform processing. In the process, Acquire video data capable of reproducing videos in which speech, images, and subtitles are synchronized in time. During video editing, the deletion intervals within the video's playback time are received. Obtain the speech-text by converting the repeated portions of the deleted interval and the subtitle interval into text. The portion of the subtitle that corresponds to the received speech text within the deletion interval and is at least part of the subtitle is deleted from the image within the deletion interval. By receiving the deletion range, the image and subtitles within the deletion range are deleted. Before deleting the portion of the subtitle corresponding to the received audio text within the deletion range from the image within the deletion range, a confirmation screen for confirming the deletion of the subtitle displays the deleted subtitle. Before deleting the aforementioned portion of the subtitles, the deletion range is adjusted in the post-editing subtitles displayed on the confirmation screen as post-editing information. The received deletion range will be used as the new partial subtitle.

7. An information processing method, wherein, The following processing is performed in this information processing method: Acquire video data capable of reproducing videos in which speech, images, and subtitles are synchronized in a time sequence; When editing a video, the deletion intervals to be deleted are received from the playback time of the video; Obtain the speech-text by converting the speech of the repeated portions of the deleted interval and the subtitle interval into text; as well as The portion of the subtitle that corresponds to the received speech text within the deletion interval and is at least part of the subtitle is deleted from the image within the deletion interval. By receiving the deletion range, the image and subtitles within the deletion range are deleted. Before deleting the portion of the subtitle corresponding to the received audio text within the deletion range from the image within the deletion range, a confirmation screen for confirming the deletion of the subtitle displays the deleted subtitle. Before deleting the aforementioned portion of the subtitles, the deletion range is adjusted in the post-editing subtitles displayed on the confirmation screen as post-editing information. The received deletion range will be used as the new partial subtitle.

8. A computer program product comprising a program that causes a computer to perform processing. In the process, Acquire video data capable of reproducing videos in which speech, images, and subtitles are synchronized in time. During video editing, the deletion intervals within the video's playback time are received. Obtain the speech-text by converting the repeated portions of the deleted interval and the subtitle interval into text. The portion of the subtitle that corresponds to the received speech text within the deletion interval and is at least part of the subtitle is deleted from the image within the deletion interval. By receiving the deletion range, the image and subtitles within the deletion range are deleted. Before deleting the portion of the subtitle corresponding to the received audio text within the deletion range from the image within the deletion range, a confirmation screen for confirming the deletion of the subtitle displays the deleted subtitle. Before deleting the aforementioned portion of the subtitles, the deletion range is adjusted in the post-editing subtitles displayed on the confirmation screen as post-editing information. The received deletion range will be used as the new partial subtitle.

Images