Video processing method and storage medium

Abstract

The application discloses a video processing method and a storage medium. The method comprises the following steps: extracting audio data from a video; converting the audio data into at least one to-be-verified subtitle of the video, and determining a start time of the to-be-verified subtitle and an end time of the to-be-verified subtitle; obtaining a similarity between the to-be-verified subtitle and a reference subtitle of the video based on the start time of the to-be-verified subtitle and the end time of the to-be-verified subtitle; and determining whether the to-be-verified subtitle is correct based on the similarity. The application solves the technical problem of low efficiency of verifying the subtitles of the video.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to a video processing method and a storage medium. Background Art

[0002] Currently, one-click video production automatically adds subtitles to the finished film based on the speech content in the video. However, the correctness of adding subtitles has a certain impact on the quality of the finished film, so it is necessary to detect the accuracy of the automatically added subtitles.

[0003] In the related art, a purely manual method is usually used to point out whether the subtitles added to the finished film are accurately synchronized after watching the video. This has the technical problem of low efficiency in verifying the subtitles of the video.

[0004] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0005] The embodiments of the present invention provide a video processing method and a storage medium to at least solve the technical problem of low efficiency in verifying subtitles of a video.

[0006] According to one aspect of an embodiment of the present invention, a method for processing video subtitles is provided. The method may include: extracting audio data from a video; converting the audio data into at least one subtitle to be verified for the video, and determining the start time and end time of the subtitle to be verified; obtaining a similarity between the subtitle to be verified and a reference subtitle of the video based on the start time and end time of the subtitle to be verified; and determining whether the subtitle to be verified is correct based on the similarity.

[0007] According to one aspect of an embodiment of the present invention, a video processing method is also provided. The method may include: obtaining a video by calling a first interface, wherein the first interface includes a first parameter, the parameter value of the first parameter being the video; extracting audio data from the video; converting the audio data into at least one subtitle to be verified for the video, and determining the start time and end time of the subtitle to be verified; obtaining the similarity between the subtitle to be verified and a reference subtitle of the video based on the start time and end time of the subtitle to be verified; determining whether the subtitle to be verified is correct based on the similarity, and obtaining a verification result; and outputting the verification result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of the second parameter being the verification result.

[0008] According to one aspect of an embodiment of the present invention, a video subtitle processing method is also provided. The method may include: obtaining a target short video from a short video platform; extracting audio data from the target short video; converting the audio data into at least one subtitle to be verified for the target short video, and determining the start time and end time of the subtitle to be verified; obtaining the similarity between the subtitle to be verified and the reference subtitle of the target short video based on the start time and end time of the subtitle to be verified; determining whether the subtitle to be verified is correct based on the similarity, and obtaining a verification result; returning the verification result to the short video platform, wherein, when the verification result indicates that the subtitle to be verified is incorrect, the verification result is used to enable the short video platform to adjust the subtitle to be verified; when the verification result indicates that the subtitle to be verified is correct, the verification result is used to enable the short video platform to display the subtitle to be verified when playing the target short video.

[0009] According to one aspect of an embodiment of the present invention, a method for processing video subtitles is also provided. The method may include: displaying a video on the operation interface in response to a video input instruction on the operation interface; and displaying a verification result of a subtitle to be verified in the video on the operation interface in response to a subtitle verification instruction on the operation interface, wherein the verification result is determined based on the similarity between the subtitle to be verified and a reference subtitle of the video, the similarity being based on the start time and the end time of the subtitle to be verified, and the subtitle to be verified being converted from audio data of the video.

[0010] According to one aspect of an embodiment of the present invention, a method for processing video subtitles is also provided. The method may include: in response to a video input command on a virtual reality device, displaying a video image representing a virtual reality scene on a display screen of the virtual reality device; in response to a subtitle verification command on the virtual reality device, displaying a verification result of the subtitles to be verified in the video on the display screen, wherein the verification result is determined based on the similarity between the subtitles to be verified and a reference subtitle of the video, the similarity being determined based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified being converted from audio data of the video; and in response to the verification result indicating that the subtitles are correct, displaying the correct subtitles on the display screen.

[0011] According to one aspect of an embodiment of the present invention, a method for processing video subtitles is also provided. The method may include: in response to a video input command on an augmented reality device, displaying a video obtained by filming a real scene on a display screen of the augmented reality device; in response to a subtitle verification command on the augmented reality device, displaying a verification result of the subtitles to be verified in the video on the display screen, wherein the verification result is determined based on the similarity between the subtitles to be verified and a reference subtitle of the video, the similarity being determined based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified being converted from audio data of the video; and in response to the verification result indicating that the subtitles are correct, displaying the correct subtitles in the video on the display screen.

[0012] According to one aspect of an embodiment of the present invention, a video subtitle processing device is also provided. The device may include: a first extraction unit for extracting audio data from a video; a first conversion unit for converting the audio data into at least one subtitle to be verified for the video, and determining the start time and end time of the subtitle to be verified; a first acquisition unit for obtaining a similarity between the subtitle to be verified and a reference subtitle of the video based on the start time and end time of the subtitle to be verified; and a first verification unit for determining whether the subtitle to be verified is correct based on the similarity.

[0013] According to one aspect of an embodiment of the present invention, a video processing device is also provided. The device may include: a second acquisition unit, configured to acquire a video by calling a first interface, wherein the first interface includes a first parameter, the parameter value of which is the video; a second extraction unit, configured to extract audio data from the video; a second conversion unit, configured to convert the audio data into at least one subtitle to be verified of the video, and determine the start time and end time of the subtitle to be verified; a third acquisition unit, configured to acquire the similarity between the subtitle to be verified and a reference subtitle of the video based on the start time and end time of the subtitle to be verified; a first determination unit, configured to determine whether the subtitle to be verified is correct based on the similarity, and obtain a verification result; and a first output unit, configured to output the verification result by calling a second interface, wherein the second interface includes a second parameter, the parameter value of which is the verification result.

[0014] According to one aspect of an embodiment of the present invention, a video subtitle processing device is also provided. The device may include: a fourth acquisition unit for acquiring a target short video from a short video platform; a third extraction unit for extracting audio data from the target short video; a third conversion unit for converting the audio data into at least one subtitle to be verified of the target short video, and determining the start time and end time of the subtitle to be verified; a fifth acquisition unit for obtaining the similarity between the subtitle to be verified and the reference subtitle of the target short video based on the start time and end time of the subtitle to be verified; a second determination unit for determining whether the subtitle to be verified is correct based on the similarity and obtaining a verification result; and a first return unit for returning the verification result to the short video platform, wherein, when the verification result indicates that the subtitle to be verified is incorrect, the verification result is used to enable the short video platform to adjust the subtitle to be verified; and when the verification result indicates that the subtitle to be verified is correct, the verification result is used to enable the short video platform to display the subtitle to be verified when playing the target short video.

[0015] According to one aspect of an embodiment of the present invention, a video subtitle processing device is provided. The device may include: a first display unit configured to display a video on the operation interface in response to a video input instruction applied to the operation interface; and a second display unit configured to display a verification result of a subtitle to be verified in the video on the operation interface in response to a subtitle verification instruction applied to the operation interface, wherein the verification result is determined based on the similarity between the subtitle to be verified and a reference subtitle of the video, the similarity being determined based on the start time and the end time of the subtitle to be verified, and the subtitle to be verified being converted from audio data of the video.

[0016] According to one aspect of an embodiment of the present invention, a video subtitle processing device is also provided. The device may include: a third display unit for displaying a video image representing a virtual reality scene on a display screen of the virtual reality device in response to a video input instruction on a virtual reality device; a fourth display unit for displaying a verification result of a subtitle to be verified in the video on the display screen in response to a subtitle verification instruction on the virtual reality device, wherein the verification result is determined based on the similarity between the subtitle to be verified and a reference subtitle of the video, the similarity being determined based on the start time and the end time of the subtitle to be verified, and the subtitle to be verified being converted from audio data of the video; and a fifth display unit for displaying the correct subtitle on the display screen in response to the verification result indicating that the subtitle is correct.

[0017] According to one aspect of an embodiment of the present invention, a video subtitle processing device is also provided. The device may include: a sixth display unit for displaying a video image obtained by filming a real scene on a display screen of the augmented reality device in response to a video input instruction on the augmented reality device; a seventh display unit for displaying a verification result of the subtitles to be verified in the video on the display screen in response to a subtitle verification instruction on the augmented reality device, wherein the verification result is determined based on the similarity between the subtitles to be verified and a reference subtitle of the video, the similarity being determined based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified being converted from audio data of the video; and an eighth display unit for displaying the correct subtitles in the video image on the display screen in response to the verification result indicating that the subtitles are correct.

[0018] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored program, wherein when the program is executed, the device where the storage medium is located is controlled to execute any of the above-mentioned video processing methods.

[0019] According to another aspect of an embodiment of the present invention, a processor is further provided, and the processor is configured to run a program, wherein the program executes any one of the above-mentioned video processing methods when running.

[0020] According to another aspect of an embodiment of the present invention, a video processing system is also provided, including: a processor; a memory connected to the processor, and configured to provide the processor with instructions for processing the following processing steps: extracting audio data from a video; converting the audio data into at least one segment of subtitles to be verified for the video, and determining a start time and an end time of the subtitles to be verified; obtaining a similarity between the subtitles to be verified and a reference subtitle of the video based on the start time and the end time of the subtitles to be verified; and determining whether the subtitles to be verified are correct based on the similarity.

[0021] In an embodiment of the present invention, audio data is extracted from a video; the audio data is converted into at least one subtitle to be verified for the video, and the start time and end time of the subtitle to be verified are determined; based on the start time and end time of the subtitle to be verified, the similarity between the subtitle to be verified and the reference subtitle of the video is obtained; and based on the similarity, whether the subtitle to be verified is correct is determined. In other words, the present invention determines the correctness of the subtitle to be verified in the video based on the similarity comparison between the start and end times of the subtitle to be verified in the video and the reference subtitle, without manual intervention, thereby achieving the technical effect of improving the efficiency of subtitle verification for the video and solving the technical problem of low subtitle verification efficiency for the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0023] Figure 1 is a hardware structure block diagram of a computer terminal (or mobile device) according to a video processing method according to an embodiment of the present invention;

[0024] Figure 2 is a flowchart of a video subtitle processing method according to an embodiment of the present invention;

[0025] Figure 3 is a flow chart of a video processing method according to an embodiment of the present invention;

[0026] Figure 4 is a flowchart of another video subtitle processing method according to an embodiment of the present invention;

[0027] Figure 5 is a flowchart of another video subtitle processing method according to an embodiment of the present invention;

[0028] Figure 6 is a schematic diagram of searching for the start time of user subtitles according to an embodiment of the present invention;

[0029] Figure 7 is a schematic diagram of a sliding similarity comparison according to an embodiment of the present invention;

[0030] Figure 8 is a schematic diagram of a video subtitle processing device according to an embodiment of the present invention;

[0031] Figure 9 is a schematic diagram of a video processing device according to an embodiment of the present invention;

[0032] Figure 10 is a schematic diagram of another video subtitle processing device according to an embodiment of the present invention;

[0033] Figure 11 is a schematic diagram of another video subtitle processing device according to an embodiment of the present invention;

[0034] Figure 12 It is a structural block diagram of a computer terminal according to an embodiment of the present invention. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0036] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following interpretations:

[0038] Automatic Speech Recognition (ASR), a technology that converts human speech into text;

[0039] The open source computer program (FFMPEG) is a set of computer programs that can be used to record, convert, and stream digital audio and video. The open source computer program can be called using the Lesser General Public License (LGPL) or the General Public License (GPL).

[0040] Audio sampling rate refers to how many times the recording device samples the analog signal per unit time. The higher the sampling frequency, the more realistic and natural the waveform of the mechanical wave will be.

[0041] Audio channel, sound channel refers to the independent audio signals collected or played back at different spatial locations when recording or playing sound. Therefore, the number of channels is the number of sound sources when recording sound or the number of corresponding speakers when playing back sound.

[0042] Edit Distance, also known as Levenshtein Distance, is the minimum number of edit operations required to change one string into another, and the greater the distance between them, the more different they are.

[0043] Finished video, the processed video generated by the user using the original video to add special effects or subtitles, is called finished video.

[0044] Optical Character Recognition (OCR) is a process of checking characters printed on paper by electronic devices such as scanners or digital cameras, and then translating shapes into computer text by character recognition method.

[0045] User subtitle file, which can be simply understood as the most accurate subtitle file, is used to compare the accuracy and real-time performance with the subtitles in the finished video, and the user subtitle file contains subtitle text, start time and end time.

[0046] Embodiment 1

[0047] According to the embodiments of the present application, a method for processing video is also provided. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a group of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a different order from here.

[0048] The method provided by the embodiment one of the present application can be executed in a mobile terminal, a computer terminal or a similar operation device. Figure 1 A hardware structure block diagram of a computer terminal (or mobile device) for implementing the video processing method is shown. As shown in the figure, Figure 1 The computer terminal 10 (or mobile device 10) can include one or more processors (the processor can include but is not limited to a microprocessor MCU or a programmable logic device FPGA processing device), a memory 104 for storing data, and a transmission module 106 for communication function. In addition, it can also include a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which can be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned electronic device. For example, the computer terminal 10 can also include more or less components than those shown in the figure, or have a structure different from that shown in the figure. Figure 1 Figure 1 ​Different configurations shown.

[0049] It should be noted that the one or more processors and / or other data processing circuits described above may generally be referred to herein as "data processing circuitry". The data processing circuitry may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuitry may be a single independent processing module, or may be incorporated in whole or in part into any of the other components in the computer terminal 10 (or mobile device). As described in the embodiments of the present application, the data processing circuitry serves as a processor control (e.g., selection of a variable resistor terminal path connected to an interface).

[0050] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the video processing method in the embodiment of the present invention. The processor executes the software programs and modules stored in the memory 104 to perform various functional applications and data processing, thereby implementing the above-mentioned video processing method. The memory 104 may include high-speed random access memory and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 104 may further include memory remotely located relative to the processor, and these remote memories may be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0051] The transmission device 106 is configured to receive or transmit data via a network. A specific example of the aforementioned network may include a wireless network provided by the communications provider of the computer terminal 10. In one embodiment, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to enable communication with the Internet. In another embodiment, the transmission device 106 may be a radio frequency (RF) module, which is configured to communicate with the Internet wirelessly.

[0052] The display may be, for example, a touch screen liquid crystal display (LCD) that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0053] It should be noted that, in some optional embodiments, the above Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuits), software elements (including computer code stored on a computer-readable medium), or a combination of hardware elements and software elements. Figure 1This is merely one example of a particular embodiment and is intended to illustrate the types of components that may be present in the aforementioned computer device (or mobile device).

[0054] exist Figure 1 Under the operating environment shown, this application provides Figure 2 It should be noted that the video subtitle processing method of this embodiment can be Figure 1 The illustrated embodiment is executed by a mobile terminal.

[0055] Figure 2 FIG. 1 is a flow chart of a video subtitle processing method according to an embodiment of the present invention. Figure 2 As shown, the method may include the following steps:

[0056] Step S202: extract audio data from the video.

[0057] In the technical solution provided in the above step S202 of the present invention, a video to be processed is selected, and the selected video is processed to extract audio data from the video, wherein the audio data can be 1600 samples, signed 16-bit little-endian pulse code modulation (PCM) format data.

[0058] Optionally, the video is processed by FFMpeg to extract audio data in a 1600-sample, signed 16-bit little-endian pulse code modulation (PCM) format.

[0059] Step S204: convert the audio data into at least one subtitle to be verified of the video, and determine the start time and the end time of the subtitle to be verified.

[0060] In the technical solution provided in the above step S204 of the present invention, the acquired audio data is converted into at least one segment of subtitles to be verified of the video, and the start time and the end time of the acquired subtitles to be verified are determined, wherein the target subtitles can be ASR subtitles, which can be subtitles obtained after the audio data is recognized by automatic speech recognition technology.

[0061] Optionally, a video to be processed is selected, audio data of the video is obtained, the audio data of the video is recognized using automatic speech recognition technology, and at least one segment of subtitles to be verified of the video is obtained. At the same time, the start time and end time of the subtitles to be verified of the audio data are extracted using automatic speech recognition technology, so as to achieve the purpose of determining the start time and end time of the subtitles to be verified.

[0062] Step S206 : obtaining the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time of the subtitle to be verified and the end time of the subtitle to be verified.

[0063] In the technical solution provided in step S206 of the present invention, the similarity between the subtitles to be verified and the reference subtitles of the video is obtained based on the start time and the end time of the subtitles to be verified, where the reference subtitles can be subtitles provided by the user, also called user subtitles.

[0064] Optionally, use reference subtitles to find the paragraph containing the start time, the middle time, and the end time in the subtitles to be verified. The reference subtitles can be slid along the subtitles to be verified from the leftmost side to the right, or from right to left, to find the starting point of the maximum similarity in the merged subtitles to be verified, and obtain the similarity between the subtitles to be verified and the reference subtitles of the video.

[0065] Step S208: determining whether the subtitle to be verified is correct based on the similarity.

[0066] In the technical solution provided in step S208 of the present invention, whether the subtitles to be verified are correct is determined based on the target similarity to obtain a verification result. For example, if the similarity is greater than a certain set threshold, the subtitles to be verified are determined to be correct; if the similarity is less than the certain set threshold, the subtitles to be verified are determined to be incorrect.

[0067] Optionally, the highest similarity of the sliding match is determined. If the highest similarity of the sliding match is greater than a set threshold, it can be determined that the difference between the start time of the subtitle matching point to be verified and the start time of the reference subtitle is small, and the subtitle to be verified can be determined to be correct; if the highest similarity of the sliding match is less than the set threshold, it can be determined that the difference between the start time of the subtitle matching point to be verified and the start time of the user subtitle is large, and the subtitle to be verified can be determined to be wrong.

[0068] Optionally, if the target similarity between the subtitles to be verified and the reference subtitles of the video is high, then it is verified that the reference subtitles of the video are consistent with the subtitles to be verified, and the content of the subtitles to be verified can be obtained to be accurate and synchronized, and the subtitles to be verified are correct.

[0069] Through the above steps S202 to S208 of the present application, audio data is extracted from the video; the audio data is converted into at least one subtitle to be verified of the video, and the start time and end time of the subtitle to be verified are determined; based on the start time and end time of the subtitle to be verified, the similarity between the subtitle to be verified and the reference subtitle of the video is obtained; and based on the similarity, whether the subtitle to be verified is correct is determined. In other words, the present invention determines the correctness of the subtitle to be verified in the video based on the similarity comparison between the start and end time of the subtitle to be verified and the reference subtitle, without manual work, thereby achieving the technical effect of improving the efficiency of verifying the subtitles of the video and solving the technical problem of low efficiency in verifying the subtitles of the video.

[0070] The above method of this embodiment is further described below.

[0071] As an optional implementation, in step S206, the similarity between the to-be-verified subtitle and the reference subtitle is obtained based on the start time of the to-be-verified subtitle and the end time of the to-be-verified subtitle, including: in response to the target time of the reference subtitle being between the start time of the to-be-verified subtitle and the end time of the to-be-verified subtitle, the similarity between the to-be-verified subtitle and the reference subtitle is obtained, wherein the target time includes the start time of the reference subtitle and / or the end time of the reference subtitle.

[0072] In this embodiment, when the start time of the reference subtitle and / or the end time of the reference subtitle is between the start time of the to-be-verified subtitle and the end time of the to-be-verified subtitle, the reference subtitle and the to-be-verified subtitle are compared to obtain the similarity between the to-be-verified subtitle and the reference subtitle, wherein the similarity can be a value expressed by a percentage, and the greater the percentage, the higher the similarity.

[0073] As an optional implementation, the at least one to-be-verified subtitle includes at least a first to-be-verified subtitle and a second to-be-verified subtitle, and in response to the target time of the reference subtitle being between the start time of the to-be-verified subtitle and the end time of the to-be-verified subtitle, the similarity between the to-be-verified subtitle and the reference subtitle is obtained, including: in response to the start time of the reference subtitle being between the start time of the first to-be-verified subtitle and the end time of the first to-be-verified subtitle, and the end time of the reference subtitle being between the start time of the second to-be-verified subtitle and the end time of the second to-be-verified subtitle, the first to-be-verified subtitle and the second to-be-verified subtitle are merged to obtain a target merged subtitle; and the target merged subtitle and the reference subtitle are compared to obtain the similarity.

[0074] In this embodiment, when the start time of the reference subtitle is between the start time of the first to-be-verified subtitle and the end time of the first to-be-verified subtitle, and the end time of the reference subtitle is between the start time of the second to-be-verified subtitle and the end time of the second to-be-verified subtitle, the first to-be-verified subtitle and the second to-be-verified subtitle are merged to obtain a target merged subtitle; and the target merged subtitle and the reference subtitle are compared to obtain the similarity, wherein one video has multiple to-be-verified subtitles.

[0075] Optionally, based on the reference subtitle as an outer traversal condition, the reference subtitle is used to find the paragraph where the start time is located, the paragraph where the intermediate time is located, and the paragraph where the end time is located in the to-be-verified subtitle, and the subtitles of the found multiple to-be-verified subtitle segments are merged to obtain a target merged subtitle, and the target merged subtitle and the reference subtitle are compared to obtain the similarity.

[0076] As an optional implementation, the target merged subtitles and the reference subtitles are compared to obtain similarity, including: determining, in the target merged subtitles, a sub-subtitle to be verified having the maximum similarity with a sub-reference subtitle in the reference subtitles; and determining the maximum similarity corresponding to the sub-subtitle to be verified as the similarity.

[0077] In this embodiment, the target merged subtitles and the reference subtitles are compared, and the sub-subtitle to be verified having the maximum similarity with the sub-reference subtitle in the reference subtitle is determined in the target merged subtitle, and the maximum similarity corresponding to the sub-subtitle to be verified is determined as the similarity to obtain the similarity.

[0078] Optionally, the reference subtitles can be slid along the target merged subtitles from the leftmost side to the right, and an edit distance calculation is performed each time a word is slid until the maximum similarity point is found. Then, the sub-subtitle to be verified with the maximum similarity between the sub-reference subtitle in the reference subtitle can be determined in the target merged subtitle, and the maximum similarity can be obtained, and the maximum similarity corresponding to the sub-subtitle to be verified is determined as the similarity.

[0079] As an optional embodiment, determining whether the subtitles to be verified are correct based on similarity includes: in response to the similarity being greater than a first target threshold and the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being less than a second target threshold, determining that the sub-subtitle to be verified is correct; in response to the similarity being not greater than the first target threshold, and / or the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being not less than the second target threshold, determining that the sub-subtitle to be verified is incorrect.

[0080] In this embodiment, when the similarity between the reference subtitle and the subtitle to be verified is greater than a first target threshold, and the difference between the start time of the subtitle to be verified and the start time of the subreference subtitle is less than a second target threshold, the subtitle to be verified is determined to be correct; when the similarity between the reference subtitle and the subtitle to be verified is not greater than the first target threshold, and / or the difference between the start time of the subtitle to be verified and the start time of the subreference subtitle is not less than the second target threshold, the subtitle to be verified is determined to be incorrect, wherein the start time of the subtitle to be verified may be the starting point of the maximum similarity in the subtitle to be verified; the first target threshold may be a similarity threshold set according to actual conditions, and may be expressed in percentage; the second target threshold may be a specific value set according to actual conditions.

[0081] Optionally, when the similarity between the reference subtitles and the subtitles to be verified is greater than a first target threshold, and the difference between the start time of the matching point of the subtitles to be verified and the start time of the user subtitles is less than a second target threshold, it can be determined that the synchronization of the subtitles to be verified is accurate; when the similarity between the reference subtitles and the subtitles to be verified is not greater than the first target threshold, and / or, and the difference between the start time of the matching point of the subtitles to be verified and the start time of the user subtitles is not less than the second target threshold, it can be determined that the subtitles to be verified are incorrect.

[0082] As an optional implementation, determining a sub-subtitle to be verified in a target merged subtitle with the maximum similarity with a sub-reference subtitle in a reference subtitle includes: determining multiple sub-merged subtitles to be compared in the target merged subtitle based on the subtitle length of the subreference subtitle; obtaining the similarity between each sub-merged subtitle and the sub-reference subtitle to obtain multiple similarities; and determining the subtitle corresponding to the maximum similarity among the multiple similarities as the sub-subtitle to be verified.

[0083] In this embodiment, multiple sub-merged subtitles to be compared are determined in the target merged subtitles based on the subtitle length of the sub-reference subtitles, and the similarity between each sub-merged subtitle and the sub-reference subtitle is obtained to obtain multiple similarities; the sub-subtitle corresponding to the maximum similarity among the multiple similarities is determined as the sub-subtitle to be verified, so as to achieve the purpose of determining the sub-subtitle to be verified in the target merged subtitles with the maximum similarity between the sub-reference subtitle in the reference subtitles.

[0084] Optionally, multiple sub-reference subtitles are obtained from the reference subtitles, and based on the subtitle lengths of the obtained sub-reference subtitles, they are slid from the leftmost side to the rightmost side in the target merged subtitles. An edit distance calculation is performed each time a character is slid to determine multiple sub-merged subtitles to be compared, and the similarity between each sub-merged subtitle and the sub-reference subtitle is obtained to obtain multiple similarities. The sub-subtitle corresponding to the maximum similarity among the multiple similarities is determined as the sub-subtitle to be verified, so as to achieve the purpose of determining the sub-subtitle to be verified in the target merged subtitle that has the maximum similarity with the sub-reference subtitle in the reference subtitle.

[0085] As an optional implementation manner, the difference between every two adjacent subtitles is a target unit subtitle.

[0086] In this embodiment, the difference between every two adjacent subtitles is a target unit subtitle, wherein the target unit subtitle can be a sliding unit word number.

[0087] Optionally, based on the subtitle length of the obtained sub-reference subtitle, slide from the leftmost side to the right side in the target merged subtitle. Each time you slide one word, you get a sub-subtitle. It should be noted that sliding one word here is just an example and can be set according to actual conditions.

[0088] As an optional implementation, the subtitles to be verified include at least a third segment of subtitles to be verified. In response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, the first segment of subtitles to be verified and the second segment of subtitles to be verified are merged to obtain a target merged subtitle, including: in response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, the middle time of the reference subtitle being between the start time of the third segment of subtitles to be verified and the end time of the third segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, the first segment of subtitles to be verified, the third segment of subtitles to be verified and the second segment of subtitles to be verified are merged to obtain a target merged subtitle.

[0089] In this embodiment, when the start time of the reference subtitle is between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, the middle time of the reference subtitle is between the start time of the third segment of subtitles to be verified and the end time of the third segment of subtitles to be verified, and the end time of the reference subtitle is between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, the first segment of subtitles to be verified, the third segment of subtitles to be verified and the second segment of subtitles to be verified are merged to obtain the target merged subtitle.

[0090] Optionally, the start time of the reference subtitle is searched in the subtitle segment to be verified to determine the first subtitle segment to be verified, wherein the start time of the reference subtitle is between the start time of the first subtitle segment to be verified and the end time of the first subtitle segment to be verified; then the subtitle segment to be verified corresponding to the middle segment where the reference subtitle is located is searched to determine the third subtitle segment to be verified, wherein the middle time of the reference subtitle is between the start time of the third subtitle segment to be verified and the end time of the third subtitle segment to be verified; finally, the end time of the reference subtitle is determined in the matching segment in the subtitle segment to be verified to obtain the second subtitle segment to be verified, wherein the end time of the reference subtitle is between the start time of the second subtitle segment to be verified and the end time of the second subtitle segment to be verified.

[0091] Optionally, the first segment of subtitles to be verified, the third segment of subtitles to be verified, and the second segment of subtitles to be verified are merged to obtain a target merged subtitle.

[0092] As an optional implementation, converting the audio data into at least one subtitle to be verified for the video includes: converting the audio data into at least one original subtitle; adjusting or deleting invalid subtitles in the original subtitles to obtain the subtitles to be verified.

[0093] In this embodiment, the audio data is converted into at least one original subtitle, and invalid subtitles in the original subtitle are adjusted or deleted, such as empty subtitles, spaces, punctuation marks, and the like, to obtain a to-be-verified subtitle.

[0094] Optionally, the original subtitle is traversed, spaces are matched by regular expressions, and punctuation marks are removed to obtain the to-be-verified subtitle.

[0095] As an optional implementation, an original reference subtitle of a video is obtained, and invalid subtitles in the original reference subtitle are adjusted or deleted to obtain a reference subtitle.

[0096] In this embodiment, an original reference subtitle of a video is obtained, and invalid subtitles in the original reference subtitle are adjusted or deleted, such as empty subtitles, spaces, punctuation marks, and the like, to obtain a reference subtitle.

[0097] Optionally, the original reference subtitle is traversed, spaces are matched by regular expressions, and punctuation marks are removed to obtain the reference subtitle.

[0098] This embodiment compares the start and end times of the to-be-verified subtitle of the video with the reference subtitle based on similarity, determines the correctness of the to-be-verified subtitle of the video, and does not require manual operation, thereby achieving the technical effect of improving the efficiency of verifying the subtitle of the video and solving the technical problem of low efficiency of verifying the subtitle of the video.

[0099] The embodiment of the application also provides another video processing method.

[0100] Figure 3 is a flowchart of another video subtitle processing method according to the embodiment of the application. As shown in Figure 3 , the method can include the following steps:

[0101] In step S302, a video is obtained by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the video.

[0102] In the technical solution provided in the above step S302 of the application, the first interface can be an interface for data interaction between a server and a client. The client can pass at least one load into the first interface as a first parameter of the first interface, to achieve the purpose of obtaining the video.

[0103] Optionally, the first parameter of the first interface is called to achieve the purpose of obtaining the first video.

[0104] In step S304, audio data is extracted from the video.

[0105] In the technical solution provided in the above step S304 of the present invention, a video to be processed is selected, and audio data is extracted from the selected video.

[0106] Step S306 : converting the audio data into at least one subtitle to be verified of the video, and determining the start time and the end time of the subtitle to be verified.

[0107] In the technical solution provided in the above step S306 of the present invention, the acquired audio data is converted into at least one subtitle to be verified of the video, and the start time and the end time of the acquired subtitle to be verified are determined.

[0108] Optionally, a video to be processed is selected, audio data of the video is obtained, the obtained audio data is recognized using automatic speech recognition technology, and at least one segment of subtitles to be verified of the target video is obtained. At the same time, the start time and end time of the subtitles to be verified of the audio data are extracted using automatic speech recognition technology, so as to achieve the purpose of determining the start time and end time of the subtitles to be verified.

[0109] Step S308: obtaining the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time of the subtitle to be verified and the end time of the subtitle to be verified.

[0110] In the technical solution provided in step S308 of the present invention, reference subtitles are used to search for the paragraph containing the start time, the paragraph containing the middle time, and the paragraph containing the end time in the subtitles to be verified. The reference subtitles can be slid from the leftmost side to the right side, or from the right to the left side, along the subtitles to be verified to find the starting point with the maximum similarity in the merged subtitles to be verified, and the similarity between the subtitles to be verified and the reference subtitles of the video can be obtained.

[0111] Step S310: determining whether the subtitle to be verified is correct based on the similarity, and obtaining a verification result.

[0112] In the technical solution provided in step S310 of the present invention, whether the subtitles to be verified are correct is determined based on the target similarity to obtain a verification result. For example, if the similarity is greater than a certain set threshold, the subtitles to be verified are determined to be correct; if the similarity is less than the certain set threshold, the subtitles to be verified are determined to be incorrect.

[0113] Step S312: Output the verification result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the verification result.

[0114] In the technical solution provided in the above step S312 of the present invention, the second interface can be an interface for data interaction between the server and the client. The server can output the verification result by calling the second interface, and the second parameter is used as a parameter of the second interface to achieve the purpose of outputting the verification result.

[0115] In this embodiment, a video is obtained by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the video; audio data is extracted from the video; the audio data is converted into at least one segment of subtitles to be verified of the video, and the start time of the subtitles to be verified and the end time of the subtitles to be verified are determined; based on the start time of the subtitles to be verified and the end time of the subtitles to be verified, the similarity between the subtitles to be verified and the reference subtitles of the video is obtained; based on the similarity, it is determined whether the subtitles to be verified are correct to obtain a verification result; the verification result is output by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the verification result, thereby achieving the technical effect of improving the efficiency of verifying the subtitles of the video and solving the technical problem of low efficiency in verifying the subtitles of the video.

[0116] The embodiment of the present invention also provides another video processing method applied to a live broadcast scenario.

[0117] Figure 4 FIG. 1 is a flow chart of another video subtitle processing method according to an embodiment of the present invention. Figure 4 As shown, the method may include the following steps:

[0118] Step S402: Obtain the target short video from the short video platform.

[0119] In the technical solution provided in the above step S402 of the present invention, the target short video to be processed is obtained from the short video platform.

[0120] Step S404: extract audio data from the target short video.

[0121] In the technical solution provided in the above step S404 of the present invention, a target short video to be processed is selected and audio data is extracted from the selected target short video.

[0122] Optionally, the target video is processed by FFMpeg to extract audio data in a format of 1600 samples, signed 16-bit little-endian pulse code modulation (PCM).

[0123] Step S406: convert the audio data into at least one subtitle to be verified of the target short video, and determine the start time and the end time of the subtitle to be verified.

[0124] In the technical solution provided in the above step S406 of the present invention, the audio data of the selected target short video is extracted to obtain audio data, and the obtained audio data is converted into at least one subtitle to be verified of the target short video, and the start time and the end time of the subtitle to be verified are determined.

[0125] Optionally, select the target short video to be processed, obtain the audio data of the target short video, use automatic speech recognition technology to identify the audio data of the target video, obtain at least one segment of subtitles to be verified of the target video, and at the same time, use automatic speech recognition technology to extract the start time and end time of the subtitles to be verified of the audio data, so as to achieve the purpose of determining the start time and end time of the subtitles to be verified.

[0126] Step S408 : obtaining the similarity between the subtitle to be verified and the reference subtitle of the target short video based on the start time of the subtitle to be verified and the end time of the subtitle to be verified.

[0127] In the technical solution provided in the above step S408 of the present invention, based on the start time of the subtitles to be verified and the end time of the subtitles to be verified, the similarity between the subtitles to be verified and the reference subtitles of the target short video is obtained, wherein the reference subtitles can be subtitles provided by the user, which can also be called user subtitles.

[0128] Optionally, use reference subtitles to find the paragraph containing the start time, the middle time, and the end time in the subtitles to be verified. The reference subtitles can be slid along the subtitles to be verified from the leftmost side to the right, or from right to left, to find the starting point of the maximum similarity in the merged subtitles to be verified, and obtain the similarity between the subtitles to be verified and the reference subtitles of the target short video.

[0129] Step S410: determining whether the subtitle to be verified is correct based on the similarity, and obtaining a verification result.

[0130] In the technical solution provided in the above step S410 of the present invention, whether the subtitles to be verified are correct is determined based on the target similarity to obtain a verification result. For example, if the similarity is greater than a certain set threshold, the subtitles to be verified are determined to be correct; if the similarity is less than the certain set threshold, the subtitles to be verified are determined to be incorrect.

[0131] Optionally, the highest similarity of the sliding match is determined. If the highest similarity of the sliding match is greater than a set threshold, it is determined that the difference between the start time of the subtitle matching point to be verified and the start time of the reference subtitle is small, and the subtitle to be verified is determined to be correct; if the highest similarity of the sliding match is less than the set threshold, it is determined that the difference between the start time of the subtitle matching point to be verified and the start time of the reference subtitle is large, and the subtitle to be verified is wrong.

[0132] Step S412, returning the verification result to the short video platform, wherein, when the verification result is used to indicate that the subtitles to be verified are incorrect, the verification result is used to enable the short video platform to adjust the subtitles to be verified; when the verification result is used to indicate that the subtitles to be verified are correct, the verification result is used to enable the short video platform to display the subtitles to be verified when playing the target short video.

[0133] In the technical solution provided in the above step S412 of the present invention, the verification result is returned to the short video platform. When the subtitles to be verified are incorrect, the verification result is displayed on the short video platform interface. The short video platform adjusts the subtitles to be verified based on the verification result. After the adjustment is correct, the adjusted subtitles to be verified are displayed on the target short video being played.

[0134] Optionally, when the subtitles to be verified are correct, the verification results are returned to the interface of the short video platform, and the short video platform displays the subtitles to be verified when playing the target short video.

[0135] The embodiment of the present invention also provides another video subtitle processing method from the human-computer interaction side.

[0136] Figure 5 FIG. 1 is a flow chart of another video subtitle processing method according to an embodiment of the present invention. Figure 5 As shown, the method may include the following steps:

[0137] Step S502 : responding to a video input instruction on the operation interface, displaying the video on the operation interface.

[0138] In the technical solution provided in the above step S502 of the present invention, the video input instruction can be triggered by the user to trigger a request to display the video, so that this embodiment responds to the video input instruction acting on the interactive interface and displays the video on the operation interface.

[0139] Step S504, in response to the subtitle verification instruction on the operation interface, displays the verification result of the subtitles to be verified of the video on the operation interface, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0140] In the technical solution provided in step S504 of the present invention, in response to a subtitle verification instruction applied to an operation interface, a verification result of the subtitles to be verified of the video is displayed on the operation interface, wherein the verification result is determined based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity being obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0141] The embodiment of the present application further provides another video subtitle processing method applied to a virtual reality (VR) device.

[0142] In the embodiment, in response to a video input instruction acting on the virtual reality device, a picture of a video for presenting a virtual reality scene is displayed on a display screen of the virtual reality device.

[0143] Optionally, the video input instruction can be triggered by a user, and the video input instruction is used to trigger a request for displaying the video, and the video input instruction acting on the virtual reality device is selected, and in response to the video input instruction acting on the virtual reality device, the picture of the video for presenting the virtual reality scene is displayed on the display screen of the virtual reality device.

[0144] In the embodiment, in response to a subtitle checking instruction acting on the virtual reality device, a checking result of a to-be-checked subtitle of the video is displayed on the display screen, wherein the checking result is determined based on a similarity between the to-be-checked subtitle and a reference subtitle of the video, and the similarity is obtained based on a start time of the to-be-checked subtitle and an end time of the to-be-checked subtitle, and the to-be-checked subtitle is converted from audio data of the video.

[0145] Optionally, the audio data of the video is converted to obtain the to-be-checked subtitle, and in response to the subtitle checking instruction acting on the virtual reality device, the to-be-checked subtitle of the video displayed on the display screen is checked, which can include determining the start time of the to-be-checked subtitle and the end time of the to-be-checked subtitle, determining the similarity between the to-be-checked subtitle and the reference subtitle of the video based on the similarity between the to-be-checked subtitle and the reference subtitle of the video, determining the checking result based on the similarity between the to-be-checked subtitle and the reference subtitle of the video, and displaying the checking result on the display screen.

[0146] Optionally, a video to be displayed is selected, audio data of the video to be displayed is obtained, the audio data of the video is recognized using automatic speech recognition technology, at least one subtitle to be verified of the video is obtained, and the start time and end time of the subtitle to be verified of the audio data are extracted using automatic speech recognition technology to obtain the start time of the subtitle to be verified and the end time of the subtitle to be verified, and a reference subtitle is used to search for the paragraph where the start time, the paragraph where the middle time, and the paragraph where the end time are located in the subtitle to be verified, and the reference subtitle can be slid from the leftmost side to the right side, or from right to left, along the subtitle to be verified to find the starting point of the maximum similarity in the merged subtitle to be verified, and the similarity between the subtitle to be verified and the reference subtitle of the video is obtained, and whether the subtitle to be verified is correct is determined based on the target similarity to obtain a verification result, and in response to a subtitle verification instruction acting on the virtual reality device, the verification result of the subtitle to be verified of the video is displayed on the display screen.

[0147] In this embodiment, in response to the verification result indicating that the subtitles are correct, the correct subtitles are displayed on the display screen.

[0148] Optionally, the highest similarity of the sliding match is determined. If the highest similarity of the sliding match is greater than a set threshold, it can be determined that the difference between the start time of the subtitle matching point to be verified and the start time of the reference subtitle is small, and it can be determined that the subtitle to be verified is correct. In response to the verification result indicating that the subtitle is correct, the correct subtitle is displayed on the display screen.

[0149] As an optional implementation, in response to the verification result indicating that the subtitles are incorrect, subtitles adjusted based on the verification result are displayed on the display screen.

[0150] Optionally, if the highest similarity of the sliding match is less than a set threshold, it can be judged that the start time of the subtitle matching point to be verified is significantly different from the start time of the user subtitle, and it can be determined that the subtitle to be verified is wrong. In response to the verification result indicating that the subtitle is wrong, the video subtitle is adjusted based on the verification result, and the subtitle adjusted based on the verification result is displayed on the display screen.

[0151] As an optional implementation, the virtual reality device is a head-mounted reality device.

[0152] In this embodiment, the virtual reality device can be a head-mounted reality device, for example, a head-mounted display with eye tracking, or a virtual reality device such as virtual glasses, a virtual reality all-in-one machine, etc. that can implement the video subtitle processing method in a virtual reality scene. Examples will not be given one by one this time.

[0153] The embodiment of the present invention determines the correctness of the subtitles to be verified in the video of the virtual reality scene based on the similarity comparison between the start and end times of the subtitles to be verified and the reference subtitles, without manual work, thereby achieving the technical effect of improving the efficiency of verifying the subtitles of the video of the virtual reality scene and solving the technical problem of low efficiency in verifying the subtitles of the video of the virtual reality scene.

[0154] An embodiment of the present invention further provides another video subtitle processing method applied to an augmented reality (AR) display device.

[0155] In this embodiment, in response to a video input instruction applied to the augmented reality device, a video image obtained by shooting a real scene is displayed on the display screen of the augmented reality device.

[0156] Optionally, the video input instruction can be triggered by the user to trigger a request to display a video, select the video input instruction on the augmented reality device, respond to the video input instruction acting on the augmented reality device, and display the video image obtained by shooting the real scene on the display screen of the augmented reality device.

[0157] In this embodiment, in response to a subtitle verification instruction acting on an augmented reality device, a verification result of the subtitles to be verified of the video is displayed on a display screen, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time of the subtitles to be verified and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0158] Optionally, the audio data of the video is converted to obtain subtitles to be verified, and in response to a subtitle verification instruction acting on the augmented reality device, the subtitles to be verified of the video displayed on the display screen are verified, which may include determining the start time of the subtitles to be verified and the end time of the subtitles to be verified, determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video to obtain the similarity between the subtitles to be verified and the reference subtitles of the video, determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video to obtain a verification result, and displaying the verification result on the display screen.

[0159] Optionally, a real scene is filmed to obtain a video image, audio data of the video to be displayed on the display screen of the augmented reality device is obtained, the audio data of the video is recognized using automatic speech recognition technology, and at least one subtitle to be verified of the video is obtained. At the same time, the start time and end time of the subtitle to be verified of the audio data are extracted using automatic speech recognition technology to obtain the start time of the subtitle to be verified and the end time of the subtitle to be verified. The reference subtitle is used to search the subtitle to be verified for the paragraph containing the start time, the paragraph containing the middle time, and the paragraph containing the end time. The reference subtitle can be slid from the leftmost side to the right side, or from right to left, along the subtitle to be verified to find the starting point of the maximum similarity in the merged subtitle to be verified, and the similarity between the subtitle to be verified and the reference subtitle of the video is obtained. Based on the target similarity, it is determined whether the subtitle to be verified is correct, and a verification result is obtained. In response to the subtitle verification instruction acting on the augmented reality device, the verification result of the subtitle to be verified of the video is displayed on the display screen.

[0160] In this embodiment, in response to the verification result indicating that the subtitles are correct, the correct subtitles are displayed in the video frame on the display screen.

[0161] Optionally, the highest similarity of the sliding match is determined. If the highest similarity of the sliding match is greater than a set threshold, it can be determined that the start time of the subtitle matching point to be verified is small compared to the start time of the reference subtitle, and it can be determined that the subtitle to be verified is correct. In response to the verification result indicating that the subtitle is correct, the correct subtitle is displayed in the video screen on the display screen.

[0162] The embodiment of the present invention compares the start and end times of the subtitles to be verified of the video obtained by shooting the real scene with the reference subtitles for similarity, and determines the correctness of the subtitles to be verified of the video obtained by shooting the real scene without manual work, thereby achieving the technical effect of improving the verification efficiency of the subtitles of the video of the augmented reality device, and solving the technical problem of low efficiency in verifying the subtitles of the video of the augmented reality device.

[0163] In an embodiment of the present invention, in response to a video input instruction applied to an operation interface, a video is displayed on the operation interface; in response to a subtitle verification instruction applied to the operation interface, a verification result of the subtitles to be verified of the video is displayed on the operation interface, wherein the verification result is determined based on the similarity between the subtitles to be verified and the reference subtitles of the video, wherein the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video. In other words, the present invention determines the correctness of the target subtitles of the target video based on the similarity comparison between the start and end times of the target subtitles of the target video and the target reference subtitles, without manual intervention, thereby achieving the technical effect of improving the efficiency of subtitle verification of the video and solving the technical problem of low efficiency of subtitle verification of the video.

[0164] Example 2

[0165] The following further introduces the preferred implementation of the above method of this embodiment, specifically using a method for verifying the correctness of subtitles in a finished video.

[0166] Currently, one-click video editing automatically adds subtitles to the finished video based on the host's speech in the video. The correctness of the subtitles has a significant impact on the quality of the user's finished video, so a certain level of detection capability is required. However, there are few solutions for verifying the correctness of subtitles embedded in finished videos, especially efficient and accurate solutions.

[0167] The accuracy of subtitles has a crucial impact on users' video viewing and comprehension. Automatically adding subtitles can lead to issues such as subtitle accuracy not meeting standards and time synchronization between subtitles and actual speech. For example, when a live video introduces, "This sweater has a high collar and sequins. Please take the number 2 link," the actual subtitles in the automatically subtitled video read, "This trade belongs to an elderly woman named Dai Liangpian. Please queue up for the number Qin Meng." The introduction in the live video is spoken between 3 and 8 seconds, but appears in the subtitles between 7 and 12 seconds.

[0168] In the related art, a purely manual method is used to verify the correctness of video subtitles, that is, after watching the video, the correctness and synchronization problems of the subtitles in the finished film are pointed out, but this method has the problem of extremely low efficiency; alternatively, a text recognition solution is used, that is, by splitting the video into frames and recording the time of each frame, and then using a text recognition device to extract the subtitles of each frame and compare them with the subtitle file provided by the user, but this solution has the problem of being unable to accurately detect the correctness of time and having large errors.

[0169] The present invention is based on automatic speech recognition technology and a subtitle accuracy and real-time verification strategy, that is, to decompose the accuracy and synchronization of the voice and subtitle text in the video into the accuracy and time consistency of the text and timestamp converted from the speech and the user subtitles. Among them, the subtitle text detection adopts sliding similarity comparison, and the time accuracy is based on the sliding similarity to identify the subtitle content based on automatic speech recognition technology, and find a more accurate start time of the user subtitle, thereby greatly improving the accuracy.

[0170] The above method of this embodiment is further introduced below.

[0171] The first step is to extract the voice from the video to achieve the purpose of extracting the audio stream.

[0172] Extract the audio stream through FFMpeg and save the extracted audio stream as a 1600-sample, signed 16-bit little-endian pulse code modulation (PCM) pure audio format. For example, extract the voice from the video file and save it as a pulse code modulation pure audio format.

[0173] Step 2: Use automatic speech recognition technology to extract the text and the start and end time of the pure audio file in the pulse code modulation format, which is used for verification and comparison with the user's subtitles.

[0174] Send the pulse code modulated audio file as the data source to the automatic speech recognition technology service, receive and parse the subtitle analysis results returned by the automatic speech recognition technology service, where "text" in the subtitle analysis results is the text of the recognized speech paragraph, "beginTime" is the start time of the paragraph, and "endTime" is the end time of the paragraph.

[0175] For example, the text "Fashionable children's sports shoes, wear unlimited vitality", beginTime: 0, endTime: 1200, can be expressed as: the subtitle "Fashionable children's sports shoes, wear unlimited vitality" starts at 0 seconds and ends at 1200 seconds.

[0176] For example, the text "Fashionable children's sports shoes, wear out unlimited vitality. The entire upper of this shoe is made of nubuck leather" has a beginTime of 1200 and an endTime of 5300, which means that the subtitle "Fashionable children's sports shoes, wear out unlimited vitality. The entire upper of this shoe is made of nubuck leather" starts at 1200 seconds and ends at 5300 seconds.

[0177] For example, text: "Nubuck leather upper is more durable", beginTime: 5200, endTime: 7500, can mean: the subtitle "Nubuck leather upper is more durable" starts at 5200 seconds and ends at 7500 seconds.

[0178] For example, text: "There will be wrinkles", beginTime: 7500, endTime: 8000, can mean: the subtitle "There will be wrinkles" starts at 7500 seconds and ends at 8000 seconds.

[0179] The third step is to remove and optimize invalid subtitles in the subtitles recognized by the automatic speech recognition technology and the user subtitles, mainly targeting empty subtitles, the end time of the user subtitles being longer than the subtitle time recognized by the automatic speech recognition technology, the start time of the subtitles not being 0, and the presence of punctuation marks, spaces, etc.

[0180] Optionally, the subtitles generated by automatic speech recognition and user subtitles are traversed, and punctuation marks such as spaces are removed through regular matching.

[0181] Optionally, when the start time of the automatic speech recognition subtitles is not 0, but the absolute value of the difference between the start time and 0 is less than the threshold time, the start time is forcibly set to 0; when the end time of the user subtitles is greater than the subtitle time recognized by the automatic speech recognition, the end time of the subtitles recognized by the automatic speech recognition is added to the threshold time.

[0182] The fourth step is to verify the correctness and synchronization of the subtitle file provided by the user and the sound content in the video. When the subtitle file provided by the user is consistent with the sound content in the video, it means that the content is accurate and synchronized.

[0183] Optionally, based on the user subtitles as the outer traversal conditions, the following operations are performed in the subtitles obtained by automatic speech recognition: First, the user subtitles are used to search for the paragraph where the start time, the paragraph where the middle time, and the paragraph where the end time are located in the subtitles obtained by automatic speech recognition (ASR subtitles), and the subtitles of the multiple subtitle segments found by automatic speech recognition are merged, and single-word time extraction and other operations are performed to retain them for later use; Second, a sliding matching scheme is used to find the maximum similarity starting point of the current user subtitles in the merged automatic speech recognition subtitles, and the starting time of the matching point of the automatic speech recognition subtitles is calculated by the single-word timestamp for later use; Third, it is determined that the highest similarity of the sliding matching is greater than the user threshold, and it is determined that the difference between the starting time of the automatic speech recognition subtitle matching point and the starting time of the user subtitle does not exceed the threshold.

[0184] Optionally, after traversing all user subtitles, the number of failures is determined, that is, the similarity and start time difference of subtitles that do not meet the threshold are determined. If any exist, the correctness detection of the completed subtitles is determined to have failed, otherwise the detection is successful.

[0185] The following is an explanation of the user subtitle start time search and sliding comparison solution in step 4.

[0186] First, use user subtitles to find the start time paragraph, middle time paragraph, and end time paragraph in the subtitles generated by automatic speech recognition (ASR subtitles). Figure 6 FIG. 1 is a schematic diagram of searching for the start time of user subtitles according to an embodiment of the present invention. Figure 6 As shown, the starting time period corresponding to the user subtitles on the right is searched in the first ASR subtitle segment on the left, and then the middle time period corresponding to the user subtitles is searched in the second ASR subtitle segment. Finally, the ending time period corresponding to the user subtitles is searched in the third ASR subtitle segment. Figure 6 The three ASR subtitles in the middle are the user subtitle matching paragraphs.

[0187] Second, use the sliding matching scheme to find the starting point with the maximum similarity between the current user's subtitles and the merged automatic speech recognition subtitles. Figure 7 is a schematic diagram of a sliding similarity comparison according to an embodiment of the present invention, such as Figure 7 As shown in the figure, the user subtitles slide along the ASR subtitles from the leftmost side to the right. A similarity calculation is performed every time a word is slid. The maximum similarity point can be found by sliding from left to right.

[0188] The above embodiment utilizes a start and end time interval search and merging scheme to improve detection accuracy. At the same time, sliding similarity comparison plus ASR recognition of single-word time is used to calculate subtitle synchronization to improve detection accuracy, thereby achieving the technical effect of improving the efficiency of verifying video subtitles and solving the technical problem of low efficiency in verifying video subtitles.

[0189] It should be noted that for the aforementioned method embodiments, for simplicity of description, they are all expressed as a series of action combinations. However, those skilled in the art should be aware that the present invention is not limited by the order of the actions described, because according to the present invention, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0190] Through the description of the above embodiments, those skilled in the art can clearly understand that the video processing method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0191] Example 3

[0192] According to an embodiment of the present invention, there is also provided a method for implementing the above Figure 2 The video subtitle processing device of the video subtitle processing method shown.

[0193] Figure 8 FIG. 1 is a schematic diagram of a video subtitle processing device according to an embodiment of the present invention. Figure 8 As shown, the video subtitle processing apparatus 800 may include: a first extraction unit 802 , a first conversion unit 804 , a first acquisition unit 806 and a first verification unit 808 .

[0194] The first extraction unit 802 is configured to extract audio data from the video.

[0195] The first conversion unit 804 is configured to convert the audio data into at least one subtitle to be verified of the video, and determine a start time and an end time of the subtitle to be verified.

[0196] The first acquiring unit 806 is configured to acquire the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time of the subtitle to be verified and the end time of the subtitle to be verified.

[0197] The first verification unit 808 is configured to determine whether the subtitle to be verified is correct based on the similarity.

[0198] It should be noted that the first extraction unit 802, first conversion unit 804, first acquisition unit 806, and first verification unit 808 described above correspond to steps S202 to S208 in Example 1. The examples and application scenarios implemented by these four units and the corresponding steps are the same, but are not limited to those disclosed in Example 1. It should be noted that the above units, as part of the apparatus, can be run in the computer terminal 10 provided in Example 1.

[0199] According to an embodiment of the present invention, there is also provided a method for implementing the above Figure 3The video processing method shown is a video processing device.

[0200] Figure 9 FIG. 1 is a schematic diagram of another video processing device according to an embodiment of the present invention. Figure 9 As shown, the video processing device 900 may include: a second acquiring unit 902 , a second extracting unit 904 , a second converting unit 906 , a third acquiring unit 908 , a first determining unit 910 and a first output unit 912 .

[0201] The second acquiring unit 902 is configured to acquire a video by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the video.

[0202] The second extraction unit 904 is configured to extract audio data from the video.

[0203] The second conversion unit 906 is configured to convert the audio data into at least one subtitle to be verified of the video, and determine a start time and an end time of the subtitle to be verified.

[0204] The third obtaining unit 908 is configured to obtain the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time and the end time of the subtitle to be verified.

[0205] The first determining unit 910 is configured to determine whether the subtitles to be verified are correct based on the similarity, and obtain a verification result.

[0206] The first output unit 912 is configured to output the verification result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the verification result.

[0207] It should be noted that the second acquisition unit 902, second extraction unit 904, second conversion unit 906, third acquisition unit 908, first determination unit 910, and first output unit 912 correspond to steps S302 to S312 in Example 1. The examples and application scenarios implemented by these six units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above-mentioned units, as part of the apparatus, can be run in the computer terminal 10 provided in Example 1.

[0208] According to an embodiment of the present invention, there is also provided a method for implementing the above Figure 4 The video subtitle processing device of the video subtitle processing method shown.

[0209] Figure 10 FIG. 1 is a schematic diagram of another video subtitle processing device according to an embodiment of the present invention. Figure 10As shown, the video subtitle processing apparatus 1000 may include: a fourth acquiring unit 1002 , a third extracting unit 1004 , a third converting unit 1006 , a fifth acquiring unit 1008 , a second determining unit 1010 and a first returning unit 1012 .

[0210] The fourth acquisition unit 1002 is used to acquire the target short video from the short video platform.

[0211] The third extraction unit 1004 is used to extract audio data from the target short video.

[0212] The third conversion unit 1006 is configured to convert the audio data into at least one subtitle to be verified of the target short video, and determine a start time and an end time of the subtitle to be verified.

[0213] The fifth obtaining unit 1008 is configured to obtain the similarity between the subtitle to be verified and the reference subtitle of the target short video based on the start time and the end time of the subtitle to be verified.

[0214] The second determining unit 1010 is configured to determine whether the subtitle to be verified is correct based on the similarity, and obtain a verification result.

[0215] The first return unit 1012 is used to return the verification result to the short video platform, wherein, when the verification result is used to indicate that the subtitles to be verified are incorrect, the verification result is used to enable the short video platform to adjust the subtitles to be verified; when the verification result is used to indicate that the subtitles to be verified are correct, the verification result is used to enable the short video platform to display the subtitles to be verified when playing the target short video.

[0216] It should be noted that the fourth acquisition unit 1002, third extraction unit 1004, third conversion unit 1006, fifth acquisition unit 1008, second determination unit 1010, and first return unit 1012 correspond to steps S402 to S412 in Example 1. The examples and application scenarios implemented by these six units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above-mentioned units, as part of the apparatus, can be run in the computer terminal 10 provided in Example 1.

[0217] According to an embodiment of the present invention, there is also provided a method for implementing the above Figure 5 The video subtitle processing device of the video subtitle processing method shown.

[0218] Figure 11 FIG. 1 is a schematic diagram of another video subtitle processing device according to an embodiment of the present invention. Figure 11 As shown, the video processing device 1100 may include: a first display unit 1102 and a second display unit 1104 .

[0219] The first display unit 1102 is configured to respond to a video input instruction applied to the operation interface and display the video on the operation interface.

[0220] The second display unit 1104 is used to respond to a subtitle verification instruction applied to the operation interface and display a verification result of the subtitles to be verified of the video on the operation interface, wherein the verification result is determined based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity being obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0221] It should be noted that the first display unit 1102 and the second display unit 1104 correspond to steps S502 to S504 in Example 1. The examples and application scenarios implemented by the two units and the corresponding steps are the same, but are not limited to the contents disclosed in Example 1. It should be noted that the above units, as part of the device, can be run in the computer terminal 10 provided in Example 1.

[0222] According to an embodiment of the present invention, a video subtitle processing device of the video subtitle processing method is also provided.

[0223] The video subtitle processing device may include: a third display unit, a fourth display unit and a fifth display unit.

[0224] The third display unit is used to respond to the video input instruction acting on the virtual reality device and display the video image used to present the virtual reality scene on the display screen of the virtual reality device.

[0225] The fourth display unit is used to respond to the subtitle verification instruction acting on the virtual reality device and display the verification result of the subtitles to be verified in the video on the display screen, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0226] The fifth display unit is configured to display the correct subtitles on a display screen in response to the verification result indicating that the subtitles are correct.

[0227] It should be noted here that the above-mentioned unit as a part of the device can be run in the computer terminal 10 provided in the first embodiment.

[0228] According to an embodiment of the present invention, a video subtitle processing device of the video subtitle processing method is also provided.

[0229] The video subtitle processing device may include: a sixth display unit, a seventh display unit, and an eighth display unit.

[0230] The sixth display unit is used to respond to the video input instruction acting on the augmented reality device and display the video image obtained by shooting the real scene on the display screen of the augmented reality device.

[0231] A seventh display unit is used to respond to a subtitle verification instruction acting on the augmented reality device and display a verification result of the subtitles to be verified in the video on the display screen, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0232] The eighth display unit is configured to display the correct subtitles on the video screen in response to the verification result indicating that the subtitles are correct.

[0233] It should be noted here that the above-mentioned unit as a part of the device can be run in the computer terminal 10 provided in the first embodiment.

[0234] In the video processing device of this embodiment, a first extraction unit extracts audio data from a video; a first conversion unit converts the audio data into at least one subtitle to be verified for the video, and determines the start time and end time of the subtitle to be verified; a first acquisition unit obtains the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time and end time of the subtitle to be verified; and a first verification unit determines whether the subtitle to be verified is correct based on the similarity. In other words, the present invention determines the correctness of the target subtitle of the target video based on the similarity comparison between the start and end time of the target subtitle and the target reference subtitle, without manual intervention, thereby achieving the technical effect of improving the efficiency of video subtitle verification and solving the technical problem of low efficiency in video subtitle verification.

[0235] Example 4

[0236] An embodiment of the present invention may provide a video processing system, which may include a computer terminal, which may be any computer terminal device in a computer terminal group. Optionally, in this embodiment, the computer terminal may also be replaced by a terminal device such as a mobile terminal.

[0237] Optionally, in this embodiment, the computer terminal may be located in at least one network device among a plurality of network devices of a computer network.

[0238] In this embodiment, the computer terminal can execute the program code of the following steps in the video processing method: extracting audio data from the video; converting the audio data into at least one subtitle to be verified of the video, and determining the start time and end time of the subtitle to be verified; obtaining the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time and end time of the subtitle to be verified; and determining whether the subtitle to be verified is correct based on the similarity.

[0239] Optionally, Figure 12 1 is a block diagram of a computer terminal according to an embodiment of the present invention. Figure 12 As shown, the computer terminal A may include: one or more (only one is shown in the figure) processors 1202 , a memory 1204 , and a transmission device 1206 .

[0240] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the video processing method and device in the embodiments of the present invention. The processor executes the software programs and modules stored in the memory to perform various functional applications and video processing, thereby implementing the above-mentioned video processing method. The memory can include high-speed random access memory and can also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory can further include memory remotely located relative to the processor, and these remote memories can be connected to the computer terminal A via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0241] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: extracting audio data from the video; converting the audio data into at least one subtitle to be verified of the video, and determining the start time and end time of the target subtitle; obtaining the similarity between the subtitle to be verified and the reference subtitle of the video based on the start time and end time of the subtitle to be verified; and determining whether the subtitle to be verified is correct based on the similarity.

[0242] Optionally, the processor may further execute program code of the following steps: in response to a target time of the reference subtitle being between the start time of the subtitle to be verified and the end time of the subtitle to be verified, obtaining the similarity between the subtitle to be verified and the reference subtitle, wherein the target time includes the start time of the reference subtitle and / or the end time of the reference subtitle.

[0243] Optionally, the processor may further execute the program code of the following steps: in response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, the first segment of subtitles to be verified and the second segment of subtitles to be verified are merged to obtain a target merged subtitle; and the target merged subtitle is compared with the reference subtitle to obtain a similarity.

[0244] Optionally, the processor may further execute program code of the following steps: determining, in the target merged subtitle, a sub-subtitle to be verified having the maximum similarity with a sub-reference subtitle in the reference subtitle; and determining the maximum similarity corresponding to the sub-subtitle to be verified as the similarity.

[0245] Optionally, the processor may further execute program code for the following steps: in response to the similarity being greater than a first target threshold, and the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being less than a second target threshold, determining that the sub-subtitle to be verified is correct; in response to the similarity being not greater than the first target threshold, and / or the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being not less than the second target threshold, determining that the sub-subtitle to be verified is incorrect.

[0246] Optionally, the processor may also execute the program code of the following steps: determining multiple sub-merged subtitles to be compared in the target merged subtitles based on the subtitle length of the sub-reference subtitles; obtaining the similarity between each sub-merged subtitle and the sub-reference subtitle to obtain multiple similarities; and determining the sub-subtitle corresponding to the maximum similarity among the multiple similarities as the sub-subtitle to be verified.

[0247] Optionally, the processor may also execute the following program code: in response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, the middle time of the reference subtitle being between the start time of the third segment of subtitles to be verified and the end time of the third segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, then merging the first segment of subtitles to be verified, the third segment of subtitles to be verified and the second segment of subtitles to be verified to obtain a target merged subtitle.

[0248] Optionally, the processor may further execute program codes of the following steps: converting the audio data into at least one original subtitle; and adjusting or deleting invalid subtitles in the original subtitles to obtain subtitles to be verified.

[0249] Optionally, the processor may further execute program codes of the following steps: obtaining original reference subtitles of the video; and adjusting or deleting invalid subtitles in the original reference subtitles to obtain reference subtitles.

[0250] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the video by calling the first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the video; extract audio data from the video; convert the audio data into at least one segment of subtitles to be verified of the video, and determine the start time of the subtitles to be verified and the end time of the subtitles to be verified; based on the start time of the subtitles to be verified and the end time of the subtitles to be verified, obtain the similarity between the subtitles to be verified and the reference subtitles of the video; determine whether the subtitles to be verified are correct based on the similarity, and obtain the verification result; output the verification result by calling the second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the verification result.

[0251] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: obtain the target short video from the short video platform; extract audio data from the target short video; convert the audio data into at least one subtitle to be verified of the target short video, and determine the start time of the subtitle to be verified and the end time of the subtitle to be verified; based on the start time of the subtitle to be verified and the end time of the subtitle to be verified, obtain the similarity between the subtitle to be verified and the reference subtitle of the target short video; determine whether the subtitle to be verified is correct based on the similarity, and obtain the verification result; return the verification result to the short video platform, wherein, when the verification result is used to indicate that the subtitle to be verified is incorrect, the verification result is used to enable the short video platform to adjust the subtitle to be verified, and when the verification result is used to indicate that the subtitle to be verified is correct, the verification result is used to enable the short video platform to display the subtitle to be verified when playing the target short video.

[0252] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: responding to the video input instruction acting on the operation interface, displaying the video on the operation interface; responding to the subtitle verification instruction acting on the operation interface, displaying the verification result of the subtitles to be verified in the video on the operation interface, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time of the subtitles to be verified and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0253] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to a video input instruction acting on the virtual reality device, display a picture of the video used to present the virtual reality scene on the display screen of the virtual reality device; in response to a subtitle verification instruction acting on the virtual reality device, display the verification result of the subtitles to be verified of the video on the display screen, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video; in response to the verification result indicating that the subtitles are correct, display the correct subtitles on the display screen.

[0254] Optionally, the processor may further execute program code for the following steps: in response to the verification result indicating that the subtitles are incorrect, displaying subtitles adjusted based on the verification result on a display screen.

[0255] As an optional example, the processor can call the information and application stored in the memory through the transmission device to perform the following steps: in response to the video input instruction acting on the augmented reality device, the picture of the video obtained by shooting the real scene is displayed on the display screen of the augmented reality device; in response to the subtitle verification instruction acting on the augmented reality device, the verification result of the subtitle to be verified of the video is displayed on the display screen, wherein the verification result is determined based on the similarity between the subtitle to be verified and the reference subtitle of the video, and the similarity is obtained based on the start time and the end time of the subtitle to be verified, and the subtitle to be verified is converted from the audio data of the video; in response to the verification result indicating that the subtitle is correct, the correct subtitle is displayed in the picture of the video on the display screen.

[0256] An embodiment of the present invention provides a video processing method that determines the correctness of target subtitles of a target video based on a similarity comparison between the start and end times of target subtitles of a target video and target reference subtitles without manual effort, thereby achieving the technical effect of improving the efficiency of verifying subtitles of a video and solving the technical problem of low efficiency in verifying subtitles of a video.

[0257] It can be understood by those skilled in the art that Figure 12 The structure shown is for illustration only, and the computer terminal may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a handheld computer, a mobile Internet device (MID), a PAD, or other terminal devices. Figure 12 It does not limit the structure of the above-mentioned computer terminal. For example, the computer terminal A may also include Figure 12More or fewer components (such as network interfaces, display devices, etc.) shown in, or with Figure 12 Different configurations shown.

[0258] A person skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0259] Example 4

[0260] The embodiment of the present invention further provides a computer-readable storage medium. Optionally, in this embodiment, the computer-readable storage medium can be used to store the program code executed by the video processing method provided in the first embodiment.

[0261] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a computer terminal group in a computer network, or in any mobile terminal in a mobile terminal group.

[0262] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: extracting audio data from a video; converting the audio data into at least one segment of subtitles to be verified of the video, and determining the start time of the subtitles to be verified and the end time of the subtitles to be verified; obtaining the similarity between the subtitles to be verified and the reference subtitles of the video based on the start time of the subtitles to be verified and the end time of the subtitles to be verified; and determining whether the subtitles to be verified are correct based on the similarity.

[0263] Optionally, the computer-readable storage medium may further execute program code for the following steps: in response to a target time of the reference subtitle being between the start time of the subtitle to be verified and the end time of the subtitle to be verified, obtaining the similarity between the subtitle to be verified and the reference subtitle, wherein the target time includes the start time of the reference subtitle and / or the end time of the reference subtitle.

[0264] Optionally, the computer-readable storage medium may further execute program code for the following steps: in response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, merging the first segment of subtitles to be verified and the second segment of subtitles to be verified to obtain a target merged subtitle; and comparing the target merged subtitle with the reference subtitle to obtain a similarity.

[0265] Optionally, the computer-readable storage medium may further execute program code for the following steps: determining, in the target merged subtitle, a sub-subtitle to be verified having the maximum similarity with a sub-reference subtitle in the reference subtitle; and determining the maximum similarity corresponding to the sub-subtitle to be verified as the similarity.

[0266] Optionally, the computer-readable storage medium may further execute program code for the following steps: in response to the similarity being greater than a first target threshold and the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being less than a second target threshold, determining that the sub-subtitle to be verified is correct; in response to the similarity being not greater than the first target threshold and / or the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being not less than the second target threshold, determining that the sub-subtitle to be verified is incorrect.

[0267] Optionally, the computer-readable storage medium may further execute program code for the following steps: determining multiple sub-merged subtitles to be compared in the target merged subtitles based on the subtitle length of the sub-reference subtitles; obtaining the similarity between each sub-merged subtitle and the sub-reference subtitle to obtain multiple similarities; and determining the sub-subtitle corresponding to the maximum similarity among the multiple similarities as the sub-subtitle to be verified.

[0268] Optionally, the computer-readable storage medium may further execute program code for the following steps: in response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, the middle time of the reference subtitle being between the start time of the third segment of subtitles to be verified and the end time of the third segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, then merging the first segment of subtitles to be verified, the third segment of subtitles to be verified, and the second segment of subtitles to be verified to obtain a target merged subtitle.

[0269] Optionally, the computer-readable storage medium may further execute program code for the following steps: converting the audio data into at least one original subtitle; and adjusting or deleting invalid subtitles in the original subtitles to obtain subtitles to be verified.

[0270] Optionally, the computer-readable storage medium may further execute program code for the following steps: obtaining original reference subtitles of the video; and adjusting or deleting invalid subtitles in the original reference subtitles to obtain reference subtitles.

[0271] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining a video by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the video; extracting audio data from the video; converting the audio data into at least one segment of subtitles to be verified of the video, and determining the start time of the subtitles to be verified and the end time of the subtitles to be verified; obtaining the similarity between the subtitles to be verified and the reference subtitles of the video based on the start time of the subtitles to be verified and the end time of the subtitles to be verified; determining whether the subtitles to be verified are correct based on the similarity, and obtaining a verification result; outputting the verification result by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the verification result.

[0272] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: obtaining a target short video from a short video platform; extracting audio data from the target short video; converting the audio data into at least one subtitle to be verified of the target short video, and determining the start time of the subtitle to be verified and the end time of the subtitle to be verified; obtaining the similarity between the subtitle to be verified and the reference subtitle of the target short video based on the start time of the subtitle to be verified and the end time of the subtitle to be verified; determining whether the subtitle to be verified is correct based on the similarity, and obtaining a verification result; returning the verification result to the short video platform, wherein, when the verification result is used to indicate that the subtitle to be verified is incorrect, the verification result is used to enable the short video platform to adjust the subtitle to be verified, and when the verification result is used to indicate that the subtitle to be verified is correct, the verification result is used to enable the short video platform to display the subtitle to be verified when playing the target short video.

[0273] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: in response to a video input instruction acting on an operation interface, displaying a video on the operation interface; in response to a subtitle verification instruction acting on the operation interface, displaying a verification result of the subtitles to be verified of the video on the operation interface, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video.

[0274] As an optional example, a computer-readable storage medium is configured to store program code for executing the following steps: in response to a video input instruction acting on a virtual reality device, displaying a picture of a video for presenting a virtual reality scene on a display screen of the virtual reality device; in response to a subtitle verification instruction acting on the virtual reality device, displaying a verification result of the subtitles to be verified of the video on the display screen, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity being obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified being converted from the audio data of the video; in response to the verification result indicating that the subtitles are correct, displaying the correct subtitles on the display screen.

[0275] Optionally, the computer-readable storage medium may further execute program code for the following steps: in response to the verification result indicating that the subtitles are incorrect, displaying subtitles adjusted based on the verification result on a display screen.

[0276] As an optional example, a computer-readable storage medium is configured to store program code for performing the following steps: in response to a video input instruction acting on an augmented reality device, displaying a video image obtained by shooting a real scene on a display screen of the augmented reality device; in response to a subtitle verification instruction acting on the augmented reality device, displaying a verification result of the subtitles to be verified of the video on the display screen, wherein the verification result is determined based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity is obtained based on the start time and the end time of the subtitles to be verified, and the subtitles to be verified are converted from the audio data of the video; in response to the verification result indicating that the subtitles are correct, displaying the correct subtitles in the image of the video on the display screen.

[0277] The serial numbers of the above embodiments of the present invention are for description only and do not represent the advantages or disadvantages of the embodiments.

[0278] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0279] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0280] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0281] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0282] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0283] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. A method for processing video subtitles, characterized in that: include: Extract audio data from video; Converting the audio data into at least one subtitle to be verified of the video, and determining a start time and an end time of the subtitle to be verified; Obtaining a similarity between the subtitle to be verified and a reference subtitle of the video based on a start time of the subtitle to be verified and an end time of the subtitle to be verified; Determining whether the subtitle to be verified is correct based on the similarity; Wherein, based on the start time and the end time of the subtitle to be verified, obtaining the similarity between the subtitle to be verified and the reference subtitle of the video, includes: in response to the start time of the reference subtitle being between the start time of the first subtitle to be verified and the end time of the first subtitle to be verified, and the end time of the reference subtitle being between the start time of the second subtitle to be verified and the end time of the second subtitle to be verified, merging the first subtitle to be verified and the second subtitle to be verified to obtain a target merged subtitle; sliding the reference subtitle along the target merged subtitle; when the reference subtitle slides one word relative to the target merged subtitle, determining the similarity between the reference subtitle and the corresponding word in the target merged subtitle after sliding; obtaining the maximum similarity point of the reference subtitle in the target merged subtitle based on the similarity between the reference subtitle and the corresponding word; based on the maximum similarity point, comparing the target merged subtitle with the reference subtitle to obtain the similarity; After determining the start time and the end time of the subtitle to be verified, the method further includes: in response to the end time of the reference subtitle being greater than the end time of the subtitle to be verified, adding the end time of the subtitle to be verified to a threshold time.

2. The method according to claim 1, characterized in that Comparing the target merged subtitles with the reference subtitles based on the maximum similarity point to obtain the similarity includes: Based on the maximum similarity point, determining, in the target merged subtitle, a sub-subtitle to be verified having the maximum similarity with the sub-reference subtitle in the reference subtitle; The maximum similarity corresponding to the subtitle to be verified is determined as the similarity.

3. The method according to claim 2, characterized in that Determining whether the subtitle to be verified is correct based on the similarity includes: In response to the similarity being greater than a first target threshold, and the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being less than a second target threshold, determining that the sub-subtitle to be verified is correct; In response to the similarity being not greater than the first target threshold, and / or the difference between the start time of the sub-subtitle to be verified and the start time of the sub-reference subtitle being not less than the second target threshold, it is determined that the sub-subtitle to be verified is erroneous.

4. The method according to claim 2, characterized in that Determining, in the target merged subtitle, a sub-subtitle to be verified having the greatest similarity with a sub-reference subtitle in the reference subtitle, includes: Determining a plurality of sub-merged subtitles to be compared in the target merged subtitles based on the subtitle length of the sub-reference subtitles; Obtaining a similarity between each of the sub-merged subtitles and the sub-reference subtitles to obtain a plurality of similarities; The sub-subtitle corresponding to the maximum similarity among the multiple similarities is determined as the sub-subtitle to be verified.

5. The method according to claim 4, characterized in that The difference between each two adjacent sub-merged subtitles is a target unit subtitle.

6. The method according to claim 1, characterized in that The subtitles to be verified include at least one third subtitle segment to be verified. In response to the start time of the reference subtitle being between the start time of the first subtitle segment to be verified and the end time of the first subtitle segment to be verified, and the end time of the reference subtitle being between the start time of the second subtitle segment to be verified and the end time of the second subtitle segment to be verified, the first subtitle segment to be verified and the second subtitle segment to be verified are merged to obtain a target merged subtitle, including: In response to the start time of the reference subtitle being between the start time of the first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, the middle time of the reference subtitle being between the start time of the third segment of subtitles to be verified and the end time of the third segment of subtitles to be verified, and the end time of the reference subtitle being between the start time of the second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, the first segment of subtitles to be verified, the third segment of subtitles to be verified and the second segment of subtitles to be verified are merged to obtain the target merged subtitle.

7. The method according to any one of claims 1 to 6, characterized in that Converting the audio data into at least one subtitle to be verified for the video includes: Converting the audio data into at least one original subtitle; Invalid subtitles in the original subtitles are adjusted or deleted to obtain the subtitles to be verified.

8. The method according to any one of claims 1 to 6, characterized in that The method further comprises: Obtaining original reference subtitles of the video; Invalid subtitles in the original reference subtitles are adjusted or deleted to obtain the reference subtitles.

9. A video processing method, characterized in that: include: Acquire the video by calling a first interface, wherein the first interface includes a first parameter, and a parameter value of the first parameter is the video; extracting audio data from the video; Converting the audio data into at least one subtitle to be verified of the video, and determining a start time and an end time of the subtitle to be verified; Obtaining a similarity between the subtitle to be verified and a reference subtitle of the video based on a start time of the subtitle to be verified and an end time of the subtitle to be verified; Determining whether the subtitles to be verified are correct based on the similarity, and obtaining a verification result; Outputting the verification result by calling a second interface, wherein the second interface includes a second parameter, and a parameter value of the second parameter is the verification result; Wherein, based on the start time and the end time of the subtitle to be verified, obtaining the similarity between the subtitle to be verified and the reference subtitle of the video, includes: in response to the start time of the reference subtitle being between the start time of the first subtitle to be verified and the end time of the first subtitle to be verified, and the end time of the reference subtitle being between the start time of the second subtitle to be verified and the end time of the second subtitle to be verified, merging the first subtitle to be verified and the second subtitle to be verified to obtain a target merged subtitle; sliding the reference subtitle along the target merged subtitle; when the reference subtitle slides one word relative to the target merged subtitle, determining the similarity between the reference subtitle and the corresponding word in the target merged subtitle after sliding; obtaining the maximum similarity point of the reference subtitle in the target merged subtitle based on the similarity between the reference subtitle and the corresponding word; based on the maximum similarity point, comparing the target merged subtitle with the reference subtitle to obtain the similarity; After determining the start time and the end time of the subtitle to be verified, the method further includes: in response to the end time of the reference subtitle being greater than the end time of the subtitle to be verified, adding the end time of the subtitle to be verified to a threshold time.

10. A method for processing video subtitles, characterized in that: include: Obtain target short videos from short video platforms; Extracting audio data from the target short video; Converting the audio data into at least one subtitle to be verified of the target short video, and determining a start time and an end time of the subtitle to be verified; Based on the start time of the subtitle to be verified and the end time of the subtitle to be verified, obtaining the similarity between the subtitle to be verified and the reference subtitle of the target short video; Determining whether the subtitles to be verified are correct based on the similarity, and obtaining a verification result; Returning the verification result to the short video platform, wherein, when the verification result indicates that the subtitles to be verified are incorrect, the verification result is used to enable the short video platform to adjust the subtitles to be verified; and when the verification result indicates that the subtitles to be verified are correct, the verification result is used to enable the short video platform to display the subtitles to be verified when playing the target short video; Wherein, based on the start time and the end time of the subtitle to be verified, obtaining the similarity between the subtitle to be verified and the reference subtitle of the video, includes: in response to the start time of the reference subtitle being between the start time of the first subtitle to be verified and the end time of the first subtitle to be verified, and the end time of the reference subtitle being between the start time of the second subtitle to be verified and the end time of the second subtitle to be verified, merging the first subtitle to be verified and the second subtitle to be verified to obtain a target merged subtitle; sliding the reference subtitle along the target merged subtitle; when the reference subtitle slides one word relative to the target merged subtitle, determining the similarity between the reference subtitle and the corresponding word in the target merged subtitle after sliding; obtaining the maximum similarity point of the reference subtitle in the target merged subtitle based on the similarity between the reference subtitle and the corresponding word; based on the maximum similarity point, comparing the target merged subtitle with the reference subtitle to obtain the similarity; After determining the start time and the end time of the subtitle to be verified, the method further includes: in response to the end time of the reference subtitle being greater than the end time of the subtitle to be verified, adding the end time of the subtitle to be verified to a threshold time.

11. A method for processing video subtitles, characterized in that: include: In response to a video input instruction applied to an operation interface, displaying a video on the operation interface; In response to a subtitle verification instruction on the operation interface, a verification result of the subtitles to be verified of the video is displayed on the operation interface, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, and the similarity is obtained in response to the start time of the reference subtitles being between the start time of a first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, and the end time of the reference subtitles being between the start time of a second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, then merging the first segment of subtitles to be verified and the second segment of subtitles to be verified to obtain a target merged subtitle; sliding the reference subtitles along the target merged subtitles; and determining the similarity between the reference subtitles and the corresponding words in the target merged subtitles after sliding each time the reference subtitles slide one word relative to the target merged subtitles; and obtaining the similarity based on the similarities between the reference subtitles and the corresponding words; the subtitles to be verified are converted from the audio data of the video; The method further includes: in response to the end time of the reference subtitle being greater than the end time of the subtitle to be verified, adding the end time of the subtitle to be verified to a threshold time.

12. A method for processing video subtitles, characterized in that: include: In response to a video input instruction applied to a virtual reality device, displaying a video image representing a virtual reality scene on a display screen of the virtual reality device; In response to a subtitle verification instruction applied to the virtual reality device, a verification result of the subtitles to be verified of the video is displayed on the display screen, wherein the verification result is obtained by determining the subtitles to be verified based on the similarity between the subtitles to be verified and the reference subtitles of the video, the similarity being obtained in response to the start time of the reference subtitles being between the start time of a first segment of subtitles to be verified and the end time of the first segment of subtitles to be verified, and the end time of the reference subtitles being between the start time of a second segment of subtitles to be verified and the end time of the second segment of subtitles to be verified, then merging the first segment of subtitles to be verified and the second segment of subtitles to be verified to obtain a target merged subtitle; sliding the reference subtitles along the target merged subtitles; and determining the similarity between the reference subtitles and the corresponding words in the target merged subtitles after sliding each time the reference subtitles slide relative to the target merged subtitles; and obtaining the result based on the similarities between the reference subtitles and the corresponding words respectively. In response to the verification result indicating that the subtitles are correct, displaying the correct subtitles on the display screen; The method further includes: in response to the end time of the reference subtitle being greater than the end time of the subtitle to be verified, adding the end time of the subtitle to be verified to a threshold time.

13. The method according to claim 12, characterized in that The method further comprises: In response to the verification result indicating that the subtitles are incorrect, the subtitles adjusted based on the verification result are displayed on the display screen.

14. The method according to claim 12, characterized in that The virtual reality device is a head-mounted reality device.

15. A method for processing video subtitles, characterized in that: include: In response to a video input instruction applied to an augmented reality device, displaying a video image obtained by shooting a real scene on a display screen of the augmented reality device; In response to a subtitle verification instruction applied to the augmented reality device, a verification result of the subtitles to be verified of the video is displayed on the display screen, wherein the verification result is obtained by determining the subtitles to be verified based on a similarity between the subtitles to be verified and a reference subtitle of the video, wherein the similarity is obtained by merging the first segment of subtitles to be verified and the second segment of subtitles to be verified to obtain a target merged subtitle in response to a start time of the reference subtitle being between a start time of a first segment of subtitles to be verified and an end time of the first segment of subtitles to be verified, and an end time of the reference subtitle being between a start time of a second segment of subtitles to be verified and an end time of the second segment of subtitles to be verified; sliding the reference subtitle along the target merged subtitle; and determining a similarity between the reference subtitle and a corresponding word in the target merged subtitle after sliding each time the reference subtitle slides one word relative to the target merged subtitle; and obtaining the similarity based on the similarities between the reference subtitle and the corresponding words. In response to the verification result indicating that the subtitles are correct, displaying the correct subtitles in a picture of the video on the display screen; The method further includes: in response to the end time of the reference subtitle being greater than the end time of the subtitle to be verified, adding the end time of the subtitle to be verified to a threshold time.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium includes a stored program, wherein when the program is executed by a processor, the device where the computer-readable storage medium is located is controlled to execute the method according to any one of claims 1 to 15.

Images