Video subtitle recognition method, device, electronic device and storage medium

Abstract

The present disclosure proposes a method, device, electronic device and storage medium for identifying video subtitles, which relates to the field of computer technology, and in particular to the fields of artificial intelligence technology such as AI medical treatment, computer vision, large models, and deep learning. It includes: performing optical character recognition on a video, obtaining a first phrase set contained in the subtitle area of ​​the video and the first position information corresponding to each first phrase; determining the number of characters contained in each first phrase and the area corresponding to the first position information; filtering the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the area to obtain the first subtitle; identifying the audio data associated with the video to obtain the second subtitle; inputting the first subtitle, the second subtitle and the preset prompt word into the large model to obtain the target subtitle of the video. As a result, the accuracy and recall rate of subtitle recognition are improved, and the overall effect of subtitle processing is improved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the fields of artificial intelligence technology such as AI medicine, computer vision, image processing, large models, and deep learning, and specifically to a method, device, electronic device, and storage medium for recognizing video subtitles. Background Art

[0002] In the video field, the accuracy and recall rate of subtitle extraction and recognition process are the key factors that determine the video quality. Summary of the invention

[0003] The present disclosure aims to solve one of the technical problems in the related art at least to some extent.

[0004] According to a first aspect of the present disclosure, a method for identifying video subtitles is provided, comprising:

[0005] Performing optical character recognition on the video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set, wherein the first position information includes a positioning point coordinate;

[0006] Determining the number of characters included in each of the first phrases and the area corresponding to the first position information;

[0007] filtering the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region to obtain a first subtitle;

[0008] Identifying audio data associated with the video to obtain a second subtitle;

[0009] The first subtitle, the second subtitle and the preset prompt words are input into the large model to obtain the target subtitle of the video.

[0010] According to a second aspect of the present disclosure, a video subtitle recognition device is provided, comprising:

[0011] A first acquisition module, configured to perform optical character recognition on a video to acquire a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set, wherein the first position information includes a positioning point coordinate;

[0012] A determination module, used to determine the number of characters contained in each of the first phrases and the area corresponding to the first position information;

[0013] A second acquisition module, configured to filter the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region, so as to acquire a first subtitle;

[0014] A third acquisition module, used for identifying the audio data associated with the video to obtain a second subtitle;

[0015] The fourth acquisition module is used to input the first subtitles, the second subtitles and preset prompt words into the large model to obtain target subtitles for the video.

[0016] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0017] at least one processor; and

[0018] a memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the video subtitle recognition method as described in the first aspect.

[0020] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the video subtitle recognition method as described in the first aspect.

[0021] According to a fifth aspect of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by a processor, implement the steps of the video subtitle recognition method as described in the first aspect.

[0022] The video subtitle recognition method, device, electronic device and storage medium provided by the present disclosure have the following beneficial effects:

[0023] In the present disclosure, firstly, optical character recognition is performed on the video to obtain the first phrase set contained in the subtitle area of ​​the video and the first position information corresponding to each first phrase in the first phrase set, then the number of characters contained in each first phrase and the area corresponding to the first position information are determined, and then the first phrase set is filtered according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included and the area, so as to obtain the first subtitle, and then the audio data associated with the video is recognized to obtain the second subtitle, and finally the first subtitle, the second subtitle and the preset prompt word are input into the large model to obtain the target subtitle of the video. Thus, by using the large model, the subtitles recognized by OCR and the subtitles and prompt words recognized by audio are processed to obtain the target subtitle of the video, thereby improving the accuracy and recall rate of subtitle recognition and improving the overall effect of subtitle processing.

[0024] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, which are used to better understand the present solution and do not constitute a limitation of the present disclosure, wherein:

[0026] Figure 1 is a flowchart of a method for recognizing video subtitles according to an embodiment of the present disclosure;

[0027] Figure 2 is a flowchart of a method for recognizing video subtitles according to another embodiment of the present disclosure;

[0028] Figure 3 is a flowchart of a method for recognizing video subtitles according to another embodiment of the present disclosure;

[0029] Figure 4 is a flowchart of a method for recognizing video subtitles according to another embodiment of the present disclosure;

[0030] Figure 5 is a flowchart of a method for recognizing video subtitles according to another embodiment of the present disclosure;

[0031] Figure 6 is a flowchart of a method for recognizing video subtitles according to another embodiment of the present disclosure;

[0032] Figure 7 is a structural schematic diagram of a video subtitle recognition device proposed according to an embodiment of the present disclosure;

[0033] Figure 8 A block diagram of an exemplary electronic device suitable for implementing embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0034] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0035] The embodiments of the present disclosure relate to the fields of artificial intelligence technologies such as computer vision, large models, and deep learning.

[0036] Artificial Intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence.

[0037] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sounds. The ultimate goal of deep learning is to enable machines to have analytical learning capabilities like humans and to be able to recognize data such as text, images, and sounds.

[0038] Computer vision refers to the use of cameras and computers to replace the human eye to identify, track and measure targets, and further perform graphic processing so that the computer processing becomes an image that is more suitable for human eye observation or transmission to instrument detection.

[0039] Image processing is a technique that uses computers to analyze images to achieve the desired results. It is also called image processing. Image processing generally refers to digital image processing. A digital image is a large two-dimensional array obtained by shooting with industrial cameras, video cameras, scanners and other equipment. The elements of the array are called pixels, and their values ​​are called grayscale values. Image processing technology generally includes three parts: image compression, enhancement and restoration, matching, description and recognition.

[0040] AI healthcare refers to the application of artificial intelligence technology and big data services in the medical industry based on the Internet, through the construction of infrastructure and data collection, to improve the diagnostic efficiency and service quality of the medical industry.

[0041] Large models refer to machine learning models with a large number of parameters and complex structures. These models can be applied to process large-scale data and complex problems.

[0042] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0043] The following describes a video subtitle recognition method, device, electronic device, and storage medium according to an embodiment of the present disclosure with reference to the accompanying drawings.

[0044] It should be noted that the execution subject of the video subtitle recognition method of this embodiment is a video subtitle recognition device, which can be implemented by software and / or hardware, and can be configured in an electronic device, which can include but is not limited to a terminal, a server, etc. In the disclosed embodiment, the video subtitle recognition device is configured in a subtitle recognition system as an example for description.

[0045] Figure 1 It is a flowchart of a method for recognizing video subtitles proposed according to an embodiment of the present disclosure.

[0046] like Figure 1 As shown, the video subtitle recognition method includes:

[0047] S101: Perform optical character recognition on a video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set.

[0048] The first position information includes the coordinates of the positioning point.

[0049] In the disclosed embodiment, the subtitle recognition system may first extract the text portion of all frames in the video using optical character recognition (OCR) technology to determine all phrases contained in the video. Then, based on the first position information of the extracted phrases and the preset subtitle area range of the video, the first phrase set contained in the subtitle area is obtained from all the extracted phrases.

[0050] It should be noted that the first position information corresponding to the first phrase can be any information that can determine the position of the first phrase in the video frame. For example, the first position information can include the coordinates of the vertices of two diagonal corners of a positioning frame; or, if each positioning frame is of a fixed size, the first position information can also be the coordinates of the center point of the positioning frame where the first phrase is located, etc., and the present disclosure does not limit this.

[0051] S102: Determine the number of characters included in each first phrase and the area corresponding to the first position information.

[0052] In the disclosed embodiment, the subtitle recognition system may calculate the area of ​​the region corresponding to the first position information according to the positioning frame determined by the first position information. For example, the area of ​​the region corresponding to the first position information may be determined by calculating the product of the length and width of the positioning frame.

[0053] It should be noted that the number of characters contained in each first phrase may be the same or different, and the present disclosure does not limit this.

[0054] S103: filtering the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region to obtain a first subtitle.

[0055] In the disclosed embodiment, the subtitle recognition system can determine the relative positions between the positioning points according to the coordinates of the positioning points corresponding to the first phrases, and then determine the arrangement direction of the video subtitle content and the arrangement direction of each first phrase content based on the relative positions. After that, the first phrases that are different from the arrangement direction of the video subtitle content can be concentrated and filtered.

[0056] Alternatively, the font area corresponding to each first character may be calculated according to the number of characters included in the first phrase and the corresponding area, and then based on the reference font area, the first phrases in the first phrase set whose corresponding font areas are too large or too small may be filtered out.

[0057] It should be noted that after the subtitle recognition system obtains the first phrase set after noise reduction, it can obtain the first subtitle by concatenating all the first phrases in the first phrase set.

[0058] In the disclosed embodiment, after filtering the first phrase set, the filtered phrases may be spliced ​​based on the filtered first phrases and the corresponding first position information to obtain the first subtitles.

[0059] In the disclosed embodiment, the subtitle recognition system can obtain the sequence of the first phrases according to the first position information of the first phrases, and then can splice the filtered first phrases according to the sequence to obtain the first subtitles, thereby improving the accuracy and reliability of the extracted subtitles while ensuring the semantic logic of the extracted subtitles.

[0060] S104: Identify audio data associated with the video to obtain a second subtitle.

[0061] In the disclosed embodiment, the subtitle recognition system can extract audio data of the video and convert it into text to obtain second subtitles.

[0062] S105: Input the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain target subtitles for the video.

[0063] In the disclosed embodiment, the subtitle recognition system can input the prompt words input by the user or the prompt words determined according to the video into the large model together with the first subtitles and the second subtitles, complete a higher level of subtitle content understanding and rewriting based on the first subtitles and the second subtitles, and obtain the target subtitles output by the large model.

[0064] Optionally, a preset prompt word may be determined according to attribute information of the video.

[0065] The attribute information of the video may include at least one of the type of the video, the field to which the video content belongs, the length of the video, and the generation time of the video.

[0066] It should be noted that the type of video may be film and television, popular science, education, etc., and the field to which it belongs may be medicine, aerospace, navigation, sports, etc., and the present disclosure does not limit this. In addition, due to the different lengths of videos, the content that needs to be repaired in the subtitles may not be exactly the same. Therefore, for videos of 3 minutes, 10 minutes, etc., the input prompt words may be different. Or, for the generation time of the video, such as videos from the 1980s, considering that people's speaking habits are different from those of modern people, the generation time of the video can also be added to the prompt words. By considering the various attribute information of the video when determining the prompt words, the key information of the video can be summarized more comprehensively and accurately, thereby improving the reliability and diversity of the prompt words.

[0067] In the disclosed embodiment, preset prompt words are determined based on the attribute information of the video, which ensures a strong correlation between the prompt words and the video content, avoids the problem of semantic deviation when outputting subtitles by a large model, and improves the recall rate and accuracy of the subtitles.

[0068] In this embodiment, the subtitle recognition system first performs optical character recognition on the video, obtains the first phrase set contained in the subtitle area of ​​the video and the first position information corresponding to each first phrase in the first phrase set, then determines the number of characters contained in each first phrase and the area corresponding to the first position information, and then filters the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area to obtain the first subtitle, then recognizes the audio data associated with the video to obtain the second subtitle, and finally inputs the first subtitle, the second subtitle, and the preset prompt word into the large model to obtain the target subtitle of the video. Thus, by using the large model, the subtitles recognized by OCR and the subtitles and prompt words recognized by audio are processed to obtain the target subtitle of the video, thereby improving the accuracy and recall rate of subtitle recognition and improving the overall effect of subtitle processing.

[0069] Figure 2 It is a flowchart of a method for recognizing video subtitles proposed in another embodiment of the present disclosure.

[0070] like Figure 2 As shown, the video subtitle recognition method includes:

[0071] S201: Perform optical character recognition on a video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set.

[0072] The first position information includes the coordinates of the positioning point.

[0073] S202: Determine the number of characters included in each first phrase and the area corresponding to the first position information.

[0074] The description of S201 and S202 can be specifically referred to the above embodiment, which will not be repeated here.

[0075] S203: Determine a first arrangement direction of subtitle content in the video.

[0076] The arrangement direction may be from left to right, or from top to bottom, etc.

[0077] It should be noted that the first arrangement direction of the subtitle content may be pre-set. Alternatively, because the subtitle content contained in a video usually accounts for a large proportion of the total character content contained therein, the first arrangement direction of the subtitle content may also be determined based on the relative positions of most of the first phrases in the identified first phrase set. For example, if most of the first phrase contents in the first phrase set are arranged "from left to right", then the first arrangement direction of the subtitle content may be determined to be "from left to right".

[0078] It should be noted that the descriptions such as "from left to right", "from top to bottom", "from bottom to top" in this disclosure are only a way to describe the arrangement direction of the subtitle content and cannot be used as a restrictive description of the arrangement direction of the subtitle content.

[0079] Optionally, if the first phrase corresponds to at least two positioning point coordinates, the first relative position between the positioning points corresponding to the first phrase can be determined based on the positioning point coordinates corresponding to the first phrase, and then when the ratio between the first number of first phrases corresponding to any first relative position and the second number is greater than a first threshold, the first arrangement direction of the subtitle content in the video can be determined based on any first relative position.

[0080] The second number is the number of first phrases included in the first phrase set.

[0081] In the disclosed embodiment, the subtitle recognition system may first determine the relationship between the two diagonal vertex coordinates of each first phrase based on the coordinates of the positioning points of each first phrase (for example, the coordinates of the two diagonal vertices of the first phrase). For example, the first threshold is 0.6, and it is determined by judging that in the first phrase set, the first relative positions between the two diagonal vertex coordinates of the first phrase A and the first phrase B are respectively "the latter vertex is located at the lower right of the former vertex", and the first relative position between the two diagonal vertex coordinates of the first phrase C is "the latter vertex is located at the lower left of the former vertex". In other words, the number of first phrases corresponding to "the latter vertex is located at the lower right of the former vertex" is 2, and the total number of phrases is 3, 2 / 3>0.6, so it can be determined that the first arrangement direction of the subtitle content in the video is "the latter vertex is located at the lower right of the former vertex", that is, the subtitles are arranged from left to right.

[0082] In the disclosed embodiment, by determining the relative positions between the anchor points of most phrases and determining the arrangement direction of the subtitle content, the accuracy of subtitle recognition is further improved, providing more accurate and reliable conditions for noise filtering in subtitles.

[0083] S204: Determine a second arrangement direction of the content in the first phrase according to the coordinates of the positioning point corresponding to the first phrase.

[0084] In the disclosed embodiment, after confirming the arrangement direction of the video subtitle content, the subtitle recognition system can filter out noise in the first phrase by confirming the second arrangement direction of the content in the first phrase.

[0085] Optionally, when the positioning point coordinates are the coordinates of a specified position in the area where the first phrase is located, the second relative position between the positioning points corresponding to the first phrase can be determined based on the positioning point coordinates corresponding to the first phrase, and then the second arrangement direction of the content in the first phrase can be determined based on the second relative position.

[0086] The coordinates of the designated position may be the coordinates of the upper left vertex where the phrase starts and the coordinates of the lower right vertex where the phrase ends in the area where each phrase is located.

[0087] In the disclosed embodiment, the subtitle recognition system can determine the relationship between the two vertices of each first phrase according to the coordinates of the upper left vertex and the lower right vertex of the area where the first phrase is located. For example, when the second relative position between the coordinates of the two vertices of the first phrase D is "the upper left vertex is located below the left of the lower right vertex", the second arrangement direction of the content in the first phrase can be determined to be "the upper left vertex is located below the left of the lower right vertex", that is, the first phrase is arranged from bottom to top.

[0088] In the disclosed embodiment, by using the coordinates of the specified position in the first phrase to determine the second arrangement direction of the content in the first phrase, the accuracy of determining the character arrangement in a single phrase is improved, providing conditions for filtering phrase noise in subtitles.

[0089] S205: Filter out the first phrases in a second arrangement direction different from the first arrangement direction to obtain a filtered first phrase set.

[0090] For example, when the first arrangement direction of the subtitle content in the video is from left to right, and the second arrangement direction of the content in the first phrase D is from bottom to top, the subtitle recognition system can consider that the first phrase D is not subtitle content and filter it out.

[0091] S206: Generate a first subtitle based on the filtered first phrase set.

[0092] In the embodiment of the present disclosure, the subtitle recognition system may sequentially splice the first phrases based on the first position information corresponding to the first phrases in the filtered first phrase set to generate the first subtitles.

[0093] Optionally, the subtitle recognition system can also determine the order of the first position information according to the first arrangement direction of the subtitle content, and then splice the filtered first phrases based on the order to obtain the first subtitles, thereby improving the accuracy and reliability of the extracted subtitles while ensuring the semantic logic of the extracted subtitles.

[0094] For example, the filtered first phrase set includes the first phrase A, the first phrase B, and the first phrase C, and the corresponding first position information is coordinate a, coordinate b, and coordinate c respectively. When the first arrangement direction of the subtitle content is from left to right, since the order of change of coordinate a, coordinate b, and coordinate c from left to right is coordinate b, coordinate a, coordinate c, the first subtitle spliced ​​based on this order is the first phrase B+first phrase A+first phrase C.

[0095] S207: Identify the audio data associated with the video to obtain a second subtitle.

[0096] S208: Input the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain target subtitles for the video.

[0097] The description of the above S206-S208 can be specifically referred to the above embodiment, and will not be repeated here.

[0098] In this embodiment, the subtitle recognition system first determines the first arrangement direction of the subtitle content in the video, and then determines the second arrangement direction of the content in the first phrase according to the coordinates of the positioning point corresponding to the first phrase, and then filters out the first phrases in the second arrangement direction different from the first arrangement direction to obtain a filtered first phrase set, and generates the first subtitle based on the filtered first phrase set. In this way, the noise phrases contained in the subtitles can be filtered based on the arrangement mode, which reduces the invalid data in the extracted subtitles and further improves the accuracy and reliability of subtitle recognition.

[0099] Figure 3 It is a flowchart of a method for recognizing video subtitles proposed in another embodiment of the present disclosure.

[0100] like Figure 3 As shown, the video subtitle recognition method includes:

[0101] S301: Perform optical character recognition on a video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set.

[0102] The first position information includes the coordinates of the positioning point.

[0103] S302: Determine the number of characters included in each first phrase and the area corresponding to the first position information.

[0104] The description of S301 and S302 can be specifically referred to the above embodiment, which will not be repeated here.

[0105] S303: Determine a font area corresponding to the first phrase according to the number of characters included in the first phrase and the area corresponding to the first position information.

[0106] In the disclosed embodiment, the subtitle recognition system can calculate the font area corresponding to the first phrase by using the number of characters included in the first phrase and the area corresponding to the first position information. For example, the font area corresponding to the first phrase can be determined by dividing the area by the number of characters.

[0107] S304: Determine a reference font area of ​​the subtitles according to the font areas corresponding to the first phrases.

[0108] In the disclosed embodiment, the subtitle recognition system can determine the reference font area of ​​the subtitles by calculating the average value of the font areas corresponding to all first phrases, or counting the most frequent value among the font areas corresponding to all first phrases, and so on.

[0109] Optionally, the mean or median of the font areas corresponding to the first phrases in the first phrase set may be determined as the reference font area.

[0110] In the disclosed embodiment, the subtitle recognition system may add up the font areas corresponding to all first phrases in the first phrase set, divide the sum by the number of first phrases to obtain the mean value, and determine it as the reference font area. Alternatively, the font areas corresponding to all first phrases in the first phrase set may be arranged in order of size to obtain the median value of the font area as the reference font area. This may make the determined reference font area more consistent with the font size in the video subtitles, and provide a more reasonable basis for subsequent phrase filtering.

[0111] Optionally, when the ratio of the third number of first phrases whose corresponding font areas are all the same to the second number is greater than a third threshold, the font areas corresponding to the third number of first phrases may be determined as reference font areas.

[0112] The second number is the number of first phrases included in the first phrase set. The third threshold may be a fixed value or a value determined according to the noise level of the video background. For example, when the background noise level is higher, the value of the third threshold may be larger, and the present disclosure does not limit this.

[0113] In the disclosed embodiment, the subtitle recognition system can count the third number of first phrases with the same corresponding font area. For example, the third threshold is 0.6. When the number of first phrases with a corresponding font area of ​​X in the first phrase set is 4, and there are 5 phrases in the first phrase set, according to 4 / 5>0.6, it can be determined that the reference font area is X.

[0114] In the disclosed embodiment, by determining the same font area corresponding to most of the first phrases in the first phrase set as the reference font area of ​​the video subtitles, the determined reference font area can be made more representative, providing a more reasonable data basis for subsequent phrase filtering.

[0115] S305: When the absolute value of the difference between the font area corresponding to any first phrase and the reference font area is greater than a second threshold, filter out any first phrase to obtain a filtered first phrase set.

[0116] In the disclosed embodiment, the subtitle recognition system can pre-set a second threshold value. When the absolute value of the difference between the font area corresponding to any first phrase and the reference font area is greater than the second threshold value, it means that the font area corresponding to the first phrase is too large or too small, which may be a tiny font or noise in the video background. Therefore, it can be filtered out to eliminate the interference of non-subtitle content and obtain a first subtitle with higher accuracy after noise reduction.

[0117] S306: Generate a first subtitle based on the filtered first phrase set.

[0118] S307: Identify the audio data associated with the video to obtain a second subtitle.

[0119] S308: Input the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain the target subtitle of the video.

[0120] The description of the above S306-S308 can be specifically referred to the above embodiment, which will not be repeated here.

[0121] In this embodiment, the subtitle recognition system first determines the font area corresponding to the first phrase according to the number of characters included in the first phrase and the area corresponding to the first position information, and then determines the reference font area of ​​the subtitle according to the font area corresponding to each first phrase. After that, when the absolute value of the difference between the font area corresponding to any first phrase and the reference font area is greater than the second threshold, any first phrase is filtered out, and the first subtitle is generated based on the filtered first phrase set. Thus, based on the font area, the noise of the first phrase is removed, and the original video subtitle after noise reduction is obtained, which reduces the interference of the video background and further improves the accuracy and reliability of subtitle recognition.

[0122] Figure 4 It is a flowchart of a method for recognizing video subtitles proposed in another embodiment of the present disclosure.

[0123] like Figure 4 As shown, the video subtitle recognition method includes:

[0124] S401: Perform optical character recognition on a video to obtain a second phrase set contained in the video and second position information corresponding to each second phrase in the second phrase set.

[0125] The second phrase set refers to a collection of all phrases extracted from all frames of the video, and may include phrases in the subtitle area and the background area.

[0126] S402: Filter the second phrase set based on the matching degree between each second position information and the subtitle area.

[0127] The subtitle area may be a preset range, or may be a subtitle area range of a video identified by a subtitle recognition system through image recognition, for example, the bottom, middle, left side, etc. of the video screen, which is not limited in the present disclosure.

[0128] In the disclosed embodiment, the subtitle recognition system can calculate the overlapping area between the positioning box determined by each second position information and the subtitle area based on the second position information corresponding to each second phrase to obtain the matching degree. When the overlapping area is small, that is, the matching degree is low, the corresponding second phrase can be deleted to obtain a filtered second phrase set.

[0129] In the disclosed embodiment, by filtering the position information that does not match the subtitle area, the noise reduction of the phrase set is achieved, the reliability of subtitle recognition is further improved, and stable and reliable data is provided for subsequent subtitle processing.

[0130] S403: Determine the filtered candidate phrases.

[0131] The candidate phrase is a second phrase corresponding to the second position information in the subtitle area.

[0132] S404: Determine the semantic similarity between every two adjacent candidate phrases.

[0133] In the disclosed embodiment, since the phrases extracted by optical character recognition may have semantic duplication, the subtitle recognition system may calculate the semantic similarity between every two adjacent candidate phrases to determine whether there is semantic duplication between the candidate phrases.

[0134] S405: When the similarity between any two candidate phrases is greater than a fourth threshold, remove any one of the two candidate phrases to obtain a first phrase set and first position information corresponding to each first phrase in the first phrase set.

[0135] In the disclosed embodiment, when the similarity between any two candidate phrases is greater than a fourth threshold, the subtitle recognition system may deem that the two candidate phrases have the same semantics, and thus remove any one of the two candidate phrases to obtain a phrase set in which the similarity between any adjacent candidate phrases is less than the fourth threshold as the first phrase set, and use the second position information corresponding to the candidate phrase as the first position information corresponding to the corresponding first phrase in the first phrase set.

[0136] In the disclosed embodiment, by calculating the semantic similarity between adjacent phrases, semantically repeated words in the phrase set are filtered out, thereby improving the quality and reliability of subtitle extraction, reducing duplicate data in subtitle processing, and providing conditions for improving processing efficiency.

[0137] S406: Determine the number of characters included in each first phrase and the area corresponding to the first position information.

[0138] S407: Filter the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region to obtain a first subtitle.

[0139] S408: Identify the audio data associated with the video to obtain a second subtitle.

[0140] S409: Input the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain the target subtitle of the video.

[0141] The description of the above S406-S409 can be specifically referred to the above embodiment, which will not be repeated here.

[0142] In this embodiment, by screening the position information that does not match the subtitle area, the noise reduction of the phrase set is achieved, and by calculating the semantic similarity between adjacent phrases, the semantically repeated words in the phrase set are filtered out, thereby improving the quality and reliability of subtitle extraction, reducing noise data and repeated data in subtitle processing, and providing conditions for improving processing efficiency and stability.

[0143] Figure 5 It is a flowchart of a method for recognizing video subtitles proposed in another embodiment of the present disclosure.

[0144] like Figure 5 As shown, the video subtitle recognition method includes:

[0145] S501: Perform optical character recognition on a video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set.

[0146] The description of S501 above can be specifically referred to the above embodiment, which will not be repeated here.

[0147] S502: When the field to which the video content belongs is the medical field, obtain a medical field reference word set.

[0148] Among them, the medical field reference word set is a word set generated based on medical field data, which may include various professional word sets in the medical field, such as drug names, disease names, drug formulas, etc., and the present disclosure does not limit this.

[0149] In the present disclosure, when the video content of the subtitles to be recognized is a medical-related video, such as a disease prevention promotional video, or a live doctor's consultation, etc., the subtitle recognition system can obtain a reference word set in the medical field to calibrate the written form of the medical terms involved in the video content, such as the name of the disease, or the name of the drug, etc.

[0150] S503: Match each first word group with each reference word in the reference word set.

[0151] In the disclosed embodiment, the subtitle recognition system can determine whether the first phrase extracted by OCR technology is in the correct form by matching each first phrase in the extracted first phrase set with each reference word in the reference phrase set, and determining whether there is a reference word in the reference phrase set that is identical to or has a higher matching degree with the first phrase.

[0152] S504: When any first word group is not included in the reference word set and the matching degree between any first word group and any reference word is greater than a fifth threshold, update any first word group based on any reference word.

[0153] It should be noted that the subtitle recognition system can determine the similarity between the first phrase and the reference word by calculating the similarity between the characters included in the first phrase and the characters in the reference word, and the present disclosure does not limit this.

[0154] In the embodiment of the present disclosure, when any first phrase is not included in the reference word set, and the matching degree between any first phrase and any reference word is greater than the fifth threshold, it may indicate that any first phrase is a professional phrase in the medical field, but contains erroneous characters, so any first phrase can be replaced by the closest reference word.

[0155] The size of the fifth threshold can be set as needed, and the present disclosure does not limit this.

[0156] For example, if the first phrase "Antheophylline" fails to find the same reference word in the reference word set, but a reference word "aminophylline" with a matching degree greater than the fifth threshold can be determined in the reference phrase through character matching, then the first phrase "Antheophylline" can be updated with "aminophylline". Alternatively, the first phrase "Mecobalamin" can be updated with the reference word "Mecobalamin".

[0157] It should be noted that if a first phrase is not included in the reference phrase set, and the matching degree between the first phrase and each reference phrase in the reference phrase set is relatively small, it means that the first phrase is not a phrase in the medical field.

[0158] S505: Determine the number of characters included in each updated first phrase and the area corresponding to the first position information.

[0159] S506: Filter the updated first phrase set according to at least one of the coordinates of the positioning point corresponding to the updated first phrase, the number of characters included, and the area of ​​the region to obtain a first subtitle.

[0160] S507: Identify the audio data associated with the video to obtain a second subtitle.

[0161] It should be noted that in the embodiments of the present disclosure, the second subtitle of the speech recognition can also be corrected according to the reference word set in the medical field. Since there may be incorrect readings of some medical terms in the video, for example, reading mecobalamin as "jiaguan'an", which may cause the subtitle recognition system to obtain incorrect characters "an" in the second subtitle after recognizing the audio data, the incorrect characters in the second subtitle can be corrected using the reference word set in the medical field.

[0162] S508: Input the first subtitle, the second subtitle, and the preset prompt words into the large model to obtain the target subtitle of the video.

[0163] It should be noted that the subtitle recognition system can also fine-tune the parameters of the large model based on the reference word set in the medical field to further improve the applicability and reliability of the subtitles generated by the large model.

[0164] For the descriptions of S505 - S508 above, specific reference can be made to the above embodiments and will not be elaborated here.

[0165] In this embodiment, when the field to which the video content belongs is the medical field, the subtitle recognition system first obtains the reference word set in the medical field, then matches each first phrase with each reference word in the reference word set, and then, when none of the first phrases are included in the reference word set and the matching degree between any first phrase and any reference word is greater than the matching degree with other reference words, updates any first phrase based on any reference word. Thus, the applicability of the subtitle recognition method in the present disclosure in the medical scenario can be realized, and the professionalism and reliability of the subtitle recognition result can be further improved.

[0166] Figure 6 It is a schematic flowchart of the method for recognizing video subtitles proposed in another embodiment of the present disclosure.

[0167] As Figure 6 shown, the method for recognizing video subtitles includes:

[0168] S601: Perform optical character recognition on the video to obtain the first phrase set included in the subtitle area of the video and the first position information corresponding to each first phrase in the first phrase set.

[0169] For the description of S601 above, specific reference can be made to the above embodiments and will not be elaborated here.

[0170] S602: When any first phrase contains a preset symbol and the field to which the video content belongs is the medical field, obtain the reference word pair set in the medical field.

[0171] The preset characters may be any medical symbols related to the medical field, and may be composed of letters, numbers or abbreviations, such as drug symbols PO (abbreviation of per os, indicating oral administration), IV (intravenous, indicating intravenous injection), ICD (abbreviation of International Classification of Diseases and Related Health Problems), etc. The reference word pair set includes reference words and associated medical symbols.

[0172] S603: Match the preset symbol with each medical symbol in the reference word pair set.

[0173] S604: When any medical symbol matches a preset symbol, a reference word associated with any medical symbol is used to replace the preset symbol in any first phrase to obtain an updated first phrase.

[0174] For example, when the preset symbol "PO" included in the first phrase can find the same medical symbol "PO" in the reference word pair set, the reference word "oral" associated with the medical symbol "PO" can be used to replace the preset symbol to update the first phrase.

[0175] S605: Determine the number of characters included in each updated first phrase and the area corresponding to the first position information.

[0176] S606: Filter the updated first phrase set according to at least one of the coordinates of the positioning point corresponding to the updated first phrase, the number of characters included, and the area of ​​the region to obtain a first subtitle.

[0177] S607: Identify the audio data associated with the video to obtain a second subtitle.

[0178] S608: Input the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain the target subtitle of the video.

[0179] The description of the above S605-S608 can be specifically referred to the above embodiment, which will not be repeated here.

[0180] In this embodiment, when a preset symbol is included in any first phrase, and the field to which the video content belongs is the medical field, the subtitle recognition system first obtains a reference word pair set in the medical field, and then matches the preset symbol with each medical symbol in the reference word pair set. After that, when any medical symbol matches the preset symbol, the reference word associated with any medical symbol is used to replace the preset symbol in any first phrase to obtain an updated first phrase. Thus, by updating the first phrase based on the reference word associated with the symbol, the readability of the subtitle recognition result is improved, and the user experience is optimized.

[0181] Figure 7 It is a structural diagram of a video subtitle recognition device proposed in an embodiment of the present disclosure.

[0182] like Figure 7 As shown, the video subtitle recognition device 700 includes:

[0183] A first acquisition module 701 is used to perform optical character recognition on the video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set, wherein the first position information includes a positioning point coordinate;

[0184] A determination module 702, configured to determine the number of characters contained in each first phrase and the area corresponding to the first position information;

[0185] A second acquisition module 703 is used to filter the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region, so as to obtain a first subtitle;

[0186] A third acquisition module 704 is used to identify audio data associated with the video to obtain a second subtitle;

[0187] The fourth acquisition module 705 is used to input the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain the target subtitle of the video.

[0188] In some embodiments, the second acquisition module 703 is specifically used to:

[0189] Determine the first arrangement direction of subtitle content in the video;

[0190] Determining a second arrangement direction of the content in the first phrase according to the coordinates of the positioning point corresponding to the first phrase;

[0191] filtering out the first phrases in a second arrangement direction different from the first arrangement direction to obtain a filtered first phrase set;

[0192] A first subtitle is generated based on the filtered first phrase set.

[0193] In some embodiments, the second acquisition module 703 is specifically used to:

[0194] Determining first relative positions between the positioning points corresponding to the first phrase according to the coordinates of the positioning points corresponding to the first phrase;

[0195] When the ratio of the first number of first phrases corresponding to any first relative position to the second number is greater than a first threshold, a first arrangement direction of the subtitle content in the video is determined according to any first relative position, wherein the second number is the number of first phrases included in the first phrase set.

[0196] In some embodiments, the second acquisition module 703 is specifically used to:

[0197] In a case where the coordinates of the positioning points are the coordinates of the designated position of the area where the first phrase is located, determining the second relative positions between the positioning points corresponding to the first phrase according to the coordinates of the positioning points corresponding to the first phrase;

[0198] A second arrangement direction of the content in the first phrase is determined according to the second relative position.

[0199] In some embodiments, the second acquisition module 703 is specifically used to:

[0200] Determining a font area corresponding to the first phrase according to the number of characters included in the first phrase and the area corresponding to the first position information;

[0201] Determining a reference font area of ​​the subtitles according to the font area corresponding to each first phrase;

[0202] When the absolute value of the difference between the font area corresponding to any first phrase and the reference font area is greater than a second threshold, filtering out any first phrase to obtain a filtered first phrase set;

[0203] A first subtitle is generated based on the filtered first phrase set.

[0204] In some embodiments, the second acquisition module 703 is specifically used to:

[0205] The mean value or median value of the font areas corresponding to the first phrases in the first phrase set is determined as the reference font area.

[0206] In some embodiments, the second acquisition module 703 is specifically used to:

[0207] When the ratio of the third number of first phrases whose corresponding font areas are the same to the second number is greater than a third threshold, the font areas corresponding to the third number of first phrases are determined as the reference font area, wherein the second number is the number of first phrases included in the first phrase set.

[0208] In some embodiments, the first acquisition module 701 is specifically used to:

[0209] Performing optical character recognition on the video to obtain a second phrase set contained in the video and second position information corresponding to each second phrase in the second phrase set;

[0210] Based on the matching degree between each second position information and the subtitle area, the second phrase set is filtered to determine the first phrases located in the subtitle area and the first position information corresponding to each first phrase.

[0211] In some embodiments, the first acquisition module 701 is specifically used to:

[0212] Determine the filtered candidate phrases;

[0213] Determine the semantic similarity between each two adjacent candidate phrases;

[0214] When the similarity between any two candidate phrases is greater than a fourth threshold, any one of the two candidate phrases is removed to obtain the first phrase set and the first position information corresponding to each first phrase in the first phrase set.

[0215] In some embodiments, the fourth acquisition module 705 is further used to:

[0216] According to the attribute information of the video, the preset prompt word is determined.

[0217] In some embodiments, in the above-mentioned video subtitle recognition device 700, the attribute information of the video includes at least one of the following: the type of the video, the field to which the video content belongs, the length of the video, and the generation time of the video.

[0218] In some embodiments, the first acquisition module 701 is further used to:

[0219] When the field to which the video content belongs is the medical field, obtaining a reference word set in the medical field;

[0220] matching each first word group with each reference word in the reference word set;

[0221] When any first word group is not included in the reference word set and the matching degree between any first word group and any reference word is greater than a fifth threshold, any first word group is updated based on any reference word.

[0222] In some embodiments, the first acquisition module 701 is further used to:

[0223] When any of the first word groups contains a preset symbol and the field to which the video content belongs is the medical field, obtaining a reference word pair set in the medical field, wherein the reference word pair set includes reference words and associated medical symbols;

[0224] Matching the preset symbol with each medical symbol in the reference word pair set;

[0225] In the case that any medical symbol matches a preset symbol, a reference word associated with any medical symbol is used to replace the preset symbol in any first phrase to obtain an updated first phrase.

[0226] It should be noted that the above explanation of the video subtitle recognition method is also applicable to the video subtitle recognition device of this embodiment, and will not be repeated here.

[0227] In this embodiment, the subtitle recognition system first performs optical character recognition on the video, obtains the first phrase set contained in the subtitle area of ​​the video and the first position information corresponding to each first phrase in the first phrase set, then determines the number of characters contained in each first phrase and the area corresponding to the first position information, and then filters the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area to obtain the first subtitle, then recognizes the audio data associated with the video to obtain the second subtitle, and finally inputs the first subtitle, the second subtitle, and the preset prompt word into the large model to obtain the target subtitle of the video. Thus, by using the large model, the subtitles recognized by OCR and the subtitles and prompt words recognized by audio are processed to obtain the target subtitle of the video, thereby improving the accuracy and recall rate of subtitle recognition and improving the overall effect of subtitle processing.

[0228] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0229] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0230] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0231] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0232] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as a method for recognizing video subtitles. For example, in some embodiments, the method for recognizing video subtitles may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the method for recognizing video subtitles described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform a method for recognizing video subtitles in any other appropriate manner (e.g., by means of firmware).

[0233] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0234] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0235] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0236] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0237] The systems and techniques described herein can be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), the Internet, and a blockchain network.

[0238] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship between the client and the server is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server may also be a server of a distributed system, or a server combined with a blockchain.

[0239] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0240] In addition, the terms "first" and "second" are used for descriptive purposes only and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In the description of the present disclosure, the words "if" and "if" used can be interpreted as "at the time of" or "when" or "in response to determination" or "under the circumstances of".

[0241] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for recognizing video subtitles, comprising: Performing optical character recognition on the video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set, wherein the first position information includes a positioning point coordinate; Determining the number of characters included in each of the first phrases and the area corresponding to the first position information; filtering the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region to obtain a first subtitle; Identifying audio data associated with the video to obtain a second subtitle; Inputting the first subtitle, the second subtitle and the preset prompt words into the macro model to obtain target subtitles for the video; The filtering of the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region to obtain the first subtitle includes: Determine a first relative position between the anchor points corresponding to the first phrase according to the anchor point coordinates corresponding to the first phrase, wherein the anchor point coordinates corresponding to the first phrase include two diagonal vertex coordinates of the first phrase; When a ratio between a first number of first phrases corresponding to any first relative position and a second number is greater than a first threshold, determining a first arrangement direction of subtitle content in the video according to any first relative position, wherein the second number is the number of first phrases included in the first phrase set; In the case where the positioning point coordinates are the coordinates of the designated position of the area where the first phrase is located, determining the second relative position between the positioning points corresponding to the first phrase according to the positioning point coordinates corresponding to the first phrase, wherein the coordinates of the designated position include the coordinates of the upper left vertex where the phrase starts and the coordinates of the lower right vertex where the phrase ends in the area where each phrase is located; determining a second arrangement direction of the content in the first phrase according to the second relative position; filtering out first phrases in a second arrangement direction different from the first arrangement direction to obtain a filtered first phrase set; The first subtitle is generated based on the filtered first phrase set.

2. The method of claim 1, wherein: The filtering process of the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region to obtain the first subtitle comprises: Determining a font area corresponding to the first phrase according to the number of characters included in the first phrase and the area corresponding to the first position information; Determining a reference font area of ​​the subtitle according to the font area corresponding to each of the first phrases; When the absolute value of the difference between the font area corresponding to any first phrase and the reference font area is greater than a second threshold, filtering out any first phrase to obtain a filtered first phrase set; The first subtitle is generated based on the filtered first phrase set.

3. The method of claim 2, wherein: The step of determining a reference font area of ​​the subtitle according to the font area corresponding to each of the first phrases includes: The mean value or median value of the font areas corresponding to the first phrases in the first phrase set is determined as the reference font area.

4. The method of claim 2, wherein: The step of determining a reference font area of ​​the subtitle according to the font area corresponding to each of the first phrases includes: When the ratio of the third number of first phrases whose corresponding font areas are the same to the second number is greater than a third threshold, the font areas corresponding to the third number of first phrases are determined as the reference font area, wherein the second number is the number of first phrases included in the first phrase set.

5. The method according to any one of claims 1 to 4, wherein: The performing optical character recognition on the video to obtain a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set includes: Performing optical character recognition on the video to obtain a second phrase set contained in the video and second position information corresponding to each second phrase in the second phrase set; filtering the second phrase set based on a matching degree between each second position information and the subtitle area; Determine the filtered candidate phrases; Determine the semantic similarity between each two adjacent candidate phrases; In the case where the similarity between any two candidate phrases is greater than a fourth threshold, any one of the two candidate phrases is removed to obtain the first phrase set and the first position information corresponding to each first phrase in the first phrase set, wherein, in the case where the similarity between any two candidate phrases is greater than the fourth threshold, it is determined that the two candidate phrases have the same semantics, and any one of the two candidate phrases is removed to obtain a phrase set in which the similarity between any adjacent candidate phrases is less than the fourth threshold as the first phrase set, and the second position information corresponding to the candidate phrase is used as the first position information corresponding to the corresponding first phrase in the first phrase set.

6. The method according to any one of claims 1 to 4, wherein: Before inputting the first subtitle, the second subtitle and the preset prompt words into the large model, the method further includes: The preset prompt word is determined according to the attribute information of the video.

7. The method of claim 6, wherein: The attribute information of the video includes at least one of the following: the type of the video, the field to which the video content belongs, the length of the video, and the generation time of the video.

8. The method according to any one of claims 1 to 4, wherein: After obtaining the first phrase set contained in the subtitle area of ​​the video and the first position information corresponding to each first phrase in the first phrase set, the method further includes: When the field to which the video content belongs is the medical field, obtaining a reference word set in the medical field; matching each of the first word groups with each reference word in the reference word set; When any first word group is not included in the reference word set and the matching degree between any first word group and any reference word is greater than a fifth threshold, the any first word group is updated based on the any reference word.

9. The method according to any one of claims 1 to 4, wherein: After obtaining the first phrase set contained in the subtitle area of ​​the video and the first position information corresponding to each first phrase in the first phrase set, the method further includes: When any of the first word groups contains a preset symbol and the field to which the video content belongs is the medical field, obtaining a reference word pair set in the medical field, wherein the reference word pair set includes reference words and associated medical symbols; Matching the preset symbol with each medical symbol in the reference word pair set; In the case that any medical symbol matches the preset symbol, the preset symbol in any first phrase is replaced by a reference word associated with the medical symbol to obtain an updated first phrase.

10. A video subtitle recognition device, comprising: A first acquisition module, configured to perform optical character recognition on a video to acquire a first phrase set contained in a subtitle area of ​​the video and first position information corresponding to each first phrase in the first phrase set, wherein the first position information includes a positioning point coordinate; A determination module, used to determine the number of characters contained in each of the first phrases and the area corresponding to the first position information; A second acquisition module, configured to filter the first phrase set according to at least one of the coordinates of the positioning point corresponding to the first phrase, the number of characters included, and the area of ​​the region, so as to acquire a first subtitle; A third acquisition module, used for identifying the audio data associated with the video to obtain a second subtitle; A fourth acquisition module, used for inputting the first subtitle, the second subtitle and a preset prompt word into a large model to obtain a target subtitle of the video; Wherein, the second acquisition module is specifically used for: Determine a first relative position between the anchor points corresponding to the first phrase according to the anchor point coordinates corresponding to the first phrase, wherein the anchor point coordinates corresponding to the first phrase include two diagonal vertex coordinates of the first phrase; When a ratio between a first number of first phrases corresponding to any first relative position and a second number is greater than a first threshold, determining a first arrangement direction of subtitle content in the video according to any first relative position, wherein the second number is the number of first phrases included in the first phrase set; In the case where the positioning point coordinates are the coordinates of the designated position of the area where the first phrase is located, determining the second relative position between the positioning points corresponding to the first phrase according to the positioning point coordinates corresponding to the first phrase, wherein the coordinates of the designated position include the coordinates of the upper left vertex where the phrase starts and the coordinates of the lower right vertex where the phrase ends in the area where each phrase is located; determining a second arrangement direction of the content in the first phrase according to the second relative position; filtering out first phrases in a second arrangement direction different from the first arrangement direction to obtain a filtered first phrase set; The first subtitle is generated based on the filtered first phrase set.

11. The device of claim 10, wherein: The second acquisition module is specifically used for: Determining a font area corresponding to the first phrase according to the number of characters included in the first phrase and the area corresponding to the first position information; Determining a reference font area of ​​the subtitle according to the font area corresponding to each of the first phrases; When the absolute value of the difference between the font area corresponding to any first phrase and the reference font area is greater than a second threshold, the any first phrase is filtered out.

12. The device of claim 11, wherein: The second acquisition module is specifically used for: The mean value or median value of the font areas corresponding to the first phrases in the first phrase set is determined as the reference font area.

13. The device of claim 11, wherein: The second acquisition module is specifically used for: When the ratio of the third number of first phrases whose corresponding font areas are the same to the second number is greater than a third threshold, the font areas corresponding to the third number of first phrases are determined as the reference font area, wherein the second number is the number of first phrases included in the first phrase set.

14. The device according to any one of claims 10 to 13, wherein: in, The first acquisition module is specifically used to: Performing optical character recognition on the video to obtain a second phrase set contained in the video and second position information corresponding to each second phrase in the second phrase set; filtering the second phrase set based on a matching degree between each second position information and the subtitle area; Determine the filtered candidate phrases; Determine the semantic similarity between each two adjacent candidate phrases; In the case where the similarity between any two candidate phrases is greater than a fourth threshold, any one of the two candidate phrases is removed to obtain the first phrase set and the first position information corresponding to each first phrase in the first phrase set, wherein, in the case where the similarity between any two candidate phrases is greater than the fourth threshold, it is determined that the two candidate phrases have the same semantics, and any one of the two candidate phrases is removed to obtain a phrase set in which the similarity between any adjacent candidate phrases is less than the fourth threshold as the first phrase set, and the second position information corresponding to the candidate phrase is used as the first position information corresponding to the corresponding first phrase in the first phrase set.

15. The device according to any one of claims 10 to 13, wherein: The fourth acquisition module is further used for: The preset prompt word is determined according to the attribute information of the video.

16. The device of claim 15, wherein: The attribute information of the video includes at least one of the following: the type of the video, the field to which the video content belongs, the length of the video, and the generation time of the video.

17. The device according to any one of claims 10 to 13, wherein: The first acquisition module is further used for: When the field to which the video content belongs is the medical field, obtaining a reference word set in the medical field; matching each of the first word groups with each reference word in the reference word set; When any first word group is not included in the reference word set and the matching degree between any first word group and any reference word is greater than a fifth threshold, the any first word group is updated based on the any reference word.

18. The device according to any one of claims 10 to 13, wherein: The first acquisition module is further used for: When any of the first word groups contains a preset symbol and the field to which the video content belongs is the medical field, obtaining a reference word pair set in the medical field, wherein the reference word pair set includes reference words and associated medical symbols; Matching the preset symbol with each medical symbol in the reference word pair set; In the case that any medical symbol matches the preset symbol, the preset symbol in any first phrase is replaced by a reference word associated with the medical symbol to obtain an updated first phrase.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the video subtitle recognition method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: in, The computer instructions are used to enable the computer to execute the video subtitle recognition method according to any one of claims 1 to 9.

21. A computer program product, characterized in that The invention comprises a computer program, which, when executed by a processor, implements the steps of the video subtitle recognition method according to any one of claims 1 to 9.

Images