Subtitle recognition method and device and related product

Abstract

The invention discloses a subtitle recognition method and device and a related product. The invention relates to data calculation in the technical field of cloud. In the method, the boundary of a character area in a to-be-processed image is identified; and obtaining skew data of the character area based on the boundary of the character area. And if the skew data of the character region meets the caption skew constraint condition, determining the character region as a caption region in the to-be-processed image. The caption skew constraint condition is set according to the skew characteristic of the caption area. Therefore, the skew data of the character area is obtained through the boundary of the character area in the to-be-processed image, and the subtitle area in the to-be-processed image can be finally determined based on whether the skew degree of the character area accords with the skew characteristic of the subtitle area. Therefore, the subtitle area added to the to-be-processed image in the later period and the character area contained in the to-be-processed image can be effectively distinguished, and the subtitle recognition accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a subtitle recognition method, device and related products. Background Art

[0002] With the rapid development of computer technology and communication technology, multimedia data such as images, sounds and videos have become important carriers in the fields of information storage and information transmission. For example, video data can bring users intuitive visual and auditory experience, allowing users to quickly learn external information. In order to help users quickly understand the information in the video, subtitles are usually added to the video later. For example, in movies or TV series, subtitles are added to the dialogues of characters in the video later.

[0003] However, when users are watching videos, they may not be able to understand the subtitles in the video because they are in different regions and use different languages, or the brightness of the subtitles in the video may affect the viewing effect. Therefore, it is necessary to identify the subtitles in the video, and then perform subtitle processing such as subtitle brightness adjustment and subtitle translation. In the related art, image text recognition and other methods are usually used to identify the subtitles of video frames in the video. However, in addition to the subtitles added later, the text in the video also includes the text that exists in the video content itself. The above-mentioned image text recognition method may mistakenly identify non-subtitle text in the video as subtitles, which can easily lead to a reduction in the accuracy of subtitle recognition. Summary of the invention

[0004] The embodiments of the present application provide a subtitle recognition method, device and related products, which are intended to effectively distinguish subtitles added later from images waiting to be processed in video frames, thereby improving the accuracy of subtitle recognition.

[0005] The first aspect of the present application provides a subtitle recognition method, comprising:

[0006] Identify the boundaries of text areas in the image to be processed;

[0007] Based on the boundary of the text area, obtaining the skew data of the text area; the skew data of the text area is used to characterize the skew degree of the text area;

[0008] If the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

[0009] A second aspect of the present application provides a subtitle recognition device, comprising:

[0010] A boundary recognition module, used to recognize the boundary of the text area in the image to be processed;

[0011] A skew data acquisition module, used to obtain skew data of the text area based on the boundary of the text area; the skew data of the text area is used to characterize the skew degree of the text area;

[0012] The subtitle region determining module is used to determine the text region as the subtitle region in the image to be processed if the skew data of the text region meets the subtitle skew constraint condition.

[0013] A third aspect of the present application provides a subtitle recognition device, the device comprising a processor and a memory:

[0014] The memory is used to store a computer program and transmit the computer program to the processor;

[0015] The processor is used to execute the steps of the subtitle recognition method provided in the first aspect according to the instructions in the computer program.

[0016] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program, and when the computer program is executed by a subtitle recognition device, the steps of the subtitle recognition method provided in the first aspect are implemented.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a subtitle recognition device, implements the steps of the subtitle recognition method provided in the first aspect.

[0018] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0019] In the technical solution of the present application, the boundary of the text area in the image to be processed is first identified. Based on the boundary of the text area, the skew data of the text area is obtained. The boundary of the identified text area can be used as analysis data for analyzing the skew degree of the text area, and the skew data of the text area is used to characterize the skew degree of the text area. Finally, if the skew data of the text area meets the subtitle skew constraint condition, the text area can be determined as the subtitle area in the image to be processed. Since the subtitles are added to the video frames in the video to wait for the image to be processed later, in order to improve the user's visual experience, the subtitle area is usually placed horizontally or vertically. The subtitle skew constraint condition is set according to the skew characteristics of the subtitle area. Therefore, the skew data of the text area is obtained through the boundary of the text area in the image to be processed, and the subtitle area in the image to be processed can be finally determined based on whether the skew degree of the text area meets the skew characteristics of the subtitle area. Therefore, the present application can effectively distinguish the subtitle area added to the image to be processed later and the text area contained in the image to be processed, thereby improving the accuracy of subtitle recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A scene architecture diagram of a subtitle recognition method provided in an embodiment of the present application;

[0021] Figure 2 A flowchart of a subtitle recognition method provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a method for identifying the boundary of a text area in an image to be processed provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of a specific method for identifying the boundary of a text area in an image to be processed provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the boundary intersection of a text area in an image to be processed provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the midpoint of the boundary of a text area in an image to be processed provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the angles between a pair of relatively positioned boundaries of a text area and a reference straight line provided in an embodiment of the present application;

[0027] Figure 8 A flowchart of a specific subtitle recognition method provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of translating a subtitle area provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of a subtitle recognition device provided in an embodiment of the present application;

[0030] Fig.11 A schematic diagram of the structure of the server in the embodiment of the present application;

[0031] Fig.12 A schematic diagram of the structure of a terminal device in an embodiment of the present application. DETAILED DESCRIPTION

[0032] At present, if you need to identify subtitles in a video, you usually use an image text recognition method to recognize the text in the video frame. However, in addition to the subtitles added later, the video also has text related to the video content. For example, in videos such as movies or TV series, when the scene in the video is on the street, various store signs will be displayed in the video. If the above-mentioned image text recognition method is used, the text in the store signs in the video frame will also be recognized, that is, the non-subtitle text in the video will be mistakenly recognized as subtitles, which will easily reduce the accuracy of subtitle recognition.

[0033] In view of the above problems, a subtitle recognition method, device and related products are provided in the embodiments of the present application, the purpose of which is to effectively distinguish the subtitles added later in the image to be processed from the text in the image to be processed, and to improve the accuracy of subtitle recognition. In the technical solution provided by the present application, the boundary of the text area in the image to be processed is first identified. Based on the boundary of the text area, the skew data of the text area is obtained. The skew data of the text area is used to characterize the skew degree of the text area. Finally, if the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed. In order to improve the user's visual experience, the subtitle area added to the video frame in the video to be processed is usually placed horizontally or vertically. The subtitle skew constraint condition is set according to the skew characteristics of the subtitle area. Therefore, the skew degree of the text area is obtained by the boundary of the text area in the image to be processed, and the subtitle area in the image to be processed can be finally determined based on whether the skew degree of the text area meets the skew characteristics of the subtitle area. Therefore, the present application can effectively distinguish the subtitle area added to the image to be processed later and the text area contained in the image to be processed, thereby improving the accuracy of subtitle recognition.

[0034] The execution subject of the subtitle recognition method provided in the embodiment of the present application may be a terminal device. As an example, the terminal device may specifically include but is not limited to a mobile phone, a desktop computer, a tablet computer, a laptop computer, a PDA, an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, etc. The execution subject of the subtitle recognition method provided in the embodiment of the present application may also be a server. In addition, the subtitle recognition method provided in the embodiment of the present application may also be executed collaboratively by a terminal device and a server, for example, the server first identifies the boundary of the text area in the image to be processed, and the server obtains the skew data of the text area based on the boundary of the text area, and then the server sends the skew data of the text area to the terminal device; if the skew data of the text area meets the subtitle skew constraint condition, the terminal device determines the text area as the subtitle area in the image to be processed. Among them, the terminal and the server may be directly or indirectly connected by wired or wireless communication, and the present application does not limit this. Therefore, in the embodiment of the present application, the implementation subject of the technical solution of the present application is not limited.

[0035] Figure 1 The following is an exemplary scenario architecture diagram of a subtitle recognition method, which includes a server and various terminal devices. Figure 1 The server shown can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. In addition, the server can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0036] Next, taking the terminal device as the execution subject, the subtitle recognition method provided in the embodiment of the present application is specifically introduced.

[0037] See also Figure 2 , which is a flow chart of a subtitle recognition method provided by an embodiment of the present application. Figure 2 The subtitle recognition method shown includes the following steps:

[0038] S201: Identify the boundary of the text area in the image to be processed.

[0039] In an embodiment of the present application, the image to be processed refers to an image containing a text area. For example, the image to be processed can be a video frame in a video, or it can be an image taken by a user, and the present application does not limit this. The text area refers to the area containing text in the image to be processed, which is usually composed of a border and the text contained in the border. It should be noted that the present application does not limit the number of characters contained in the text area. The boundary of the text area refers to the border containing text in the image to be processed, such as the border of a quadrilateral containing text in the image to be processed, etc.

[0040] In a possible implementation of the present application, the step of identifying the boundary of the text area in the image to be processed can be specifically divided into the following steps: first, perform image text segmentation on the image to be processed to obtain the text segmentation result corresponding to the image to be processed; then perform binarization processing on the text segmentation result to obtain the binarization processing result of the image to be processed to mark the text area; finally, determine the boundary of the text area in the image to be processed based on the binarization processing result of the image to be processed. For details, please refer to Figure 3 , which is a schematic diagram of a method for identifying the boundary of a text area in an image to be processed provided in an embodiment of the present application.

[0041] For ease of understanding, the following describes a specific method for identifying the boundaries of text areas in an image to be processed. Figure 4 , which is a schematic diagram of a specific method for identifying the boundary of a text area in an image to be processed provided by an embodiment of the present application. Figure 4 As shown, the image features are first extracted through the feature pyramid to obtain the image features, and the image features of the image to be processed are adaptively predicted by threshold to obtain the threshold feature map, and the threshold feature map and the image features are the text segmentation results; then the threshold feature map and the image features are differentiably binarized to obtain the binarization processing result of the image to be processed, and finally, the binarization processing result of the image to be processed is marked to determine the boundary of the text area in the image to be processed.

[0042] It should be noted that the implementation method of identifying the boundary of the text area in the image to be processed given in the above introduction is only an exemplary description and does not represent all implementation methods of the embodiments of the present application.

[0043] It can be seen that, through the above-mentioned text area boundary recognition method, the boundary of the text area containing text in the image to be processed can be accurately recognized, which is beneficial to the execution of subsequent subtitle recognition method steps.

[0044] S202: Obtain skew data of the text area based on the boundary of the text area.

[0045] The skew data of the text area is used to represent the skew degree of the text area.

[0046] In actual applications, the subtitle area added to the video frame in the video to be processed is usually placed horizontally or vertically. For example, in videos such as movies or TV series, usually after the shooting is completed, the subtitles are processed for the dialogue of the characters in the video, the narration of the video content, etc., that is, a horizontally placed subtitle area is added to the video, and the subtitle area includes text that matches the dialogue of the characters or the narration of the video content, so as to bring a good visual experience to users watching the video. In videos such as music videos of songs, the lyrics of the song are generally added to the left or right side of the video. The lyrics are in the vertically placed subtitle area to avoid blocking the picture in the music video as much as possible, giving users a better visual experience.

[0047] Therefore, the degree of skewness of the text area is closely related to whether it is a subtitle area added later. Based on the boundary of the text area identified in the above steps, skewness data used to characterize the degree of skewness of the text area can be obtained.

[0048] In the embodiment of the present application, there are multiple possible implementation methods for the step of obtaining the skew data of the text area based on the boundary of the text area in S202 mentioned above, which are introduced below. It should be noted that the implementation methods given in the following introduction are only for exemplary description and do not represent all implementation methods of the embodiment of the present application.

[0049] Then the first optional implementation method of S202 is: based on the positions of multiple reference points on the boundary of the text area, obtain the skew data of the text area. Among them, the reference point is a pre-set pixel point for determining the skew degree of the text area. For example, the reference point can be a pixel point on the boundary of the text area, such as a boundary intersection or a boundary midpoint, etc.; the reference point can also be a pixel point in the text area or a pixel point outside the text area, such as a pixel point on the center line of the text area or a pixel point on the extension line of the center line of the text area, etc. This application does not limit the number of reference points and the position of the reference points. In this way, based on the reference points on the boundary of the text area, skew data that can accurately represent the skew degree of the text area is obtained, and then the subtitle area in the image to be processed is accurately identified.

[0050] In a possible implementation of the present application, the multiple reference points on the boundary of the text area may include multiple boundary intersections on the boundary of the text area. A boundary intersection refers to the intersection of two boundaries in the text area. Taking the text area as a quadrilateral as an example, in the embodiment of the present application, the multiple boundary intersections may include the boundary intersection of the left boundary of the text area and the upper boundary of the text area, that is, the upper left boundary intersection; the boundary intersection of the left boundary of the text area and the lower boundary of the text area, that is, the lower left boundary intersection; the boundary intersection of the right boundary of the text area and the upper boundary of the text area, that is, the upper right boundary intersection; and the boundary intersection of the right boundary of the text area and the lower boundary of the text area, that is, the lower right boundary intersection. Accordingly, the step of obtaining the skew data of the text area based on the positions of the multiple reference points on the boundary of the text area may specifically be: based on the coordinate value of the upper left boundary intersection, the coordinate value of the lower left boundary intersection, the coordinate value of the upper right boundary intersection, and the coordinate value of the lower right boundary intersection, obtain the skew data of at least one pair of two boundaries in relative positions of the text area. Among them, the image to be processed is placed in the coordinate system, and the pixel points corresponding to the lower left corner, upper left corner or the center point of the image to be processed can be set as the coordinate origin. Therefore, the coordinate value of the boundary intersection is the horizontal distance value and the vertical distance value of the pixel point corresponding to the boundary intersection relative to the coordinate origin. It can be seen that through multiple boundary intersections on the boundary of the text area, the skew data of multiple boundaries of the text area can be obtained. The skew data of multiple boundaries can accurately represent the skew degree of the text area, and then distinguish the subtitle area in the image to be processed based on the skew degree of the text area, thereby improving the accuracy of subtitle area recognition.

[0051] It is understandable that the subtitle area in the image to be processed may include monolingual subtitles or bilingual subtitles. The number of characters in different languages ​​corresponding to the same sentence is different. The subtitle area added to the image to be processed later may be a monolingual subtitle in a rectangular shape or a bilingual subtitle in a trapezoidal shape. Therefore, when determining the degree of skewness of the text area, the skew data of each boundary of the text area can be determined, or the skew data of a pair of two boundaries in relative positions of the text area can be determined. For example, for a horizontally placed text area, the skew data of the upper boundary and the skew data of the lower boundary of the text area can be determined, or the skew data of each boundary (i.e., the upper boundary, the lower boundary, the left boundary, and the right boundary) of the text area can be determined. For a vertically placed text area, the skew data of the left boundary and the skew data of the right boundary of the text area can be determined, or the skew data of each boundary (i.e., the upper boundary, the lower boundary, the left boundary, and the right boundary) of the text area can be determined. This application does not limit this.

[0052] As an example, the coordinates of the upper left corner boundary intersection in the text area are (x1, y1), the coordinates of the lower left corner boundary intersection are (x2, y2), the coordinates of the upper right corner boundary intersection are (x3, y3), and the coordinates of the lower right corner boundary intersection are (x4, y4). Then, we can first calculate the difference between the horizontal coordinate value x1 of the upper left corner boundary intersection and the horizontal coordinate value x2 of the lower left corner boundary intersection (x1-x2), and then calculate the difference between the vertical coordinate value y1 of the upper left corner boundary intersection and the vertical coordinate value y2 of the lower left corner boundary intersection (y1-y2). Then, using the ratio of the difference between the vertical coordinates and the difference between the horizontal coordinates (y1-y2) / (x1-x2), we can calculate the slope of the left boundary of the text area as the skew data. Similarly, based on the coordinate values ​​of the upper left corner boundary intersection, the lower left corner boundary intersection, the upper right corner boundary intersection, and the lower right corner boundary intersection, the slopes of multiple boundaries of the text area can be obtained as the skew data of the text area.

[0053] For details, please refer to Figure 5 , which is a schematic diagram of the boundary intersection of a text area in an image to be processed provided by an embodiment of the present application. Figure 5As shown, the image to be processed is a video frame in a video, and the video frame is in a scene in which student A submits a sports registration form to student B in a school gymnasium. The video frame includes text area 1 "This is my registration form", the coordinates of the upper left corner boundary intersection of text area 1 are (x1, y1), the coordinates of the lower left corner boundary intersection are (x2, y2), the coordinates of the upper right corner boundary intersection are (x3, y3), and the coordinates of the lower right corner boundary intersection are (x4, y4). The video frame also includes text area 2 "Registration Office", the coordinates of the upper left corner boundary intersection of text area 2 are (x5, y5), the coordinates of the lower left corner boundary intersection are (x6, y6), the coordinates of the upper right corner boundary intersection are (x7, y7), and the coordinates of the lower right corner boundary intersection are (x8, y8).

[0054] In another possible implementation of the present application, the multiple reference points on the boundary of the text area also include multiple boundary midpoints on the boundary of the text area. The boundary midpoint is the midpoint on a boundary of the text area. For example, if the coordinates of the upper left corner boundary intersection of the text area are (x1, y1), and the coordinates of the lower left corner boundary intersection are (x2, y2), then the coordinates of the boundary midpoint of the left boundary of the text area are ((x1+x2) / 2, (y1+y2) / 2). Correspondingly, S202 can specifically be: based on the coordinate values ​​of the boundary midpoints on at least one pair of two boundaries of the text area that are in relative positions, obtain the skew data of at least one pair of two boundaries of the text area that are in relative positions. As described above, when the text area is a quadrilateral, the text area includes two pairs of two boundaries that are in relative positions. If the skew data of the text area needs to be obtained, at least the skew data of at least one pair of two boundaries of the text area that are in relative positions needs to be obtained.

[0055] As an example, see Figure 6 , which is a schematic diagram of the boundary midpoint of a text area in an image to be processed provided in an embodiment of the present application. Figure 6 The scenario shown is similar to the one above. Figure 5The scene shown is the same, and the video frame includes text area 1 "This is my registration form" and text area 2 "Registration Office". The coordinates of the boundary midpoint of the left boundary of text area 1 are (x1, y1), and the coordinates of the boundary midpoint of the right boundary are (x2, y2). The boundary midpoint of the left boundary and the boundary midpoint of the right boundary can be connected to obtain a midpoint line, and the slope of the midpoint line is calculated (y2-y1) / (x2-x1), and the slope of the midpoint line is used as the skew data of the upper boundary and the lower boundary of text area 1. Similarly, the boundary midpoint coordinates of the left boundary of text area 2 are (x3, y3), and the boundary midpoint coordinates of the right boundary are (x4, y4). Then the boundary midpoint of the left boundary and the boundary midpoint of the right boundary can be connected to obtain a midpoint line, and the slope of the midpoint line is calculated (y4-y3) / (x4-x3), and the slope of the midpoint line is used as the skew data of the upper boundary and the lower boundary of text area 2.

[0056] In addition, a second optional implementation of S202 is: determine the angle between at least one pair of two boundaries of the text area in relative positions and the reference straight line. That is to say, the angle is used as the skew data of the text area. Among them, the reference straight line is a straight line used to calculate the skewness of the boundary of the text area. For example, the reference straight line can be a horizontal straight line, and the reference straight line can also be a vertical straight line perpendicular to the horizontal straight line; the reference straight line can also be a horizontal straight line and a vertical straight line, and the present application does not limit the number of reference straight lines. When the reference straight line is a horizontal straight line, the skew data of the upper boundary and the skew data of the lower boundary of the text area can be obtained; when the reference straight line is a vertical straight line perpendicular to the horizontal straight line, the skew data of the left boundary and the skew data of the right boundary of the text area can be obtained.

[0057] As an example, see Figure 7 , which is a schematic diagram of the angles between a pair of relatively positioned boundaries of a text area and a reference straight line provided in an embodiment of the present application. Figure 7 The scene shown is similar to Figure 5 and Figure 6 The scene shown is the same, and the image to be processed is a video frame in the video, which is in the scene where student A submits a sports registration form to student B in the school gymnasium. The video frame includes text area 1 "This is my registration form" and text area 2 "Registration Office". Reference straight line 1 is a horizontal straight line. According to the reference straight line 1, the skew data of the upper boundary and the skew data of the lower boundary of the text area 1 are both angle 0; the skew data of the upper boundary and the skew data of the lower boundary of the text area 2 are both angle A.

[0058] It should be noted that, for the two optional implementation methods of the step of obtaining the skew data of the text area based on the boundary of the text area in S202 above, the terminal device can select one or combine multiple methods to implement, and this application does not impose any restrictions on this.

[0059] S203: If the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

[0060] Among them, the subtitle skew constraint condition is a constraint condition pre-set based on the skew characteristics of the subtitle area, and is used to distinguish the subtitle area in the image to be processed and the text area contained in the image to be processed itself. For example, if the skew data of the text area is the slope of the boundary of the text area, the subtitle skew constraint condition can be the slope threshold of the boundary; if the skew data of the text area is the angle between the boundary and the reference line, the subtitle skew condition can be the degree threshold; if the skew data of the text area is the coordinate offset value, the subtitle skew constraint condition can be the offset value threshold. This application does not limit the subtitle constraint condition, and it can match the skew data of the text area.

[0061] Based on the above example, when the skew data of the text area is the boundary slope of at least one pair of two boundaries of the text area in relative positions, S203 can be specifically that if the boundary slope of at least one pair of two boundaries of the text area in relative positions meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed. As an example, when the two boundaries in relative positions are the upper boundary and the lower boundary of the text area, the subtitle skew constraint condition can be that the absolute value of the boundary slope of the upper boundary and the absolute value of the boundary slope of the lower boundary are both less than or equal to the slope threshold; when the two boundaries in relative positions are the left boundary and the right boundary of the text area, the subtitle skew constraint condition can be that the absolute value of the boundary slope of the left boundary and the absolute value of the boundary slope of the right boundary are both greater than or equal to the slope threshold, or are both positive infinity (that is, the left boundary and the right boundary are both vertical straight lines perpendicular to the horizontal straight line).

[0062] In a possible implementation of the present application, based on the above example, when the skew data of the text area is the skew data of at least one pair of two boundaries of the text area in relative positions, S203 can be specifically as follows: if the skew data of at least one pair of two boundaries of the text area in relative positions meets the subtitle skew constraint condition, the text area is determined to be the subtitle area in the image to be processed. Therefore, through the multiple boundary midpoints on the boundary of the text area, the skew data of the multiple boundaries of the text area can also be obtained to accurately represent the skew degree of the text area, thereby distinguishing the subtitle area in the image to be processed based on the skew degree of the text area, thereby improving the accuracy of subtitle recognition. As an example, Figure 6The slope of the midpoint line shown is used as the skew data of the upper and lower boundaries of the text area 2, and the subtitle skew constraint condition can be that the absolute value of the slope of the midpoint line is less than or equal to the slope threshold.

[0063] In a possible implementation of the present application, based on the above example, when the skew data of the text area is the angle between at least one pair of two boundaries of the text area in relative positions and the reference straight line, S203 can be specifically: if the angle between at least one pair of two boundaries of the text area in relative positions and the reference straight line is less than or equal to the first degree threshold, the text area is determined as the subtitle area of ​​the image to be processed. It can be seen that the angle between the reference straight line and the boundary of the text area is used as the skew data, and the subtitle area is effectively distinguished based on the first degree threshold, thereby improving the accuracy of subtitle recognition.

[0064] As an example, when the two boundaries in relative position are the upper boundary and the lower boundary of the text area, the subtitle skew constraint condition may be that the angle between the upper boundary and the horizontal straight line and the angle between the lower boundary and the horizontal straight line are both less than or equal to a first degree threshold; when the two boundaries in relative position are the left boundary and the right boundary of the text area, the subtitle skew constraint condition may be that the angle between the left boundary and the vertical straight line (i.e., the straight line perpendicular to the horizontal straight line) and the angle between the right boundary and the vertical straight line are both less than or equal to the first degree threshold.

[0065] Considering that in actual applications, in order to provide users with a clear and definite visual experience, the subtitles added to the image to be processed later will be adjusted based on the size of the image to be processed, so that the subtitles will not occupy too much of the screen, affecting the content display of the image to be processed, nor will they occupy too little of the screen, causing the user to be unable to see the text in the subtitles, thereby affecting the user's viewing experience. Therefore, when determining the subtitle area, not only the degree of skewness of the text area can be considered, but also the ratio of the area height of the text area to the height of the image to be processed, and / or the ratio of the area of ​​the text area to the area of ​​the image to be processed can be comprehensively considered.

[0066] Therefore, in a possible implementation of the present application, the above S203 may specifically be: if the ratio of the area height of the text area to the image height of the image to be processed is greater than or equal to the preset height ratio, and the skew data of the text area satisfies the subtitle skew constraint condition, then the text area can be determined as the subtitle area in the image to be processed. Among them, the preset height ratio may be a ratio pre-set by a technician, or may be a ratio determined by the terminal device according to the height of the image to be processed. The present application does not limit the specific value of the preset height ratio. It can be seen that, on the basis of judging the degree of skewness of the text area, the height ratio of the text area in the image to be processed is judged, and the skew characteristics and height ratio characteristics of the subtitle area are comprehensively considered, thereby further improving the accuracy of subtitle recognition.

[0067] Similarly, when the text area is placed vertically, the ratio of the area width of the text area to the image width of the image to be processed can also be calculated. When the ratio of the widths of the two is greater than or equal to the preset width ratio, and the skew data of the text area meets the subtitle skew constraint condition, the text area can be determined as the subtitle area in the image to be processed. The present application does not limit the specific value of the preset width ratio.

[0068] As an example, the preset height ratio may be 0.03, the image height of the image to be processed is h, the skew data of the text area satisfies the subtitle skew constraint, and the text area is a regular rectangular area, the coordinates of the upper left corner boundary intersection of the text area are (x1, y1), and the coordinates of the lower left corner boundary intersection are (x1, y2), then the area height of the text area is |y2-y1|, so when the ratio of the area height of the text area to the image height of the image to be processed satisfies Formula 1, the text area can be determined as the subtitle area in the image to be processed. Formula 1 is as follows:

[0069]

[0070] In another possible implementation of the present application, the above S203 may also be specifically as follows: if the ratio of the area of ​​the text area to the area of ​​the image to be processed is between the first area ratio and the second area ratio, and the skew data of the text area meets the subtitle skew constraint, then the text area is determined as the subtitle area in the image to be processed; the second area ratio is greater than the first area ratio. Among them, the present application does not limit the first area ratio and the second area ratio, which can be pre-set by technicians based on experience, or pre-set by the terminal device based on the size of the image to be processed. It can be seen that on the basis of judging the degree of skewness of the text area, judging the area ratio of the text area in the image to be processed, and comprehensively considering the skew characteristics and area ratio characteristics of the subtitle area, can further improve the accuracy of subtitle recognition.

[0071] As an example, the first area ratio may be 0.0009, the second area ratio may be 0.14, the image height of the image to be processed is h, the image width of the image to be processed is w, and the area of ​​the image to be processed is (h×w). The skew data of the text area satisfies the subtitle skew constraint, and the text area is a regular rectangular area, the coordinates of the upper left corner boundary intersection of the text area are (x1, y1), the coordinates of the lower left corner boundary intersection are (x1, y2), the coordinates of the upper left corner boundary intersection are (x3, y1), and the coordinates of the lower left corner boundary intersection are (x3, y2). Therefore, when the ratio of the area of ​​the text area to the area of ​​the image to be processed satisfies Formula 2, the text area can be determined as the subtitle area in the image to be processed. Formula 2 is as follows:

[0072]

[0073] It is understandable that when watching a video, the user may encounter a situation where the text in the subtitle area is not clearly visible because the brightness of the subtitle area in the video frame is similar to that of the video frame. Therefore, the subtitle recognition method provided in the embodiment of the present application can be used to identify the subtitle area in the video frame, and then perform subtitle extraction and subtitle enhancement processing on the subtitle area.

[0074] Therefore, if the subtitle area needs to be enhanced, the subtitle recognition method may further include: first determining the coordinate values ​​of multiple rectangular boundary intersections of the minimum circumscribed rectangle of the subtitle area; then updating the coordinate values ​​of multiple boundary intersections on the boundary of the subtitle area based on the coordinate values ​​of the multiple rectangular boundary intersections to obtain an updated subtitle area, that is, updating the coordinate value of the upper left corner boundary intersection of the subtitle area to the coordinate value of the upper left corner rectangular boundary intersection of the minimum circumscribed rectangle, and the process of updating the coordinate values ​​of other boundary intersections is similar, which will not be repeated here. Subsequently, the updated upper left corner boundary intersection can be connected with the updated upper right corner boundary intersection and the updated lower left corner boundary intersection respectively, and then the updated lower right corner boundary intersection can be connected with the updated upper right corner boundary intersection and the updated lower left corner boundary intersection respectively, so as to obtain the updated subtitle area. Extract the updated subtitle area. Enhance the updated subtitle area to obtain the enhanced processed image to be processed. Thus, the coordinate values ​​of the boundary intersections of the original subtitle area are updated and reset to obtain the updated subtitle area which is a regular rectangle, which can simplify the difficulty of extracting the subtitle area and is conducive to the step of performing the enhancement processing of the subtitle area.

[0075] It is understandable that in the process of calculating the slopes of multiple boundaries based on the coordinate values ​​of multiple boundary intersections, it may be possible to encounter a situation where the slope cannot be calculated (i.e., the slope is positive infinity or the slope is negative infinity), and the calculation process is too cumbersome, and when the image to be processed includes a large number of text areas, it is easy to occupy computing resources. Therefore, in order to solve this problem, an embodiment of the present application provides a specific subtitle recognition method, when the skew data of the text area is obtained based on the coordinate values ​​of multiple boundary intersections of the text area, the horizontal coordinate offset value of two boundary intersections on the same boundary, or the vertical coordinate offset value of two boundary intersections on the same boundary can be used as the skew data of the text area.

[0076] For ease of understanding, the following embodiments take the scenario in which the image to be processed is a video frame to be processed and subtitles in the video frame to be processed need to be translated as an example to specifically illustrate the specific subtitle recognition method provided in the embodiments of the present application.

[0077] For details, please refer to Figure 8 A flowchart of a specific subtitle recognition method is shown, and the specific subtitle recognition method includes the following steps:

[0078] S801: Identify the boundary of a text area in a video frame to be processed.

[0079] The video frame to be processed in the video may be obtained first, and the boundary of the text area in the video frame may be identified.

[0080] S802: Acquire a first horizontal coordinate offset value between an upper left corner boundary intersection point and a lower left corner boundary intersection point, and a second horizontal coordinate offset value between an upper right corner boundary intersection point and a lower right corner boundary intersection point.

[0081] The first horizontal axis offset value is used to represent the degree of deflection of the left border of the text area, and the second horizontal axis offset value is used to represent the degree of deflection of the right border of the text area.

[0082] S803: Obtain a first ordinate offset value between the upper left corner boundary intersection point and the upper right corner boundary intersection point, and a second ordinate offset value between the lower left corner boundary intersection point and the lower right corner boundary intersection point.

[0083] The first ordinate offset value is used to represent the degree of inclination of the upper boundary of the text area, and the second ordinate offset value is used to represent the degree of inclination of the lower boundary of the text area.

[0084] S804: Use the first horizontal coordinate offset value as the skew data of the left boundary of the text area, use the first horizontal coordinate offset value as the skew data of the right boundary of the text area, use the first vertical coordinate offset value as the skew data of the upper boundary of the text area, and use the second vertical coordinate offset value as the skew data of the lower boundary of the text area.

[0085] Based on the above description, different coordinate offset values ​​respectively represent the offset degrees of different boundaries of the text area, so the first horizontal coordinate offset value, the second horizontal coordinate offset value, the first horizontal coordinate offset value and the second horizontal coordinate offset value can all be used as the skew data of the text area.

[0086] S805: If the absolute value of the first coordinate difference, the absolute value of the second coordinate difference, the absolute value of the third coordinate difference and the absolute value of the fourth coordinate difference are respectively less than their corresponding difference thresholds, the ratio of the area height of the text area to the image height of the video frame to be processed is greater than or equal to the preset height ratio, and the ratio of the area of ​​the text area to the area of ​​the video frame to be processed is between the first area ratio and the second area ratio, then the text area is determined as a subtitle area in the video frame to be processed.

[0087] Among them, the second area ratio is greater than the first area ratio. It should be emphasized that the execution order of determining the slope data of the text area, the area height of the text area, and the area of ​​the text area in S805 is not limited. It can be determined in a preset order whether the slope data of the text area meets the subtitle slope constraint, the comparison of the area height of the text area with the image height of the video frame to be processed, and the comparison of the area of ​​the text area with the area of ​​the video frame to be processed, or it can be performed simultaneously.

[0088] S806: Determine the coordinate values ​​of the intersection points of multiple rectangular boundaries of the minimum bounding rectangle of the subtitle area.

[0089] It is understandable that, in view of the viewing needs of users, if the subtitle area needs to be processed by subtitle enhancement, subtitle translation, etc., it is necessary to extract the text area that has been determined as the subtitle area. However, the subtitle area may be an irregular rectangular area, which may easily bring difficulties to the subtitle extraction, and thus affect the execution of subsequent steps such as special subtitle processing. Therefore, the coordinate values ​​of multiple boundary intersections of the subtitle area can be reset and updated to obtain a subtitle area that is a regular rectangular area. The minimum circumscribed rectangle of the subtitle area can not only avoid the missing text in the subtitle area, but also determine the minimum regular rectangular area including the subtitle area, reduce the area of ​​the extracted subtitle area, and improve the convenience of subtitle extraction.

[0090] It should be noted that the coordinate values ​​of the intersection points of multiple rectangular boundaries of other rectangles containing the subtitle area may also be determined, and this application does not impose any limitation on this.

[0091] S807: Based on the coordinate values ​​of the plurality of rectangular boundary intersection points, the coordinate values ​​of the plurality of boundary intersection points on the boundary of the subtitle area are updated to obtain an updated subtitle area.

[0092] Update the coordinate value of the upper left corner boundary intersection of the subtitle area to the coordinate value of the upper left corner boundary intersection of the minimum bounding rectangle, update the coordinate value of the upper right corner boundary intersection of the subtitle area to the coordinate value of the upper right corner boundary intersection of the minimum bounding rectangle, update the coordinate value of the lower left corner boundary intersection of the subtitle area to the coordinate value of the lower left corner boundary intersection of the minimum bounding rectangle, and update the coordinate value of the lower right corner boundary intersection of the subtitle area to the coordinate value of the lower right corner boundary intersection of the minimum bounding rectangle.

[0093] S808: Extracting the updated subtitle area.

[0094] Based on the multiple boundary intersection points after the coordinate values ​​of the subtitle area are updated, an updated subtitle area is extracted.

[0095] S809: Translate the updated subtitle area.

[0096] It is understandable that when watching a video, the user may encounter a situation where the language of the subtitle area in the video frame is different from the language the user is proficient in, resulting in the user being unable to understand the text in the subtitle area. Therefore, the subtitle recognition method provided in the embodiment of the present application can be used to update the coordinate values ​​of the boundary intersection points of the subtitle area after identifying the subtitle area in the video frame, and extract a more regular subtitle area, so as to smoothly perform steps such as subtitle extraction and subtitle translation processing.

[0097] S810: Displaying the translated subtitle area in the video frame to be processed.

[0098] For example, the translated subtitle area can be displayed below the original subtitle area to form bilingual subtitles; the translated subtitle area can also be covered above the original subtitle area, which is not limited in this application.

[0099] As an example, see Fig. 9 , which is a schematic diagram of a translation process for a subtitle area provided by an embodiment of the present application. Fig. 9 As shown in FIG, the scene in the video frame is inside the school gymnasium, where student A hands over a sports registration form to classmate B. Fig. 9 As shown, the text area "This is my registration form" is determined to be the subtitle area. The coordinates of the upper left corner boundary intersection of the subtitle area are (x1, y1), the coordinates of the lower left corner boundary intersection are (x2, y2), the coordinates of the upper right corner boundary intersection are (x3, y3), and the coordinates of the lower right corner boundary intersection are (x4, y4). The coordinates of the upper left corner boundary intersection of the minimum circumscribed rectangle are (a1, b1), the coordinates of the lower left corner boundary intersection are (a1, b2), the coordinates of the upper right corner boundary intersection are (a2, b1), and the coordinates of the lower right corner boundary intersection are (a2, b2). Fig. 9As shown, the coordinates of the upper left corner boundary intersection of the subtitle area coincide with the coordinates of the upper left corner boundary intersection of the minimum circumscribed rectangle, and the coordinates of the upper right corner boundary intersection of the subtitle area coincide with the coordinates of the upper right corner boundary intersection of the minimum circumscribed rectangle. Therefore, the coordinates of the lower left corner boundary intersection and the coordinates of the lower right corner boundary intersection of the subtitle area can be updated according to the coordinates (a1, b2) of the lower left corner boundary intersection of the minimum circumscribed rectangle and the coordinates (a2, b2) of the lower right corner boundary intersection to obtain an updated subtitle area; then extract the updated subtitle area, translate the text "This is my registration form" in the subtitle area, and obtain the text "This is my registration form" in the translated subtitle area; then display the translated subtitle area below the original subtitle area in the video frame to form a bilingual subtitle.

[0100] Based on the subtitle recognition method described in S801-S810, it is only necessary to calculate the coordinate offset values ​​between the multiple boundary intersections on the boundary of the text area as the skew data of the text area, and the calculation process is simple, which can save computing resources and help improve the speed of subtitle recognition. In addition, by resetting the coordinate values ​​of the multiple boundary intersections of the subtitle area through the coordinate values ​​of the multiple rectangular boundary intersections of the minimum circumscribed rectangle, a more regular subtitle area can be obtained, which is conducive to executing the steps of subtitle area extraction and subtitle area translation processing.

[0101] Based on the subtitle recognition method provided in the above embodiment, the present application also provides a subtitle recognition device. The subtitle recognition device provided in the embodiment of the present application is specifically introduced below.

[0102] See also Fig.10 , which is a schematic diagram of the structure of a subtitle recognition device provided by an embodiment of the present application. Fig.10 As shown, the subtitle recognition device 100 specifically includes:

[0103] A boundary recognition module 110 is used to recognize the boundary of the text area in the image to be processed;

[0104] The skew data acquisition module 120 is used to obtain skew data of the text area based on the boundary of the text area; the skew data of the text area is used to characterize the skew degree of the text area;

[0105] The subtitle region determining module 130 is configured to determine the text region as the subtitle region in the image to be processed if the skew data of the text region satisfies the subtitle skew constraint condition.

[0106] As an implementation manner, the above-mentioned skew data acquisition module 120 can be specifically used for:

[0107] Based on the positions of a plurality of reference points on the boundary of the text area, the skew data of the text area is obtained.

[0108] As an implementation manner, the multiple reference points on the boundary of the text area include multiple boundary intersections on the boundary of the text area; the multiple boundary intersections include an upper left boundary intersection, a lower left boundary intersection, an upper right boundary intersection, and a lower right boundary intersection;

[0109] Accordingly, the skew data acquisition module 120 may be specifically used for:

[0110] Based on the coordinate values ​​of the upper left corner boundary intersection, the lower left corner boundary intersection, the upper right corner boundary intersection and the lower right corner boundary intersection, the skew data of at least one pair of two boundaries in relative positions of the text area are obtained.

[0111] As an implementation manner, the above-mentioned skew data acquisition module 120 may specifically include:

[0112] A first offset value obtaining unit, used to obtain a first horizontal coordinate offset value between the upper left corner boundary intersection point and the lower left corner boundary intersection point, and a second horizontal coordinate offset value between the upper right corner boundary intersection point and the lower right corner boundary intersection point;

[0113] A second offset value obtaining unit, used to obtain a first ordinate offset value between an upper left corner boundary intersection point and an upper right corner boundary intersection point, and a second ordinate offset value between a lower left corner boundary intersection point and a lower right corner boundary intersection point;

[0114] a skew data determining unit, configured to use the first horizontal coordinate offset value as the skew data of the left boundary of the text area, the second horizontal coordinate offset value as the skew data of the right boundary of the text area, the first vertical coordinate offset value as the skew data of the upper boundary of the text area, and the second vertical coordinate offset value as the skew data of the lower boundary of the text area;

[0115] Accordingly, the subtitle area determination module 130 may be specifically used for:

[0116] If the absolute value of the first coordinate difference, the absolute value of the second coordinate difference, the absolute value of the third coordinate difference and the absolute value of the fourth coordinate difference are all smaller than their corresponding difference thresholds, the text area is determined as a subtitle area in the image to be processed.

[0117] As an implementation manner, the multiple reference points on the boundary of the text area include multiple boundary midpoints on the boundary of the text area; the skew data acquisition module 120 is specifically used to:

[0118] Obtaining skew data of at least one pair of two boundaries of the text area at relative positions based on coordinate values ​​of boundary midpoints of at least one pair of two boundaries of the text area at relative positions;

[0119] Accordingly, the subtitle area determination module 130 may be specifically used for:

[0120] If the skew data of at least one pair of two boundaries in relative positions of the text area meet the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

[0121] As an implementation manner, the above-mentioned skew data acquisition module 120 is specifically used for:

[0122] Determine the angle between at least one pair of two relatively positioned boundaries of the text area and the reference straight line;

[0123] Accordingly, the subtitle area determination module 130 may be specifically used for:

[0124] If the angles between at least one pair of two relatively positioned boundaries of the text area and the reference straight line are both less than or equal to a first degree threshold, the text area is determined as a subtitle area of ​​the image to be processed.

[0125] As an implementation manner, the subtitle area determination module 130 may be specifically used for:

[0126] If the ratio of the area height of the text area to the image height of the image to be processed is greater than or equal to the preset height ratio, and the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

[0127] As an implementation manner, the subtitle area determination module 130 may be specifically used for:

[0128] If the ratio of the area of ​​the text area to the area of ​​the image to be processed is between the first area ratio and the second area ratio, and the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed; the second area ratio is greater than the first area ratio.

[0129] As an implementation manner, the subtitle area determination module 130 may be specifically used for:

[0130] If the ratio of the region height of the text region to the image height of the image to be processed is greater than or equal to a preset height ratio, the ratio of the area of ​​the text region to the area of ​​the image to be processed is between a first area ratio and a second area ratio, and the skew data of the text region satisfies the subtitle skew constraint condition, then the text region is determined as a subtitle region in the image to be processed; the second area ratio is greater than the first area ratio.

[0131] As an implementation manner, the subtitle recognition device 100 may further include:

[0132] A first coordinate value determination module, used to determine the coordinate values ​​of a plurality of rectangular boundary intersections of the minimum circumscribed rectangle of the subtitle area;

[0133] A first coordinate value updating module, used for updating the coordinate values ​​of a plurality of boundary intersection points on the boundary of the subtitle area based on the coordinate values ​​of a plurality of rectangular boundary intersection points, to obtain an updated subtitle area;

[0134] A first subtitle region extraction module, used for extracting an updated subtitle region;

[0135] The enhancement processing module is used to perform enhancement processing on the updated subtitle area to obtain an enhanced image to be processed.

[0136] As an implementation manner, the subtitle recognition device 100 may further include:

[0137] A second coordinate value determination module, used to determine the coordinate values ​​of the intersection points of multiple rectangular boundaries of the minimum circumscribed rectangle of the subtitle area;

[0138] A second coordinate value updating module is used to update the coordinate values ​​of multiple boundary intersection points on the boundary of the subtitle area based on the coordinate values ​​of the multiple rectangular boundary intersection points to obtain an updated subtitle area;

[0139] A second subtitle region extraction module, used for extracting an updated subtitle region;

[0140] A translation processing module, used for translating the updated subtitle area;

[0141] The subtitle area display module is used to display the translated subtitle area in the image to be processed.

[0142] As an implementation manner, the above-mentioned boundary recognition module may specifically include:

[0143] An image segmentation unit is used to perform image text segmentation on the image to be processed and obtain a text segmentation result corresponding to the image to be processed;

[0144] A binarization processing unit is used to perform binarization processing on the text segmentation result to obtain a binarization processing result of the image to be processed;

[0145] The boundary determination unit is used to determine the boundary of the text area in the image to be processed according to the binarization processing result of the image to be processed.

[0146] An embodiment of the present application provides a subtitle recognition device, which may be a server. Fig.11: is a schematic diagram of a server structure provided in an embodiment of the present application. The server 900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 922 (for example, one or more processors) and memory 932, and one or more storage media 930 (for example, one or more mass storage devices) storing application programs 942 or data 944. Among them, the memory 932 and the storage medium 930 can be short-term storage or permanent storage. The program stored in the storage medium 930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 922 can be configured to communicate with the storage medium 930 to execute a series of instruction operations in the storage medium 930 on the server 900.

[0147] The server 900 may also include one or more power supplies 926, one or more wired or wireless network interfaces 950, one or more input and output interfaces 958, and / or one or more operating systems 941, such as Windows Server 2000. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0148] The CPU 922 is used to execute the following steps:

[0149] Identify the boundaries of text areas in the image to be processed;

[0150] Based on the boundary of the text area, skew data of the text area is obtained; the skew data of the text area is used to characterize the skew degree of the text area;

[0151] If the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

[0152] The present application embodiment also provides another subtitle recognition device, which can be a terminal device. Fig.12 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. Take the terminal device as a mobile phone as an example:

[0153] Fig.12 The block diagram shows a partial structure of a mobile phone provided by an embodiment of the present application. Fig.12The mobile phone includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Fig.12 The mobile phone structure shown in the figure does not constitute a limitation on the mobile phone, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0154] Combine the following Fig.12 A detailed introduction to the various components of the mobile phone:

[0155] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: LowNoiseAmplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications may use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0156] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the mobile phone by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0157] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the mobile phone. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into contact coordinates, and then sends it to the processor 1080, and can receive and execute commands sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0158] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the mobile phone. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Fig.12 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the mobile phone.

[0159] The mobile phone may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the mobile phone is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary. It can be used for applications that identify the posture of the mobile phone (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometer, tapping), etc.; as for other sensors that can be configured in the mobile phone, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0160] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the mobile phone. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another mobile phone through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0161] WiFi is a short-range wireless transmission technology. The mobile phone can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Fig.12 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the mobile phone and can be omitted as needed without changing the essence of the invention.

[0162] The processor 1080 is the control center of the mobile phone. It uses various interfaces and lines to connect various parts of the entire mobile phone. By running or executing software programs and / or modules stored in the memory 1020, and calling data stored in the memory 1020, it executes various functions of the mobile phone and processes data, thereby collecting overall data and information of the mobile phone. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0163] The mobile phone also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0164] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0165] In the embodiment of the present application, the processor 1080 included in the mobile phone also has the following functions:

[0166] Identify the boundaries of text areas in the image to be processed;

[0167] Based on the boundary of the text area, skew data of the text area is obtained; the skew data of the text area is used to characterize the skew degree of the text area;

[0168] If the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

[0169] The embodiment of the present application further provides a computer-readable storage medium for storing a computer program. When the computer program is executed on a subtitle recognition device, the subtitle recognition device executes any implementation of a subtitle recognition method described in the aforementioned embodiments.

[0170] The embodiment of the present application also provides a computer program product including a computer program. When the computer program product is run on a subtitle recognition device, the subtitle recognition device executes any implementation of a subtitle recognition method described in the aforementioned embodiments.

[0171] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0172] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the system is only a logical function division. There may be other division methods in actual implementation, such as multiple systems can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0173] The systems described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0175] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store computer programs.

[0176] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A subtitle recognition method, characterized in that: include: Identify the boundaries of text areas in the image to be processed; Based on the boundary of the text area, obtaining the skew data of the text area; The skew data of the text area is used to characterize the skew degree of the text area; If the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

2. The method according to claim 1, characterized in that The obtaining the skew data of the text area based on the boundary of the text area specifically includes: Based on the positions of a plurality of reference points on the boundary of the text area, the skew data of the text area is obtained.

3. The method according to claim 2, characterized in that The multiple reference points on the border of the text area include multiple border intersections on the border of the text area; the multiple border intersections include an upper left corner border intersection, a lower left corner border intersection, an upper right corner border intersection, and a lower right corner border intersection; The obtaining the skew data of the text area based on the positions of a plurality of reference points on the boundary of the text area specifically includes: Based on the coordinate values ​​of the upper left corner boundary intersection, the lower left corner boundary intersection, the upper right corner boundary intersection and the lower right corner boundary intersection, the skew data of at least one pair of two boundaries in relative positions of the text area are obtained.

4. The method according to claim 3, characterized in that The step of obtaining the skew data of at least one pair of two relatively positioned boundaries of the text area based on the coordinate value of the upper left corner boundary intersection, the coordinate value of the lower left corner boundary intersection, the coordinate value of the upper right corner boundary intersection, and the coordinate value of the lower right corner boundary intersection specifically includes: Obtaining a first horizontal coordinate offset value between the upper left corner boundary intersection point and the lower left corner boundary intersection point, and a second horizontal coordinate offset value between the upper right corner boundary intersection point and the lower right corner boundary intersection point; Obtaining a first ordinate offset value between the upper left corner boundary intersection point and the upper right corner boundary intersection point, and a second ordinate offset value between the lower left corner boundary intersection point and the lower right corner boundary intersection point; Using the first horizontal coordinate offset value as the skew data of the left boundary of the text area, using the second horizontal coordinate offset value as the skew data of the right boundary of the text area, using the first vertical coordinate offset value as the skew data of the upper boundary of the text area, and using the second vertical coordinate offset value as the skew data of the lower boundary of the text area; If the skew data of the text area satisfies the subtitle skew constraint condition, determining the text area as the subtitle area in the image to be processed specifically includes: If the absolute value of the first coordinate difference, the absolute value of the second coordinate difference, the absolute value of the third coordinate difference and the absolute value of the fourth coordinate difference are all smaller than their corresponding difference thresholds, the text area is determined as a subtitle area in the image to be processed.

5. The method according to claim 2, characterized in that: The multiple reference points on the boundary of the text area include multiple boundary midpoints on the boundary of the text area; and obtaining the skew data of the text area based on the positions of the multiple reference points on the boundary of the text area specifically includes: Obtaining skew data of at least one pair of two boundaries of the text area at relative positions based on coordinate values ​​of boundary midpoints of at least one pair of two boundaries of the text area at relative positions; If the skew data of the text area satisfies the subtitle skew constraint condition, determining the text area as the subtitle area in the image to be processed specifically includes: If the skew data of at least one pair of two boundaries of the text area at relative positions meet the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

6. The method according to claim 1, characterized in that The obtaining the skew data of the text area based on the boundary of the text area specifically includes: Determine the angle between at least one pair of two relatively positioned boundaries of the text area and the reference straight line; If the skew data of the text area satisfies the subtitle skew constraint condition, determining the text area as the subtitle area in the image to be processed specifically includes: If the angles between at least one pair of two relatively positioned boundaries of the text area and the reference straight line are both less than or equal to a first degree threshold, the text area is determined as a subtitle area of ​​the image to be processed.

7. The method according to claim 1, characterized in that If the skew data of the text area satisfies the subtitle skew constraint condition, determining the text area as the subtitle area in the image to be processed specifically includes: If the ratio of the area height of the text area to the image height of the image to be processed is greater than or equal to a preset height ratio, and the skew data of the text area meets the subtitle skew constraint condition, the text area is determined as the subtitle area in the image to be processed.

8. The method according to claim 1, characterized in that If the skew data of the text area satisfies the subtitle skew constraint condition, determining the text area as the subtitle area in the image to be processed specifically includes: If the ratio of the area of ​​the text area to the area of ​​the image to be processed is between a first area ratio and a second area ratio, and the skew data of the text area satisfies the subtitle skew constraint, the text area is determined as a subtitle area in the image to be processed; the second area ratio is greater than the first area ratio.

9. The method according to claim 1, characterized in that: If the skew data of the text area satisfies the subtitle skew constraint condition, determining the text area as the subtitle area in the image to be processed specifically includes: If the ratio of the region height of the text region to the image height of the image to be processed is greater than or equal to a preset height ratio, the ratio of the area of ​​the text region to the area of ​​the image to be processed is between a first area ratio and a second area ratio, and the skew data of the text region satisfies the subtitle skew constraint condition, then the text region is determined as a subtitle region in the image to be processed; the second area ratio is greater than the first area ratio.

10. The method according to claim 1, characterized in that After determining the text area as the subtitle area in the image to be processed, the method further includes: Determine the coordinate values ​​of the intersection points of multiple rectangular boundaries of the minimum bounding rectangle of the subtitle area; Based on the coordinate values ​​of the plurality of rectangular boundary intersection points, the coordinate values ​​of the plurality of boundary intersection points on the boundary of the subtitle area are updated to obtain an updated subtitle area; Extracting the updated subtitle area; The updated subtitle area is enhanced to obtain an enhanced image to be processed.

11. The method according to claim 1, characterized in that: After determining the subtitle area as the subtitle area in the image to be processed, the method further includes: Determine the coordinate values ​​of the intersection points of multiple rectangular boundaries of the minimum bounding rectangle of the subtitle area; Based on the coordinate values ​​of the plurality of rectangular boundary intersection points, the coordinate values ​​of the plurality of boundary intersection points on the boundary of the subtitle area are updated to obtain an updated subtitle area; Extracting the updated subtitle area; Performing translation processing on the updated subtitle area; The translated subtitle area is displayed in the image to be processed.

12. The method according to claim 1, characterized in that The step of identifying the boundary of the text area in the image to be processed specifically includes: Performing image and text segmentation on the image to be processed to obtain a text segmentation result corresponding to the image to be processed; Binarization is performed on the text segmentation result to obtain a binarization result of the image to be processed; According to the binarization result of the image to be processed, the boundary of the text area in the image to be processed is determined.

13. A subtitle recognition device, characterized in that: include: A boundary recognition module, used to recognize the boundary of the text area in the image to be processed; A skew data acquisition module, used for acquiring skew data of the text area based on the boundary of the text area; The skew data of the text area is used to characterize the skew degree of the text area; The subtitle region determining module is used to determine the text region as the subtitle region in the image to be processed if the skew data of the text region meets the subtitle skew constraint condition.

14. A subtitle recognition device, characterized in that: The device comprises a processor and a memory: The memory is used to store a computer program and transmit the computer program to the processor; The processor is configured to execute the steps of the subtitle recognition method according to any one of claims 1 to 12 according to the instructions in the computer program.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store a computer program, and when the computer program is executed by a subtitle recognition device, the steps of the subtitle recognition method according to any one of claims 1 to 12 are implemented.

Images