Subtitle extraction method and device

Abstract

The invention discloses a subtitle extraction method and device, electronic equipment and a readable storage medium, and belongs to the technical field of electronic equipment. The method comprises the steps that at least one key frame group is determined in a first mode, the key frame group comprises at least one first image frame in a first video, and subtitle information in each first image frame in the same key frame group is the same; extracting subtitle information of any image frame in each key frame group to obtain subtitle information of the first video; wherein the first mode comprises any one of the following steps: determining based on a gray difference value between pixel points in an image frame in the first video; and determining based on the edge similarity of the adjacent image frames in the first video.

Description

Technical Field

[0001] The present application belongs to the technical field of electronic equipment, and specifically relates to a subtitle extraction method, device, electronic equipment and readable storage medium. Background Art

[0002] With the development of electronic device technology, more and more users need to use electronic devices to watch videos. In one scenario, users need to obtain subtitle information in a video, but some videos do not come with a dedicated subtitle file. In this regard, in related technologies, optical character recognition (OCR) technology is usually used to extract subtitle information contained in an image frame from the image frame.

[0003] However, when there are too many video frames, it will take a lot of time to detect a video frame by frame, extract the subtitle information in each image frame through OCR technology, and merge the same results, making the efficiency of subtitle extraction by electronic devices too low. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a subtitle extraction method, device, electronic device and readable storage medium, which can improve the efficiency of electronic devices in extracting subtitles from videos.

[0005] In a first aspect, an embodiment of the present application provides a subtitle extraction method, which includes: using a first method to determine at least one key frame group, the key frame group includes at least one image frame in a first video, and the subtitle information in each image frame in the same key frame group is the same; extracting the subtitle information of any image frame in each key frame group to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between pixels in the image frame in the first video; based on the edge similarity of adjacent image frames in the first video.

[0006] In a second aspect, an embodiment of the present application provides a subtitle extraction device, which includes: a processing module and an extraction module; the processing module is used to determine at least one key frame group using a first method, the key frame group includes at least one image frame in a first video, and the subtitle information in each image frame in the same key frame group is the same; the extraction module is used to extract the subtitle information of any image frame in each key frame group to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between pixels in the image frame in the first video; based on the edge similarity of adjacent image frames in the first video.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the method described in the first aspect.

[0010] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method described in the first aspect.

[0011] In an embodiment of the present application, the electronic device determines at least one key frame group by adopting a first method, wherein the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; the subtitle information of any image frame in each key frame group is extracted to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between the pixels in the image frames in the first video; based on the edge similarity of adjacent image frames in the first video. In this scheme, on the one hand, since there is a large grayscale difference between the grayscale value of the pixel points in the area where the same subtitle information is located and the grayscale value of the pixel points in other areas, the electronic device can determine whether the subtitle information areas between the image frames are the same by calculating the grayscale difference of the statistical image frames, thereby determining a group of image frames as a key frame group. On the other hand, since the edges of the same subtitle information areas are the same, the electronic device determines the subtitle information areas with high edge similarity as the same key frame group by comparing the edge similarities of adjacent image frames. In this way, after the electronic device has determined multiple key frame groups with the same subtitle information in the first video, it can extract a frame of image frames from each key frame group to extract subtitles, while ensuring the accuracy of the subtitle information, reducing the number of image frames for extracting subtitles, thereby improving the efficiency of the electronic device in extracting subtitles from the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is one of the flow charts of a subtitle extraction method provided in an embodiment of the present application;

[0013] Figure 2This is a second flow chart of a subtitle extraction method provided in an embodiment of the present application;

[0014] Figure 3 This is a third flow chart of a subtitle extraction method provided in an embodiment of the present application;

[0015] Figure 4 This is a fourth flow chart of a subtitle extraction method provided in an embodiment of the present application;

[0016] Figure 5 This is one of the structural schematic diagrams of a subtitle extraction device provided in an embodiment of the present application;

[0017] Figure 6 This is a second structural diagram of a subtitle extraction device provided in an embodiment of the present application;

[0018] Figure 7 This is one of the hardware structure diagrams of an electronic device provided in an embodiment of the present application;

[0019] Figure 8 This is the second schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0021] Some terms and nouns involved in the embodiments of the present application are explained below.

[0022] Grayscale value: In digital image processing, grayscale value is a numerical value used to represent the brightness or grayscale level of each pixel in the image. In a typical grayscale image, the grayscale value of each pixel is usually represented by an integer between 0 and 255, where 0 represents black and 255 represents white. Other values ​​represent different degrees of gray.

[0023] Edge detection: Edge detection is a technique in digital image processing that is used to identify the boundaries or edges between different areas in an image. Image edges are usually places where pixel intensity changes abruptly, which may represent the boundaries of objects or important features in the image.

[0024] Optical Character Recognition (OCR): The process of processing and identifying images or videos containing text content and extracting the text and layout information contained therein. Currently, there are many mature self-developed or industry solutions.

[0025] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0026] The terms "at least one (item)", "at least one of" and the like in the specification and claims of the present application refer to any one, any two or a combination of more than two of the objects included therein. For example, at least one (item) of a, b, and c can be represented by: "a", "b", "c", "a and b", "a and c", "b and c" and "a, b and c", where a, b, and c can be single or multiple. Similarly, "at least two (items)" refers to two or more, and its meaning is similar to that of "at least one (item)".

[0027] The logo in this application refers to text, symbols, images, etc. used to indicate information. Logos or other containers can be used as carriers to display information, including but not limited to text logos, image logos, symbol logos, etc.

[0028] It should be noted that the subtitle extraction method provided in the embodiments of the present application can be executed by electronic devices such as mobile phones, tablet computers, laptop computers, PDAs, and vehicle-mounted electronic devices. In some embodiments of the present application, the subtitle extraction method provided in the embodiments of the present application is described by taking the electronic device as the execution subject to execute the subtitle extraction method as an example.

[0029] The subtitle extraction method, device, electronic device and readable storage medium provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings through specific embodiments and their application scenarios.

[0030] When users watch videos, they sometimes need to obtain some subtitle information in the video. However, some video files do not come with subtitle files, and subtitle information cannot be directly obtained from the subtitle files. In this case, the electronic device can use OCR technology to extract subtitles, that is, to process and recognize images or videos containing text content, and extract the subtitle information contained therein, such as text and layout information.

[0031] Since the number of frames in a video is generally 25 frames or 30 seconds, for a 40-minute video, the total number of frames can reach 60,000 frames. If an electronic device uses OCR technology to extract subtitles and merges the same results, the result will undoubtedly be the most accurate, but this will be very time-consuming. Of course, it is also possible to obtain a group of image frames at a certain time interval and perform OCR recognition on any frame in the group of image frames, such as performing OCR recognition every 0.1s, which will greatly reduce the recognition time. However, since the time when the subtitles appear is not necessarily an integer of 0.1s, the recognized subtitle information may not be accurate enough.

[0032] In this regard, in the subtitle extraction method provided by the embodiment of the present application, the electronic device determines at least one key frame group by adopting a first method, the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; extracts the subtitle information of any image frame in each key frame group to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between the pixels in the image frame in the first video; based on the edge similarity of adjacent image frames in the first video. In this scheme, on the one hand, since there is a large grayscale difference between the grayscale value of the pixel point in the area where the same subtitle information is located and the grayscale value of the pixel point in other areas, the electronic device can determine whether the subtitle information area between the image frames is the same by calculating the grayscale difference of the statistical image frames, thereby determining a group of image frames as a key frame group. On the other hand, since the edges of the same subtitle information area are the same, the electronic device determines the subtitle information area with high edge similarity as the same key frame group by comparing the edge similarity of adjacent image frames. In this way, after the electronic device has determined multiple key frame groups with the same subtitle information in the first video, it can extract a frame of image frames from each key frame group to extract subtitles, while ensuring the accuracy of the subtitle information, reducing the number of image frames for extracting subtitles, thereby improving the efficiency of the electronic device in extracting subtitles from the video.

[0033] The execution subject of the subtitle extraction method provided in the embodiment of the present application may be a subtitle extraction device. Exemplarily, the subtitle extraction device may be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The subtitle extraction method provided in the embodiment of the present application will be described below by taking an electronic device as an example.

[0034] The present application embodiment provides a subtitle extraction method. Figure 1 FIG. 1 is a flowchart of a subtitle extraction method provided by an embodiment of the present application, which can be applied to electronic devices. Figure 1 As shown, the subtitle extraction method provided in the embodiment of the present application may include the following steps 201 and 202:

[0035] Step 201: The electronic device determines at least one key frame group using a first method.

[0036] In some embodiments of the present application, the above-mentioned key frame group includes at least one image frame in the first video.

[0037] In some embodiments of the present application, the subtitle information in each image frame in the same key frame group is the same.

[0038] For example, the subtitle information of each image frame in one key frame group is "Hello", and the subtitle information of each image frame in another key frame group is "Goodbye".

[0039] In some embodiments of the present application, the first method includes any one of the following:

[0040] Based on the grayscale difference between pixels in the image frame in the first video;

[0041] Based on edge similarities of adjacent image frames in the first video.

[0042] In a possible embodiment, the electronic device obtains the maximum grayscale difference of all pixels of each image frame in a group of image frames, and then determines the subtitle area in the image frame based on the maximum grayscale difference of the pixels, and then determines that the group of image frames is the same key frame group based on the pixel position of the maximum grayscale difference of the pixels in the subtitle area in the image frame.

[0043] Exemplarily, the maximum grayscale difference is a grayscale difference with the largest difference between the grayscale value of a pixel and the grayscale values ​​of surrounding pixels.

[0044] It is understandable that since subtitles are generally white, that is, the grayscale value is 0, the grayscale value difference from other areas of the image frame is usually too large. Therefore, this embodiment can determine whether the subtitle information between image frames is the same by judging the difference between the grayscale values, thereby determining whether a group of image frames is a key frame group.

[0045] In another possible embodiment, the electronic device performs edge detection on each image frame to identify the boundaries or edges of different areas in the image to find the subtitle area of ​​an image frame, and obtains the edge similarity between the two adjacent image frames by comparing the edge areas between two adjacent image frames. When the edge similarity is greater than a preset threshold, the electronic device determines the two adjacent image frames as two image frames in a key frame group.

[0046] It is understandable that if the subtitle information in the same key frame group is the same, the edge area will also be the same. Therefore, the electronic device can obtain the edge similarity of two adjacent image frames by edge detection of the image frames, thereby determining that the two adjacent image frames are in the same key frame group.

[0047] Step 202: The electronic device extracts subtitle information of any image frame in each key frame group to obtain subtitle information of the first video.

[0048] In some embodiments of the present application, the subtitle information includes at least one of the following: text in the image frame, and text layout information in the image frame.

[0049] In some embodiments of the present application, the electronic device uses OCR technology to extract subtitle information from any image frame in each key frame group, and then arranges and combines the subtitle information corresponding to each group of key frames to obtain all subtitle information of the first video.

[0050] In the subtitle extraction method provided in the embodiment of the present application, the electronic device determines at least one key frame group by adopting the first method, the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; extracts the subtitle information of any image frame in each key frame group to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between the pixels in the image frame in the first video; based on the edge similarity of adjacent image frames in the first video. In this scheme, on the one hand, since there is a large grayscale difference between the grayscale value of the pixel point in the area where the same subtitle information is located and the grayscale value of the pixel point in other areas, the electronic device can determine whether the subtitle information area between the image frames is the same by calculating the grayscale difference of the statistical image frames, thereby determining a group of image frames as a key frame group. On the other hand, since the edges of the same subtitle information area are the same, the electronic device determines the subtitle information area with high edge similarity as the same key frame group by comparing the edge similarity of adjacent image frames. In this way, after the electronic device has determined multiple key frame groups with the same subtitle information in the first video, it can extract a frame of image frames from each key frame group to extract subtitles, while ensuring the accuracy of the subtitle information, reducing the number of image frames for extracting subtitles, thereby improving the efficiency of the electronic device in extracting subtitles from the video.

[0051] The following two specific examples are used to exemplarily illustrate the process of determining key frames by the electronic device in the embodiments of the present application using different first methods.

[0052] Embodiment 1: The first method is based on the grayscale difference between pixels in the image frame in the first video.

[0053] Optionally, in some embodiments of the present application, combined with Figure 1 ,like Figure 2 As shown, before the above step 201, the subtitle extraction method provided by the embodiment of the present application further includes step 301:

[0054] Step 301: The electronic device acquires M first image frames within a time window.

[0055] Wherein, M is a positive integer.

[0056] In some embodiments of the present application, the time window is the duration for acquiring a group of image frames.

[0057] Exemplarily, the time window may be 1s, 3s or 10s. Specifically, it may be a default setting of the electronic device or a setting of the user.

[0058] Exemplarily, when the time window is 3 seconds and the image frame rate is 25 frames per second, the electronic device can acquire an image frame set containing 75 image frames.

[0059] It should be noted that when this method is started for the first time, a set of image frames will be obtained based on the time window starting from the first frame at the beginning of the video. If it is not started for the first time, it will start from the set starting picture.

[0060] In this way, the electronic device obtains a group of image frames in a time window to simultaneously determine whether the subtitle information between the group of image frames is the same. If they are the same, they are determined to be the same key frame group, thereby reducing the time required to determine the key frame group and improving the efficiency of the electronic device in extracting subtitles.

[0061] Further, in some embodiments of the present application, combined with Figure 2 ,like Figure 3 As shown, the above step 201 can be specifically implemented through the following steps 201a to 201d.

[0062] Step 201a: The electronic device determines a first grayscale difference matrix according to each first image frame.

[0063] In some embodiments of the present application, the first grayscale difference matrix includes at least one first matrix element.

[0064] In some embodiments of the present application, the first matrix element is determined by a first difference value corresponding to a pixel point at the same position in the corresponding first image frame.

[0065] In some embodiments of the present application, the first difference is the maximum difference among the grayscale value differences between a pixel and each of the adjacent pixels.

[0066] In some embodiments of the present application, taking a first image frame as an example, the electronic device grayscales the first image frame to obtain the grayscale value of each pixel in the first image frame. Since the first image frame is a rectangle, assuming that its width is W and its length is H, a grayscale value matrix of size W*H will be obtained. As shown in Table 1 below, each table is used to represent a pixel in the first image frame, G hw Represents a grayscale value.

[0067]

[0068]

[0069] Table 1 Gray value matrix of image frame with width W and height H

[0070] For each pixel of the first image frame, the maximum value of the grayscale difference between the pixel and the surrounding pixels is calculated to obtain a first grayscale difference matrix as shown in Table 2. hw It is the maximum grayscale difference between the grayscale value of the pixel at the Hth row and the Wth column and the grayscale values ​​of the surrounding pixels, that is, the first matrix element mentioned above.

[0071] <![CDATA[D 11 ]]> <![CDATA[D 12 ]]> … <![CDATA[D 1w-1 ]]> <![CDATA[D 1w ]]> <![CDATA[D 21 ]]> <![CDATA[D 22 ]]> … <![CDATA[D 2w-1 ]]> <![CDATA[D 2w ]]> … … <![CDATA[D MN ]]> … … <![CDATA[D (h-1)1 ]]> <![CDATA[D (h-1)2 ]]> … <![CDATA[D (h-1)(w-1) ]]> <![CDATA[D (h-1)w ]]> <![CDATA[D h1 ]]> <![CDATA[D h2 ]]> … <![CDATA[D h(w-1) ]]> <![CDATA[D hw ]]>

[0072] Table 2 The first grayscale difference matrix of the image frame with width W and height H

[0073] For example, in the first image frame, the gray value of the Mth row and the Nth column is G MN The grayscale value of all surrounding pixels is: G M-1N-1 , G M-1N , G M-1N+1 , G MN-1 , G MN+1 , G M+1N-1 , G M+1N , G M+1N+1 , then the maximum grayscale difference between the grayscale value of the pixel and the grayscale values ​​of the surrounding pixels is calculated as: D MN =Max(Abs(G M-1N-1 -G MN ), Abs(G M-1N -G MN ),Abs(G M-1N+1 -G MN ), Abs(G MN-1 -G MN ), Abs(G MN+1 -G MN ), Abs(G M+1N-1 -G MN ), Abs(G M+1N -G MN ), Abs(G M+1N+1 -G MN )). Where Abs is the absolute value.

[0074] It can be understood that the electronic device repeats the above process to obtain the first grayscale difference matrix of all first image frames within the time window.

[0075] Step 201b: The electronic device determines a quantity matrix according to the M grayscale difference matrices.

[0076] In some embodiments of the present application, the above-mentioned number matrix includes at least one second matrix element.

[0077] In some embodiments of the present application, the second matrix element is determined by the number of third matrix elements in the M first matrix elements at the same position in the M grayscale difference matrices.

[0078] In some embodiments of the present application, the third matrix element is greater than or equal to a preset grayscale difference.

[0079] In some embodiments of the present application, the preset grayscale difference value may be a default setting of the electronic device or a user-defined setting.

[0080] In some embodiments of the present application, the electronic device counts the number of first matrix elements at the same position in each grayscale difference matrix, that is, whether the maximum grayscale difference of the pixel point at this position is greater than or equal to the number of pixel points with a preset grayscale difference, to obtain the above-mentioned quantity matrix.

[0081] Exemplarily, taking the above-mentioned set of image frames as 75 frames as an example, each image frame corresponds to a first grayscale difference matrix, and the format in which the matrix elements at the same position in each first grayscale difference matrix are greater than the preset grayscale value is counted one by one. Specifically, if the first matrix element of the first first grayscale difference matrix is ​​greater than or equal to the preset grayscale difference, it is recorded as 1; if it is less than the preset grayscale difference, it is recorded as 0. The first matrix elements of all first grayscale difference matrices are counted to obtain a quantity value, namely the above-mentioned second matrix element, to obtain the quantity matrix shown in Table 3 below.

[0082] For example, the calculation method of the second matrix element corresponding to the pixel point in the Mth row and the Nth column is: MN =(D in the first grayscale difference matrix of the first image frame MN Is the value of greater than or equal to the preset grayscale value? If it is greater than or equal to the preset grayscale value, it is 1. Otherwise, it is 0) + (D in the first grayscale difference matrix of the first image frame MN Is the value of greater than or equal to the preset grayscale value? If it is greater than or equal to the preset grayscale value, it is 1. Otherwise, it is 0) +… + (D in the first grayscale difference matrix of the first image frame MN Is the value of greater than or equal to the preset grayscale value? If it is greater than or equal to the preset grayscale value, it is 1. Otherwise, it is 0). To obtain the second matrix element in each quantity matrix.

[0083] <![CDATA[C 11 ]]> <![CDATA[C 12 ]]> … <![CDATA[C 1w-1 ]]> <![CDATA[C 1w ]]> <![CDATA[C 21 ]]> <![CDATA[C 22 ]]> … <![CDATA[C 2w-1 ]]> <![CDATA[C 2w ]]> … … <![CDATA[C MN ]]> … … <![CDATA[C (h-1)1 ]]> <![CDATA[C (h-1)2 ]]> … <![CDATA[C (h-1)(w-1) ]]> <![CDATA[C (h-1)w ]]> <![CDATA[C h1 ]]> <![CDATA[C h2 ]]> … <![CDATA[Ch (w-1) ]]> <![CDATA[C hw ]]>

[0084] Table 3 The number matrix of a set of image frames with width W and height H

[0085] In the quantity matrix shown in Table 3, C 11 To C hwThe H*W points represent this group of image frames respectively, that is, the number of points in all the first image frames whose maximum grayscale difference between this point in the Hth row and Wth column and the surrounding points is greater than or equal to the preset grayscale value.

[0086] Step 201c: The electronic device determines a first number of second matrix elements in at least one second matrix element that is greater than or equal to a first preset number value.

[0087] In some embodiments of the present application, the first preset quantity value may be user-defined or a default setting of the electronic device.

[0088] Exemplarily, the first preset number value may generally be set to a half or two-thirds of the number of all first image frames.

[0089] In some embodiments of the present application, the electronic device counts the number of second matrix elements in the quantity matrix that are greater than or equal to a first preset quantity value, that is, the above-mentioned first quantity.

[0090] Exemplarily, the first quantity calculation method is: Total = (C 11 Is it greater than or equal to the first preset value, if yes, then it is 1, otherwise it is 0)+(C 12 Is it greater than or equal to the first preset value, if yes, then it is 1, otherwise it is 0)+…+(C h(W-1) Is it greater than or equal to the first preset value, if yes, then it is 1, otherwise it is 0)+(C hw Is it greater than or equal to a first preset quantity value, if yes, it is 1, otherwise it is 0).

[0091] Step 201d: When the first number is greater than or equal to a second preset number, the electronic device determines the M first image frames as a key frame group.

[0092] In some embodiments of the present application, the second preset quantity value may be user-defined or a default setting of the electronic device.

[0093] Exemplarily, the second preset value may be generally set to one tenth, one eighth, etc. of the total number of pixels in the image frame. In other words, it can be considered that the second preset value is generally the number of pixels in the image frame occupied by the subtitle area.

[0094] In this way, the electronic device obtains the image frame containing the subtitles and its duration by calculating the grayscale difference of the pixels in the image frame, and then performs an OCR operation on one image frame in this group of key frames to obtain the subtitle information. It is no longer necessary to perform OCR operation on each image frame one by one, which greatly reduces the workload of subtitle extraction.

[0095] Optionally, after the above step 201d, the subtitle extraction method provided by the embodiment of the present application may further include the following steps 401 and 402.

[0096] Step 401: The electronic device determines a second grayscale difference matrix according to a second image frame.

[0097] In some embodiments of the present application, the second image frame is an image frame following the last image frame of the key frame group.

[0098] In some embodiments of the present application, the second grayscale difference matrix includes at least one fourth matrix element.

[0099] In some embodiments of the present application, the fourth matrix element is determined by a second difference value corresponding to a pixel point at the same position in the corresponding second image frame.

[0100] In some embodiments of the present application, the second difference is the maximum difference between the grayscale value differences of a pixel and each of the adjacent pixels.

[0101] It should be noted that the process of determining the second grayscale difference matrix is ​​the same as the above step 201a. The specific process can refer to the above step 201a and will not be repeated here.

[0102] Step 402: When the number of the fifth matrix elements is greater than the third preset number, the electronic device determines the second image frame as an image frame in the key frame group.

[0103] In some embodiments of the present application, the fifth matrix element is a matrix element having the same pixel information as the pixel point corresponding to the third matrix element.

[0104] In some embodiments of the present application, the pixel information of the above-mentioned pixel point is the position information of the pixel point in the image frame, that is, the coordinate information, for example, (1,1) represents the coordinate information of the first pixel point.

[0105] Exemplarily, the number n of pixels at the same position of the pixels corresponding to the first grayscale difference matrix and the second grayscale difference matrix is ​​compared. For example, the pixel points corresponding to the matrix elements greater than or equal to the preset grayscale value in the first grayscale difference matrix are (1,1), (2,2), (3,3), (4,4), and the pixel points corresponding to the matrix elements greater than or equal to the preset grayscale value in the second grayscale difference matrix are (1,1), (2,2), (3,3), (5,5), then (1,1), (2,2), (3,3) are the same points. If the number of the same pixels is greater than the third preset number value, it is considered that the second image frame is an image frame in the key frame group to which the previous image frame belongs.

[0106] Exemplarily, after determining that the second image frame is an image frame in the key frame group to which the previous image frame belongs, the process continues to determine whether the next image frame of the second image frame also belongs to this key frame group, until it is determined that a certain image frame does not belong to this key frame group, and then continues to acquire the next group of image frames.

[0107] In this way, after acquiring a group of image frames and determining that the group of image frames is a key frame, the electronic device can further determine whether subsequent image frames belong to the key frame group only based on the grayscale difference matrix of the image frames, thereby avoiding omissions.

[0108] Embodiment 2: The first method is based on edge similarity of adjacent image frames in the first video.

[0109] Optionally, in some embodiments of the present application, combined with Figure 1 ,like Figure 4 As shown, the above step 201 can be specifically implemented through the following steps 201e to 201g.

[0110] Step 201e: The electronic device determines position information of edge pixels in each image frame of the first video.

[0111] In some embodiments of the present application, the edge pixel points are pixel points corresponding to the boundary or edge between two different areas in the image frame.

[0112] In some embodiments of the present application, the above-mentioned position information is pixel coordinate information of edge pixels.

[0113] In some embodiments of the present application, the electronic device may use an edge detection method, such as Laplacian, to obtain position information of edge pixels in each image frame.

[0114] Exemplarily, the position information of the edge pixel point may be as shown in the edge matrix of Table 4 below, where a matrix element in the edge matrix is ​​used to indicate whether a pixel point is edge similarity.

[0115] <![CDATA[EK 11 ]]> <![CDATA[EK 12 ]]> … <![CDATA[EK 1w-1 ]]> <![CDATA[EK 1w <!-- 8 -->]]> <![CDATA[EK 21 ]]> <![CDATA[EK 22 ]]> … <![CDATA[EK 2w-1 ]]> <![CDATA[EK 2w ]]> … … <![CDATA[EK MN ]]> … … <![CDATA[EK (h-1)1 ]]> <![CDATA[EK (h-1)2 ]]> … <![CDATA[EK (h-1)(w-1) ]]> <![CDATA[EK (h-1)w ]]> <![CDATA[EK h1 ]]> <![CDATA[EK h2 ]]> … <![CDATA[EK h(w-1) ]]> <![CDATA[EK hw ]]>

[0116] Table 4 Edge matrix corresponding to the Kth image frame

[0117] For EK 11 To EK hw The values ​​of these H*W elements are only 1 and 0. 1 indicates that the pixel is an edge pixel, and 0 indicates that the pixel is a non-edge pixel.

[0118] Step 201f: The electronic device determines edge similarity between the third image frame and the fourth image frame based on position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame.

[0119] In some embodiments of the present application, the edge similarity is used to represent the subtitle similarity between two adjacent image frames.

[0120] In some embodiments of the present application, the edge similarity between the two adjacent image frames can be obtained by taking the intersection of the position information of the edge pixels of the two adjacent image frames.

[0121] Optionally, the above step 201f can be specifically implemented through the following steps A1 and A2.

[0122] Step A1: The electronic device determines a second quantity according to position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame.

[0123] In some embodiments of the present application, the second number is the number of edge pixel points having the same position information in the third image frame and the fourth image frame.

[0124] Exemplarily, the calculation method of the second number may be: assuming that the image frames are the Kth image frame and the K+1th image frame, the value of the point in the Mth row and Nth column in the edge matrix is ​​determined as follows: That is, if the pixel at the Mth row and the Nth column of the Kth image frame and the K+1th image frame are both edge pixels and have a value of 1, the value of DeltaMN is 1. Then, the number of pixels with a DeltaMN value of 1 is counted, which is the second number.

[0125] Step A2: When the second number is greater than or equal to a fourth preset number, the electronic device determines that the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold.

[0126] In some embodiments of the present application, the fourth preset quantity value may be a default setting of the electronic device or may be user-defined.

[0127] Exemplarily, the fourth preset value may be generally set to one tenth, one eighth, etc. of the total number of pixels in the image frame. In other words, it can be considered that the fourth preset value is generally the number of pixels in the image frame occupied by the subtitle area.

[0128] Step 201g: When the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold, the electronic device determines the third image frame and the fourth image frame as a key frame group.

[0129] Exemplarily, the electronic device counts the number of DeltaMN values ​​1 in the third key frame and the fourth key frame. If the number is greater than or equal to the fourth preset value, it is considered that the edge similarity between the third image frame and the fourth image frame is greater than or equal to the preset threshold, that is, the edges of the two images are similar.

[0130] Exemplarily, the electronic device then determines the edge similarity of the fourth image frame and the fifth image frame until a certain image has edges that are not similar to its previous image, and then all the previous image frames are determined as image frames in the same key frame group.

[0131] Thus, this embodiment uses edge detection technology to compare the differences between adjacent pictures to determine the key frame group, so that there is no need to perform OCR operation on each image frame one by one, thereby reducing the workload of subtitle extraction.

[0132] Each of the above-mentioned method embodiments, or various possible implementation methods in each method embodiment, can be executed separately, or any two or more of them can be executed in combination with each other. The specific implementation method can be determined according to actual usage requirements, and the embodiments of the present application do not limit this.

[0133] The subtitle extraction method provided in the embodiment of the present application can be executed by an electronic device or a subtitle extraction device. In the embodiment of the present application, the subtitle extraction device provided in the embodiment of the present application is described by taking the subtitle extraction method executed by the subtitle extraction device as an example.

[0134] Figure 5 FIG. 2 shows a possible structural diagram of a subtitle extraction device involved in an embodiment of the present application. Figure 5 As shown, the subtitle extraction device 700 may include: a processing module 701 and an extraction module 702 .

[0135] Among them, the above-mentioned processing module 701 is used to determine at least one key frame group using a first method, and the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; the above-mentioned extraction module 702 is used to extract the subtitle information of any image frame in each key frame group determined by the above-mentioned processing module 701 to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between pixel points in the image frame in the first video; based on the edge similarity of adjacent image frames in the first video.

[0136] Optionally, in some embodiments of the present application, the first method is based on the grayscale difference between pixels in the image frame in the first video; Figure 5 ,like Figure 6As shown, the above-mentioned device 700 also includes: an acquisition module 703; the acquisition module 703 is used to acquire M first image frames in the time window before the above-mentioned processing module 701 adopts the first method to determine at least one key frame group, where M is a positive integer; the above-mentioned processing module 701 is specifically used to: determine a first grayscale difference matrix according to each first image frame, the first grayscale difference matrix includes at least one first matrix element, the first matrix element is determined by the first difference corresponding to a pixel point at the same position in the corresponding first image frame, and the first difference is the maximum difference between the grayscale value differences of a pixel point and each adjacent pixel point; determine a quantity matrix according to the M grayscale difference matrices, the quantity matrix includes at least one second matrix element, the second matrix element is determined by the number of third matrix elements in the M first matrix elements at the same position in the M grayscale difference matrices, and the third matrix element is greater than or equal to the preset grayscale difference; determine a first number of second matrix elements in at least one second matrix element that is greater than or equal to the first preset number value; when the first number is greater than or equal to the second preset number value, determine the M first image frames as a key frame group.

[0137] Optionally, in some embodiments of the present application, the above-mentioned processing module 701 is also used to determine the second grayscale difference matrix according to the second image frame after determining the M first image frames as a key frame group when the first number is greater than or equal to the second preset number, the second grayscale difference matrix includes at least one fourth matrix element, the fourth matrix element is determined by the second difference corresponding to a pixel point at the same position in the corresponding second image frame, and the second difference is the maximum difference between the grayscale value differences of a pixel point and each adjacent pixel point; the above-mentioned processing module 701 is also used to determine the second image frame as an image frame in the key frame group when the number of the fifth matrix elements is greater than the third preset number; wherein the fifth matrix element is a matrix element having the same pixel information as the pixel point corresponding to the third matrix element, and the second image frame is the image frame after the last image frame in the key frame group.

[0138] Optionally, in some embodiments of the present application, the first method is based on edge similarity of adjacent image frames in the first video;

[0139] The processing module 701 is specifically used for:

[0140] Determine position information of edge pixels in each image frame of the first video;

[0141] Determining edge similarity between the third image frame and the fourth image frame based on position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame;

[0142] When the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold, the third image frame and the fourth image frame are determined as a key frame group.

[0143] Optionally, in some embodiments of the present application, the processing module 701 is specifically used to:

[0144] Determine a second number according to position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame, where the second number is the number of edge pixels with the same position information in the third image frame and the fourth image frame;

[0145] When the second number is greater than or equal to a fourth preset number, it is determined that the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold.

[0146] In the subtitle extraction device provided in the embodiment of the present application, the subtitle extraction device determines at least one key frame group by adopting a first method, wherein the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; the subtitle information of any image frame in each key frame group is extracted to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between the pixels in the image frames in the first video; based on the edge similarity of adjacent image frames in the first video. In this scheme, the subtitle extraction device, on the one hand, since there is a large grayscale difference between the grayscale value of the pixel points in the area where the same subtitle information is located and the grayscale value of the pixel points in other areas, the subtitle extraction device can determine whether the subtitle information areas between the image frames are the same by calculating the grayscale difference of the statistical image frames, thereby determining a group of image frames as a key frame group. On the other hand, since the edges of the same subtitle information areas are the same, the subtitle extraction device determines the subtitle information areas with high edge similarity as the same key frame group by comparing the edge similarities of adjacent image frames. In this way, after determining multiple key frame groups with the same subtitle information in the first video, the subtitle extraction device can extract a frame of image frames from each key frame group to extract subtitles, while ensuring the accuracy of the subtitle information, reducing the number of image frames for extracting subtitles, thereby improving the efficiency of the subtitle extraction device in extracting subtitles from the video.

[0147] The subtitle extraction device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a palm computer, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0148] The subtitle extraction device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0149] The subtitle extraction device provided in the embodiment of the present application can implement each process implemented in the embodiment of the subtitle extraction method and achieve the same technical effect. To avoid repetition, it will not be described here.

[0150] Alternatively, if Figure 7 As shown, the embodiment of the present application further provides an electronic device 800, including a processor 801 and a memory 802, wherein the memory 802 stores a program or instruction that can be executed on the processor 801, and when the program or instruction is executed by the processor 801, each step of the above-mentioned subtitle extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0151] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0152] Figure 8 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of the present application.

[0153] The electronic device 100 includes but is not limited to components such as a radio frequency unit 101, a network module 102, an audio output unit 103, an input unit 104, a sensor 105, a display unit 106, a user input unit 107, an interface unit 108, a memory 109, and a processor 110.

[0154] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 110 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0155] The processor 110 is used to determine at least one key frame group using a first method, wherein the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; the processor 110 is used to extract the subtitle information of any image frame in each key frame group determined by the processor 110 to obtain the subtitle information of the first video; the first method includes any one of the following: based on the grayscale difference between pixels in the image frame in the first video; based on the edge similarity of adjacent image frames in the first video.

[0156] Optionally, in some embodiments of the present application, the first method is based on the grayscale difference between pixels in the image frame in the first video; the processor 110 is also used to obtain M first image frames in the time window before the processor 110 adopts the first method to determine at least one key frame group, where M is a positive integer; the processor 110 is specifically used to: determine a first grayscale difference matrix according to each first image frame, the first grayscale difference matrix includes at least one first matrix element, the first matrix element is determined by the first difference corresponding to a pixel point at the same position in the corresponding first image frame, and the first grayscale difference matrix includes at least one first matrix element. A difference is the maximum difference between the grayscale value differences of a pixel point and each adjacent pixel point; a quantity matrix is ​​determined according to M grayscale difference matrices, the quantity matrix includes at least one second matrix element, the second matrix element is determined by the number of third matrix elements in M ​​first matrix elements with the same position in the M grayscale difference matrices, and the third matrix element is greater than or equal to a preset grayscale difference; a first number of second matrix elements greater than or equal to a first preset number value in at least one second matrix element is determined; when the first number is greater than or equal to the second preset number value, the M first image frames are determined as a key frame group.

[0157] Optionally, in some embodiments of the present application, the processor 110 is further used to determine a second grayscale difference matrix according to the second image frame after determining M first image frames as a key frame group when the first number is greater than or equal to a second preset number, the second grayscale difference matrix including at least one fourth matrix element, the fourth matrix element being determined by a second difference corresponding to a pixel at the same position in the corresponding second image frame, the second difference being the maximum difference between the grayscale value differences of a pixel and each adjacent pixel; the processor 110 is further used to determine the second image frame as an image frame in the key frame group when the number of fifth matrix elements is greater than a third preset number; wherein the fifth matrix element is a matrix element having the same pixel information as the pixel corresponding to the third matrix element, and the second image frame is an image frame after the last image frame in the key frame group.

[0158] Optionally, in some embodiments of the present application, the first method is based on edge similarity of adjacent image frames in the first video;

[0159] The processor 110 is specifically configured to:

[0160] Determine position information of edge pixels in each image frame of the first video;

[0161] Determining edge similarity between the third image frame and the fourth image frame based on position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame;

[0162] When the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold, the third image frame and the fourth image frame are determined as a key frame group.

[0163] Optionally, in some embodiments of the present application, the processor 110 is specifically configured to:

[0164] Determine a second number according to position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame, where the second number is the number of edge pixels with the same position information in the third image frame and the fourth image frame;

[0165] When the second number is greater than or equal to a fourth preset number, it is determined that the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold.

[0166] In the electronic device provided in the embodiment of the present application, the electronic device determines at least one key frame group by adopting a first method, wherein the key frame group includes at least one image frame in the first video, and the subtitle information in each image frame in the same key frame group is the same; the subtitle information of any image frame in each key frame group is extracted to obtain the subtitle information of the first video; wherein the first method includes any one of the following: based on the grayscale difference between the pixels in the image frames in the first video; based on the edge similarity of adjacent image frames in the first video. In this scheme, on the one hand, since there is a large grayscale difference between the grayscale value of the pixel points in the area where the same subtitle information is located and the grayscale value of the pixel points in other areas, the electronic device can determine whether the subtitle information areas between the image frames are the same by calculating the grayscale difference of the statistical image frames, thereby determining a group of image frames as a key frame group. On the other hand, since the edges of the same subtitle information areas are the same, the electronic device determines the subtitle information areas with high edge similarity as the same key frame group by comparing the edge similarities of adjacent image frames. In this way, after the electronic device has determined multiple key frame groups with the same subtitle information in the first video, it can extract a frame of image frames from each key frame group to extract subtitles, while ensuring the accuracy of the subtitle information, reducing the number of image frames for extracting subtitles, thereby improving the efficiency of the electronic device in extracting subtitles from the video.

[0167] It should be understood that in the embodiment of the present application, the input unit 104 may include a graphics processor (Graphics Processing Unit, GPU) 1041 and a microphone 1042, and the graphics processor 1041 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 106 may include a display panel 1061, and the display panel 1061 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 107 includes a touch panel 1071 and at least one of other input devices 1072. The touch panel 1071 is also called a touch screen. The touch panel 1071 may include two parts: a touch detection device and a touch controller. Other input devices 1072 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0168] The memory 109 can be used to store software programs and various data. The memory 109 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 109 may include a volatile memory or a non-volatile memory, or the memory 109 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 109 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0169] The processor 110 may include one or more processing units; optionally, the processor 110 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 110.

[0170] The embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned subtitle extraction method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0171] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0172] The embodiment of the present application further provides a chip, which includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned subtitle extraction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0173] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0174] The embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned subtitle extraction method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0175] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0176] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A subtitle extraction method, characterized in that: The method comprises: In a first manner, at least one key frame group is determined, wherein the key frame group includes at least one image frame in the first video, and subtitle information in each image frame in the same key frame group is the same; Extracting subtitle information of any image frame in each of the key frame groups to obtain subtitle information of the first video; The first method includes any one of the following: Based on the grayscale difference between pixels in the image frame in the first video; Based on edge similarities of adjacent image frames in the first video.

2. The method according to claim 1, characterized in that The first method is based on the grayscale difference between pixels in the image frame in the first video; Before adopting the first manner to determine at least one key frame group, the method further includes: Acquire M first image frames within the time window, where M is a positive integer; The first method is used to determine at least one key frame group, including: Determine a first grayscale difference matrix according to each of the first image frames, the first grayscale difference matrix comprising at least one first matrix element, the first matrix element being determined by a first difference corresponding to a pixel point at the same position in the corresponding first image frame, the first difference being the maximum difference between grayscale value differences between the pixel point and each of the adjacent pixels; Determine a quantity matrix according to the M grayscale difference matrices, the quantity matrix comprising at least one second matrix element, the second matrix element being determined by the number of third matrix elements in the M first matrix elements having the same position in the M grayscale difference matrices, the third matrix element being greater than or equal to a preset grayscale difference; Determining a first number of the second matrix elements in at least one of the second matrix elements that is greater than or equal to a first preset number value; When the first number is greater than or equal to a second preset number, M first image frames are determined as one key frame group.

3. The method according to claim 2, characterized in that In the case where the first number is greater than or equal to a second preset number, after determining the M first image frames as one key frame group, the method further includes: Determine a second grayscale difference matrix according to the second image frame, wherein the second grayscale difference matrix includes at least one fourth matrix element, and the fourth matrix element is determined by a second difference corresponding to a pixel point at the same position in the corresponding second image frame, and the second difference is a maximum difference between grayscale value differences of the pixel point and each adjacent pixel point; When the number of the fifth matrix elements is greater than the third preset number, determining the second image frame as an image frame in the key frame group; The fifth matrix element is a matrix element having the same pixel information as that of a pixel point corresponding to the third matrix element, and the second image frame is an image frame subsequent to the last image frame of the key frame group.

4. The method according to claim 1, characterized in that: The first method is based on edge similarity of adjacent image frames in the first video; The first method is used to determine at least one key frame group, including: Determine position information of edge pixels in each image frame of the first video; Determining edge similarity between the third image frame and the fourth image frame based on position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame; In a case where the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold, the third image frame and the fourth image frame are determined as one of the key frame groups.

5. The method according to claim 4, characterized in that The determining the edge similarity between the third image frame and the fourth image frame based on the position information of the edge pixel points in the third image frame and the position information of the edge pixel points in the fourth image frame includes: Determine a second number according to position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame, where the second number is the number of edge pixels with the same position information in the third image frame and the fourth image frame; When the second number is greater than or equal to a fourth preset number, it is determined that the edge similarity between the third image frame and the fourth image frame is greater than or equal to the preset threshold.

6. A subtitle extraction device, characterized in that: The subtitle extraction device comprises: a processing module and an extraction module; The processing module is used to determine at least one key frame group in a first manner, wherein the key frame group includes at least one image frame in the first video, and the subtitle information in each of the image frames in the same key frame group is the same; The extraction module is used to extract the subtitle information of any image frame in the key frame group determined by each processing module to obtain the subtitle information of the first video; The first method includes any one of the following: Based on the grayscale difference between pixels in the image frame in the first video; Based on edge similarities of adjacent image frames in the first video.

7. The device according to claim 6, characterized in that The first method is based on the grayscale difference between pixels in the image frame in the first video; the device also includes: an acquisition module; The acquisition module is used to acquire M first image frames in the time window before the processing module adopts the first method to determine at least one key frame group, where M is a positive integer; The processing module is specifically used for: Determine a first grayscale difference matrix according to each of the first image frames, the first grayscale difference matrix comprising at least one first matrix element, the first matrix element being determined by a first difference corresponding to a pixel point at the same position in the corresponding first image frame, the first difference being the maximum difference between grayscale value differences between the pixel point and each of the adjacent pixels; Determine a quantity matrix according to the M grayscale difference matrices, the quantity matrix comprising at least one second matrix element, the second matrix element being determined by the number of third matrix elements in the M first matrix elements having the same position in the M grayscale difference matrices, the third matrix element being greater than or equal to a preset grayscale difference; Determining a first number of the second matrix elements in at least one of the second matrix elements that is greater than or equal to a first preset number value; When the first number is greater than or equal to a second preset number, M first image frames are determined as one key frame group.

8. The device according to claim 7, characterized in that The processing module is further configured to determine, after determining the M first image frames as one key frame group, a second grayscale difference matrix according to a second image frame when the first number is greater than or equal to a second preset number, the second grayscale difference matrix including at least one fourth matrix element, the fourth matrix element being determined by a second difference corresponding to a pixel point at the same position in the corresponding second image frame, the second difference being the maximum difference between grayscale value differences of the one pixel point and each adjacent pixel point; The processing module is further configured to determine the second image frame as an image frame in the key frame group when the number of the fifth matrix elements is greater than a third preset number; The fifth matrix element is a matrix element having the same pixel information as that of a pixel point corresponding to the third matrix element, and the second image frame is an image frame subsequent to the last image frame of the key frame group.

9. The device according to claim 6, characterized in that The first method is based on edge similarity of adjacent image frames in the first video; The processing module is specifically used for: Determine position information of edge pixels in each image frame of the first video; Determining edge similarity between the third image frame and the fourth image frame based on position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame; In a case where the edge similarity between the third image frame and the fourth image frame is greater than or equal to a preset threshold, the third image frame and the fourth image frame are determined as one of the key frame groups.

10. The device according to claim 9, characterized in that The processing module is specifically used for: Determine a second number according to position information of edge pixels in the third image frame and position information of edge pixels in the fourth image frame, where the second number is the number of edge pixels with the same position information in the third image frame and the fourth image frame; When the second number is greater than or equal to a fourth preset number, it is determined that the edge similarity between the third image frame and the fourth image frame is greater than or equal to the preset threshold.

Images