Method, system and device for converting video into text and medium

Abstract

The invention belongs to the technical field of image recognition, and discloses a method, system and device for converting a video into a text and a medium, and the method comprises the steps: collecting video information containing book content, and carrying out the preprocessing of the video information; performing frame-by-frame conversion on the preprocessed video to obtain a plurality of pictures; screening the obtained pictures to obtain an image with the optimal picture quality in a unit timestamp; and identifying the image with the optimal picture quality to obtain text information. According to the method, the accuracy of the OCR technology in processing book information in a video is improved, the problem that the traditional OCR technology is difficult to recognize text information in a dynamic video picture is effectively solved, the application range of the OCR technology is widened, and meanwhile, the applicability and practicability of the OCR technology are greatly improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image recognition and relates to a method, system, device and medium for converting video into text. Background Art

[0002] With the rapid development of information technology, the acquisition and dissemination of digital content has become increasingly convenient. However, traditional book materials often exist in paper form, and their digitization process usually requires manual participation, which is inefficient and costly. In recent years, with the popularization of video media, a large number of video materials containing book content have been uploaded to the Internet platform, and the book information contained in these videos needs to be effectively extracted and utilized.

[0003] In the prior art, although there are text recognition systems based on OCR technology, there are still many challenges in directly applying them to extract book information from videos. Book images in videos are often affected by multiple factors such as light, angle, and resolution, resulting in uneven image quality, which increases the difficulty of OCR recognition. Book content is presented in various forms in videos, including page turning, zooming, and rotation, which further increases the difficulty of automatic extraction. In addition, most existing OCR technologies are designed for static images, and it is difficult to recognize dynamic book images in videos. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that most of the OCR technologies in the prior art are designed for static images and it is difficult to recognize dynamic book pictures in videos, and to provide a method, system, device and medium for converting video into text.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for converting video into text, comprising:

[0007] Collect video information containing book content and pre-process the video information;

[0008] Convert the preprocessed video frame by frame to obtain several pictures;

[0009] Filter the acquired pictures to obtain the image with the best picture quality within a unit timestamp;

[0010] Recognize the image with the best picture quality and obtain text information.

[0011] A further improvement of the present invention is:

[0012] Furthermore, the video information is preprocessed, specifically: the bit rate, resolution and frame rate of the collected video are adjusted to improve the clarity of the video;

[0013] The pre-processed video is converted frame by frame to obtain a number of pictures, specifically: based on FFmpeg multimedia processing software, the frame rate and picture quality are set, and the images in the video are extracted frame by frame.

[0014] Furthermore, the acquired images are screened, specifically: identifying the timestamp information on each image, setting the length of the unit timestamp, treating the images within the unit timestamp as an image set, and determining whether the contents of the images within the unit timestamp are exactly the same; if not, comparing the percentages of different images in the images within the unit timestamp, treating the images with a smaller percentage as abnormal images, and comparing the abnormal images with the images within other unit timestamps in turn, and adding them to the unit timestamp containing the image.

[0015] Furthermore, before identifying the timestamp information on each picture, the method further includes: identifying the book text information contained in each picture, and if the book text information does not exist in the picture, deleting the picture, and the remaining pictures are pictures containing the book text information.

[0016] Furthermore, the image with the best picture quality within the unit timestamp is obtained, specifically: the images within each unit timestamp are regarded as an image set, the text information contained in the images of the image set is identified, and when it is determined that the accuracy of the identified text is less than the set accuracy threshold, the corresponding image is deleted.

[0017] Furthermore, obtaining the image with the best picture quality within the unit timestamp also includes: obtaining the clarity of the image based on the Sobel operator, and taking the image whose clarity does not exceed the clarity threshold and is ranked high as the image with the best picture quality;

[0018] The method of obtaining the clarity of the image based on the Sobel operator is specifically as follows: based on the convolution kernel of the Sobel operator and each pixel position of the image, the absolute values ​​of the horizontal and vertical gradients are calculated; the absolute values ​​of the horizontal and vertical gradients of each pixel position are added to obtain the sum of the gradients at that position; the larger the value of the gradient sum is, the higher the clarity of the image is.

[0019] Furthermore, based on the convolution kernel of the Sobel operator and each pixel position of the image, the absolute values ​​of the horizontal and vertical gradients are calculated. Specifically, the horizontal gradient Gx is obtained by convolving the horizontal Sobel kernel with the 3x3 neighborhood centered on (i, j) in the image to obtain the horizontal gradient value Gx(i, j) at the position; the vertical gradient Gy is obtained by convolving the vertical Sobel kernel with the 3x3 neighborhood centered on (i, j) in the image to obtain the vertical gradient value Gy(i, j) at the position; for each pixel position, the absolute values ​​of the horizontal and vertical gradients are calculated, namely |Gx(i, j)| and |Gy(i, j)|.

[0020] A system for converting video into text, comprising:

[0021] A collection module, wherein the collection module collects video information containing book content and pre-processes the video information;

[0022] A conversion module, wherein the conversion module converts the preprocessed video frame by frame to obtain a plurality of pictures;

[0023] A screening module, wherein the screening module screens the acquired pictures to obtain an image with the best picture quality within a unit timestamp;

[0024] A recognition module is used to recognize the image with the best picture quality and obtain text information.

[0025] A terminal device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0026] A computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0027] Compared with the prior art, the present invention has the following beneficial effects:

[0028] The present invention converts a dynamic video frame by frame to obtain a plurality of pictures, screens the obtained pictures, deletes pictures without book text information, obtains the clarity of the pictures through the Sobel operator, and then obtains the image with the best picture quality within a unit time stamp; recognizes the image with the best picture quality to obtain text information. The present invention can improve the accuracy of OCR in extracting book information from videos, and improves the application scope and applicability of OCR. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0030] Figure 1 A schematic diagram of a process of converting video into text according to the present invention;

[0031] Figure 2 The present invention is a schematic diagram of the structure of a system for converting video into text. DETAILED DESCRIPTION

[0032] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0033] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0035] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0036] In addition, if the term "horizontal" appears, it does not mean that the component is required to be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0037] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0038] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0039] See also Figure 1 The present invention discloses a method for converting video into text, comprising:

[0040] S101, collecting video information containing book content and preprocessing the video information;

[0041] Adjust the bit rate, resolution and frame rate of the captured video to improve the clarity of the video;

[0042] S102, converting the preprocessed video frame by frame to obtain a number of pictures;

[0043] Based on FFmpeg multimedia processing software, set the frame rate and image quality, and extract images from the video frame by frame.

[0044] After extracting the image, the image is converted to a grayscale image, and Gaussian blur is applied to denoise the image. The Canny edge detector is used to detect the image edge, and morphological operations (dilation and erosion) are used to connect the edges and draw the edge contours to obtain the image after image enhancement.

[0045] S103, screening the acquired pictures to acquire the image with the best picture quality within a unit time stamp;

[0046] The obtained images are screened, specifically: identifying the timestamp information on each image, setting the length of the unit timestamp, treating the images within the unit timestamp as an image set, and determining whether the contents of the images within the unit timestamp are exactly the same; if not, comparing the percentages of different images in the images within the unit timestamp, treating the images with a smaller percentage as abnormal images, and comparing the abnormal images with the images within other unit timestamps in turn, and adding them to the unit timestamp containing the image.

[0047] Before identifying the timestamp information on each picture, the method further includes: identifying the book text information contained in each picture. If the book text information does not exist in the picture, the picture is deleted, and the remaining pictures are pictures containing the book text information.

[0048] Get the image with the best image quality within a unit timestamp, specifically: regard the image within each unit timestamp as an image set, recognize the text information contained in the image of the image set, and delete the corresponding image when the accuracy of the recognized text is less than the set accuracy threshold. When the book content includes page turning, zooming, rotation, etc., these cannot exceed the accuracy threshold, and the corresponding image will be deleted.

[0049] Obtaining an image with the best picture quality within a unit timestamp, further comprising: obtaining the clarity of the image based on a Sobel operator, and taking the image whose clarity does not exceed a clarity threshold and is ranked high as the image with the best picture quality;

[0050] The method of obtaining the clarity of the image based on the Sobel operator is specifically as follows: based on the convolution kernel of the Sobel operator and each pixel position of the image, the absolute values ​​of the horizontal and vertical gradients are calculated; the absolute values ​​of the horizontal and vertical gradients of each pixel position are added to obtain the sum of the gradients at that position; the larger the value of the gradient sum is, the higher the clarity of the image is.

[0051] Based on the convolution kernel of the Sobel operator and each pixel position of the image, the absolute values ​​of the horizontal and vertical gradients are calculated. Specifically, the horizontal gradient Gx is the horizontal gradient value Gx(i,j) obtained by convolving the horizontal Sobel kernel with the 3x3 neighborhood centered on (i,j) in the image; the vertical gradient Gy is the vertical gradient value Gy(i,j) obtained by convolving the vertical Sobel kernel with the 3x3 neighborhood centered on (i,j) in the image; for each pixel position, the absolute values ​​of the horizontal and vertical gradients are calculated, namely |Gx(i,j)| and |Gy(i,j)|.

[0052] The Sobel operator has two 3x3 convolution kernels, one for detecting horizontal edges (Gx) and the other for detecting vertical edges (Gy).

[0053] Among them, the Sobel kernel in the horizontal direction (Gx) is:

[0054]

[0055] The Sobel kernel in the vertical direction (Gy) is:

[0056]

[0057] The horizontal Sobel kernel is convolved with the 3x3 neighborhood centered at (i, j) in the image to obtain the horizontal gradient value Gx(i, j) at that position. The vertical Sobel kernel is convolved with the 3x3 neighborhood centered at (i, j) in the image to obtain the vertical gradient value Gy(i, j) at that position.

[0058] Among them, with pixel (i, j) as the center, the 3x3 pixel neighborhood around it includes 9 pixels in the range of (i-1, j-1) to (i+1, j+1).

[0059] S104, identifying the image with the best picture quality to obtain text information.

[0060] See also Figure 2 The present invention discloses a system for converting video into text, comprising:

[0061] A collection module, wherein the collection module collects video information containing book content and pre-processes the video information;

[0062] A conversion module, wherein the conversion module converts the preprocessed video frame by frame to obtain a plurality of pictures;

[0063] A screening module, wherein the screening module screens the acquired pictures to obtain an image with the best picture quality within a unit timestamp;

[0064] A recognition module is used to recognize the image with the best picture quality and obtain text information.

[0065] Example:

[0066] The present invention discloses a method for converting video into text, comprising:

[0067] Turn the pages of the book one by one, and turn on the mobile phone camera to record the turning of the pages of the book; upload the recorded video to the server, and the server uses Python to take screenshots of the received video frame by frame, filter the obtained screenshots, and identify the book text information contained in each picture. If the book text information does not exist in the picture, the picture will be deleted, and the remaining pictures are pictures containing book text information.

[0068] Identify the timestamp information on each image, set the length of the unit timestamp, treat the images in the unit timestamp as an image set, and determine whether the image content in the unit timestamp is exactly the same; if not, compare the percentage of different images in the unit timestamp, treat the images with a smaller percentage as abnormal images, compare the abnormal images with the images in other unit timestamps in turn, and add them to the unit timestamp containing the image. Treat the images in each unit timestamp as an image set, identify the text information contained in the images in the image set, and delete the corresponding images when the accuracy of the identified text is less than the set accuracy threshold. Crop the boundaries of the filtered images and identify the page numbers of the images. Sort the images from small to large according to the size of the page, convert the screenshot JPG format into PDF format, and make the PDF format into a book; perform text recognition on the book and convert it into word and other formats and send it to the client for download. Save the received video on the server, convert the video, and then send it to the APP for download.

[0069] The terminal device provided in an embodiment of the present invention. The terminal device of this embodiment includes: a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps in the above-mentioned method embodiments are implemented. Alternatively, when the processor executes the computer program, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0070] The computer program may be divided into one or more modules / units, and the one or more modules / units are stored in the memory and executed by the processor to accomplish the present invention.

[0071] The terminal device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0072] The processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0073] The memory may be used to store the computer programs and / or modules, and the processor implements various functions of the terminal device by running or executing the computer programs and / or modules stored in the memory and calling the data stored in the memory.

[0074] If the module / unit integrated in the terminal device is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0075] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for converting video into text, characterized in that: include: Collect video information containing book content and pre-process the video information; Convert the preprocessed video frame by frame to obtain several pictures; Filter the acquired pictures to obtain the image with the best picture quality within a unit timestamp; Recognize the image with the best picture quality and obtain text information.

2. The method for converting video into text according to claim 1, characterized in that: The preprocessing of the video information specifically includes: adjusting the bit rate, resolution and frame rate of the collected video to improve the clarity of the video; The pre-processed video is converted frame by frame to obtain a number of pictures, specifically: based on FFmpeg multimedia processing software, the frame rate and picture quality are set, and the images in the video are extracted frame by frame.

3. The method for converting video into text according to claim 1, characterized in that: The obtained pictures are screened, specifically, as follows: identifying the timestamp information on each picture, setting the length of the unit timestamp, treating the pictures in the unit timestamp as an image set, and determining whether the contents of the pictures in the unit timestamp are completely the same; if not, comparing the percentages of different pictures in the pictures in the unit timestamp, treating the pictures with a smaller percentage as abnormal pictures, comparing the abnormal pictures with the pictures in other unit timestamps in turn, and adding them to the unit timestamp containing the picture.

4. The method for converting video into text according to claim 3, characterized in that: Before identifying the timestamp information on each picture, the method further includes: identifying the book text information contained in each picture. If the book text information does not exist in the picture, the picture is deleted, and the remaining pictures are pictures containing the book text information.

5. The method for converting video into text according to claim 4, characterized in that: The method of obtaining the image with the best picture quality within a unit timestamp is specifically as follows: the images within each unit timestamp are regarded as an image set, the text information contained in the images of the image set is identified, and when it is determined that the accuracy of the identified text is less than a set accuracy threshold, the corresponding image is deleted.

6. A method for converting video into text according to claim 5, characterized in that: The step of obtaining the image with the best picture quality within the unit timestamp further includes: obtaining the clarity of the image based on a Sobel operator, and taking the image with a clarity that does not exceed a clarity threshold and is ranked high as the image with the best picture quality; The method of obtaining the clarity of the image based on the Sobel operator is specifically as follows: based on the convolution kernel of the Sobel operator and each pixel position of the image, the absolute values ​​of the horizontal and vertical gradients are calculated; the absolute values ​​of the horizontal and vertical gradients of each pixel position are added to obtain the sum of the gradients at that position; the larger the value of the gradient sum is, the higher the clarity of the image is.

7. The method for converting video into text according to claim 6, characterized in that: The convolution kernel based on the Sobel operator and each pixel position of the image calculates the absolute values ​​of the horizontal and vertical gradients, specifically: the horizontal gradient Gx is the horizontal gradient value Gx(i,j) of the position obtained by convolving the Sobel kernel in the horizontal direction with the 3x3 neighborhood centered on (i,j) in the image; the vertical gradient Gy is the vertical gradient value Gy(i,j) of the position obtained by convolving the Sobel kernel in the vertical direction with the 3x3 neighborhood centered on (i,j) in the image; for each pixel position, the absolute values ​​of the horizontal and vertical gradients are calculated, i.e., |Gx(i,j)| and |Gy(i,j)|.

8. A system for converting video into text, characterized in that: include: A collection module, wherein the collection module collects video information containing book content and pre-processes the video information; A conversion module, wherein the conversion module converts the preprocessed video frame by frame to obtain a plurality of pictures; A screening module, wherein the screening module screens the acquired pictures to obtain an image with the best picture quality within a unit timestamp; A recognition module is used to recognize the image with the best picture quality and obtain text information.

9. A terminal device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images