Video detection method and apparatus, electronic device, and storage medium

Abstract

The application provides a video detection method, device, electronic equipment and storage medium, the method comprises: performing page turning action detection on a to-be-detected video to obtain an image sequence of each page in the to-be-detected video in a non-page turning action; determining a representative image of each page from each image based on visual features and frequency domain features of each image in the image sequence and semantic features of recognized text in each image; and performing image processing based on the representative image of each page. The method, device, electronic equipment and storage medium provided by the application determine the representative image of each page from the image sequence of each page obtained by page turning action detection, perform image processing based on the representative image, realize page turning detection in a full-automatic process, can adapt to a large number of page turning action detection scenarios, and ensure the reliability and rationality of selection of the representative image of each page, thereby ensuring the reliability of image processing on the page content contained in the to-be-detected video.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and in particular to a video detection method and device, electronic equipment and storage medium. BACKGROUND

[0002] Currently, page turning detection is performed on a video scene to obtain pages with different contents. The page turning detection is mainly based on manual clicking to confirm a page turning button, determine the images corresponding to the pages before and after page turning, and perform subsequent image processing to realize functions such as character recognition and topic cataloging.

[0003] However, the above-mentioned page turning detection method requires a large amount of manual operation and is not suitable for tasks requiring a large number of page turning actions. In addition, the page images confirmed manually may have poor image quality, and the reliability of subsequent image processing is not high enough. SUMMARY

[0004] The present application provides a video detection method and device, electronic equipment and storage medium to solve the defects of manual page turning detection in the prior art, which is complicated and has low reliability.

[0005] The present application provides a video detection method, comprising:

[0006] detecting a page turning action in a to-be-detected video to obtain an image sequence of each page in the to-be-detected video in a non-page turning action;

[0007] determining a representative image of each page from the images based on visual features and frequency domain features of the images and semantic features of recognized text in the images;

[0008] performing image processing based on the representative images of each page.

[0009] According to the video detection method provided by the present application, the representative image of each page is determined from the images based on the visual features and frequency domain features of the images and the semantic features of the recognized text in the images, comprising:

[0010] determining a deep feature of each image and a hidden layer feature of the image sequence based on the visual features, frequency domain features and semantic features of the images in the image sequence;

[0011] determining the representative image of each page from the images based on the similarity between the deep features of the images and the hidden layer feature of the image sequence.

[0012] According to the video detection method provided in the application, the deep features of each image in the image sequence are determined based on the visual features, frequency domain features and semantic features of each image in the image sequence, and the hidden layer features of the image sequence are determined.

[0013] The deep features of each image in the image sequence are determined based on the visual features, frequency domain features and semantic features of each image in the image sequence, and the hidden layer features of the image sequence are determined based on the text line position features of the recognized text in each image.

[0014] According to the video detection method provided in the application, the deep features of each image in the image sequence are determined based on the visual features, frequency domain features and semantic features of each image in the image sequence, and the hidden layer features of the image sequence are determined based on the text line position features of the recognized text in each image, comprising:

[0015] The fusion features of each image are obtained by fusing the visual features, frequency domain features, semantic features and text line position features of each image;

[0016] The deep features of each image are obtained by performing feature extraction on the fusion features of each image.

[0017] The hidden layer features of the image sequence are obtained by performing sequence encoding on the image sequence based on the fusion features of each image.

[0018] According to the video detection method provided in the application, the flipping action detection is performed on the to-be-detected video to obtain the image sequence of each page in the to-be-detected video in a non-flipping action, comprising:

[0019] The first preset number of characters of each text line in each image in the to-be-detected video are recognized to obtain each text line in each image in the to-be-detected video.

[0020] The matching is performed on each text line in adjacent images in the to-be-detected video to obtain the matching result of adjacent images.

[0021] Based on the matching result of adjacent images in the to-be-detected video, the images in the to-be-detected video in a non-flipping action are determined, and based on the images in a non-flipping action, the image sequence of each page is determined.

[0022] According to the video detection method provided in the application, the matching is performed on each text line in adjacent images in the to-be-detected video to obtain the matching result of adjacent images, comprising:

[0023] The matching is performed on each text line in one image and each text line in another image in the adjacent images to obtain the line matching result of each text line in the adjacent images.

[0024] Determine the matching result of the adjacent images based on the line matching result of each line of text.

[0025] According to the video detection method provided in the application, the matching result of the adjacent images is determined based on the line matching result of each line of text, which includes:

[0026] Determine the number of lines of text that match successfully based on the line matching result of each line of text.

[0027] Determine the matching result of the adjacent images based on the number of lines of text that match successfully and the number of lines of text of each adjacent image.

[0028] The application further provides a video detection device, which includes:

[0029] A page turning action detection unit detects the page turning action of the video to be detected to obtain the image sequence of each page in the video to be detected under non-page turning action;

[0030] A representative image determination unit determines the representative image of each page from each image based on the visual feature and frequency domain feature of each image and the semantic feature of the recognized text in each image.

[0031] An image processing unit processes the image based on the representative image of each page.

[0032] The application further provides an electronic device, which includes a camera, a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the video detection method as described above when executing the program.

[0033] The application further provides a non-transitory computer readable storage medium, which stores a computer program executable on a processor to implement the video detection method as described above.

[0034] The application further provides a computer program product, which includes a computer program executable on a processor to implement the video detection method as described above.

[0035] The video detection method, device, electronic equipment and storage medium provided by the application are used for detecting the image sequence of each page obtained by a page turning action, determining a representative image of each page based on the visual features and frequency domain features of each image in the image sequence and the semantic features of the recognized text in each image, performing image processing based on the representative images of each page, realizing page turning detection in a full-automatic process, without manual participation, being able to adapt to a large number of page turning action detection scenes, and ensuring the reliability and rationality of the selection of the representative images of each page, and further ensuring the reliability of image processing on the page content contained in the video to be detected. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort on the basis of these drawings.

[0037] Figure 1 is one of the flowcharts of the video detection method provided by the application;

[0038] Figure 2 is the flowchart of determining the representative image of each page provided by the application;

[0039] Figure 3 is the flowchart of determining the depth features of each image and the hidden layer features of the image sequence provided by the application;

[0040] Figure 4 is the flowchart of determining the image sequence of each page under a non-page turning action provided by the application;

[0041] Figure 5 is the second flowchart of the video detection method provided by the application;

[0042] Figure 6 is the structural schematic diagram of the video detection device provided by the application;

[0043] Figure 7 is the structural schematic diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0045] The current page turning detection method needs a large amount of manual operation, is not suitable for tasks that need a large number of page turning actions, and the page images confirmed manually may have the problem of poor image quality, and the reliability of subsequent image processing is not high enough.

[0046] In view of the above problems, Figure 1 One of the flowcharts of the video detection method provided by the present application is shown in Figure 1 The method comprises the following steps.

[0047] In step 110, page turning action detection is performed on the to-be-detected video to obtain image sequences of each page in the to-be-detected video in a non-page turning action state.

[0048] Specifically, the to-be-detected video is a video obtained by real-time shooting in the process of a user performing a page turning action, and the to-be-detected video can contain one or more page turning actions. For one of the page turning actions, the images related to it in the to-be-detected video contain images of the current page content, images of the page turning action and images of the next page content.

[0049] When image processing is performed on the page content contained in the to-be-detected video, the images that are really needed to be used are the images other than the page turning action. It can be understood that in the to-be-detected video, valuable page content, i.e. text and / or image information, cannot be extracted from the images in the page turning action. Therefore, before image processing is performed on the page content contained in the to-be-detected video, the images in the page turning action in the to-be-detected video need to be detected, the image sequence of the page turning action is stripped from the to-be-detected video, and the remaining image sequence is obtained as the image sequence of each page. When image processing needs to be performed on the to-be-detected video, the image sequence or a certain representative image of each page in the to-be-detected video in a non-page turning action state is processed.

[0050] Here, the page turning action detection can be implemented by relying on the similarity between adjacent images in the video to be detected. It can be understood that the higher the similarity between adjacent images, the lower the probability that the adjacent images are under the page turning action, and vice versa. The lower the similarity between adjacent images, the higher the probability that the adjacent images are under the page turning action. The similarity between adjacent images can be reflected as image similarity or as the similarity of the text contained in the images. Thus, when calculating the similarity, image feature extraction can be performed on two adjacent images respectively, and then the extracted image features are applied to calculate the similarity; or text recognition can be performed on the adjacent images respectively, and then semantic feature extraction is performed on the recognized text, and the extracted semantic features are applied to calculate the similarity. The embodiments of the present application do not make specific limitations on this.

[0051] Step 120, determining the representative image of the page from each image in the image sequence based on the visual features and frequency domain features of each image in the image sequence and the semantic features of the recognized text in each image;

[0052] Here, the image sequence of each page obtained in step 110 includes one or more image sequences of the page. It can be understood that the page and the image sequence are one-to-one corresponding, i.e., one page corresponds to one image sequence.

[0053] For any one page, the image sequence of the page usually contains multiple repeated images of the page, and the image quality of these repeated images is uneven. If image processing is performed on each of these repeated images, not only multiple resources are consumed to perform content extraction, but also the completeness and accuracy of the extracted content are not high, thereby resulting in low reliability of image processing. Therefore, in order to improve the reliability of image processing of the images corresponding to the page content contained in the video to be detected, the high-quality images of each page, i.e., the representative images of each page, can be determined from the image sequences of each page under the non-page turning action. The reliability of image processing of the representative images of each page is higher than that of the other repeated images of each page.

[0054] Further, for any page, the representative image in the image sequence of the page can be determined by the visual features of each image in the image sequence, the frequency domain features of each image, and the semantic features of the recognized text in each image. Here, the frequency domain features of the image can be the frequency domain spectrum obtained by Fourier transform of the image, or the features obtained by further feature extraction based on Fourier transform. For example, the frequency domain spectrum can be fed into a visual pre-training model such as ImageNet to obtain. The frequency domain features reflect the overall sharpness of the image. For example, the frequency domain spectrum obtained by Fourier transform can reflect the distribution of frequency components of the image. The more high-frequency components, the more likely it is a clear area. Conversely, the more low-frequency components, the more likely it is a fuzzy area.

[0055] Through the frequency domain features of the image, images with high overall sharpness can be screened out. However, when performing image processing on the video to be detected, the image content needs to be extracted, and the text information or image information in the image needs to be clear. Therefore, the visual features of the image can be used to determine the local sharpness of the image. Here, the visual features of the image reflect the local sharpness of the image, which can be obtained by visual feature extraction of the image. Specifically, the image can be fed into a visual pre-training model such as ImageNet to obtain. It can be understood that the higher the local sharpness of the image, the higher the accuracy and completeness of the extracted page content, and the higher the probability of being a representative image.

[0056] In addition, by determining the overall sharpness of the image and the sharpness of a certain part of the image, it is ensured that image processing can be performed on a representative image with high quality. However, in order to further ensure the accuracy and completeness of the extracted content, it is possible that a certain part of the image is clear enough, but due to the problem of the video recording angle, part of the image is missing, such as partial text content missing. To solve this problem, the semantic features of the recognized text in the image can also be used to determine. The semantic features of the recognized text can reflect whether the text content in the image is complete. Specifically, the semantic features of the recognized text can be extracted by using a semantic extraction model such as BERT (Bidirectional Encoder Representation from Transformers) on the recognized text in the image. It can be understood that the image with more complete text semantics reflected by the semantic features of the recognized text has a higher probability of being a representative image compared to other images in the image sequence of this page.

[0057] Step 130, based on the representative image of each page, performing image processing.

[0058] Specifically, the representative image of each page obtained by image selection from the image sequence of each page in step 120 is an image whose overall clarity and local clarity are higher than those of other images in the image sequence of each page, and the semantic content of the recognized text contained in the image is also higher than that of other images. After obtaining the high-quality representative image, it can be used as the object of image processing to realize the subsequent process of page turning detection. The image processing here can be target detection on the image to realize the point reading function, or text recognition on the image, or used as the object of other image processing methods, which is not limited in the present application.

[0059] The method provided by the embodiment of the present application is based on the visual features and frequency domain features of each image in the image sequence of each page obtained by page turning action detection, and the semantic features of the recognized text in each image, to determine the representative image of each page from each image, and based on the representative image of each page, to perform image processing, realizing page turning detection in a fully automatic process without human intervention, which can adapt to a large number of page turning action detection scenarios, and ensures the reliability and rationality of the selection of the representative image of each page, and further ensures the reliability of image processing on the page content contained in the video to be detected.

[0060] Based on any of the above embodiments, Figure 2 The flowchart for determining the representative image of each page provided by the present application is shown in Figure 2 As shown in step 120, comprising:

[0061] Step 210, based on the visual features, frequency domain features and semantic features of each image in the image sequence, determining the depth features of each image and the hidden layer features of the image sequence;

[0062] Specifically, after obtaining the visual features, frequency domain features and semantic features of each image in the image sequence of any page, the depth features of each image in the image sequence of the page and the hidden layer features of the entire image sequence can be obtained based on this. It can be understood that the depth features here are for a single image, and the depth features of a single image can be obtained by splicing or other feature fusion of the visual features, frequency domain features and semantic features of the single image, or further feature extraction can be performed on the fused features. It can be understood that the depth features of the image fuse the features that can reflect the overall clarity of the image, the clarity of the local image and the semantic completeness of the recognized text in the image, and the features are strengthened to represent the key part of the image. The clarity and text completeness information of the image can be represented by the depth features of the image.

[0063] In addition, the hidden layer feature herein refers to an overall feature of the image sequence of the page, which can reflect the overall definition of the images of the entire image sequence, the definition of the local images, and the feature of the semantic completeness of the recognized text embodied in the image sequence level, and can be obtained by encoding the image sequence of the page through a neural network structure such as an LSTM (Long Short Term Memory) model or an RNN (Recurrent Neural Network).

[0064] In step 220, the representative image of the page is determined from the images based on the similarity between the depth feature of each image and the hidden layer feature of the image sequence.

[0065] Specifically, after obtaining the depth feature of each image in the image sequence and the hidden layer feature of the image sequence, the similarity between the depth feature of each image and the hidden layer feature of the image sequence can be calculated, respectively. Here, the greater the similarity between the depth feature of any image and the hidden layer feature of the image sequence, the more the features of the image conform to the features of the entire image sequence, and thus the image is more likely to be the representative image in the image sequence of the page. The similarity can be calculated through the following formula:

[0066]

[0067] wherein h i represents the hidden layer feature of the image sequence corresponding to the i th page, v j represents the depth feature of the j th image in the image sequence, represents the similarity between the hidden layer feature of the image sequence corresponding to the i th page and the depth feature of the j th image in the image sequence.

[0068] The method provided by the embodiment of the application determines the depth feature of each image and the hidden layer feature of the image sequence through the visual feature, the frequency domain feature, and the semantic feature of the recognized text of each image in the image sequence, and selects the image with the depth feature most similar to the features of the image sequence at the sequence level as the representative image by calculating the similarity between the depth feature of the image and the hidden layer feature of the image sequence, thereby improving the reliability of image processing through the representative image.

[0069] The visual features, frequency domain features and semantic features of the image can basically reflect the basic quality information of the image, but there can still be cases where the text content is missing without affecting the overall text semantic, that is, the text content can be complete in terms of semantics, but there are missing characters, especially at the corners of the page, so the reliability of judging the completeness of the text only based on the semantic features of the text needs to be improved.

[0070] To solve the above problems, based on any of the above embodiments, step 210 comprises:

[0071] Based on the visual features, frequency domain features and semantic features of each image in the image sequence, and the text line position features of the recognized text in each image, the depth features of each image and the hidden layer features of the image sequence are determined.

[0072] Specifically, to further improve the completeness evaluation of the text in the image and further improve the reliability of image processing, the completeness information of the recognized text can also be further determined according to the position information of the text in the image. Specifically, the Position Embedding algorithm can be used to encode the position of each line of text in the image to obtain the text line position features of the text. It can be understood that by using the visual features, frequency domain features and semantic features of each image, and the text line position features of the recognized text in each image, the depth features of each image and the hidden layer features of the image sequence are determined, which has a wider range and can more reliably filter out images close to the content of the page.

[0073] Based on any of the above embodiments, Figure 3 The flowchart for determining the depth features of the image and the hidden layer features of the image sequence provided by the present application is shown in Figure 3 As shown in step 210, based on the visual features, frequency domain features and semantic features of each image in the image sequence, and the text line position features of the recognized text in each image, the depth features of each image and the hidden layer features of the image sequence are determined, which comprises:

[0074] Step 310, fusing the visual features, frequency domain features, semantic features and text line position features of each image to obtain the fusion features of each image;

[0075] Specifically, the obtained visual features, frequency domain features, semantic features and text line position features of each image can be fused. The fusion method here can be direct splicing or weighting of the obtained features. It can be understood that the fusion features here are for a single image and contain information about the overall sharpness of the image, the sharpness of the local image and the semantic completeness of the text in the image, which can fully reflect the quality of the image itself.

[0076] Step 320, feature extraction is performed on the fusion features of the images to obtain deep features of the images.

[0077] Specifically, after obtaining the fusion features of the images, further feature extraction can be performed to obtain the deep features of the images. It can be understood that the deep features strengthen the representation of the information of the images on the basis of the fusion features, and can more specifically reflect the features of the images in the quality level. The feature extraction can be implemented by a CNN model, or can be implemented by other types of neural network models.

[0078] After determining the fusion features of the images, the hidden layer features of the image sequence corresponding to the images also need to be determined, which can be implemented by step 330.

[0079] Step 330, sequence encoding is performed on the image sequence based on the fusion features of the images to obtain hidden layer features of the image sequence.

[0080] Specifically, after obtaining the fusion features of the images, sequence encoding can be performed on the image sequence based on the fusion features of the images to obtain the hidden layer features of the image sequence, for example, the image sequence can be encoded by an LSTM model.

[0081] It can be understood that the deep features of the images and the hidden layer features of the corresponding image sequence are determined based on the fusion features of the images, and there is no specified order, and the deep features of the images or the hidden layer features of the corresponding image sequence can be determined in any priority or in parallel.

[0082] Based on any of the above embodiments, Figure 4 A flowchart for determining the image sequence of each page under the non-page turning action is provided in the present application, as shown in Figure 4 Step 110 includes:

[0083] Step 410, the first predetermined number of characters in each text line in each image in the to-be-detected video are recognized to obtain each text line in each image in the to-be-detected video.

[0084] Specifically, considering that the text lines and the number of texts in each image in the video to be detected are relatively large, and the characters in each text line are basically not repeated, the first preset number of characters in each text line in each image can be used to represent each text line in each image in the video to be detected. The preset number can be determined according to the average number of characters in each text line in the image. It can be understood that the greater the average number of characters in each text line, the greater the preset number corresponding to the text line. Alternatively, the preset number can be fixed, for example, the preset number can be fixed at 2 or 3, that is, only the first two or three characters in each line are recognized.

[0085] It can be understood that, compared with recognizing the complete text line, only recognizing the first preset number of characters in each text line can greatly reduce the amount of text recognition and improve the efficiency of text recognition.

[0086] Step 420, matching each text line in adjacent images in the video to be detected to obtain a matching result of the adjacent images;

[0087] In the video to be detected, adjacent images can correspond to the same page or different pages in a page turning action. Therefore, after obtaining each text line in each image in the video to be detected, each text line in adjacent images can be matched to obtain a matching result of the adjacent images. It can be understood that the higher the matching degree of the adjacent images, the greater the possibility that the adjacent images correspond to the same page, and the lower the matching degree, the smaller the possibility that the adjacent images correspond to the same page.

[0088] In this process, since only the first preset number of characters in each text line obtained by recognition is used for matching each text line, the amount of matching data of each text line is also limited, and the matching efficiency of each text line is ensured.

[0089] Step 430, determining images in a non-page turning action in the video to be detected based on the matching result of the adjacent images in the video to be detected, and determining an image sequence of each page based on the images in the non-page turning action.

[0090] Specifically, based on the matching results of the adjacent images of the to-be-detected video, it can be determined whether the adjacent images are images of the same page, and then the adjacent images of the same page are put into the image sequence of the corresponding page, and the adjacent images of different pages are determined as images under non-page turning actions. For example, it is determined through matching that the matching results of the Nth image and the N+1th image, the N+1th image and the N+2th image, and the N+2th image and the N+3th image are all images of non-same text, so it can be determined that the N+1th to N+3th images are images under page turning actions, and then the 1st to Nth images are an image sequence of the same page, and the N+4th image to the next image of non-same text are an image sequence of another page.

[0091] The method provided by the embodiment of the present application can obtain each text line in each image in the to-be-detected video by recognizing a preset number of characters in each text line in each image in the to-be-detected video, so that the number of texts for matching is reduced as much as possible, the calculation amount required for matching is reduced, and the matching speed of each text line is improved. The matching of each text line in adjacent images in the to-be-detected video is performed to obtain the matching result of the adjacent images, the images under non-page turning actions in the to-be-detected video are determined based on the matching result of the adjacent images in the to-be-detected video, and the image sequence of each page is determined based on the images under non-page turning actions, thereby improving the reliability of image processing of each page in the to-be-detected video.

[0092] Based on any of the above embodiments, step 420 comprises:

[0093] The matching of each text line in one image in the adjacent images with each text line in another image is performed to obtain the line matching result of each text line in the adjacent images.

[0094] The matching result of the adjacent images is determined based on the line matching result of each text line.

[0095] Specifically, for two images in adjacent images, the line matching result of each text line of the two images can be obtained by matching each text line in one image with each text line in another image. For example, the line matching result of a text line in adjacent images can be obtained by matching any text line in one image with each text line in another image. The line matching result of each text line can be the result of matching all text lines including the text line with the same line number and the text line with a non-same line number in another image, and the matching of the text line is successful only when any one of the text lines is matched.

[0096] Then, according to the line matching results of the lines of text of the adjacent images, the matching results of the adjacent images can be determined. Generally, there can be a case that part of the text in the images is blurred, and a preset value can be determined as the percentage of the number of lines of text with matching success in the total number of lines of text of the adjacent images. If the percentage is greater than the preset value, it can be determined that the matching result of the adjacent images is the images with the same text; if the percentage is less than the preset value, it can be determined that the matching result of the adjacent images is the images with non-same text.

[0097] Based on any of the above embodiments, the matching results of the adjacent images are determined based on the line matching results of the lines of text, including:

[0098] Based on the line matching results of the lines of text, the number of lines of text with matching success is determined.

[0099] Based on the number of lines of text with matching success and the number of lines of text of the adjacent images respectively, the matching results of the adjacent images are determined.

[0100] Specifically, the number of lines of text with matching success of the adjacent images can be calculated through the line matching results of the lines of text. It can be understood that the greater the number of lines of text with matching success of the adjacent images, the more likely the matching result of the adjacent images is the images with the same text; and the smaller the number of lines of text with matching success of the adjacent images, the more likely the matching result of the adjacent images is the images with non-same text.

[0101] In order to avoid the matching error of the adjacent images due to the different number of lines of text contained in different pages, the number of lines of text of the two images in the adjacent images is additionally counted. The maximum value of the number of lines of text of the two images or the average value of the number of lines of text of the two images can be selected as the line number reference value. The ratio of the number of lines of text with matching success to the line number reference value is used as the basis for measuring the matching result of the adjacent images, thereby improving the reliability of the matching.

[0102] Based on any of the above embodiments, Figure 5 The flowchart of the video detection method provided by the present application is shown in Figure 2, which includes: Figure 5 as shown, including:

[0103] In step 510, the page turning action detection of the video to be detected is performed to obtain the image sequence of each page in the video to be detected under the non-page turning action;

[0104] Step 510 can be implemented by the following steps: first, identifying the first preset number of characters of each text line in each image in the to-be-detected video as each text line in each image in the to-be-detected video. Specifically, text line detection can be performed on each image, and text recognition is performed on each text line. To improve the efficiency of text line detection, only two characters are decoded for each text line, and the two characters represent the entire text line.

[0105] Next, the text lines in adjacent images in the to-be-detected video are matched to obtain a matching result of the adjacent images. Specifically, one text line is taken from each of the adjacent images, respectively. If the texts of the two text lines are the same, the matching number is increased by one. After traversing all the text lines of the adjacent images, the larger one of the matching number and the number of text lines of the adjacent images is divided, and if the result is greater than 0.7, it is considered that the adjacent images belong to the same text.

[0106] Finally, based on the matching result of the adjacent images in the to-be-detected video, images in the to-be-detected video under non-page turning action are determined, and based on the images under non-page turning action, image sequences of each page are determined.

[0107] Through step 510, it can be determined which images correspond to the same page content. The core goal of the video detection method provided by the application is to select the best image from the image sequences of the same page content as the input of the subsequent image processing task. The subsequent task can be used for text recognition, image correction, etc.

[0108] Step 520, based on the visual features, frequency domain features and semantic features of each image in the image sequence, and the text line position features of the recognized text in each image, determine the deep features of each image and the hidden layer features of the image sequence;

[0109] In step 520, first, the visual features, frequency domain features, semantic features and text line position features of each image are obtained, and each feature is spliced and fused to obtain the fusion features of each image.

[0110] Next, feature extraction is performed on the fusion features of each image to obtain the deep features of each image. Based on the fusion features of each image, sequence encoding is performed on the image sequence to obtain the hidden layer features of the image sequence. It can be understood that there is no specific order relationship between obtaining the deep features of each image and the hidden layer features of the image sequence.

[0111] Step 530, based on the similarity between the deep features of each image and the hidden layer features of the image sequence, determine the representative image of the page from each image sequence;

[0112] The representative image of the page can be determined through a representative image generation model based on deep learning of multi-modal features. The representative image generation model can be obtained through pre-training. The pre-training process can include the following steps. First, sample data is collected, and image features of representative images of pages corresponding to the sample data are extracted through One Hot coding and manual annotation. Then, according to the similarity between the deep features of each image and the hidden layer features of the image sequence, and the difference between the image features of the representative images obtained through manual annotation, the loss function is used to iteratively update the parameters of the training model, and finally the representative image generation model is obtained.

[0113] In step 540, image processing is performed based on the representative images of the pages.

[0114] According to any of the above embodiments, Figure 6 is a structural schematic diagram of a video detection device provided by the present application, as Figure 6 shown, the device comprises:

[0115] The page-turning action detection unit 610 detects the page-turning action of the video to be detected to obtain an image sequence of each page in a non-page-turning action in the video to be detected.

[0116] The representative image determination unit 620 determines a representative image of the page based on the visual features and frequency domain features of each image in the image sequence, and the semantic features of the recognized text in each image.

[0117] The image processing unit 630 performs image processing based on the representative images of the pages.

[0118] The device provided by the embodiment of the present application is used for the image sequence of each page obtained through page-turning action detection. The representative image of the page is determined from each image based on the visual features and frequency domain features of each image in the image sequence, and the semantic features of the recognized text in each image. Image processing is performed based on the representative images of the pages. The page-turning detection in the full-automatic process is realized without human intervention. The detection scene of a large number of page-turning actions can be adapted. The reliability and rationality of the selection of the representative images of the pages are ensured, and the reliability of the image processing of the page content contained in the video to be detected is ensured.

[0119] According to any of the above embodiments, the representative image determination unit is specifically configured to:

[0120] Determine the deep features of each image and the hidden layer features of the image sequence based on the visual features, frequency domain features and semantic features of each image in the image sequence.

[0121] determine the representative image of the page from the images based on similarity between the depth features of the images and the hidden layer features of the image sequence.

[0122] According to any one of the above embodiments, the representative image determination unit is specifically configured to:

[0123] determine the depth features of the images and the hidden layer features of the image sequence based on the visual features, frequency domain features and semantic features of the images in the image sequence, and the text line position features of the recognized texts in the images.

[0124] According to any one of the above embodiments, the representative image determination unit is specifically configured to:

[0125] fuse the visual features, frequency domain features, semantic features and text line position features of the images to obtain fusion features of the images;

[0126] perform feature extraction on the fusion features of the images to obtain the depth features of the images;

[0127] perform sequence coding on the image sequence based on the fusion features of the images to obtain the hidden layer features of the image sequence.

[0128] According to any one of the above embodiments, the page turning action detection unit is specifically configured to:

[0129] recognize the first preset number of characters of each text line in each image in the to-be-detected video to obtain each text line in each image in the to-be-detected video;

[0130] match the text lines in adjacent images in the to-be-detected video to obtain a matching result of the adjacent images;

[0131] determine the images in the to-be-detected video that are in a non-page turning action based on the matching result of the adjacent images in the to-be-detected video, and determine the image sequence of each page based on the images in the non-page turning action.

[0132] According to any one of the above embodiments, the page turning action detection unit is specifically configured to:

[0133] match each text line in one image of the adjacent images with each text line in another image to obtain a line matching result of the text lines in the adjacent images;

[0134] determine the matching result of the adjacent images based on the line matching result of the text lines.

[0135] According to any one of the above embodiments, the page turning action detection unit is specifically configured to:

[0136] determine a number of line texts matched successfully based on the line matching result of each line text;

[0137] determine a matching result of the adjacent images based on the number of line texts matched successfully and the number of line texts of each of the adjacent images.

[0138] Figure 7 An example of a schematic diagram of an entity structure of an electronic device is shown in Figure 7 As shown in the figure, the electronic device can include a processor 710, a communications interface 720, a memory 730, and a communications bus 740, wherein the processor 710, the communications interface 720, and the memory 730 complete mutual communication through the communications bus 740. The processor 710 can invoke a logical instruction in the memory 730 to execute a video detection method, which includes: performing page turning action detection on a to-be-detected video to obtain an image sequence of each page in the to-be-detected video in a non-page turning action; determining a representative image of each page from each image in the image sequence based on visual features and frequency domain features of each image and semantic features of recognized text in the image; and performing image processing based on the representative image of each page.

[0139] In addition, the logical instruction in the memory 730 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0140] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the video detection method provided by any of the above methods, which comprises: performing page turning action detection on a to-be-detected video to obtain image sequences of each page in the to-be-detected video in a non-page turning action state; determining a representative image of each page from the images in the image sequences based on visual features and frequency domain features of each image and semantic features of recognized text in the images; and performing image processing based on the representative images of each page.

[0141] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, enables a computer to perform the video detection method provided by any of the above methods, which comprises: performing page turning action detection on a to-be-detected video to obtain image sequences of each page in the to-be-detected video in a non-page turning action state; determining a representative image of each page from the images in the image sequences based on visual features and frequency domain features of each image and semantic features of recognized text in the images; and performing image processing based on the representative images of each page.

[0142] The device embodiments described above are merely illustrative, wherein the units illustrated as separate components can or can not be physically separated, and the components illustrated as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected to achieve the purpose of the embodiment according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0143] From the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0144] It should be pointed out finally that the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit the same; and although the present application has been described in detail with reference to the foregoing embodiments, it should be appreciated by those skilled in the art that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features thereof can be replaced equivalently; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A video detection method, characterized in that, include: Page-turning motion detection is performed on the video to be detected to obtain the image sequence of each page in the video to be detected that is not in a page-turning motion. Based on the visual features, frequency domain features, and semantic features of the text identified in each image in the image sequence, the depth features of each image and the hidden layer features of the image sequence are determined. Based on the similarity between the depth features of each image and the hidden layer features of the image sequence, the representative image of the page is determined from each image; Image processing is performed based on representative images from each page.

2. The video detection method according to claim 1, characterized in that, The step of determining the depth features of each image and the hidden layer features of the image sequence based on the visual features, frequency domain features, and semantic features of the recognized text in each image in the image sequence includes: Based on the visual features, frequency domain features, and semantic features of each image in the image sequence, as well as the text line position features of the identified text in each image, the depth features of each image and the hidden layer features of the image sequence are determined.

3. The video detection method according to claim 2, characterized in that, The determination of the depth features of each image and the hidden layer features of the image sequence based on the visual features, frequency domain features, and semantic features of each image in the image sequence, as well as the text line position features of the identified text in each image, includes: By fusing the visual features, frequency domain features, semantic features, and text line position features of each image, the fused features of each image are obtained. Feature extraction is performed on the fusion features of the images to obtain the depth features of each image; Based on the fusion features of each image, the image sequence is sequence encoded to obtain the hidden layer features of the image sequence.

4. The video detection method according to any one of claims 1 to 3, characterized in that, The step involves detecting page-turning actions in the video to be detected, resulting in an image sequence of each page in the video that is not undergoing a page-turning action, including: Identify the first preset number of characters of each text line in each image of the video to be detected to obtain the text of each line in each image of the video to be detected; The text in each line of adjacent images in the video to be detected is matched to obtain the matching results of adjacent images; Based on the matching results of adjacent images in the video to be detected, the images in the video to be detected that are not in a page-turning state are determined, and based on the images in a page-turning state, the image sequence of each page is determined.

5. The video detection method according to claim 4, characterized in that, The step of matching each line of text in adjacent images of the video to be detected to obtain the matching results of adjacent images includes: The text in one of the adjacent images is matched with the text in the other image to obtain the line matching result of the text in the adjacent images. Based on the line matching results of each line of text, the matching results of the adjacent images are determined.

6. The video detection method according to claim 5, characterized in that, Determining the matching result of adjacent images based on the line matching results of each line of text includes: Based on the line matching results of each line of text, determine the number of lines of text that are successfully matched; The matching result of the adjacent images is determined based on the number of successfully matched lines of text and the number of lines of text in each of the adjacent images.

7. A video detection device, characterized in that, include: The page-turning action detection unit performs page-turning action detection on the video to be detected, and obtains the image sequence of each page in the video to be detected that is not in a page-turning action. Representative image units are determined, and based on the visual features, frequency domain features, and semantic features of the text identified in each image in the image sequence, the depth features of each image and the hidden layer features of the image sequence are determined. Based on the similarity between the depth features of each image and the hidden layer features of the image sequence, the representative image of the page is determined from each image; The image processing unit performs image processing based on the representative images of each page.

8. An electronic device comprising a camera, a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The camera is used to capture the video to be detected; When the processor executes the program, it implements the video detection method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the video detection method as described in any one of claims 1 to 6.

Images