Image processing method, device and computer equipment

Abstract

The embodiments of the present application disclose an image processing method, apparatus and computer equipment, wherein the method includes: obtaining a first image frame and a second image frame adjacent to each other in a video to be processed; if the first image frame and the second image frame both include text content, determining a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; if the first image frame and the second image frame both include text content, determining a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame; based on the similarity relationship between the first image frame and the second image frame, updating a similarity relationship set of the video to be processed, wherein the similarity relationship recorded in the similarity relationship set can be used to determine a key image frame of the video to be processed, thereby improving the accuracy of key image frame extraction for the video file.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an image processing method, apparatus, and computer equipment. Background Art

[0002] Key image frames in video files can be used in a variety of scenarios, such as selecting video cover art and editing videos. The history of image processing technology reveals the existence of numerous classic image processing models and algorithms for determining image similarity and key image frames based on this similarity. However, the accuracy of key image frames extracted using traditional image processing methods is generally low. Therefore, improving the accuracy of key image frame extraction in video files has become a current research hotspot. Summary of the Invention

[0003] The embodiments of the present application provide an image processing method, apparatus, and computer device that can improve the accuracy of key image frame extraction from video files.

[0004] In one aspect, an embodiment of the present application provides an image processing method, comprising:

[0005] Obtaining a first image frame and a second image frame adjacent to each other in the video to be processed;

[0006] If both the first image frame and the second image frame include text content, determining a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame;

[0007] If neither the first image frame nor the second image frame includes text content, determining a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame;

[0008] Based on the similarity relationship between the first image frame and the second image frame, the similarity relationship set of the video to be processed is updated, and the similarity relationship set is used to record the similarity relationship between any two adjacent image frames in the video to be processed. The similarity relationship recorded in the similarity relationship set is used to determine the key image frame of the video to be processed.

[0009] On the other hand, an embodiment of the present application provides an image processing device, including:

[0010] An acquisition unit, configured to acquire a first image frame and a second image frame adjacent to each other in a video to be processed;

[0011] a determining unit configured to determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame if both the first image frame and the second image frame include text content;

[0012] The determining unit is further configured to determine a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame if neither the first image frame nor the second image frame includes text content;

[0013] A processing unit is used to update the similarity relationship set of the video to be processed based on the similarity relationship between the first image frame and the second image frame, wherein the similarity relationship set is used to record the similarity relationship between any two adjacent image frames in the video to be processed, and the similarity relationship recorded in the similarity relationship set is used to determine the key image frame of the video to be processed.

[0014] In one embodiment, the determining unit is further configured to perform:

[0015] If either the first image frame or the second image frame includes text content, and the other image frame does not include text content, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0016] In another embodiment, the first text feature includes a first number of text boxes in the first image frame, and the second text feature includes a second number of text boxes in the second image frame. One text box is used to represent a text display area, and the text display area is used to display text content. The determining unit is further configured to perform:

[0017] If the first number and the second number are different, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0018] In yet another embodiment, the first text feature further includes first position information of a text box in the first image frame, and the second text feature further includes second position information of a text box in the second image frame; and the determining unit is further configured to perform:

[0019] If the first number and the second number are the same, determining a matching relationship between the first location information and the second location information;

[0020] If the matching relationship between the first position information and the second position information is not matching, it is determined that the similarity relationship between the first image frame and the second image frame is not similar.

[0021] In yet another embodiment, the first text feature further includes first text content in a text box in the first image frame, and the second text feature further includes second text content in a text box in the second image frame; and the determining unit is further configured to perform:

[0022] If the matching relationship between the first position information and the second position information is a match, determining the text similarity between the first text content and the second text content;

[0023] If the text similarity is greater than a text similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is similar;

[0024] If the text similarity is less than or equal to the text similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0025] In yet another embodiment, the processing unit is further configured to execute:

[0026] Adjusting the first image frame and the second image frame to a consistent size;

[0027] Performing text region detection on the first image frame to obtain a text frame in the first image frame; and performing text region detection on the second image frame to obtain a text frame in the second image frame;

[0028] Perform text recognition on the text box in the first image frame to obtain the first text content, and perform text recognition on the text box in the second image frame to obtain the second text content.

[0029] In yet another embodiment, the image feature includes a first image feature of the first image frame and a second image feature of the second image frame; and the determining unit is further configured to perform:

[0030] If the image similarity between the first image feature and the second image feature is less than or equal to an image similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is dissimilar;

[0031] If the image similarity between the first image feature and the second image feature is greater than the image similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be similar.

[0032] In another embodiment, the video to be processed includes N image frames, the first image frame and the second image frame belong to the N image frames, and the N image frames are arranged in the order of the appearance time of each of the N image frames in the video to be processed, where N is an integer greater than 1; the processing unit is further configured to perform:

[0033] Acquire a similarity relationship between every two adjacent image frames in the N image frames from the similarity relationship set;

[0034] Based on a similarity relationship between every two adjacent image frames in the N image frames, the N image frames are divided into M image frame groups; wherein each image frame group includes one or more image frames, and when any image frame group includes multiple image frames, the multiple image frames are continuous, and the similarity relationship between every two image frames is similar, and M is a positive integer, and M is less than or equal to N;

[0035] An image frame is obtained from each image frame group as a key image frame.

[0036] In one aspect, the present application provides a computer device, comprising:

[0037] a processor adapted to execute one or more computer programs;

[0038] A computer storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded and executed by the processor:

[0039] Acquire a first image frame and a second image frame adjacent to each other in the video to be processed; if both the first image frame and the second image frame include text content, determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; if neither the first image frame nor the second image frame includes text content, determine a similarity relationship between the first image frame and the second image feature of the second image frame; based on the similarity relationship between the first image frame and the second image frame, update a similarity relationship set of the video to be processed, the similarity relationship set being used to record a similarity relationship between any two adjacent image frames in the video to be processed, and the similarity relationship recorded in the similarity relationship set being used to determine a key image frame of the video to be processed.

[0040] In one aspect, the present application provides a computer storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded and executed by a processor:

[0041] Acquire a first image frame and a second image frame adjacent to each other in the video to be processed; if both the first image frame and the second image frame include text content, determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; if neither the first image frame nor the second image frame includes text content, determine a similarity relationship between the first image frame and the second image feature of the second image frame; based on the similarity relationship between the first image frame and the second image frame, update a similarity relationship set of the video to be processed, the similarity relationship set being used to record a similarity relationship between any two adjacent image frames in the video to be processed, and the similarity relationship recorded in the similarity relationship set being used to determine a key image frame of the video to be processed.

[0042] In one aspect, the present application provides a computer program product or computer program, the computer program product comprising a computer program, the computer program being stored in a computer storage medium; a processor of a computer device reading the computer program from the computer storage medium, and the processor executing the computer program, causing the computer device to perform:

[0043] Acquire a first image frame and a second image frame adjacent to each other in the video to be processed; if both the first image frame and the second image frame include text content, determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; if neither the first image frame nor the second image frame includes text content, determine a similarity relationship between the first image frame and the second image feature of the second image frame; based on the similarity relationship between the first image frame and the second image frame, update a similarity relationship set of the video to be processed, the similarity relationship set being used to record a similarity relationship between any two adjacent image frames in the video to be processed, and the similarity relationship recorded in the similarity relationship set being used to determine a key image frame of the video to be processed.

[0044] In the present application, the computer device determines the similarity between two image frames based on the first text feature and the second text feature when both adjacent image frames include text content, so that the method provided by the present application can recall key image frames with text content; and because the computer device can determine the similarity between two image frames based on image features when both adjacent image frames do not include text content, the present application takes into account the situation where the computer device determines the similarity between two image frames when the image frames only include image content, thereby improving the accuracy of the computer device in extracting key image frames from video files to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0046] Figure 1 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0047] Figure 2 is a schematic diagram of a text box provided in an embodiment of the present application;

[0048] Figure 3 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0049] Figure 4a is a schematic diagram of a color histogram provided in an embodiment of the present application;

[0050] Figure 4b is a schematic diagram of a pixel unit (cell) and a pixel block (block) provided in an embodiment of the present application;

[0051] Figure 5a is a schematic diagram of a kernel filtering image provided by an embodiment of the present application;

[0052] Figure 5b This is a schematic diagram of gradient direction division provided in an embodiment of the present application;

[0053] Figure 6 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0054] Figure 7 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The embodiment of the present application provides an image processing solution based on OCR technology and image analysis technology, wherein OCR (Optical Character Recognition) technology refers to: a technology for analyzing, identifying and processing image files of text materials to obtain text and layout information. It can be understood that the OCR technology can be used to recognize the text in the image and obtain the text in the image returned in the form of text. The above-mentioned image analysis technology refers to: a technology that uses mathematical models and image processing technology to analyze the underlying features and upper-level structures, thereby extracting information with certain intelligence. In specific applications, the focus of image analysis technology is on studying the content of the image, which includes but is not limited to the use of various image processing technologies, but compared with image processing technology, image analysis technology is more inclined to analyze, interpret and recognize the content of the image. The main process of image analysis can be: input, segmentation, recognition, and interpretation.

[0056] The general principle of the image processing solution proposed in the embodiment of the present application can be as follows: for each two adjacent image frames (first image frame, second image frame) in a video file, according to whether the two image frames both include text content, a corresponding similarity relationship determination method is selected to determine the similarity relationship between the two image frames (such as: similar, dissimilar), and further based on the similarity relationship between the two adjacent image frames, the key image frame in the video file is determined. Specifically, when both image frames include text content, the similarity relationship determination method adopted can be: based on the first text feature of the text content in the first image frame and the second text feature of the text content in the second image frame, the similarity relationship between the two image frames is determined; when both image frames do not include text content, the similarity relationship determination method adopted can be: based on the first image feature of the first image frame and the second image feature of the second image frame, the similarity relationship between the two image frames is determined. It is not difficult to see that the image processing solution provided by the present application can be used to determine the similarity relationship between cross-modal data, such as determining the similarity relationship between two images that include both image content and text content. Therefore, it can be understood that the use of the present application to determine the similarity relationship between two images can effectively improve the accuracy of the determination result (i.e., the similarity relationship between the two images), which can further improve the accuracy of key image frame extraction for video files.

[0057] In specific applications, the above-mentioned image processing solution can be executed by a computer device, which can be a terminal or a server. The terminal may include but is not limited to: smartphones, tablets, laptops, desktop computers, smart TVs, etc.; a variety of client applications (APPs) can run in the terminal, such as multimedia playback clients, social clients, browser clients, information streaming clients, educational clients, etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, etc.

[0058] Based on the above description, it is not difficult to understand that the image processing solution provided by the present application can be applied to multiple scenarios, such as: video cover selection scenario, video condensation scenario, and video editing scenario. Optionally, when the above image processing solution is applied to the video cover selection scenario, the computer device can first obtain one or more key image frames of the video file through the above image processing solution. If there is only one key image frame obtained, the computer device can directly use the key image frame as the video cover. If there are multiple key image frames obtained, the computer device can perform image quality evaluation processing on the multiple key image frames, and then select the key image frame with the highest image quality from the multiple key image frames as the video cover of the video file.

[0059] Optionally, when the above-mentioned image processing scheme is applied to a video condensation scenario, the computer device can obtain one or more key image frames of the video file through the above-mentioned image processing scheme, and compose a new target video (i.e., the video file after video condensation) based on the one or more key image frames. Since the key image frame can represent the content of the video frame at the time node near the key image frame, it can be understood that the target video composed of key image frames can more completely summarize the main content of the video file. It can be further understood that the image processing scheme can effectively improve the work efficiency of relevant personnel when used in security video display scenarios.

[0060] Optionally, when the above-mentioned image processing scheme is applied to a video editing scene, the computer device can first obtain one or more key image frames of the video file through the above-mentioned image processing scheme, and then set the duration of the editing segment (such as 30S), and then select the video segment containing the largest number of key image frames (the duration of the video segment is 30S) as the target segment, and the target segment can be used as the key highlight segment of the video file. Of course, the present application can also be used for other similarity judgment scenarios, such as: for judging whether the text content of two pictures is the same, whether the faces of two pictures are similar, whether the clothes contained in two pictures are similar, etc., and the present application will not give detailed examples here.

[0061] Based on the description of the principles of the above image processing scheme, this application provides an image processing method, which can be executed by the above-mentioned computer device; see Figure 1 As shown, the image processing method may include the following steps:

[0062] S101: Acquire a first image frame and a second image frame that are adjacent to each other in a video to be processed.

[0063] The first image frame and the second image frame can be selected from N image frames, and these N image frames can be obtained by the computer device decoding the video to be processed and performing frame extraction on the video to be processed. For example, the computer device can use OpenCV (open source computer vision library) to decode the video to be processed; OpenCV refers to the Intel open source computer vision library, which consists of a series of C functions and a small number of C++ classes and implements many common algorithms in image processing and computer vision. In this application, the image processing module and video analysis module provided by OpenCV are mainly utilized. The image processing module includes but is not limited to linear and nonlinear image filtering, geometric image transformations (resizing, affine and perspective warping, general table-based remapping), color space conversion, histograms, etc.; the video analysis module includes but is not limited to motion estimation, background subtraction, and object tracking algorithms. Based on this, the application can quickly and efficiently decode the video to be processed to obtain N image frames, thereby enabling the application to quickly obtain the first image frame and the second image frame. In addition, the N image frames mentioned above can be arranged in the order of time in which the image frames extracted appear in the video to be processed. Based on this, the first image frame and the second image frame can refer to two adjacent image frames among the N image frames.

[0064] In specific applications, the computer device may perform frame extraction processing on the video to be processed in the following two ways, but not limited to:

[0065] The first method is that the computer device uses a full frame sampling method to extract frames from the video to be processed. Full frame sampling can be understood as: the frequency of the computer device extracting image frames is the same as the frame rate of the video. For example, when the frame rate of the video to be processed is 25 frames per second, the computer device can extract 25 image frames per second. Then, for a video with a duration of m seconds, the N image frames obtained by the computer device after sampling and frame extraction are specifically: 25*m video frames. It is not difficult to understand that the computer device uses this method to extract frames from the video to be processed, which enables the computer device to determine more comprehensive key image frames from the N image frames obtained by frame extraction. The second method is that the computer device uses a sampling and frame sampling method to extract frames from the video to be processed. Sampling and frame extraction can be understood as: when the frame rate of the video to be processed is n (n is a positive integer), k image frames are extracted per second, and k is a positive integer less than n. Then, for a video with a duration of m seconds, the N image frames obtained by the computer device after sampling and frame extraction are specifically: k*m video frames. Based on this, it is not difficult to understand that the computer device uses a sampling frame extraction method to extract frames from the video to be processed, which can effectively reduce the amount of calculation and alleviate the data processing burden of the computer device.

[0066] S102 : If both the first image frame and the second image frame include text content, determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame.

[0067] In practical applications, it is not difficult to understand that an image frame can include both text content and image content. Based on this, the computer device can first determine whether the first image frame includes text content, and determine whether the second image frame includes text content; further, when both the first image frame and the second image frame include text content, the computer device can determine the similarity relationship between the first image frame and the second image frame based on the first text feature of the first image frame and the second text feature of the second image frame. Experiments have also shown that when performing text recognition on the text display area in the image frame, the accuracy of the recognition result obtained by using text box detection is higher than the accuracy of the recognition result obtained by directly performing text recognition on the text display area. For example, the text feature of any image frame can be obtained by the computer device using OCR recognition technology to extract the text feature of any image frame.

[0068] Specifically, the first text feature of the first image frame mentioned above may include but is not limited to: the first number of text boxes in the first image frame, the first position information of each text box in the first number of text boxes, and the first text content of each text box; the second text feature of the second image frame may include but is not limited to: the second number of text boxes in the second image frame, the second position information of each text box in the second number of text boxes, and the second text content of each text box. Figure 2 As shown in 211, it can be seen that the text box 211 is a part of the display area 21 of the text content in the image frame, and the text box can be used to represent a text display area, and the text display area is used to display text content. For example, if the computer device uses OCR technology to extract text features from the image frame, then the number of text boxes (such as the first number, the second number) and the position information (such as the first position information, the second position information) can be obtained after the computer device uses the FAST-RCNN algorithm to detect the text area of ​​the image frame, and the text content of each text box (such as the first text content, the second text content) can be obtained after the computer device uses the CNN (Convolutional Neural Networks, Convolutional Neural Networks) algorithm, the LSTM (Long-Short Term Memory, Long Short-Term Memory Artificial Neural Network) algorithm, and the CTC (Connectionist Temporal Classification, Connectionist Temporal Classification) algorithm to perform text recognition on each text box.

[0069] S103 : If neither the first image frame nor the second image frame includes text content, determining a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame.

[0070] The similarity relationship between the first image frame and the second image frame may be determined by the computer device based on the similarity between the first image frame and the second image frame. Specifically, if the image similarity between the first image frame and the second image frame is less than or equal to an image similarity threshold, the computer device may determine that the similarity relationship between the first image frame and the second image frame is dissimilar; correspondingly, if the image similarity between the first image frame and the second image frame is greater than the image similarity threshold, the computer device may determine that the similarity relationship between the first image frame and the second image frame is similar.

[0071] In one embodiment, the similarity between the first image frame and the second image frame can be obtained by: a computer device uses a SIFT (Scale-invariant feature transform) algorithm or a SURF (Speeded Up Robust Features) algorithm to evaluate the similarity of the first image frame and the second image frame. The main principles of these two algorithms are as follows: by extracting feature points from the image frame, and then constructing a feature vector of the image frame based on the extracted feature points, the computer device can obtain the feature vector of the first image frame (i.e., the first image feature) and the feature vector of the second image frame (i.e., the second image feature), and then calculate the distance between the two feature vectors (such as Hamming distance, Euclidean distance, cosine distance, etc.) to obtain the image similarity between the first image frame and the second image frame.

[0072] In another embodiment, the image similarity between the first image frame and the second image frame may also be determined by a computer device based on PSNR (Peak Signal to Noise Ratio) or SSIM (Structural Similarity). The principles of both the PSNR algorithm and the SSIM algorithm are to evaluate the similarity between the first image frame and the second image frame at the pixel level.

[0073] In another embodiment, the image similarity between the first image frame and the second image frame can also be obtained by: the computer device evaluates the similarity of the first image frame and the second image frame based on a deep learning algorithm. Exemplarily, the deep learning algorithm adopted by the computer device can be a twin network algorithm. Then, when the deep learning algorithm adopted by the computer device is a twin network algorithm, the computer device can first extract the first image features of the first image frame and the second image features of the second image frame respectively, and merge the first image features and the second image features; further, after the computer device merges the first image features and the second image features, the similarity between the first image frame and the second image frame can be calculated based on the target image features after the feature merging.

[0074] It should be noted that when the image processing method provided in this application is used for online business (such as online video editing, online video cover selection), since online business needs to focus on the recognition speed of key image frames, therefore, in this case, among the above three image similarity calculation methods, the priority of the method in which the computer device uses a deep learning algorithm to calculate the image similarity of the first image frame and the second image frame can be set to the lowest, that is, when this application is used for online business, the other two image similarity calculation methods can be given priority.

[0075] In practical applications, image features can be represented by color histograms and gradient histograms. Based on this, the computer device can splice the color histogram of the image frame and the gradient histogram of the image frame, and then use the features obtained by the splicing process as the image features of the image frame. Then, the computer device can further determine the similarity relationship between the two image frames by calculating the cosine distance between the two image features; specifically, when the cosine distance is greater than or equal to a certain parameter (such as a distance threshold parameter), the computer device can determine that the similarity relationship between the two image frames is similar; when the cosine distance is less than the certain parameter, the computer device can determine that the similarity relationship between the two image frames is dissimilar. For example, the range of the cosine distance can be [-1, 1], and the distance threshold parameter can be 0.7. That is to say, when the cosine distance between the first image feature and the second image feature is greater than or equal to 0.7, the computer device can determine that the first image frame and the second image frame are similar; when the cosine distance between the first image feature and the second image feature is less than 0.7, the computer device can determine that the first image frame and the second image frame are dissimilar.

[0076] In one embodiment, it is easy to understand that when only one of the first image frame and the second image frame includes text content, the computer device can directly determine that the similarity relationship between the first image frame and the second image frame is dissimilar.

[0077] S104: Based on the similarity relationship between the first image frame and the second image frame, update the similarity relationship set of the video to be processed.

[0078] Among them, the similarity relationship set is used to record the similarity relationships between any two adjacent image frames in the video to be processed. Then, the computer device can determine the key image frames of the video to be processed based on the similarity relationships recorded in the similarity relationship set. Exemplarily, the computer device can represent the similarity relationship between two image frames as similar or dissimilar in the similarity relationship set based on two different identifiers. Each element in the similarity relationship set corresponds to an image frame. For example, the computer device can use the identifier "0" and / or the identifier "1" to represent the similarity relationship between two image frames in the similarity relationship set. For instance, assume that any element in the similarity relationship set can be represented as Ri. Then, Ri can represent whether the i-th image frame among N image frames is similar to the (i - 1)-th image frame, where i is a positive integer and i < N. Then, if the similarity relationship between the i-th image frame and the (i - 1)-th image frame is similar, then Ri = R(i - 1); if the similarity relationship between the i-th image frame and the (i - 1)-th image frame is dissimilar, then Ri + R(i - 1) = 1, that is: Ri and R(i + 1) are inverted, which means: when Ri = 0, R(i - 1) = 1; when Ri = 1, R(i - 1) = 0.

[0079] Based on this, when the computer device determines the key image frames of the video to be processed based on the similarity relationships recorded in the similarity relationship set, it can be specifically implemented in the following manner: The computer device determines the similarity relationships between every two adjacent image frames from the similarity relationship set, and divides the N image frames into M (M is a positive integer and M ≤ N) image frame groups based on the similarity relationships between every two adjacent image frames. Each image frame group includes one or more image frames. Then, when there are multiple image frames in any image frame group, the similarity relationship between every two image frames is similar, that is: these multiple image frames are all represented by the same identifier. Then, it is easy to understand that these multiple image frames in any one image frame are multiple consecutive image frames among the N image frames. Further, the computer device can obtain one image frame from each image frame group as the key image frame. Exemplarily, the computer device can use the first image frame in each image frame group as the key image frame, or can use any image frame in each image frame group as the key image frame. This application does not limit this.

[0080] The above method is described in detail below with reference to specific examples. Assume that the N images are specifically: {R1, R2, R3, R4, R5, R6, R7, R8, R9}, and the initial definition is R1 = 1; then, when the similarity relationship set is {1, 1, 0, 0, 1, 1, 1, 1, 0}, since R1 = R2, it can be understood that: the first key image frame and the second key image frame are similar; since R2 + R3 = 1, it can be understood that: the second key image frame and the third key image frame are not similar. In the above manner, the computer device can obtain the similarity relationship between each two adjacent image frames in the N image frames based on the similarity relationship set corresponding to the N image frames. It should be noted that the computer device can also initially define R1=0. In this case, the similarity relationship set corresponding to {R1, R2, R3, R4, R5, R6, R7, R8, R9} can be expressed as: {0, 0, 1, 1, 0, 0, 0, 0, 1}. Based on this, the computer device can divide these N image frames into 4 image frame groups, namely {R1, R2}, {R3, R4}, {R5, R6, R7, R8}, and {R9}; then, based on this, when the computer device takes the first image frame in each image frame group as the key image frame, the key image frame corresponding to {R1, R2, R3, R4, R5, R6, R7, R8, R9} can be {R1, R3, R, R9}; when the computer device takes any image frame in each image frame group as the key image frame, the key image frame corresponding to {R1, R2, R3, R4, R5, R6, R7, R8, R9} can be {R1, R3, R, R9}, or {R1, R4, R6, R9}, etc.

[0081] In an embodiment of the present application, when both image frames in a video file include text content, the computer device can use OCR technology to obtain text features corresponding to the text display area of ​​the image frame, and then determine the similarity relationship between the two image frames based on the first text feature of the first image frame and the second text feature of the second image frame. Therefore, it can be seen from the above that the text recognition result obtained by the computer device using OCR technology to perform text box recognition has a high accuracy. Therefore, this can make the similarity relationship between the two image frames determined by the computer device based on the OCR technology more accurate, so that the computer device can effectively recall the key image frames with text content. When both image frames in the video file do not include text content, the computer device determines the similarity relationship between the two image frames by obtaining the first image feature of the first image frame and the second image feature of the second image frame. This allows the present application to take into account the situation where the computer device determines the similarity relationship between the two image frames when the image frames only include image content, and to a certain extent improves the accuracy of the computer device when extracting key image frames from video files.

[0082] Based on the above description, the embodiment of the present application provides another image processing method, which can also be executed by the computer device mentioned above; see Figure 3 , the image processing method may include the following steps:

[0083] S301: Acquire a first image frame and a second image frame that are adjacent to each other in a video to be processed.

[0084] In one embodiment, the relevant embodiments of step S301 can refer to the relevant description of step S101, and this application will not repeat them here.

[0085] S302: Adjust the first image frame and the second image frame to the same size.

[0086] To improve the accuracy of text content recognition by the computer device on image frames, the computer device may resize (i.e., change the size of the image frames) each image frame before performing text content recognition on the image frames to adjust the image frames to the same size. Text content recognition may include text box detection and text box text recognition. Furthermore, the computer device may optionally adjust the contrast, brightness, saturation, etc. of each image frame when resizing the image frames, which is not limited in this application.

[0087] S303 , performing text region detection on the first image frame to obtain a text frame in the first image frame; and performing text region detection on the second image frame to obtain a text frame in the second image frame.

[0088] Based on the description of step S102, it can be seen that in a specific application, the computer device can perform text region detection on the image frame based on the Fast-RCNN technology to obtain one or more text boxes corresponding to the image frame. Among them, Fast-RCNN can avoid repeated convolution calculations by inputting the image frame into the convolutional network for convolution, thereby greatly reducing the amount of calculation of the computer device; in addition, Fast-RCNN adopts ROIpooling (i.e., Region of Interest pooling), which can convert each candidate text box into the same size; using SVD (Singular Value Decomposition) instead of the fully connected layer in Fast-RCNN can effectively improve the calculation speed; the increase of the data set can effectively improve the mAP (Meanaverage precision), thereby improving the accuracy of the computer device in recognizing text boxes.

[0089] Then, based on the above description, it can be seen that the computer device can use Fast-RCNN technology to perform text area recognition on the first image frame to obtain a first number of text boxes in the first image frame; the computer device can also use Fast-RCNN technology to perform text recognition on the second image frame to obtain a second number of text boxes in the second image frame.

[0090] S304 , performing text recognition on the text frame in the first image frame to obtain first text content, and performing text recognition on the text frame in the second image frame to obtain second text content.

[0091] As can be seen from the description of step S102, in one embodiment, the computer device can combine CNN technology, LSTM technology, and CTC technology to perform text recognition on each text frame in the first image frame (or the second image frame). CNN is a type of feedforward neural network with a deep structure that includes convolution calculations. The use of CNN technology can effectively reduce the dimensionality of large-scale image frames into small-scale data and effectively retain image features, thereby allowing the computer device to ensure the efficiency of convolution processing while taking into account the accuracy of text recognition results to a certain extent. In addition, it can be seen from the above that the first image frame and the second image frame are two adjacent image frames among N image frames, and the N image frames are arranged in chronological order in the video to be processed. Therefore, since the LSTM network is very suitable for classification, processing and prediction based on time series data, and CTC is a way to avoid manual alignment of input and output, it is very suitable for OCR technology. Therefore, when the computer device uses OCR technology to process the image frame, it combines CNN technology, LSTM technology and CTC technology to perform text recognition on each text box in the first image frame (or second image frame), so that the accuracy of the text recognition result obtained by the computer device can be guaranteed, and thus the accuracy of the similarity relationship between the first image frame and the second image frame determined by the computer device can be guaranteed.

[0092] S305: If the first number and the second number are the same, determine a matching relationship between the first location information and the second location information.

[0093] As can be seen from the foregoing, the text features of the first image frame include a first number of text boxes in the first image frame and first position information of the text boxes in the first image frame, and the text features of the second image frame include a second number of text boxes in the second image frame and second position information of the text boxes in the second image frame. The first position information may include position information of each text box in the first number of text boxes, and the second position information may include position information of each text box in the second number of text boxes.

[0094] Then, the computer device can first determine whether the first number and the second number are the same. If the first number and the second number are not the same, the computer device can directly determine that the similarity relationship between the first image frame and the second image frame is dissimilar; if the first number and the second number are the same, the computer device can further determine the similarity relationship between the first image frame and the second image frame based on the matching relationship between the first position information and the second position information.

[0095] In a specific embodiment, the method for a computer device to determine the matching relationship between the first position information and the second position information may include the following steps (1)-(3), wherein: (1) the computer device obtains the position information of each text box in a first number of text boxes corresponding to the first image frame, and obtains the position information of each text box in a second number of text boxes corresponding to the second image frame; (2) the computer device establishes a corresponding relationship between the first number of text boxes and the second number of text boxes; (3) the computer device determines the matching relationship between the position information of the two text boxes with a corresponding relationship based on whether the position error between the position information of the two text boxes with a corresponding relationship is less than an error threshold. Wherein, exemplarily, when the computer device determines the matching relationship between the position information of the two text boxes, the loss function adopted may be a cross-entropy loss function, a weighted cross-entropy loss function, or the like. It should be noted that, for the above-mentioned steps (1) and (2), the computer device may first execute (1) and then execute (2); of course, the computer device may also first execute (2) and then execute (1); this application does not impose any specific restrictions on the execution order of the above-mentioned steps (1) and (2).

[0096] S306: If the matching relationship between the first position information and the second position information is a match, determine the text similarity between the first text content and the second text content.

[0097] Based on the above description, it can be seen that the text features of the first image frame also include: the first text content of the text box in the first image frame, and the text features of the second image frame also include: the second text content of the text box in the second image frame, wherein the first text content can include: the text content of each text box in the first number of text boxes, and the second text content can include: the text content of each text box in the second number of text boxes. For the sake of convenience, it is assumed below that any text box in the first image frame is represented as: text box A, and it is assumed that the text box in the second image frame that has a corresponding relationship with text box A is represented as: text box B, then it can be seen that text box A and text box B have a corresponding relationship, then, text box A and text box B can also be called: a pair of text boxes with a corresponding relationship. Then, it is not difficult to understand that for the first image frame and the second image frame, the computer device can obtain the first number (or second number) of text boxes with a corresponding relationship.

[0098] When a computer device determines the matching relationship between the position information of two corresponding text boxes based on whether the position error between the position information of the two corresponding text boxes is less than an error threshold, if the position error between the upper left corner position of text box A and the upper left corner position of text box B is less than the error threshold, and the position error between the lower right corner position of text box A and the lower right corner position of text box B is less than the error threshold, the computer device can determine that the matching relationship between the position information of text box A and the position information of text box B is a match. Exemplarily, the error threshold can be 5px, where px (Pixel) refers to the smallest unit of a digital image or graphic that can be displayed and represented on a digital display device. Accordingly, it is not difficult to understand that if the position error between the upper left corner position of text box A and the upper left corner position of text box B is greater than or equal to the error threshold, or if the position error between the lower right corner position of text box A and the lower right corner position of text box B is greater than or equal to the error threshold, the computer device can determine that the matching relationship between the position information of text box A and the position information of text box B is a mismatch.

[0099] In practical applications, it is not difficult to understand that the computer device can determine the matching relationship between the position information of each pair of text boxes with a corresponding relationship in the two text boxes of the first number pair (or second number pair) mentioned above by the above method. Then, further, if the matching relationship between the position information of any pair of text boxes with a corresponding relationship is not matched, the computer device can directly determine that the similarity relationship between the first image frame and the second image frame is not similar. Correspondingly, if the matching relationship between the position information of each pair of text boxes with a corresponding relationship is a match, the computer device can further determine the similarity relationship between the first image frame and the second image frame based on the text similarity between the first text content of the text box in the first image frame and the second text content of the text box in the second image frame. Specifically, the computer device can determine the similarity relationship between the first image frame and the second image frame based on the text similarity between each pair of text boxes with a corresponding relationship. Among them, when the computer device determines the text similarity between each pair of text boxes with a corresponding relationship, the loss function adopted can be a softmax loss function (normalized loss function) or a weighted softmax loss function (weighted normalized loss function).

[0100] S307: If the text similarity is greater than the text similarity threshold, determine that the similarity relationship between the first image frame and the second image frame is similar.

[0101] The computer device may perform text similarity evaluation on each pair of corresponding text boxes. For ease of explanation, this application further assumes that the text content displayed in text box A is text content a, and the text content displayed in text box B is text content b. Then, the computer device may first perform text recognition on text box A to obtain text content a, and perform text recognition on text box B to obtain text content b, and then the computer device may calculate the text similarity between text content a and text content b.

[0102] Then, if the text similarity between each pair of corresponding text boxes in the first number of pairs (or second number of pairs) mentioned above is greater than the first similarity threshold, the computer device can determine that the text similarity between the first text content and the second text content is greater than the text similarity threshold, and then the computer device can determine that the similarity relationship between the first image frame and the second image frame is similar. Wherein, for example, the first similarity threshold mentioned above can be: the computer device is set while ensuring that the text error between the two text contents of each pair of corresponding text boxes is within 2 characters. That is to say, if the number of characters with difference between the text content a and the text content b mentioned above is 2, the computer device determines that the text similarity between the text content a and the text content b is less than or equal to the first similarity threshold; if the number of characters with difference between the text content a and the text content b mentioned above is 0 or 1, the computer device determines that the text similarity between the text content a and the text content b is greater than the first similarity threshold.

[0103] S308 : If the text similarity is less than or equal to the text similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is dissimilar.

[0104] Based on the above description, it is not difficult to understand that if among the first number pairs (or second number pairs) of text boxes with corresponding relationships mentioned above, the text similarity between any pair of corresponding text boxes is less than or equal to the first similarity threshold, the computer device can determine that the text similarity between the first text content and the second text content is less than the text similarity threshold, and then the computer device can determine that the similarity relationship between the first image frame and the second image frame is dissimilar.

[0105] S309 : If neither the first image frame nor the second image frame includes text content, determining a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame.

[0106] As can be seen from step S103, the computer device can splice the color histogram and gradient histogram of the image frame to obtain the image features of the image frame. The color histogram includes: the color histogram of the R channel, the color histogram of the G channel, and the color histogram of the B channel. Then, the computer device can splice the color histograms of these three channels to obtain a complete color histogram corresponding to the image frame. For example, for Figure 4a After the computer device extracts the image features of the image frame 41 shown in FIG. Figure 4a Color histograms of the three channels shown in 42.

[0107] The following will explain in detail how a computer device obtains a color histogram with specific examples: the computer device can first adjust the size of the image frame to the target size, such as adjusting it to 720*480px. Further, the computer device can divide the image frame into several 16*16px small units (or called: cells, i.e., cell units), and then the computer device obtains the 16*16 pixel values ​​included in each small unit, and each pixel value can be any one of the 256 values ​​from 0 to 255; further, the computer device can divide 0-255 into 9 pixel value intervals, i.e., the range of 0-255 is evenly mapped to the 1-9 pixel value interval, and then the computer device counts the number of pixel values ​​corresponding to each pixel value interval based on the 16*16 pixel values, and constructs a histogram by sliding the window. Each slide can obtain a feature vector with a length of 9. Then, it is not difficult to understand that a histogram constructed each time a slide is constructed is a color histogram corresponding to a small unit. Of course, the number of pixel value intervals can be other values ​​(such as 10, 16, etc.), and the sliding length when sliding the window can be set according to actual needs. For example, in this application, the sliding length can be set to 16px.

[0108] Based on the above description, assuming that the image frames are all adjusted to 720*480px, the sliding length is 16px, and the size of each cell is 16*16px; then, the computer device can further use 2*2 cells as a block (pixel block). For example, the block can be as follows Figure 4b As shown in 43, the cell can be Figure 4bAs shown in 44, it can be seen that each block includes 4 cells. Based on this, it is not difficult to understand that the feature vector of each block can be obtained by splicing the feature vectors of 4 cells. Therefore, the length of the feature vector of a block can be 4*9=36, where "4" refers to the number of cells and "9" refers to the length of the feature vector of each cell. Since the feature vector of the cell is obtained by sliding the window, it is not difficult to understand that the feature vector of the block can also be obtained by sliding the window. For example, the sliding length can also be 16px. In this case, for an image frame with a size of 720*480px, it can include 720 / 16-1=44 blocks in the horizontal direction and 480 / 16-1=29 blocks in the vertical direction. Then, the total number of blocks included in the image frame is 44*29. Furthermore, the computer device can merge the feature vectors corresponding to these 44*29 blocks to obtain the color histogram of a channel corresponding to the image frame. That is to say, the length of the feature vector corresponding to the color histogram of each channel can be 44*29*9*4, where "44*29" is the number of blocks, "4" is the number of cells included in each block, and "9" is the length of the feature vector of each cell; then, it can be further understood that the length of the feature vector corresponding to the color histogram of an image frame is 44*29*9*4*3, where 44*29*9*4 refers to the length of the feature vector corresponding to the color histogram of each channel, and "3" refers to the three channels of R, G, and B.

[0109] From the above description, it can be seen that the computer device can not only obtain the color histogram, but also the gradient histogram of the image frame. The most important concept of the gradient histogram is the gradient, and the gradient contains two quantities, namely: direction and size. The ladder (i.e., direction) can be divided into horizontal and vertical directions. Taking the horizontal direction as an example, the ladder in the horizontal direction represents the difference between the current position and the values ​​on the left and right sides of the horizontal direction. The larger the difference, the larger the gradient. Then, the computer device can obtain the gradient histogram by calculating the horizontal gradient and the vertical gradient. The specific calculation method can be shown in Formula 1:

[0110] Grad 2 =grad H 2 +grad V 2 Formula 1

[0111] Among them, Grad represents the gradient of the gradient histogram, Grad H Represents the horizontal gradient of the gradient histogram, Grad Vrepresents the vertical gradient of the gradient histogram. In addition, the computer device can use Figure 5a The kernel filter image shown is used to calculate the horizontal and vertical gradients, such as the horizontal gradient at the Xi position can be calculated using Figure 5a The kernel shown in 51 is used to calculate, that is: -X(i-1)+X(i-1). However, it should be noted that experiments have shown that the computer device in this embodiment can ignore the gradient size when constructing the feature vector of the gradient histogram.

[0112] The following will describe in detail how a computer device obtains a gradient histogram with a specific example: the computer device first adjusts the size of the image frame to the target size, such as to 720*480px. Further, the computer device can divide the image frame into several 16*16px cells. Since the gradient histogram includes the size and direction of the gradient, it is not difficult to understand that each cell includes 512 pixel values ​​(i.e., 16*16 values ​​for indicating the size of the gradient, and 16*16 values ​​for indicating the direction of the gradient, a total of 16*16*2=512). In actual applications, the gradient direction of each cell can be divided into z (z is a positive integer) direction blocks (such as Figure 5b As shown in FIG, the gradient direction and magnitude in the cell are used to perform weighted projection on the z direction blocks so that each cell can generate a z-dimensional feature vector. For example, z can be 9, that is, the computer device can divide 360 ​​degrees into 9 direction blocks, which are then used to project the gradient direction. The direction of the gradient histogram can be calculated using Equation 2:

[0113] θ=arctan(grad V +grad H ) Formula 2

[0114] Among them, θ represents the angle corresponding to the gradient direction, Grad H Represents the horizontal gradient of the gradient histogram, Grad V Represents the vertical gradient of the gradient histogram.

[0115] Then, based on the above description, it is not difficult to understand that when constructing a gradient histogram, the computer device can only use the values ​​of the 256 gradient directions in each cell; it should be noted that, in the present application, the computer device can take the absolute value of the gradient direction and then project it. That is to say, in the embodiment of the present application, the angle range of the gradient direction in each cell is [0°, 180°]. Then, similarly, the computer device can divide 0-180 into 9 angle intervals, that is, the angle range of 0°-180° is evenly mapped to 1-9 angle intervals (that is, [0°, 20°], [20°, 40°], [40°, 60°], [60°, 80°], [80°, 100°], [100°, 120°], [120°, 140°, [140°, 160°], [160°, 180°]), and then the computer device counts the number of pixels corresponding to each angle interval based on the values ​​of the 256 gradient directions, and constructs a gradient histogram by sliding the window. Each slide can obtain a feature vector of length 9. In this case, for an image frame with a size of 720*480px, it can include 720 / 16-1=44 blocks in the horizontal direction and 480 / 16-1=29 blocks in the vertical direction. Therefore, the total number of blocks included in the image frame can be 44*29. Furthermore, the computer device can merge the feature vectors corresponding to these 44*29 blocks to obtain the gradient histogram corresponding to the image frame. In other words, the length of the feature vector corresponding to the gradient histogram of the image frame can be 44*29*9*4, where 44*29 is the number of blocks, "4" is the number of cells included in each block, and "9" is the length of the feature vector of each cell.

[0116] S310: Based on the similarity relationship between the first image frame and the second image frame, update the similarity relationship set of the video to be processed.

[0117] In one embodiment, the relevant embodiments of step S310 can refer to the relevant description of step S104, and this application will not repeat them here.

[0118] In the embodiment of the present application, a computer device uses FAST-RCNN technology to perform text box detection on an image frame, thereby effectively improving the speed and accuracy of text box detection. In addition, since the computer device also determines that the text similarity between the text contents of two text boxes is greater than a first similarity threshold when the text contents of the two text boxes differ by one character, it can be seen that the image processing method provided by the present application increases the case where the Hamming loss is 1, thereby increasing the probability that the text similarity between the first text content and the second text content is greater than the first similarity threshold. Therefore, the recall rate of the key image frames of the video to be processed is further effectively improved. At the same time, since the present application determines the similarity relationship between the first image frame and the second image frame based on image features by a computer device when neither the first image frame nor the second image frame includes text content, the present application takes into account the computer device's way of determining the similarity relationship between the two image frames in three situations: neither image frame includes text content, one of the two image frames includes text content, and both image frames include text content. This enables the present application to have a higher accuracy and recall rate when used for key image frame extraction, which greatly saves the computational amount and computational time of the computer device for text recognition in the processed video to a certain extent.

[0119] Based on the description of the above image processing method, the present application also discloses an image processing device, which can be a computer program (including program code) running in the above-mentioned computer device. The image processing device can perform the following steps: Figure 1 and Figure 3 For the image processing method shown, see Figure 6 The image processing device 60 may at least include: an acquisition unit 601, a determination unit 602 and a processing unit 603.

[0120] An acquisition unit 601 is configured to acquire a first image frame and a second image frame adjacent to each other in a video to be processed;

[0121] a determining unit 602 configured to determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame if both the first image frame and the second image frame include text content;

[0122] The determining unit 602 is further configured to determine a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame if neither the first image frame nor the second image frame includes text content;

[0123] Processing unit 603 is used to update the similarity relationship set of the video to be processed based on the similarity relationship between the first image frame and the second image frame. The similarity relationship set is used to record the similarity relationship between any two adjacent image frames in the video to be processed. The similarity relationship recorded in the similarity relationship set is used to determine the key image frame of the video to be processed.

[0124] In one embodiment, the determining unit 602 is further configured to execute:

[0125] If either the first image frame or the second image frame includes text content, and the other image frame does not include text content, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0126] In another embodiment, the first text feature includes a first number of text boxes in the first image frame, and the second text feature includes a second number of text boxes in the second image frame. One text box is used to represent a text display area, and the text display area is used to display text content. The determining unit 602 is further configured to execute:

[0127] If the first number and the second number are different, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0128] In another embodiment, the first text feature further includes first position information of a text box in the first image frame, and the second text feature further includes second position information of a text box in the second image frame; and the determining unit 602 is further configured to execute:

[0129] If the first number and the second number are the same, determining a matching relationship between the first location information and the second location information;

[0130] If the matching relationship between the first position information and the second position information is not matching, it is determined that the similarity relationship between the first image frame and the second image frame is not similar.

[0131] In another embodiment, the first text feature further includes the first text content of the text box in the first image frame, and the second text feature further includes the second text content of the text box in the second image frame; the determining unit 602 is further configured to execute:

[0132] If the matching relationship between the first position information and the second position information is a match, determining the text similarity between the first text content and the second text content;

[0133] If the text similarity is greater than a text similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is similar;

[0134] If the text similarity is less than or equal to the text similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0135] In yet another embodiment, the processing unit 602 is further configured to execute:

[0136] Adjusting the first image frame and the second image frame to a consistent size;

[0137] Performing text region detection on the first image frame to obtain a text frame in the first image frame; and performing text region detection on the second image frame to obtain a text frame in the second image frame;

[0138] Perform text recognition on the text box in the first image frame to obtain the first text content, and perform text recognition on the text box in the second image frame to obtain the second text content.

[0139] In yet another embodiment, the determining unit 602 is further configured to execute:

[0140] If the image similarity between the first image feature and the second image feature is less than or equal to an image similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is dissimilar;

[0141] If the image similarity between the first image feature and the second image feature is greater than the image similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be similar.

[0142] In another embodiment, the video to be processed includes N image frames, the first image frame and the second image frame belong to the N image frames, and the N image frames are arranged in the order of the appearance time of each of the N image frames in the video to be processed, where N is an integer greater than 1; the processing unit 603 is further configured to execute:

[0143] Acquire a similarity relationship between every two adjacent image frames in the N image frames from the similarity relationship set;

[0144] Based on a similarity relationship between every two adjacent image frames in the N image frames, the N image frames are divided into M image frame groups; wherein each image frame group includes one or more image frames, and when any image frame group includes multiple image frames, the multiple image frames are continuous, and the similarity relationship between every two image frames is similar, and M is a positive integer, and M is less than or equal to N;

[0145] An image frame is obtained from each image frame group as a key image frame.

[0146] According to one embodiment of the present application, Figure 1 and Figure 3 The steps involved in the method shown can be performed by Figure 6 The image processing apparatus 60 shown in FIG. Figure 1 The step S101 shown can be performed by Figure 6 The image processing device 60 shown in FIG. 6 is executed by the acquisition unit 601; Steps S102 to S103 can be performed by Figure 6 The image processing device 60 shown in FIG. 6 is used to determine the unit 602; step S104 can be performed by Figure 6 The processing unit 603 in the image processing device 60 shown in FIG. Figure 3 Step S301 shown can be performed by Figure 6 The image processing device 60 shown in FIG. 6 is executed by the acquisition unit 601; Steps S302 to S304 can be performed by Figure 6 The processing unit 603 in the image processing device 60 shown in FIG. 6 is used to perform the operation; steps S305 to S309 can be performed by Figure 6 The image processing device 60 shown in FIG. 6 is used to determine the unit 602; step S310 can be performed by Figure 6 The processing unit 603 in the image processing device 60 shown is executed.

[0147] According to another embodiment of the present application, Figure 6 The various units in the image processing device 60 shown are divided based on logical functions. The above-mentioned units can be individually or all combined into one or more other units to form a structure, or one (or some) of the units can be further divided into multiple functionally smaller units to form a structure, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. In other embodiments of the present application, the above-mentioned image processing device 60 can also include other units. In actual applications, these functions can also be assisted by other units and can be achieved by the collaboration of multiple units.

[0148] According to another embodiment of the present application, the program can be executed by running on a general computing device such as a computer including a central processing unit (CPU), a random access memory (RAM), a read-only memory (ROM) and other processing elements and storage elements. Figure 1 or Figure 3 A computer program (including program code) for each step of the method shown is used to construct Figure 6The image processing apparatus 60 shown in the figure can be used to implement the image processing method of the embodiment of the present application. The computer program can be recorded on a computer storage medium, for example, and loaded into the computing device through the computer storage medium and run therein.

[0149] In the present application, the image processing device determines the similarity between two image frames based on text features when both adjacent image frames include text content, so that the method provided by the present application can recall key image frames with text content; and because the image processing device can determine the similarity between two image frames based on image features when both adjacent image frames do not include text content, the present application takes into account the situation where the image processing device determines the similarity between two image frames when the image frames only include image content, thereby improving the accuracy of the image processing device in extracting key image frames from video files to a certain extent.

[0150] Based on the relevant descriptions of the above method embodiments and apparatus embodiments, the present application also provides a computer device, see Figure 7 The computer device 70 at least includes a processor 701 and a computer storage medium 702 , and the processor 701 and the computer storage medium 702 of the computer device 70 may be connected via a bus or other means.

[0151] Among them, the computer storage medium 702 mentioned above is a memory device in the computer device 70, which is used to store programs and data. It is understandable that the computer storage medium 702 here can include both the built-in storage medium in the computer device 70 and the extended storage medium supported by the computer device 70. The computer storage medium 702 provides a storage space, which stores the operating system of the computer device 70. In addition, one or more computer programs suitable for being loaded and executed by the processor 701 are also stored in the storage space. These computer programs can be one or more program codes. It should be noted that the computer storage medium here can be a high-speed RAM memory or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one computer storage medium located away from the aforementioned processor. The processor 701 (or CPU (Central Processing Unit)) is the computing core and control core of the computer device 70, which is suitable for implementing one or more computer programs, specifically suitable for loading and executing one or more computer programs to implement the corresponding method flow or corresponding function.

[0152] In one embodiment, the processor 701 may load and execute one or more computer programs stored in the computer storage medium 702 to implement the above-mentioned Figure 1 and Figure 3 The corresponding method steps in the method embodiment shown; in a specific implementation, one or more computer programs in the computer storage medium 702 are loaded by the processor 701 and execute the following steps:

[0153] Acquire a first image frame and a second image frame adjacent to each other in the video to be processed; if both the first image frame and the second image frame include text content, determine a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; if neither the first image frame nor the second image frame includes text content, determine a similarity relationship between the first image frame and the second image feature of the second image frame; based on the similarity relationship between the first image frame and the second image frame, update a similarity relationship set of the video to be processed, the similarity relationship set being used to record a similarity relationship between any two adjacent image frames in the video to be processed, and the similarity relationship recorded in the similarity relationship set being used to determine a key image frame of the video to be processed.

[0154] In one embodiment, the processor 701 is further configured to load and execute:

[0155] If either the first image frame or the second image frame includes text content, and the other image frame does not include text content, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0156] In yet another embodiment, the first text feature includes a first number of text boxes in the first image frame, and the second text feature includes a second number of text boxes in the second image frame. One text box is used to represent a text display area, and the text display area is used to display text content. The processor 701 is further used to load and execute:

[0157] If the first number and the second number are different, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0158] In yet another embodiment, the first text feature further includes first position information of a text box in the first image frame, and the second text feature further includes second position information of a text box in the second image frame; the processor 701 is further configured to load and execute:

[0159] If the first number and the second number are the same, determining a matching relationship between the first location information and the second location information;

[0160] If the matching relationship between the first position information and the second position information is not matching, it is determined that the similarity relationship between the first image frame and the second image frame is not similar.

[0161] In yet another embodiment, the first text feature further includes first text content in a text box in the first image frame, and the second text feature further includes second text content in a text box in the second image frame; the processor 701 is further configured to load and execute:

[0162] If the matching relationship between the first position information and the second position information is a match, determining the text similarity between the first text content and the second text content;

[0163] If the text similarity is greater than a text similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is similar;

[0164] If the text similarity is less than or equal to the text similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

[0165] In yet another embodiment, the processor 701 is further configured to load and execute:

[0166] Adjusting the first image frame and the second image frame to a consistent size;

[0167] Performing text region detection on the first image frame to obtain a text frame in the first image frame; and performing text region detection on the second image frame to obtain a text frame in the second image frame;

[0168] Perform text recognition on the text box in the first image frame to obtain the first text content, and perform text recognition on the text box in the second image frame to obtain the second text content.

[0169] In yet another embodiment, the processor 701 is further configured to load and execute:

[0170] If the image similarity between the first image feature and the second image feature is less than or equal to an image similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is dissimilar;

[0171] If the image similarity between the first image feature and the second image feature is greater than the image similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be similar.

[0172] In another embodiment, the video to be processed includes N image frames, the first image frame and the second image frame belong to the N image frames, and the N image frames are arranged in the order of the appearance time of each of the N image frames in the video to be processed, where N is an integer greater than 1; the processor 701 is further used to load and execute:

[0173] Acquire a similarity relationship between every two adjacent image frames in the N image frames from the similarity relationship set;

[0174] Based on a similarity relationship between every two adjacent image frames in the N image frames, the N image frames are divided into M image frame groups; wherein each image frame group includes one or more image frames, and when any image frame group includes multiple image frames, the multiple image frames are continuous, and the similarity relationship between every two image frames is similar, and M is a positive integer, and M is less than or equal to N;

[0175] An image frame is obtained from each image frame group as a key image frame.

[0176] In the present application, the computer device determines the similarity between two image frames based on text features when both adjacent image frames include text content, so that the method provided by the present application can recall key image frames with text content; and because the computer device can determine the similarity between two image frames based on image features when both adjacent image frames do not include text content, the present application takes into account the situation where the computer device determines the similarity between two image frames when the image frames only include image content, thereby improving the accuracy of the computer device in extracting key image frames from video files to a certain extent.

[0177] The present application also provides a computer storage medium storing a computer program for the above-mentioned image processing method. The computer program includes program instructions. When one or more processors load and execute the program instructions, the image processing method described in the embodiment can be implemented, which is not repeated here. The description of the beneficial effects of adopting the same method is not repeated here. It is understood that the program instructions can be deployed and executed on one or more devices that can communicate with each other.

[0178] It should be noted that, according to one aspect of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor in a computer device reads the computer instructions from the computer-readable storage medium and then executes the computer instructions, thereby enabling the computer device to perform the above-mentioned Figure 1 and Figure 3 The image processing method shown in the embodiment is provided in various optional manners.

[0179] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium. When executed, the computer program can include the processes in the above-described image processing method embodiments. The computer-readable storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0180] The above disclosure is only a partial embodiment of the present application, and it is certainly not intended to limit the scope of the rights of the present application. A person skilled in the art can understand that all or part of the processes of the above embodiments and equivalent changes made in accordance with the claims of the present application are still within the scope of the invention.

Claims

1. An image processing method, characterized in that: include: Obtaining a first image frame and a second image frame adjacent to each other in a video to be processed; the video to be processed includes N image frames, the first image frame and the second image frame belong to the N image frames, and the N image frames are arranged in a chronological order of appearance of each image frame in the video to be processed, where N is an integer greater than 1; According to whether both the first image frame and the second image frame include text content, a corresponding similarity relationship determination strategy is selected to determine a similarity relationship between the first image frame and the second image frame; wherein the similarity relationship determination strategy includes: If both the first image frame and the second image frame include text content, determining a similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; If neither the first image frame nor the second image frame includes text content, determining a similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame; If any one of the first image frame and the second image frame includes text content, and the other image frame does not include text content, determining that the similarity relationship between the first image frame and the second image frame is dissimilar; Based on the similarity relationship between the first image frame and the second image frame, updating the similarity relationship set of the video to be processed, the similarity relationship set being used to record the similarity relationship between any two adjacent image frames in the video to be processed, the similarity relationship set including an identifier corresponding to each image frame in the N image frames, if the identifiers of the any two adjacent image frames are the same, then the similarity relationship between the any two adjacent image frames is similar; if the identifiers of the any two adjacent image frames are different, then the similarity relationship between the any two adjacent image frames is dissimilar; the similarity relationship recorded in the similarity relationship set is used to determine the key image frame of the video to be processed; Acquire a similarity relationship between every two adjacent image frames in the N image frames from the similarity relationship set; Based on a similarity relationship between every two adjacent image frames in the N image frames, the N image frames are divided into M image frame groups; wherein each image frame group includes one or more image frames, and when any image frame group includes multiple image frames, the multiple image frames are continuous, and the similarity relationship between every two image frames is similar, and M is a positive integer, and M is less than or equal to N; An image frame is obtained from each image frame group as a key image frame.

2. The method according to claim 1, characterized in that The first text feature includes a first number of text boxes in the first image frame, and the second text feature includes a second number of text boxes in the second image frame, where one text box represents a text display area, and the text display area is used to display text content. Determining the similarity relationship between the first image frame and the second image frame based on the first text feature of the first image frame and the second text feature of the second image frame includes: If the first number and the second number are different, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

3. The method according to claim 2, characterized in that The first text feature further includes first position information of a text box in the first image frame, and the second text feature further includes second position information of a text box in the second image frame; and determining the similarity relationship between the first image frame and the second image frame based on the first text feature of the first image frame and the second text feature of the second image frame includes: If the first number and the second number are the same, determining a matching relationship between the first location information and the second location information; If the matching relationship between the first position information and the second position information is not matching, it is determined that the similarity relationship between the first image frame and the second image frame is not similar.

4. The method according to claim 3, characterized in that The first text feature further includes first text content of a text box in the first image frame, and the second text feature further includes second text content of a text box in the second image frame; and determining the similarity relationship between the first image frame and the second image frame based on the first text feature of the first image frame and the second text feature of the second image frame further includes: If the matching relationship between the first position information and the second position information is a match, determining the text similarity between the first text content and the second text content; If the text similarity is greater than a text similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is similar; If the text similarity is less than or equal to the text similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be dissimilar.

5. The method according to claim 4, characterized in that The method further comprises: Adjusting the first image frame and the second image frame to a consistent size; Performing text region detection on the first image frame to obtain a text frame in the first image frame; and performing text region detection on the second image frame to obtain a text frame in the second image frame; Perform text recognition on the text box in the first image frame to obtain the first text content, and perform text recognition on the text box in the second image frame to obtain the second text content.

6. The method according to claim 1, wherein The determining, based on a first image feature of the first image frame and a second image feature of the second image frame, a similarity relationship between the first image frame and the second image frame includes: If the image similarity between the first image feature and the second image feature is less than or equal to an image similarity threshold, determining that the similarity relationship between the first image frame and the second image frame is dissimilar; If the image similarity between the first image feature and the second image feature is greater than the image similarity threshold, the similarity relationship between the first image frame and the second image frame is determined to be similar.

7. An image processing device, characterized in that: include: an acquisition unit, configured to acquire a first image frame and a second image frame adjacent to each other in a video to be processed; the video to be processed includes N image frames, the first image frame and the second image frame belong to the N image frames, the N image frames are arranged in a chronological order of appearance of each of the N image frames in the video to be processed, where N is an integer greater than 1; a determination unit, configured to select a corresponding similarity determination strategy to determine a similarity relationship between the first image frame and the second image frame based on whether both the first image frame and the second image frame include text content; wherein the similarity determination strategy includes: if both the first image frame and the second image frame include text content, determining the similarity relationship between the first image frame and the second image frame based on a first text feature of the first image frame and a second text feature of the second image frame; if neither the first image frame nor the second image frame includes text content, determining the similarity relationship between the first image frame and the second image frame based on a first image feature of the first image frame and a second image feature of the second image frame; and if either the first image frame or the second image frame includes text content and the other does not include text content, determining the similarity relationship between the first image frame and the second image frame as dissimilar; a processing unit, configured to update a similarity relationship set of the video to be processed based on the similarity relationship between the first image frame and the second image frame, wherein the similarity relationship set is used to record the similarity relationship between any two adjacent image frames in the video to be processed, and the similarity relationship set includes an identifier corresponding to each image frame in the N image frames, and if the identifiers of the any two adjacent image frames are the same, then the similarity relationship between the any two adjacent image frames is similar; if the identifiers of the any two adjacent image frames are different, then the similarity relationship between the any two adjacent image frames is dissimilar; the similarity relationship recorded in the similarity relationship set is used to determine a key image frame of the video to be processed; The processing unit is further used to obtain the similarity relationship between every two adjacent image frames in the N image frames from the similarity relationship set; based on the similarity relationship between every two adjacent image frames in the N image frames, divide the N image frames into M image frame groups; wherein each image frame group includes one or more image frames, when any image frame group includes multiple image frames, the multiple image frames are continuous, and the similarity relationship between every two image frames is similar, M is a positive integer, and M is less than or equal to N; and obtain one image frame from each image frame group as a key image frame.

8. A computer device, characterized in that: The computer device comprises: a processor adapted to execute one or more computer programs; A computer storage medium storing one or more computer programs, wherein the one or more computer programs are suitable for being loaded by the processor and executing the image processing method according to any one of claims 1 to 6.

9. A computer program product comprising a computer program, characterized in that The computer program is stored in a computer storage medium and is suitable for being read and executed by a processor of a computer device, so that the computer device executes the image processing method according to any one of claims 1 to 6.

Images