Video image detection method, device, and electronic equipment
By acquiring the feature map of the video image and calculating the similarity ratio of its hash matrix, it automatically detects whether the video frame is a static frame, solving the problems of misjudgment and manual operation in the prior art, and improving the detection efficiency.
Patent Information
- Application Number
- CN202010879412.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-27
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2040-08-27
AI Technical Summary
The existing video image detection methods tend to ignore changes in video frames within a small range, resulting in misjudgment, and require manual operations to affect the user experience.
By obtaining the feature map of the video image to be detected, the hash matrix is determined according to the discrete cosine transformation rule, and the similarity ratio between the feature map and the target feature map is calculated based on the hash matrix. If the similarity ratio is greater than the preset threshold, it is judged as video Still frame image.
It realizes automatic detection of whether there are still frames in video frames, saving manpower review costs and improving audit efficiency.
Smart Images

Figure CN112115295B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and more specifically, to a video image detection method, device, electronic device and storage medium. Background Art
[0002] With the development of multimedia technology, users can shoot and share videos anytime and anywhere. In order to ensure the correctness of the shared videos and attract more users to watch, the videos can be tested before they are released, such as detecting whether there are video still frames (video still frames include green screen, black screen, frozen and other error types of video images), so that the videos can be processed in time according to the detection results. However, the existing detection methods are prone to ignore the changes in video frames in a small range and thus make misjudgments, and most of them require manual operation, which seriously affects the user experience. Summary of the invention
[0003] In view of the above problems, the present application proposes a video image detection method, device, electronic device and storage medium to improve the above problems.
[0004] In a first aspect, an embodiment of the present application provides a video image detection method, the method comprising: obtaining a video image to be detected; obtaining a feature map corresponding to the video image to be detected; determining a hash matrix corresponding to the feature map according to a discrete cosine transform rule; calculating a similarity ratio between the feature map and a target feature map based on the hash matrix; if the similarity ratio is greater than a preset threshold, using the video image to be detected as a video still frame image.
[0005] In the second aspect, an embodiment of the present application provides a video image detection device, which includes: a first acquisition module, used to acquire a video image to be detected; a second acquisition module, used to acquire a feature map corresponding to the video image to be detected; a determination module, used to determine a hash matrix corresponding to the feature map according to a discrete cosine transform rule; a calculation module, used to calculate a similarity ratio between the feature map and a target feature map based on the hash matrix; and a detection module, used to use the video image to be detected as a video still frame image if the similarity ratio is greater than a preset threshold.
[0006] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory and one or more processors; one or more programs are stored in the memory and configured to be executed by one or more processors, and the one or more programs are configured to execute the method described in the first aspect above.
[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, wherein when the program code is executed by a processor, the method described in the first aspect is executed.
[0008] A video image detection method, device, electronic device and storage medium provided by the embodiment of the present application. The method obtains the video image to be detected, and then obtains the feature map corresponding to the video image to be detected, and then determines the hash matrix corresponding to the feature map according to the discrete cosine transform rule, and then calculates the similarity ratio between the feature map and the target feature map based on the hash matrix. If the similarity ratio is greater than the preset threshold, the video image to be detected is used as a video still frame image. Thus, through the above-mentioned method, when the feature map corresponding to the video image to be detected is obtained, the hash matrix corresponding to the feature map can be determined according to the discrete cosine transform rule, and then the similarity ratio between the feature map and the target feature map can be calculated based on the hash matrix. When the similarity ratio is greater than the preset threshold, the video image to be detected can be used as a video still frame image, and the automatic review of whether the video frame image has a still frame is realized, which saves the manpower review cost of short video operation and improves the review efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.
[0010] Figure 1 A schematic diagram of an application environment provided by an embodiment of the present application is shown.
[0011] Figure 2 A method flow chart of a video image detection method provided by an embodiment of the present application is shown.
[0012] Figure 3 Shows Figure 2 Flow chart of the method of step S120 in .
[0013] Figure 4 A schematic diagram of the structure of a specified neural network model provided in an embodiment of the present application is shown.
[0014] Figure 5 An example diagram of mean merging of feature maps to be processed provided in an embodiment of the present application is shown.
[0015] Figure 6 Shows Figure 2 Flow chart of the method of step S130 in .
[0016] Figure 7 An example diagram of the processing process for obtaining a hash matrix corresponding to a feature map corresponding to a video image to be detected provided in an embodiment of the present application is shown.
[0017] Figure 8 Shows Figure 2 Flow chart of the method of step S140 in .
[0018] Fig. 9 An example diagram of the detection process for still frame detection of a video image provided by an embodiment of the present application is shown.
[0019] Fig.10 A method flow chart of a video image detection method provided by another embodiment of the present application is shown.
[0020] Fig.11 A structural block diagram of a video image detection device provided in an embodiment of the present application is shown.
[0021] Fig.12 A structural block diagram of an electronic device provided in an embodiment of the present application is shown.
[0022] Fig.13 A storage unit for storing or carrying program codes for implementing the video image detection method according to an embodiment of the present application is shown. DETAILED DESCRIPTION
[0023] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0024] With the development of multimedia technology, users can shoot and share videos anytime and anywhere. In order to ensure the accuracy of the shared videos and attract more users to watch, the videos can be tested before they are put online, such as detecting whether there are still frames in the video, so that the video can be processed in time according to the detection results. However, the existing detection methods are prone to ignore the changes of video frames in a small range and thus cause misjudgment, and most of them require manual operation, which seriously affects the user experience.
[0025] In order to improve the above-mentioned problems, the inventors have found through long-term research that it is possible to obtain a video image to be detected, and then obtain a feature map corresponding to the video image to be detected, and then determine the hash matrix corresponding to the feature map according to the discrete cosine transform rule, and then calculate the similarity ratio between the feature map and the target feature map based on the hash matrix. If the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image. Thus, in the above manner, it is achieved that when the feature map corresponding to the video image to be detected is obtained, the hash matrix corresponding to the feature map can be determined according to the discrete cosine transform rule, and then the similarity ratio between the feature map and the target feature map can be calculated based on the hash matrix. When the similarity ratio is greater than the preset threshold, the video image to be detected can be used as a video still frame image, and automatic review of whether the video frame image has a still frame is achieved, saving the manpower review cost of short video operation, and thus improving the review efficiency. Therefore, a video image detection method, device, electronic device and storage medium provided in an embodiment of the present application are proposed.
[0026] In order to facilitate the detailed description of the present application, an application environment in an embodiment of the present application is first introduced below with reference to the accompanying drawings.
[0027] See also Figure 1 , is a schematic diagram of an application environment of a video image detection method provided in an embodiment of the present application, such as Figure 1 As shown, the application environment can be understood as a network system 10 provided in an embodiment of the present application, and the network system 10 includes: a first user terminal 11, a server 12 and a second user terminal 13. Among them, the server 12 can be a server (network access server), or a server cluster (cloud server) composed of several servers, or a cloud computing center (database server). The first user terminal 11 and the second user terminal 13 can be any device with communication and storage functions, including but not limited to PC (Personal Computer), PDA (tablet computer), smart TV, smart phone, smart wearable device or other smart communication device with network connection function.
[0028] In this embodiment, the first user terminal 11 can be used to record or shoot a short video, and share the recorded or shot short video. The server 12 is used to review the short video shot by the user through the first user terminal 11 (i.e., detect whether there are erroneous video frame images such as video still frames), and push the short video that has passed the review to the logged-in user of the second user terminal 13 for viewing. Optionally, the number of the first user terminal 11 and the second user terminal 13 can be one or more, and there is no specific limitation. Optionally, in some embodiments, the first user terminal 11 and the second user terminal 13 can be the same user terminal; in other embodiments, the first user terminal 11 and the second user terminal 13 can be different user terminals. In this way, when the short video shot by the first user terminal 11 is reviewed and approved by the server 12, the server 12 can push the short video that has passed the review to the first user terminal 11 and the second user terminal 13.
[0029] The video image detection method provided in this embodiment can be applied to short video sharing scenarios. Before a short video is shared, the server can automatically perform still frame detection on the short video, and the short video will be pushed only after the detection is passed, thereby avoiding affecting the user viewing experience of the short video due to the existence of erroneous video frame images such as video still frames. At the same time, the server can automatically detect whether the short video includes video still frames, which can greatly reduce the labor review cost and thus improve the review efficiency.
[0030] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.
[0031] See also Figure 2 , shows a flow chart of a video image detection method provided by an embodiment of the present application. This embodiment provides a video image detection method that can be applied to a server. The method includes:
[0032] Step S110: Acquire the video image to be detected.
[0033] Optionally, the video image to be detected can be a video or short video that is about to be launched (or released), for example, a short video shot by a user can be used as the video image to be detected. Optionally, the video image to be detected can be a video image shot and stored in advance, or a video image shot in real time.
[0034] As a method, before the video is pushed to the user, the video to be pushed can be obtained as the video image to be detected, so that before the video (or short video) is pushed to the user, it can be detected whether there is a state such as a still picture or a black screen, and then the video can be processed according to the detection result to improve the user's viewing experience. Optionally, the number of image frames and the length of the content of the video image to be detected can be unlimited, for example, it can be a certain 50-frame image in a video (the specific number of frames can be unlimited, for example, it can also be a frame of image), or it can be a whole one-minute short video.
[0035] Step S120: Acquire a feature map corresponding to the video image to be detected.
[0036] In this embodiment, the feature map is a feature map of width and height dimensions. Different feature maps corresponding to different video images to be detected may be different. The specific acquisition process of the feature map corresponding to the video image to be detected is described as follows:
[0037] See also Figure 3 As one approach, step S120 may include:
[0038] Step S121: Decode the video image to be detected, input the decoded video image to be detected into a specified neural network model, and obtain a feature map to be processed output by the specified neural network model.
[0039] Optionally, in order to facilitate statistics on the reasons for the stillness or black screen of the video image to be detected, the video image to be detected can be decoded, and then the decoded video image to be detected is input into a specified neural network model to obtain a feature map to be processed output by the specified neural network model. Among them, the relevant principles and specific decoding process of decoding the video image to be detected can refer to the relevant technology, which will not be repeated here.
[0040] Optionally, the specified neural network model may be a CNN pre-trained model, such as an InceptionResNetv2 model. For some specific implementations, see Figure 4 , which shows a schematic diagram of the structure of a specified neural network model provided in an embodiment of the present application, such as Figure 3As shown, the specified neural network model can be the first 5 layers of the CNN pre-trained model, that is, the specified neural network model in this embodiment can include convolution layer 1, convolution layer 2, convolution layer 3, two-dimensional pooling layer 1, convolution layer 4, convolution layer 5 and two-dimensional pooling layer 2. Among them, convolution layer 1 can include two-dimensional convolution layer 1, batch normalization layer 1 and activation layer 1; convolution layer 2 can include two-dimensional convolution layer 2, batch normalization layer 2 and activation layer 2; convolution layer 3 can include two-dimensional convolution layer 3, batch normalization layer 3 and activation layer 3; convolution layer 4 can include two-dimensional convolution layer 4, batch normalization layer 4 and activation layer 4; convolution layer 5 can include two-dimensional convolution layer 5, batch normalization layer 5 and activation layer 5.
[0041] As a method, the decoded video image to be detected can be input into the convolution layer 1 of the specified neural network model, and then the output of the convolution layer 1 is input into the convolution layer 2, the output of the convolution layer 2 is input into the convolution layer 3, the output of the convolution layer 3 is input into the two-dimensional pooling layer 1, the output of the two-dimensional pooling layer 1 is input into the convolution layer 4, the output of the convolution layer 4 is input into the convolution layer 5, the output result of the convolution layer 5 is input into the two-dimensional pooling layer 2, and the output result of the two-dimensional pooling layer 2 is used as the feature map to be processed output by the specified neural network model. Optionally, the feature map to be processed can be a feature map of three dimensions, for example, it can be a feature map of HxWxC dimensions. For example, in a specific application scenario, assuming that the video image to be detected is a, a can be scaled to 224x224x3 dimensions. After a is input into the specified neural network model, the feature map dimension of each frame image can be output as a feature map to be processed of 25x25x192.
[0042] Step S122: Calculate the mean value corresponding to the feature map to be processed along the specified direction to obtain the feature map corresponding to the video image to be detected.
[0043] Optionally, the specified direction may be the third dimension direction of the feature map to be processed. For example, if the feature map to be processed is an HxWxC-dimensional feature map, the specified direction may be the C-dimensional direction. As a method, the mean corresponding to the feature map to be processed may be calculated along the specified direction, and the feature map obtained by merging the mean values may be used as the feature map corresponding to the video image to be detected.
[0044] For example, in a specific application scenario, see Figure 5 , shows an example diagram of mean merging of feature maps to be processed provided by an embodiment of the present application, such as Figure 5 As shown, if the feature map to be processed is a HxWxC-dimensional feature map (which may include multiple images), the mean of the feature map to be processed in the H dimension and the W dimension can be calculated along the C dimension, and the calculated means can be merged, and the merged HxW-dimensional feature map can be used as the feature map corresponding to the video image to be detected.
[0045] Step S130: Determine a hash matrix corresponding to the feature map according to a discrete cosine transform rule.
[0046] Optionally, the discrete cosine transform rule in this embodiment (assuming it is represented by Fu) can be:
[0047]
[0048]
[0049] Wherein, u represents the index value of the index matrix, f(i) represents the value in the matrix position corresponding to the current index value i (for example, if i is 3, then f(i) is the corresponding matrix value), c(u) represents a variable coefficient (relative to u), and N represents the value of the HxW matrix. Optionally, in some embodiments, if the feature map is a single-frame image, the above-mentioned discrete cosine transform rule can be used for calculation.
[0050] See also Figure 6 As one approach, step S130 may include:
[0051] Step S131: Calculate a first matrix corresponding to the feature map along a first dimension according to a discrete cosine transform rule.
[0052] Optionally, the first dimension may be a width dimension. As a method, a first matrix corresponding to a feature map corresponding to the video image to be detected may be calculated along the width dimension according to the above discrete cosine transform rule. For example, in a specific embodiment, assuming that the first dimension is the H dimension of the aforementioned HxW dimension, a discrete cosine transform may be performed on the feature map corresponding to the video image to be detected along the H dimension according to the above discrete cosine transform rule, and the calculated matrix result may be used as the first matrix. For the specific calculation principle and calculation process, reference may be made to the relevant technology, which will not be described in detail here.
[0053] Step S132: Calculate a second matrix corresponding to the first matrix along a second dimension according to the discrete cosine transform rule.
[0054] Optionally, the second dimension may be a height dimension. If only a discrete cosine transform is performed on the feature map corresponding to the video image to be detected, if the image content variation range between different video frames is small, the obtained feature map may not accurately reflect whether the video image is a still frame image or whether it includes a still frame image. As a way to improve this problem, this embodiment can calculate a second matrix corresponding to the first matrix along the second dimension according to the discrete cosine transform rule, that is, perform another discrete cosine transform on the result after the first discrete cosine transform.
[0055] For example, based on the above example, as a method, the second matrix corresponding to the first matrix can be calculated along the W dimension according to the aforementioned discrete cosine transform rule. Exemplarily, assuming that the second matrix can be represented by TD, and the feature map corresponding to the video image to be detected can be represented by mf, the process of performing discrete cosine transform on the feature map corresponding to the video image to be detected in this embodiment can be expressed as the following formula:
[0056] TD=DCT(DCT(mf,axis=0),axis=1).
[0057] Wherein, DCT represents discrete cosine transform processing, axis=0 represents calculation along the H dimension, and axis=1 represents calculation along the W dimension.
[0058] Optionally, in some possible implementations, the feature map corresponding to the video image to be detected may be first discrete cosine transformed along the W dimension, and then the result of the processing along the W dimension may be subjected to a secondary discrete cosine transform along the H dimension. The specific order may not be limited.
[0059] Step S133: Obtain average values corresponding to the elements of the second matrix.
[0060] Optionally, the second matrix may include multiple elements, and the specific number of rows and columns of the second matrix may not be limited. As a method, the average value (which may be expressed as AVG) corresponding to the elements of the second matrix may be obtained based on the following formula:
[0061]
[0062] Among them, x i represents the value of index i in the aforementioned second matrix (the dimension for calculating the average value may not be limited), and N represents the value of the HxW matrix.
[0063] Step S134: Obtain a target matrix based on the second matrix and the average value.
[0064] As an implementation method, the target matrix (F(u,v)) can be obtained by subtracting the elements of the TD matrix (i.e., the aforementioned second matrix) from the aforementioned average value, obtaining a matrix after the elements are changed, and then using the obtained matrix as the target matrix. The specific calculation formula is as follows:
[0065] F(u,v)=TD(u,v)-AVG,u∈[0,H)and v∈[0,W)
[0066] Among them, u and v represent the index of each dimension respectively, H represents the H dimension, and W represents the W dimension. Optionally, the target matrix is a HxW dimensional matrix. In order to facilitate the subsequent calculation of the hash value, all elements in the target matrix can be judged, that is, whether the elements are greater than 0. The judgment formula is as follows:
[0067]
[0068] Optionally, if the element is greater than 0, the corresponding element is marked as 1, and if the element is less than 0, the corresponding element is marked as 0, and finally a HxW dimensional matrix including only elements 0 and 1 is obtained, and the HxW dimensional matrix including only elements 0 and 1 is used as the target matrix.
[0069] Step S135: Perform dimensionality reduction processing on the target matrix to obtain a hash matrix corresponding to the feature map.
[0070] Optionally, in order to improve the computational efficiency, the target matrix can be reduced to a one-dimensional matrix (in the process of dimensionality reduction, the horizontal dimension reduction can be performed first and then the vertical dimension reduction or the vertical dimension reduction can be performed first and then the horizontal dimension reduction, and the specific order can be not limited), and the one-dimensional matrix is used as the hash matrix corresponding to the feature map. Optionally, the specific principle of dimensionality reduction and the dimensionality reduction process can refer to the relevant technology, which will not be repeated here.
[0071] For example, in a specific application scenario, see Figure 7 , shows an example diagram of the processing process of obtaining a hash matrix corresponding to a feature map corresponding to a video image to be detected provided by an embodiment of the present application, such as Figure 7 As shown, the feature map corresponding to the video image to be detected (i.e. Figure 7 The merged feature map shown in the figure is first processed by a discrete cosine transform along the H dimension (i.e. Figure 7 As shown, a DCT is first performed along the H dimension), and then a discrete cosine transform is performed on the result along the W dimension (i.e. Figure 7 As shown, DCT is performed again along the W dimension to obtain a TD matrix, and then the elements in the TD matrix are averaged, each element in the TD matrix is subtracted from the average value, and the result after the subtraction is marked. Specifically, the elements in the result matrix after the subtraction whose values are greater than 0 can be marked as 1, and the elements in the result matrix after the subtraction whose values are less than 0 can be marked as 0, so that an HxW dimensional matrix including only elements 0 and 1 can be obtained, and then the HxW dimensional matrix can be reduced in dimensionality to obtain an HxW dimensional one-dimensional matrix.
[0072] Step S140: Calculating a similarity ratio between the feature map and a target feature map based on the hash matrix.
[0073] Optionally, the target feature map is a video frame different from the feature map. For example, assuming that the feature map is a feature map corresponding to the current video frame, the target feature map may be a feature map corresponding to a video frame before the current video frame, or may be a feature map corresponding to a video frame after the current video frame.
[0074] As a method, whether a video frame is a still frame can be determined by calculating the image content similarity between feature maps of different video frames. Specifically, the similarity ratio between the feature map and the target feature map can be calculated based on the hash matrix. If the similarity ratio is greater than a preset threshold, the current video frame can be determined to be a still frame, otherwise the current video frame is determined to be a non-still frame. The specific calculation process is as follows:
[0075] See also Figure 8 As one approach, step S140 may include:
[0076] Step S141: Obtain a hash value corresponding to the hash matrix.
[0077] Optionally, after obtaining the hash matrix, a hash value corresponding to the hash matrix can be obtained based on a related hash algorithm. The specific calculation process can refer to related technologies and will not be described in detail here.
[0078] Step S142: Subtract the hash value from the hash value corresponding to the target feature map to obtain a difference vector between the feature map and the target feature map.
[0079] Optionally, the difference vector includes the difference vectors between the current video frame and all other video frames. For example, in a specific implementation, assuming that a certain frame image is recorded as x, the hash vector value of x can be subtracted from the hash vector value of another frame image (assuming it is recorded as y) to obtain a vector difference of HxW dimensions, and the number of non-zero values of the vector difference is counted to obtain the difference value h between x and y (others), which is expressed as follows:
[0080] xy diff =x hash -y hash ;
[0081] h = count_nozero(xy diff ).
[0082] Optionally, if h is less than a set parameter threshold (such as 12), then x and y can be considered to be similar images and the value is set to 1, otherwise it is 0. The difference value vector hv between x and all image frames except x can be calculated by the above calculation method, with a dimension of 1x(N-1), where N represents the number of input video image frames. Optionally, the calculation formula of the difference value vector hv is as follows:
[0083]
[0084] Among them, h thresh represents the set parameter threshold, idx represents the index value of x in the image frame set, and h u represents the h value of the image frame x and index u, and N represents the number of all image frames. Optionally, the difference value vector hv(u) corresponding to different image frames is different.
[0085] Step S143: Obtaining a similarity ratio between the feature map and the target feature map based on the difference vector.
[0086] As a way, we can count the number of non-zero values of hv(u) and divide it by (N-1) to get the corresponding similarity ratio p, which is calculated as follows:
[0087]
[0088] Optionally, the target feature map can be a multi-frame image. If the similarity ratio is greater than a preset threshold, the maximum vector value in the similarity ratios corresponding to the multiple frames of the image can be obtained, wherein each frame of the image corresponds to a similarity ratio, and different similarity ratios correspond to different vector values. As a method, if the maximum vector value is greater than a preset threshold, the video image to be detected can be used as a video still frame image, thereby improving the detection accuracy of the video still frame. Specifically, as an implementation method, the similarity ratios corresponding to each frame of the image can be composed into a vector P, and the maximum value max_p in the vector P can be obtained (the calculation formula is as follows).
[0089] P=[p0,p1,...,p n ],n∈[0,N]P max =max(P).
[0090] Where N represents the number of all image frames, p n Represents the p value of the previous formula for the nth image frame.
[0091] Step S150: If the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image.
[0092] Optionally, the specific value of the preset threshold can be set according to actual conditions, for example, the preset threshold can be 0.9. As a way, if the similarity ratio is greater than the preset threshold, the video image to be detected can be used as a video still frame image. The specific judgment formula is as follows:
[0093]
[0094] Among them, 1 indicates that there is a still frame in the video, and 0 indicates that there is no still frame in the video. threshrepresents the similarity ratio threshold, p max Represents the maximum vector difference.
[0095] Below Fig. 9 Take this example to illustrate this embodiment:
[0096] like Fig. 9 As shown, after sending the video to the server, the server can decode the video according to the pre-stored decoding algorithm to obtain a batch of image frames, input the batch of image frames into the cropped InceptionResNetv2 model (i.e., the first 5 convolutional layers of the InceptionResNetv2 model, which can be pre-trained), and then use the improved perceptual hash algorithm to calculate the image hash value of the batch of image frames. The specific calculation process may include inputting the batch of video frame features, then performing two discrete cosine transforms on the video frame features, and then calculating the average value of the elements in the result matrix after the two discrete cosine transforms, calculating the difference between the result matrix after the two discrete cosine transforms and the average value, and then obtaining the hash value of the video frame according to the positive and negative conditions of the difference. Optionally, the hash values between different image frames can be compared to obtain the maximum similarity ratio, and then determine whether the maximum similarity ratio exceeds a threshold, and then determine whether the image frame is a video still frame, thereby achieving fast and effective video frame detection. The specific process can refer to the above description and will not be repeated here.
[0097] The video image detection method provided in this embodiment obtains the video image to be detected, and then obtains the feature map corresponding to the video image to be detected, and then determines the hash matrix corresponding to the feature map according to the discrete cosine transform rule, and then calculates the similarity ratio between the feature map and the target feature map based on the hash matrix. If the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image. Thus, through the above method, when the feature map corresponding to the video image to be detected is obtained, the hash matrix corresponding to the feature map can be determined according to the discrete cosine transform rule, and then the similarity ratio between the feature map and the target feature map can be calculated based on the hash matrix. When the similarity ratio is greater than the preset threshold, the video image to be detected can be used as a video still frame image, and automatic review of whether there is a still frame in the video frame image is achieved, saving the human review cost of short video operation, and thus improving the review efficiency.
[0098] See also Fig.10 , shows a flow chart of a video image detection method provided by another embodiment of the present application. This embodiment provides a video image detection method that can be applied to a server. The method includes:
[0099] Step S210: Acquire the video image to be detected.
[0100] Step S220: Obtain a feature map corresponding to the video image to be detected.
[0101] Step S230: Determine a hash matrix corresponding to the feature map according to a discrete cosine transform rule.
[0102] Step S240: Calculate a similarity ratio between the feature map and the target feature map based on the hash matrix.
[0103] Step S250: If the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image.
[0104] Step S260: filtering the video still frame image.
[0105] As a method, after determining that an image frame is a video still frame, the image frame can be filtered. For example, the push frequency of the video corresponding to the image frame can be reduced or its push cycle can be increased. Alternatively, the video can be pushed only to the user who provided the video, and not to other users, so as to avoid other users from seeing such videos including video still frames, so as to improve the user's viewing experience. Optionally, the video still frame can also be cropped, and then the video content can be analyzed. If the content correlation between the remaining image frames is greater than a specified threshold, the video can be pushed to other users. The specific value of the specified threshold is not limited.
[0106] The video image detection method provided in this embodiment realizes automatic review of whether there are still frames in the video frame images, saves the manpower review cost of short video operations, and further improves the review efficiency.
[0107] See also Fig.11 , is a structural block diagram of a video image detection device provided in an embodiment of the present application. This embodiment provides a video image detection device 300, which runs on a server. The device 300 includes: a first acquisition module 310, a second acquisition module 320, a determination module 330, a calculation module 340 and a detection module 350:
[0108] The first acquisition module 310 is used to acquire a video image to be detected.
[0109] The second acquisition module 320 is used to acquire a feature map corresponding to the video image to be detected.
[0110] As a method, the second acquisition module 320 can be used to decode the video image to be detected, input the decoded video image to be detected into a specified neural network model, and obtain the feature map to be processed output by the specified neural network model; calculate the mean corresponding to the feature map to be processed along the specified direction to obtain the feature map corresponding to the video image to be detected.
[0111] The determination module 330 is used to determine the hash matrix corresponding to the feature map according to the discrete cosine transform rule.
[0112] As a method, the determination module 330 can be used to calculate a first matrix corresponding to the feature map along a first dimension according to a discrete cosine transform rule; calculate a second matrix corresponding to the first matrix along a second dimension according to the discrete cosine transform rule; obtain an average value corresponding to elements of the second matrix; obtain a target matrix based on the second matrix and the average value; perform dimensionality reduction processing on the target matrix to obtain a hash matrix corresponding to the feature map.
[0113] Among them, obtaining the target matrix based on the second matrix and the average value may include: subtracting the elements of the second matrix from the average value respectively to obtain a matrix after the elements are changed; and using the matrix as the target matrix.
[0114] The calculation module 340 is used to calculate the similarity ratio between the feature map and the target feature map based on the hash matrix.
[0115] As a method, the calculation module 340 can be used to obtain a hash value corresponding to the hash matrix; subtract the hash value from the hash value corresponding to the target feature map to obtain a difference vector between the feature map and the target feature map; and obtain a similarity ratio between the feature map and the target feature map based on the difference vector.
[0116] The detection module 350 is configured to use the video image to be detected as a video still frame image if the similarity ratio is greater than a preset threshold.
[0117] Optionally, the target feature map can be a multi-frame image. In this way, the detection module 350 can be used to obtain the maximum vector value in the similarity ratios corresponding to the multiple frame images, each frame image corresponds to a similarity ratio, and different similarity ratios correspond to different vector values; if the maximum vector value is greater than a preset threshold, the video image to be detected is used as a video still frame image.
[0118] Optionally, the device 300 may further include a filtering processing module for performing filtering processing on the video still frame image.
[0119] The present embodiment provides a video image detection device, which obtains a video image to be detected, and then obtains a feature map corresponding to the video image to be detected, and then determines a hash matrix corresponding to the feature map according to a discrete cosine transform rule, and then calculates the similarity ratio between the feature map and the target feature map based on the hash matrix. If the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image. Thus, through the above method, when the feature map corresponding to the video image to be detected is obtained, the hash matrix corresponding to the feature map can be determined according to the discrete cosine transform rule, and then the similarity ratio between the feature map and the target feature map can be calculated based on the hash matrix. When the similarity ratio is greater than the preset threshold, the video image to be detected can be used as a video still frame image, and automatic review of whether there is a still frame in the video frame image is achieved, saving the human review cost of short video operation, and thus improving the review efficiency.
[0120] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.
[0121] In several embodiments provided in the present application, the coupling or direct coupling or communication connection between the modules shown or discussed may be an indirect coupling or communication connection through some interfaces, devices or modules, which may be electrical, mechanical or other forms.
[0122] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.
[0123] See also Fig.12 Based on the above-mentioned video image detection method and device, the embodiment of the present application also provides an electronic device 100 that can execute the above-mentioned video image detection method. The electronic device 100 includes a memory 102 and one or more (only one is shown in the figure) processors 104 coupled to each other, and the memory 102 and the processor 104 are connected by a communication line. The memory 102 stores a program that can execute the content of the above-mentioned embodiment, and the processor 104 can execute the program stored in the memory 102.
[0124] Among them, the processor 104 may include one or more processing cores. The processor 104 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 104 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 104 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing display content; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 104, but may be implemented separately through a communication chip.
[0125] The memory 102 may include a random access memory (RAM) or a read-only memory (ROM). The memory 102 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 102 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the aforementioned embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.
[0126] Please refer to Fig.13 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable storage medium 400 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.
[0127] The computer readable storage medium 400 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 400 includes a non-transitory computer-readable storage medium. The computer readable storage medium 400 has storage space for program code 410 that performs any method step of the above method. These program codes can be read from or written to one or more computer program products. The program code 410 can be compressed, for example, in an appropriate form.
[0128] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.
[0129] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A video image detection method, characterized in that: The method comprises: Acquire the video image to be detected; Acquire a feature map corresponding to the video image to be detected; Determine a hash matrix corresponding to the feature map according to a discrete cosine transform rule, wherein the determining the hash matrix corresponding to the feature map according to the discrete cosine transform rule includes: calculating a first matrix corresponding to the feature map along a first dimension according to the discrete cosine transform rule, wherein the first dimension is a width dimension; calculating a second matrix corresponding to the first matrix along a second dimension according to the discrete cosine transform rule, wherein the second dimension is a height dimension; obtaining an average value corresponding to elements of the second matrix; obtaining a target matrix based on the second matrix and the average value; and performing dimensionality reduction processing on the target matrix to obtain a hash matrix corresponding to the feature map; The similarity ratio between the feature map and the target feature map is calculated based on the hash matrix, and the similarity ratio between the feature map and the target feature map is calculated based on the hash matrix, including: obtaining a hash value corresponding to the hash matrix; subtracting the hash value from the hash value corresponding to the target feature map to obtain a difference vector between the feature map and the target feature map; obtaining the similarity ratio between the feature map and the target feature map based on the difference vector, wherein the calculation formula for obtaining the similarity ratio between the feature map and the target feature map based on the difference vector is as follows: Wherein, p represents a similarity ratio, N represents the number of all image frames, and hv(u) represents a difference vector; If the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image; The method further comprises: The video still frame image is filtered.
2. The method according to claim 1, characterized in that The acquiring a target matrix based on the second matrix and the average value includes: Subtracting the elements of the second matrix from the average value respectively to obtain a matrix after the elements are changed; The matrix is taken as the target matrix.
3. The method according to claim 1, characterized in that The target feature map is a multi-frame image, and if the similarity ratio is greater than a preset threshold, the video image to be detected is used as a video still frame image, including: Obtaining the maximum vector value in the similarity ratios respectively corresponding to the multiple frames of images, each frame of image corresponds to a similarity ratio, and different similarity ratios correspond to different vector values; If the maximum vector value is greater than a preset threshold, the video image to be detected is used as a video still frame image.
4. The method according to claim 1, characterized in that: The acquiring of a feature map corresponding to the video image to be detected includes: Decoding the video image to be detected, inputting the decoded video image to be detected into a specified neural network model, and obtaining a feature map to be processed output by the specified neural network model; The mean value corresponding to the feature map to be processed is calculated along the specified direction to obtain the feature map corresponding to the video image to be detected.
5. A video image detection device, characterized in that: The device comprises: A first acquisition module, used to acquire a video image to be detected; A second acquisition module is used to acquire a feature map corresponding to the video image to be detected; A determination module, used to determine a hash matrix corresponding to the feature map according to a discrete cosine transform rule, wherein the determining the hash matrix corresponding to the feature map according to the discrete cosine transform rule includes: calculating a first matrix corresponding to the feature map along a first dimension according to the discrete cosine transform rule, wherein the first dimension is a width dimension; calculating a second matrix corresponding to the first matrix along a second dimension according to the discrete cosine transform rule, wherein the second dimension is a height dimension; obtaining an average value corresponding to elements of the second matrix; obtaining a target matrix based on the second matrix and the average value; and performing dimensionality reduction processing on the target matrix to obtain a hash matrix corresponding to the feature map; A calculation module is used to calculate the similarity ratio between the feature map and the target feature map based on the hash matrix, wherein the calculation of the similarity ratio between the feature map and the target feature map based on the hash matrix includes: obtaining a hash value corresponding to the hash matrix; subtracting the hash value from the hash value corresponding to the target feature map to obtain a difference vector between the feature map and the target feature map; obtaining the similarity ratio between the feature map and the target feature map based on the difference vector, wherein the calculation formula for obtaining the similarity ratio between the feature map and the target feature map based on the difference vector is as follows: Wherein, p represents a similarity ratio, N represents the number of all image frames, and hv(u) represents a difference vector; A detection module, configured to use the video image to be detected as a video still frame image if the similarity ratio is greater than a preset threshold; The filtering processing module is used to filter the still frame image of the video.
6. An electronic device, characterized in that: including one or more processors and memory; One or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to execute the method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program code, wherein when the program code is executed by a processor, the method according to any one of claims 1 to 4 is executed.
