Video frame filtering method, device and equipment and computer medium

Abstract

The invention provides a video frame filtering method, device and equipment and a computer readable medium, and the method comprises the steps: obtaining a to-be-processed cut frame from a live video stream, and calculating the similarity between the to-be-processed cut frame and a historical submitted cut frame through employing a partition comparison method; and then, according to a similarity calculation result, it is determined that the to-be-processed cut frames are discarded or transferred to a machine review system, so that part of the cut frames can be discarded according to the similarity between the to-be-processed cut frames and historical submitted cut frames, the number of the cut frames transferred to the machine review system is reduced, the load of the machine review system is reduced, and the machine review efficiency is improved. Besides, the frame cutting is divided into a plurality of independent areas for similarity comparison, and the similarity of each area is calculated independently, so that the influence of a stable area in the live broadcast picture on a changing area can be avoided, the really changed part can be accurately identified, the accuracy of similar frame judgment is improved, and the user experience is improved. And the robustness and the efficiency are enhanced.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a video frame filtering method, device, equipment and computer-readable medium. Background Art

[0002] This section is intended to provide a background or context to the embodiments of the application that are recited in the claims. The description herein is not admitted to be prior art by inclusion in this section.

[0003] With the rapid development of the live streaming industry, the security audit business of live streaming platforms faces severe challenges. Traditional audit methods are inefficient and costly when dealing with massive live video streams. Specifically, the video stream of each live broadcast room needs to be cut into frames to form a series of static images so that a preliminary risk assessment can be performed through the machine audit model. If the model detects potential risks, the cut frames will be sent for manual review for further confirmation.

[0004] At present, due to the rapid growth of live broadcast volume, more and more frames need to be reviewed. The existing machine review technology seems to be unable to cope with high-frequency and large-scale frame data, especially during the peak period of live broadcast. A large number of frames flood into the review system, which can easily cause system bottlenecks, resulting in review delays or even omissions, affecting the content quality and user safety of the live broadcast platform.

[0005] In view of the above problems, it is necessary to provide a technical solution that can improve the efficiency of live broadcast machine review. Summary of the invention

[0006] Multiple aspects of the present application provide a video frame filtering method, apparatus, device and computer-readable storage medium to improve the efficiency of live broadcast machine review.

[0007] In one aspect of the present application, a video frame filtering method is provided, wherein the method comprises:

[0008] Get the frames to be processed from the live video stream;

[0009] The partition comparison method is used to calculate the similarity between the to-be-processed cut frame and the historical cut frame submitted for review, and a similarity calculation result is obtained;

[0010] Determine whether to discard the pending cut frame or transfer it to the machine review system according to the similarity calculation result.

[0011] Another aspect of the present application provides a video frame filtering device, wherein the device comprises:

[0012] A frame acquisition module is used to obtain the frame to be processed from the live video stream;

[0013] A frame comparison module is used to calculate the similarity between the frame to be processed and the historical frame submitted for review by using a partition comparison method to obtain a similarity calculation result;

[0014] The frame cutting filtering module is used to determine whether to discard the to-be-processed frame cutting or transfer it to the machine review system according to the similarity calculation result.

[0015] Another aspect of the present application provides an electronic device, the electronic device comprising:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the video frame filtering method as described above.

[0019] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the video frame filtering method as described above.

[0020] In the solution provided in the embodiment of the present application, after obtaining the to-be-processed cut frame from the live video stream, the partition comparison method is used to calculate the similarity between the to-be-processed cut frame and the historical cut frame submitted for review, and then according to the similarity calculation result, it is determined whether to discard the to-be-processed cut frame or transfer it to the machine review system. In this way, some cut frames can be discarded according to the degree of similarity between the to-be-processed cut frame and the historical cut frame submitted for review, thereby realizing pre-filtering of similar frames and reducing the number of cut frames transferred to the machine review system, thereby reducing the load of the machine review system and improving the efficiency of machine review. In addition, considering that in live broadcast scenes, video frames often contain some relatively stable areas (such as backgrounds, fixed decorative elements, etc.), and the parts that have actually changed (such as character movements, etc.) are the objects that need to be focused on. If the cut frames are compared for similarity as a whole, the high similarity in the stable areas may cause the overall similarity score to be high, thereby masking the actual changed parts. The present application divides the image into multiple independent areas for similarity comparison, and calculates the similarity of each area separately. In this way, the influence of stable areas on changing areas in the live broadcast picture can be avoided, ensuring that the parts that have actually changed can be accurately identified, thereby improving the accuracy of similar frame judgment, as well as enhancing robustness and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0022] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0023] Figure 1 A flowchart of a video frame filtering method provided in one embodiment of the present application;

[0024] Figure 2 A schematic diagram for describing frequency domain transformation provided in an embodiment of the present application;

[0025] Figure 3 A schematic diagram for describing a partitioned frequency domain transformation provided in an embodiment of the present application;

[0026] Figure 4 A schematic diagram of the structure of a video frame filtering device provided in one embodiment of the present application;

[0027] Figure 5 A schematic diagram of the structure of an electronic device suitable for implementing the solution in the embodiments of the present application;

[0028] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0031] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0032] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0033] Taking into account the challenges faced by the live broadcast machine review system when processing massive live video streams, the embodiments of the present application provide a video frame filtering method, device, equipment and computer-readable storage medium to improve the efficiency of live broadcast machine review. After obtaining the cut frame to be processed from the live video stream, the partition comparison method is used to calculate the similarity between the cut frame to be processed and the historical cut frame submitted for review, and then the cut frame to be processed is determined to be discarded or transferred to the machine review system based on the similarity calculation result. In this way, some cut frames can be discarded based on the similarity between the cut frame to be processed and the historical cut frame submitted for review, thereby achieving pre-filtering of similar frames and reducing the number of cut frames transferred to the machine review system, thereby reducing the load of the machine review system and improving the efficiency of the machine review. In addition, considering that in live broadcast scenes, video frames often contain some relatively stable areas (such as backgrounds, fixed decorative elements, etc.), and the parts that have actually changed (such as character movements, etc.) are the objects that need to be focused on. If the cut frames are compared for similarity as a whole, the high similarity in the stable areas may cause the overall similarity score to be high, thereby masking the actual changed parts. The present application divides the image into multiple independent areas for similarity comparison, and calculates the similarity of each area separately. In this way, the influence of stable areas on changing areas in the live broadcast picture can be avoided, ensuring that the parts that have actually changed can be accurately identified, thereby improving the accuracy of similar frame judgment, as well as enhancing robustness and efficiency.

[0034] The following is an exemplary description of the video frame filtering method, apparatus, device and computer-readable storage medium provided in the present application.

[0035] In actual scenarios, the execution subject of the video frame filtering method provided in the embodiment of the present application can be a user device, or a device formed by integrating a user device and a network device through a network, or it can also be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0036] Figure 1 The processing flow of a video frame filtering method provided by an embodiment of the present application is shown, and the method at least includes the following processing steps:

[0037] Step S101, obtaining a frame to be processed from a live video stream.

[0038] In live video processing, "cut frames" refer to single static images cut from a continuous video stream. When a video is transmitted or processed, image frames are usually extracted from the video stream at a certain frequency (such as a certain number of frames per second). These image frames are called "cut frames".

[0039] Frames play an important role in the live security review business because they are the basic unit for both machine and manual review. By breaking down the video stream into a series of frames, each frame can be independently assessed for risk. If the machine review system detects a potential risk, the frame will be sent to manual review for further confirmation.

[0040] Among them, the machine review system is a software system for automatically reviewing video content. In the live broadcast industry, it is mainly used to quickly process frame cuts in a large number of video streams. It cuts the video stream into a series of static images (i.e., frames), and then performs image processing, feature extraction, and model analysis on these frames to identify potential risk factors, such as pornography, violence, etc., and decides whether the frame cuts need further manual review or are directly approved / rejected based on the risk assessment results.

[0041] Step S102, using a partition comparison method to calculate the similarity between the to-be-processed cut frame and the historical cut frame submitted for review.

[0042] Among them, the partition comparison method refers to dividing the image into multiple independent areas and performing independent similarity comparison on each area. This method can avoid mutual interference between different areas and improve the accuracy of similarity comparison. Especially for live broadcast scenes, video frames often contain some relatively stable areas (such as backgrounds, fixed decorative elements, etc.), and the parts that have actually changed (such as character movements, etc.) are the objects that need to be focused on. If the cut frames are compared for similarity as a whole, the high similarity of the stable areas may cause the overall similarity score to be high, thereby masking the actual changed parts. The present application divides the image into multiple independent areas for similarity comparison, and calculates the similarity of each area separately. In this way, the influence of stable areas on changing areas in the live broadcast picture can be avoided, ensuring that the parts that have actually changed can be accurately identified, thereby improving the accuracy of similar frame judgment, as well as enhancing robustness and efficiency.

[0043] The above-mentioned historical submitted frames refer to the frames that were transferred to the machine review system for review before the pending frames, and submitted for review means sent to the machine review system for review.

[0044] Step S103: Determine whether to discard the to-be-processed cut frame or transfer it to a machine review system according to the similarity calculation result.

[0045] In this step, the similarity calculation result can be compared with a preset similarity threshold. If it is higher than the threshold, it is judged that the to-be-processed cut frame is highly similar to the historical cut frame submitted for review. Otherwise, it is judged that the to-be-processed cut frame is not similar enough to the historical cut frame submitted for review.

[0046] If it is determined based on the similarity calculation result that the to-be-processed cut frame is highly similar to the historically submitted cut frame, the to-be-processed cut frame can be discarded to reduce unnecessary machine review burden; on the contrary, if the to-be-processed cut frame is not similar enough to the historically submitted cut frame, there may be new content or risks, and the to-be-processed cut frame should be handed over to the machine review system for further review.

[0047] The above-mentioned video frame filtering method provided in the embodiment of the present application obtains the cut frame to be processed from the live video stream, and then uses the partition comparison method to calculate the similarity between the cut frame to be processed and the historical cut frame submitted for review, and then determines whether to discard the cut frame to be processed or transfer it to the machine review system according to the similarity calculation result. In this way, some cut frames can be discarded according to the similarity between the cut frame to be processed and the historical cut frame submitted for review, thereby realizing pre-filtering of similar frames, reducing the number of cut frames transferred to the machine review system, thereby reducing the load of the machine review system and improving the efficiency of machine review. In addition, considering that in live broadcast scenes, video frames often contain some relatively stable areas (such as backgrounds, fixed decorative elements, etc.), and the parts that have actually changed (such as character movements, etc.) are the objects that need to be focused on. If the cut frames are compared for similarity as a whole, the high similarity in the stable areas may cause the overall similarity score to be high, thereby masking the actual changed parts. The present application divides the image into multiple independent areas for similarity comparison, and calculates the similarity of each area separately. In this way, the influence of stable areas on changing areas in the live broadcast picture can be avoided, ensuring that the parts that have actually changed can be accurately identified, thereby improving the accuracy of similar frame judgment, as well as enhancing robustness and efficiency.

[0048] Next, an embodiment of performing similarity comparison using the partition comparison method in the above step S102 is described.

[0049] In some modified implementations, the calculating of the similarity between the to-be-processed frame and the historical frame submitted for review by using a partition comparison method may include:

[0050] Dividing the to-be-processed cut frame into a plurality of regions;

[0051] For each of the regions, calculating the region similarity between the region and the corresponding region in the historical cut frame submitted for review;

[0052] The similarity calculation result is determined according to the region similarity corresponding to each of the regions.

[0053] Among them, the division method and number of the above-mentioned areas can be flexibly set according to the actual application scenario. For example, the grid division method can be used to divide the area into nine grids, four grids, sixteen grids, etc., or the division method based on the interest area can be used to divide the area according to the interest points or interest areas in the image, so as to separate the interest area from the non-interest area and calculate the similarity separately. In addition, the area division method corresponding to the current live broadcast type can be selected for division according to the live broadcast type. For example, according to the characteristics of the live broadcast screen of different live broadcast types, the corresponding area division method and number can be set in advance for various live broadcast types, and then, according to the live broadcast type of the live video stream, the corresponding area division method and number are selected for area division to improve the accuracy of similarity comparison; in addition, the area division method corresponding to the current audit dimension can be selected for division according to the audit dimension. For example, the corresponding area division method and number can be set in advance for various audit dimensions, and then, according to the current audit dimension, the corresponding area division method and number are selected for area division to improve the accuracy of similarity comparison from the perspective of machine review needs.

[0054] The above-mentioned regional similarity is the similarity between a certain area in the to-be-processed cut frame and an area of ​​corresponding position and size in the historical cut frame submitted for review. In practical applications, the to-be-processed cut frame and the historical cut frame submitted for review are generally divided using the same regional division method. Therefore, the similarity comparison can be performed directly.

[0055] In addition, since the to-be-processed cut frame is divided into multiple regions for similarity comparison, the calculated regional similarities are also multiple, and the aforementioned similarity calculation result refers to the macroscopic similarity between the to-be-processed cut frame and the historical cut frame submitted for review. Therefore, it is necessary to determine the similarity calculation result based on the regional similarity. There are many ways to determine the similarity calculation result. For example, a weighted average method can be used to calculate the comprehensive similarity as the similarity calculation result. The regional similarity of a specific area can be selected as the similarity calculation result based on the area of ​​interest. Or, based on the sorting result of the regional similarity, the minimum regional similarity is selected as the similarity calculation result to highlight the degree of non-similarity. All of these can achieve the purpose of the embodiments of the present application from different dimensions. Those skilled in the art can flexibly determine the method for determining the similarity calculation result based on actual needs.

[0056] Regarding the method for calculating the regional similarity, based on the above implementation, in some modified implementations, the calculation of the regional similarity between the region and the corresponding region in the historical cut frame submitted for review includes:

[0057] Using a perceptual hash algorithm to extract fingerprint information of the area;

[0058] The region similarity is calculated based on the fingerprint information of the region and the fingerprint information of the corresponding region in the historical cut frame submitted for review.

[0059] Among them, the perceptual hashing algorithm (pHash for short) is a similarity detection technology for images. It processes the image, extracts its perceptual features, and converts them into a hash value of fixed length. This hash value can reflect the key features of the image. By comparing the perceptual hash values ​​of two images, it is possible to quickly determine whether they are similar enough. The above-mentioned perceptual features refer to the key characteristics used to describe the image in image processing, which may include but are not limited to color distribution, texture, edge, shape, etc. The above-mentioned fingerprint information is the hash value extracted by the perceptual hashing algorithm.

[0060] Based on the above fingerprint information, the above regional similarity can be calculated using the Hamming distance. The Hamming distance refers to the number of different characters in the corresponding positions of two equal-length strings. For example, the fingerprint information of region A in the current pending frame is 10101010, and the fingerprint information of the corresponding region a in the corresponding historical frame submitted for review is 10101011. The number of bits of difference between 10101010 and 10101011 is 1, and the Hamming distance between the two is 1. Since the fingerprint information calculated by the perceptual hash algorithm is a hash value of fixed length, it is very suitable for calculating the similarity using the Hamming distance.

[0061] It should be noted that for the historical cut frames submitted for review, they have also been partitioned and fingerprint information extracted as pending cut frames before submission for review. Therefore, after confirming that they will be transferred to the machine review system for review, the various areas of the historical cut frames submitted for review and their corresponding fingerprint information can be stored. When it is necessary to calculate the similarity between the pending cut frames and the historical cut frames submitted for review, the fingerprint information of each area stored in advance can be directly extracted for calculation, thereby avoiding the redundant step of repeatedly calculating the fingerprint information for the historical cut frames submitted for review. Since reading the pre-stored fingerprint information is more convenient and efficient than recalculating the fingerprint information, the system computing overhead can be reduced and the overall processing efficiency can be improved.

[0062] By using the perceptual hashing algorithm to extract the fingerprint information of each region and calculating the regional similarity based on the fingerprint information, this embodiment can significantly improve the accuracy of similarity detection. Specifically, the perceptual hashing algorithm can not only capture the key features of the image, including color distribution, texture, edge, shape, etc., but also keep its fingerprint information relatively stable even if the image is slightly modified (such as compression, cropping, rotation, etc.), thereby improving the accuracy of similarity detection. In addition, for historical review frames, it is only necessary to calculate and store the fingerprint information of each region when it is first submitted for review. When it is necessary to compare the similarity with the frame to be processed, the pre-stored fingerprint information can be directly used without repeated calculation, which significantly reduces the system's computing overhead and improves processing efficiency. Since reading the pre-stored fingerprint information is more convenient and efficient than recalculating the fingerprint information, it can speed up the similarity calculation and improve the overall processing efficiency.

[0063] In order to extract the fingerprint information of the area, it is first necessary to map the image of the area from the spatial domain to the frequency domain. This process can be achieved by discrete cosine transform (DCT) or discrete Fourier transform (DFT). The conversion process from spatial domain mapping to frequency domain can capture the frequency components of the image, especially high-frequency signals, which usually correspond to details and edge features in the image. Based on this, in some modified implementations, the use of perceptual hashing algorithm to extract the fingerprint information of the area may include:

[0064] Mapping the image of the region from the spatial domain to the frequency domain to obtain a frequency domain signal;

[0065] extracting a high frequency signal from the frequency domain signal;

[0066] A hash calculation is performed according to the high-frequency signal to obtain fingerprint information of the area.

[0067] The image data in the spatial domain exists in the form of a two-dimensional array of pixels, and each pixel represents the color information of a certain position in the image. By applying discrete cosine transform (DCT) or discrete Fourier transform (DFT), the information of these pixels can be converted into frequency domain signals, so that the information of the image is represented as a series of frequencies and corresponding amplitudes. In this way, the original image is decomposed into its basic frequency patterns, where the low-frequency components represent the basic structure of the image, while the high-frequency components correspond to the details and edges in the image. This conversion helps to extract key features in the image, especially high-frequency signals, which can reflect the details and edges in the image, thus providing a basis for subsequent feature extraction and similarity calculation.

[0068] The frequency domain signal contains various frequency components of the image, including low-frequency signals (representing the general outline of the image) and high-frequency signals (representing the details of the image). In order to extract key detail features, usually only the high-frequency signal part needs to be focused on. Extracting high-frequency signals can be achieved by setting a threshold, for example, only retaining coefficients in the frequency domain signal that are higher than a certain threshold, which can effectively extract the key features in the image.

[0069] After extracting the high-frequency signal, it can be hashed to generate a fixed-length hash value, i.e., fingerprint information. Hash calculation converts the high-frequency signal into a binary sequence, such as determining the value of each bit by comparing the size of adjacent coefficients. The generated hash value can reflect the key features of the image, thereby improving the accuracy of similarity detection.

[0070] You can refer to Figure 2 For understanding, it shows a schematic diagram for describing frequency domain transformation provided by an embodiment of the present application, such as Figure 2 As shown in Figure 1, it transforms the image from the spatial domain to the frequency domain using discrete cosine transform (DCT) to separate the low-frequency and high-frequency components of the image. Low-frequency signals usually represent large smooth areas of the image, such as facial contours or background colors, while high-frequency signals correspond to detailed parts of the image, such as eye textures or sharp edges.

[0071] In the frequency domain, low-frequency signals occupy the upper left corner, while high-frequency signals are distributed in the lower right corner. If the image has low detail, its spectrum energy will be mainly concentrated in the low-frequency area; conversely, when the image is rich in detail, the energy will spread to the high-frequency area. By focusing on high-frequency signals, the subtle structures in the image can be retained in a targeted manner, especially the sensitive information in small fonts that may be contained in live broadcasts, thereby achieving effective filtering of high-frequency data.

[0072] It should be noted that, during the image processing process, extracting only low-frequency signals will cause image distortion, which will appear as a blurred visual effect. On the contrary, extracting high-frequency signals can reveal the edges and contours of the image, which is an aspect of particular concern in the embodiments of the present application. By analyzing these contour information, it is possible to effectively identify the slight differences between the cut frames, thereby maintaining the integrity of key details in the process of filtering similar frames based on the scheme of the present application.

[0073] The embodiment of the present application enhances the extraction and utilization of high-frequency information through frequency domain conversion, providing strong support for the efficient review of live content and the accurate identification of sensitive information. This strategy not only improves the efficiency of the filtering process, but also ensures the keen capture of detail differences. It is an important technical means to optimize the live security review process.

[0074] The above about Figure 2The description is only an exemplary description of the frequency domain transformation and the high-frequency signal. Since the embodiment of the present application adopts the partition comparison method to calculate the frame similarity, when the perceptual hash algorithm is used to extract the fingerprint information, it is also performed in different regions. Correspondingly, the mapping of the regional image from the spatial domain to the frequency domain is also performed in different regions. Figure 3 A schematic diagram for describing the partition frequency domain transformation provided by an embodiment of the present application is shown. It can be seen that the embodiment of the present application performs frequency domain transformation on each region and extracts high-frequency signals to generate fingerprint information. By dividing the image into multiple independent regions and extracting fingerprint information for similarity calculation in a targeted manner, the errors caused by mutual interference between different regions can be effectively avoided, thereby improving the overall detection accuracy.

[0075] Based on any of the above implementations, in some modified implementations, for each of the regions, calculating the region similarity between the region and a corresponding region in a historical frame submitted for review includes:

[0076] A plurality of computing heads are used to calculate in parallel the regional similarity between each of the regions and the corresponding region in the historical cut frame submitted for review, wherein each of the computing heads calculates the regional similarity of one of the regions.

[0077] In order to further improve the computing efficiency, this implementation introduces a multi-head parallel computing method. The core idea of ​​this method is to use multiple computing units to perform similarity calculations simultaneously in a multi-core processor or distributed computing environment, and each computing unit is responsible for calculating the similarity of a specific area. Among them, a computing head refers to a software module or hardware unit that performs a specific computing task.

[0078] In specific implementation, after the to-be-processed cut frame is divided into multiple regions, a computing head is assigned to each region, and each computing head independently performs the frequency domain transformation, feature extraction, and similarity calculation tasks for the region assigned to it, and all computing heads work simultaneously, and then the results are aggregated to obtain the similarity calculation results of the to-be-processed cut frame and the historical cut frame submitted for review. Since multiple computing heads work in parallel, and the amount of calculation of each computing head is greatly reduced compared to the amount of calculation of the entire cut frame, the efficiency of similarity calculation can be effectively improved and the time consumption of calculation can be reduced. Taking the area division method of the nine-square grid as an example, 9 computing heads are used for similarity calculation, and the overall time consumption can be reduced to one-tenth compared to no division.

[0079] In addition, based on any of the foregoing implementations, in some modified implementations, determining the similarity calculation result according to the region similarity corresponding to each of the regions may include:

[0080] Performing weighted calculation according to the regional similarity corresponding to each of the regions to obtain a comprehensive similarity;

[0081] The comprehensive similarity is determined as a similarity calculation result.

[0082] Different weights can be assigned according to the importance of the regions. For example, if a live broadcast focuses on the faces of people in the video, a higher weight can be assigned to the regions containing the faces of people.

[0083] Once the similarity and corresponding weight of each sub-region are determined, the comprehensive similarity can be calculated by weighted average. The formula can be expressed as:

[0084]

[0085] Where: n represents the total number of regions; weight i Represents the weight of the i-th region; regional similarity i Represents the region similarity of the i-th region.

[0086] Then, the calculated comprehensive similarity can be determined as the similarity calculation result between the to-be-processed cut frame and the historical cut frame submitted for review. If the comprehensive similarity exceeds a preset threshold, the two frames can be considered similar and do not require machine review. The to-be-processed cut frame can be discarded and not submitted for review; otherwise, they are considered dissimilar and need to be transferred to the machine review system for machine review. The above thresholds can be flexibly set and adjusted according to actual needs, and are not limited in the embodiments of the present application.

[0087] The above implementation method, by dividing the video frames into regions and considering the importance of each region, can more accurately judge the similarity between video frames, thereby improving the accuracy of similarity calculation. In addition, the distribution of weights can be flexibly adjusted according to specific needs, which enables the algorithm to better adapt to different application scenarios and improve flexibility. In addition, even if some areas are affected by noise or deformation, other parts can still provide useful information, thereby improving the overall robustness.

[0088] Based on the above implementation, in some modified implementations, the weighted calculation is performed according to the regional similarity corresponding to each of the regions to obtain the comprehensive similarity, including:

[0089] Obtain weight distribution information according to the live broadcast type and / or review dimension of the current live broadcast stream;

[0090] Determine a weight corresponding to each of the regions according to the weight allocation information;

[0091] The regional similarity corresponding to each of the regions is weightedly calculated according to the weight to obtain a comprehensive similarity.

[0092] Considering that different types of live streams may focus on different areas, for example, in game live broadcasts, the player's operating area may be more important; in news live broadcasts, the news anchor's facial area may be the focus; in education live broadcasts, the teacher's explanation area or blackboard area may be more important. Therefore, weights can be assigned to each area according to the type of live broadcast, and the weights of important areas are assigned higher. In practical applications, according to the characteristics of the live broadcast screens of different types of live broadcasts, weight distribution information corresponding to specific regional division methods can be set in advance for various types of live broadcasts. The weight distribution information contains the weight corresponding to each area. Then, according to the live broadcast type of the live video stream, the weight distribution information corresponding to the current regional division method is selected, and the weight distribution information is used to determine the weight corresponding to each of the said areas to perform weighted calculation of regional similarity.

[0093] In addition, in actual applications, according to the live broadcast review requirements, there will be different review dimensions corresponding to different types of live broadcast violations, such as OCR (Optical Character Recognition), pornography, violence, etc., and the areas where the illegal content appears in different review dimensions are also different. In order to achieve the pre-filtering purpose of the machine review system, the embodiment of the present application also needs to focus on the illegal content in the process of comparing similar frames. Therefore, weights can also be assigned to each area according to the review dimension. For example, if the review dimension is to detect text content, then the area containing text (such as the title on the screen, bullet screen, etc.) should be given a higher weight; if the review dimension is to detect pornographic content, in this case, the exposed part of the human body (such as the chest, buttocks, etc.) will be given a higher weight; if the review dimension is to detect violent content, since violent behavior usually occurs in areas where people move violently, the area containing people's activities will be given a higher weight. Based on this, technicians in this field can also set corresponding regional division methods and weight distribution information for various review dimensions in advance according to the live broadcast review requirements. The weight distribution information contains the weight corresponding to each area, and then, according to the current review dimension, select the weight distribution information corresponding to the current regional division method for weighted calculation of regional similarity.

[0094] In addition, based on the above description, technicians in this field can also combine the live broadcast type and the review dimension to determine the weight distribution information, and achieve more precise weight distribution through more complex combination relationships to further improve the accuracy of similarity calculation, which will not be repeated here.

[0095] The above implementation of the present application can significantly improve the accuracy of the algorithm by determining the weight of each area according to different live broadcast types and audit dimensions, so that it can more accurately identify and judge the similarity between video frames, especially in scenarios involving illegal content, and can more accurately detect potential problem areas. At the same time, the determination of weights can flexibly respond to changes in the content and audit requirements of different types of live broadcasts, thereby improving the adaptability and scalability of this solution. In addition, even if some areas are affected by noise or occlusion, the high weights of important areas can ensure the stability of the overall similarity calculation, thereby improving the robustness of this solution.

[0096] Based on the above implementation, in some modified implementations, determining the weight distribution information according to the live broadcast type and / or audit dimension of the current live broadcast stream includes:

[0097] Determine model input data according to the live broadcast type and / or audit dimension of the current live broadcast stream;

[0098] The model input data is input into a pre-trained attention model, and the attention model is used to output weight allocation information.

[0099] Among them, the attention model is a neural network model built based on the attention mechanism. Its core idea is to enable the model to dynamically assign different weights or attention levels according to different parts of the input data. This mechanism enables the model to pay more attention to the key parts of the input data. In the embodiment of the present application, through pre-training, the trained attention model can determine the importance of each area according to the live broadcast type and / or audit dimension of the current live broadcast stream. Specifically, the model input data is first determined according to the live broadcast type and / or audit dimension, and then these model input data are input into the pre-trained attention model. The attention model dynamically assigns a weight to each area by learning the correlation between different areas in the video frame. These weights reflect the importance of each area relative to other areas in the current live broadcast type and / or audit dimension.

[0100] In specific implementation, you can first build an attention model (such as a model based on the Transformer architecture), which can learn the importance of different areas in the video frame and dynamically assign weights to each area based on the content of the video frame. Next, prepare a training data set, including video frames and their corresponding labels or key information (such as violation types, important areas, etc.), and divide the video frames into multiple regions to extract the features of each region. Then, use the training data set to train the attention model to let the model learn how to assign weights to each region. During the training process, the model automatically adjusts the weight assignment based on the content of the video frame to maximize the prediction accuracy. Finally, after the training is completed, the model input data is input into the attention model, and the attention model can be used to output the weight assignment information for each region.

[0101] In the embodiment of the present application, by introducing an attention model, the importance weights of different areas can be dynamically adjusted according to business needs. In live broadcast scenarios, areas with non-core content such as barrages and sliding subtitles can be given lower weights through the attention mechanism, while the center of the screen or other key areas that may contain illegal information are given higher weights. In this way, it is possible to more accurately identify and filter out cut frames with high similarity, while ensuring that key information is not ignored, significantly improving the accuracy and efficiency of similar frame comparisons.

[0102] It should be noted that in actual applications, multiple model channels can be set for different live broadcast types and / or audit dimensions, for example, game live broadcast model channels, education live broadcast model channels, military live broadcast model channels, OCR model channels, vulgar model channels, etc., and each model channel is set with an attention model. In this way, each attention model is only responsible for the weight allocation of a corresponding model channel. Based on this, more targeted training can be carried out to improve the accuracy of the attention model. When it is necessary to determine the weight allocation information, the model input data is input into the attention model of the corresponding model channel to use the attention model of the model channel to output more accurate weight allocation information. At the same time, the attention model of each model channel can also be iteratively updated to further improve the rationality and accuracy of the weight allocation.

[0103] For example, in the live broadcast business, for different types of violations (such as OCR, pornography, and violence), corresponding model types are set for risk identification, and whether the cut frame enters the current model channel is determined by the filtering algorithm. The attention model in the model channel allocates regional similarity weights according to the model recognition type, live broadcast room partition and other live broadcast room background information, thereby calculating the comprehensive similarity. For example, the cut frame belongs to the "virtual anchor" partition. When the cut frame enters the OCR model channel, according to the live broadcast room information and OCR model capabilities, the attention mechanism will significantly strengthen the similarity weight of the text area on both sides and weaken the similarity weight of the middle image area; and when the cut frame enters the vulgar model channel, the attention mechanism will strengthen the similarity weight of the middle image area and weaken the similarity weight of the text area on both sides. Among them, the attention model is trained based on the business history violation annotation data, with the violation type (which can correspond to the audit dimension), live broadcast room partition (which can correspond to the live broadcast type) and other live broadcast room background information as input, and the violation area as output, and the weight is allocated to fit the importance of the area.

[0104] Based on any of the above implementations, in some modified implementations, the similarity between the to-be-processed frame and the historical frame submitted for review is calculated by using a partition comparison method to obtain a similarity calculation result, including:

[0105] The partition comparison method is used to calculate the similarity between the to-be-processed frame and at least one historical frame submitted for review in the set of historical frames submitted for review, and at least one similarity calculation result is obtained;

[0106] The step of determining, according to the similarity calculation result, whether to discard the to-be-processed cut frame or to transfer it to the machine review system comprises:

[0107] If any of all similarity calculation results obtained by calculation meets the preset similarity determination condition, the frame to be processed is discarded;

[0108] If all the similarity calculation results calculated do not meet the preset similarity determination conditions, the to-be-processed cut frame is handed over to the machine review system.

[0109] Among them, the historical cut frame set submitted for review can be determined according to the time window or according to the quantity. For example, it can be a set of cut frames submitted for review within a specified historical time window, such as a set of cut frames submitted for review within 10 minutes before the cut frame to be processed. For example, it can also be a set of 100 cut frames before the cut frame to be processed, etc. The embodiments of the present application are not limited to this, and those skilled in the art can flexibly set it according to actual needs, which are all within the protection scope of the present application.

[0110] In this implementation, the to-be-processed cut frame is compared with the historical cut frames submitted for review in the historical cut frame set for similarity one by one, and each comparison will generate a similarity calculation result. In order to improve the overall video frame filtering efficiency, each similarity calculation result can be compared with the preset similarity judgment condition. When it is found that any similarity calculation result meets the preset similarity judgment condition, the to-be-processed cut frame can be judged as a similar frame without the need for subsequent similarity comparison with the remaining historical cut frames submitted for review. Then, the to-be-processed cut frame is discarded based on this, and the comparison process of the to-be-processed cut frame is terminated in advance. When selecting the historical cut frames submitted for review to be compared from the historical cut frame set, it can start from the historical cut frame submitted for review that is closest to the to-be-processed cut frame, and select in order from near to far. Since the probability that the to-be-processed cut frame is similar to the adjacent cut frames is high, this implementation can find similar frames more quickly and save a lot of similarity comparison work with the remaining historical cut frames submitted for review, thereby improving the processing efficiency as a whole.

[0111] In addition, the above-mentioned preset similarity determination condition can be implemented by using a similarity determination threshold. For example, if the similarity calculation result is higher than the similarity determination threshold, it is determined that the preset similarity determination condition is met, and the pending cut frame can be discarded. For another example, if the similarity calculation result is lower than the similarity determination threshold, it is determined that the preset similarity determination condition is not met, and the pending cut frame needs to be transferred to the machine review system for machine review. The embodiment of the present application does not limit the specific value of the above-mentioned similarity determination threshold. Those skilled in the art can flexibly set it according to actual needs, which will not be repeated here.

[0112] Through this embodiment, by comparing the similarity between the to-be-processed cut frame and at least one historical cut frame submitted for review in the set of historical cut frames submitted for review, the to-be-processed cut frame can be compared with more historical cut frames submitted for review, thereby avoiding the misjudgment of the scheme of comparing with a single historical cut frame submitted for review due to the frequent changes in the live broadcast screen, thereby improving the stability, accuracy and robustness of the overall similar frame filtering; in addition, since the preset similarity judgment condition is immediately compared after each comparison, the comparison process can be terminated in advance once the similarity result that meets the preset condition is found, thereby significantly improving the processing efficiency and resource utilization. In addition, by starting the comparison from the most recent historical cut frame submitted for review, similar frames can be found more quickly and the comparison step with the remaining historical cut frames submitted for review can be omitted, thereby further improving the system response speed and accuracy, as well as reducing the system calculation amount and system load.

[0113] Based on the same inventive concept, a video frame filtering device is also provided in an embodiment of the present application. The method corresponding to the device may be the video frame filtering method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The video frame filtering device provided in an embodiment of the present application can implement the aforementioned video frame filtering method, and the video frame filtering device can be implemented by software, hardware, or a combination of software and hardware. For example, the video frame filtering device may include integrated or separate functional modules or units to perform the corresponding steps in the aforementioned methods. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts may refer to the partial description of the method embodiment. The device embodiment described below is merely illustrative. Please refer to Figure 4 , the video frame filtering device may include:

[0114] The frame acquisition module 101 is used to acquire the frame to be processed from the live video stream;

[0115] The frame comparison module 102 is used to calculate the similarity between the frame to be processed and the historical frame submitted for review by using a partition comparison method to obtain a similarity calculation result;

[0116] The frame cutting filtering module 103 is used to determine whether to discard the frame cutting to be processed or transfer it to the machine review system according to the similarity calculation result.

[0117] In some modified implementations, the frame comparison module 102 includes:

[0118] A frame cutting partitioning unit, used for dividing the frame cutting to be processed into a plurality of regions;

[0119] A partition calculation unit, used for calculating, for each of the regions, the region similarity between the region and the corresponding region in the historical cut frame submitted for review;

[0120] The similarity calculation unit is used to determine a similarity calculation result according to the region similarity corresponding to each of the regions.

[0121] In some implementation variations, the partition calculation unit includes:

[0122] A fingerprint information extraction subunit, used to extract fingerprint information of the area using a perceptual hash algorithm;

[0123] The region similarity calculation subunit is used to calculate the region similarity based on the fingerprint information of the region and the fingerprint information of the corresponding region in the historical cut frame submitted for review.

[0124] In some modified implementations, the fingerprint information extraction subunit includes:

[0125] A frequency domain conversion subunit, used for mapping the image of the region from the spatial domain to the frequency domain to obtain a frequency domain signal;

[0126] A high-frequency signal extraction subunit, used to extract a high-frequency signal from the frequency domain signal;

[0127] The hash calculation subunit is used to perform hash calculation according to the high-frequency signal to obtain fingerprint information of the area.

[0128] In some implementation variations, the partition calculation unit includes:

[0129] The multi-head computing subunit is used to use multiple computing heads to parallelly calculate the regional similarity between each of the regions and the corresponding region in the historical cut frame submitted for review, wherein each of the computing heads calculates the regional similarity of one of the regions.

[0130] In some modified implementations, the similarity calculation unit includes:

[0131] A comprehensive calculation subunit, used for performing weighted calculation according to the regional similarity corresponding to each of the regions to obtain a comprehensive similarity;

[0132] The comprehensive similarity determination subunit is used to determine the comprehensive similarity as the similarity calculation result.

[0133] In some implementation variations, the comprehensive calculation subunit includes:

[0134] The weight distribution information acquisition subunit is used to determine the weight distribution information according to the live broadcast type and / or audit dimension of the current live broadcast stream;

[0135] A weight determination subunit, used to determine the weight corresponding to each of the regions according to the weight allocation information;

[0136] The weighted calculation subunit is used to perform weighted calculation on the region similarity corresponding to each of the regions according to the weight to obtain a comprehensive similarity.

[0137] In some modified implementations, the weight distribution information acquisition subunit includes:

[0138] An input data determination subunit, used to determine model input data according to the live broadcast type and / or audit dimension of the current live broadcast stream;

[0139] The attention processing subunit is used to input the model input data into a pre-trained attention model and use the attention model to output weight allocation information.

[0140] In some modified implementations, the frame comparison module 102 includes:

[0141] A set comparison unit, configured to compare the to-be-processed cut frame with at least one historical cut frame submitted for review in the set of historical cut frames submitted for review by using a partition comparison method for multiple comparisons, and obtain at least one similarity calculation result;

[0142] The frame cutting filter module 103 includes:

[0143] A frame cutting discarding unit, configured to discard the frame cutting to be processed if any one of all similarity calculation results calculated meets a preset similarity determination condition;

[0144] The frame cutting recovery unit is used to transfer the frame cutting to be processed to the machine review system if all the similarity calculation results calculated do not meet the preset similarity judgment conditions.

[0145] The video frame filtering device provided in the embodiment of the present application is based on the same inventive concept as the video frame filtering method provided in the aforementioned embodiment of the present application and has the same beneficial effects.

[0146] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the video frame filtering method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the video frame filtering methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0147] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0148] Figure 5The structure of an electronic device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the electronic device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM 1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0149] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0150] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0151] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0152] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination therewith.

[0153] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0154] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0155] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0156] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0157] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0158] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0159] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0160] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0161] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0162] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0163] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A video frame filtering method, wherein: The method comprises: Get the frames to be processed from the live video stream; The partition comparison method is used to calculate the similarity between the to-be-processed cut frame and the historical cut frame submitted for review, and a similarity calculation result is obtained; Determine whether to discard the pending cut frame or transfer it to the machine review system according to the similarity calculation result.

2. The method according to claim 1, wherein: The partition comparison method is used to calculate the similarity between the to-be-processed frame and the historical frame submitted for review, and the similarity calculation result is obtained, including: Dividing the to-be-processed cut frame into a plurality of regions; For each of the regions, calculating the region similarity between the region and the corresponding region in the historical cut frame submitted for review; The similarity calculation result is determined according to the region similarity corresponding to each of the regions.

3. The method according to claim 2, wherein: The calculating of the regional similarity between the region and the corresponding region in the historical cut frame submitted for review includes: Using a perceptual hash algorithm to extract fingerprint information of the area; The region similarity is calculated based on the fingerprint information of the region and the fingerprint information of the corresponding region in the historical cut frame submitted for review.

4. The method according to claim 3, wherein: The extracting fingerprint information of the area by using a perceptual hash algorithm includes: Mapping the image of the region from the spatial domain to the frequency domain to obtain a frequency domain signal; extracting a high frequency signal from the frequency domain signal; A hash calculation is performed according to the high-frequency signal to obtain fingerprint information of the area.

5. The method according to claim 2, wherein: The step of calculating, for each region, the region similarity between the region and a corresponding region in a historical frame submitted for review includes: A plurality of computing heads are used to calculate in parallel the regional similarity between each of the regions and the corresponding region in the historical cut frame submitted for review, wherein each of the computing heads calculates the regional similarity of one of the regions.

6. The method according to claim 2, wherein: The determining of the similarity calculation result according to the region similarity corresponding to each of the regions includes: Performing weighted calculation according to the regional similarity corresponding to each of the regions to obtain a comprehensive similarity; The comprehensive similarity is determined as a similarity calculation result.

7. The method according to claim 6, wherein: The weighted calculation is performed according to the regional similarity corresponding to each of the regions to obtain the comprehensive similarity, including: Determine weight distribution information according to the live broadcast type and / or review dimension of the current live broadcast stream; Determine a weight corresponding to each of the regions according to the weight allocation information; The regional similarity corresponding to each of the regions is weightedly calculated according to the weight to obtain a comprehensive similarity.

8. The method according to claim 7, wherein: Determining the weight distribution information according to the live broadcast type and / or audit dimension of the current live broadcast stream includes: Determine model input data according to the live broadcast type and / or audit dimension of the current live broadcast stream; The model input data is input into a pre-trained attention model, and the attention model is used to output weight allocation information.

9. The method according to claim 1, wherein: The partition comparison method is used to calculate the similarity between the to-be-processed frame and the historical frame submitted for review, and the similarity calculation result is obtained, including: The partition comparison method is used to calculate the similarity between the to-be-processed frame and at least one historical frame submitted for review in the set of historical frames submitted for review, and at least one similarity calculation result is obtained; The step of determining, according to the similarity calculation result, whether to discard the to-be-processed cut frame or to transfer it to the machine review system comprises: If any of all similarity calculation results obtained by calculation meets the preset similarity determination condition, the frame to be processed is discarded; If all the similarity calculation results calculated do not meet the preset similarity determination conditions, the to-be-processed cut frame is handed over to the machine review system.

10. A video frame filtering device, wherein: The device comprises: A frame acquisition module is used to obtain the frame to be processed from the live video stream; A frame comparison module is used to calculate the similarity between the frame to be processed and the historical frame submitted for review by using a partition comparison method to obtain a similarity calculation result; The frame cutting filtering module is used to determine whether to discard the to-be-processed frame cutting or transfer it to the machine review system according to the similarity calculation result.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

12. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 9.

Images