Systems and methods for digital fingerprinting of media content - Patents.com
The described method improves content filtering by preprocessing media files to generate resistant fingerprints, addressing inefficiencies in detecting copyrighted and prohibited content on online platforms.
Patent Information
- Application Number
- JP2024549581
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-01-14
- Filing Date
- 2022-11-07
- Publication Date
- 2024-11-15
AI Technical Summary
Existing content filtering systems for online media platforms are inefficient in detecting copyrighted and prohibited content due to high false positives and negatives, computational overhead, and ease of circumvention through re-encoding and metadata manipulation.
A method involving preprocessing media files to create modified copies, generating video and audio fingerprints using perceptual hashing, and performing comparisons with previously identified fingerprints to determine prohibited content.
Enhances accuracy and reduces computational load by creating irreversible fingerprints that are resistant to common content manipulation techniques, effectively identifying prohibited content.
Smart Images

Figure 2024542867000001_ABST
Abstract
Description
[Technical field]
[0001] Priority This application claims priority to U.S. Provisional Application No. 63 / 276,883, filed November 8, 2021, and to U.S. Nonprovisional Application No. 17 / 576,666, filed January 14, 2022, which also claims priority to U.S. Provisional Application No. 63 / 276,883, filed November 8, 2021, the contents of all of which are incorporated by reference herein in their entireties.
[0002] The present disclosure relates to digital fingerprinting of media content, and more particularly, to detection of copyrighted media content and / or other prohibited content using a combination of audio and video fingerprinting. [Background technology]
[0003] For online media platforms, particularly hosts of Video Sharing Platforms (VPS), where website users can upload content for distribution, ensuring that the uploaded content complies with rules, laws, and / or regulations is a challenging problem. For many such websites, the amount of content uploaded per minute is not possible for an individual censor to review in a month. To check content for copyright infringement or content violations, various forms of automated filtering may be used that flag content for further review. However, such automated filtering is often inaccurate and results in too many false positives and / or false negatives depending on how stringent the filtering process is configured. Many content fingerprinting solutions, such as file hashes, can be easily circumvented by re-encoding, truncation, and / or metadata modification. In addition, such automated filtering can easily result in excessive computations, since as the number of known content increases, so does the number of comparisons required to ensure that new content is free of copyright and other prohibited content. Summary of the Invention
[0004] Additional features and advantages of the present disclosure will be set forth in the description which follows, and will be understood in part from the description, or may be learned by practice of the principles disclosed herein. The features and advantages of the present disclosure may be realized and obtained by means of the instruments and combinations particularly pointed out in the appended claims. These and other features of the present disclosure will become more fully apparent from the following description and the appended claims, or may be learned by practice of the principles set forth herein.
[0005] A system, method, and non-transitory computer readable storage medium are disclosed that provide technical solutions to the described technical problems. A method for implementing the concepts disclosed herein includes, in a computer system, receiving a media file including video and audio; pre-processing, by at least one processor, the media file to generate an altered copy of the video, an altered copy of the audio, and keyframes within the video; creating, by the at least one processor, a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of distinct segments of the altered copy of the video; creating, by the at least one processor, an audio fingerprint of the altered copy of the audio, the audio fingerprint including a second vector of distinct segments of the altered copy of the video; creating, by the at least one processor, a keyframe fingerprint based on the keyframes; and generating, by the at least one processor, a comparison media file. performing, by the at least one processor, a video comparison of the video fingerprint with a video fingerprint identified prior to the comparison media file; performing, by the at least one processor, an audio comparison of the audio fingerprint with an audio fingerprint identified prior to the comparison media file; performing, by the at least one processor, a keyframe comparison of the keyframe fingerprint with a keyframe fingerprint identified prior to the comparison media file; and determining, by the at least one processor, that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison.
[0006] A system configured to implement the concepts disclosed herein may include at least one processor and a non-transitory computer readable storage medium having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to: receive a media file including video and audio; pre-process the media file to generate an altered copy of the video, an altered copy of the audio, and keyframes within the video; create a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, where the video fingerprint includes a first vector of distinct segments of the altered copy of the video; and generate a modified copy of the audio and keyframes within the video. The method performs operations including: creating an audio fingerprint of the altered copy, where the audio fingerprint includes a second vector of individual segments of the altered copy of the video; creating a keyframe fingerprint based on the keyframes; performing a video comparison of the video fingerprint with a video fingerprint identified prior to the comparison media file; performing an audio comparison of the audio fingerprint with a keyframe fingerprint identified prior to the comparison media file; performing a keyframe comparison of the keyframe fingerprint with a keyframe fingerprint identified prior to the comparison media file; and determining that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison.
[0007] A non-transitory computer readable storage medium configured as disclosed herein may store instructions that, when executed by a computing device, cause the computing device to perform operations including receiving a media file including video and audio; pre-processing the media file to generate an altered copy of the video, an altered copy of the audio, and keyframes within the video; creating a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of individual segments of the altered copy of the video; creating an audio fingerprint of the altered copy of the audio, the audio fingerprint including a second vector of individual segments of the altered copy of the video; creating a keyframe fingerprint based on the keyframes; performing a video comparison of the video fingerprint with a previously identified video fingerprint of the comparison media file; performing an audio comparison of the audio fingerprint with a previously identified audio fingerprint of the comparison media file; performing a keyframe comparison of the keyframe fingerprint with a previously identified keyframe fingerprint of the comparison media file; and determining that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison. [Brief description of the drawings]
[0008] [Figure 1] FIG. 1 illustrates an exemplary system embodiment. [Diagram 2] FIG. 1 illustrates an example of comparing a known fingerprint with a new fingerprint. [Diagram 3] FIG. 2 illustrates an example of pre-processing a video to create a video fingerprint. [Figure 4] FIG. 13 is a diagram showing an example of creating a keyframe fingerprint. [Diagram 5] FIG. 2 illustrates an example of pre-processing a segment of a video. [Figure 6A] FIG. 13 is a diagram showing an example of a comparison of audio fingerprints. [Figure 6B]FIG. 13 shows an example of a comparison of video fingerprints. [Figure 6C] FIG. 13 illustrates an example of a comparison of keyframe fingerprints. [Figure 7] FIG. 13 is a diagram showing an example of fingerprint comparison. [Figure 8] FIG. 1 illustrates a first exemplary user interface displaying matches. [Figure 9A] FIG. 11 illustrates a second exemplary user interface displaying matches. [Figure 9B] FIG. 13 illustrates a third exemplary user interface displaying matches. [Figure 10] FIG. 1 illustrates an exemplary system architecture. [Figure 11] FIG. 1 illustrates an exemplary method embodiment. [Figure 12] FIG. 1 illustrates an exemplary computer system. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0009] Various embodiments of the present disclosure are described in detail below. While specific implementations are described, it should be understood that this description is done for purposes of example only. Other components and configurations may be used without departing from the spirit and scope of the present disclosure.
[0010] One exemplary, non-limiting practical application to the above technical problem is to compare the "fingerprint" of uploaded media content when the media is uploaded to an online platform with stored fingerprints of known content that has already been identified as copyrighted and / or prohibited content (child pornography, non-consensual media, etc.). Prohibited content, as defined herein, can include any content that the media file owner or uploader is not legally permitted to share, including copyrighted media, videos or images of minor sexual content, videos or images obtained without consent, etc. The overall process for identifying prohibited content can be summarized as follows: (1) pre-processing all visual (image and / or video) and audio submissions into a common format (generating modified copies of the media file submissions); (2) fingerprinting the common format video and audio submissions so that the content is transformed into a unique, irreversible quantity; (3) if the content is to be protected, storing the fingerprints in a database (such as, but not limited to, the Approximate Nearest Neighbor Database (ANN DB)) that can be used to quickly find similar elements; (4) identifying matching or highly similar content in the database, optionally performing a second verification of the match, and reporting the match to the user.
[0011] Thereafter, only media content that does not contain copyrighted or other prohibited material is allowed to be stored on the media platform. A fingerprint is an abstract, irreversible representation (e.g., integral with video content) of a scene, segment, clip, and / or sub-portion of a media content. For example, since video and sound are sequences of events (images for video, waveforms for audio), a sequence of fingerprints is computed for both the video and audio portions of the media content. During this matching step, this sequence may then be compared with known fingerprints of previously processed media content. The system may support many fingerprinting engines, causing multiple types of fingerprints to be computed during the protection or scanning process. This large number of fingerprints per item or sub-item of content increases the confidence and accuracy of decisions made by the matching engines in subsequent stages of the scanning process.
[0012] Consider the role of the disclosed fingerprinting method and system within the content suitability process. First, new content uploads are stored and encoded into different file formats to support many devices (PC (Personal Computer), mobile, etc.). These uploads and subsequent encoded files can be stored locally and cannot be accessed by unauthorized personnel or systems. At this point, the upload / encode process passes the files through different gates / systems that can scan the content and report results before review by a suitability team. The fingerprinting method and system described herein is one of many gate systems that can be used alone or in combination with other gate systems to provide results to a suitability team that can allow the content to be distributed or prohibit the distribution of the content.
[0013] For example, a VSP encoder may send all uploads to the system, whether they are images or videos. After the system receives the content, it may be pre-processed to standardize the format of the content before it is sent to the different fingerprinting and matching engines in the system. At this point, the VSP encoder waits until the content is scanned by the system before moving it along the review process. After the system scans the content, the system calls back the encoder with a response. For videos, there are three possible types of responses: · Match: One or more videos match the uploaded video, and the system is confident that this match is accurate. · Suspicious: One or more videos match the uploaded video, and the system is not sure if this match is accurate. · No Match: There are no videos that match the uploaded video.
[0014] If a match or suspicious response is returned, the video is moved to a separate queue where suspicious matches are reviewed by a compatibility team. The process for images can be similar, but instead of three different responses as described above, multiple distance indicators can be provided by different engines for the image that is most likely to match the protected catalog. Since the response is a distance indicator, the lower this indicator, the more likely the image in the catalog is a match. If the distance indicator is low enough to cause a match, the image is moved to a separate queue where suspicious matches are reviewed by a compatibility team.
[0015] The fingerprinting system disclosed herein can be a standalone component integrated by the VSP, thus avoiding any dependencies. In other words, the system can be completely decoupled from the VSP, and thus can be easily integrated by partners without software or framework dependencies. As described further herein, unless expressly stated otherwise, a "system" refers to either a standalone fingerprinting system including one or more fingerprinting engines, or a VSP upload and compatibility system including a fingerprinting system including one or more fingerprinting engines.
[0016] When content is received by the system, it is first pre-processed and standardized before converting the content into a fingerprint. As part of the standardization, if the content contains both audio and video (such as in a movie), the audio and video portions may be separated before being standardized. Note that in some situations and configurations, the content may be video-only (images, GIFs, or video with no audio) or audio-only. After the content is pre-processed and standardized, parallel processes may take the processed content and apply transformations to the processed content to create various fingerprints.
[0017] As an example of audio pre-processing, audio signals can be extracted to raw WAV (or other audio format) files, and (if stereo audio is present) both channels can be merged into a single mono audio file. Any signal below a certain amplitude threshold is considered noise and removed. For example, a noise gate (-90dB) can be applied to the audio track to remove background interference that would prevent accurate matching. Silent portions of the track can also be removed, as silence itself does not constitute information. Audio sample rates can be standardized to a common sample rate, which is predetermined before pre-processing begins. The resulting audio signal can then be downsampled and stored to disk.
[0018] Video tracks go through similar pre-processing steps. For example, all input files can be downsampled to a fixed resolution of RxC (rows x columns) and N frames per second, and this modified file can be stored to disk. This also ensures that very large video files (such as 4K formats) are significantly reduced in size for the next step in the pipeline. In addition, uniform background colors (such as black bars that appear in portrait videos) are detected and removed. If the background is removed, the modified video is also saved to disk and passed to the next pre-processing step.
[0019] Other types of pre-processing of video content (such as images or GIFs) may also be performed: images are treated as a sequence of frames, with a static image (e.g., a JPEG file) considered to contain only one frame, while a GIF animation is considered to contain N frames. All subsequent operations downstream of the pipeline may be generalized using images as a video-like sequence.
[0020] The image pre-processing step may include the following steps: For each frame in the image, single color background bands are detected and removed. Then, every frame is saved as an individual file (such as a PNG file) and the resulting collection of files is passed forward in the pipeline. Other steps that may be included are that the resolution may be reduced, each submitted image generates a derivative image that may be flipped and rotated to increase the chances of a match, and / or automatic landscape or portrait correction.
[0021] After the content is processed, fingerprinting engines (computer processor executable code / algorithms) are made available for the image, video, and / or audio aspects of the content. These separate engines may create a mathematical representation (a "fingerprint") of the content using one or more media-specific engines that compare the fingerprint to known fingerprints stored in a database. The known fingerprints may correspond to copyrighted material and / or protected content. While examples of image, video, and audio fingerprinting engines are provided, it is noted that these engines are exemplary and that the system may include other types of fingerprinting engines known to those skilled in the art.
[0022] With respect to images, various exemplary types of fingerprinting engines are available to convert the image into a fingerprint that can be compared to a protected content catalog. Any combination of image fingerprinting engines may be deployed by the system, such that the system may use a single image fingerprinting engine, multiple image fingerprinting engines (in series or parallel), or no image fingerprinting engines, depending on the particular configuration of the system and the needs based on the content under review. The purpose of hashing image information is to obtain a numerical representation that compresses the content of the image in a way that is irreversible and searchable in a database. At the same time, the hash (fingerprint) should be resistant (to some degree) to modifications of the image content, such as scaling, changing brightness, contrast, color, adding or removing watermarks, etc. So-called perceptual hashes have these desirable properties.
[0023] A first exemplary image fingerprinting engine uses a differential hash or "dHash". The idea behind this hash is that the image is first converted to grayscale and then reduced to an 8x9 grid, where every cell (new pixel) in this grid contains the average gray level value of the pixels in the original image contained on that image region. This 8x9 grid can be two 8x8 grids (left grid (L) and right grid (R)) that overlap in all instances except for one column. The system can then compare the gray level values for every cell in every grid and write down a 1 if L>R (cells in the left grid have a larger gray level value than cells in the right grid) or write down a 0 if R>=L. This creates a 64-bit hash based on the L / R comparison of the input image. If the image contained multiple frames (such as in a GIF or video segment), this operation is performed for every frame in the image.
[0024] Upon ingestion, a 64-bit hash (or hashes) of this is generated and then stored in the ANN DB.
[0025] Upon query, the hash of the frame being queried is matched against previously stored hashes / fingerprints present in the ANN DB. An exact match is not required since images may differ slightly, so a reasonable predefined threshold (up to N bits of difference) can determine if a match exists. Also, as an added protection against image modification, hashes of the queried image in both mirrored and rotated form may be queried.
[0026] While the first image fingerprinting engine is simple yet effective, the system can also use a second image fingerprinting engine, in which a convolutional neural network (CNN) called a Deep Perceptual Hasher (DPH) is trained to transform images in such a way that similar images have similar outputs, helping to match video content even when the video content has been altered. The input of the DPH model is an image or video frame, and the output of the model is an N-dimensional embedding, such as a vector of 128-bit floating-point numbers (128 floats). For speed, a smaller CNN architecture, such as MobileNetV2, can be trained on subject-specific images related to the type of content being protected.
[0027] Training of the DPH convolutional neural network uses a triplet loss function, and at each training step, the DPH model outputs (1) a training image, (2) a randomly transformed (e.g., cropped, padded, rotated, flipped, slightly color changed, etc.) training image, and (3) an embedding of another image with the same random transformation. The DPH model then learns that the embedding between the training image and its transformation should be closer than the other image. For best results, the other image should be similar to the training image, which may be determined by the perceptual hashing mentioned above. This training step is repeated multiple times over thousands of images until the loss score reaches a plateau. When the plateau is reached, the DPH convolutional neural network may be considered trained.
[0028] The fingerprint / hash is obtained by using the DPH engine, and the image is simply passed through the DPH model to obtain an embedding, also known as a vector formation. The secured image embedding is stored in the ANN DB. Any query image can be embedded using the DPH, and the resulting fingerprint is searched against the ANN DB to find a match below a distance threshold. The distance can be, for example, the Hamming distance between two fingerprints or any other type of distance measure. Additional non-limiting exemplary distance measures between fingerprints can include weighted correlation distance, quadratic form distance, Hausdorff distance, etc.
[0029] With regard to video fingerprinting, the video fingerprinting engine is very similar to the fingerprinting engine used for static images, since video is a sequence of images. In the first example, a video file is downsampled to R frames per second, but if every frame is hashed, there may be too many hashes per file. Thus, the system can use a moving window to calculate the average of frames in an S-second region and hash this average frame using the same 8×9 system described above for images, then move forward M seconds. M and S are chosen so that there is some overlap between regions. In this way, even an hour-long video will only produce a few hundred hashes, making the database size manageable even when millions of videos are ingested.
[0030] If background bands are detected and removed during the pre-processing stage, a hash of the original video file (including the bands) may also be calculated. The system may also calculate hashes of the mirrored video, and these hashes are only used during querying to establish a match.
[0031] Upon ingestion, the hashes are added to the ANN DB. Only direct hashes (not mirrored video hashes) are used in this operation.
[0032] At query time, the direct hashes and mirror-inverted hashes are queried against the ANN DB to retrieve the video with the most matches. These candidates are checked during a second stage to ensure that the matches have some degree of temporal coherence (e.g., that the matches are not scattered across the timeline, do not belong to the same time instant, etc.). After the noisy candidates are removed, the filtered matches are returned. If two sets of hashes are received (one set of hashes of unchanged files, another set of hashes of files with background removed), two queries are performed and the results are aggregated so that the aggregation operation is transparent to the user.
[0033] In some configurations, the window length S is predetermined, such as 5 seconds or 10 seconds long, etc. In other configurations, the system uses a window length that can vary based on the size of the file being analyzed.
[0034] The system can also define an overlap parameter, for example in seconds, i.e. each window overlaps with the previous window by a certain number of seconds. An average frame is created using all frames within a window of a predefined length. For each window a segment fingerprint is generated and the system moves the window in a way that it overlaps with the previous window by the configured number of seconds, continuing until no frames are left to read.
[0035] Another exemplary video fingerprinting engine can use a process similar to a "Deep Perceptual Hash" of the video file. For efficiency, not every frame of the video can be embedded in the DPH model. Selected frames, called "keyframes," are extracted from the video and embedded in the DPH model. The contrast within each frame, measured as the root mean square error of the grayscale frame, can be calculated, and the frame with the most noticeable contrast within a sliding window of S seconds can be selected as the keyframe. For example, if the sliding window is 5 seconds long, the system can calculate the amount of contrast for each frame and then select the frame within that window with the highest amount of contrast as the keyframe. The system can then move the sliding window and continue identifying keyframes.
[0036] The embeddings of the keyframes can be (1) stored in the ANN DB for protection or (2) queried against the ANN DB to find a match. If the keyframe match after querying clearly belongs to a protected video, a match for this video is reported. If there are no salient videos and the match is less clear, the matched video may be considered a candidate and sent to an additional match confirmation step using time series alignment (see MASS description below). In some configurations, the same DPH model is used for images and videos, so that embeddings from images (videos) can be searched against protected embeddings of videos (images) if requested.
[0037] Although an example of identifying key frames based on grayscale contrast is provided above, the identification of key frames analyzed by the DPH algorithm may be accomplished in any manner known to one skilled in the art. A further exemplary manner in which key frames may be identified includes the following steps: (1) Score each frame based on the difference between the frame and the previous frame and the color change within the frame. For example, the system may receive frames within a segment / particular sub-portion of a video, group the frames into buckets of time T (e.g., T=5 seconds), and pick the top N frames with high scores within each bucket as key frames. (2) Process the frames of the video to determine a time when a predetermined amount of contrast between frames is detected. For example, the system may compare a first key frame with each subsequent frame in the video until the amount of contrast between the first key frame and the subsequent frame has a predetermined amount of contrast. After a predetermined amount of contrast is detected in the subsequent frame, the subsequent frame is identified as the next key frame, and the process may continue throughout the remainder of the video. In other configurations, contrast may be determined based on time such that a measurement of contrast is received every period t until a predetermined amount of contrast is detected. For example, the system may measure contrast every second until a threshold amount of contrast is detected. (3) Within video compression, there are I-frames (intra-coded pictures) that contain a complete image such as a JPG or BMP image file, P-frames (predicted pictures) that hold only the changes in the image from the previous frame, and B-frames (bi-directionally predicted pictures) that use the difference between the current frame and both the preceding and following frames to specify content. In some configurations, the system may identify I-frames as key frames.
[0038] A third example of video fingerprinting also relies on contrast. As previously described, frames of a video are identified to determine when a predetermined amount of contrast between frames is detected, and those points in time. However, in this example, the system generates a fingerprint based on the points in time when sufficient contrast is detected. The generated fingerprint allows the system to compare the temporal changes in contrast between the video to a derived fingerprint, which is also a list of values associated with a particular segment, each value indicating the time at which those points of contrast were detected.
[0039] Another video fingerprint that can be generated is a face print (also known as a face embedding), which is unique and irreversible for each person. Faces can be detected in images and video frames using a Multi-Task Cascaded Convolutional Neural Network (MTCNN). Face detection is performed on the original content file, since the face needs to be clear. From the detection, facial landmarks can be used to calculate an angle score to determine if the face is positioned more frontally or showing a side view. The detected face is cropped and passed through multiple embedding and expression models. The embedding models are each trained on a different dataset of faces, and each output a person-specific embedding, typically a 512-bit float vector. The expression models are trained on faces with different facial gestures (e.g., happiness, surprise, sadness, anger), and output the probability of each of the particular facial gestures. For example, if there are eight gestures in the model, the output will be eight different probabilities.
[0040] Faces in the image are detected and passed through the embedding model and the expression model. Face embeddings, expression probabilities, and angle scores are stored for the protected content (actual face cropping is not preserved). Protected face embeddings are preferably stored in the corresponding ANN DB, although other types of databases may be used. In a query, faces in the image are detected, embeddings are extracted, and those embeddings are searched in the protected ANN DB. Those faces found within a distance threshold are considered a match. As a second validation, matching faces must also have similar expression and angle scores.
[0041] Matching faces in videos is slightly different from images because there may be many frames in a video that contain faces. Every N seconds a frame of the video is extracted and face detection, embedding and expression model is performed on each frame. For protected content, there are too many faces to store all embeddings, so filtering is required. Based on a similarity threshold determined for each embedding model, similar faces can be clustered and then the best face in each cluster can be selected based on a "matchability" score. This matchability score is important in predicting how well a face will match its mirror-inverted self and keeping only those faces that can be matched accurately. The data associated with the best face is then stored and the embedding is maintained in the corresponding ANN DB (no cropping of the actual face is maintained).
[0042] In a query, more faces in the video than in the capture can be maintained, and each can be searched in the ANN DB. As with images, faces detected within a distance threshold are considered a match, and these matches must have similar facial expressions and angle scores to pass secondary validation. Because the same face model is used for images and videos, embeddings from images (videos) can be searched against secured embeddings of videos (images).
[0043] After a video fingerprint is generated (in the form of a list or array of image fingerprints corresponding to segments / sub-portions / keyframes of the video), the system can compare the video fingerprint to known video fingerprints stored in a database.
[0044] With regard to computing an audio fingerprint, it may be more complicated than a video or image, since audio content is a sequence of very short duration values corresponding to changes in air pressure, and even small imperceptible changes can significantly alter these values. To get around such limitations, the system may use a noise gate during pre-processing to remove noise above and / or below a certain volume. This pre-processing may also remove very high and very low frequencies. A Fast-Fourier Transform (FFT) may be applied to extract a spectrogram of the audio content. Thus, the basic steps of an audio fingerprinting algorithm may be as follows: (1) Calculate the spectrogram of the audio signal. (2) Find the highest amplitude spectral peaks, separated by a certain distance. To avoid generating too few hashes when the file is mostly silent, or too many hashes when there is a lot of voice activity, amplitude filtering may be performed dynamically based on the overall voice level in the file. A voice activity threshold (for determining the overall voice level) may be predetermined before starting the pre-processing process.
[0045] The system can, for example, use a peak detection algorithm to extract amplitude peaks, and the locations of those peaks can serve as input to a hash function, the output of which is the audio fingerprint (a hash function is a one-way mathematical function that converts an array of bytes into a small unique identifier).
[0046] Unlike perceptual hash functions used in image and / or video analysis, audio hashes / fingerprints do not provide a level of similarity between outputs, i.e., they may only be used to search for exact matches. To address this issue, two aspects may be used. First, by consistently pre-processing the audio in a pre-defined manner (i.e., using consistent noise gates, frequency filtering, etc.), it is ensured that the audio being analyzed is analyzed in the same manner as previously identified audio. Second, the system analyzes specific segments / sub-portions of audio tracks that have a pre-defined length. If the hash output of a 5-second audio clip is compared to the hash output of a 3-second clip, audio matching will not work. To accommodate this, the system may utilize a pre-defined set duration of audio content. If necessary, the system may adjust the audio encoding during pre-processing to ensure that the audio content meets the required formatting standards.
[0047] The result of Audio Fingerprinting is an Audio Fingerprint consisting of a list or array of hashed audio fingerprints of subportions / segments of the entire audio track.
[0048] After the pre-processing of the audio steps mentioned above, an audio fingerprint may be created by generating pairs of spectral peaks. The peaks are defined by their time and frequency location (t, f). Two peaks P1, P2 are defined by (t1, f1), (t2, f2), where P2 occurs later in time than P1. A hash (also known as a fingerprint) is defined by (f1, f2, t2-t1). The system may also store t1 as the time at which the fingerprint appears in the file. To simplify this further, the initial triplet (f1, f2, t2-t1) may be encoded as a single integer.
[0049] Upon capture, the fingerprint is stored in a database, preferably a key-value database, although other options are possible. Since an exact match is required here, an ANN DB is not required.
[0050] At query time, the input audio fingerprint is matched against the DB and the number of matches is counted. Content / videos with sufficient matches are considered as candidates and become input to the second stage of the query algorithm. This second stage checks if the candidates are valid by making sure that the matches have temporal coherence and form a continuous sequence. After the correct candidates are validated, the algorithm returns them to the user as a list of possible banned content or returns nothing if no matching videos were found.
[0051] In some configurations, alternative audio fingerprinting methods are used to generate audio quad hashes that are more robust to audio modifications. In such configurations, a spectrogram is computed from the preprocessed audio signal and the point of maximum amplitude is selected. Peaks in groups of four are then selected with the condition that two of them (A-bottom left, B-top right) must be in opposite corners of a rectangle and another two (C,D) must be located inside. Point A's (x,y) coordinates are set as (0,0), B's as (1,1), and the coordinates of C and D are normalized based on A and B to make C' and D'. In this way, the (x,y) values of C and D are strictly contained within the range (0,1). These four data points (x_C', y_C', x_D', y_D') form one audio quad hash.
[0052] For each file, two sets of audio quad hashes are generated: a smaller set (the "reference" hashes) containing up to R hashes per second, and a larger set (the "query" hashes) containing up to Q (Q>>R) hashes per second. Also, the parameters that control how spectrogram peaks are selected are allowed to vary slightly between the reference and the query hashes. The idea behind this is that the small differences between both hash sets make the method robust to audio changes.
[0053] At ingestion time, the reference hashes are added to the ANN DB. At query time, two sequential operations are performed: (1) an ANN query is performed using the reference DB with the reference hashes of the input video (which contains the reference hashes of all clips that should be flagged), giving a list of possible candidates; (2) a verification algorithm is then performed for each selected candidate using a sample of the reference hashes from the candidate and the query hash of the input video. Such verification may be performed by any suitable comparison algorithm.
[0054] After the image, video, and / or audio tracks of a piece of content have been processed and fingerprinted by the process described above, those fingerprints are stored in the system's database. At this point, the content may be discarded and removed from the system's infrastructure. However, the system then uses a matching engine to compare fingerprints between the protected catalog (copyrighted material and / or other prohibited content) and the newly uploaded content.
[0055] Contrary to classical hash functions, where small changes in the input result in large changes to the resulting hash value, perceptual hashes only change slightly when the input changes slightly. Thus, by considering the perceptual hash as a high-dimensional vector corresponding to an image or segment of a video, a video / image fingerprint can determine whether the image is within some range of similarity to a known video / image fingerprint. If the vectors are close to each other, they are likely to represent similar content. Thus, for each query, a fast nearest neighbor database is used to find the closest match to the image / video fingerprint of a particular subject. A similar process can be used for audio tracks, where many candidates for a match are identified and then further processed and compared to rank the candidates with the highest likelihood of matching.
[0056] Each time a query to the system is performed, the content is downloaded, pre-processed, fingerprinted, and queried against a database containing the protected catalog. The most likely fingerprints can be grouped by content (video, image, or audio), and an overlap can be calculated between the fingerprint results and the query, preferably in seconds, although other metrics are possible. For example, based on the overlap of video and audio fingerprints, the system can calculate an overlap of 300 seconds (5 minutes) between two pieces of content. These metrics and the overlap can be converted into a summary response in which all matching content is ranked according to the amount of time (seconds) of overlap. In other configurations, this ranking can be based on a percentage of similarity or other similarity metrics. In other words, when a video is uploaded and multiple protected videos are matched, the video with the longest matching length (contiguous or non-contiguous) can be returned to the integrator in descending order. In some configurations, this list can be presented to a user via a User Interface (UI), where the user can review the comparison between the original content and the matching content and determine whether the new content should be added to the database and made available for distribution. In other configurations, any content having a level of similarity above a threshold amount (such as 50%) can be automatically removed.
[0057] When a new piece of content is detected (whether or not it corresponds to prohibited content), a fingerprint may be automatically added to the system catalog of content (note: this content is not stored in the system, only the fingerprint is stored in the system). When a new piece of content is determined to correspond to known prohibited content (e.g., containing copyrighted material, adult content, non-consensual content, etc.), not only is a fingerprint added to the system catalog, but all previous queries matching this new content are notified and / or removed. For example, if a video was uploaded to a VSP or social media platform partner a few weeks ago, and a similar video is protected a few weeks later, the original upload / query is notified by the system. This functionality is implemented via a callback to the system integrator.
[0058] In some configurations, if the content appears to be prohibited content based on the similarity of the video fingerprints, the system can trigger additional verification using time series alignment against the contrast signal created to select key frames. This contrast signal can be an irreversible one-dimensional time series of the video. A particular time series can be compared to a candidate time series using Mueen's algorithm for similarity search (MASS) and dynamic time warping (DTW). MASS aligns two time series by transforming the time series into the frequency domain and calculating the Euclidean distance between the frequencies. DTW finds the best way to map one time series to another. MASS and DTW each produce a distance measure that is used to determine whether the candidates match. In particular, MASS and DTW align smaller chunks of a particular queried time series to the candidate time series and then determine whether these alignments are (1) within a distance threshold and (2) whether the chunks form a contiguous alignment. Varying the chunk size allows an additional layer of validation that the time series are properly aligned. In other words, the MASS and DTW algorithms can calculate how geometrically similar a particular segment of audio or video is to a known segment, generating all of the distances from the query to subsequences of the long time series. However, such alignment is computationally expensive and cannot be performed for all protected videos, but only for a candidate subset.
[0059] Due to the computational cost of running MASS, an example system flow when MASS is executed may be as follows: (1) receive content, (2) separate video and audio content, (3) preprocess video and audio content, (4) generate fingerprints of the preprocessed video and audio content, (5) determine that the video or audio exceeds a threshold level of similarity to known prohibited content, (6) initiate execution of the MASS algorithm on the video or audio based on exceeding the threshold, and (7) receive MASS distance results. If the segment is deemed not to match when the predefined MASS distance is calculated for a particular segment, other similarity results may be presented to MASS.
[0060] In one example, time series alignment (MASS and / or DTW) is performed on the contrast signal of the video match candidates, but the same alignment method can be performed on the mono audio signal to confirm the audio match.
[0061] After the fingerprint is generated, the system can attempt to identify a match with a previously stored fingerprint. In the case of an image or GIF, all matching algorithms (basic hashing, DPH, face detection) can be run in parallel and the results from all of the matching algorithms can be used to determine if a match is found.
[0062] Matching logic for videos (or other content that contains both video and audio data) is a bit more complex. On the one hand, matches from video frames and matches from audio signals are desirable. Matches that contain only audio information or only video information may be correct but may require further verification to reduce the number of matches that must be manually reviewed. Also, MASS and DTW are computationally expensive and cannot be performed for every single video in the DB. One possible approach to these problems is: (1) perform parallel queries of basic video hash, basic audio hash, audio quads, and keyframes, and (2) if there is a match between the basic video hash and either the basic audio hash or audio quads on the same video, this is considered a match. If there is no match at all and the keyframes also yield low match counts, the video is declared good. If there is a match only on the video signal, or only on the audio signal, or no match on either but the keyframes of the DPH return high match counts, the candidate is sent to MASS and DTW time series alignment for verification. If the MASS output and the DTW output both align, a match is declared; if not, the file is suspect and must be manually reviewed.
[0063] As previously mentioned, the generated video and audio fingerprints can be lists or sequences of fingerprints of segments or sub-parts, so identifying a match against a fingerprint of a single segment may result in false positives. To improve matching, the system can look for a match of a contiguous sequence of fingerprints of those segments. For example, if a sequence of segments of a video fingerprint includes "A", "B", "C" and "D" as segment fingerprints, respectively, resulting in a fingerprint of ABCD, the system can look for a contiguous sequence of either the entire fingerprint (ABCD) (which can be disrupted by removing or replacing the fingerprint of a single segment) and / or a portion of the fingerprint (e.g., ABC or BCD) (which is more difficult to disrupt). This also ensures that the video is matched, even if more content is added before or after a particular segment, with prohibited content still discoverable.
[0064] Having generally described the system, the disclosure now turns to the examples shown in the figures. FIG. 1 illustrates an embodiment of an exemplary system 100. As shown in the figure, a media file 102 is uploaded to the system 100, and the media file is preprocessed (104). For example, if the media file is a moving image that contains both audio and video components, preprocessing may include filtering the audio through a noise gate, removing various high and / or low frequencies in the case of audio, ensuring that the encoding matches a common / desired format, etc. Examples of video preprocessing may include removing banding from the video, removing / changing colors to grayscale, changing the encoding to a common / desired format, etc. The system 100 then performs a perceptual hash 112 on the preprocessed video 106 to generate a video hash 116, which is a video fingerprint. The system 100 also performs a hash 114 on the preprocessed audio 108 to generate an audio hash 118, which is an audio fingerprint. In this example, the system 100 also identifies keyframes 110 from within the preprocessed video, and the system performs a perceptual hash 126 (such as a DPH embedding) on the keyframes 110 to generate keyframe hashes 128, which are keyframe fingerprints.
[0065] Next, the system 100 compares (130, 132, 134) the video hash 116, audio hash 118, and keyframe hash 128 to the known content fingerprint 120, respectively. In some configurations, these comparisons 130, 132, 134 may occur simultaneously, while in other configurations, the comparisons 130, 132, 134 may occur sequentially. For example, the known (audio) fingerprint 120 may be compared to the newly generated audio fingerprint 118 at the same time that the known (video) fingerprint 120 is compared to the newly generated video fingerprint 116. Alternatively, the known (keyframe) fingerprint 120 may not be compared to the newly generated keyframe fingerprint 128 until the comparison 130 of the known (video) fingerprint 120 to the newly generated video fingerprint 116 is completed. In some configurations, this sequential process of fingerprint comparison may depend on a threshold level of similarity that has been determined by the initial fingerprint comparison. For example, keyframe comparison 134 may not occur unless video comparison 130 and / or audio comparison 132 are first completed and indicate that potentially prohibited content has been detected.
[0066] After the comparisons 130, 132, 134 are completed, the system 100 may filter 122 the content based on the comparisons 130, 132, 134. For example, if the comparisons 130, 132, 134 indicate that the newly submitted media file 102 is 100% similar to a known copyrighted file, the system may immediately remove the media file 102 and / or provide a notice to the user attempting to upload the media file 102 that the copyrighted material being uploaded that the user does not own is prohibited (and thus is prohibited content). If the comparisons 130, 132, 134 indicate that the media file 102 may contain other prohibited content, such as underage sexual content, but the comparisons 130, 132, 134 are not entirely clear, the filter 122 may request a human review of the content. If the comparisons 130, 132, 134 indicate that the content is new, the system may store the fingerprints 116, 118, 128 in a database, and the fingerprints 116, 118, 128 may be used in inspecting future media files. After the comparisons 130, 132, 134 are complete and the filtering process 122 does not identify the content as prohibited for copyright or content reasons, the system 100 may provide an indication 124 to the VSP that the content 102 is ready for upload, or may begin uploading if the system 100 is configured to upload the content 124.
[0067] An example of comparing a known fingerprint to a new fingerprint is shown in Figure 2. As shown, a video 208 is being compared to previously enrolled / fingerprinted videos 202. Each of the previously enrolled videos 202 has been hashed / fingerprinted / converted into an embedding (a mathematical vector, as discussed above) and those fingerprints are stored in an ANN DB 206, referred to as the ANN DB.
[0068] Upon receiving a new video 208, the system generates a hash / embedding / fingerprint 210 of the new video and submits the fingerprint 210 as a query to the ANN DB 206. The fingerprint 210 is compared to known videos 202 and content fingerprints 204. If the comparison does not reveal a clear match, the new video 208 can proceed to candidate verification 212, where additional information about the new video 208 may be required. Additionally, unless the fingerprint 210 of the new video 208 is an exact match to the known fingerprint 204 stored in the ANN DB 206, the new fingerprint 210 may be stored in the ANN DB 206 for comparison against future content uploads.
[0069] 3 shows an example of pre-processing a video to create a video fingerprint. In some configurations, the order or inclusion of various steps may be changed. Where applicable, the same process or a slightly modified process may also be applied to individual images.
[0070] When the system receives video content 304, the video content 304 may be normalized to a reduced frame rate and resolution. This normalization can be to convert the video content 304 to grayscale 302. This sampling of the content before generating the fingerprint ensures consistency between content items. The system then detects the length of the video and identifies portions in the video that contain unbroken bands / blocks of color 306 (the bands / blocks 306 do not necessarily have to be black as shown). The bands do not necessarily have to be on the horizontal sides of the image content, but can also be located vertically (at the top or bottom) of the content, on only one side of the content (only on the top, bottom, left, or right), or any combination of these. The system identifies the corresponding coordinates of the blocks and generates a block cropped, band-free image / content 308 for each frame in which the block is present.
[0071] The remaining content is then divided into a grid 310. As shown in the figure, the grid is 8x9, although in other configurations a different number of cells are possible. With regard to the number of cells, if the system is planning to use an 8x9 grid for video content, then consistency between fingerprinting instances is crucial to ensure a proper match, such that all video content needs to be evaluated using the 8x9 grid.
[0072] As mentioned above, each cell in the 8×9 grid contains the average gray level value of the pixels in the original image 308 that are included on that image region. This 8×9 grid 310 can be two 8×8 grids (left grid (L) and right grid (R)) that overlap in all cases except one column. The system can then compare the gray level values for each cell in every grid and write down a 1 if L>R (cells in the left grid have a larger gray level value than cells in the right grid) or write down a 0 if R>-L. This creates a 64-bit hash based on the L / R comparison of the input image. If the image contained multiple frames (such as in a GIF (Graphics Interchange Format) or video segment), this operation is performed for all frames in the image.
[0073] Upon capture, this 64-bit hash (or hashes) is generated and then stored in the ANN DB. As an example, the first value of the fingerprint corresponds to the top left cell [1,1] of the 8x8 grid of 8x8 comparisons, the second value is the cell immediately to the right [1,2], the third value is to the right of that [1,3], etc., the ninth value is the first cell of the second row [2,1], etc. In this way, we finally have a 64-bit long number (floating point) that constitutes the fingerprint of this image, or averaged frame in a video segment. It is noted that in other configurations, the way in which the fingerprint is formed from the cells, or the number of bits in the fingerprint, can vary while still being within the scope of this disclosure.
[0074] Figure 4 shows an example of creating a keyframe fingerprint. In this example, a video includes a series of frames 402. The system can identify special frames (keyframes 404) from the series of frames 402 and compress the keyframes 404 using deep learning. The system can extract the keyframes 404 as shown in the figure and then create a 128-bit floating point embedding (fingerprint) 408 using a deep perceptual hash 406 algorithm as previously described.
[0075] In some configurations, the keyframes 404 may be extracted by the system by: (1) scoring each frame based on the difference between the frame and the previous frame and the color change within the frame; and (2) grouping the frames into buckets of time T (e.g., a window of T=5 seconds) and picking the top N highest scoring frames in each bucket as keyframes (e.g., for a single keyframe 404, use N=1). After the keyframes 404 are extracted, in some configurations, the system may further compress the keyframes 404 into a smaller embedding (such as a 64-bit float for a 128-bit embedding 408).
[0076] Figure 5 illustrates an example of pre-processing a segment 510 of a video 508. As shown, a series of frames 502 in the segment 510 are received, and the system may normalize the frames before performing the keyframe identification described in Figure 4. After the frames are normalized with respect to size, color (converted to grayscale, etc.), the system may calculate 504 the grayscale contrast between the frames 502 to generate a frame contrast difference 506. The contrast difference 506 may be used to calculate a fingerprint of the segment 510, as previously described.
[0077] FIG. 6A shows an example of an audio fingerprint comparison 602. This figure is a portion of a user interface (UI) that a user can use to see, for example, how the content being evaluated by the system compares to a number of other files 604. In this example, the audio fingerprint has an 88.04% similarity to "File 9," indicated by the black line 614 in the "File 9" row. No significant similarities are shown with the other compared files. A user can use this information to determine whether the content being evaluated contains prohibited content. In some configurations, if the similarity is not very precise, the user interface can use different color tones to indicate the level of similarity of a particular section.
[0078] For example, FIG. 6B shows an example of a video fingerprint comparison 606. As with FIG. 6A, this figure can be part of a UI used during the content evaluation process. In this example, there is a band 616 on the right side, which shows that the level of similarity between two pieces of content can be indicated by a color gradient. As shown by the solid black line 618, "File 9" still shows a high level of similarity, but there are also multiple other pieces of content that contain matching segments of other files 608, such as File 4, which in this case contains two discontinuous portions 620 with a high degree of similarity, and File 14 622, which contains a single unbroken portion, but has a lower level of similarity (shown in light gray). Again, the user can use this information to determine whether the content being evaluated contains prohibited content.
[0079] Figure 6C shows an example of a keyframe fingerprint comparison 610 for the same content for which an audio comparison was shown in Figure 6A and a video comparison was shown in Figure 6B. However, as shown, the keyframe fingerprint comparison does not reveal any similarities with any of the files under consideration 612. Again, a user can use this information to determine whether the content being evaluated contains prohibited content.
[0080] 7 shows an example of a fingerprint comparison. As shown, the submitted video content portion includes frames associated with different segments 702, 704, 706. The system calculates average values for each segment, generating average1 708, average2 710, and average3 712. Each of these averaged frames 708, 710, 712 are submitted to a perceptual hashing algorithm 714, respectively, resulting in a segment fingerprint "ABCD" 716 corresponding to segment 702, a segment fingerprint "ABCE" 718 corresponding to segment 704, and a segment fingerprint "XBCE" 720 corresponding to segment 706.
[0081] A complete fingerprint 722 of the video content is shown as a list of fingerprints of individual segments, with "ABCD" immediately followed by "ABCE" followed by "XBCE". The system compares the fingerprints of the series of segments 724 to other known fingerprints 726, 728, 730. As shown, fingerprint (1) 726 has a 100% match 732 with respect to the series of segments, i.e., each of the fingerprints 716, 718, 720 is present in fingerprint 726 in the same (in this case consecutive) temporal order as in the submitted video content. Fingerprint (2) 728 also has a 100% match 734 with the fingerprints of the same segments and in the same temporal order as in the submitted video content. However, the match is not at the same temporal location in the video as fingerprint (1) 726; instead, the match is earlier in the known content (indicated by fewer dots preceding the fingerprints of the series of segments). Such a match may occur, for example, if a portion of the video has been removed from the video associated with the second fingerprint 728. Fingerprint (3) 730 contains segment fingerprints "ABCD" and "ABCE", but a slightly different segment fingerprint "ZBCE" as a subsequent segment fingerprint. Thus, the system assigned an 80% match 736 between the submitted video content and this fingerprint 730. The system can provide notification to the conformance review team regarding the matching similarities 732, 734, 736 and / or make a decision regarding banning the content based on the detected similarities.
[0082] 8 illustrates a first exemplary user interface (UI) 810 displaying matching portions between uploaded and known content for different types of fingerprint comparisons. In this example, the user interface 810 includes portions associated with an audio fingerprint comparison 802, a video fingerprint comparison 804, a MASS comparison 806, and a keyframe fingerprint comparison 808. As shown, there are audio 802, video 804, and keyframe 808 comparison portions that match (812) between the uploaded and known content. A user reviewing the comparisons 802, 804, 806, 808 via the UI 810 can examine each matching section in determining whether the content being evaluated includes prohibited content.
[0083] FIG. 9A illustrates a second exemplary user interface 900 that displays matching portions 922. Within this UI, the user can filter content that is an exact match 902 for the content under investigation, suspicious content, etc. Similarly, the user can filter the matching content based on engine type 904 (e.g., only content that matches based on audio fingerprint comparison, and / or only content that matches based on video fingerprint comparison), threshold amount 906, and only results with more than x% matches are displayed (the user can set the threshold x). The UI can rank the compared content, with the most similar matches 910 displayed first, the second most similar matches 912 displayed second, and so on. In the illustrated configuration, the user can have the option to view an animated video comparison 908 of the content being examined and the known content. The UI can also display metadata (title, uploading user's name, identification token, upload date, etc.) and matching portions / segments of the different comparisons 922. For example, a chart similar to the UI shown in Figure 8 may be included here, providing the user with an indication of the portions of the content that match the known content with respect to audio, video, MASS, keyframe, and / or other comparisons. The UI may also display a percentage amount identifying how much of the known content matches the content under review, and provide the user with options such as "copy response," "confirm match," "reject match," etc.
[0084] 9B shows a third exemplary user interface displaying matches. In some configurations, this third example can be a continuation of the second example shown in FIG. 9A. Again, different comparisons 922 are displayed with user options such as "copy response" 916, "confirm match" 918, and "reject match" 920. In this case, however, only one matching video 914 exists.
[0085] FIG. 10 illustrates an exemplary system architecture. As shown, protection and query requests are sent by an “Integrator” 1002 (such as any VSP or third-party application) to a “Load Balancer” 1004. The Load Balancer redirects the requests to a cluster of web servers 1006 that act as “API endpoints”. These web servers validate the requests and then send them via an “Event Queue” 1008 that is distributed to “Compute Nodes” 1010. A “Processing Cluster” 1012 then attempts to match the uploaded content using a “Fingerprint” 1018 stored in the Storage Cluster. After the protection or query request is completed, the data is sent back to the Integrator 1002 using a “Callback” 1016.
[0086] Infrastructure metrics may be continuously monitored 1014 to ensure that the system infrastructure remains operational at all times. If infrastructure metrics deviate from normal, indicating that an error is occurring, an alert may be triggered to notify appropriate personnel. Protection requests, inquiry requests, and reports 1020 regarding system performance may be provided at regular intervals, or whenever an event meets a threshold level of immediacy.
[0087] All computational and storage resources are highly scalable and capable of supporting large volumes of content and fingerprinting activity. All systems can be continuously backed up and disaster recovery strategies are in place to prevent any data loss in the event of a major infrastructure incident.
[0088] FIG. 11 illustrates an embodiment of an exemplary method that may be performed, for example, by a computer system. As shown, the method may include receiving, at the computer system, a media file including video and audio (1102), pre-processing, by at least one processor, the media file (1104), generating an altered copy of the video (1106), an altered copy of the audio (1108), and keyframes within the video (1110). The system may then create, by at least one processor, a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of distinct segments of the altered copy of the video (1112). Similarly, the system may create, by at least one processor, an audio fingerprint of the altered copy of the audio (1114), the audio fingerprint including a second vector of distinct segments of the altered copy of the video, and create, by at least one processor, a keyframe fingerprint based on the keyframes (1116).
[0089] Once the fingerprints are created, the system may perform, by the at least one processor, a video comparison (1118) of the video fingerprint with a previously identified video fingerprint of the comparison media file, an audio comparison (1120) of the audio fingerprint with a previously identified audio fingerprint of the comparison media file, and a keyframe comparison (1122) of the keyframe with previously identified keyframes of the comparison media file. The comparison media file may be, for example, another image, video, or other media content to which the current media file is being compared. Typically, the comparison media file has already been analyzed such that the fingerprints (audio, video, images, keyframes, etc.) have already been generated and stored. These previously identified audio fingerprints, video fingerprints, and keyframe fingerprints may then be used in each comparison to the fingerprint of the media file being analyzed. The system may then determine, by the at least one processor, that the media file contains prohibited content based on at least one of the video comparison, audio comparison, and keyframe comparison (1124). Prohibited content may include any content that the media file owner or uploader is not legally permitted to share, including copyrighted media, videos or images of underage sexual content, videos or images obtained without consent, etc.
[0090] An illustrated method, in another configuration, includes receiving, at a computer system, a media file including video and audio; pre-processing, by at least one processor, the media file to generate an altered copy of the video, an altered copy of the audio, and keyframes within the video; creating, by the at least one processor, a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of distinct segments of the altered copy of the video; creating, by the at least one processor, an audio fingerprint of the altered copy of the audio, the audio fingerprint including a second vector of distinct segments of the altered copy of the video; creating, by the at least one processor, a keyframe fingerprint based on the keyframes; and based on the video comparison, generating, by the at least one processor, a keyframe fingerprint based on the keyframes using at least one of MASS (Mueen's Similarity Search Algorithm) and DTW (Dynamic Time Warp). performing, by the at least one processor, a video comparison of the video fingerprint with a video fingerprint identified prior to the comparison media file; performing, by the at least one processor, an audio comparison of the audio fingerprint with an audio fingerprint identified prior to the comparison media file; performing, by the at least one processor, a key frame comparison of the key frames with key frames identified prior to the comparison media file; performing, by the at least one processor, a second video comparison of the second video fingerprint with a second video fingerprint identified prior to the comparison media file, where the second video fingerprint identified prior to the comparison media file was generated using at least one of MASS and DTW; and determining, by the at least one processor, that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, the key frame comparison, and the second video comparison.
[0091] Another alternative version of the illustrated method may be receiving, at a computer system, a media file containing video and audio; creating, by at least one processor of the computer system, a video fingerprint, an audio fingerprint, and a keyframe fingerprint of the media file; performing, by the at least one processor, a video comparison of the video fingerprint with previously identified video fingerprints; performing, by the at least one processor, an audio comparison of the audio fingerprint with previously identified audio fingerprints; performing, by the at least one processor, a keyframe comparison of the keyframe fingerprint with previously identified keyframe fingerprints; and determining, by the at least one processor, that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison.
[0092] Yet another alternative version of the illustrated method may include receiving, at a computer system, a media file containing video and audio; pre-processing, by at least one processor, the media file to generate a pre-processed media file containing the pre-processed video and the pre-processed audio; creating, by the at least one processor, a first video fingerprint of the pre-processed video using a perceptual hashing algorithm, creating an audio fingerprint of the pre-processed audio, and creating a second video fingerprint of the pre-processed video using keyframes in the pre-processed video; comparing the first video fingerprint, the second video fingerprint, and the audio fingerprint to known fingerprints to obtain a comparison result; and determining, by the at least one processor, that the media file contains prohibited content based on the comparison.
[0093] In configurations used for image analysis (or other visual media), pre-processing, fingerprinting, and comparison may be coordinated as disclosed above.
[0094] In some configurations, the video comparison identifies individual video segments within the individual segments that meet a predetermined threshold level of video similarity between the video fingerprint and a previously identified video fingerprint of the comparison media file, the audio comparison identifies individual audio segments within the individual segments that meet a predetermined threshold level of audio similarity between the audio fingerprint and a previously identified audio fingerprint of the comparison media file, and the keyframe comparison identifies individual keyframes that meet a predetermined threshold level of keyframe similarity between the keyframes and a previously identified keyframe of the comparison media file.
[0095] In some configurations, pre-processing of the media file may include parallel processes for creating an altered copy of the video and an altered copy of the audio, and for identifying key frames within the video.
[0096] In some configurations, the performance of the video comparison, audio comparison, and keyframe comparison occurs in parallel.
[0097] In some configurations, pre-processing the media file to generate an altered copy of the video may include performing at least one of: changing, by at least one processor, a frame rate of the video to a predetermined frame rate; changing, by at least one processor, a resolution of the video to a predetermined resolution; and removing, by at least one processor, banding from at least one side of the video.
[0098] In some configurations, pre-processing the media file to generate an altered copy of the audio may include converting the audio to a frequency domain by a Fast Fourier Transform performed by at least one processor to generate frequency domain audio, and filtering, by the at least one processor, from the frequency domain audio at least one of sounds above a first predetermined frequency, sounds above a first predetermined decibel, sounds below a second predetermined frequency, and sounds below the second predetermined decibel.
[0099] In some configurations, pre-processing of a media file to generate key frames in a video may include identifying, by at least one processor, a predetermined amount of entropy that should be present between key frames; identifying, by at least one processor, a first frame in a segment of the video as a key frame; adding, by at least one processor, the key frame to a number of key frames over the length of the video; comparing, by at least one processor, sequentially subsequent frames from the key frame until the predetermined amount of entropy is detected between the key frame and a subsequent frame; and repeatedly identifying, by at least one processor, the subsequent frame as the first key frame to generate the key frames.
[0100] In some configurations, the illustrated method may further include creating, by the at least one processor, a second video fingerprint of the altered copy of the video using MASS (Mueen's Similarity Search Algorithm) based on the video comparison, and performing, by the at least one processor, a second video comparison of the second video fingerprint with a previously identified second video fingerprint of the comparison media file, where the previously identified second video fingerprint of the comparison media file was generated using MASS, and determining that the media file contains prohibited content is further based on the second video comparison.
[0101] In some configurations, creating the video fingerprint may further include detecting a face in the altered video by at least one processor, cropping the face from the altered video by at least one processor, and creating a face print of the face by at least one processor performing a perceptual hashing algorithm, the video fingerprint further including the face print, and the video comparison further includes comparing the face print to known face prints and storing the face print in a database storing known face prints upon not detecting a match in the known face prints.
[0102] In some configurations, the audio comparison, video comparison, and keyframe comparison include comparing successive sub-portions of the audio fingerprint, successive sub-portions of the video fingerprint, or successive sub-portions of the keyframe fingerprint with successive sub-portions of the audio fingerprint identified prior to the comparison media file, successive sub-portions of the video fingerprint identified prior to the comparison media file, or successive sub-portions of the keyframe fingerprint identified prior to the comparison media file, respectively.
[0103] With reference to FIG. 12, an exemplary system includes a general-purpose computing device 1200 including a processing unit (CPU or processor) 1220 and a system bus 1210 coupling various system components to the processor 1220, including a system memory 1230 such as read-only memory (ROM) 1240 and random access memory (RAM) 1250. The system 1200 may include a cache of high-speed memory connected directly to the processor 1220, connected in close proximity, or integrated as part of the processor 1220. The system 1200 copies data from the memory 1230 and / or storage device 1260 to the cache for faster access by the processor 1220. In this manner, the cache provides improved performance and prevents delays to the processor 1220 while waiting for data. These and other modules may control or be configured to control the processor 1220 to perform various actions. Other system memories 1230 may also be used. The memory 1230 may include multiple different types of memory having different performance characteristics. It may be understood that the present disclosure may operate on a computing device 1200 including two or more processors 1220, or on a group or cluster of computing devices networked together to provide greater processing power. The processor 1220 may include any general-purpose processor and hardware module, or a special-purpose processor where software instructions are embedded into the actual processor design, in addition to software modules such as module 1 1262, module 2 1264, and module 3 1266 stored in storage device 1260 configured to control the processor 1220. The processor 1220 may essentially be a fully self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.
[0104] The system bus 1210 may be any of several types of bus structures, including a memory bus or memory controller, a peripheral bus, and a local bus using any of a variety of bus architectures. A basic input / output system (BIOS), stored in ROM 1240 or the like, may provide basic routines that help transfer information between elements within the computing device 1200, such as during start-up. The computing device 1200 further includes a storage device 1260, such as a hard disk drive, a magnetic disk drive, an optical disk drive, a tape drive, or the like. The storage device 1260 may include software modules 1262, 1264, 1266 for controlling the processor 1220. Other hardware or software modules are contemplated. The storage device 1260 is connected to the system bus 1210 by a drive interface. The drives and associated computer-readable storage media provide non-volatile storage of computer-readable instructions, data structures, program modules, and other data for the computing device 1200. In one aspect, a hardware module that performs a particular function includes software components stored on a tangible computer-readable storage medium in association with the necessary hardware components, such as the processor 1220, the bus 1210, the display 1270, etc., to perform the function. In another aspect, the system may employ a processor and a computer-readable storage medium for storing instructions that, when executed by the processor, cause the processor to perform a method or other particular action. Depending on the type of device, such as whether the device 1200 is a small handheld computing device, a desktop computer, or a computer server, basic components and appropriate variations are contemplated.
[0105] Although the example embodiments described herein employ a hard disk 1260, other types of computer readable media capable of storing data accessible by a computer may be used in the exemplary operating environment, such as a magnetic cassette, a flash memory card, a digital versatile disk, a cartridge, a random access memory (RAM) 1250, and a read only memory (ROM) 1240. Tangible computer readable storage medium, computer readable storage device, or computer readable memory device explicitly excludes media such as ephemeral waves, energy, carrier signals, electromagnetic waves, and the signals themselves.
[0106] To enable user interaction with the computing device 1200, the input device 1290 represents any number of input mechanisms, such as a microphone for speech, a touch screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. The output device 1270 can also be one or more of a number of output mechanisms known to those skilled in the art. In some cases, a multimodal system allows a user to provide multiple types of input to communicate with the computing device 1200. The communication interface 1280 typically controls and manages user input and system output. There is no limitation to operation with any particular hardware configuration, and thus the basic features of this specification can be easily substituted for improved hardware or firmware configurations as they are developed.
[0107] Use of language such as "at least one of X, Y, and Z," "at least one of X, Y, or Z," "at least one or more of X, Y, and Z," "at least one or more of X, Y, or Z," or "at least one of X, Y, and / or Z" is intended to include both single items (e.g., only X, or only Y, or only Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase "at least one of" and similar phrases are not intended to convey a requirement that each possible item must be present, although each possible item may be present.
[0108] The various embodiments described above are provided merely as examples and should not be construed as limiting the scope of the present disclosure. Various modifications and changes may be made to the principles described herein without departing from the spirit and scope of the present disclosure, other than in accordance with the example embodiments and applications illustrated and described herein.
Claims
1. Receiving, at a computer system, a media file including video and audio; pre-processing, by at least one processor, the media file; a modified copy of said video; a modified copy of said audio; and Keyframes in the video and creating, by the at least one processor, a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of distinct segments of the altered copy of the video; creating, by the at least one processor, an audio fingerprint of the altered copy of the audio, the audio fingerprint including a second vector of the individual segments of the altered copy of the video; generating, by the at least one processor, a keyframe fingerprint based on the keyframes; performing, by the at least one processor, a video comparison of the video fingerprint with a previously identified video fingerprint of a comparison media file; performing, by the at least one processor, an audio comparison of the Audio Fingerprint with previously identified Audio Fingerprints of the comparison media file; performing, by the at least one processor, a keyframe comparison of the keyframe fingerprint with previously identified keyframe fingerprints of the comparison media file; determining, by the at least one processor, that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison; A method comprising:
2. said video comparison identifying individual video segments within said individual segments that satisfy a predetermined threshold level of video similarity between said video fingerprint and said previously identified video fingerprint of said comparison media file; said audio comparison identifying individual audio segments within said individual segments that satisfy a predetermined threshold level of audio similarity between said Audio Fingerprint and said previously identified Audio Fingerprint of said comparison media file; The method of claim 1 , wherein the keyframe comparison identifies individual keyframes that satisfy a predetermined threshold level of keyframe similarity between the keyframes and the previously identified keyframes of the comparison media file.
3. 2. The method of claim 1, wherein the pre-processing of the media file includes parallel processes for creating the modified copy of the video and the modified copy of the audio and for identifying the key frames within the video.
4. The method of claim 1 , wherein the performing of the video comparison, the audio comparison, and the keyframe comparison occurs in parallel.
5. said pre-processing of said media file to generate said modified copy of said video, changing, by the at least one processor, a frame rate of the video to a predetermined frame rate; changing, by the at least one processor, a resolution of the video to a predetermined resolution; and removing, by said at least one processor, banding from at least one side of said video. The method of claim 1 , comprising performing at least one of the following:
6. said pre-processing of said media file to generate said modified copy of said audio, transforming the audio into a frequency domain by a fast Fourier transform performed by the at least one processor to generate a frequency domain audio; from the frequency domain audio, a sound exceeding a first predetermined frequency; a sound exceeding a first predetermined decibel; a sound below a second predetermined frequency; and A sound below a second predetermined decibel. and filtering out at least one of The method of claim 1 , comprising:
7. 20. The method of claim 19, wherein the pre-processing of the media file to generate the keyframes in the video comprises: identifying, by the at least one processor, a predetermined amount of entropy that should be present between key frames; identifying, by the at least one processor, a first frame in a segment of video as a key frame; Over the length of the video, adding, by the at least one processor, the first frame to the key frames; comparing, by the at least one processor, successive frames from the first frame until the predetermined amount of entropy is detected between the first frame and the subsequent frame; and identifying, by the at least one processor, the subsequent frame as the first frame. Repeatedly, generating said key frames; The method of claim 1 , comprising:
8. creating, by the at least one processor, a second video fingerprint of the altered copy of the video based on the video comparison using MASS (Mueen's Similarity Search Algorithm); performing, by the at least one processor, a second video comparison of the second video fingerprint with a previously identified second video fingerprint of the comparison media file; wherein the previously identified second video fingerprint of the comparison media file is generated using MASS; The method of claim 1 , wherein the determining that the media file contains prohibited content is further based on the second video comparison.
9. said generating said video fingerprint further comprising: detecting, by the at least one processor, a face in the altered video; and cropping, by the at least one processor, the face from the modified video; and generating, by the at least one processor executing the perceptual hashing algorithm, a face print of the face; the video fingerprint further comprises the face print; The video comparison comparing said face print to known face prints; The method of claim 1 , further comprising storing the face print in a database storing the known face prints upon not finding a match within the known face prints.
10. said audio comparison, said video comparison, and said keyframe comparison; with a contiguous sub-portion of the previously identified audio fingerprint of the comparison media file, a contiguous sub-portion of the previously identified video fingerprint of the comparison media file, or a contiguous sub-portion of the previously identified keyframe fingerprint of the comparison media file. The method of claim 1 , comprising comparing successive subportions of the audio fingerprint, the video fingerprint, or the keyframe fingerprint, respectively.
11. At least one processor; and a non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the at least one processor to: receiving a media file including video and audio; pre-processing the media file; a modified copy of said video; a modified copy of said audio; and Keyframes in the video and creating a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of distinct segments of the altered copy of the video; creating an audio fingerprint of the altered copy of the audio, the audio fingerprint including a second vector of the individual segments of the altered copy of the video; generating a keyframe fingerprint based on the keyframes; performing a video comparison of said video fingerprint with a previously identified video fingerprint of a comparison media file; performing an audio comparison of the Audio Fingerprint with a previously identified Audio Fingerprint of the comparison media file; performing a keyframe comparison of the keyframe fingerprint with previously identified keyframe fingerprints of the comparison media file; determining that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison; A system that causes a system to perform an operation including:
12. said video comparison identifying individual video segments within said individual segments that satisfy a predetermined threshold level of video similarity between said video fingerprint and said previously identified video fingerprint of said comparison media file; said audio comparison identifying individual audio segments within said individual segments that satisfy a predetermined threshold level of audio similarity between said Audio Fingerprint and said previously identified Audio Fingerprint of said comparison media file; 12. The system of claim 11, wherein the keyframe comparison identifies individual keyframes that meet a predetermined threshold level of keyframe similarity between the keyframes and the previously identified keyframes of the comparison media file.
13. 12. The system of claim 11, wherein the pre-processing of the media file includes parallel processes for creating the modified copy of the video and the modified copy of the audio and for identifying the key frames within the video.
14. The system of claim 11 , wherein the performing of the video comparison, the audio comparison, and the keyframe comparison occurs in parallel.
15. said pre-processing of said media file to generate said modified copy of said video, changing the frame rate of the video to a predetermined frame rate; changing the resolution of the video to a predetermined resolution; and Removing bands from at least one side of the video. The system of claim 11 , further comprising:
16. said pre-processing of said media file to generate said modified copy of said audio, transforming the audio into a frequency domain by a fast Fourier transform performed by the at least one processor to generate a frequency domain audio; From the frequency domain sound, a sound exceeding a first predetermined frequency; a sound exceeding a first predetermined decibel; a sound below a second predetermined frequency; and A sound below a second predetermined decibel. and filtering out at least one of The system of claim 11 , comprising:
17. 20. The method of claim 19, wherein the pre-processing of the media file to generate the keyframes in the video comprises: Identifying a predetermined amount of entropy that should be present between key frames; Identifying a first frame in a segment of video as a key frame; Over the length of the video, adding said key frame to said plurality of key frames; comparing successive frames from the first frame until the predetermined amount of entropy is detected between the first frame and the subsequent frame; and Identifying the subsequent frame as the first frame. Repeatedly, generating said key frames; The system of claim 11 , comprising:
18. Additional instructions are stored on the non-transitory computer readable storage medium, the instructions, when executed by the at least one processor, causing the at least one processor to: creating a second video fingerprint of the modified copy of the video based on the video comparison using MASS (Mueen's Similarity Search Algorithm); performing a second video comparison of the second video fingerprint with a previously identified second video fingerprint of the comparison media file; wherein the previously identified second video fingerprint of the comparison media file is generated using MASS; The system of claim 11 , wherein the determining that the media file contains prohibited content is further based on the second video comparison.
19. said generating said video fingerprint further comprising: Detecting a face in the altered video; and cropping out the face from the altered video; and generating a face print for the face by executing the perceptual hashing algorithm; the video fingerprint further comprises the face print; The video comparison comparing said face print to known face prints; The system of claim 11 , further comprising storing the face print in a database storing the known face prints upon not finding a match within the known face prints.
20. A non-transitory computer readable storage medium storing instructions that, when executed by at least one processor, cause the at least one processor to: receiving a media file including video and audio; pre-processing the media file; a modified copy of said video; a modified copy of said audio; and Keyframes in the video and creating a video fingerprint of the altered copy of the video using a perceptual hashing algorithm, the video fingerprint including a first vector of distinct segments of the altered copy of the video; creating an audio fingerprint of the altered copy of the audio, the audio fingerprint including a second vector of the individual segments of the altered copy of the video; generating a keyframe fingerprint based on the keyframes; performing a video comparison of said video fingerprint with a previously identified video fingerprint of a comparison media file; performing an audio comparison of the Audio Fingerprint with a previously identified Audio Fingerprint of the comparison media file; performing a keyframe comparison of the keyframe fingerprint with previously identified keyframe fingerprints of the comparison media file; determining that the media file contains prohibited content based on at least one of the video comparison, the audio comparison, and the keyframe comparison; 16. A non-transitory computer readable storage medium for causing a computer to perform operations including:
21. Receiving, at a computer system, a media file including video and audio; pre-processing, by at least one processor, the media file to generate a pre-processed media file comprising pre-processed video and pre-processed audio; creating, by the at least one processor, a first video fingerprint of the preprocessed video using a perceptual hashing algorithm, an audio fingerprint of the preprocessed audio, and a second video fingerprint of the preprocessed video using keyframes within the preprocessed video; comparing the first video fingerprint, the second video fingerprint, and the audio fingerprint with known fingerprints to obtain a comparison result; determining, by the at least one processor, that the media file contains prohibited content based on the comparison; and A method comprising:
Citation Information
Patent Citations
Family histogram-based technology for detecting commercials and other video content
JP2005513663A
Extraction of features from video and audio signal content for reliable signal identification.
JP2010512057A
Media channel identification using video multi-match detection and audio fingerprint-based disambiguation
JP2019507515A
Method, system, and medium for detecting stereoscopic video by fingerprinting portions of video frames
JP2020513705A
Fingerprinting and matching of content of a multi-media file
US20170185675A1
