AI-based detection of anomalies in audiovisual data

Abstract

Using artificial intelligence (AI)-based techniques to detect glitches in audiovisual data can improve the reliability of glitch detection and reduce the cost of system validation and / or maintenance. An AI-based method for detecting glitches in an image can include extracting one or more features from the image, providing the image and the extracted features as input to a model, and generating, by the model, a classification output indicating whether the image is glitched. An AI-based method for detecting glitches in an audio data segment can include generating an image comprising a spectrogram of the audio data segment, providing the image as input to a model, and generating, by the model, a classification output indicating whether the audio data segment represented by the image is glitched. A record of glitches can be generated, and a validation status of a system under test (SUT) can be determined based on the record.

Description

Cross-references to related applications

[0001] This application claims priority to U.S. Patent Application No. 18 / 619,793, filed March 28, 2024, under Agent’s File No. 211011-703700 / US, entitled “AI-Based Detection of Anomalies in Audiovisual Data,” the contents of which are hereby incorporated herein by reference to the fullest extent permitted by applicable law. Background Technology

[0002] Many computer systems process audiovisual data (e.g., audio data and / or image data). Processing audiovisual (“AV”) data may involve: capturing audio signals or images as digital audio data or digital image data; generating synthetic audio data or synthetic image data; compressing / decompressing, encoding / decoding, scaling, enlarging, or otherwise modifying audio data or image data; analyzing audio data or image data (e.g., computer vision, face recognition, object detection and classification, pattern recognition, etc.); outputting audio data or image data (e.g., via speakers or display devices); etc. Computer systems may use many components in hardware, firmware, and software to perform AV data processing operations. A failure in any of these components may introduce anomalies (e.g., defects) into the AV data. Attached Figure Description

[0003] The accompanying drawings illustrate several exemplary implementations and are part of the specification. Together with the following description, these drawings demonstrate and explain the various principles of this disclosure.

[0004] Figure 1 This is a block diagram of an example of a verification system.

[0005] Figure 2 This is a block diagram of an example glitch detection system.

[0006] Figure 3 This is a block diagram of an example of a convolutional neural network (CNN).

[0007] Figure 4 This is a block diagram of another example of a burr detection system.

[0008] Figure 5 This is a block diagram of yet another example of a burr detection system.

[0009] Figure 6 This is a flowchart of an example method for verifying the system under test (SUT).

[0010] Figure 7This is a flowchart of another example method used to verify the SUT.

[0011] Figure 8 This is a block diagram of an example computing device.

[0012] Throughout the accompanying drawings, the same reference characters and descriptions indicate similar but not necessarily identical elements. While the examples described herein are susceptible to various modifications and alternatives, specific embodiments have been shown by way of example in the drawings and will be described in detail herein. However, the specific embodiments of the examples described herein are not intended to be limited to the specific forms disclosed. Rather, this disclosure covers all modifications, equivalents, and alternatives falling within the scope of the appended claims. Detailed Implementation

[0013] This disclosure relates in general to artificial intelligence (AI) based techniques for detecting anomalies in audiovisual data (e.g., audio data and / or image data). In some examples, these techniques are used to detect “glitch” in audiovisual data (e.g., anomalies in audiovisual (“AV”) data that produce defects in the visual properties of images represented by AV data and / or in the auditory properties of sounds represented by AV data).

[0014] Suppliers of computer systems capable of processing AV data typically attempt to validate such systems by monitoring for glitches in their processing of AV data. This monitoring can help diagnose the causes of glitches and allow for the replacement or redesign of faulty components. However, reliably detecting glitches in AV data can be difficult and laborious. Existing glitch detection processes may involve multiple people manually monitoring audio and video output from a computer system over long periods (e.g., hundreds or thousands of hours). However, reliably detecting glitches that only occur intermittently or over short periods remains challenging. Therefore, there is a need for a glitch detection system capable of monitoring and reliably detecting glitches in large volumes of AV data.

[0015] In some examples, a spur detection system includes a feature extraction component, a spur detection model, and a logging component. The spur detection system monitors for the presence of spurs in an image. This monitoring may include evaluating whether each image contains spurs. Image evaluation may include extracting one or more features from the image by the feature extraction component. Image evaluation may also include providing the image and the extracted features as input to the spur detection model. The spur detection model generates a classification output indicating whether an image contains spurs. The monitoring may also include the logging component generating a record of images classified as spurious by the model. This record may be provided to the user.

[0016] In some examples, the spur detection model includes a neural network, and providing an image and extracted features as input to the spur detection model includes providing the image and extracted features as input to the input layer of the neural network. In some examples, providing an image and extracted features as input to the spur detection model includes providing at least one of the image or extracted features as input to one or more hidden layers of the neural network. In some examples, the features include a first feature indicating the geometric properties of the image, a second feature indicating multiple pixel intensity gradients derived from the image, and / or a third feature characterizing the anomalousness of multiple pixel intensity values ​​derived from the image. Extracting the first feature may involve applying a Fourier transform to the image. Extracting the second feature may involve generating an orientation gradient histogram (HOG) of the pixel intensities of the image. Extracting the third feature may involve calculating a pixel-level anomalous score of the pixel intensities of the image.

[0017] In some examples, the image monitored by the glitch detection system corresponds to image data processed by a computer system. In other examples, the image monitored by the glitch detection system represents audio data processed by a computer system. For example, segments of audio data can be converted into images (e.g., spectrogram images), and the glitch detection system can detect glitch in the image to indicate glitch in the corresponding audio data.

[0018] In some examples, a glitch detection system includes a feature extraction component, a glitch detection model, and a logging component. The glitch detection system monitors audio data for the presence of glitch. This monitoring may include evaluating whether segments of audio data contain glitch. The feature extraction component may include an audio-to-image converter. Evaluation of segments of audio data may include extracting images (e.g., spectrogram images) representing the segments of audio data from the audio-to-image converter. Evaluation of segments of audio data may also include providing the images as input to the glitch detection model. In some examples, evaluation of segments of audio data may also include the feature extraction component extracting one or more features from the image and providing the features as additional input to the glitch detection model. The glitch detection model generates a classification output indicating whether segments of audio data corresponding to the images contain glitch. The monitoring may also include the logging component generating a record of audio data segments classified as glitch-containing by the model. This record may be provided to a user.

[0019] Using the AI-based glitch detection techniques disclosed herein, defects (e.g., vulnerabilities or design flaws) in hardware, firmware, and software-based computer components performing AV data processing operations can be detected. In some examples, these techniques are used to detect such defects during the post-silicon phase of computer system verification (e.g., by the hardware and / or software provider) or during the operation of the computer system (e.g., by the end user or technician). Using the disclosed techniques to monitor and detect AV data processing defects can improve the reliability of glitch detection and reduce the cost of system verification and / or maintenance. For example, providing an image and specific features derived from the image as input to a model facilitates the detection of specific types of glitch that might be difficult for a human observer to detect reliably.

[0020] In some examples, data generated by a glitch detection system can be used to control or improve computer system verification processes (e.g., post-silicon verification processes). For instance, some glitch detection systems not only detect glitch but also identify the "glitch type" of the detected glitch. In some examples, the source of the glitch can be located (e.g., to a specific component or set of components in the system under test), and / or the root cause of the glitch can be identified based on its glitch type. In this way, the glitch detection techniques disclosed herein can improve the speed and reduce the cost of computer system verification processes. The burr detection techniques disclosed herein do not involve automating burr detection techniques previously practiced manually by humans using only computers as tools. Instead, the disclosed systems and methods enable computers to reliably detect burrs using new techniques that differ from and improve upon those previously practiced manually by humans.

[0021] refer to Figures 1 to 5 and Figure 8 This disclosure provides a detailed description of an example system for burr detection. Combined with Figures 6 to 7 Provide a detailed description of the corresponding computer-implemented method.

[0022] In some aspects, the techniques described herein relate to a computer-implemented spur detection method comprising: for each of a plurality of images, evaluating whether the corresponding image is spurious, the evaluation comprising: extracting one or more features from the image; providing the image and the one or more features extracted from the image as multiple inputs to at least one model; and generating a classification output by the at least one model indicating whether the image is spurious; generating one or more records identifying one or more of the one or more images, each of the one or more images being classified as spurious by the at least one model; and determining the verification status of the system under test (SUT) based on the one or more records.

[0023] In some aspects, the techniques described herein relate to a spur detection method, wherein the at least one model includes a neural network, and wherein providing the image and the one or more features extracted from the image as the plurality of inputs to the at least one model includes providing the plurality of inputs to the neural network through an input layer.

[0024] In some respects, the techniques described herein relate to a spur detection method, wherein the neural network is a convolutional neural network (CNN), wherein the CNN includes one or more convolutional layers, the one or more convolutional layers including a first convolutional layer, and wherein the one or more features are inserted into the CNN at the input of the first convolutional layer.

[0025] In some aspects, the techniques described herein relate to a burr detection method, which further includes: obtaining first audiovisual data derived from second audiovisual data processed by a computer system, wherein the second audiovisual data includes image data, and wherein the first audiovisual data includes the plurality of images.

[0026] In some respects, the techniques described herein relate to a burr detection method, wherein the first audiovisual data is derived from the second audiovisual data via a deduplication process.

[0027] In some aspects, the techniques described herein relate to a glitch detection method, which further includes: obtaining first audiovisual data derived from second audiovisual data processed by a computer system, wherein the second audiovisual data includes audio data, and wherein the first audiovisual data includes a set of images representing a corresponding set of segments of the audio data.

[0028] In some respects, the techniques described herein relate to a glitch detection method, wherein obtaining the first audiovisual data includes: obtaining a set of audio data segments; and generating a set of images representing the corresponding set of audio data segments, wherein each image in the set of images corresponds to a corresponding audio data segment in the set of audio data segments and includes a spectrogram of the corresponding audio data segment.

[0029] In some respects, the techniques described herein relate to a glitch detection method, wherein: the plurality of images includes the set of images representing the corresponding set of audio data segments, the set of images including a first image representing a first audio data segment, and the first image is classified by the at least one model as having glitch, indicating that the first audio data segment has glitch.

[0030] In some aspects, the technology described herein relates to a glitch detection method, wherein the at least one model includes at least one first model, wherein the one or more records include one or more first records, and wherein the method further includes: for each audio data segment in the set of audio data segments, evaluating whether the corresponding audio data segment is glitch-prone, including: providing the image including the spectrogram of the audio data segment as input to at least one second model; and generating a classification output by the at least one second model indicating whether the audio data segment represented by the image is glitch-prone; generating one or more second records identifying one or more audio data segments in the set of audio data segments, each of the one or more audio data segments being classified as glitch-prone by the at least one second model; and providing the one or more second records to a user.

[0031] In some aspects, the techniques described herein relate to a spur detection method, wherein the one or more features include a first feature indicating one or more frequency domain properties of the image, a second feature indicating a plurality of pixel intensity gradients derived from the image, and / or a third feature characterizing the anomalousness of the plurality of pixel intensity values ​​derived from the image.

[0032] In some respects, the techniques described herein relate to a spur detection method, wherein extracting one or more features includes: extracting the first feature based on a Fourier transform of the image; extracting the second feature based on an orientation histogram of the plurality of pixel intensity gradients; and / or extracting the third feature based on a plurality of anomaly scores of the respective plurality of pixel intensity values.

[0033] In some respects, the technique described herein relates to a spur detection method, wherein: for each of the one or more images classified as having spurs by the at least one model, the classification output further indicates one or more probabilities that the image has spurs of one or more image spur types.

[0034] In some aspects, the techniques described herein relate to a burr detection method, wherein the one or more image burr types include stripe merging burrs, color-changing burrs, dotted line burrs, line pixelation burrs, Morse code burrs, parallel line burrs, radial dotted line burrs, random patch burrs, regular triangulation burrs, shader burrs, shape burrs, square patch burrs, stuttering burrs, texture protrusion burrs, and / or triangular burrs.

[0035] In some aspects, the technology described herein relates to a verification system comprising: a spur detection system communicatively coupled to a system under test (SUT), the spur detection system including at least one processor and at least one computer-readable storage medium having instructions encoded thereon, the instructions causing the at least one processor, when executed by the at least one processor, to perform operations including: obtaining first audiovisual data derived from second audiovisual data processed by the SUT, wherein the second audiovisual data includes image data, and wherein the first audiovisual data includes a plurality of images; evaluating, for each of the plurality of images, whether the corresponding image is spurious, the evaluation including: extracting one or more features from the image; providing the image and the one or more features extracted from the image as a plurality of inputs to at least one model; and generating a classification output by the at least one model indicating whether the image is spurious; generating one or more records identifying one or more of the plurality of images, each of the one or more images being classified as spurious by the at least one model; and determining the verification state of the SUT based on the one or more records.

[0036] In some aspects, the technology described herein relates to a computer-implemented glitch detection method comprising: evaluating whether a given audio data segment is glitch-prone for each of a plurality of audio data segments, the evaluation comprising: generating an image representing the audio data segment, the image including a spectrogram of the audio data segment; providing the image including the spectrogram of the audio data segment as input to at least one model; and generating a classification output by the at least one model indicating whether the audio data segment represented by the image is glitch-prone; generating one or more records identifying one or more audio data segments among the plurality of audio data segments, each of the one or more audio data segments being classified as glitch-prone by the at least one model; and determining the verification status of the system under test (SUT) based on the one or more records.

[0037] In some respects, the techniques described herein relate to a spur detection method, wherein at least one model comprises a convolutional neural network (CNN).

[0038] In some aspects, the techniques described herein relate to a spur detection method, which further includes: for each of the plurality of audio data segments: extracting one or more features from the audio data segment and / or from the image representing the audio data segment; and providing the extracted one or more features as one or more additional inputs to the at least one model.

[0039] In some respects, the technique described herein relates to a glitch detection method, wherein: for each of the one or more audio data segments classified as glitch-bearing by the at least one model, the classification output further indicates one or more probabilities that the audio data segment has one or more types of glitch.

[0040] In some respects, the techniques described herein relate to a glitch detection method, wherein the one or more audio glitch types include buzzing glitch type, intermittent glitch type, noise mixing glitch type, and clipping glitch type.

[0041] In some aspects, the technology described herein relates to a verification system comprising: a glitch detection system communicatively coupled to a system under test (SUT), the glitch detection system including at least one processor and at least one computer-readable storage medium having instructions encoded thereon, the instructions causing the at least one processor, when executed by the at least one processor, to perform operations including: obtaining audio data processed by the SUT; evaluating whether the corresponding audio data segment is glitch-prone for each of a plurality of segments of the audio data, the evaluation including: generating an image representing the audio data segment, the image including a spectrogram of the audio data segment; providing the image including the spectrogram of the audio data segment as input to at least one model; and generating a classification output by the at least one model indicating whether the audio data segment represented by the image is glitch-prone; generating one or more records identifying one or more audio data segments of the plurality of audio data segments, each of the one or more audio data segments being classified as glitch-prone by the at least one model; and determining the verification status of the SUT based on the one or more records.

[0042] Figure 1This is a block diagram of an example verification system 100. In some examples, verification system 100 verifies a system under test (SUT) 110. Generally, verifying a SUT may include any process that verifies that the SUT performs as expected (e.g., operates according to the system's performance specifications). In some examples, SUT 110 includes one or more (e.g., all) components of a system of computing devices or communication-coupled (e.g., network-connected) computing devices involved in the processing of audiovisual (AV) data (e.g., combined to perform a specific task or a set of application-related tasks). Components involved in the processing of AV data may include components that perform operations related to: capturing audio signals or images as digital audio data or digital image data; generating synthetic audio data or synthetic image data; compressing, decompressing, encoding, decoding, scaling, enlarging, or otherwise modifying audio data or image data; analyzing audio data or image data (e.g., in conjunction with computer vision, facial recognition, object detection and classification, pattern recognition, natural language processing, etc.); outputting audio data or image data (e.g., via a speaker or display device); etc. Verification of SUT 110 may involve monitoring for glitches in audiovisual (AV) data 114 processed by SUT 110 (e.g., by one or more hardware, firmware, and / or software-based components of SUT 110).

[0043] In some examples, the verification system 100 includes: a verification client 112 configured to run on one or more computing devices in the same computing environment as SUT 110; and / or a glitch detection system 120 configured to run on one or more computing devices different from SUT 110. The glitch detection system 120 may be communicatively coupled to SUT 110 via one or more wired and / or wireless networks, local area networks and / or wide area networks (including one or more public or private communication networks, corporate networks and / or the Internet). Any functionality of the verification system 100 may be implemented, in whole or in part, on the verification client 112 or the glitch detection system 120. In some examples, the verification client 112 sends some or all of the AV data 114 processed by SUT 110 to the glitch detection system 120, which checks the AV data 114 for glitch presence and generates a record 122 of the AV data 114 and / or the detected glitch.

[0044] In some examples, the glitch detection system 120 is implemented wholly or partially in a cloud computing environment. For example, the glitch detection system 120 may be implemented using resources from one or more data centers. The data centers may be communicatively coupled to the SUT 110 via one or more communication networks, as indicated above. In some examples, the data center includes nodes and a controller that allocates the node resources (e.g., storage resources, processing resources, etc.) to applications (e.g., the glitch detection application of the glitch detection system 120) or tasks (e.g., tasks performed by the glitch detection system 120). Each node in the data center can be any suitable type of computing device (e.g., server, personal computer, desktop computer, laptop computer, mobile device, etc.). Figure 8 The nodes can be computer-readable storage media (e.g., network-attached storage) or computing devices (e.g., 800, etc.). Nodes can be organized in any suitable manner (e.g., distributed, rack-mounted, network-connected, etc.). In some examples, one or more virtual machines can run on one or more nodes in a data center, and the data center controller can assign applications or tasks to the virtual machines.

[0045] In some examples, the verification system 100 also includes a verification client 132 configured to run on a client device 130 that is different from the SUT 110 and different from the spur detection system 120. The verification client 132 may provide, for example, a user interface through which a user can initiate or control the verification of the SUT 110 and / or view the records 122 generated by the spur detection system 120. In other examples, such a user interface may be provided by the verification client 112 and / or by the spur detection system 120.

[0046] Audiovisual data 114 may include audio data and / or image data. Audio data may include: data-encoded audio signals (e.g., audio signals representing music, speech, or other sounds; audio tracks or other audio portions of video; etc.), data derived from or otherwise related to the processing of audio signals, or information derived therefrom. Image data may include data-encoded images (e.g., photographs, video frames, computer-generated images, etc.), data derived from or related to the processing of images, or information derived therefrom. Some non-limiting examples of types of audiovisual data may include audio streams, audio files, image files, video streams, video files, or any portion thereof.

[0047] Glitches in AV data can include any anomalies in the AV data that cause defects in the visual properties of the image represented by the AV data and / or in the audible properties of the sound represented by the AV data. Such anomalies may be caused, for example, by a fault or design flaw in the component that processes the AV data. Some non-limiting examples of visual properties of an image may include: the size, color depth, resolution, brightness, orientation, etc. of the image; the size, shape, color, position, etc. of any object depicted in the image; the size, color, brightness, intensity, position, etc. of any pixel in the image; or any other visible property of the image (when displayed). Some non-limiting examples of audible properties of an audio segment may include: the pitch, timbre, loudness, etc. of each sound (including background noise) in the audio segment; the amplitude, frequency, spectrum, etc. of each sound wave (including the sound wave that produces background noise) in the audio segment; the intelligibility of spoken words in the audio segment; or any other audible or physical property of any sound or set of sounds in the audio segment (when played).

[0048] In some examples, the verification system 100 can detect any suitable type of glitches in the image data, including: shader artifacts (“shader glitch”), shape artifacts (“shape glitch”), color-shifting artifacts (“color-shifting glitch”), Morse code patterns (“Morse code glitch”), patterning artifacts (“patterning glitch”); dotted line artifacts (“dotted line glitch”), radial dotted line artifacts (“radial dotted line glitch”), parallel line artifacts (“parallel line glitch”), triangulation artifacts (“triangulation glitch”, e.g., regular triangulation glitch”), line pixelation artifacts (“line pixelation glitch”), screen stuttering artifacts (“screen stuttering glitch”), screen tearing artifacts (“screen tearing glitch”), square patch artifacts (“square patch glitch”), blur artifacts (“blur glitch”), random patch artifacts (“random patch glitch”), stripe merging artifacts (“stripe merging glitch”), texture abruptness artifacts (“texture abruptness glitch”), etc.

[0049] Shader artifacts can include visible artifacts associated with shading miscoloration. A “shader program” is a program that executes on a graphics processor (e.g., a graphics processing unit (“GPU”) to perform graphics functions such as transforming vertex coordinates (“vertex shader program”), shading pixels (“pixel shader program”), etc. An image includes shader artifacts when one or more polygons in the image are shaded inappropriately. Visually, instances of such shading miscoloration appear in the image as polygonal shapes of different colors that blend together or show a gradual fading in certain directions.

[0050] An image includes shape artifacts when one or more shapes (e.g., polygons, monochrome shapes) are inappropriately included in the image (e.g., at random or pseudo-random locations). An image includes color artifacts when the color and / or intensity of a cluster of pixels are set to inappropriate values, or when the image's original color palette is altered to make the colors in the image appear incorrect, inconsistent, or unnaturally exaggerated.

[0051] An image contains Morse code glitches when dots and / or dashes resembling Morse code are inappropriately included in it. More code glitches are typically caused by malfunctions in the image rendering process or in the display hardware. For example, Morse code glitches can appear in an image when a set of memory cells in the graphics processor becomes stuck (e.g., persistently storing the same value despite attempts to overwrite it with a new value), causing pixels corresponding to the stuck values ​​to be displayed instead of pixels corresponding to the actual image. In various examples, GPU operation at speeds or temperatures exceeding the GPU's design constraints can also result in the display of Morse code patterns.

[0052] Patterned glitch occurs when a repeating pattern of dots, dashes, and / or lines is inappropriately included in an image (e.g., superimposed on portions of the original image). Repeating dots, dashes, and / or lines may be evenly spaced and / or appear in rows or columns. Morse code glitch is an example of patterned glitch. Like Morse code glitch, patterned glitch can be caused by a set of memory cells in the graphics processor becoming stuck.

[0053] An image includes dotted line artifacts when one or more dotted lines are inappropriately included in the image. In some examples, the positions and slopes of the dotted lines may appear irrelevant or uncorrelated (e.g., random or pseudo-random). In the case of radial dotted line artifacts, the dotted lines may be radial lines emanating from a single point. An image includes parallel line artifacts when two or more parallel lines are inappropriately included in the image. In some examples, the parallel lines have a uniform color. An image includes triangulation artifacts when a grid of triangles appears inappropriately throughout the image (or a portion of the image).

[0054] An image includes line pixelation artifacts when random colors are inappropriately assigned to pixels (or clusters of pixels) in a band (or "stripes") of an image. An image includes screen stuttering artifacts when adjacent columns and rows (referring to individual lines or bands in a vertical or horizontal direction) are inappropriately swapped with each other. An image includes screen tearing artifacts when two consecutive frames of a video are rendered in the same image, such that one part of the image shows a scene at one time while another part shows a scene at a different time.

[0055] An image includes square patch artifacts when uniformly or nearly uniformly colored square patches are inappropriately present in the image. An image includes blur artifacts when at least a portion of the image is inappropriately blurred. An image includes random patch artifacts when uniformly or nearly uniformly colored random-shaped patches are inappropriately present in the image. A patch can be "randomly shaped" in the sense that the individual patches are irregularly shaped and / or that different patches have different shapes.

[0056] An image includes stripe merging glitch when a portion of a first image is inappropriately included in a portion of a second image (e.g., displayed in place of a portion of the second image). The inappropriately included portion can be stripes or bands of the first image (e.g., horizontal stripes or bands). In some examples, the first and second images are identical or highly similar (e.g., two different frames of a video). In some examples, the size (e.g., height) and positioning of the stripes or bands change intermittently (e.g., sporadically and / or randomly). Stripe merging glitch can occur when the image buffer inconsistently fails and recovers, resulting in partially updated display. When two consecutive frames of a video are rendered such that an object has a low-resolution texture in one frame and a high-resolution texture in the next, the image includes texture spurious glitch.

[0057] In some examples, the verification system 100 can detect any suitable type of glitches in an audio segment, including buzzing glitches, intermittent glitches, noise mixing glitches, clipping glitches, etc. A buzzing glitch occurs when an audio segment inappropriately produces a persistent (e.g., continuous or intermittent) sound characterized by buzzing (e.g., humming) noise ranging from low to medium frequencies. In some examples, the volume and / or pitch of the buzzing noise may change as the buzzing noise stops and starts again, but typically remains constant during periods when the buzzing noise is continuous.

[0058] Intermittent spikes occur when temporary anomalies in an audio signal occur at irregular intervals. In some examples, intermittent spikes are characterized by their unpredictable and transient nature. In other examples, intermittent spikes manifest as brief distortion, transient sound loss, burst noise, or any other sudden anomaly in the audio signal.

[0059] Noise mixing glitches occur when external noise is unintentionally mixed with the main audio signal (e.g., blended). In some examples, noise mixing glitches are characterized by the presence of interfering noise superimposed on the main audio signal (e.g., electrostatic discharge, hissing, crackling, popping, and / or other forms of audio distortion).

[0060] Clipping glitches occur when the amplitude of an audio signal exceeds the amplitude range of the audio processing system (e.g., the maximum amplitude). In some examples, clipping glitches are characterized by peaks (e.g., local maxima and / or local minima) in the audio waveform representing the clipped audio segment. Clipping glitches typically result in harsh, distorted sound.

[0061] As used herein, the term "glitched" can refer to an image or sound that includes glitches, or it can refer to AV data that includes defects that produce glitches when AV data (e.g., via a display device or speaker) is output. The term "glitched-free" can refer to an image, sound, or AV data item that is not glitched. The phrases "classified as glitched" or "considered glitched" can refer to an image, sound, or AV data item that has been marked as glitched (e.g., by a glitching detection model).

[0062] Record 122 generated by glitch detection system 120 may include any suitable record of AV data 114 processed by glitch detection system 120 and / or glitch detected by glitch detection system 120. In some examples, glitch detection system 120 generates record 122 for each AV data item processed by glitch detection system (e.g., received, subjected to a glitch detection process, etc.). Such a record may indicate whether glitch detection system 120 classifies the corresponding AV data item as glitch-prone, and if so, may indicate what type of glitch was detected in the AV data item. In some examples, glitch detection system 120 generates such a record for each AV data item classified as glitch-prone. In some examples, record 122 includes an aggregate record that indicates, for example, the duration of the time period during which the burr detection system 120 monitors for defects in the AV data 114 processed by the SUT 110, the amount of AV data 114 processed by the burr detection system (e.g., number of images, number of audio clips, etc.) (e.g., within a time period or for a specific SUT 110), etc.

[0063] In some examples, the process of validating SUT 110 is controlled or improved based on records 122 generated by the burr detection system 120. For example, based on the records 122 of detected burrs (and optionally, their burr types), the source of the burr can be located (e.g., to a specific component or set of components of SUT 110) and / or the root cause of the burr can be identified. In some examples, the location of the source of the burr and / or the identification of the root cause of the burr are performed automatically by the burr detection system 120, which can then (e.g., via a verification client) notify the user that the burr may be caused by a defect in a specific component or set of components. In some examples, the location of the source of the burr and / or the identification of the root cause of the burr are performed by the user based on the inspection of the records 122 of detected burrs.

[0064] Figure 2 This is a block diagram of an example burr detection system 200 (e.g., burr detection system 120), which is configured to monitor AV data 201 (e.g., AV data 114). Figure 2 In one example, the glitch detection system 200 includes a data preparation module 202, a feature extraction module 204, a glitch detection model 206, and a recorder 208. The data preparation module 202 processes AV data 201 to produce one or more AV data items 212. AV data items 212 may be provided as input to the feature extraction module 204 and / or to the glitch detection model 206. In some examples, AV data 201 includes AV data items 212, and the data preparation module 202 is omitted or bypassed. The feature extraction module 204 extracts one or more features 214 from the AV data items 212. The extracted features 214 may be provided as input to the glitch detection model 206. The glitch detection model 206 may classify the AV data items 212 as glitch-free or glitch-free based on the AV data items 212 and / or the extracted features 214. In some examples, the glitch detection model may generate a classification output 216 indicating whether the AV data item 212 contains glitch, and if so, what type of glitch the model detected in that AV data item. A recorder 208 may generate a record 122 of the AV data 201 processed by the glitch detection system 200. The recorder may store such records in, for example, a database or any other suitable data storage system. This document further describes in detail some examples of the components of the glitch detection system 200.

[0065] In some examples, data preparation module 202 provides a portion of AV data 201 as AV data item 212. The remaining portion of AV data 201 may be discarded or provided to recorder 208 (e.g., to enable the recorder to maintain a recording of the discarded portion of the AV data). In some scenarios, portions of the AV data processed by the SUT may be largely static. For example, AV data 201 may comprise a sequence of many identical or nearly identical images. Repeatedly checking identical or nearly identical images for glitches can be an inefficient use of computational resources for a glitch detection system. To avoid such inefficiency, data preparation module 202 may perform a deduplication operation on AV data 201. Any suitable deduplication operation can be used. In some examples, data preparation module 202 divides the images in AV data 201 into buckets based on the similarity of the images. Any suitable metric for image similarity can be used. In some examples, data preparation module 202 (1) generates a histogram (e.g., pixel color histogram, pixel intensity histogram, etc.) for each image in AV data 201, (2) calculates a hash value (e.g., perceptual hash or “pHash”) representing an attribute (e.g., a “fingerprint”) of the image, (3) searches for image pairs such that the similarity (e.g., overlap) between the histograms of the images exceeds a first threshold (e.g., 90%) and the distance between the hash values ​​of the images (e.g., Hamming distance) is less than a second threshold (e.g., 100), and (4) assigns any such image pairs to a common bucket. In some examples, the data preparation module selects (e.g., samples) one image from each bucket and provides the selected image (along with any images not assigned to any shared bucket) as AV data item 212. In some examples, to limit the number of pairwise comparisons performed as part of a deduplication operation, data preparation module 202 may divide AV data 201 into processing windows and apply the deduplication operation only to images within the same processing window. In some examples, the data preparation module 202 provides any image from the AV data item 212 as input to the feature extraction module 204 and the spur detection model 206.

[0066] In some examples, the data preparation module 202 divides the AV data 201 into a set of AV data items 212. For example, the AV data 201 may include an audio file or an audio stream. To facilitate glitch detection, the data preparation module 202 may divide the audio file (or stream) into a set of audio segments. In some examples, the audio segments have approximately equal durations (e.g., 5 seconds). AV data items 212 may include audio segments. In some examples, the data preparation module 202 provides any audio file, stream, or segment from the AV data items 212 as input to the feature extraction module 204.

[0067] Still referencing Figure 2The burr detection system 200 may include one or more feature extraction modules 204. Each feature extraction module 204 receives AV data item 212 as input, extracts one or more features 214 from the AV data item 212, and provides the extracted features 214 as input to the burr detection model 206. Any suitable feature 214 can be extracted from the AV data item 212, and any suitable data analysis (e.g., feature extraction) technique can be used to extract the feature 214.

[0068] In some examples, for AV data item 212 of encoded image I1, feature 214 and / or feature extraction techniques may include one or more of the following: edge detection algorithms, corner detection algorithms, blob detection algorithms, ridge detection algorithms, Hough transform (e.g., generalized Hough transform), structure tensor (e.g., generalized structure tensor), affine invariant feature detection algorithms (e.g., affine shape adaptation, Harris affine region detection, Hessian affine region detection, etc.), feature description algorithms (e.g., scale-invariant feature transform (SIFT), accelerated robust feature (SURF), gradient location and orientation histogram (GLOH), orientation gradient histogram (HOG)), Richardson-Lucy deconvolution, segmentation, etc.

[0069] In some examples, for the AV data item 212 of the encoded image I1, the extracted feature 214 can be an image I2 (e.g., a frequency domain image or Fourier domain image corresponding to the spatial domain image I1), which can be extracted by applying a Fourier transform (FT) (e.g., Discrete Fourier Transform (DFT), Two-Dimensional (2D) DFT, Discrete-Time Fourier Transform (DTFT), Fast Fourier Transform (FFT), etc.) to the image I1. Such features may be particularly useful for detecting glitch that exhibits a periodic spatial pattern (e.g., Morse code glitch, parallel line glitch) and / or glitch that exhibits well-defined edges (e.g., shader glitch, shape glitch), as well as other types of glitch. In some examples, such features are useful for identifying the geometric characteristics of the image I1 (e.g., the presence of line segments or geometric shapes in the image I1).

[0070] In some examples, for AV data item 212 encoding image I1, the extracted features 214 can indicate the location of edges within image I1. For example, the extracted features 214 can be an orientation histogram of the intensity gradients of pixels in image I1 (e.g., an orientation gradient histogram (HOG)). In some examples, HOG analysis is applied to aggregate pixel intensities (e.g., grayscale pixel intensities derived from pixels in image I1) or to individual pixel intensities for one or more color channels (e.g., red / green / blue channels) of pixels in image I1. The HOG for image I1 can be calculated by dividing the image into small regions, determining the magnitude and orientation of the pixel intensity gradients within each patch, and summarizing the results. Other edge detection techniques can be used. Features indicating the location of edges can be particularly useful for detecting parallel line spurs and / or triangulation spurs, as well as other types of spurs.

[0071] In some examples, for AV data item 212 of encoded image I1, the extracted features 214 can indicate a pixel-level anomaly metric (PAM) for the pixels of the image. In some examples, PAM can be calculated by approximating the distribution of pixel intensities (e.g., grayscale intensity or the intensity of individual color channels) and assigning an anomaly score to each pixel based on the degree to which the pixel intensity deviates from the estimated global distribution. Other techniques for scoring the anomalies of pixels in an image can be used. Features indicating a pixel-level anomaly metric can be particularly useful for detecting stripe merging spikes and / or stuttering spikes, as well as other types of spikes.

[0072] In some examples, for AV data item 212 encoding an audio segment, feature 214 may include one or more of the following: a time-frequency representation of the waveform of the audio segment (e.g., an audio signal waveform) (e.g., waveform graph, spectrogram, Mel spectrogram, constant Q transform, etc.), frequency domain properties of the waveform of the audio segment (e.g., bandgap energy ratio, spectral centroid, spectral flux, etc.), and time domain properties of the waveform of the audio segment (e.g., zero-crossing rate, amplitude envelope, root mean square (RMS) energy, etc.). In some examples, the frequency domain properties can be obtained by applying a Fourier transform to the waveform. In some examples, the time-frequency representation of the waveform can be obtained by applying a short-time Fourier transform (STFT) to the waveform.

[0073] In some examples, for AV data item 212 encoding an audio segment, the extracted features 214 may include a spectrogram representing the waveform of the audio segment (e.g., a time-domain audio signal waveform). The spectrogram may be an image depicting the frequency spectrum of the audio segment over time (e.g., over the duration of the audio segment). The spectrogram of the audio segment can be obtained by applying an STFT to the waveform representing the audio segment. In some examples, the spectrogram is a logarithmic spectrogram (e.g., where frequencies are converted to a logarithmic scale), such as a Mel spectrogram (e.g., where frequencies are converted to a Mel scale).

[0074] In some examples, for AV data item 212 of the encoded image or extracted features 214 of the encoded image (e.g., spectrogram), feature extraction module 204 may extract a resized version of the input image as feature 214. The resized version of the input image may be referred to as the "resized image." In some examples, the resized image has a different (e.g., lower) pixel resolution than the input image. In some examples, the resized image has a pixel resolution of 480 x 480 pixels or 224 x 224 pixels. In some examples, the resized image and the input image have different aspect ratios. For example, the input image may have an aspect ratio of 1920:1080, and the resized image may have an aspect ratio of 1:1. In some examples, the resized image is generated by a machine learning model (e.g., a resizing model). Providing a resized image (instead of the input image) as input to the spur detection model 206 can significantly reduce the memory utilization of the spur detection model, reduce the time and / or computational resources used to train the spur detection model, and / or reduce the time and / or computational resources used by the trained spur detection model to generate inferences.

[0075] Still referencing Figure 2 The glitch detection system 200 may include one or more glitch detection models 206. Each glitch detection model 206 may classify AV data item 212 as glitch-free or glitch-free based on AV data item 212 and / or extracted features 214. In some examples, the glitch detection model 206 generates a classification output 216 indicating whether the AV data item 212 has glitch, and if so, what type of glitch the model 206 detected in the AV data item 212. In some examples, the same glitch detection model 206 is used to detect glitch in image and audio clips. In some examples, different glitch detection models 206 are used to detect glitch in image and audio clips.

[0076] The glitch detection model 206 may include any suitable type of machine learning model and may have any suitable model architecture. In some examples, the glitch detection model 206 includes a convolutional neural network (CNN) or has a CNN-based architecture (e.g., VGG16, EfficientNet, ResNet, Inception, InceptionResNet, etc.). In some examples, the glitch detection model includes a residual neural network (RNN) or has an RNN-based architecture (e.g., ResNet, InceptionResNet, etc.). See below for reference. Figure 3 This describes some aspects of CNN 306. See below for reference. Figure 4 and Figure 5 Some examples describing burr detection models (406, 506).

[0077] Any suitable technique (including supervised, unsupervised, self-supervised, and semi-supervised techniques) can be used to train a glitch detection model. In some examples, training the model involves: obtaining an AV dataset, fitting the model to the training portion of the AV dataset (“training data”), validating the model on the validation portion of the AV dataset (“validation data”), and testing the model on the test portion of the AV dataset (“test data”). The AV dataset may include input samples of AV data items with and without glitch, and corresponding output samples (e.g., ground truth output samples) indicating whether an AV data item has glitch, and if so, what type of glitch is present.

[0078] Such input samples can be obtained using any suitable technique. In some examples, input samples include AV data captured from a computing device (e.g., images captured from the frame buffer of a computing device, audio data captured from the loopback of the audio processing stack of a computing device, etc.). Output samples corresponding to such input samples can be generated manually (e.g., by a human reviewer of the captured images). In some examples, input samples include glitched images or audio clips previously submitted to a vulnerability tracking system. Output samples corresponding to such input samples can also be obtained from a vulnerability tracking system. In some examples, input samples include synthetic glitched AV data (e.g., a previously glitch-free image to which an image glitch injection tool (e.g., Glitchify) has injected one or more glitches, a previously glitch-free audio clip to which an audio glitch injection tool has injected one or more glitches, etc.). Output samples corresponding to synthetic glitched AV data can be generated automatically by the glitch injection tool or by the software controlling the glitch injection tool.

[0079] Fitting a spur detection model to training data may involve adjusting the values ​​of the hyperparameters of the training algorithm and / or the parameters of the model so that the model learns the relationship between the input samples and output samples of the training portion of the dataset. Validating the model on validation data may involve using the model to generate output samples corresponding to the input samples of the validation data (e.g., the output samples could be classification outputs generated by the model) and evaluating the model's performance based on a comparison of the model-generated output samples with the corresponding ground truth classifications. In some examples, the training and validation steps are performed iteratively until the model exhibits an acceptable level of performance. Testing the model on test data may involve using the model to generate output samples corresponding to the input samples of the test dataset, where the input samples of the test dataset have not been used during the training and validation steps.

[0080] Figure 3 A block diagram of a Convolutional Neural Network (CNN) 306 is shown. CNNs and their variants are commonly used for image recognition, classification, and / or segmentation. Figure 3 In the example, the input to CNN 306 is image 301. Image 301 can have any suitable height (e.g., number of pixels in the vertical direction), width (e.g., number of pixels in the horizontal direction), and depth (e.g., number of channels, such as color channels). CNN 306 includes two feature extraction stages (320, 350), each of which includes a convolutional layer and a pooling layer. The first feature extraction stage 320 generates one or more feature maps 340 that identify low-level features 344 of image 301. Specifically, the convolutional layer of the first feature extraction stage 320 generates one or more feature maps 330 that identify low-level features 332 of image 301 by convolving a portion 312 of image 301 with a convolutional kernel, and the pooling layer of the first feature extraction stage performs pooling (e.g., downsampling) operations on a portion 334 of feature map 330 to extract low-level features 344 of feature map 340. Similarly, the second feature extraction stage 350 generates one or more feature maps 370 of high-level features 374 of the identifier image 301. Specifically, the convolutional layer of the second feature extraction stage 350 generates one or more feature maps 360 of high-level features 362 of the identifier image 301 by convolving a portion 342 of the low-level feature map 340 with a convolutional kernel, and the pooling layer of the second feature extraction stage performs pooling (e.g., downsampling) operations on a portion 364 of the feature map 360 to extract the high-level features 374 of the feature map 370. Figure 3 In the example, the CNN also includes two hidden layers 380 and 385 (e.g., fully connected layers set between the input and output layers) and an output layer 390.

[0081] Figure 4A block diagram of an example glitch detection system 400 is shown. In some examples, the glitch detection system 400 is configured to detect and classify image glitch in an input image 412 and / or detect and classify audio glitch in an audio segment represented by the input image 412. The glitch detection system 400 may include one or more feature extraction modules 404 (e.g., feature extraction module 204) and a glitch detection model 406.

[0082] In some examples, image 412 is provided as input to feature extraction module 404. Feature extraction module 404 may extract one or more features 414 (e.g., features 414a, 414b, 414c, 214, etc.) from image 412. In other examples (e.g., when image 412 represents an audio segment), the feature extraction module may be omitted or bypassed.

[0083] The spur detection model 406 includes a CNN-based model 420. Image 412 is provided as input to the CNN-based model 420. In some examples, the spur detection model 406 includes one or more feature processing pipelines 440 (e.g., feature processing pipelines 440a, 440b, 440c, etc.). Extracted features 414 may be provided as input to the feature processing pipeline 440. Therefore, the input to the spur detection model 406 may include image 412 and features 414. In some examples, the extracted features 414 include features indicating the frequency domain properties of image 412 (e.g., a frequency domain representation of image 412, such as a frequency domain image corresponding to image 412, where image 412 is a spatial domain image), features indicating the location of edges within image 412 (e.g., an orientation histogram of the intensity gradients of pixels in image 412), and features indicating a pixel-level anomaly metric (PAM) for pixels in the image. In other examples, one or more feature processing pipelines in feature processing pipeline 440 may be omitted or bypassed.

[0084] In some examples, the CNN-based model 420 is an Inception-ResNet model with stages 421 through 430 (e.g., an Inception-ResNet-v2 model). Stage 421 can be an input layer. Stage 422 can be an Inception-ResNet Stem (e.g., an initial set of layers preceding the first Inception block). Stage 423 can be the first Inception block (e.g., a 5 x Inception-ResNet-A block). Stage 424 can be the first reduction block (e.g., a Reduction-A block). Stage 425 can be the second Inception block (e.g., a 10 x Inception-ResNet-B block). Stage 426 can be the second reduction block (e.g., a Reduction-B block). Stage 427 can be the third Inception block (e.g., a 5 x Inception-ResNet-C block). Stage 428 can be a pooling layer (e.g., an average pooling layer). Stage 429 can be a dropout layer. Stage 430 can be an activation layer (e.g., a softmax layer). The output of the activation layer can be the model's classification output (e.g., classification output 216).

[0085] In some examples, each feature processing pipeline in feature processing pipeline 440 may have the same architecture as the initial sequence of two or more stages of the CNN-based model 420. For example, feature processing pipeline 440 may include stage 441 (e.g., stage 441a, 441b, or 441c) and stage 442 (e.g., stage 442a, 442b, or 442c), where stage 441 and stage 442 may be the input layer and the Inception-ResNet Stem, respectively. In some examples, feature processing pipeline 440 may also include stage 443 (e.g., stage 443a, 443b, or 443c), where stage 443 is the first Inception block. In some examples, the feature processing pipeline 440 also includes stage 444 (e.g., stage 444a, 444b, or 444c) and stage 445 (e.g., stage 445a, 445b, or 445c), where stage 444 and stage 445 can be a first reduction block and a second inception block. In some examples, the feature processing pipeline 440 also includes stage 446 (e.g., stage 446a, 446b, or 446c) and stage 447 (e.g., stage 447a, 447b, or 447c), where stage 446 and stage 447 can be a second reduction block and a third inception block.

[0086] In some examples, the output of a stage (e.g., 442, 443, 445, or 447) of the feature processing pipeline 440 can be fed to a CNN-based model 420, which can combine (e.g., aggregate) the output of the feature processing pipeline stage with the output of the corresponding stage of the CNN-based model 420, and provide the aggregated output as input to the next stage of the CNN-based model 420. In this way, the feature processing pipeline stages can perform early-stage processing of features 414 in parallel with the early-stage processing of image 412 by the CNN-based model, and the data generated by those parallel early-stage processing tasks can be aggregated and processed together by later stages of the CNN-based model. In some examples, all feature processing pipelines 440 have the same architecture (e.g., the same number of stages). In other examples, different feature processing pipelines 440 may have different numbers of stages.

[0087] In some examples (e.g., when image 412 represents an audio segment), feature extraction module 404 and feature processing pipeline 440 can be omitted or bypassed. In such scenarios, placeholder data (e.g., "padding") can be provided to CNN-based model 420 instead of the output from feature processing pipeline 440.

[0088] In some examples, the spur detection system 400 includes a recorder (e.g., recorder 208). In some examples, the spur detection system 400 is communicatively coupled to the system under test (SUT) (e.g., SUT 110), image 412 is extracted from AV data processed by the SUT, and the verification status of the SUT is determined by the spur detection system based on the recording generated by the recorder.

[0089] Figure 5 A block diagram of an example glitch detection system 500 is shown. In some examples, the glitch detection system 500 is a specific configuration of the glitch detection system 200. In some examples, the glitch detection system 500 is configured to detect and classify glitch in audio data 501 based on an image 514 representing a segment 512 of audio data 501. The glitch detection system 500 may include a data preparation module 502 (e.g., data preparation module 202), one or more feature extraction modules 504 (e.g., feature extraction module 204), a glitch detection model 506, and a recorder 508 (e.g., recorder 208).

[0090] In some examples, the data preparation module 502 divides the audio data 501 into a set of audio data segments 512. In some examples, the feature extraction module 504 extracts at least one feature corresponding to each audio data segment in the audio data segments. In some examples, the extracted features include an image 514 representing the waveform of the audio data segment 512 (e.g., a spectrogram image of the waveform).

[0091] In some examples, the glitch detection model 506 includes a CNN-based model. In some examples, the glitch detection model 506 generates a classification output 516 that indicates whether the audio data segment 512 contains glitch, and if so, what type of glitch the model 506 has detected in the audio data segment 512.

[0092] In some examples, the glitch detection system 500 is communicatively coupled to the system under test (SUT) (e.g., SUT 110), audio data 501 is extracted from AV data processed by the SUT, and the verification status of the SUT is determined by the glitch detection system based on the recording generated by the recorder 508.

[0093] Figure 6 A flowchart of verification method 600 is shown. In some examples, verification method 600 may be performed to verify the AV data processing of the system under test (SUT). In some examples, verification method 600 may be performed by any suitable system (e.g., a glitch detection system) or utilized by any suitable system. Verification method 600 may include steps 610 to 670. Some examples of steps 610 to 670 are further described in detail below.

[0094] In step 610, the system obtains an image. In some examples, the image is included in the AV data processed by the SUT. In some examples, the image includes a spectrogram of the waveform of an audio data segment processed by the SUT.

[0095] In step 620, the system extracts one or more features from the image. This document describes feature extraction techniques and some examples of the extracted features. In some examples (e.g., where the image represents a waveform of an audio data segment), step 620 may be omitted.

[0096] In step 630, the system provides the image and the extracted features (if any) as input to the spur detection model (e.g., model 420). Some examples of spur detection models are described in this paper.

[0097] In step 640, the spur detection model generates a classification output indicating whether the image contains spurs. In some examples, the classification output indicating whether the image contains spurs also indicates what type of spurs the model has detected in the image (or in the corresponding audio clip).

[0098] In step 650, the system determines if more images are available for glitch detection. If so, the system applies steps 610 to 640 to those images. If no more images are available for glitch detection, or in parallel with applying steps 610 to 640 to additional images, the system proceeds to step 660. In step 660, the system generates a record of the processed image and the corresponding classification output.

[0099] In step 670, the system determines the verification status of the SUT based on the generated records. In some examples, the SUT includes one or more (e.g., all) AV data processing components of a computer system. In some examples, the verification status of the SUT may indicate whether the SUT passes or fails the verification test. In some examples, the SUT fails the verification test if the number of detected glitches exceeds a threshold number of glitches (e.g., N glitches, where N is any suitable non-negative integer), or if the rate of detected glitches exceeds a threshold rate of glitches (e.g., R glitches per minute, where R is any suitable non-negative number). In some examples, the SUT passes the verification test if it does not fail the verification test. In some examples, the verification status of the SUT may indicate the need for additional careful review of the SUT. In some examples, if the number of detected spikes exceeds a spike threshold (e.g., N spikes, where N is any suitable non-negative integer), or if the rate of detected spikes exceeds a spike threshold rate (e.g., R spikes per minute, where R is any suitable non-negative number), it is necessary to perform additional careful review of the SUT. In some examples, the system notifies the user of the SUT's verification status.

[0100] In some examples, determining the verification status of the SUT includes: locating the source of one or more glitches based on the generated records (e.g., identifying a specific component or set of components of the SUT as a possible source of a glitch) and / or identifying the root cause of one or more glitches (e.g., identifying a specific module or facility of a component of the SUT as a possible cause of a glitch). In some examples, the location of the glitch source and / or the identification of the root cause of the glitch are performed automatically by the glitch detection system based on the type of glitch detected.

[0101] Figure 7A flowchart of verification method 700 is shown. In some examples, verification method 700 may be performed to verify the AV data processing of the system under test (SUT). In some examples, verification method 700 may be performed by any suitable system (e.g., a glitch detection system) or utilized by any suitable system. Verification method 700 may include steps 710 to 770. Some examples of steps 710 to 770 are further described in detail below.

[0102] In step 710, the system obtains an audio data segment. In some examples, the audio data segment is derived from the audio data processed by the SUT.

[0103] In step 720, the system generates an image representing the audio data segment. In some examples, the image includes a spectrogram of the waveform of the audio data segment.

[0104] In step 730, the system provides an image as input to a spur detection model (e.g., model 506). Some examples of spur detection models are described herein.

[0105] In step 740, the glitch detection model generates a classification output indicating whether the audio data segment represented by the image contains glitch. In some examples, the classification output indicating that the audio data segment contains glitch also indicates what type of glitch the model has detected in the audio data segment.

[0106] In step 750, the system determines whether there are more audio data segments available for glitch detection. If so, the system applies steps 710 to 740 to those audio data segments. If no more audio data segments are available for glitch detection, or in parallel with applying steps 710 to 740 to additional audio data segments, the system proceeds to step 760. In step 760, the system generates records of the processed audio data segments and corresponding classification outputs.

[0107] In step 770, the system determines the verification status of the SUT based on the generated records. Some examples of techniques for determining the verification status of the SUT based on records generated by the glitch detection system have been described above.

[0108] The techniques operating according to the principles described herein can be implemented in any suitable manner. While the foregoing disclosures illustrate various specific implementations using particular block diagrams, flowcharts, and examples, each block diagram component, flowchart step, operation, and / or component described and / or illustrated herein can be implemented individually and / or collectively using a wide range of hardware, software, or firmware (or any combination thereof). Furthermore, any disclosure of components contained within other components should be considered, in nature, non-limiting examples, as many other architectures can be implemented to achieve the same functionality.

[0109] The above discussion includes flowcharts illustrating the steps and actions of the instruction scheduling method. The processing and decision boxes in the flowcharts represent steps and actions that can be included in algorithms that perform these processes. Algorithms derived from these processes can be implemented as software integrated with and directing the operation of one or more single-purpose or multi-purpose processors (e.g., central processing unit (CPU), graphics processing unit (GPU), tensor processing unit (TPU), hardware accelerator, etc.), can be implemented as functionally equivalent circuits such as digital signal processing (DSP) circuits, field-programmable gate arrays (FPGAs), or application-specific integrated circuits (ASICs), or can be implemented in any other suitable manner. It should be understood that the flowcharts included herein do not depict the syntax or operations of any particular circuit or any particular programming language or type of programming language. Rather, the flowcharts exemplify functional information that a person skilled in the art can use to construct circuits or implement computer software algorithms to perform the processing of a particular device that performs the type of techniques described herein. It should also be understood that, unless otherwise specified herein, the specific order of steps and / or actions described in each flowchart is merely an example of an implementable algorithm and can vary in specific implementations and embodiments of the principles described herein.

[0110] Therefore, in some implementations, the techniques described herein can be embodied in computer-executable instructions implemented as software, including as application software, system software, firmware, middleware, embedded code, or any other suitable type of software. Such computer-executable instructions can be written using a variety of suitable programming languages ​​and / or any of the programming or scripting tools, and can also be compiled into executable machine language code or intermediate code that executes on a framework or virtual machine.

[0111] When the techniques described herein are embodied as computer-executable instructions, these instructions can be implemented in any suitable manner, including as multiple functional facilities, each providing one or more operations to perform the execution of an algorithm according to these techniques. However, an instantiated “functional facility” is a structural component of a computer system that, when integrated with and executed by one or more computers, enables the computers to perform a specific operational role. A functional facility can be part or all of a software element. For example, a functional facility can be implemented as a function of a process, or as a discrete process, or as any other suitable processing unit. If the techniques described herein are implemented as multiple functional facilities, each functional facility can be implemented in its own way; none of these need to be implemented in the same way. Furthermore, these functional facilities can be executed in parallel and / or serially as appropriate, and can exchange information with each other using shared memory on the computer on which they are executed, using message passing protocols, or in any other suitable manner.

[0112] Typically, functional facilities include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. The functionality of functional facilities can generally be combined or distributed as needed within the system in which they operate. In some implementations, one or more functional facilities performing the techniques described herein can together form a complete software package. In alternative implementations, these functional facilities may be adapted to interact with other unrelated functional facilities and / or processes to implement software program applications, such as verification clients (112 or 132), glitch detection systems (120, 200, 400, or 500), data preparation modules (202 or 502), feature extraction modules (204, 404, or 504), glitch detection models (206, 406, or 506), recorders (208 or 508), etc. In other implementations, functional facilities may be adapted to interact with other functional facilities to form an operating system, including those available from Microsoft in Redmond, Washington. ® Microsoft ® Windows obtained by the Corporation of Redmond, Washington ® Operating system. In other words, in some implementations, functionalities may be implemented as part of the operating system or outside of it.

[0113] This document has described some exemplary functional facilities for performing one or more tasks. However, it should be understood that the described functional facilities and task divisions are merely illustrative of the types of functional facilities that can implement the exemplary techniques described herein, and implementation is not limited to any particular number, division, or type of functional facilities. In some specific implementations, all functionality may be implemented in a single functional facility. It should also be understood that in some specific implementations, some of the functional facilities described herein may be implemented together with or separately from other functional facilities (i.e., as a single unit or separate units), or some of these functional facilities may be omitted.

[0114] In some implementations, computer-executable instructions that implement the techniques described herein (when implemented as one or more functional facilities or in any other way) may be encoded on one or more computer-readable media to provide functionality to the media. Computer-readable media include magnetic media such as hard disk drives, optical media such as optical discs (CDs) or digital universal discs (DVDs), persistent or non-persistent solid-state storage (e.g., flash memory, magnetic RAM, etc.), or any other suitable storage media. Such computer-readable media may be implemented in any suitable manner, including as described below. Figure 8The computer-readable storage medium 806 (i.e., as part of the computing device 800) or as a separate, independent storage medium. As used herein, a “computer-readable medium” (also referred to as a “computer-readable storage medium”) means a tangible storage medium. A tangible storage medium is non-transitory and has at least one physical structural component. In a “computer-readable medium” as used herein, at least one physical structural component has at least one physical property that can be changed in some way during the process of creating a medium with embedded information, the process of recording information thereon, or any other process of encoding information into the medium. For example, the magnetization state of a portion of the physical structure of a computer-readable medium can be changed during the recording process.

[0115] Furthermore, some of the aforementioned technologies involve storing information (e.g., data and / or instructions) in certain ways for use by these technologies. In some specific implementations of these technologies (such as those implemented as computer-executable instructions), information may be encoded on a computer-readable storage medium. Where a particular structure is described herein in an advantageous format for storing that information, these structures can be used to give the information a physical organization when encoded on the storage medium. These advantageous structures can then provide functionality to the storage medium by influencing the operation of one or more processors interacting with the information; for example, by improving the efficiency of computer operations performed by the processors.

[0116] In this context, the technology may be embodied in some, but not all, specific implementations of computer-executable instructions that can be executed on one or more suitable computing devices operating in any suitable computer system, or one or more computing devices (or one or more processors of one or more computing devices) may be programmed to execute the computer-executable instructions. When the instructions are stored in a manner accessible to the computing device / processor, such as in local memory (e.g., on-chip cache or instruction register, computer-readable storage medium accessible via a bus, computer-readable storage medium accessible via one or more networks and accessible by the device / processor, etc.), the computing device or processor may be programmed to execute the instructions. The functional facilities including these computer-executable instructions may be integrated with and direct the operation of: a single multipurpose programmable digital computer device; a coordinated system of two or more multipurpose computer devices sharing processing power and jointly executing the technologies described herein; a single computer device or a coordinated system of computer devices (located in the same location or geographically distributed) dedicated to executing the technologies described herein; one or more field-programmable gate arrays (FPGAs) or any other suitable system for executing the technologies described herein.

[0117] Figure 8An exemplary embodiment of a computing device is illustrated, which presents the form of a computing device 800 that can be used in a system implementing the techniques described herein, although other embodiments are also possible. It should be understood that... Figure 8 It is neither intended to depict the necessary components for a computing device to operate according to the principles described herein, nor is it a comprehensive description.

[0118] Computing device 800 may include at least one processor 802, a network adapter 804, and a computer-readable storage medium 806. Computing device 800 may be, for example, a desktop or laptop computer, a personal digital assistant (PDA), a smartphone, a server, a wireless access point or other networking element, or any other suitable computing device. Network adapter 804 may be any suitable hardware and / or software enabling computing device 800 to communicate wired and / or wirelessly with any other suitable computing device via any suitable computing network. Computing network may include wireless access points, switches, routers, gateways, and / or other networking equipment, and one or more suitable wired and / or wireless communication media (including the Internet) for exchanging data between two or more computers. Computer-readable medium 806 may be adapted to store data to be processed and / or instructions to be executed by one or more processors 802. Processor 802 performs data processing and instruction execution. Data and instructions may be stored on computer-readable storage medium 806.

[0119] Data and instructions stored on computer-readable storage medium 806 may include computer-executable instructions that implement techniques operating according to the principles described herein. Figure 8 In the example, computer-readable storage medium 806 stores computer-executable instructions that implement various facilities and store various information as described above, including verification client 808, burr detection system 810 (e.g., burr detection system 120, 200, 400 or 500), burr detection model 812 (e.g., burr detection model 206, 406 or 506) and / or burr detection facilities for performing burr detection methods (600 or 700), etc.

[0120] Although Figure 8 Not illustrated, but computing devices may additionally have one or more components and peripheral devices (including input and output devices). These devices may be used to present a user interface, etc. Examples of output devices that can be used to provide a user interface include printers or displays for visual presentation of output, and speakers or other sound-generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards and pointing devices (such as mice, touchpads, and digitizers). As another example, computing devices may receive input information via voice recognition or in other audible formats.

[0121] Implementation schemes in which the technology is implemented in circuits and / or computer-executable instructions have been described. It should be understood that some implementation schemes may be in the form of methods, and at least one example of such a method has been provided. Actions performed as part of a method may be ordered in any suitable manner. Therefore, implementation schemes in which actions are performed in an order different from the illustrative order can be constructed, which may include the simultaneous execution of several actions, even if they are shown as sequential actions in the illustrative implementation scheme.

[0122] The various aspects of the above embodiments can be used individually, in combination, or in various arrangements not specifically discussed in the foregoing embodiments; therefore, their application is not limited to the details and arrangements of the components set forth in the foregoing description or illustrated in the accompanying drawings. For example, an aspect described in one embodiment can be combined in any way with aspects described in other embodiments.

[0123] The use of ordinal terms such as “first,” “second,” and “third” to modify claim elements in claims does not imply any priority, precedence, or order of one claim element relative to another, or the chronological order of the execution of method actions. Rather, it serves only as a label to distinguish one claim element with a specific name from another element with the same name (but using ordinal terms), thus differentiating claim elements.

[0124] Furthermore, the wording and terminology used herein are for descriptive purposes and should not be considered restrictive. The use of “including,” “comprising,” “having,” “containing,” “involving,” and their variations herein is intended to cover the items listed thereafter and their equivalents, as well as additional items.

[0125] The word “exemplary” is used herein to mean used as an example, instance, or illustration. Therefore, any implementation, particular embodiment, process, feature, etc., described herein as exemplary should be understood as an exemplary example and should not be construed as a preferred or advantageous example unless otherwise stated.

[0126] The phrase “and / or” as used in the specification and claims should be understood to mean “any one or both” of the elements so combined, that is, elements that are combined in some cases and separate in others. Multiple elements listed with “and / or” should be interpreted in the same way, that is, “one or more” of the elements so combined. In addition to the elements specifically identified by the “and / or” clause, other elements may optionally be present, whether related to or unrelated to those specifically identified. Thus, as a non-limiting example, when used in conjunction with open-ended language such as “comprising,” a reference to “A and / or B” may refer only to A in one embodiment (optionally including elements other than B); while in another embodiment it may refer to B; in yet another embodiment, only to B (optionally including elements other than A); in yet another embodiment, for both A and B (optionally including other elements); and so on.

[0127] Unless otherwise indicated, the terms “connected to” and “coupled to” (and their derivatives) as used in the specification and claims shall be deemed to allow both direct and indirect (i.e., via other elements or components) connections.

[0128] Therefore, having described several aspects of at least one embodiment, it should be understood that various changes, modifications, and improvements will readily occur to those skilled in the art. Such changes, modifications, and improvements are intended to be part of this disclosure and are intended to fall within the spirit and scope of the principles described herein. Thus, the foregoing description and figures are merely illustrative.

Claims

1. A computer-implemented burr detection method, the burr detection method comprising: For each of a plurality of images, the evaluation includes assessing whether the corresponding image has burrs. Extract one or more features from the image; The image and one or more features extracted from the image are provided as multiple inputs to at least one model; and The at least one model generates a classification output indicating whether the image has jagged edges; Generate one or more records identifying one or more images from the plurality of images, each of the one or more images being classified as having spiky features by the at least one model; and The verification status of the system under test (SUT) is determined based on one or more records.

2. The spur detection method of claim 1, wherein the at least one model comprises a neural network, and wherein providing the image and the one or more features extracted from the image as the plurality of inputs to the at least one model comprises providing the plurality of inputs to an input layer of the neural network.

3. The spur detection method according to claim 2, wherein the neural network is a convolutional neural network (CNN), wherein the CNN includes one or more convolutional layers, the one or more convolutional layers include a first convolutional layer, and wherein the one or more features are inserted into the CNN at the input of the first convolutional layer.

4. The burr detection method according to claim 1, further comprising: Obtain first audiovisual data derived from second audiovisual data processed by a computer system, wherein the second audiovisual data includes image data, and wherein the first audiovisual data includes the plurality of images.

5. The burr detection method according to claim 4, wherein the first audiovisual data is derived from the second audiovisual data via a deduplication process.

6. The burr detection method according to claim 1, further comprising: First audiovisual data is obtained derived from second audiovisual data processed by a computer system, wherein the second audiovisual data includes audio data, and wherein the first audiovisual data includes a set of images representing a corresponding set of segments of the audio data.

7. The burr detection method according to claim 6, wherein obtaining the first audiovisual data includes: Obtain the set of audio data segments; as well as Generate a set of images representing the corresponding set of audio data segments, wherein each image in the set of images corresponds to a corresponding audio data segment in the set of audio data segments and includes a spectrogram of the corresponding audio data segment.

8. The burr detection method according to claim 7, wherein: The plurality of images includes the set of images representing the corresponding set of audio data segments. The set of images includes a first image representing a first audio data segment, and The classification of the first image as having spikes by the at least one model indicates that the first audio data segment has spikes.

9. The spur detection method according to claim 8, wherein the classification output corresponding to the first image further indicates one or more probabilities that the first audio data segment has one or more types of audio spurs.

10. The burr detection method according to claim 7, wherein the at least one model includes at least one first model, wherein the one or more records include one or more first records, and wherein the method further comprises: For each audio data segment in the set of audio data segments, evaluate whether the corresponding audio data segment has glitches, including: The image, including the spectrogram of the audio data segment, is provided as input to at least one second model; and The at least one second model generates a classification output indicating whether the audio data segment represented by the image has glitches; Generating one or more records identifying one or more audio data segments from the set of audio data segments; secondly, each of the one or more audio data segments is classified as spiking by the at least one second model; and Provide the user with one or more second records.

11. The spur detection method according to claim 1, wherein the one or more features include a first feature indicating one or more frequency domain properties of the image, a second feature indicating a plurality of pixel intensity gradients derived from the image, and / or a third feature characterizing the anomalousness of the plurality of pixel intensity values ​​derived from the image.

12. The burr detection method according to claim 11, wherein extracting the one or more features includes: The first feature is extracted based on the Fourier transform of the image; The second feature is extracted based on the orientation histogram of the multiple pixel intensity gradients; And / or The third feature is extracted based on multiple anomaly scores of the corresponding multiple pixel intensity values.

13. The burr detection method according to claim 1, wherein: For each of the one or more images classified as having spikes by the at least one model, the classification output further indicates one or more probabilities that the image has spikes of one or more image spike types.

14. The burr detection method according to claim 13, wherein the one or more image burr types include stripe merging burr, color-changing burr, dotted line burr, line pixelation burr, Morse code burr, parallel line burr, radial dotted line burr, random patch burr, regular triangular partitioning burr, shader burr, shape burr, square patch burr, stuttering burr, texture protrusion burr, and / or triangular burr.

15. A verification system, the verification system comprising: A burr detection system communicatively coupled to a system under test (SUT), the burr detection system including at least one processor and at least one computer-readable storage medium having instructions encoded thereon, the instructions causing the at least one processor to perform operations when executed by the at least one processor, the operations including: Obtain first audiovisual data derived from second audiovisual data processed by the SUT, wherein the second audiovisual data includes image data, and wherein the first audiovisual data includes a plurality of images; For each of the plurality of images, an evaluation is performed to determine whether the corresponding image contains jagged edges, the evaluation including: Extract one or more features from the image; The image and one or more features extracted from the image are provided as multiple inputs to at least one model; and The at least one model generates a classification output indicating whether the image has jagged edges; Generate one or more records identifying one or more images from the plurality of images, each of the one or more images being classified as having spiky features by the at least one model; and The verification status of the SUT is determined based on the one or more records.

Images