Automatically assessing the significance of pixel anomalies

Abstract

According to one embodiment, a system for performing automatic significance assessment of pixel anomalies includes a computing platform having a hardware processor and a system memory storing software code. The hardware processor is configured to execute the software code to analyze whether there is a pixel anomaly in an image, obtain a significance standard of the image when the analysis of the image detects the presence of the pixel anomaly, and classify the pixel anomaly as one of a significant anomaly or a harmless anomaly based on the significance standard of the image. The hardware processor is also configured to execute the software code to ignore the pixel anomaly when the pixel anomaly is classified as a harmless anomaly, and mark the pixel anomaly when the pixel anomaly is classified as a significant anomaly.

Description

Technical Field

[0001] The present disclosure relates to automatic assessment of significance of pixel anomalies. Background Art

[0002] Pixel errors often occur in images, but can be difficult and costly to identify and correct. For example, anomalous pixels in a video frame can be introduced by many different processes in the video production pipeline. Final quality procedures to detect and correct such errors are usually completed before the video is finally released.

[0003] In traditional techniques, abnormal pixel detection is usually performed by inspectors. Typically, these inspectors are tasked with checking every frame of every video hundreds of times before the video is finally distributed. Due to the strong reliance on human involvement, traditional pixel error detection and correction methods are expensive and time-consuming, which is not desirable. In addition, not all pixel anomalies require correction. For example, certain pixel anomalies can be reasonably ignored based on their location in the image and their relationship to particularly important features in the image. In other words, not all pixel errors are significant enough to be worth the cost associated with their correction. Therefore, there is a need in the art for an automatic solution that can accurately assess the significance of abnormal pixel errors detected in an image. Summary of the invention

[0004] The present disclosure provides systems and methods for performing automatic assessment of significance of pixel anomalies, substantially as shown in and / or described in connection with at least one of the accompanying drawings and as more fully set forth in the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Figure 1A An exemplary system for performing automatic significance assessment of pixel anomalies according to one embodiment is shown;

[0006] Figure 1B An exemplary system for performing automatic significance assessment of pixel anomalies according to another embodiment is shown;

[0007] Figure 2 According to one embodiment, a suitable Figure 1A and Figure 1B exemplary software code executed by the hardware processor of the illustrated system;

[0008] Figure 3 FIG. 1 shows a schematic diagram of a circuit according to an embodiment of the present invention. Figure 2 and Figure 3 An exemplary graphical user interface (GUI) provided by the software code shown;

[0009] Figure 4A flowchart presenting an exemplary method for performing automatic significance assessment of pixel anomalies is shown according to one embodiment; and

[0010] Figure 5 A flow chart presenting an exemplary method for performing automatic significance assessment of pixel anomalies is shown according to another embodiment. DETAILED DESCRIPTION

[0011] The following description contains specific information related to the embodiments in the present disclosure. Those skilled in the art will recognize that the present disclosure can be implemented in a manner different from that specifically discussed herein. The drawings and detailed descriptions thereof in this application are only for exemplary embodiments. Unless otherwise specified, the same or corresponding elements in the drawings may be represented by the same or corresponding reference numerals. In addition, the drawings and illustrations in this application are generally not drawn to scale and are not intended to correspond to actual relative sizes.

[0012] The present application discloses a system and method for performing an automatic assessment of the significance of pixel anomalies, which overcomes the shortcomings and deficiencies in conventional techniques. It should be noted that the method disclosed in the present application can be performed as a substantially automated process by a substantially automated system. It should also be noted that the term "automated" as used in the present application refers to systems and processes that do not require human user participation in performing image inspection. Although in some embodiments, inspectors can check the performance of the automated system described herein, human participation is optional. Therefore, the method described in the present application can be performed under the control of the hardware processing components of the disclosed automated system.

[0013] Figure 1A An exemplary system 100 for performing automatic assessment of the significance of pixel anomalies according to one embodiment is shown. As described below, in some embodiments, the system 100 can be implemented using a computer server accessible via a local area network (LAN), or can be implemented as a cloud-based system. Figure 1A As shown, system 100 includes a computing platform 102 having a hardware processor 104 and a system memory 106 implemented as a non-transitory storage device storing software code 110 that provides a graphical user interface (GUI) 130. In addition, in some embodiments, system memory 106 may also store a saliency criteria database 108.

[0014] like Figure 1A As further shown, system 100 is implemented in a use environment having a communication network 140 and a user system 150 having a display 158. User system 150 is configured to be used by user 144. Figure 1AAlso shown are a network communication link 142 that interactively connects the user system 150 and the system 100 via the communication network 140, one or more images 160 (hereinafter referred to as "image 160"), significance criteria 146 for evaluating pixel anomalies in the image 160, one or more corrections 148 to the image 160, and a report 170 marking one or more significant pixel anomalies in the image 160.

[0015] Note that image 160 may correspond to a single digital image, such as a digital photograph, or to video content including one or more video frames or one or more video stream segments, such as one or more video "shots" or one or more video scenes. It is also noted that a "shot" as used in this application refers to a sequence of video frames captured from a unique camera perspective without editing and / or other film transitions. Thus, in various embodiments, image 160 may correspond to a single video frame, a single video shot including multiple individual video frames, or to one or more scenes including multiple shots.

[0016] It is also noted that, although for conceptual clarity, the present application refers to the software code 110 as being stored in the system memory 106, more generally, the system memory 106 may take the form of any computer-readable non-transitory storage medium. The expression "computer-readable non-transitory storage medium" used in the present application refers to any medium other than a carrier wave or other temporary signal that provides instructions to the hardware processor 104 of the computing platform 102. Therefore, a computer-readable non-transitory medium may correspond to various types of media, such as volatile media and non-volatile media. Volatile media may include dynamic memory, such as dynamic random access memory (dynamic RAM), while non-volatile memory may include optical, magnetic, or electrostatic storage devices. Common forms of computer-readable non-transitory media include, for example, optical disks, RAM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), and flash memory.

[0017] Also note that although Figure 1A The software code 110 is described as being stored entirely in the system memory 106, but this representation is provided merely to aid conceptual clarity. More generally, the system 100 may include one or more computing platforms 102, such as computer servers, which may be co-located or may form an interactively linked but distributed system, such as a cloud-based system.

[0018] Thus, hardware processor 104 and system memory 106 may correspond to distributed processor and memory resources within system 100. Therefore, it should be understood that various features of software code 110, such as those described below with reference to Figure 2The software modules, described in more detail, may be stored and / or executed using distributed memory and / or processor resources of system 100 .

[0019] according to Figure 1A In the illustrated embodiment, a user 144 may interact with the system 100 via the communication network 140 using a user system 150. In one such embodiment, the computing platform 102 may correspond to one or more network servers that may be accessed, for example, via a packet-switched network such as the Internet. Alternatively, the computing platform 102 may correspond to one or more computer servers that support a wide area network (WAN), a local area network (LAN), or are included in another type of limited distributed or private network.

[0020] Although user system 150 is Figure 1A 140 is shown as a desktop computer, but this representation is also provided only as an example. More generally, user system 150 can be any suitable mobile or fixed computing device or system that implements data processing capabilities sufficient to provide a user interface, support connection to communication network 140, and implement the functions attributed to user system 150 herein. For example, in other embodiments, user system 150 can take the form of, for example, a laptop computer, a tablet computer, or a smart phone. User 144 can interact with system 100 using user system 150 to perform an automatic assessment of the significance of pixel anomalies detected by system 100 in image 160 using software code 110 executed by hardware processor 104.

[0021] Note that in various embodiments, when a report 170 marking one or more significant pixel anomalies in an image 160 is generated using the software code 110, the report 170 can be stored in the system memory 106 and / or can be copied to a non-volatile memory. Alternatively or additionally, in some embodiments, the report 170 can be displayed to the user 144 via the user system 150 and the display 158 of the GUI 130.

[0022] With respect to the display 158 of the user system 150, the display 158 may be physically integrated with the user system 150, or may be communicatively coupled to the user system 150, but physically separate from the user system 150. For example, where the user system 150 is implemented as a smartphone, a laptop, or a tablet computer, the display 158 will typically be integrated with the user system 150. Conversely, where the user system 150 is implemented as a desktop computer, the display 158 may take the form of a monitor that is separate from the user system 150 in the form of a computer tower. Furthermore, the display 158 may be implemented as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, or any other suitable display screen that performs a physical conversion of a signal to light.

[0023] In general, one or more pixels within an image may be identified as abnormal, i.e., as having a defect, based on their context. For example, a human observer will typically identify an image region corresponding to one or more pixels as abnormal based on the color of the one or more pixels relative to the surrounding pixels in the image. If the color of an image region differs too much from its surroundings to a human observer, an anomaly may exist. As described above, in conventional techniques, abnormal pixel detection is typically performed by human inspectors. Typically, these inspectors are tasked with checking every frame of every video hundreds of times before it is finally distributed. Traditional pixel error detection and correction methods are expensive and time-consuming due to their strong reliance on human involvement, which is not desirable.

[0024] In addition, as described above, not all pixel anomalies require correction. For example, some pixel anomalies may be reasonably ignored based on their location in the foreground or background of the image, and their relationship to particularly important features or main characters in the image. That is, not all pixel errors are significant enough to be worth the cost associated with their correction. Therefore, the system 100 is configured to accurately assess the significance of abnormal pixel errors detected in the image 160, and only mark those pixel errors that meet the significance criteria, which may be predetermined or may be dynamically determined based on the user 144's input to the GUI 130. Regarding the concept of pixel anomaly significance, Cong Runmin et al. described image saliency detection and determination in a publication entitled "Review of Visual Saliency Detection with Comprehensive Information". (See Cong Runmin, Lei Jianjun, Fu Huazhu, Cheng Mingming, Lin Weisi, and Huang Qingming, IEEE Transactions on Circuits and Systems for Video Technology, October 2019, Vol. 29, No. 1), the entire text of which is incorporated herein by reference.

[0025] Note that while in some embodiments, the saliency criteria 146 applied by the system 100 may be dynamically determined based on user input to the GUI 130, in other embodiments, the saliency may be dynamically calculated based on the previous and / or surrounding context of the image. That is, the dynamic saliency criteria 146 may be automatically adjusted based on the content of the image 160. For example, a pixel anomaly with a particular residual value (where the residual is the difference between the expected value of the pixel and the actual value of the pixel) may not be considered salient in an action scene that includes significant movement, but may be considered salient when the same residual value anomaly occurs in a scene with little movement and relatively stillness. In another example, when a pixel anomaly appears at the background edge of an image depicting a prominent focal character, the pixel anomaly may be considered insignificant, but another image may not include the focal character, and the saliency criteria 146 will then be adjusted to reflect the difference in the content of the image 160.

[0026] Such dynamic thresholds can be implemented in several different ways. In the use case where the images 160 include a sequence of video frames from a movie, one approach includes using machine learning to train a model that predicts human visual attention to various segments (e.g., shots or scenes) of a movie. Each shot or scene can be analyzed using the predictive model to identify an average score of attention, and the saliency criterion 146 can be dynamically determined based on, for example, two or three standard deviations from that average. Since the distribution will be calculated for each shot or scene, the saliency criterion 146 will also change for each shot or scene.

[0027] Optionally, in embodiments where the system 100 determines saliency based on a predetermined value such as "outside the character outline" versus "inside the character outline", object detection and structured metadata may be used to perform an additional series of checks, such as <center portion of screen>, <large object takes up at least 60% of the screen>, <animated object>, <minor character>, <major character>, etc. In these embodiments, the system 100 may determine which metadata category the image 160 corresponds to, and therefore which saliency criteria 146 to use to determine pixel anomaly saliency. For example, a variety of computer vision techniques, including machine learning convolutional neural networks, may be used to determine whether the image 160 corresponds to any of the above metadata categories, and if so, which regions of the image 160 correspond.

[0028] Figure 1B An exemplary system (ie, user system 150) for performing automatic assessment of the significance of pixel anomalies according to another embodiment is shown. Note that Figure 1B Zhong You and Figure 1A Any features identified by those same reference numerals shown in the corresponding figures correspond to those previously described features, respectively, and may share any properties attributed to those corresponding features by the present disclosure. Figure 1A In addition to the features shown, Figure 1B User system 150 is shown as including a computing platform 152 having a hardware processor 154 and system memory 156 implemented as a non-transitory storage device. Figure 1B In the exemplary embodiment shown, system memory 156 stores the significance criteria database 108 and software code 110 that provides the GUI 130 . Figure 1B Also shown in FIG. 1 are user 144 , image 160 , significance criteria 146 , and report 170 .

[0029] Figure 1B The embodiment shown is similar to Figure 1AThe illustrated embodiment differs in that the significance criteria database 108 and the software code 110 are stored on a user system 150, and the software code 110 is executed locally using a hardware processor 154 of a computing platform 152. Figure 1B As shown, in some embodiments, the user system 150 may include Figure 1A All or substantially all of the features and functionality of system 100.

[0030] Reference now Figure 2 , Figure 2 According to one embodiment, a suitable Figure 1A and Figure 1B Example software code executed by the hardware processor of the system shown. Figure 2 As shown, the software code 210 is configured to receive one or more images 260 (hereinafter referred to as "images 260") and generate a report 270 that marks any significant pixel anomalies detected in the images 260. Figure 2 As further shown, the software code 210 includes an anomaly detection module 212, an optional segmentation module 214, a significance analysis module 216, a reporting module 218, and a GUI 230. In addition, Figure 2 One or more pixel anomalies 222 , segmentation data 224 , significance criteria 246 , one or more significant pixel anomalies 226 , and report 270 of marking one or more significant pixel anomalies 226 in image 260 . Figure 2 Also shown is a saliency criteria database 208 , optionally accessed by software code 210 , and one or more corrections 248 to image 260 received via GUI 230 .

[0031] Image 260, report 270, software code 210, GUI 230, significance criteria database 208, and significance criteria 246 generally correspond to Figure 1A and Figure 1B 146. Thus, the image 260, report 270, software code 210, GUI 230, saliency criteria database 208, and saliency criteria 246 may share any features imparted by the present disclosure to the respective image 160, report 170, software code 110, GUI 130, saliency criteria database 108, and saliency criteria 146. That is, like the software code 210, in addition to providing the GUI 130, the software code 110 may include features corresponding to the anomaly detection module 212, the optional segmentation module 214, the saliency analysis module 216, and the reporting module 218, respectively.

[0032] also, Figure 2 The one or more corrections 248 in generally correspond to Figure 1A One or more corrections 148 in , and those corresponding features may share any characteristics assigned to any corresponding feature by the present disclosure. Figure 1A and Figure 1B As a result of correction input by user 144 to user system 150, one or more corrections 148 / 248 may be received from user system 150 via GUI 130 / 230.

[0033] Figure 3 An exemplary GUI 330 provided by software code 110 / 210 is shown according to one embodiment. Figure 3 As shown, before or after classifying pixel anomalies 364 as significant or harmless (not significant) based on significance criteria 346, GUI 330 can display report 370, which includes a list of pixel anomalies 364 detected in images 360a, 360b, and 360c (hereinafter referred to as "images 360a-360c"), namely, pixel anomalies 364a, 364b, 364c, and 364d (hereinafter referred to as "pixel anomalies 364a-364d"). In these figures, images 360a-360c generally correspond to FIG. 1, Figure 2 and Figure 3 360a-360c and report 370 may include any characteristics imparted to the corresponding images 160 / 260 and reports 170 / 270 by the present disclosure, and vice versa. Figure 3 GUI 330 and significance criteria 346 in FIG. 1 generally correspond to FIG. Figure 2 and Figure 3 Thus, the GUI 130 / 230 and the prominence criteria 146 / 246 may share any characteristics imparted to the corresponding GUI 330 and the prominence criteria 346 by the present disclosure, and vice versa.

[0034] Combined with reference Figure 1A , Figure 1B , Figure 2 and Figure 3 Note that in some embodiments, the saliency criteria 146 / 246 / 346 of the images 160 / 260 / 360a-360c may be predetermined saliency criteria and may be stored in the saliency criteria database 108 / 208. However, in other embodiments, the saliency criteria 146 / 246 / 346 may be obtained as one or more inputs from the user 144. Figure 3As shown, input to GUI 330 may result in one or more of image region criteria 346a, character criteria 346b, foreground / background criteria 346c, or content segment criteria 346d being used to classify pixel anomalies 364 detected in images 360a-360c. Figure 3 In the exemplary use case shown, the user 144 has selected the character criterion 346b and the content segment criterion 346d. In contrast, the image area criterion 346a and the foreground / background criterion 346c are displayed in gray, indicating that they are not currently applied.

[0035] like Figure 3 As further shown, report 370 may flag the presence of a significant pixel anomaly in an image. For example, where pixel anomaly 364a in image 360a is classified as significant, the location of the significant pixel anomaly may be identified in image 360a using a bounding box or boundary 366a around the location of the significant pixel anomaly in image 360a.

[0036] Image region criteria 346a may allow user 144 to designate one or more regions of images 360a-360c as being more sensitive to the presence of pixel anomalies than other regions of those images. In those embodiments, pixel anomalies 364 determined to be within the region identified by user 144 may be classified as significant, while those pixel anomalies outside of the one or more sensitive regions may be classified as harmless and ignored.

[0037] Note that in some embodiments, the image region criteria 346a may explicitly identify the region of interest, such as providing xy coordinates that define the region of interest. Alternatively or additionally, the user 144 may explicitly identify the region of interest by manually drawing a box or window around the region of interest using the GUI 330. Alternatively or additionally, the image region criteria 346a may implicitly identify the region of interest, such as by specifying a color value for a region of pixels of interest.

[0038] Role criteria 346b can allow user 144 to specify one or more roles, and the appearance of these roles is particularly important for the aesthetics of images 360a-360c. In those embodiments, for example, pixel anomalies 364 determined to be located on or between roles identified by user 144 can be classified as significant, while those pixel anomalies located at least a certain threshold distance from these roles can be classified as harmless and ignored. Note that in some embodiments, the threshold distance for distinguishing potential significant anomalies and harmless anomalies can be predetermined. Alternatively or additionally, in some embodiments, such a threshold distance can be a user-selectable parameter. Alternatively or additionally, in some embodiments, the threshold distance for distinguishing potential significant anomalies and harmless anomalies can be dynamically determined based on the content type of images 360a-360c.

[0039] In one embodiment, a proportional pixel area around a character's outline may establish a threshold distance that distinguishes potentially significant anomalies from harmless anomalies. By way of example only, 2% or other larger percentages of a character's estimated area may extend the threshold distance more for larger or closer characters that take up more space, and less for smaller or farther away characters.

[0040] The foreground / background criteria 346c may allow the user 144 to designate one of the foreground or background of the images 360a-360c as being more important to the aesthetics of one or more of the images 360a-360c. In the case where the user 144 designates the foreground as being more important, pixel anomalies located in the foreground may be classified as significant pixel anomalies, while pixel anomalies located in the background may be classified as harmless and ignored. Conversely, in the case where the user 144 designates the background as being more important, pixel anomalies located in the background may be classified as significant pixel anomalies, while pixel anomalies located in the foreground may be classified as harmless and ignored.

[0041] The content segment criteria 346d may allow the user 144 to specify the significance criteria 346 by content segment, such as one or more frames, shots, or scenes of a video. That is, in embodiments where the image 160 / 260 or one of the images 360a-360c is a video frame, the content segment criteria 346d for the image may be determined based on one of a video shot that includes the video frame or a video scene that includes the video frame. For example, in some use cases, the user 144 may choose to specify the significance criteria 346 once for a shot, scene, or any other user-selected content segment (e.g., a portion of a shot or scene, or multiple shots or scenes). In those use cases, the same significance criteria may be automatically applied to each frame in the user-selected segment, thereby advantageously alleviating the user 144 from the burden of specifying the significance criteria 346 for each frame individually. However, in some use cases, the user 144 may prefer to specify the significance criteria 346 on a frame-by-frame basis, in which case the present automatic solution for assessing the significance of pixel anomalies enables the user 144 to do so.

[0042] For example, in the use case where user 144 selects content segment criterion 346d, user 144 may change other prominence criteria 346 between segments of content including images 360a-360c. Additionally, in some implementations, GUI 130 / 230 / 330 may be configured to prompt user 144 to provide input for selecting one or more prominence criteria, either because the end of a segment has been reached, or based on any other relevant criteria.

[0043] Will refer to Figure 4Combined with Figure 1A , Figure 1B , Figure 2 and Figure 3 The functionality of the software code 110 / 210 and the GUI 130 / 230 / 330 is further described. Figure 4 A flowchart 480 is shown that presents an exemplary method for performing automatic significance assessment of pixel anomalies, according to one embodiment. Figure 4 In the method outlined in FIG. 48 , it is noted that certain details and features have been omitted from flowchart 480 in order to avoid obscuring the discussion of the described embodiments in this application.

[0044] Flowchart 480 begins by analyzing images 160 / 260 / 360a-360c for one or more pixel anomalies 222 (operation 481). Figure 1A and Figure 2 As shown, in one embodiment, user 144 may interact with system 100 using user system 150 to evaluate the significance of one or more pixel anomalies 222 included in image 160 / 260, each of which may be a digital photograph or one or more video frames, for example. Figure 1A As further shown, user 144 may transmit image 160 from user system 150 to system 100 via communication network 140 and network communication link 142. Alternatively, image 160 may be received from a third party source or may be stored in system memory 106.

[0045] Alternatively, as shown in Figures 1 and Figure 2 As shown, in some embodiments, the user 144 can utilize the processor and memory resources of the user system 150 to locally evaluate the significance of one or more pixel anomalies 222 included in the image 160 / 260 using the user system 150. In those embodiments, the image 160 / 260 can be obtained from the system memory 156 of the user system 150.

[0046] Combined with reference Figure 1A , Figure 1B and Figure 2, the image 160 / 260 may be analyzed for one or more pixel anomalies 222 using an anomaly detection module 212 of software code 110 / 210 executed by hardware processor 104 of system 100 or hardware processor 154 of user system 150. Note that operation 481 may be performed using any suitable pixel error detection method, including but not limited to the method described in U.S. Patent Application No. 16 / 243,650, filed on January 9, 2019, entitled “Pixel Error Detection System” or the method described in U.S. Patent Application No. 16 / 836,453, filed on March 31, 2020, entitled “Automated Pixel Error Detection Using an Inpainting Neural Network”, both of which are incorporated herein by reference in their entirety.

[0047] When the analysis of the image 160 / 260 detects the presence of the pixel anomaly 222, the flowchart 480 continues with obtaining the significance criteria 146 / 246 / 346 for the image 160 / 260 / 360a-360c (operation 482). As described above, in some embodiments, the significance criteria 146 / 246 for the image 160 / 260 / 360a-360c may be predetermined significance criteria and may be stored in the significance criteria database 108 / 208. In some of these embodiments, as Figure 1A As shown, operation 482 may correspond to retrieving or otherwise obtaining, by software code 110 executed by hardware processor 104, the significance criteria 146 from a significance criteria database 108 stored in system memory 106. However, in other implementations, significance criteria database 108 may not be stored in system memory 106, but may be remote from system 100. In those implementations, hardware processor 104 may execute software code 110 to obtain significance criteria 146 by accessing significance criteria database 108 via communication network 140 and network communication link 142.

[0048] refer to Figure 3 In other embodiments, the significance criteria 346 may be obtained as one or more inputs to the GUI 330. Figure 3 As shown, input to GUI 330 may result in one or more of image region criteria 346a, character criteria 346b, foreground / background criteria 346c, or content segment criteria 346d being applied to classify pixel anomalies detected in images 360a-360c, as described above.

[0049] Note that while the exemplary method outlined in flowchart 480 describes analyzing the images 160 / 260 / 360a-360c for the presence of one or more pixel anomalies 222 in operation 481 and subsequently obtaining the significance criteria 146 / 246 / 346 for the images 160 / 260 / 360a-360c in operation 482, in some implementations, operation 482 may be performed prior to operation 481. Identifying the region of interest images 160 / 260 / 360a-360c first and then analyzing those particular regions of interest for the presence of one or more pixel anomalies 222 may advantageously reduce the computational overhead required to detect significantly anomalous pixels.

[0050] Continuing with the exemplary method outlined in flowchart 480, the pixel anomaly 222 detected in operation 481 is classified as one of a significant anomaly or a harmless anomaly based on the significance criteria 146 / 246 / 346 of the image 160 / 260 / 360a-360c (operation 483). The classification of the pixel anomaly 222 detected in operation 481 may be performed using a significance analysis module 216 of software code 110 / 210 executed by a hardware processor 104 of system 100 or a hardware processor 154 of user system 150.

[0051] When the pixel anomaly 222 is classified as a harmless anomaly in operation 483 (operation 484), the flowchart 480 continues to ignore the pixel anomaly 222 detected in operation 481. As described above, not all pixel anomalies 222 in the image 160 / 260 / 360a-360c are worth the time and / or cost associated with their correction. Therefore, those pixel anomalies 222 determined to not meet the significance criteria obtained in operation 482 can be ignored without substantially detracting from the aesthetics of the image 160 / 260. Operation 484 can be performed using the significance analysis module 216 of the software code 110 / 210, which is executed by the hardware processor 104 of the system 100 or the hardware processor 154 of the user system 150.

[0052] When the pixel anomaly 222 is classified as a significant pixel anomaly 226 in operation 484 (operation 485), the flowchart 480 continues to mark the pixel anomaly 222 detected in operation 481. As described above, the pixel anomaly 222 in the image 160 / 260 is significant enough, such as based on its location in the image 160 / 260 or its proximity to a feature character or object in the image 160 / 260, to be worth the time and / or cost associated with its correction. Therefore, those pixel anomalies 222 determined to meet the significance criteria obtained in operation 482 can be marked in the image 160 / 260 and included in the report 170 / 270 / 370 output by the GUI 130 / 230 / 330. Operation 485 may generate a report 170 / 270 and may be performed using a reporting module 218 of software code 110 / 210 executed by a hardware processor 104 of the system 100 or by a hardware processor 154 of the user system 150 .

[0053] Note that the report 170 / 270 marking the pixel anomalies 226 detected in the images 160 / 260 / 360a-360c can be an interactive report, enabling the user 144 to switch and view: a) all detected pixel anomalies, b) only pixel anomalies classified as significant pixel anomalies, optionally with some marking or color distinction between significant anomalies and harmless anomalies. In some embodiments, the report 170 / 270 can also include a timestamp for each frame including a significant pixel anomaly or any pixel anomaly, a visual representation of the frame with the abnormal area identified, for example, with a circle or box, and optionally a close-up representation of each abnormal area for enhanced visual confirmation.

[0054] It is also noted that while the exemplary method outlined in flowchart 480 describes ignoring detected pixel anomalies 222 when pixel anomalies 222 are classified as harmless in operation 484, and subsequently marking detected pixel anomalies 222 when pixel anomalies 222 are classified as significant in operation 485, this order is provided only as an example. In various embodiments of the present method, operations 484 and 485 may be performed substantially simultaneously, operation 485 may precede operation 484, or one of operations 484 or 485 may be omitted. For example, in the case where detected pixel anomalies 222 are classified as significant, operation 484 may be omitted, and operation 485 may follow operation 483. Conversely, in the case where detected pixel anomalies 222 are classified as harmless, operation 485 may be omitted, and flowchart 480 may end with operation 484.

[0055] As described above, in some embodiments, the flowchart 480 may end with operation 484 or operation 485 as described above. However, in other embodiments, the flowchart 480 may continue and end with optionally applying the correction 148 / 248 to the image 160 / 260 to correct the significant anomaly 226 (operation 486). In one embodiment, for example, the correction 148 / 248 may be applied to the image 160 / 260 by the software code 110 / 210 executed by the hardware processor 104 of the system 100 or the hardware processor 154 of the user system 150 to correct the significant anomaly 226 in the automated process. In other embodiments, the significant anomaly 226 may be marked to the user 144 to correct the significant anomaly 226. In those other embodiments, the software code 110 / 210 executed by the hardware processor 104 of the system 100 or executed by the hardware processor 154 of the user system 150 may receive the correction 148 / 248 to the image 160 / 260 from the user 144 to correct the significant anomaly 226. Furthermore, in those embodiments, software code 110 / 210 executed by hardware processor 104 of system 100 or executed by hardware processor 154 of user system 150 may apply correction 148 / 248 to image 160 / 260 in response to receiving correction 148 / 248 from user 144 .

[0056] Figure 5 A flowchart 590 is shown that presents an exemplary method for performing automatic assessment of the significance of pixel anomalies according to another embodiment. Figure 4 In the method outlined in FIG. 590 , it is noted that certain details and features have been omitted from flowchart 590 in order to avoid obscuring the discussion of the described embodiments in this application.

[0057] Flowchart 590 begins by analyzing the video clips included in images 160 / 260 / 360a-360c to find out if one or more pixel anomalies 222 exist (operation 591). Figure 1A and Figure 2 As shown, in one embodiment, user 144 may interact with system 100 using user system 150 to evaluate the significance of one or more pixel anomalies 222 included in image 160 / 260, for example, image 160 / 260 may include a video clip in the form of a video shot or a video scene. Figure 1A As further shown, user 144 may transmit image 160 from user system 150 to system 100 via communication network 140 and network communication link 142. Alternatively, image 160 may be received from a third party source or may be stored in system memory 106.

[0058] Alternatively, as shown in Figures 1 and Figure 2As shown, in some embodiments, the user 144 can utilize the processor and memory resources of the user system 150 to locally evaluate the significance of one or more pixel anomalies 222 included in the image 160 / 260 using the user system 150. In those embodiments, the image 160 / 260 can be obtained from the system memory 156 of the user system 150.

[0059] Combined with reference Figure 1A , Figure 1B and Figure 2 In some embodiments, the video segments included in the images 160 / 260 may be predetermined and may be identified by the anomaly detection module 212 of the software code 110 / 210 based on metadata included in the images 160 / 260. However, in other embodiments, analyzing the video segments included in the images 160 / 260 for the presence of one or more pixel anomalies 222 may include segmenting the images 160 / 260 by shot or scene, for example using an optional segmentation module 214 of the software code 110 / 210 to generate segmentation data 224 identifying the video segments.

[0060] Therefore, according to Figure 1A In the exemplary embodiment shown, operation 591 may be performed by software code 110 / 210 executed by hardware processor 104 of system 100 and using anomaly detection module 212 alone or in combination with optional segmentation module 214. Figure 1B In the exemplary embodiment shown, operation 591 may be performed by software code 110 / 210 executed by hardware processor 154 of user system 150 and using anomaly detection module 212 alone or in combination with optional segmentation module 214. Note that while anomaly detection module 212 analyzes images 160 / 260 / 360a-360c, segment data 224 is important for embodiments where user 144 has specified what criteria to use for saliency in a particular segment, as described above. Otherwise, the system will not recognize when to reset the saliency criteria, or prompt user 144 to confirm their saliency criteria on the next segment.

[0061] Any suitable pixel error detection method may be used to analyze the video clip included in the image 160 / 260 for abnormal pixels in operation 591. By way of example only, pixel error detection may be based on the method described in U.S. Patent Application No. 16 / 243,650, filed on January 9, 2019, entitled “Pixel Error Detection System” or the method described in U.S. Patent Application No. 16 / 836,453, filed on March 31, 2020, entitled “Automated Pixel Error Detection Using an Inpainting Neural Network”, both of which are incorporated herein by reference in their entirety.

[0062] Continuing with flowchart 590, for each video segment that includes pixel anomaly 222, one or more video frames included in the segment are displayed using GUI 130 / 230 / 330 (operation 592). For example, in the case where images 160 / 260 include one of the video segments that is found to include pixel anomaly 222, and in the case where images 360a-360c are also included in the video segment that includes pixel anomaly 222, operation 592 corresponds to displaying one or more of images 360a-360c using GUI 330. Figure 1A In the exemplary embodiment shown, operation 592 may be performed by software code 110 executed by hardware processor 104 of system 100. Figure 1B In the exemplary embodiment shown, operation 592 may be performed by software code 110 executed by hardware processor 154 of user system 150 .

[0063] Combined with reference Figure 2 , Figure 3 and Figure 5 Continuing with flowchart 590, for each video segment that includes a pixel anomaly 222, a saliency criterion 246 / 346 for the video segment that includes one or more images 260 / 360a-360c displayed in operation 592 is received via GUI 330 (operation 593). As described above, input to GUI 330 may result in one or more of image region criteria 346a, character criteria 346b, foreground / background criteria 346c, or content segment criteria 346d being used to classify the pixel anomaly 222 detected in the image 260 / 360a-360c, as described above. Also refer to Figure 1A and Figure 1B ,according to Figure 1AIn the exemplary embodiment shown, operation 593 may be performed by software code 110 / 210 executed by hardware processor 104 of system 100 and using significance analysis module 216. Figure 1B In the exemplary embodiment shown, operation 593 may be performed by software code 110 / 210 executed by hardware processor 154 of user system 150 and utilizing significance analysis module 216 .

[0064] Note that although the exemplary method outlined in flowchart 590 describes analyzing the video segments included in images 160 / 260 / 360a-360c in operation 591 before receiving the significance criteria 146 / 246 / 346 in operation 593, in some embodiments, operation 593 may be performed before operation 591. Receiving the significance criteria 146 / 246 / 346 first and then analyzing the pixel anomalies of the video segments based on the significance criteria may advantageously reduce the computational overhead required to detect significant anomalous pixels. Therefore, in some embodiments, operation 593 may be performed before operation 591.

[0065] Continuing with the flowchart 590, for each video segment including the pixel anomaly 222, the pixel anomaly 222 is classified as a significant anomaly 226 or a harmless anomaly based on the significance criteria 146 / 246 / 346 of the video segment including the pixel anomaly 222 (operation 594). The classification of the pixel anomaly 222 may be performed using the significance analysis module 216 of the software code 110 / 210 executed by the hardware processor 104 of the system 100 or the hardware processor 154 of the user system 150.

[0066] Continuing with flowchart 590, for each video segment including pixel anomalies 222, when the pixel anomaly 222 is classified as a harmless anomaly in operation 594, the pixel anomaly 222 is ignored (operation 595). As described above, not all pixel anomalies 222 in the images 160 / 260 / 360a-360c are worth the time and / or cost associated with their correction. Therefore, those pixel anomalies determined to be pixel anomalies 222 that do not meet the significance criteria received in operation 591 may be ignored without substantially detracting from the aesthetics of the video segment including the images 360a-360c. Operation 595 may be performed using the significance analysis module 216 of the software code 110 / 210 executed by the hardware processor 104 of the system 100 or the hardware processor 154 of the user system 150.

[0067] Continuing with the flowchart 590, for all video segments included in the images 160 / 260 / 360a-360c, all pixel anomalies 222 that are classified as significant anomalies 226 are marked (operation 596). As described above, some pixel anomalies 222 are significant enough (e.g., based on their location in the video segments included in the images 160 / 260 / 360a-360c or their proximity to a feature character or object depicted in the video segments included in the images 160 / 260 / 360a-360c) to be worth the time and / or cost associated with their correction. Therefore, those pixel anomalies 222 in the images determined to meet the significance criteria 146 / 246 may be marked and included in the report 170 / 270 / 370 outputted via the GUI 130 / 230 / 330. Operation 596 may be performed using reporting module 218 of software code 110 / 210 executed by hardware processor 104 of system 100 or hardware processor 154 of user system 150 .

[0068] Note that although the exemplary method outlined in flowchart 590 describes ignoring detected pixel anomalies 222 when the pixel anomalies 222 are classified as harmless in operation 595, all significant anomalies 226 are subsequently marked in operation 596, and this order is provided only as an example. In various embodiments of the present method, operations 595 and 596 can be performed substantially simultaneously, operation 596 can precede operation 595, or one of operations 595 or 596 can be omitted. For example, in the case where only significant anomalies 226 are detected, operation 595 can be omitted, and operation 596 can follow operation 594. Conversely, in the case where only harmless pixel anomalies are detected, operation 596 can be omitted, and flowchart 590 can end with operation 595.

[0069] As described above, in some embodiments, the flowchart 590 may end with operation 595 or operation 596 as described above. However, in other embodiments, the flowchart 590 may continue and end with optionally applying the corrections 148 / 248 to at least some of the significant anomalies 226 in the video segments included in the images 160 / 260 / 360a-360c (operation 597). For example, in one embodiment, the corrections 148 / 248 may be applied to at least some of the significant anomalies 226 in the video segments included in the images 160 / 260 / 360a-360c by software code 110 / 210 executed by the hardware processor 104 of the system 100 or the hardware processor 154 of the user system 150 to correct those significant anomalies 226 in an automated process. In other embodiments, the significant anomalies 226 may be marked to the user 144 to correct the significant anomalies 226. In those other embodiments, the software code 110 / 210 executed by the hardware processor 104 of the system 100 or by the hardware processor 154 of the user system 150 may receive the corrections 148 / 248 from the user 144. Furthermore, in those embodiments, the software code 110 / 210 executed by the hardware processor 104 of the system 100 or by the hardware processor 154 of the user system 150 may apply the corrections 148 / 248 to at least some of the notable anomalies 226 in the video segments included in the images 160 / 260 / 360a-360c in response to receiving the corrections 148 / 248 from the user 144.

[0070] Therefore, the present application discloses a system and method for performing an automated assessment of the significance of pixel anomalies that overcomes the shortcomings and deficiencies in conventional techniques. The automated solution for assessing the significance of pixel anomalies disclosed in the present application improves upon the prior art by being able to effectively identify harmless anomalies that can be ignored without significantly affecting the aesthetics of one or more images that include such harmless anomalies. Thus, the present solution advantageously enables pixel anomaly assessment and correction to be more efficient and less costly than conventional methods.

[0071] As can be seen from the above description, various techniques can be used to implement the concepts described in this application without departing from the scope of these concepts. In addition, although these concepts have been described with specific reference to certain embodiments, it will be appreciated by those of ordinary skill in the art that changes can be made in form and detail without departing from the scope of these concepts. Therefore, the described embodiments should be illustrative and non-restrictive in all respects. It should also be understood that the application is not limited to the specific embodiments described herein, and many rearrangements, modifications and replacements may be performed without departing from the scope of the present disclosure.

Claims

1. A system comprising: A computing platform, including a hardware processor and system memory storing software code; The hardware processor is configured to execute the software code to: Detect pixel anomalies in images; obtaining one or more saliency criteria for the image, wherein the one or more saliency criteria include a specified area of ​​the image, the specified area including at least one of a character in the image, a foreground of the image, or a background of the image; determining locations of pixel anomalies in the image; After detecting the pixel anomaly, classifying the pixel anomaly as a significant anomaly or a harmless anomaly based on the one or more significance criteria of the image, wherein the pixel anomaly is classified as a significant anomaly or a harmless anomaly based on whether the determined location of the pixel anomaly is in a specified area of ​​the image; When the pixel anomaly is classified as the harmless anomaly, ignoring the pixel anomaly; as well as When the pixel anomaly is classified as the significant anomaly, the pixel anomaly is marked.

2. The system according to claim 1, wherein: Marking the pixel anomaly includes marking the pixel anomaly to a user to correct the pixel anomaly classified as the significant anomaly.

3. The system according to claim 2, wherein: The hardware processor is further configured to execute the software code to receive a correction to the image from the user for correcting the pixel anomaly classified as the significant anomaly.

4. The system according to claim 3, wherein: The hardware processor is further configured to execute the software code to apply the correction to the image.

5. The system according to claim 2, wherein: The hardware processor is further configured to execute the software code to apply corrections to the image in response to flagging the pixel anomaly to correct the pixel anomaly classified as the significant anomaly.

6. The system according to claim 2, wherein: One or more saliency criteria for the image are obtained as input from the user.

7. The system according to claim 1, wherein: The one or more saliency criteria of the image are predetermined saliency criteria.

8. The system according to claim 1, wherein: The image is a video frame, and wherein one or more saliency criteria of the image are determined based on one of a shot of a video including the video frame or a scene of the video including the video frame.

9. A method for use by a system comprising a computing platform having a hardware processor and a system memory storing software code, comprising: detecting pixel anomalies in an image by the software code executed by the hardware processor; obtaining, by the software code executed by the hardware processor, one or more saliency criteria for the image, wherein the one or more saliency criteria include a designated area of ​​the image, the designated area including at least one of a character in the image, a foreground of the image, or a background of the image; determining, by the software code executed by the hardware processor, locations of pixel anomalies in the image; After detecting the pixel anomaly, the software code executed by the hardware processor classifies the pixel anomaly as one of a significant anomaly or a harmless anomaly based on one or more significance criteria of the image, wherein the pixel anomaly is classified as a significant anomaly or a harmless anomaly based on whether the determined location of the pixel anomaly is in a specified area of ​​the image; When the pixel anomaly is classified as the harmless anomaly, the software code executed by the hardware processor ignores the pixel anomaly; as well as The software code executed by the hardware processor marks the pixel anomaly when the pixel anomaly is classified as the significant anomaly.

10. The method according to claim 9, wherein: Marking the pixel anomaly includes marking the pixel anomaly to a user to correct the pixel anomaly classified as the significant anomaly.

11. The method according to claim 10, further comprising: A correction to the image is received from the user by the software code executed by the hardware processor for correcting the pixel anomaly classified as the significant anomaly.

12. The method according to claim 11, further comprising: The correction is applied to the image by the software code executed by the hardware processor.

13. The method according to claim 10, further comprising: In response to flagging the pixel anomaly, applying a correction to the image by the software code executed by the hardware processor to correct the pixel anomaly classified as the significant anomaly.

14. The method according to claim 10, wherein: One or more saliency criteria for the image are obtained as input from a user.

15. The method according to claim 9, wherein: The one or more saliency criteria of the image are predetermined saliency criteria.

16. The method according to claim 9, wherein: The image is a video frame, and wherein one or more saliency criteria of the image are determined based on one of a shot of a video including the video frame or a scene of the video including the video frame.

17. A method for use by a system comprising a computing platform having a hardware processor and a system memory storing software code, the software code providing a graphical user interface, the method comprising: detecting, by the software code executed by the hardware processor, a pixel anomaly in a first video segment of a plurality of video segments; When the first video clip is analyzed and the pixel anomaly in the first video clip is detected, displaying at least one video frame in the first video clip via a graphical user interface by the software code executed by the hardware processor; receiving, by the software code executed by the hardware processor, via a graphical user interface, one or more saliency criteria for the first video clip, wherein the one or more saliency criteria include a designated area of ​​the first video clip, the designated area including at least one of a character in the first video clip, a foreground of the first video clip, or a background of the first video clip; After detecting the pixel anomaly, determining a location of the pixel anomaly in the first video clip and classifying the pixel anomaly as a significant anomaly or a harmless anomaly based on one or more significance criteria of the first video clip by the software code executed by the hardware processor, wherein the pixel anomaly is classified as a significant anomaly or a harmless anomaly according to whether the determined location of the pixel anomaly is in a specified area of ​​the first video clip; and When the pixel anomaly is classified as the harmless anomaly, ignoring the pixel anomaly by the software code executed by the hardware processor; When the pixel anomaly is classified as the significant anomaly, the pixel anomaly is marked by the software code executed by the hardware processor.

18. The method according to claim 17, wherein: Marking the pixel anomaly includes marking the pixel anomaly to a user to correct the pixel anomaly classified as the significant anomaly.

19. The method according to claim 18, further comprising: A correction to the first video clip is received from the user, by the software code executed by the hardware processor, for correcting the pixel anomaly classified as the significant anomaly.

20. The method according to claim 19, further comprising: In response to receiving the correction to the first video segment from the user, the correction is applied to the first video segment by the software code executed by the hardware processor.

Images