A method and device for evaluating motion blur removal, an intelligent device and a storage medium

Abstract

This application relates to the field of image display technology, and more particularly to a method, apparatus, smart device, and storage medium for evaluating ghosting removal. The method includes: acquiring a target image, the target image being determined based on a current image and a first preset number of preceding images; acquiring ghosting features of a ghosting region in the target image; determining a first ghosting level based on the ghosting features of the ghosting region in the target image, the first ghosting level being the ghosting level of the target image; and performing a ghosting removal evaluation when the first ghosting level is greater than or equal to a first preset level. This method can effectively evaluate the degree of ghosting, avoid over-removal of ghosting, and thus improve the effectiveness of ghosting removal.

Description

Technical Field

[0001] This application relates to the field of image display technology, and in particular to a method, apparatus, smart device, and storage medium for evaluating ghosting removal. Background Technology

[0002] Due to limitations in light sensing and processing capabilities, cameras may exhibit ghosting or noisy trailing effects when objects move in low-light conditions due to delayed convergence, thus affecting users' evaluation of the camera's image quality.

[0003] Currently, research on motion blur mainly focuses on the detection and removal of motion blur in single images. Inappropriate timing or intensity of motion blur removal evaluation can lead to problems such as motion blur breakage and flickering that are clearly perceptible to the human eye, resulting in unsatisfactory evaluation results for motion blur removal.

[0004] Therefore, how to avoid over-removing ghosting and improve the effectiveness of ghosting removal is a problem that needs to be considered. Summary of the Invention

[0005] In view of this, embodiments of this application provide a method, apparatus, smart device, and storage medium for evaluating ghosting removal, which can effectively evaluate the degree of ghosting removal, avoid over-removal of ghosting, and thus improve the effectiveness of ghosting removal.

[0006] The first aspect of this application provides a method for evaluating ghosting removal, including:

[0007] Acquire a target image, which is determined based on the current image and a first preset number of preceding images;

[0008] Obtain the motion blur features of the motion blur region in the target image;

[0009] Based on the motion blur characteristics of the motion blur region in the target image, a first motion blur level is determined, and the first motion blur level is the motion blur level of the target image;

[0010] When the first ghosting level is greater than or equal to the first preset level, a ghosting removal evaluation is performed.

[0011] In one possible implementation of the first aspect, acquiring the target image includes:

[0012] Obtain an initial image sequence, the initial image sequence including the first preset number of preceding images;

[0013] Obtain a trailing region image cache sequence, wherein the trailing region image cache sequence includes the trailing region image of each image in the initial image sequence;

[0014] The target image is obtained by fusing the image cache sequence of the trailing region with the current image.

[0015] In one possible implementation of the first aspect, the trailing feature includes area ratio and trailing density;

[0016] The step of obtaining the motion blur features of the motion blur region in the target image includes:

[0017] According to a preset segmentation algorithm, the trailing region in the target image is divided into several non-overlapping sub-regions;

[0018] Obtain the total number of pixels in the target image, the number of pixels in the sub-region, and the number of pixels in the trailing image;

[0019] The area ratio of the sub-region is determined based on the number of pixels in the region and the total number of pixels in the image.

[0020] The ghosting density of the sub-region is determined based on the number of ghosting pixels and the number of pixels in the region.

[0021] In one possible implementation of the first aspect, determining the first motion blur level based on the motion blur features of the motion blur region in the target image includes:

[0022] The trailing features of the trailing region in the target image are input into a pre-trained first attention model to obtain a first fusion weight. The first attention model is used to dynamically determine the first fusion weight based on the spatial correlation between each trailing feature.

[0023] The first motion blur level is determined based on the motion blur features of the motion blur region in the target image and the first fusion weight.

[0024] In one possible implementation of the first aspect, the motion blur removal evaluation includes:

[0025] Obtain a second motion blur level, where the second motion blur level is the motion blur level of the target image sequence, and the target image sequence is an image sequence consisting of the current image and a second preset number of preceding images;

[0026] If the second ghosting level is less than or equal to the second preset level, then perform the ghosting removal operation.

[0027] In one possible implementation of the first aspect, obtaining the second motion blur level includes:

[0028] Based on the motion blur characteristics of the motion blur region in the second preset number of preceding images, determine the preceding motion blur level of each preceding image in the second preset number of preceding images;

[0029] Based on the first motion blur level and the preceding motion blur level, the initial motion blur level corresponding to the target image sequence is obtained;

[0030] The initial motion blur level is input into a pre-trained second attention model to obtain a second fusion weight. The second attention model is used to dynamically determine the second fusion weight based on the temporal correlation between each motion blur level in the initial motion blur level.

[0031] The second motion blur level is obtained based on the initial motion blur level and the second fusion weight.

[0032] In one possible implementation of the first aspect, after obtaining the initial motion blur level corresponding to the target image sequence based on the first motion blur level and the preceding motion blur level, the method includes:

[0033] When the first motion blur level is greater than the first preset level, the first motion blur level is updated according to the preset reference image;

[0034] Update the initial motion blur level based on the updated first motion blur level;

[0035] The step of inputting the initial motion blur level into the pre-trained second attention model includes:

[0036] The updated initial ghosting level is input into the second attention model to obtain the second fusion weight.

[0037] A second aspect of this application provides a ghosting removal evaluation apparatus, comprising:

[0038] A target image acquisition unit is used to acquire a target image, wherein the target image is determined based on the current image and a first preset number of preceding images;

[0039] The feature acquisition unit is used to acquire the motion blur features of the motion blur region in the target image;

[0040] A level determination unit is used to determine a first motion blur level based on the motion blur characteristics of the motion blur region in the target image, wherein the first motion blur level is the motion blur level of the target image;

[0041] The ghosting removal evaluation unit is used to perform ghosting removal evaluation when the first ghosting level is greater than or equal to the first preset level.

[0042] In one possible implementation of the second aspect, the target image acquisition unit includes:

[0043] An initial sequence acquisition module is used to acquire an initial image sequence, wherein the initial image sequence includes the first preset number of preceding images;

[0044] The cache sequence acquisition module is used to acquire a cache sequence of images of the trailing region, wherein the cache sequence of images of the trailing region includes the trailing region image of each image in the initial image sequence;

[0045] The target image acquisition module is used to perform image fusion between the image cache sequence of the trailing region and the current image to obtain the target image.

[0046] In one possible implementation of the second aspect, the trailing shadow feature includes area proportion and trailing shadow density, and the feature acquisition unit includes:

[0047] The region segmentation module is used to segment the trailing region in the target image into several non-overlapping sub-regions according to a preset segmentation algorithm;

[0048] The image information acquisition module is used to acquire the total number of pixels in the target image, the number of pixels in the sub-region, and the number of pixels in the trailing image.

[0049] The proportion determination module is used to determine the area proportion of the sub-region based on the number of pixels in the region and the total number of pixels in the image.

[0050] The density determination module is used to determine the motion density of the sub-region based on the number of motion pixels and the number of pixels in the region.

[0051] In one possible implementation of the second aspect, the level determination unit includes:

[0052] The first weight determination module is used to input the trailing features of the trailing region in the target image into the pre-trained first attention model to obtain the first fusion weight. The first attention model is used to dynamically determine the first fusion weight based on the spatial correlation between each trailing feature.

[0053] The first level determination module is used to determine the first motion blur level based on the motion blur features of the motion blur region in the target image and the first fusion weight.

[0054] In one possible implementation of the second aspect, the ghosting removal evaluation unit includes:

[0055] The sequence level determination module is used to obtain the second motion blur level, which is the motion blur level of the target image sequence, and the target image sequence is an image sequence composed of the current image and a second preset number of preceding images;

[0056] The ghosting removal module is used to perform ghosting removal operation if the second ghosting level is less than or equal to the second preset level.

[0057] In one possible implementation of the second aspect, the sequence rank determination module includes:

[0058] The preceding level determination submodule is used to determine the preceding motion blur level of each preceding image in the second preset number of preceding images based on the motion blur characteristics of the motion blur region in the second preset number of preceding images.

[0059] The initial level determination submodule is used to obtain the initial motion blur level corresponding to the target image sequence based on the first motion blur level and the preceding motion blur level;

[0060] The second weight determination submodule is used to input the initial motion blur level into the pre-trained second attention model to obtain the second fusion weight. The second attention model is used to dynamically determine the second fusion weight based on the temporal correlation between each motion blur level in the initial motion blur level.

[0061] The second-level determination submodule is used to obtain the second motion blur level based on the initial motion blur level and the second fusion weight.

[0062] In one possible implementation of the second aspect, the sequence level determination module further includes:

[0063] The image level update submodule is used to update the first motion blur level according to a preset reference image when the first motion blur level is greater than the first preset level.

[0064] The initial level update submodule is used to update the initial ghosting level based on the updated first ghosting level;

[0065] The second weight determination submodule is also used to input the updated initial ghosting level into the second attention model to obtain the second fusion weight.

[0066] A third aspect of this application provides an intelligent device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of a ghosting removal evaluation method as provided in the first aspect of this application.

[0067] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a ghosting removal evaluation method as provided in the first aspect of this application.

[0068] The fifth aspect of this application provides a computer program product that, when run on a terminal device, causes the terminal device to perform the steps of a ghosting removal evaluation method described in the first aspect of this application.

[0069] In this embodiment, a target image is acquired, determined based on the current image and a first preset number of preceding images. Then, the motion blur features of the motion blur region in the target image are acquired. Based on these features, a first motion blur level is determined. When the first motion blur level is greater than or equal to a first preset level, motion blur removal evaluation is performed. This solution determines whether to trigger motion blur evaluation based on the motion blur level of the image, and then determines whether to remove the motion blur based on the evaluation result. This avoids over-removal of motion blur leading to poor removal results and improves the effectiveness of motion blur removal. Attached Figure Description

[0070] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0071] Figure 1 This is a flowchart illustrating the implementation of a motion blur removal evaluation method provided in an embodiment of this application.

[0072] Figure 2 This is a flowchart illustrating a specific implementation of step S101 in a method for evaluating ghosting removal provided in this application embodiment;

[0073] Figure 3 This is a flowchart illustrating a specific implementation of step S102 in a method for evaluating ghosting removal provided in this application embodiment;

[0074] Figure 4 This is a flowchart illustrating a specific implementation of step S103 in a method for evaluating ghosting removal provided in this application embodiment;

[0075] Figure 5 This is a flowchart illustrating a specific implementation of a motion blur removal evaluation method provided in this application embodiment.

[0076] Figure 6 This is a flowchart illustrating a specific implementation of a method for determining a second motion blur level in a motion blur removal evaluation method provided in this application embodiment;

[0077] Figure 7This is a flowchart illustrating a specific implementation of updating the initial ghosting level in a ghosting removal evaluation method provided in this application embodiment;

[0078] Figure 8 This is a structural block diagram of a motion blur removal evaluation device provided in an embodiment of this application;

[0079] Figure 9 This is a schematic diagram of a smart device provided in an embodiment of this application. Detailed Implementation

[0080] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known apparatuses, systems, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0081] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0082] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0083] It should be understood that the various method embodiments of this application provide a method for evaluating ghosting removal that is applicable to various types of smart devices that require ghosting removal.

[0084] The following describes, with reference to specific embodiments, an exemplary method for evaluating ghosting removal provided in this application.

[0085] Figure 1 The implementation flow of a ghosting removal evaluation method provided by an embodiment of this application is illustrated. The execution end of this embodiment can be a smart device. The method flow may include the following steps S101 to S104.

[0086] Step S101: Obtain the target image, which is determined based on the current image and a first preset number of preceding images.

[0087] The current image mentioned above refers to the image at the current time. For example, if the current time is t, the current image is the image at time t.

[0088] In this embodiment, the target image is determined using the current image and a first preset number of preceding images, based on the trailing regions in the current image and the preceding images. The first preset number can be 10 images.

[0089] As one possible implementation of this application Figure 2 The following is a detailed implementation flow of the method for acquiring a target image provided in the embodiments of this application:

[0090] A1: Obtain an initial image sequence, the initial image sequence including the first preset number of preceding images.

[0091] In this embodiment, a first preset number of preceding images are buffered in a circular buffer. Generally, the human eye's visual persistence time is 0.4 seconds. Calculated at a standard frame rate of 25fps, when the human eye views the current image, the first 10 images will be temporarily retained in the brain. Therefore, the length of the circular buffer is set to 10, that is, the first preset number can be 10 images.

[0092] A2: Obtain the trailing region image cache sequence, wherein the trailing region image cache sequence includes the trailing region image of each image in the initial image sequence.

[0093] In this embodiment, the trailing region image in a single image is obtained through image segmentation. The image segmentation algorithm can employ existing techniques, such as publicly available CNN algorithms. The trailing region images of each image in the initial image sequence constitute a trailing region image cache sequence, stored using a circular cache, denoted as R. t =[I t-i I t-(i-1) …I t-1 ], where I t This represents the trailing region image of the image at the current time t, where i is the length of the circular storage; for example, i = 10, I t-i This represents the trailing region image of the i-th preceding image of the trailing region image at the current time t. If no trailing region was detected in the preceding image, the buffer corresponding to that preceding image is filled with all zeros. If the circular buffer is full when storing the trailing region image, the earliest cached trailing region image is overwritten.

[0094] A3: The image cache sequence of the trailing region is fused with the current image to obtain the target image.

[0095] In this embodiment, the trailing region images in the trailing region image cache sequence are fused with the current image to obtain the target image.

[0096] In one possible implementation, the cached sequence of the trailing region images and the current image are input into a pre-trained attention model to generate the target image. The attention model dynamically determines fusion weights based on the temporal and spatial correlations between the trailing region images and the current image. Then, based on these fusion weights, the trailing region images in the cached sequence are fused pixel-by-pixel according to coordinates with the current image to obtain the target image. The attention model supports spatiotemporal attention mechanisms (such as TimeSformer, ViViT, etc.).

[0097] In some implementations, the attention model described above preferentially uses the dot product model.

[0098] In this embodiment, the persistence of vision of the human eye is fully considered. Based on the spatiotemporal attention mechanism, the images of the trailing regions in the preceding images that are temporarily retained due to visual characteristics are fused together, which is closer to the human eye's perception of the trailing of the current image and helps to improve the accuracy and effectiveness of trailing assessment.

[0099] Step S102: Obtain the motion blur features of the motion blur region in the target image.

[0100] As one possible implementation of this application, the trailing shadow feature includes area ratio and trailing shadow density; Figure 3 The following is a detailed description of the specific implementation process for obtaining the motion blur features of the motion blur region in the target image, as illustrated in the method embodiment provided in this application:

[0101] B1: According to the preset segmentation algorithm, the trailing region in the target image is segmented into several non-overlapping sub-regions.

[0102] In this embodiment, a publicly available CNN algorithm is used to segment the trailing region from the target image, extract the trailing pixels in the trailing region, and then a preset segmentation algorithm is used to divide the trailing region into several non-overlapping sub-regions. In some embodiments, the preset segmentation algorithm can be a Euclidean distance algorithm, specifically: trailing pixels with an Euclidean distance less than a preset threshold are grouped into one sub-region.

[0103] B2: Obtain the total number of pixels in the target image, the number of pixels in the sub-region, and the number of pixels in the trailing image.

[0104] B3: Determine the area percentage of the sub-region based on the number of pixels in the region and the total number of pixels in the image. Specifically, the area percentage = number of pixels in the region / total number of pixels in the image.

[0105] B4: Determine the motion blur density of the sub-region based on the number of motion blur pixels and the number of pixels in the region. Specifically, motion blur density = number of motion blur pixels / number of pixels in the region.

[0106] In this embodiment, the trailing region in the target image is first divided into several sub-regions, and then the area ratio and trailing density of the sub-regions are calculated, which helps to improve the accuracy of determining the trailing features of the target image.

[0107] Step S103: Determine a first motion blur level based on the motion blur characteristics of the motion blur region in the target image. The first motion blur level is the motion blur level of the target image.

[0108] In this embodiment, the motion blur level setting can be adjusted according to actual business needs.

[0109] As one possible implementation of this application Figure 4 The following is a detailed implementation process for determining the first motion blur level based on the motion blur features of the motion blur region in the target image, as illustrated in the method embodiment provided in this application:

[0110] C1: The motion blur features of the motion blur region in the target image are input into a pre-trained first attention model to obtain a first fusion weight. The first attention model is used to dynamically determine the first fusion weight based on the spatial correlation between the motion blur features. The first attention model is a model that supports spatial attention mechanisms.

[0111] In some implementations, the attention scoring mechanism of the first attention model mentioned above preferentially selects the dot product model.

[0112] C2: Determine the first motion blur level based on the motion blur features of the motion blur region in the target image and the first fusion weight.

[0113] For example, publicly available techniques that support spatial attention mechanisms (such as ViT, CNN, etc.) are employed to obtain the motion blur features FI of the motion blur region in the target image. t As input to the model, the first fusion weight WS is dynamically determined by the model based on the spatial correlation between each pair of inputs. t According to SI t =WS t ×FI t The first ghosting level is obtained.

[0114] Step S104: When the first ghosting level is greater than or equal to the first preset level, perform ghosting removal evaluation.

[0115] In one possible implementation, the first motion blur level is determined by feature mapping of the current image's motion blur level. The mapping weights are obtained through fitting / training with a self-built dataset.

[0116] In some implementations, the first ghosting level range is 1 to 5. For example, these 5 levels are: 1 - barely noticeable; 2 - flaws can be found with careful examination; 3 - flaws can be easily observed; 4 - obvious problems can be easily observed; 5 - serious defects can be easily observed.

[0117] In this embodiment, when the first ghosting level is greater than or equal to the first preset level, a ghosting removal evaluation is triggered. The ghosting removal evaluation assesses the feasibility of removing ghosting from the current image. In some implementations, the first preset level is level 2. That is, when the first ghosting level is greater than or equal to level 2, a ghosting removal evaluation is performed to assess whether it is necessary to remove the ghosting.

[0118] As one possible implementation of this application, such as Figure 5 As shown, the motion blur removal evaluation specifically includes:

[0119] D1: Obtain the second motion blur level, which is the motion blur level of the target image sequence. The target image sequence is an image sequence consisting of the current image and a second preset number of preceding images. The second preset number is a specific length. The second preset number is at least 25 (calculated at a standard frame rate of 25fps, i.e., 1 second of continuous video), and the specific value of the second preset number can be adjusted according to memory resources.

[0120] In this embodiment, the second motion blur level is a set of motion blur levels for the current image and a second preset number of preceding images. The motion blur levels in this set are ordered sequentially according to the preceding images, which are ordered chronologically. Each motion blur level in the second set corresponds one-to-one with a preceding image. If no preceding image exists, the preceding motion blur level is set to 0.

[0121] As one possible implementation of this application, such as Figure 6 As shown, obtaining the second motion blur level specifically includes:

[0122] D31: Based on the motion blur characteristics of the motion blur region within the second preset number of preceding images, determine the preceding motion blur level of each preceding image in the second preset number of preceding images. The specific method for determining the preceding motion blur level based on the motion blur characteristics can refer to the aforementioned method for determining the first motion blur level, and will not be repeated here.

[0123] D32: Based on the first motion blur level and the preceding motion blur level, obtain the initial motion blur level corresponding to the target image sequence. Specifically, the initial motion blur level is the set of preceding motion blur levels corresponding to the first motion blur level and the second preset number of preceding images.

[0124] D33: The initial motion blur levels are input into a pre-trained second attention model to obtain second fusion weights. The second attention model dynamically determines the second fusion weights based on the temporal correlation between the motion blur levels in the initial motion blur levels. The second attention model is a model that supports a temporal attention mechanism.

[0125] D34: The second motion blur level is obtained based on the initial motion blur level and the second fusion weight.

[0126] For example, publicly available techniques that support temporal attention mechanisms (such as ViT, RNN, etc.) are employed to determine the initial motion blur level SI. t As input to the model, the second fusion weight WS' is dynamically determined by the model based on the temporal correlation between each pair of inputs. t According to SV t =WS't×SI t The first ghosting level is obtained.

[0127] As one possible implementation of this application, such as Figure 7 As shown, after obtaining the initial motion blur level corresponding to the target image sequence based on the first motion blur level and the preceding motion blur level, the process includes:

[0128] E1: When the first motion blur level is greater than the first preset level, update the first motion blur level according to the preset reference image.

[0129] The preset reference image is a reference image without motion blur. When the first motion blur level or the motion blur level of the preceding image is greater than the first preset level, it means that motion blur removal is preferred. In this embodiment, when the first motion blur level is greater than the first preset level, the current image after motion blur removal is obtained using the reference image without motion blur. A new first motion blur level is obtained based on the motion blur characteristics of the current image after motion blur removal, or a new preceding motion blur level is obtained based on the motion blur characteristics of the preceding image after motion blur removal.

[0130] E2: Update the initial ghosting level based on the updated first ghosting level. That is, update the first ghosting level or the preceding ghosting level in the initial ghosting level.

[0131] In this embodiment, the current image after removing the motion blur is obtained using the reference image without motion blur, and the effect after removing the motion blur is used as a reference to determine whether it is necessary to remove the motion blur. This is closer to the overall perception of the video by the human eye, thereby improving the effectiveness of motion blur removal.

[0132] In this embodiment, the updated initial ghosting level is input into the second attention model to obtain the second fusion weight. The specific method for determining the second fusion weight is as described above and will not be repeated here.

[0133] D2: If the second ghosting level is less than or equal to the second preset level, then perform the ghosting removal operation.

[0134] In this embodiment, when the second ghosting level is less than or equal to the second preset level, it is determined that a ghosting removal operation will be performed. When the second ghosting level is greater than the second preset level, it is determined that a ghosting removal operation will not be performed.

[0135] In one possible implementation, the second motion blur level is determined by mapping the motion blur level of the target image sequence. The mapping weights are obtained by fitting / training a self-built dataset.

[0136] In some implementations, the second ghosting level range is 1 to 5. For example, these 5 levels are: 1 - content is fully preserved and realistic; 2 - content loss is slight but realistic; 3 - content loss is significant but realistic; 4 - content loss is slight but unrealistic; 5 - content loss is significant and unrealistic.

[0137] For example, when the first motion blur level of the target image is greater than 2, a motion blur removal evaluation is triggered. In the motion blur removal evaluation, if the motion blur level of the target image sequence is less than 4, it is considered that the video viewing experience after motion blur removal is acceptable to the human eye, and the motion blur removal operation is determined to be performed. If the motion blur level of the target image sequence is greater than or equal to 4, it means that the video viewing experience after motion blur removal is unacceptable to the human eye, and the motion blur removal operation is determined not to be performed.

[0138] In this embodiment, a target image is acquired, determined based on the current image and a first preset number of preceding images. Then, the motion blur features of the motion blur region in the target image are acquired. Based on these features, a first motion blur level is determined. When the first motion blur level is greater than or equal to a first preset level, motion blur removal evaluation is performed. This solution determines whether to trigger motion blur evaluation based on the motion blur level of the image, and then determines whether to remove the motion blur based on the evaluation result. This avoids over-removal of motion blur leading to poor removal results and improves the effectiveness of motion blur removal.

[0139] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0140] Corresponding to the ghosting removal evaluation method described in the above embodiments, Figure 8 The diagram shows a structural block diagram of a motion blur removal evaluation device provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiment of this application are shown.

[0141] Reference Figure 8 The motion blur removal evaluation device is applied to a smart device. The device includes: a target image acquisition unit 81, a feature acquisition unit 82, a grade determination unit 83, and a motion blur removal evaluation unit 84, wherein:

[0142] The target image acquisition unit 81 is used to acquire a target image, which is determined based on the current image and a first preset number of preceding images.

[0143] Feature acquisition unit 82 is used to acquire the motion blur features of the motion blur region in the target image;

[0144] The grade determination unit 83 is used to determine a first motion blur grade based on the motion blur characteristics of the motion blur region in the target image, wherein the first motion blur grade is the motion blur grade of the target image;

[0145] The ghosting removal evaluation unit 84 is used to perform ghosting removal evaluation when the first ghosting level is greater than or equal to the first preset level.

[0146] As one possible implementation of this application, the target image acquisition unit 81 includes:

[0147] An initial sequence acquisition module is used to acquire an initial image sequence, wherein the initial image sequence includes the first preset number of preceding images;

[0148] The cache sequence acquisition module is used to acquire a cache sequence of images of the trailing region, wherein the cache sequence of images of the trailing region includes the trailing region image of each image in the initial image sequence;

[0149] The target image acquisition module is used to perform image fusion between the image cache sequence of the trailing region and the current image to obtain the target image.

[0150] As one possible implementation of this application, the above-mentioned trailing shadow features include area ratio and trailing shadow density, and the feature acquisition unit 82 includes:

[0151] The region segmentation module is used to segment the trailing region in the target image into several non-overlapping sub-regions according to a preset segmentation algorithm;

[0152] The image information acquisition module is used to acquire the total number of pixels in the target image, the number of pixels in the sub-region, and the number of pixels in the trailing shadow.

[0153] The proportion determination module is used to determine the area proportion of the sub-region based on the number of pixels in the region and the total number of pixels in the image.

[0154] The density determination module is used to determine the motion density of the sub-region based on the number of motion pixels and the number of pixels in the region.

[0155] As one possible implementation of this application, the grade determination unit 83 includes:

[0156] The first weight determination module is used to input the trailing features of the trailing region in the target image into the pre-trained first attention model to obtain the first fusion weight. The first attention model is used to dynamically determine the first fusion weight based on the spatial correlation between each trailing feature.

[0157] The first level determination module is used to determine the first motion blur level based on the motion blur features of the motion blur region in the target image and the first fusion weight.

[0158] As one possible implementation of this application, the above-mentioned ghosting removal evaluation unit 84 includes:

[0159] The sequence level determination module is used to obtain the second motion blur level, which is the motion blur level of the target image sequence, and the target image sequence is an image sequence composed of the current image and a second preset number of preceding images;

[0160] The ghosting removal module is used to perform ghosting removal operation if the second ghosting level is less than or equal to the second preset level.

[0161] As one possible implementation of this application, the sequence level determination module includes:

[0162] The preceding level determination submodule is used to determine the preceding motion blur level of each preceding image in the second preset number of preceding images based on the motion blur characteristics of the motion blur region in the second preset number of preceding images.

[0163] The initial level determination submodule is used to obtain the initial motion blur level corresponding to the target image sequence based on the first motion blur level and the preceding motion blur level;

[0164] The second weight determination submodule is used to input the initial motion blur level into the pre-trained second attention model to obtain the second fusion weight. The second attention model is used to dynamically determine the second fusion weight based on the temporal correlation between each motion blur level in the initial motion blur level.

[0165] The second-level determination submodule is used to obtain the second motion blur level based on the initial motion blur level and the second fusion weight.

[0166] As one possible implementation of this application, the sequence level determination module further includes:

[0167] The image level update submodule is used to update the first motion blur level according to a preset reference image when the first motion blur level is greater than the first preset level.

[0168] The initial level update submodule is used to update the initial ghosting level based on the updated first ghosting level;

[0169] The aforementioned second weight determination submodule is also used to input the updated initial ghosting level into the second attention model to obtain the second fusion weight.

[0170] In this embodiment, a target image is acquired, determined based on the current image and a first preset number of preceding images. Then, the motion blur features of the motion blur region in the target image are acquired. Based on these features, a first motion blur level is determined. When the first motion blur level is greater than or equal to a first preset level, motion blur removal evaluation is performed. This solution determines whether to trigger motion blur evaluation based on the motion blur level of the image, and then determines whether to remove the motion blur based on the evaluation result. This avoids over-removal of motion blur leading to poor removal results and improves the effectiveness of motion blur removal.

[0171] This application embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements... Figures 1 to 7 The steps of any ghosting removal evaluation method are represented.

[0172] This application embodiment also provides a smart device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements... Figures 1 to 7 The steps of any ghosting removal evaluation method are represented.

[0173] This application also provides a computer program product that, when run on a terminal device, causes the terminal device to execute the implementation of... Figures 1 to 7The steps of any ghosting removal evaluation method are represented.

[0174] Figure 9 This is a schematic diagram of a smart device provided in an embodiment of this application. Figure 9 As shown, the intelligent device 9 of this embodiment includes: a processor 90, a memory 91, and a computer program 92 stored in the memory 91 and executable on the processor 90. When the processor 90 executes the computer program 92, it implements the steps of each of the embodiments of the above-described ghosting removal evaluation method, for example... Figure 1 Steps S101 to S104 are shown. Alternatively, when the processor 90 executes the computer program 92, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 8 The functions of units 81 to 84 are shown.

[0175] The computer program 92 can be divided into one or more modules / units, which are stored in the memory 91 and executed by the processor 90 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 92 in the smart device 9.

[0176] The processor 90 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0177] The memory 91 can be an internal storage unit of the smart device 9, such as a hard drive or memory of the smart device 9. The memory 91 can also be an external storage device of the smart device 9, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the smart device 9. Furthermore, the memory 91 can include both internal and external storage units of the smart device 9. The memory 91 is used to store the computer program and other programs and data required by the smart device. The memory 91 can also be used to temporarily store data that has been output or will be output.

[0178] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0179] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0180] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0181] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0182] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0183] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0184] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0185] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0186] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for evaluating ghosting removal, characterized in that, include: Acquire a target image, which is determined based on the current image and a first preset number of preceding images; Obtain the motion blur features of the motion blur region in the target image; Based on the motion blur characteristics of the motion blur region in the target image, a first motion blur level is determined, and the first motion blur level is the motion blur level of the target image; When the first ghosting level is greater than or equal to the first preset level, a ghosting removal evaluation is performed. The ghosting removal evaluation includes: obtaining a second ghosting level, where the second ghosting level is the ghosting level of the target image sequence, and the target image sequence is an image sequence consisting of the current image and a second preset number of preceding images; if the second ghosting level is less than or equal to the second preset level, a ghosting removal operation is performed.

2. The method as described in claim 1, characterized in that, The acquisition of the target image includes: Obtain an initial image sequence, the initial image sequence including the first preset number of preceding images; Obtain a trailing region image cache sequence, wherein the trailing region image cache sequence includes the trailing region image of each image in the initial image sequence; The target image is obtained by fusing the image cache sequence of the trailing region with the current image.

3. The method as described in claim 1, characterized in that, The trailing characteristics include area ratio and trailing density; The step of obtaining the motion blur features of the motion blur region in the target image includes: According to a preset segmentation algorithm, the trailing region in the target image is divided into several non-overlapping sub-regions; Obtain the total number of pixels in the target image, the number of pixels in the sub-region, and the number of pixels in the trailing image; The area ratio of the sub-region is determined based on the number of pixels in the region and the total number of pixels in the image. The ghosting density of the sub-region is determined based on the number of ghosting pixels and the number of pixels in the region.

4. The method as described in claim 1, characterized in that, The step of determining the first motion blur level based on the motion blur characteristics of the motion blur region in the target image includes: The trailing features of the trailing region in the target image are input into a pre-trained first attention model to obtain a first fusion weight. The first attention model is used to dynamically determine the first fusion weight based on the spatial correlation between each trailing feature. The first motion blur level is determined based on the motion blur features of the motion blur region in the target image and the first fusion weight.

5. The method as described in claim 1, characterized in that, The acquisition of the second motion blur level includes: Based on the motion blur characteristics of the motion blur region in the second preset number of preceding images, determine the preceding motion blur level of each preceding image in the second preset number of preceding images; Based on the first motion blur level and the preceding motion blur level, the initial motion blur level corresponding to the target image sequence is obtained; The initial motion blur level is input into a pre-trained second attention model to obtain a second fusion weight. The second attention model is used to dynamically determine the second fusion weight based on the temporal correlation between each motion blur level in the initial motion blur level. The second motion blur level is obtained based on the initial motion blur level and the second fusion weight.

6. The method as described in claim 5, characterized in that, After obtaining the initial motion blur level corresponding to the target image sequence based on the first motion blur level and the preceding motion blur level, the process includes: When the first motion blur level is greater than the first preset level, the first motion blur level is updated according to the preset reference image; The initial motion blur level is updated based on the updated first motion blur level; The step of inputting the initial motion blur level into the pre-trained second attention model includes: The updated initial ghosting level is input into the second attention model to obtain the second fusion weight.

7. A motion blur removal evaluation device, characterized in that, include: A target image acquisition unit is used to acquire a target image, wherein the target image is determined based on the current image and a first preset number of preceding images; The feature acquisition unit is used to acquire the motion blur features of the motion blur region in the target image; A level determination unit is used to determine a first motion blur level based on the motion blur characteristics of the motion blur region in the target image, wherein the first motion blur level is the motion blur level of the target image; A ghosting removal evaluation unit is used to perform ghosting removal evaluation when the first ghosting level is greater than or equal to a first preset level. The ghosting removal evaluation unit includes: The sequence level determination module is used to obtain the second motion blur level, which is the motion blur level of the target image sequence, and the target image sequence is an image sequence composed of the current image and a second preset number of preceding images; The ghosting removal module is used to perform ghosting removal operation if the second ghosting level is less than or equal to the second preset level.

8. A smart device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of a ghosting removal evaluation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of a ghosting removal evaluation method as described in any one of claims 1 to 6.

Images