Image processing method and apparatus, and device and medium

Abstract

The embodiments of the present disclosure relate to an image processing method and apparatus, and a device and a medium. The method comprises: acquiring a plurality of image frames obtained by photographing a target object, and determining a current target image frame to be processed from among the plurality of image frames; determining from among the plurality of image frames a current reference image frame corresponding to the target image frame; when the current reference image frame is not the target image frame, acquiring a pixel difference value between a target object area in the target image frame and a target object area in the current reference image frame, and acquiring a current noise threshold value corresponding to both the target image frame and the current reference image frame; and on the basis of the pixel difference value and the current noise threshold value, performing fusion processing on the target object area in the target image frame and the target object area in the current reference image frame, so as to obtain a fused image frame, wherein the fused image frame serves as an image used for a target object processing task.

Description

Image processing method, device, equipment and medium

[0001] This application claims priority to the Chinese invention patent application entitled “Image processing method, device, equipment and medium” and application number 202311771865.3, filed on December 21, 2023. The entire contents of that application are incorporated by reference into this application. Technical Field

[0002] The present disclosure relates to the field of computer technology, and in particular to an image processing method, apparatus, device, and medium. Background Art

[0003] In some image processing scenarios, it is necessary to capture multiple frames of images of a target object and perform tasks related to the target object based on the multiple frames of images. The inventors have found through research that image noise can have a significant impact on the performance of task execution. Taking the user's eyes as an example, when the user is gazing at a certain point, multiple images of the user's eyes can be captured to estimate the direction of sight. However, even if the positions of the camera, gaze point, eyes, etc. have not changed, under the influence of image noise, the sight direction estimated by the sight estimation model for multiple eye images will also have a certain degree of jitter, and the stability of sight estimation is poor. Some related technologies will adopt a fuzzy smoothing processing method to reduce noise for each frame of the image, but it is easy to cause problems such as reduced accuracy, and still cannot guarantee a good task execution effect. Therefore, there is an urgent need for an image processing method that can still better guarantee the performance of tasks in the presence of image noise. Summary of the Invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present disclosure provides an image processing method, apparatus, device and medium.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, the method comprising: acquiring a plurality of frames of images obtained by shooting a target object, and determining a target frame image currently to be processed in the plurality of frames; determining a current reference frame image corresponding to the target frame image from the plurality of frames; wherein the current reference frame image is a frame image shot earlier than the target frame image, or the current reference frame image is the target frame image; when the current reference frame image is not the target frame image, acquiring a pixel difference between a target object area of ​​the target frame image and a target object area of ​​the current reference frame image, and acquiring a current noise threshold corresponding to the target frame image and the current reference frame image; performing a fusion process on the target object area of ​​the target frame image and the target object area of ​​the current reference frame image according to the pixel difference and the current noise threshold to obtain a fused frame image; wherein the fused frame image is used as a frame image adopted for a target object processing task.

[0006] An embodiment of the present disclosure also provides an image processing device, comprising: a target frame image determination module, configured to acquire a plurality of frame images obtained by photographing a target object, and determine a target frame image currently to be processed in the plurality of frame images; a reference frame image determination module, configured to determine a current reference frame image corresponding to the target frame image from the plurality of frame images; wherein the current reference frame image is a frame image photographed earlier than the target frame image, or the current reference frame image is the target frame image; an acquisition module, configured to acquire, when the current reference frame image is not the target frame image, a pixel difference between a target object region of the target frame image and a target object region of the current reference frame image, and to acquire a current noise threshold corresponding to the target frame image and the current reference frame image; a fusion module, configured to fuse the target object region of the target frame image with the target object region of the current reference frame image according to the pixel difference and the current noise threshold, to obtain a fused frame image; wherein the fused frame image is used as the frame image adopted for the target object processing task.

[0007] An embodiment of the present disclosure also provides an electronic device, which includes: a processor; a memory for storing executable instructions of the processor; the processor is used to read the executable instructions from the memory and execute the instructions to implement the image processing method provided by the embodiment of the present disclosure.

[0008] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the storage medium stores a computer program, and the computer program is used to execute the image processing method provided by the embodiment of the present disclosure.

[0009] The embodiments of the present disclosure further provide a computer program product, comprising computer program instructions, which implement the image processing method provided in the embodiments of the present disclosure when the computer program instructions are executed by a processor.

[0010] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0012] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0013] FIG1 is a schematic diagram of a flow chart of an image processing method provided by an embodiment of the present disclosure;

[0014] FIG2 is a schematic flow chart of an image processing method provided by an embodiment of the present disclosure;

[0015] FIG3 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure;

[0016] FIG4 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0017] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.

[0018] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0019] Image noise is usually unavoidable, and the noise of different images also varies to a certain extent, which will affect the stability of the target object processing task. In order to improve stability, some related technologies will perform pre-processing operations such as Gaussian blur smoothing on each frame of the collected image, but this has a greater impact on the image accuracy, and further affects the subsequent task execution effect, such as affecting the accuracy of line of sight estimation. In addition, the amount of computation required for blur smoothing is large and the cost is high. In addition, for the line of sight estimation task, some related technologies adopt smoothing methods such as Kalman filtering and angular velocity smoothing for the line of sight estimation results output by the line of sight estimation model for multiple eye images, but there is still the problem of poor task execution effect. The main reason is that the smoothing coefficient needs to be manually adjusted, which has poor flexibility, and the line of sight estimation results of adjacent eye images will also interfere with each other, resulting in the final line of sight estimation result being inaccurate. In order to better ensure the task execution effect in the presence of image noise, the embodiments of the present disclosure provide an image processing method, device, equipment and medium.

[0020] The above technical solution provided by the embodiment of the present disclosure is capable of determining the current reference frame image corresponding to the target frame image currently to be processed from the multiple frames of images obtained by shooting the target object, and when the current reference frame image is not the target frame image, the target object area of ​​the target frame image and the target object area of ​​the current reference frame image are fused based on the pixel difference between the target object area of ​​the target frame image and the target object area of ​​the current reference frame image and the current noise threshold, thereby obtaining a fused frame image that can be used for the target object processing task. The above method fully considers the impact of image noise on the target object processing task, and therefore the pixel difference and noise threshold of the target object area corresponding to the target frame image and the current reference frame image are comprehensively considered for fusion processing.

[0021] The following is a detailed explanation.

[0022] Figure 1 is a flow chart of an image processing method provided by an embodiment of the present disclosure. The method can be performed by an image processing device, wherein the device can be implemented using software and / or hardware and can generally be integrated into an electronic device. As shown in Figure 1, the method mainly includes the following steps S102 to S108:

[0023] Step S102 : acquiring multiple frames of images obtained by photographing the target object, and determining a target frame image to be processed currently in the multiple frames of images.

[0024] The embodiments of the present disclosure do not limit the type of target object. For example, the target object can be a person, an animal, a vehicle, a building, or a designated object. The target object can also be a designated part of a person, such as a person's face or eyes, or a vehicle's license plate, etc., and can be flexibly set according to specific needs. In some specific implementation examples, the target object is an eye, such as a user's eyes. In actual applications, the user can wear electronic devices such as VR glasses, smart glasses, eye trackers, or other electronic devices that need to estimate the direction of the user's eye line of sight, and the camera set up in the electronic device can capture the user's eyes to obtain multiple frames of images. It should be noted that before collecting the user's eye image, a collection prompt message can be sent to the user, and the eye image can be collected after obtaining the user's authorization. The user can also be informed of the purpose of the image collection in the prompt message. In addition, the embodiments of the present disclosure do not limit the specific number of multiple frames of images obtained, and can be two or more. The specific number can be flexibly set according to actual conditions.

[0025] In some implementations, each frame of the multiple frames of images may be sequentially used as a target frame image to be currently processed from the front to the back based on the shooting time.

[0026] Step S104 , determining a current reference frame image corresponding to the target frame image from the multiple frame images; wherein the current reference frame image is a frame image shot earlier than the target frame image, or the current reference frame image is the target frame image.

[0027] In the embodiment of the present disclosure, a reference frame image is additionally provided, and the reference frame image is used to perform pixel comparison with the target frame image so as to minimize the noise difference between different images based on the pixel difference. It should be noted that the reference frame image is selected from multiple frames of images and can be changed, and each frame image in the multiple frames of images corresponds to a reference frame image. For each frame image, the corresponding reference frame image may be a frame image taken before the frame image, or the frame image itself may also serve as the reference frame image. The reference frame images corresponding to different frame images may be the same or different.

[0028] Step S106 , when the current reference frame image is not the target frame image, obtain the pixel difference between the target object area of ​​the target frame image and the target object area of ​​the current reference frame image, and obtain the current noise threshold corresponding to the target frame image and the current reference frame image.

[0029] The purpose of acquiring multiple frames of images in the embodiment of the present disclosure is to perform target object processing tasks. Taking into account the differences in noise between images, the embodiment of the present disclosure can obtain the pixel difference between the target object area of ​​the target frame image and the target object area of ​​the current reference frame image to reflect the degree of difference between the target object area of ​​the target frame image and the target object area of ​​the current reference frame image. And by combining with the current noise threshold, it helps to further determine whether the degree of difference between the target object area of ​​the target frame image and the target object area of ​​the current reference frame image is mainly due to noise difference or actual content difference, so as to carry out subsequent processing in a targeted manner. It should be noted that the current noise threshold in the embodiment of the present disclosure is not fixed. When the target frame image and the current reference frame image are different, the corresponding noise threshold may also change.

[0030] In step S108, the target object region of the target frame image is fused with the target object region of the current reference frame image based on the pixel difference and the current noise threshold to obtain a fused frame image. The fused frame image is used as the image for the target object processing task, that is, the fused frame image can replace the target frame image in the target object processing task.

[0031] As mentioned above, the pixel difference and the current noise threshold value help to further determine whether the degree of difference between the target object areas of the target frame image and the current reference frame image is mainly a noise difference or a substantial content difference, so that fusion processing can be performed based on the category to which the degree of difference belongs. In some specific implementation examples, the pixel differences mentioned above are all analyzed in units of pixels. For example, for a pixel located at the same position in the target frame image and the current reference frame image, if the pixel difference corresponding to the pixel is a noise difference, then the pixel value of the pixel in the fused frame image is the pixel value of the current reference frame image. If the pixel difference corresponding to the pixel is a substantial content difference, then the pixel value of the pixel in the fused frame image is the pixel value of the target frame image. In the above manner, when there is a noise difference between the target frame image and the current reference frame image, the fused frame image can be made to be based on the noise of the reference frame image as much as possible, that is, the noise of the reference frame image is fixed, thereby reducing the noise difference between different images and improving the stability of the target object processing task.

[0032] The embodiments of the present disclosure do not limit the target object processing task. In some specific implementation examples, the target object includes an eye, and the target object processing task includes a line of sight estimation task.

[0033] Compared to related technologies, the disclosed embodiments utilize this approach to integrate the pixel differences between the target object regions of the processed image and the reference frame image, as well as the noise threshold. This allows the resulting fused frame image to better mitigate the impact of noise differences between different images on the target object processing task. Furthermore, this approach has minimal impact on image accuracy, ensuring effective task execution even in the presence of image noise.

[0034] The aforementioned step S104, i.e., determining the current reference frame image corresponding to the target frame image from the multiple frame images, can be performed with reference to the following cases 1 and 2:

[0035] In case 1, when the target frame image is the first frame image among multiple frame images, the current reference frame image corresponding to the target frame image is determined as the target frame image, that is, the target frame image itself is used as the current reference frame image.

[0036] In the second scenario, when the target frame image is not the first frame image among multiple frames, the reference frame image corresponding to the previous frame image of the target frame image is obtained, and based on the reference frame image corresponding to the previous frame image and the target frame image, the current reference frame image corresponding to the target frame image is determined. In actual applications, the current reference frame image corresponding to the target frame image may be consistent with the reference frame image corresponding to the previous frame image, or there may be certain differences between the target frame image and the previous frame image. Therefore, the target frame image is directly used as the current reference frame image. In this case, it can be considered that the reference frame image has been updated, and the current reference frame image corresponding to the target frame image is the target frame image itself.

[0037] In the disclosed embodiment, a target object detection frame corresponding to the target frame image may be further obtained, and a current reference object detection frame corresponding to the target frame image may be determined based on the target object detection frame corresponding to the target frame image. The current reference object detection frame may be a target object detection frame of a frame image captured earlier than the target frame image, or the current reference object detection frame may be the target object detection frame of the target frame image. The target object detection frame may be obtained using an object detection algorithm, which will not be further described herein.

[0038] The primary purpose of setting a reference object detection frame in the disclosed embodiments is to further reduce the differences between different images. Specifically, the reference object detection frame is used to determine the target object region of the target frame image. In the disclosed embodiments, the target object region of the target frame image is not directly determined using the target object detection frame of the target frame image. Instead, a reference object detection frame is set. The target object region of the target frame image is determined based on the current reference object detection frame corresponding to the target frame image. The reference object detection frames corresponding to different frames may be the same or different. Furthermore, the reference object detection frame is not fixed but may be updated based on the degree of image difference. It is understood that if the difference between two frames is not significant, the pixel difference between the target object regions determined using the same reference object detection frame is typically also small, thereby effectively ensuring the stability of the target object processing task. However, if each frame image uses its own target object detection frame to determine the target object region, there will typically be a certain degree of deviation between the two target object detection frames. Even a deviation of only one pixel will ultimately result in an overall pixel difference between the target object regions determined by each, thus affecting the stability of the target object processing task.

[0039] In some specific implementation examples, the following cases (1) and (2) may be referred to for execution:

[0040] Case (1): when the target frame image is the first frame image in the multi-frame image, the current reference object detection frame corresponding to the target frame image is determined to be the target object detection frame corresponding to the target frame image.

[0041] In case (2), when the target frame image is not the first frame image in the multi-frame image, the reference object detection frame corresponding to the previous frame image of the target frame image is obtained, and the current reference object detection frame corresponding to the target frame image is determined based on the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image. The current reference object detection frame corresponding to the target frame image may still be the reference object detection frame corresponding to the previous frame image, or it may be the target object detection frame corresponding to the target frame image itself. The specific details mainly depend on the degree of similarity between the target object detection frame corresponding to the target frame image and the reference object detection frame corresponding to the previous frame image, so as to determine whether the current reference object detection frame needs to be updated.

[0042] In some specific implementation examples, determining the current reference object detection frame corresponding to the target frame image based on the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image may be performed with reference to the following steps 1, 2a, or 2b:

[0043] Step 1: Obtain the similarity between the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image, as well as a preset similarity threshold.

[0044] Exemplarily, the degree of similarity is characterized by the overlap rate between the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image. The overlap rate can be determined using the intersection-over-union algorithm, and the similarity threshold can be flexibly set according to needs. For example, the degree of similarity is the overlap rate, and the similarity threshold is 0.85.

[0045] In step 2a, if the degree of similarity is not less than the similarity threshold, the current reference object detection frame corresponding to the target frame image is determined to be the reference object detection frame corresponding to the previous frame image. In other words, if the degree of similarity is high, the current reference object detection frame corresponding to the target frame image remains unchanged from the reference object detection frame corresponding to the previous frame image.

[0046] In step 2b, if the degree of similarity is less than the similarity threshold, the current reference object detection frame corresponding to the target frame image is determined to be the target object detection frame corresponding to the target frame image. In other words, if the degree of similarity is low, it indicates that there are certain substantive differences between the target frame image and the previous frame image. Therefore, the target object detection frame corresponding to the target frame image is directly updated to be the current reference object detection frame.

[0047] Based on the above, in the second scenario, when determining the current reference frame image corresponding to the target frame image based on the reference frame image and the target frame image corresponding to the previous frame image, the current reference frame image corresponding to the target frame image can be determined from the reference frame image and the target frame image corresponding to the previous frame image based on the target object detection frame and the current reference object detection frame. In other words, whether the reference frame image needs to be updated can be determined based on the update of the reference object detection frame. In some specific implementation examples, this can be achieved by referring to the following scenarios a and b:

[0048] In case a, when the current reference object detection frame corresponding to the target frame image is the target object detection frame of the target frame image, the current reference frame image corresponding to the target frame image is determined to be the target frame image. It will be understood that if the target object detection frame of the target frame image itself serves as the current reference object detection frame, this indicates that there is a significant difference between the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image. Therefore, the reference object detection frame has been updated. In this case, the reference frame image also needs to be updated accordingly.

[0049] In case b, when the current reference object detection frame corresponding to the target frame image is not the target object detection frame of the target frame image, a target difference between the reference frame image corresponding to the previous frame image and the target frame image is obtained. Based on the target difference and a preset difference threshold, the current reference frame image corresponding to the target frame image is determined from the reference frame image corresponding to the previous frame image and the target frame image. It is understood that although the current reference object detection frame corresponding to the target frame image has not yet reached the point of being updated, there may still be a certain degree of difference between the reference frame image corresponding to the previous frame image and the target frame image. Therefore, the disclosed embodiment further obtains a target difference between the two and combines this with a difference threshold to determine whether the reference frame image needs to be updated. Exemplarily, the target difference is represented by the mean pixel difference between the reference frame image corresponding to the previous frame image and the target frame image. That is, the pixel difference of each pixel is first calculated, and then the pixel differences of each pixel are averaged to obtain the mean pixel difference. The difference threshold can be flexibly set according to needs, for example, it can be set to 8.

[0050] For ease of understanding, in some examples of the step of determining the current reference frame image corresponding to the target frame image from the reference frame image corresponding to the previous frame image and the target frame image based on the target difference and a preset difference threshold in case b, if the target difference is not greater than the preset difference threshold, the current reference frame image corresponding to the target frame image can be determined to be the reference frame image corresponding to the previous frame image; if the target difference is greater than the preset difference threshold, the current reference frame image corresponding to the target frame image can be determined to be the target frame image. In other words, if a significant difference still exists between the reference frame image corresponding to the previous frame image and the target frame image based on the difference threshold, the target frame image is used as the current reference frame image, i.e., the reference frame image is updated. If the difference is not significant, the original reference frame image is retained.

[0051] In some implementation examples of obtaining the current noise threshold corresponding to the target frame image and the current reference frame image in step S106, the maximum value of the pixel difference between the background area of ​​the target frame image and the background area of ​​the current reference frame image can be obtained, and the current noise threshold corresponding to the target frame image and the current reference frame image can be determined based on this maximum value. The background area of ​​the target frame image is the area outside the edge contour of the target object in the target frame image, and the background area of ​​the current reference frame image is the area outside the edge contour of the target object in the current reference frame image. The edge contour can be determined based on the key points of the target object's contour. For details, please refer to related art and are not limited here. It is understood that, in general, the background areas in different images do not vary much, and the main differences come from differences in the objects. For example, changes in the position or movement of the target object can cause significant differences in the corresponding pixels. Therefore, in embodiments of the present disclosure, the noise threshold can be set based on the maximum value of the pixel difference between the background area of ​​the target frame image and the background area of ​​the current reference frame image. This noise threshold can be used to determine whether differences between other pixels are noise differences or real differences. Compared to setting a fixed noise threshold, this noise threshold setting method is more reasonable and reliable.

[0052] In some specific implementations, a target object region of a target frame image and a target object region of a current reference frame image are fused based on pixel differences and a current noise threshold to obtain a fused frame image. This can be achieved as follows: for a target pixel located at the same position in the target object region of the target frame image and the target object region of the current reference frame image, if the pixel difference corresponding to the target pixel is not greater than the noise threshold, the pixel value of the target pixel in the fused frame image is determined to be the pixel value of the target pixel in the current reference frame image. If the pixel difference corresponding to the target pixel is greater than the noise threshold, the pixel value of the target pixel in the fused frame image is determined to be the pixel value of the target pixel in the target frame image. A fused frame image is then obtained based on the pixel values ​​of each target pixel in the fused frame image. It is understood that if the pixel difference is not significant, the pixel of the reference frame image is fixedly used; if the pixel difference is significant, the pixel of the target frame image is retained. This approach minimizes noise differences between different images by maximizing the noise level of the reference frame image.

[0053] In practical applications, the target object area of ​​the target frame image and the target object area of ​​the current reference frame image can be cropped separately, and then the pixel difference and noise threshold between the cropped object area images are compared to generate a fused frame image based on the cropped object area images.

[0054] The image processing method provided by the disclosed embodiment further includes: when the current reference frame image is the target frame image, obtaining a frame image used for the target object processing task based on the target object region of the target frame image. Because the current reference frame image corresponding to the target frame image is itself, even after fusion, the target object region of the target frame image remains the target object region itself, and the target object region of the target frame image can be directly cropped to obtain the frame image used for the target object processing task.

[0055] When the target frame image to be processed is the first frame image, the reference frame image corresponding to the first frame image is the first frame image itself, and the reference object detection frame corresponding to the first frame image is the target object detection frame of the first frame image itself. When the target frame image to be processed is not the first frame image, referring to the flowchart of an image processing method shown in FIG2 , the method mainly includes the following steps S202 to S218b:

[0056] Step S202 , obtaining the target frame image to be processed and the reference frame image corresponding to the previous frame image of the target frame image, and obtaining the target object detection frame of the target frame image and the reference object detection frame corresponding to the previous frame image of the target frame image.

[0057] Step S204: Determine whether the similarity between the target object detection frame of the target frame image and the reference object detection frame corresponding to the previous frame image of the target frame image is less than a preset similarity threshold. If not, proceed to step S206a. If so, proceed to step S206b.

[0058] Step S206a: Determine that the current reference object detection frame corresponding to the target frame image is the reference object detection frame corresponding to the previous frame image. Then, execute step S208.

[0059] Step S206b: Determine that the current reference object detection frame corresponding to the target frame image is the target object detection frame corresponding to the target frame image, and then execute step S212a.

[0060] Step S208 , obtaining the average pixel difference between the reference frame image and the target frame image corresponding to the previous frame image.

[0061] Step S210: Determine whether the pixel difference mean is greater than a preset difference threshold. If yes, proceed to step S212a; if no, proceed to step S212b.

[0062] In step S212a, the current reference frame image corresponding to the target frame image is determined to be the target frame image. That is, the reference frame images corresponding to the target frame image and the previous frame image are changed. Then, step S214a is executed.

[0063] In step S212b, the current reference frame image corresponding to the target frame image is determined to be the reference frame image corresponding to the previous frame image. That is, the reference frame images corresponding to the target frame image and the previous frame image are consistent and remain unchanged. Steps S214b to S218b are then executed.

[0064] Step S214a: determining the target object region corresponding to the target frame image based on the current reference object detection frame of the target frame image, and cropping the target object region of the target frame image to obtain an image used for the target object processing task.

[0065] Step S214b: determine the target object area corresponding to the target frame image based on the current benchmark object detection frame of the target frame image, and crop it to obtain a first object area map; determine the target object area corresponding to the current benchmark frame image based on the benchmark object detection frame of the current benchmark frame image, and crop it to obtain a second object area map.

[0066] Step S216b: obtaining pixel differences between the first object region map and the second object region map, and obtaining a current noise threshold corresponding to the target frame image and the current reference frame image.

[0067] Step S218b: Based on the pixel difference and the current noise threshold, the first object region map and the second object region map are fused to obtain an image used for the target object processing task. The specific fusion method can be referred to the above related content and will not be repeated here.

[0068] The above-mentioned method provided by the embodiment of the present disclosure can effectively reduce the noise difference between different images by setting the reference object detection frame and the reference frame image, and effectively ensure the execution effect of the target object processing task.

[0069] To facilitate understanding of the above-mentioned image processing method provided by the embodiments of the present disclosure, an exemplary description is provided below. Assuming that multiple frames of images are collected for the user's eyes, for the first frame image 1, the corresponding reference frame image is image 1 itself, and the corresponding reference object detection frame is also the target object detection frame of image 1 itself (referred to as detection frame 1 for short). By cropping image 1 based on detection frame 1, an eye image for input into the gaze estimation model to perform the gaze estimation task can be obtained. For image 2, the intersection-over-union algorithm can be first used to obtain the overlap ratio between the target object detection frame of image 2 (referred to as detection frame 2 for short) and detection frame 1. If the overlap ratio is not less than the preset overlap ratio threshold (corresponding to the aforementioned similarity threshold), the reference object detection frame of image 2 is still detection frame 1. At this point, the mean pixel difference between image 2 and image 1 can be further determined. If the mean pixel difference is greater than the preset difference threshold, image 2 is used as the latest reference frame image, that is, the reference frame image corresponding to image 2 is itself. If the mean pixel difference is not greater than the preset difference threshold, the reference frame image corresponding to image 2 is still image 1. Conversely, if the overlap between detection frame 2 and detection frame 1 is less than a preset overlap threshold, the reference object detection frame of image 2 is updated to detection frame 2. This means that the reference object detection frame is updated, and detection frame 2 serves as the latest reference object detection frame. When the reference frame image corresponding to image 2 is image 1, fusion can be performed based on the target object region of image 1 (determined based on detection frame 1) and the target object region of image 2 (determined based on the reference object detection frame of image 2). Specifically, assuming that the reference object detection frame of image 2 is detection frame 1, image 1 is cropped based on detection frame 1 to obtain object image 1. Image 2 is cropped based on detection frame 1 to obtain object image 2. For each pixel with the same position in object image 1 and object image 2, if the pixel difference is less than the noise threshold, the pixel value in the final fused image 2 is the pixel value of object image 1. If the pixel difference is not less than the noise threshold, the pixel value in the final fused image 2 is the pixel value of object image 2. The final fused image 2 serves as the eye image input to the gaze estimation model to perform the gaze estimation task. By analogy with the above method, each frame of the captured image can be processed accordingly to obtain an image for input to the line of sight estimation model, which will not be repeated here.

[0070] In summary, the image processing method provided by the embodiment of the present disclosure can minimize the noise difference between different images, thereby effectively ensuring the stability of the target object processing task. Moreover, compared with the fuzzy smoothing processing and other methods used in related technologies, it basically does not affect the image accuracy, and the required amount of calculation is relatively small, which helps to better ensure the task execution effect in the presence of image noise. In the line of sight estimation task, the method provided by the embodiment of the present disclosure is compared with the method of smoothing the line of sight estimation results corresponding to each frame image output by the line of sight estimation model in the related technology. There is no need to manually adjust the smoothing coefficient, and the line of sight estimation results corresponding to each frame will not interfere with each other, so the reliability of the line of sight estimation results can be better guaranteed.

[0071] Corresponding to the aforementioned image processing method, FIG3 is a schematic structural diagram of an image processing device provided by an embodiment of the present disclosure. The device can be implemented by software and / or hardware and can generally be integrated into an electronic device. As shown in FIG3 , the image processing device includes: a target frame image determination module 302, which is used to obtain multiple frames of images obtained by shooting a target object and determine the current target frame image to be processed in the multiple frames; a reference frame image determination module 304, which is used to determine a current reference frame image corresponding to the target frame image from the multiple frames; wherein the current reference frame image is a frame image shot earlier than the target frame image, or , the current reference frame image is the target frame image; an acquisition module 306 is used to obtain the pixel difference between the target object area of ​​the target frame image and the target object area of ​​the current reference frame image, and obtain the current noise threshold corresponding to the target frame image and the current reference frame image when the current reference frame image is not the target frame image; a fusion module 308 is used to fuse the target object area of ​​the target frame image with the target object area of ​​the current reference frame image according to the pixel difference and the current noise threshold to obtain a fused frame image; wherein the fused frame image is used as the frame image adopted by the target object processing task.

[0072] Compared with the related art, the embodiment of the present disclosure can comprehensively consider the pixel difference and noise threshold between the target object areas of the image to be processed and the reference frame image through the above method. The resulting fused frame image can better reduce the impact of noise from different images on the target object processing task. The above method has little effect on image accuracy, which helps to better ensure the task execution effect even in the presence of image noise.

[0073] In some implementations, the target frame image determination module 302 is specifically configured to: based on shooting time, sequentially select each frame image in the multiple frames from front to back as the target frame image to be processed currently.

[0074] In some embodiments, the reference frame image determination module 304 is specifically used to: when the target frame image is the first frame image among the multiple frame images, determine that the current reference frame image corresponding to the target frame image is the target frame image; when the target frame image is not the first frame image among the multiple frame images, obtain the reference frame image corresponding to the previous frame image of the target frame image, and determine the current reference frame image corresponding to the target frame image based on the reference frame image corresponding to the previous frame image and the target frame image.

[0075] In some embodiments, the device further includes a detection frame determination module for obtaining a target object detection frame corresponding to the target frame image, and determining a current reference object detection frame corresponding to the target frame image based on the target object detection frame corresponding to the target frame image; wherein the current reference object detection frame is a target object detection frame of a frame image shot earlier than the target frame image, or the current reference object detection frame is the target object detection frame of the target frame image.

[0076] In some embodiments, the reference frame image determination module 304 is specifically used to: determine the current reference frame image corresponding to the target frame image from the reference frame image corresponding to the previous frame image and the target frame image based on the target object detection frame and the current reference object detection frame corresponding to the target frame image.

[0077] In some embodiments, the detection frame determination module is specifically used to: when the target frame image is the first frame image in the multiple frame images, determine that the current reference object detection frame corresponding to the target frame image is the target object detection frame corresponding to the target frame image; when the target frame image is not the first frame image in the multiple frame images, obtain the reference object detection frame corresponding to the previous frame image of the target frame image, and determine the current reference object detection frame corresponding to the target frame image based on the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image.

[0078] In some embodiments, the detection frame determination module is specifically used to: obtain the degree of similarity between the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image, as well as a preset similarity threshold; when the degree of similarity is not less than the similarity threshold, determine that the current reference object detection frame corresponding to the target frame image is the reference object detection frame corresponding to the previous frame image; when the degree of similarity is less than the similarity threshold, determine that the current reference object detection frame corresponding to the target frame image is the target object detection frame corresponding to the target frame image.

[0079] In some embodiments, the degree of similarity is represented by an overlap ratio between a reference object detection frame corresponding to the previous frame image and a target object detection frame corresponding to the target frame image.

[0080] In some embodiments, the reference frame image determination module 304 is specifically used to: determine that the current reference frame image corresponding to the target frame image is the target frame image when the current reference object detection frame corresponding to the target frame image is the target object detection frame of the target frame image; and obtain the target difference between the reference frame image corresponding to the previous frame image and the target frame image when the current reference object detection frame corresponding to the target frame image is not the target object detection frame of the target frame image, and determine the current reference frame image corresponding to the target frame image from the reference frame image corresponding to the previous frame image and the target frame image based on the target difference and a preset difference threshold.

[0081] In some embodiments, the reference frame image determination module 304 is specifically used to: when the target difference is not greater than the preset difference threshold, determine that the current reference frame image corresponding to the target frame image is the reference frame image corresponding to the previous frame image; when the target difference is greater than the preset difference threshold, determine that the current reference frame image corresponding to the target frame image is the target frame image.

[0082] In some embodiments, the target difference is represented by an average pixel difference between a reference frame image corresponding to the previous frame image and the target frame image.

[0083] In some implementations, the target object region of the target frame image is determined based on a current reference object detection frame corresponding to the target frame image.

[0084] In some embodiments, the acquisition module 306 is specifically used to: obtain the maximum value of the pixel difference corresponding to the background area of ​​the target frame image and the background area of ​​the current reference frame image, so as to determine the current noise threshold corresponding to the target frame image and the current reference frame image based on the maximum value; wherein, the background area of ​​the target frame image is the area outside the edge contour of the target object in the target frame image, and the background area of ​​the current reference frame image is the area outside the edge contour of the target object in the current reference frame image.

[0085] In some embodiments, the fusion module 308 is specifically used to: for a target pixel located at the same position in the target object area of ​​the target frame image and the target object area of ​​the current reference frame image, if the pixel difference corresponding to the target pixel is not greater than the noise threshold, then determine that the pixel value of the target pixel in the fused frame image is the pixel value of the target pixel of the current reference frame image; if the pixel difference corresponding to the target pixel is greater than the noise threshold, then determine that the pixel value of the target pixel in the fused frame image is the pixel value of the target pixel of the target frame image; and obtain a fused frame image based on the pixel value of each target pixel in the fused frame image.

[0086] In some embodiments, the apparatus further includes an image acquisition module configured to obtain an image used for the target object processing task based on a target object region of the target frame image when the current reference frame image is the target frame image.

[0087] In some embodiments, the target object includes an eye; and the target object processing task includes a gaze estimation task.

[0088] The image processing device provided by the embodiments of the present disclosure can execute the image processing method provided by any embodiment of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0089] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described device embodiment can refer to the corresponding process in the method embodiment, and will not be repeated here.

[0090] An embodiment of the present disclosure provides an electronic device, which includes: a storage device storing a computer program; and a processing device configured to execute the computer program in the storage device to implement the steps of any one of the methods in the present disclosure.

[0091] Reference is now made to FIG4 , which illustrates a schematic diagram of the structure of an electronic device 400 suitable for implementing embodiments of the present disclosure. Terminal devices in embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device illustrated in FIG4 is merely an example and should not limit the functionality or scope of use of embodiments of the present disclosure.

[0092] As shown in Figure 4, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 401, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 402 or a program loaded from a storage device 408 into a random access memory (RAM) 403. Various programs and data required for the operation of electronic device 400 are also stored in RAM 403. Processing device 401, ROM 402, and RAM 403 are connected to each other via a bus 404. An input / output (I / O) interface 405 is also connected to bus 404.

[0093] Typically, the following devices may be connected to the I / O interface 405: an input device 406 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 407 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 408 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 409. The communication device 409 may allow the electronic device 400 to communicate with other devices wirelessly or by wire to exchange data. Although FIG4 shows the electronic device 400 with various devices, it should be understood that not all of the devices shown are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0094] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 409, or installed from the storage device 408, or installed from the ROM 402. When the computer program is executed by the processing device 401, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0095] In addition to the above-mentioned methods and devices, the embodiments of the present disclosure may also be a computer program product, which includes computer program instructions, which, when executed by a processor, cause the processor to perform the image processing method provided by the embodiments of the present disclosure. The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present disclosure, the programming languages ​​including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0096] In addition, the embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor is enabled to execute the image processing method provided by the embodiment of the present disclosure.

[0097] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, device or component of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0098] The embodiments of the present disclosure further provide a computer program product, including a computer program / instruction, which implements the image processing method in the embodiments of the present disclosure when executed by a processor.

[0099] It is understandable that before using the technical solutions disclosed in the various embodiments of this disclosure, the type, scope of use, usage scenarios, etc. of the personal information involved in this disclosure should be informed to the user and the user's authorization should be obtained in an appropriate manner in accordance with relevant laws and regulations.

[0100] For example, in response to a user's active request, a prompt message is sent to the user to clearly inform the user that the operation requested will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the electronic device, application, server, storage medium, or other software or hardware that performs the operations of the disclosed technical solution based on the prompt message.

[0101] As an optional but non-limiting implementation, in response to receiving a user's active request, the prompt information may be sent to the user in the form of a pop-up window, in which the prompt information may be presented in text form. Furthermore, the pop-up window may also contain a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0102] It is understandable that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of the present disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of the present disclosure.

[0103] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or device comprising the element.

[0104] The foregoing description is intended only to provide specific embodiments of the present disclosure, intended to enable those skilled in the art to understand and implement the present disclosure. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the embodiments described herein, but rather to be construed in the broadest manner consistent with the principles and novel features disclosed herein.

Claims

1. An image processing method, comprising: Acquire multiple frames of images obtained by photographing a target object, and determine a target frame image to be processed currently among the multiple frames of images; Determining a current reference frame image corresponding to the target frame image from the multiple frame images; wherein the current reference frame image is a frame image shot earlier than the target frame image, or the current reference frame image is the target frame image; In the case where the current reference frame image is not the target frame image, obtaining a pixel difference between a target object region of the target frame image and a target object region of the current reference frame image, and obtaining a current noise threshold value corresponding to the target frame image and the current reference frame image; and According to the pixel difference and the current noise threshold, the target object area of ​​the target frame image and the target object area of ​​the current reference frame image are fused to obtain a fused frame image; wherein the fused frame image is used as an image adopted for the target object processing task.

2. The method according to claim 1, wherein determining the target frame image currently to be processed among the multiple frames of images comprises: Based on the shooting time, each frame image in the multiple frame images is sequentially used as a target frame image to be processed currently from the front to the back.

3. The method according to claim 1, wherein determining a current reference frame image corresponding to the target frame image from the multiple frame images comprises: In a case where the target frame image is the first frame image in the multiple frame images, determining a current reference frame image corresponding to the target frame image as the target frame image; as well as In the case where the target frame image is not the first frame image among the multiple frame images, a reference frame image corresponding to the previous frame image of the target frame image is obtained, and based on the reference frame image corresponding to the previous frame image and the target frame image, a current reference frame image corresponding to the target frame image is determined.

4. The method according to claim 3, further comprising: Acquire a target object detection frame corresponding to the target frame image, and determine a current reference object detection frame corresponding to the target frame image based on the target object detection frame corresponding to the target frame image; wherein the current reference object detection frame is a target object detection frame of a frame image shot earlier than the target frame image, or the current reference object detection frame is a target object detection frame of the target frame image; and The step of determining the current reference frame image corresponding to the target frame image according to the reference frame image corresponding to the previous frame image and the target frame image comprises: According to the target object detection frame and the current reference object detection frame corresponding to the target frame image, the current reference frame image corresponding to the target frame image is determined from the reference frame image corresponding to the previous frame image and the target frame image.

5. The method according to claim 4, wherein determining the current reference object detection frame corresponding to the target frame image based on the target object detection frame corresponding to the target frame image comprises: In a case where the target frame image is a first frame image in the multiple frame images, determining a current reference object detection frame corresponding to the target frame image as a target object detection frame corresponding to the target frame image; as well as In the case where the target frame image is not the first frame image in the multiple frame images, a reference object detection frame corresponding to a previous frame image of the target frame image is obtained, and based on the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image, a current reference object detection frame corresponding to the target frame image is determined.

6. The method according to claim 5, wherein determining the current reference object detection frame corresponding to the target frame image according to the reference object detection frame corresponding to the previous frame image and the target object detection frame corresponding to the target frame image comprises: Obtaining a similarity between a reference object detection frame corresponding to the previous frame image and a target object detection frame corresponding to the target frame image, as well as a preset similarity threshold; When the degree of similarity is not less than the similarity threshold, determining that the current reference object detection frame corresponding to the target frame image is the reference object detection frame corresponding to the previous frame image; as well as When the degree of similarity is less than the similarity threshold, the current reference object detection frame corresponding to the target frame image is determined to be the target object detection frame corresponding to the target frame image.

7. The method according to claim 6, wherein the degree of similarity is characterized by an overlap ratio between a reference object detection frame corresponding to the previous frame image and a target object detection frame corresponding to the target frame image.

8. The method according to claim 4, wherein the step of determining the current reference frame image corresponding to the target frame image from the reference frame image corresponding to the previous frame image and the target frame image according to the target object detection frame and the current reference object detection frame corresponding to the target frame image comprises: In a case where the current reference object detection frame corresponding to the target frame image is the target object detection frame of the target frame image, determining the current reference frame image corresponding to the target frame image as the target frame image; as well as In a case where the current reference object detection frame corresponding to the target frame image is not the target object detection frame of the target frame image, a target difference between the reference frame image corresponding to the previous frame image and the target frame image is obtained, and based on the target difference and a preset difference threshold, a current reference frame image corresponding to the target frame image is determined from the reference frame image corresponding to the previous frame image and the target frame image.

9. The method according to claim 8, wherein the determining, based on the target difference and a preset difference threshold, the current reference frame image corresponding to the target frame image from the reference frame image corresponding to the previous frame image and the target frame image comprises: In a case where the target difference is not greater than the preset difference threshold, determining the current reference frame image corresponding to the target frame image as the reference frame image corresponding to the previous frame image; as well as In a case where the target difference is greater than the preset difference threshold, the current reference frame image corresponding to the target frame image is determined to be the target frame image. 10 . The method according to claim 8 , wherein the target difference is represented by an average pixel difference between a reference frame image corresponding to the previous frame image and the target frame image.

11. The method according to claim 4, wherein the target object area of ​​the target frame image is determined based on a current reference object detection frame corresponding to the target frame image.

12. The method according to claim 1, wherein the obtaining of the current noise threshold corresponding to the target frame image and the current reference frame image comprises: Acquire a maximum value of pixel difference values ​​corresponding to a background area of ​​the target frame image and a background area of ​​the current reference frame image, so as to determine a current noise threshold value corresponding to the target frame image and the current reference frame image according to the maximum value; as well as The background area of ​​the target frame image is an area outside the edge contour of the target object in the target frame image, and the background area of ​​the current reference frame image is an area outside the edge contour of the target object in the current reference frame image.

13. The method according to claim 1, wherein the step of fusing the target object region of the target frame image with the target object region of the current reference frame image according to the pixel difference and the current noise threshold to obtain a fused frame image comprises: For a target pixel located at the same position in the target object area of ​​the target frame image and the target object area of ​​the current reference frame image, if the pixel difference corresponding to the target pixel is not greater than the noise threshold, determining that the pixel value of the target pixel in the fused frame image is the pixel value of the target pixel of the current reference frame image; If the pixel difference corresponding to the target pixel is greater than the noise threshold, determining the pixel value of the target pixel in the fused frame image to be the pixel value of the target pixel in the target frame image; as well as A fused frame image is obtained based on the pixel value of each target pixel in the fused frame image.

14. The method according to claim 1, further comprising: In a case where the current reference frame image is the target frame image, an image used for the target object processing task is obtained based on a target object region of the target frame image.

15. The method of claim 1, wherein the target object comprises an eye; and wherein the target object processing task comprises a gaze estimation task.

16. An image processing device, comprising: A target frame image determination module is used to obtain multiple frames of images obtained by shooting a target object, and determine a target frame image to be processed currently in the multiple frames of images; A reference frame image determination module, used to determine a current reference frame image corresponding to the target frame image from the multiple frame images; wherein the current reference frame image is a frame image shot earlier than the target frame image, or the current reference frame image is the target frame image; an acquisition module, configured to acquire, when the current reference frame image is not the target frame image, a pixel difference between a target object region of the target frame image and a target object region of the current reference frame image, and to acquire a current noise threshold value corresponding to the target frame image and the current reference frame image; and A fusion module is used to fuse the target object area of ​​the target frame image with the target object area of ​​the current reference frame image according to the pixel difference and the current noise threshold to obtain a fused frame image; wherein the fused frame image is used as the image used for the target object processing task.

17. An electronic device, comprising: a storage device having a computer program stored thereon; as well as A processing device, used to execute the computer program in the storage device to implement the steps of the image processing method according to any one of claims 1 to 15.

18. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the image processing method according to any one of claims 1 to 15.

19. A computer program product, comprising computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the image processing method according to any one of claims 1 to 15.

Images