Image enhancement method, electronic equipment and storage medium

Abstract

The invention discloses an image enhancement method, electronic equipment and a storage medium. The method comprises the following steps: acquiring a current frame image; first differential data of the current frame image and the first reference frame image and motion detection results of multiple frames of associated images of the current frame image are synthesized, a plurality of target motion pixel points are searched from the current frame image, and the current frame image, the first reference frame image and the associated images are obtained through shooting of the same shooting device; the current frame image and the first reference frame image are obtained by shooting at different moments, the associated image is the current frame image or the number of frames differing from the current frame image is smaller than a preset frame number, and the target motion pixel point is a first motion pixel point determined based on first differential data and is a second motion pixel point determined based on a motion detection result; and performing enhancement processing on the plurality of target motion pixel points in the current frame image to obtain an enhanced image. Through the above mode, the target can be accurately enhanced so as to distinguish the target from the background.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image enhancement method, electronic device and storage medium. Background Art

[0002] Affected by factors such as the environment and the performance of the camera itself, the target object in the pictures and videos taken by the camera is very likely to blend into the background environment, resulting in the target not being prominent enough, thus affecting the visual observation effect and subsequent image processing operations. For example, infrared thermal imaging equipment relies on the thermal radiation energy of the object to form an image. When the temperature of the target object and the background environment is close, the image effect will show that the grayscale of the target and the background are close, making it impossible to effectively distinguish the target object from the background. Summary of the invention

[0003] The main technical problem solved by the present application is to provide an image enhancement method, an electronic device and a storage medium, which can accurately enhance the target to distinguish the target from the background.

[0004] To solve the above technical problems, the first aspect of the present application provides an image enhancement method, an electronic device and a storage medium, the method comprising: obtaining a current frame image; integrating first differential data between the current frame image and a first reference frame image and motion detection results of multiple frames of associated images of the current frame image, and finding a number of target motion pixel points from the current frame image, wherein the current frame image, the first reference frame image and the associated image are all captured by the same shooting device, the current frame image and the first reference frame image are captured at different times, the associated image is the current frame image or the number of frames different from the current frame image is less than a preset number of frames, the target motion pixel point is a first motion pixel point determined to belong to a moving target based on the first differential data, and is a second motion pixel point determined to belong to a moving target based on the motion detection result; enhancing the number of target motion pixel points in the current frame image to obtain an enhanced image.

[0005] To solve the above technical problems, the second aspect of the present application provides an electronic device, which includes a memory and a processor, and the memory stores program instructions; the processor is used to execute the program instructions stored in the memory to implement the method provided by the first aspect above.

[0006] In order to solve the above technical problem, the third aspect of the present application provides a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the method provided by the first aspect above.

[0007] The beneficial effect of the present application is that, different from the prior art, the present application integrates the first differential data between the current frame image and the first reference frame image, and the motion detection results of multiple frames of associated images of the current frame image, and finds out several target motion pixel points from the current frame image. The target motion pixel point is the first motion pixel point determined as belonging to the moving target based on the first differential data, and is also the second motion pixel point determined as belonging to the moving target based on the motion detection result. Combining the two methods to determine the target motion pixel point can improve the accuracy of the target motion pixel point, and then enhance the several target motion pixel points in the current frame image, which can achieve accurate enhancement of the target, and the obtained enhanced image can clearly distinguish the target from the background. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 It is a flowchart of an implementation method of an image enhancement method provided by the present application;

[0009] Figure 2 It is a flowchart of an implementation of step S12 in an implementation of an image enhancement method provided by the present application;

[0010] Figure 3 It is a flowchart of an implementation of step S13 in an implementation of an image enhancement method provided by the present application;

[0011] Figure 4 It is a flowchart of an implementation method of the first differential data acquisition method provided by the present application;

[0012] Figure 5 It is a schematic diagram of the framework structure of an embodiment of an electronic device provided by the present application;

[0013] Figure 6 It is a schematic diagram of the framework structure of an implementation scheme of a computer-readable storage medium provided in this application. DETAILED DESCRIPTION

[0014] The following is a clear and complete description of the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0015] It should be noted that in the embodiments of the present application, there are descriptions involving "first", "second", etc., which are only used for descriptive purposes and cannot be understood as indicating or implying their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" or "second" may explicitly or implicitly include at least one of the features.

[0016] In the description of the application, the meaning of "multiple" is at least two, for example, two, three, etc., unless otherwise clearly and specifically limited. In the embodiment of the present application, all directional indications (such as up, down, left, right, front, back ...) are only used to explain the relative position relationship, movement, etc. between the components under a certain specific posture (as shown in the drawings). If the specific posture changes, the directional indication also changes accordingly. In addition, the terms "including" and "having" and any of their deformations are intended to cover non-exclusive inclusions. For example, the process, method, system, product or equipment comprising a series of steps or units is not limited to the steps or units listed, but optionally also includes the steps or units not listed, or optionally also includes other steps or units inherent to these processes, methods, products or equipment.

[0017] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present invention. The appearance of the phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0018] See also Figure 1 , Figure 1 : is a flow chart of an embodiment of an image enhancement method provided by the present application, the method comprising:

[0019] S11: Get the current frame image.

[0020] This embodiment is used to enhance the target in the current frame image to highlight the target in the current frame image and distinguish the target from the background in the current frame image. It can be understood that in one embodiment, the current frame image can be the image to be enhanced, and the image to be enhanced can be the current image captured by the camera at the current moment, or it can be other than the current image. For example, the image to be enhanced can also be the image captured by the camera at a historical moment. The current frame image can be a visible light image, an infrared image, etc.

[0021] S12: searching for a number of target motion pixels from the current frame image by combining first differential data between the current frame image and the first reference frame image and motion detection results of multiple frames of associated images of the current frame image.

[0022] In one embodiment, the current frame image, the first reference frame image and the associated image are all captured by the same shooting device, and the current frame image and the first reference frame image may be captured at different times. Specifically, the first reference frame image may be an image captured at any historical moment before the current moment. For example, the first reference frame image may be a frame image captured at a previous moment adjacent to the current moment; for another example, the first reference frame image may also be a frame image captured at two moments before the current moment. The first differential data includes the difference between each pixel in the current frame image and the corresponding pixel in the first reference frame image, that is, the first differential data includes the pixel difference between each pixel in the current frame image and the corresponding pixel in the first reference frame image. Among them, the pixel in the current frame image and the corresponding pixel in the first reference frame image correspond to the same three-dimensional coordinate point in the three-dimensional world. In a specific embodiment, the current frame image is an image captured at the current moment, and the first reference frame image is an image captured at a moment before the current moment. The pixel values ​​of the two can be directly subtracted to obtain the first differential data. In another specific implementation, the first reference frame image may include multiple frames, and a differential data is calculated using the current frame image and every two frames of the multiple first reference frame images, and the multiple differential data are averaged to obtain the first differential data.

[0023] In one embodiment, the associated image may be the current frame image, and in this case, a number of target motion pixel points may be found based on the first differential data and the motion detection result of the current frame image. For example, the first differential data may be used to determine the first motion pixel point belonging to the motion target in the current frame image, and the target detection algorithm may be used to determine the second motion pixel point belonging to the motion target in the current frame image, and the pixel point belonging to both the first motion pixel point and the second motion pixel point may be used as the target motion pixel point.

[0024] In another embodiment, the number of frames that the associated image differs from the current frame image is less than the preset number of frames, wherein the associated image may be an image captured before the current frame image, and the multi-frame associated images may be multi-frame images captured within a time period. In a specific embodiment, a preset number of frames is set, and the preset number of frames may be the same as the number of multi-frame associated images, and images of the preset number of frames before the current frame image may be acquired to obtain multi-frame associated images. For example, if the preset number of frames is set to k frames, then k frames of images before the current frame image may be taken as multi-frame associated images. It can be understood that the k-frame images may be multi-frame images that are close to the shooting time of the current frame image. In another specific embodiment, the multi-frame associated images may include the current frame image, and an image whose number of frames that differ from the current frame image is less than the preset number of frames. At this time, a preset number of images before the current frame image may be acquired to obtain multi-frame associated images. Wherein, the preset number is equal to the preset number of frames minus 1.

[0025] In another specific embodiment, the preset number of frames may be less than the number of multi-frame associated images. For example, the number of multi-frame associated images is 10 frames, the preset number of frames may be 9 frames, and the multi-frame associated images may include the current frame image, and an image whose number of frames differing from the current frame image is less than the preset number of frames. The number of frames differing from the current frame image may be equal to the number of frames of the current frame image minus the number of frames of the associated image minus 1. For example, if the current frame image is the 10th frame image captured by the camera, the current frame image and the first 9 frames captured by the camera may be used as associated images, wherein the number of frames differing between the first frame image captured by the camera and the current frame image is 8 frames, which is less than the preset number of frames.

[0026] In one embodiment, the target moving pixel is a first moving pixel determined to belong to the moving target based on the first differential data, and is a second moving pixel determined to belong to the moving target based on the motion detection result. Specifically, a number of first moving pixels belonging to the moving target in the current frame image can be determined based on the first differential data, and a number of second moving pixels belonging to the moving target in the current frame image can be determined based on the motion detection result, and the pixel in the current frame image that belongs to both the first moving pixel and the second moving pixel is used as the target moving pixel.

[0027] Among them, the step of determining a number of first motion pixel points belonging to a moving target in the current frame image based on the first differential data includes: obtaining a pixel point whose differential value meets the first motion requirement as the first motion pixel point; or obtaining a pixel point whose differential value meets the first motion requirement as a candidate motion pixel point, and for each candidate motion pixel point, counting the number of candidate motion pixel points in the moving target cluster to which the candidate motion pixel point belongs, and taking the candidate motion pixel point whose number of candidate motion pixel points in the moving target cluster to which the candidate motion pixel point belongs is greater than the first threshold as the first motion pixel point. Please refer to the following for the first motion requirement and the moving target cluster, which will not be described here for the time being.

[0028] The step of determining the second motion pixel point based on the motion detection result includes: using the motion detection results of each associated image, counting the number of times each pixel point in the multi-frame associated image is detected as belonging to a moving target; determining the pixel point whose number is greater than a third threshold as the second motion pixel point. Specifically, the multi-frame associated image includes a current frame image and several images whose number of frames different from the current frame image is less than a preset number of frames. For each frame of the associated image, determine whether each pixel point in the associated image is a moving target. For the pixel point in the current frame image, obtain the associated pixel point corresponding to the pixel point in other associated images. The pixel point in the current frame image and the associated pixel point can correspond to the same position point in the three-dimensional world. Count the number of pixel points in the current frame image and the number of pixel points in the associated pixel points that are moving targets, and obtain the number of times the pixel point is detected as belonging to a moving target in the multi-frame associated image.

[0029] Among them, determining whether each pixel point in the associated image is a moving target can be determined by using an existing target detection algorithm, or by using the differential value corresponding to each pixel point. The method for calculating the differential value of each pixel point in the associated image is similar to the method for calculating the first differential data between the current frame image and the first reference frame image, and will not be repeated here.

[0030] In this embodiment, the steps of determining the first moving pixel point and determining the second moving pixel point are not in any particular order, that is, the step of determining the first moving pixel point can be performed first, or the step of determining the second moving pixel point can be performed first, or both can be performed simultaneously.

[0031] S13: performing enhancement processing on a number of target moving pixels in the current frame image to obtain an enhanced image.

[0032] In one embodiment, the area where several target motion pixels are located can be directly enhanced to obtain an enhanced image. In a specific embodiment, the differential value corresponding to each target motion pixel can be adjusted to obtain the enhanced value of each target motion pixel; the pixel value of each target motion pixel in the current frame image is superimposed with the enhanced value of each target motion pixel to obtain an enhanced image. In another specific embodiment, the enhancement value can also be pre-set, and the pixel value of each target motion pixel is superimposed with the enhancement value. In the above method, the first differential data between the current frame image and the first reference frame image, and the motion detection results of multiple frames of associated images of the current frame image are combined to find several target motion pixels from the current frame image. The target motion pixel is the first motion pixel determined to belong to the moving target based on the first differential data, and the second motion pixel determined to belong to the moving target based on the motion detection result. Combining the two methods to determine the target motion pixel can improve the accuracy of the target motion pixel, and then enhance the several target motion pixels in the current frame image, which can achieve accurate enhancement of the target, and the obtained enhanced image can clearly distinguish the target from the background.

[0033] In other embodiments, in order to make the shape of the target represented by the target motion pixel point closer to the real shape of the target, the third motion target area formed by several target motion pixel points can be expanded to obtain a fourth motion target area, wherein the pixel points of the fourth motion target area are updated to the target motion point for enhancement processing, that is, each pixel point of the fourth motion target area is enhanced. Among them, the expansion operation can set an expansion operation window, and the expansion operation window is used to determine the expansion degree. It can be understood that the expansion operation window can be set according to the actual situation. For example, if the target is a car, the car is more likely to present a horizontal shape in the image, that is, the length of the car is greater than the height of the car, then the expansion operation window m rows and n columns can be designed so that n is greater than m.

[0034] See also Figure 2 , Figure 2 : is a flow chart of step S12 in an embodiment of an image enhancement method provided by the present application. Step S12 may include:

[0035] S221: Based on first differential data between a current frame image and a first reference frame image, determine a plurality of first moving pixel points belonging to a moving target from the current frame image.

[0036] The first differential data includes a differential value between each pixel point in the current frame image and a corresponding pixel point in the first reference frame image.

[0037] In one embodiment, a pixel point whose differential value meets the first motion requirement can be obtained as the first motion pixel point. Specifically, based on the differential value corresponding to each pixel point of the current frame image, it is determined whether each pixel point of the current frame image is a first motion pixel point. If the differential value meets the first motion requirement, it is determined that the pixel point corresponding to the differential value that meets the first motion requirement in the current frame image is the first motion pixel point. Among them, the first motion requirement includes that the characterization value of the differential value is greater than the second threshold value, the characterization value is the absolute value of the differential value, or the characterization value is the value obtained after filtering the absolute value of the differential value, and the filtering can adopt any filtering method such as Gaussian filtering, mean filtering, bilateral filtering, etc. The second threshold is a preset value or a preset ratio of the maximum characterization value, and the maximum characterization value is the maximum value of the characterization values ​​of each differential value contained in the first differential data. In other embodiments, the first motion requirement may also include that the differential value is greater than the second threshold value.

[0038] In a specific embodiment, the characterization value of the differential value is a value obtained after filtering the absolute value of the differential value, and the second threshold is a preset ratio of the maximum characterization value, that is, the maximum value of the values ​​obtained after filtering the absolute value of the differential value is multiplied by the preset ratio to obtain the second threshold. If the characterization value of the differential value corresponding to the pixel point of the current frame image is greater than the second threshold, the pixel point is determined to be a first motion pixel point. It can be understood that in this embodiment, the second threshold value changes according to the change of the image. In order to avoid the second threshold value being too large or too small, the maximum threshold value and the minimum threshold value can be set to limit the second threshold value. If the calculated second threshold value is greater than the maximum threshold value, the maximum threshold value is used as the second threshold value to determine the first motion pixel point; if the calculated second threshold value is less than the minimum threshold value, the minimum threshold value is used as the second threshold value to determine the first motion pixel point.

[0039] In order to avoid using the first motion requirement to judge that the moving pixel point is an interference pixel point such as noise or bad pixel, in another embodiment, the pixel point whose differential value meets the first motion requirement can be obtained as a candidate moving pixel point; for each candidate moving pixel point, the number of candidate moving pixel points in the moving target cluster to which the candidate moving pixel point belongs is counted, and the candidate moving pixel point whose number of candidate moving pixel points in the moving target cluster to which the candidate moving pixel point belongs is greater than the first threshold is taken as the first moving pixel point. For example, for candidate moving pixel point A, the number of candidate moving pixel points in the moving target cluster to which the candidate moving pixel point A belongs is counted, and if it is greater than the first threshold, the candidate moving pixel point A is determined to be the first moving pixel point; if it is less than the first threshold, the candidate moving pixel point A is determined to be an interference pixel point.

[0040] The motion target cluster may be a region determined by a region growing method, a connected domain determined by a connected domain method, or a neighborhood window. Specifically, before counting the number of candidate motion pixel points in the motion target cluster to which the candidate motion pixel points belong, the method further includes: using a region growing method to aggregate each candidate motion pixel point into at least one region, wherein each region is a motion target cluster; or, using a connected domain-based method to form at least one connected domain using each candidate motion pixel point, wherein each connected domain is a motion target cluster; or, using a neighborhood window of each candidate motion pixel point as the motion target cluster of each candidate motion pixel point, and the size of the neighborhood window can be pre-set according to actual conditions.

[0041] S222: Based on the motion detection result, further determine a plurality of second motion pixel points from the plurality of first motion pixel points to serve as a plurality of target motion pixel points.

[0042] In one embodiment, a first moving target area formed by a number of first moving pixel points is expanded to obtain a second moving target area, and the pixel points in the second moving target area are updated to the first moving pixel points; invalid values ​​are set for the pixel points in the area other than the second moving target area in each associated image; motion detection is performed on each associated image after the invalid value is set to obtain the motion detection results of each associated image; using the motion detection results of each associated image, the number of times each first moving pixel point is detected as belonging to a moving target in multiple frames of associated images is counted; and the first moving pixel point whose number is greater than a third threshold is determined as a second moving pixel point, and used as a target moving pixel point.

[0043] The motion detection of each associated image after the invalid value is set may be to determine whether each pixel in the second moving target area of ​​each associated image belongs to a moving target. The invalid value setting may be to change the pixel value of the pixel in the area other than the second moving target area to 0.

[0044] In this embodiment, after determining a number of first motion pixel points in the current frame image, a number of target motion pixel points are further determined from the number of first motion pixel points based on the motion detection results of the first motion pixel points and multiple frame associated images. In other words, in this embodiment, the target is firstly roughly detected using differential data, interference pixel points are eliminated from the rough detection results, and then the rough detection results after eliminating interference pixel points are screened in combination with multiple frame images to obtain target motion pixel points. The accuracy of target motion pixel points determined by combining multiple methods is high, which can improve the accuracy of moving target detection.

[0045] See also Figure 3 , Figure 3 : is a flow chart of step S13 in an embodiment of an image enhancement method provided by the present application. Step S13 may include:

[0046] S331: Adjust the differential value corresponding to each target motion pixel point to obtain an enhanced value of each target motion pixel point.

[0047] In one implementation, the first global enhancement coefficient may be used to adjust the differential value corresponding to each target motion pixel to obtain the enhancement value of each target motion pixel. In a specific implementation, the first global enhancement coefficient may be multiplied by the differential value corresponding to each target motion pixel to obtain the enhancement value of each target motion pixel. It is understandable that the first global enhancement coefficient may be set according to actual needs.

[0048] In another embodiment, the local enhancement coefficient corresponding to each target motion pixel point can be used to adjust the differential value corresponding to each target motion pixel point to obtain the enhancement value of each target motion pixel point. In a specific embodiment, the local enhancement coefficient corresponding to each target motion pixel point is multiplied by the differential value corresponding to each target motion pixel point to obtain the enhancement value of each target motion pixel point. In another specific embodiment, for each target motion pixel point, the local enhancement coefficient corresponding to the target motion pixel point can also be multiplied by the second global enhancement coefficient and the differential value corresponding to the target motion pixel point to obtain the enhancement value of the target motion pixel point. It can be understood that the second global enhancement coefficient can be set according to actual needs.

[0049] Among them, the local enhancement coefficient of the target motion pixel is the ratio of the differential value corresponding to the target motion pixel or the absolute value of the differential value to a fourth threshold, the fourth threshold is a preset threshold or an adaptive threshold, the preset threshold can be set as needed, and the adaptive threshold is any one of the following or is determined based on any one of the following: a preset ratio of the maximum characterization value, the number of motion pixels in the motion target cluster where the target motion pixel is located, the number of times detected as belonging to a motion target in multiple frame associated images, a discrete degree characterization value of the differential value of each pixel within the first range of the target motion pixel, and the difference between the maximum differential value and the minimum differential value of each pixel within the second range of the target motion pixel.

[0050] In one embodiment, the fourth threshold is an adaptive threshold, and the local enhancement coefficient of the target motion pixel is the ratio of the differential value corresponding to the target motion pixel to the fourth threshold. In a specific embodiment, the fourth threshold may be a preset ratio of the maximum characterization value, the maximum characterization value is the maximum of the characterization values ​​of the differential values ​​corresponding to the pixels of the current frame image, the characterization value of the differential value may be the absolute value of the differential value, in which case the maximum characterization value is the maximum of the absolute values ​​of the differential values ​​corresponding to the pixels; the characterization value of the differential value may also be the value obtained after filtering the absolute value of the differential value, in which case the maximum characterization value is the maximum of the values ​​obtained after filtering.

[0051] In another specific implementation, the fourth threshold value may be determined based on the probability that the target motion pixel point belongs to the moving target, and the greater the probability that the target motion pixel point belongs to the moving target, the greater or smaller the fourth threshold value may be, so that the enhancement value of the target motion pixel point determined based on the fourth threshold value is greater. In one implementation, the greater the probability that the target motion pixel point belongs to the moving target, the smaller the fourth threshold value may be.

[0052] Among them, the probability that the target motion pixel point belongs to a moving target can be determined by any one of the number of motion pixels in the moving target cluster where the target motion pixel point is located, the number of times it is detected as belonging to a moving target in multiple frames of associated images, the discrete degree characterization value of the differential values ​​of each pixel point within the first range of the target motion pixel point, and the difference between the maximum differential value and the minimum differential value of each pixel point within the second range of the target motion pixel point.

[0053] The more moving pixels there are in the moving target cluster where the target moving pixel point is located, the greater the probability that the target moving pixel point belongs to the moving target. The more times the target moving pixel point is detected as belonging to the moving target in the multi-frame associated image, the greater the probability that the target moving pixel point belongs to the moving target. The greater the discrete degree representation value of the differential value of each pixel point located within the first range of the target moving pixel point, the greater the probability that the target moving pixel point belongs to the moving target, wherein the discrete degree representation value includes but is not limited to variance, standard deviation, etc. The greater the difference between the maximum differential value and the minimum differential value of each pixel point located within the second range of the target moving pixel point, the greater the probability that the target moving pixel point belongs to the moving target.

[0054] S332: Superimpose the pixel value of each target motion pixel point in the current frame image with the enhancement value of each target motion pixel point to obtain an enhanced image.

[0055] In one embodiment, the pixel value of each target motion pixel in the current frame image can be added to the enhancement value of each target motion pixel to obtain the pixel value of each target motion pixel after enhancement to obtain an enhanced image. Specifically, the pixel value of each target motion pixel can be enhanced according to the following formula to obtain an enhanced image.

[0056] Imgout=Tstr*alphaF*FdiffF+Imgcur

[0057] Among them, Imgout is the pixel value after the pixel value of the target motion pixel is enhanced, Tstr is the second global enhancement coefficient, alphaF is the local enhancement coefficient of the target motion pixel, FdiffF is the differential value corresponding to the target motion pixel, and Imgcur is the pixel value of each target motion pixel in the current frame image.

[0058] In this embodiment, the differential value corresponding to each target motion pixel is adjusted to obtain the enhancement value of each target motion pixel, and then the target motion pixel is enhanced using the enhancement value. Furthermore, a local enhancement coefficient is set so that the pixel is enhanced according to the possibility that the pixel is biased towards the moving target. The closer the pixel is to the moving target, the greater the pixel enhancement strength is, and the target can be accurately enhanced. Since the enhancement operation only enhances the moving target area and the background area is not affected, the normal image display effect of the background area is also guaranteed while ensuring that the target display effect meets expectations; in addition, the enhancement uses a local enhancement coefficient to distinguish the enhancement strength of the detected moving target area, further ensuring that the moving target is likely to be highlighted and the falsely detected interference information is not highlighted as much as possible.

[0059] In another embodiment, before adjusting the differential value corresponding to each target motion pixel point, it also includes: performing time domain smoothing on the filtered differential data of the current frame image and the filtered differential data of the second reference frame image, and using the differential value corresponding to each target motion pixel point in the time domain smoothing result as the differential value corresponding to each target motion pixel point to be adjusted. In this embodiment, the differential value corresponding to each target motion pixel point obtained by time domain smoothing is adjusted to obtain the enhanced value of each target motion pixel point, and then the pixel value of each target motion pixel point in the current frame image is superimposed with the enhanced value of each target motion pixel point to obtain an enhanced image. Through time domain smoothing, the noise and mutation in the differential data can be reduced, so that the detection result of the moving target determined by the differential data is more accurate.

[0060] The filtered differential data of the current frame image is obtained by filtering the first differential data between the current frame image and the first reference frame image, or by filtering the absolute value of the first differential data between the current frame image and the first reference frame image. The first reference frame image may be a frame image before the current frame image.

[0061] The filtered differential data of the second reference frame image is obtained by filtering the second differential data between the second reference frame image and the third reference frame image, or by filtering the absolute value of the second differential data between the second reference frame image and the third reference frame image. The second reference frame image may be a frame image before the current frame image, and the third reference frame image may be a frame image before the second reference frame image.

[0062] In other embodiments, the first reference frame image may include multiple frames, and the multiple first reference frame images may include the previous frame image of the current frame image, the previous frame image of the current frame image, etc. Differential data may be calculated for every two frames of the multiple first reference frame images, and then the calculated differential data are averaged to obtain the first differential data. Similarly, the third reference frame image may also be multiple frames, and the second differential data may be obtained in the same manner as above, which will not be described in detail here.

[0063] See also Figure 4 , Figure 4 : is a flow chart of an implementation method of a first differential data acquisition method provided by the present application. The first differential data acquisition method includes:

[0064] S41: Combine each pixel point in the current image frame with the corresponding pixel point in the first reference frame image to form a number of pixel pairs; or, pre-process the current frame image and the first reference frame image respectively, and combine each pixel point in the pre-processed current image frame with the corresponding pixel point in the pre-processed first reference frame image to form a number of pixel pairs, and the pre-processing includes filtering processing.

[0065] In one embodiment, the position of the shooting device is fixed, so the current frame image and the first reference frame image are both obtained by shooting the same area, so the corresponding pixel points in the current frame image and the first reference frame image of any three-dimensional coordinate point in the three-dimensional world are located at the same position, that is, the pixel points in the first row and first column of the current frame image and the pixel points in the first row and first column of the first reference frame image correspond to the same three-dimensional coordinate point. Therefore, the pixel points in the current image frame and the corresponding pixel points in the first reference frame image can be formed into a pixel point pair, and the two pixel points in the pixel point pair correspond to the same three-dimensional coordinate point.

[0066] It is understandable that in order to reduce the influence of irrelevant interference information such as noise on the determination of subsequent motion pixels, the current frame image and the first reference frame image can be selectively pre-processed, and then a number of pixel pairs are formed after the pre-processing. The pre-processing includes filtering, and the filtering includes but is not limited to Gaussian filtering, mean filtering, bilateral filtering, etc.

[0067] In another embodiment, the position of the shooting device is not fixed, and the moving distance of the shooting device between the current frame image and the first reference frame image can be determined based on the pixel data of the current frame image and the first reference frame image; then, the sum of the first coordinate of the pixel point in the current image frame and the moving distance is used as the second coordinate, and the pixel point at the first coordinate in the current image frame and the pixel point at the second coordinate in the first reference frame image are combined into a pixel pair.

[0068] Among them, any one of the grayscale projection method, block search matching method, optical flow method, etc. can be used to determine the moving distance. In a specific embodiment, the grayscale projection method is used to determine the moving distance of the shooting device between the current frame image and the first reference frame image based on the pixel data of the current frame image and the first reference frame image. Specifically, the maximum movement amplitude Xmax of the shooting device in the first direction is determined. The maximum movement amplitude Xmax can be preset, and the first direction moving distance dx is sequentially taken from -Xmax to Xmax, and the first direction moving distance dx is calculated as the sum of multiple column difference values ​​between the current frame image and the first reference frame image when each value is taken, wherein each column difference value represents the difference between the sum of the pixel values ​​of a first column j in the current frame image and the sum of the pixel values ​​of the corresponding second column i in the first reference frame image, and i is the sum of j and dx. Determine the maximum movement amplitude Ymax of the camera in the second direction, take values ​​of the second direction movement distance dy from -Ymax to Ymax in sequence, and calculate the second direction movement distance dy as the sum of multiple row difference values ​​between the current frame image and the first reference frame image at each value, wherein each row difference value represents the difference between the sum of pixel values ​​of a first row m in the current frame image and the sum of pixel values ​​of a corresponding second row n in the first reference frame image, where n is the sum of m and dy. Determine the movement distance based on the sum of the column difference values ​​and the sum of the row difference values.

[0069] In a specific implementation, the sum of the pixel values ​​of the first column j in the current frame image can be calculated. The sum of the pixel values ​​of the first column j can be obtained by directly summing the pixel values ​​of each pixel point in the first column j; or the column pixel mean of the pixel value of each pixel point in the first column j can be calculated first, and then the difference between the pixel value of each pixel point in the first column j and the column pixel mean can be summed, as shown in the following formula.

[0070]

[0071]

[0072] Among them, mean_col_cur(j) is the column pixel mean of the pixel values ​​of each pixel in the first column j of the current frame image, Imgcur(i,j) is the pixel value of the pixel in the i-th row and j-th column of the current frame image, M is the number of rows of the current frame image resolution, and sum_col_cur(j) is the sum of the pixel values ​​in the first column j of the current frame image.

[0073] It can be understood that the sum of the pixel values ​​corresponding to the second column i in the first reference frame image is calculated in the same way as the sum of the pixel values ​​of the first column j in the previous frame image, which will not be repeated here.

[0074] Further, the sum of the column difference values ​​corresponding to each first-direction movement distance dx is calculated according to the following formula.

[0075]

[0076] Among them, sum_col_cur(j) is the sum of the pixel values ​​of the first column j of the current frame image, sum_col_pre(j+dx) is the sum of the pixel values ​​corresponding to the second column i in the first reference frame image, i is the sum of j and dx, and diff_col(dx+Xmax+1) is the sum of the column difference values.

[0077] By the above method, after calculating the sum of the column difference values ​​corresponding to each first-direction movement distance dx, the minimum sum of the column difference values ​​can be selected, and the first-direction movement distance dx corresponding to the minimum sum of the column difference values ​​is used as the movement distance of the camera in the first direction between the current frame image and the first reference frame image. In the same way, the movement distance of the camera in the second direction between the current frame image and the first reference frame image can be determined.

[0078] S42: Obtain the pixel value difference of each pixel point pair to obtain first differential data.

[0079] In one implementation, the pixel values ​​of two pixels in each pixel pair may be subtracted to obtain first differential data corresponding to each pixel.

[0080] In this embodiment, when the position of the photographing device is fixed or moving, the first differential data can be accurately determined, and then the moving pixel points can be accurately detected using the first differential data.

[0081] See also Figure 5 , Figure 5 It is a schematic diagram of the framework structure of an embodiment of an electronic device provided in the present application.

[0082] The electronic device 50 includes a memory 51 and a processor 52. The memory 51 stores program instructions, and the processor 52 is used to execute the program instructions stored in the memory 51 to implement the steps of any of the above-mentioned method implementation methods. In a specific implementation scenario, the electronic device 50 may include, but is not limited to: a microcomputer, a server, and in addition, the electronic device 50 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0083] Specifically, the processor 52 is used to control itself and the memory 51 to implement the steps of any of the above-mentioned method implementation methods. The processor 52 can also be called a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip with signal processing capabilities. The processor 52 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 52 can be implemented by an integrated circuit chip.

[0084] See also Figure 6 , Figure 6 It is a schematic diagram of the framework structure of an implementation scheme of a computer-readable storage medium provided in this application.

[0085] The computer-readable storage medium 60 stores program instructions 61 , and when the program instructions 61 are executed by a processor, they are used to implement the steps of any of the above-mentioned method implementations.

[0086] The computer-readable storage medium 60 may specifically be a medium that can store computer programs, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or may be a server that stores the computer program. The server may send the stored computer program to other devices for execution, or may also execute the stored computer program itself.

[0087] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0088] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. An image enhancement method, characterized in that: include: Get the current frame image; Combining the first differential data between the current frame image and the first reference frame image, and the motion detection results of multiple frames of associated images of the current frame image, a number of target motion pixel points are found from the current frame image, wherein the current frame image, the first reference frame image, and the associated images are all captured by the same capturing device, the current frame image and the first reference frame image are captured at different times, the associated image is the current frame image or the number of frames different from the current frame image is less than a preset number of frames, the target motion pixel point is a first motion pixel point determined to belong to the motion target based on the first differential data, and is a second motion pixel point determined to belong to the motion target based on the motion detection result; Enhancement processing is performed on a number of target moving pixel points in the current frame image to obtain an enhanced image.

2. The method according to claim 1, characterized in that The first differential data includes a differential value between each pixel in the current frame image and a corresponding pixel in the first reference frame image; The step of determining the first moving pixel point based on the first differential data includes: Acquire a pixel point whose differential value meets a first motion requirement as the first motion pixel point; or, Pixel points whose differential values ​​satisfy the first motion requirement are obtained as candidate motion pixel points, and for each candidate motion pixel point, the number of candidate motion pixel points in the motion target cluster to which the candidate motion pixel point belongs is counted, and the candidate motion pixel point whose number of candidate motion pixel points in the motion target cluster to which it belongs is greater than a first threshold is taken as the first motion pixel point.

3. The method according to claim 2, characterized in that The first motion requirement includes that the characterization value of the differential value is greater than a second threshold, wherein the second threshold is a preset value or a preset ratio of a maximum characterization value, the characterization value is an absolute value of the differential value, or a value obtained by filtering the absolute value of the differential value, and the maximum characterization value is a maximum value among the characterization values ​​of the differential values ​​included in the first differential data; And / or, before counting the number of candidate motion pixel points in the moving target cluster to which the candidate motion pixel points belong, the method further includes: Adopting a region growing method to aggregate the candidate motion pixels into at least one region, wherein each region is regarded as a motion target cluster; or, Using each of the candidate motion pixel points to form at least one connected domain, wherein each of the connected domains serves as a motion target cluster; or, The neighborhood window of each candidate motion pixel point is used as the motion target cluster of each candidate motion pixel point.

4. The method according to claim 1, characterized in that The step of determining the second motion pixel point based on the motion detection result comprises: Using the motion detection results of each of the associated images, counting the number of times each pixel in the multiple frames of associated images is detected as belonging to a moving target; The pixel points whose number of times is greater than the third threshold are determined as second motion pixel points.

5. The method according to any one of claims 1 to 4, characterized in that: The step of synthesizing the first differential data between the current frame image and the first reference frame image and the motion detection results of multiple frames of associated images of the current frame image to find a plurality of target motion pixels from the current frame image includes: Based on first differential data between the current frame image and the first reference frame image, determining a plurality of first moving pixel points belonging to the moving target from the current frame image; Based on the motion detection result, a plurality of second motion pixel points are further determined from the plurality of first motion pixel points to serve as the plurality of target motion pixel points.

6. The method according to claim 5, characterized in that The determining, based on first differential data between the current frame image and the first reference frame image, a plurality of first moving pixel points belonging to the moving target from the current frame image comprises: Acquire the pixel point whose differential value meets the first motion requirement as the candidate motion pixel point; For each of the candidate motion pixel points, counting the number of candidate motion pixel points in the motion target cluster to which the candidate motion pixel point belongs, and taking the candidate motion pixel point whose number of candidate motion pixel points in the motion target cluster to which the candidate motion pixel point belongs is greater than a first threshold as the first motion pixel point; The further determining, based on the motion detection result, a plurality of second motion pixel points from the plurality of first motion pixel points as the plurality of target motion pixel points comprises: Performing an expansion operation on a first moving target region formed by the plurality of first moving pixel points to obtain a second moving target region, and updating the pixel points in the second moving target region to the first moving pixel points; Setting invalid values ​​for pixel points in areas other than the second moving target area in each of the associated images; Performing motion detection on each associated image after the invalid value is set, to obtain a motion detection result of each associated image; Using the motion detection results of the associated images, counting the number of times each of the first motion pixels is detected as belonging to a moving target in the multiple frames of associated images; The first moving pixel point whose number of times is greater than the third threshold is determined as the second moving pixel point and used as the target moving pixel point.

7. The method according to claim 1, characterized in that Before performing enhancement processing on a plurality of target moving pixels in the current frame image to obtain an enhanced image, the method further includes: A dilation operation is performed on a third moving target region formed by the plurality of target moving pixel points to obtain a fourth moving target region, wherein the pixel points of the fourth moving target region are updated to target moving points for performing the enhancement processing.

8. The method according to claim 1, characterized in that: The step of performing enhancement processing on a plurality of target moving pixel points in the current frame image to obtain an enhanced image includes: Adjusting the differential value corresponding to each of the target motion pixel points to obtain an enhanced value of each of the target motion pixel points; The pixel value of each of the target motion pixel points in the current frame image is superimposed with the enhancement value of each of the target motion pixel points to obtain the enhanced image.

9. The method according to claim 8, characterized in that The step of adjusting the differential value corresponding to each of the target motion pixels to obtain an enhanced value of each of the target motion pixels includes: Using the first global enhancement coefficient, respectively adjusting the differential value corresponding to each of the target motion pixel points to obtain an enhancement value of each of the target motion pixel points; or, The differential value corresponding to each of the target motion pixel points is adjusted using the local enhancement coefficient corresponding to each of the target motion pixel points to obtain the enhancement value of each of the target motion pixel points.

10. The method according to claim 9, characterized in that The local enhancement coefficient of the target motion pixel is the ratio of the differential value or its absolute value corresponding to the target motion pixel to a fourth threshold, the fourth threshold is a preset threshold or an adaptive threshold, and the adaptive threshold is any one of the following or is determined based on any one of the following: a preset ratio of the maximum characterization value, the number of motion pixels in the motion target cluster where the target motion pixel is located, the number of times the target motion pixel is detected as belonging to a motion target in the multi-frame associated image, a discrete degree characterization value of the differential value of each pixel within a first range of the target motion pixel, and a difference between a maximum differential value and a minimum differential value of each pixel within a second range of the target motion pixel; And / or, using the first global enhancement coefficient to adjust the differential value corresponding to each of the target motion pixels to obtain the enhancement value of each of the target motion pixels, including: Multiplying the first global enhancement coefficient by the difference value corresponding to each of the target motion pixels to obtain an enhancement value of each of the target motion pixels; And / or, respectively adjusting the differential value corresponding to each of the target motion pixels by using the local enhancement coefficient corresponding to each of the target motion pixels to obtain the enhancement value of each of the target motion pixels, including: The local enhancement coefficient corresponding to each target motion pixel point is multiplied by the difference value corresponding to each target motion pixel point to obtain the enhancement value of each target motion pixel point; or, For each of the target motion pixel points, a local enhancement coefficient corresponding to the target motion pixel point is multiplied by a second global enhancement coefficient and a differential value corresponding to the target motion pixel point to obtain an enhancement value of the target motion pixel point.

11. The method according to claim 8, characterized in that Before adjusting the differential value corresponding to each of the target motion pixels, the method further includes: The filtered differential data of the current frame image and the filtered differential data of the second reference frame image are time-domain smoothed, and the differential values ​​corresponding to the target motion pixel points in the time-domain smoothing result are used as the differential values ​​corresponding to the target motion pixel points for the adjustment, wherein the filtered differential data of the current frame image is obtained by filtering the first differential data or its absolute value between the current frame image and the first reference frame image, and the filtered differential data of the second reference frame image is obtained by filtering the second differential data or its absolute value between the second reference frame image and the third reference frame image.

12. The method according to claim 1, characterized in that The method further comprises: Each pixel point in the current image frame and the corresponding pixel point in the first reference frame image are combined into a plurality of pixel point pairs; or, the current frame image and the first reference frame image are pre-processed respectively, and each pixel point in the pre-processed current image frame and the corresponding pixel point in the pre-processed first reference frame image are combined into a plurality of pixel point pairs, wherein the pre-processing includes filtering processing; Obtain the pixel value difference of each of the pixel point pairs to obtain the first differential data.

13. The method according to claim 12, characterized in that The step of combining each pixel point in the current image frame with a corresponding pixel point in the first reference frame image to form a plurality of pixel point pairs includes: Determining a moving distance of the photographing device between the current frame image and the first reference frame image based on pixel data of the current frame image and the first reference frame image; The sum of the first coordinate of the pixel point in the current image frame and the moving distance is used as the second coordinate, and the pixel point at the first coordinate in the current image frame and the pixel point at the second coordinate in the first reference frame image form the pixel point pair.

14. The method according to claim 13, characterized in that The determining, based on pixel data of the current frame image and the first reference frame image, a moving distance of the photographing device between the current frame image and the first reference frame image comprises: Determine the maximum movement amplitude Xmax of the shooting device in the first direction, take values ​​of the movement distance dx in the first direction from -Xmax to Xmax in sequence, and calculate the sum of multiple column difference values ​​between the current frame image and the first reference frame image when the movement distance dx in the first direction takes each value, wherein each column difference value represents the difference between the sum of pixel values ​​of a first column j in the current frame image and the sum of pixel values ​​of a corresponding second column i in the first reference frame image, where i is the sum of j and dx; Determine the maximum movement amplitude Ymax of the camera in the second direction, take values ​​of the movement distance dy in the second direction from -Ymax to Ymax in sequence, and calculate the sum of multiple row difference values ​​between the current frame image and the first reference frame image when the movement distance dy in the second direction takes each value, wherein each row difference value represents the difference between the sum of pixel values ​​of a first row m in the current frame image and the sum of pixel values ​​of a corresponding second row n in the first reference frame image, where n is the sum of m and dy; The moving distance is determined based on the sum of the column difference values ​​and the sum of the row difference values.

15. An electronic device, characterized in that: including memory and processor, The memory stores program instructions; The processor is used to execute the program instructions stored in the memory to implement the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the method according to any one of claims 1 to 14.

Images