Method for camera image turbulence mitigation based on jitter information

Abstract

The application provides an image processing method, which comprises: when turbulence detection is required, determining that a PTZ camera is not in a motion state; based on shaking information corresponding to the PTZ camera, performing anti-shaking processing on a first image collected by the PTZ camera to obtain an anti-shaking image corresponding to the first image, determining a motion vector diagram corresponding to the anti-shaking image, and storing the motion vector diagram in a designated storage medium; the shaking information is collected by a shaking detection sensor; a motion vector diagram corresponding to N-1 frames before the first image is obtained; based on the N motion vector diagrams, a turbulence influence factor corresponding to the anti-shaking image is determined, a turbulence reduction operation is performed on the anti-shaking image based on the turbulence influence factor, and a turbulence-reduced image corresponding to the first image is obtained. Through the technical scheme of the application, the turbulence effect of the anti-shaking image is reduced, and the image quality is significantly improved.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image processing method. Background Technology

[0002] Atmospheric turbulence causes uneven air density, resulting in different optical refractive indices in different parts of a hot air mass. This causes direct light rays passing through the hot air mass to be deflected, bent, and scattered, leading to distortion, deformation, and blurring of observed objects. This is known as the turbulence effect. In other words, images captured by cameras exhibit turbulence effects, resulting in severe blurring of target objects, poor imaging performance, and poor image quality. Summary of the Invention

[0003] According to one aspect of this disclosure, an image processing method is applied to a PTZ camera, the method comprising:

[0004] When turbulence detection is required, it is determined that the PTZ camera is not in motion, wherein the motion state refers to any of the states of pitch, yaw or zoom switching.

[0005] Based on the jitter information corresponding to the PTZ camera, the first image captured by the PTZ camera is subjected to jitter stabilization processing to obtain the jitter-stabilized image corresponding to the first image, and the motion vector diagram corresponding to the jitter-stabilized image is determined and stored in a specified storage medium; wherein, the jitter information is collected by a jitter detection sensor;

[0006] Obtain the motion vector diagrams corresponding to the N-1 frames preceding the first image;

[0007] Based on the N frames of motion vector graphics, the turbulence influence factor corresponding to the image after image stabilization is determined. Based on the turbulence influence factor, a turbulence reduction operation is performed on the image after image stabilization to obtain the image after turbulence reduction corresponding to the first image. The turbulence reduction operation is used to reduce the turbulence effect in the image after image stabilization.

[0008] According to another aspect of this disclosure, a PTZ camera with turbulence detection includes:

[0009] A shake detection sensor is used to determine the shake information of the PTZ camera;

[0010] The processor is used to execute the following methods:

[0011] In response to an externally input turbulence detection command, determine whether the PTZ camera is in motion, wherein the motion state represents any one of pitch motion, yaw motion, or magnification switching:

[0012] the PTZ camera is in a motion state, the turbulence detection instruction is not executed; and

[0013] the PTZ camera is not in a motion state, the turbulence detection instruction is executed, including:

[0014] obtaining an N-frame image sequence, wherein a collection time of a last frame in the image sequence is not earlier than a time when the PTZ camera receives the turbulence detection instruction;

[0015] based on the jitter information, performing a de-jittering processing on the N-frame image sequence to obtain N-frame de-jittered images, and determining N-frame motion vector maps corresponding to the N-frame de-jittered images;

[0016] determining a motion vector length target value based on the N-frame motion vector maps;

[0017] determining a threshold interval in which the motion vector length target value is located, and determining a turbulence intensity corresponding to the threshold interval; wherein the threshold interval and the turbulence intensity have a one-to-one correspondence;

[0018] determining a turbulence influence factor based on the turbulence intensity and a pixel size of the PTZ camera; and

[0019] based on the turbulence influence factor, performing a turbulence alleviation operation on the de-jittered images to obtain turbulence alleviated images, the turbulence alleviation operation being used to alleviate the turbulence effect in the de-jittered images.

[0020] As can be seen from the above technical solutions, in the embodiments of the present application, the turbulence influence factor can be determined based on the N-frame motion vector maps, the turbulence alleviation operation is performed on the de-jittered images based on the turbulence influence factor to obtain the turbulence alleviated images, so as to alleviate the turbulence effect in the de-jittered images, alleviate the turbulence effect of the de-jittered images, make the target object clear, and have good imaging performance and good image quality. Based on the characteristics of the turbulence on the image, the motion interference is removed, the turbulence phenomenon is effectively improved, and the image quality is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the description of the embodiments of the present application or the prior art will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can also be obtained by those skilled in the art based on these drawings.

[0022] Figure 1 is a flowchart of an image processing method in an embodiment of the present application;

[0023] Figure 2is a structural schematic diagram of a camera in an embodiment of the present application;

[0024] Figure 3 is a functional schematic diagram of a main processing device in an embodiment of the present application;

[0025] Figure 4 is a flow schematic diagram of a turbulence detection operation in an embodiment of the present application;

[0026] Figure 5 is a geometric model schematic diagram of a PTZ spherical coordinate system in an embodiment of the present application;

[0027] Figure 6 is a geometric model schematic diagram of a PTZ camera magnification in an embodiment of the present application;

[0028] Figure 7 is a schematic diagram of an image anti-shake process in an embodiment of the present application;

[0029] Figure 8 is a storage schematic diagram of a motion vector map in an embodiment of the present application;

[0030] Figure 9A is a flow schematic diagram of a turbulence mitigation operation in an embodiment of the present application;

[0031] Figure 9B is a flow schematic diagram of an image enhancement operation in an embodiment of the present application. DETAILED DESCRIPTION

[0032] The terminology used in the present application merely describes specific embodiments, and is not intended to limit the present application. The singular forms "a," "an," and "the" used in the present application and claims are intended to include plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in the present application means any or all possible combinations of one or more associated listed items.

[0033] It should be understood that, although the terms first, second, third, etc. can be used in this application to describe various information, these terms are not intended to limit the information. These terms are only used to distinguish one piece of information from another piece of information. For example, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information, without departing from the scope of the present application. In addition, depending on the context, the word "if" used herein can be interpreted as "when" or "upon determining" or "in response to determining".

[0034] This application proposes an image processing method that can be applied to cameras (such as PTZ cameras, pan-tilt cameras, etc., taking PTZ cameras as an example). See [link to relevant documentation]. Figure 1 As shown, the method includes:

[0035] Step 101: When turbulence detection is required, determine whether the PTZ camera is in motion. If not, i.e., the PTZ camera is not in motion, proceed to step 102. Here, "motion state" refers to any of the following: pitch, roll, or zoom switching.

[0036] Step 102: If it is determined that the PTZ camera is not in motion, perform image stabilization processing on the first image acquired by the PTZ camera based on the shake information corresponding to the PTZ camera to obtain the image after image stabilization, and determine the motion vector map corresponding to the image after image stabilization, and store the motion vector map in the specified storage medium; wherein, the PTZ camera may include a shake detection sensor, and the shake information can be acquired by the shake detection sensor, that is, the shake information is acquired through the shake detection sensor.

[0037] Step 103: Obtain the motion vector maps corresponding to the N-1 frames preceding the first image. These N-1 frames of motion vector maps and the motion vector map corresponding to the first image together form the N-frame motion vector map. In other words, the N-frame motion vector map can include the motion vector map corresponding to the first image and the motion vector maps corresponding to the N-1 frames of image stabilization images preceding the image stabilization image in the specified storage medium.

[0038] Step 104: Determine the turbulence influence factor corresponding to the image after image stabilization based on the N-frame motion vector map. Perform turbulence reduction operation on the image after image stabilization based on the turbulence influence factor to obtain the turbulence-reduced image corresponding to the first image. Then, the target image corresponding to the first image can be generated based on the turbulence-reduced image.

[0039] For example, images captured by a camera exhibit turbulence. Even after image stabilization, the stabilized image still exhibits turbulence. Turbulence mitigation is used to reduce the turbulence effect in the stabilized image. Since turbulence interferes with the image, mitigating the turbulence effect in the stabilized image reduces this interference, resulting in better image quality.

[0040] For example, the horizontal rotation P value, vertical rotation T value, and magnification Z value corresponding to the PTZ camera can be obtained; if at least one of the horizontal rotation P value, vertical rotation T value, and magnification Z value changes, it is determined that the PTZ camera is in motion. Otherwise, if none of the horizontal rotation P value, vertical rotation T value, and magnification Z value changes, it is determined that the PTZ camera is not in motion.

[0041] For example, determining the turbulence influence factor corresponding to the image after the anti-shake based on the N-frame motion vector diagram can include but is not limited to: determining a motion vector length target value based on the N-frame motion vector diagram; wherein the motion vector length target value is the sum of the motion vector lengths corresponding to the N-frame motion vector diagram; for each motion vector diagram, the motion vector diagram includes motion vectors corresponding to a plurality of image blocks, and the motion vector length corresponding to the motion vector diagram is determined based on the length value of the motion vector corresponding to each image block. Then, determine the threshold interval in which the motion vector length target value is located, and determine the turbulence intensity corresponding to the threshold interval; wherein the threshold interval and the turbulence intensity have a one-to-one correspondence. After obtaining the turbulence intensity, the quotient value of the turbulence intensity and the pixel size of the PTZ camera can be determined as the turbulence influence factor.

[0042] For example, performing turbulence mitigation operation on the image after the anti-shake based on the turbulence influence factor to obtain the turbulence mitigation image corresponding to the first image can include but is not limited to: determining an initial residual image based on the image after the anti-shake and a reference image, and the reference image can be the previous frame of the image after the anti-shake. Perform large-scale filtering on the initial residual image to obtain a large-scale residual image, and perform small-scale filtering on the initial residual image to obtain a small-scale residual image. Based on the turbulence influence factor, the preset scale factor, the large-scale residual image and the small-scale residual image, determine the target residual image corresponding to the first image. Based on the target residual image, the image after the anti-shake and the reference image, determine the turbulence mitigation image.

[0043] For example, based on the turbulence influence factor, the preset scale factor, the large-scale residual image and the small-scale residual image, determining the target residual image corresponding to the first image can include but is not limited to: determining a first weight coefficient corresponding to the large-scale residual image and a second weight coefficient corresponding to the small-scale residual image based on the turbulence influence factor and the preset scale factor; wherein the larger the turbulence influence factor, the larger the first weight coefficient, and the smaller the second weight coefficient. Perform weighted operation based on the large-scale residual image, the first weight coefficient, the small-scale residual image and the second weight coefficient to obtain the target residual image.

[0044] For example, the first weight coefficient can be the product value between the turbulence influence factor and the preset scale factor, and the second weight coefficient can be the difference value between 1 and the first weight coefficient.

[0045] For example, based on the turbulence influence factor, the preset scale factor, the large-scale residual image and the small-scale residual image, determining the target residual image corresponding to the first image can include but is not limited to:

[0046] The following formula is used to determine the target residual image corresponding to the first image:

[0047] fusDiff = smallDiff * (1 - K * ratio) + largeDiff * K * ratio

[0048] wherein K represents a turbulence influence factor, ratio represents a preset ratio factor, smallDiff represents a small-scale residual image, largeDiff represents a large-scale residual image, and fusDiff represents a target residual image.

[0049] For example, determining a turbulence mitigated image based on the target residual image, the post-stabilization image, and the reference image can include, but is not limited to, determining a motion coefficient based on an average noise size of the target residual image and the post-stabilization image; determining a third weight coefficient corresponding to the post-stabilization image and a fourth weight coefficient corresponding to the reference image based on the motion coefficient, and performing a weighting operation based on the post-stabilization image, the third weight coefficient, the reference image, and the fourth weight coefficient to obtain the turbulence mitigated image.

[0050] For example, after performing the turbulence mitigation operation on the post-stabilization image based on the turbulence influence factor to obtain the turbulence mitigated image corresponding to the first image, a target image corresponding to the first image can also be generated based on the turbulence mitigated image. For example, an image enhancement factor is determined based on the turbulence influence factor and a target distance, the target distance being a distance between a target object in the first image and the PTZ camera; wherein the larger the turbulence influence factor, the larger the image enhancement factor, and the larger the target distance, the larger the image enhancement factor. A high-frequency detail image corresponding to the first image is generated based on the turbulence mitigated image, and an enhanced image is generated based on the high-frequency detail image and the image enhancement factor; and the target image is generated based on the turbulence mitigated image and the enhanced image.

[0051] For example, storing the motion vector map into the designated storage medium can include, but is not limited to, determining a motion vector variance corresponding to the motion vector map, if the motion vector variance is less than a first threshold, then the motion vector map is prohibited from being stored into the designated storage medium, and if the motion vector variance is not less than the first threshold, then the motion vector map is stored into the designated storage medium. Alternatively, a motion vector length corresponding to the motion vector map is determined, if the motion vector length is greater than a second threshold, then the motion vector map is prohibited from being stored into the designated storage medium, and if the motion vector length is not greater than the second threshold, then the motion vector map is stored into the designated storage medium. Alternatively, a motion vector variance and a motion vector length corresponding to the motion vector map are determined, if the motion vector variance is less than a first threshold and / or the motion vector length is greater than a second threshold, then the motion vector map is prohibited from being stored into the designated storage medium, and if the motion vector variance is not less than the first threshold and the motion vector length is not greater than the second threshold, then the motion vector map is stored into the designated storage medium.

[0052] Exemplarily, the post-stabilized image can include a plurality of image blocks, the motion vector map can include a plurality of motion vectors corresponding to the plurality of image blocks; and storing the motion vector map into the specified storage medium can include, but is not limited to: for each motion vector, if a length corresponding to the motion vector is greater than a third threshold, the motion vector is removed from the motion vector map; if the length corresponding to the motion vector is not greater than the third threshold, the motion vector is retained in the motion vector map, to obtain a processed motion vector map (i.e., each motion vector is processed as described above); and the processed motion vector map is stored into the specified storage medium.

[0053] Exemplarily, the post-stabilized image can include a plurality of image blocks, the motion vector map can include a plurality of motion vectors corresponding to the plurality of image blocks, the motion vectors correspond to the image blocks one by one, and based on this: for each image block in the post-stabilized image, N motion vectors corresponding to the image block can be determined from N motion vector maps, a turbulence influence factor corresponding to the image block is determined based on the N motion vectors, a turbulence mitigation operation is performed on the image block based on the turbulence influence factor, to obtain a turbulence mitigated image block, and a target image block is generated based on the turbulence mitigated image block; and a target image is generated based on the target image block corresponding to each image block in the post-stabilized image. For the processing process of each image block, refer to the above embodiments.

[0054] As can be seen from the above technical solutions, in the embodiments of the present application, the turbulence influence factor corresponding to the post-stabilized image can be determined based on N motion vector maps, a turbulence mitigated image is obtained by performing a turbulence mitigation operation on the post-stabilized image based on the turbulence influence factor, and a target image corresponding to the first image is generated based on the turbulence mitigated image, so that the turbulence effect in the post-stabilized image is mitigated, the turbulence effect of the post-stabilized image is mitigated, the target object in the target image is clear, the imaging performance is good, and the image quality is good. Based on the characteristics of turbulence on the image, the motion disturbance is removed, the turbulence phenomenon is effectively improved, and the image quality is significantly improved.

[0055] In the embodiments of the present application, a PTZ camera with turbulence detection is provided, which includes:

[0056] A jitter detection sensor is configured to determine jitter information of the PTZ camera.

[0057] A processor is configured to execute the following method:

[0058] In response to an externally input turbulence detection instruction, it is determined whether the PTZ camera is in a motion state, wherein the motion state represents any one of a pitching motion, a panning motion, or a zooming motion:

[0059] When the PTZ camera is in the motion state, the turbulence detection instruction is not executed; and

[0060] When the PTZ camera is not in a motion state, a turbulent flow detection instruction is executed, including:

[0061] An N-frame image sequence is acquired, where the acquisition time of the last frame in the image sequence is not earlier than the time when the PTZ camera receives the turbulent flow detection instruction;

[0062] Based on the jitter information, anti-jitter processing is performed on the N-frame image sequence to obtain N frames of anti-jitter images, and N frames of motion vector graphs corresponding to the N frames of anti-jitter images are determined;

[0063] A motion vector length target value is determined based on the N frames of motion vector graphs;

[0064] A threshold interval in which the motion vector length target value is located is determined, and a turbulent flow intensity corresponding to the threshold interval is determined; the threshold interval and the turbulent flow intensity have a one-to-one correspondence;

[0065] A turbulent flow influence factor is determined based on the turbulent flow intensity and the pixel size of the PTZ camera (i.e., the quotient value of the turbulent flow intensity and the pixel size of the PTZ camera is taken as the turbulent flow influence factor); and

[0066] Based on the turbulent flow influence factor, a turbulent flow mitigation operation is performed on the anti-jitter image to obtain a turbulent flow mitigated image, and the turbulent flow mitigation operation is used to mitigate the turbulent flow effect in the anti-jitter image. For example, the turbulent flow mitigation operation is performed on the anti-jitter image corresponding to the last frame in the image sequence to obtain the turbulent flow mitigated image corresponding to the last frame in the image sequence.

[0067] It should be noted that when each frame of image is acquired by the PTZ camera, the N-frame image sequence corresponding to the frame of image (i.e., the frame of image and the previous N-1 frames of image form an image sequence) can be constructed in the above manner, and then the turbulent flow mitigated image corresponding to the frame of image can be obtained in the above manner.

[0068] The above technical solutions of the embodiments of the present application will be described below in combination with specific application scenarios.

[0069] Atmospheric turbulent flow can cause uneven air density, resulting in different optical refractive indices of different parts of a hot air mass, thereby causing the straight light rays passing through the hot air mass to be deflected, bent and scattered, and further causing the observed object scene to be distorted and blurred, i.e., the turbulent flow effect. In order to process the turbulent flow effect in the image, an image processing method is proposed in the embodiments of the present application, which can eliminate the interference of motion based on the presentation of the characteristics of the turbulent flow on the image, effectively improve the turbulent flow phenomenon, and significantly improve the image quality.

[0070] For example, the image processing method can be applied to a camera, such as a PTZ camera, a pan-tilt-zoom camera, etc., of course, other types of cameras are also possible, which are not limited. Referring toFigure 2 As shown in FIG. 1, a structural schematic diagram of a camera is shown, which can include a lens, a photosensitive element, a shake detection sensor, and a main processing device. The camera can collect an image through the lens and the photosensitive element, which is referred to as an original image (i.e., the first image in the above embodiment, and the original image is taken as an example for subsequent description), and input the original image to the main processing device. The camera can collect shake information through the shake detection sensor, and input the shake information to the main processing device. The main processing device can perform anti-shake processing on the original image based on the shake information to obtain an anti-shake image, and implement an image processing method based on the anti-shake image, i.e., perform operations such as turbulence mitigation on the anti-shake image, so as to obtain a target image corresponding to the original image, which is an image with improved turbulence phenomenon, and is the final output image of the main processing device.

[0071] For example, the input data of the main processing device is the original image and the shake information, based on which, referring to FIG. 2, the main processing device can perform operations such as turbulence detection, turbulence mitigation, and image enhancement. Figure 3 As shown in FIG. 2, the main processing device is used to implement operations such as turbulence detection, turbulence mitigation, and image enhancement. For the turbulence detection operation, in combination with the motion of the camera, the shake of the camera, and the characteristics of turbulence imaging, the turbulence intensity is effectively refined, and a turbulence influence factor corresponding to the turbulence intensity is determined. For the turbulence mitigation operation, the image is subjected to turbulence mitigation based on the turbulence influence factor, so as to restore the image picture. For the image enhancement operation, the image is enhanced to avoid the problem of local unclarity caused by turbulence.

[0072] First, the turbulence detection operation.

[0073] For example, the turbulence intensity can be judged according to the image, and the motion and shake of the device itself can interfere with the detection of the turbulence intensity, such as the motion of the PTZ camera, the motion of the PTZ camera, the installation shake of the PTZ camera or the gimbal camera, and the motion of the foreground in the fixed scene, etc., which can interfere with the detection of the turbulence intensity. Therefore, before judging the turbulence intensity according to the image, preprocessing is needed to eliminate the interference, i.e., to remove the motion and shake of the camera and other interference to avoid interference with the detection of the turbulence intensity.

[0074] When the eye observes a moving object, the scene of the object forms a series of continuously changing images on the retina of the eye, and this series of continuously changing information constantly flows through the retina, which is referred to as optical flow. Since the motion of turbulence is disordered, the optical flow field caused by turbulence tends to be a zero vector on a certain time axis. Since the motion vector diagram can reflect the optical flow field caused by turbulence, the turbulence intensity can be judged based on the motion vector diagram corresponding to the original image.

[0075] Based on the above principle, the flowchart of the turbulence detection operation is shown in FIG. 3. Figure 4As shown, the process includes:

[0076] Step 401, determine whether it is necessary to perform the turbulence detection.

[0077] For example, the user can send a turbulence detection command to the main processor, and the main processor determines that it is necessary to perform the turbulence detection after receiving the turbulence detection command. Alternatively, the main processor can perform the turbulence detection by default. Alternatively, other ways of triggering the turbulence detection can also be used, which are not limited.

[0078] Step 402, when it is necessary to perform the turbulence detection, determine whether the camera is in a motion state, which can also be referred to as an adjustment state, i.e., determine whether the camera is in an adjustment state. If the camera is a PTZ camera, the motion state refers to any state of the pitch motion, the yaw motion, or the zoom ratio switching.

[0079] If yes, step 403 can be performed; if no, step 404 can be performed.

[0080] For example, if the camera is in a motion state, the motion of the camera itself needs to be removed, i.e., the turbulence detection is not performed, and if the camera is not in a motion state, the turbulence detection is performed. Therefore, it is necessary to determine whether the camera is in a motion state. For example, for a PTZ camera, whether the PTZ camera is in a motion state is determined, and the determination method of whether the PTZ camera is in a motion state is not described herein.

[0081] For example, the PTZ camera has jitter during rotation, the lens group of the PTZ camera is unstable during zooming (e.g., the motor lead screw is bent, causing the lens group to move in a non-straight line and to shake; the guide shaft hole and the guide shaft have a large matching amount, causing the lens group to shake; and the lens has internal interference, causing the lens group to be forced to swing), and therefore, the PTZ camera is in a motion state, and the turbulence detection is not performed.

[0082] For example, in order to determine whether the PTZ camera is in a motion state, the horizontal rotation P value, the vertical rotation T value, and the zoom ratio Z value corresponding to the PTZ camera can be obtained. If at least one of the horizontal rotation P value, the vertical rotation T value, and the zoom ratio Z value changes, it is determined that the PTZ camera is in a motion state, and step 403 is performed. Otherwise, if none of the horizontal rotation P value, the vertical rotation T value, and the zoom ratio Z value changes, it is determined that the PTZ camera is not in a motion state, and step 404 is performed.

[0083] Referring to Figure 5As shown in the figure, it is a geometric model diagram of PTZ spherical coordinate system, the position of PTZ camera is the center O point of the hemisphere, the horizontal direction of the spherical coordinate system of the hemisphere with PTZ camera as the center of gravity is composed of mutually perpendicular u and v directions, the vertical direction of the spherical coordinate system is w direction perpendicular to u and v directions, OC is the optical axis of the lens of PTZ camera. Assuming that the three-dimensional coordinates of a certain object in the imaging picture of PTZ camera are (u, v, w), the horizontal rotation P value is the rotation angle of the hemisphere along the horizontal direction with O point (PTZ camera), the vertical rotation T value is the rotation angle of the hemisphere along the vertical direction with O point (PTZ camera), then:

[0084] P = arctan (v / u)

[0085]

[0086] Assuming that the field of view range of PTZ camera is the ground plane, referring to Figure 6 As shown in the figure, it is a geometric model diagram of PTZ camera magnification, line segment BD represents the ground, point A is the installation position of PTZ camera, the installation height of PTZ camera is H, that is, the length of line segment AB is H, which can be a known value, line segment AB is perpendicular to line segment BD, point D is the position of the reference target on the ground when calibrating the magnification, then:

[0087]

[0088] As described above, the horizontal rotation P value, the vertical rotation T value and the magnification Z value can be obtained, after obtaining the horizontal rotation P value, the vertical rotation T value and the magnification Z value, it can be known whether the horizontal rotation P value changes, whether the vertical rotation T value changes and whether the magnification Z value changes.

[0089] Step 403, empty the frame pool.

[0090] For example, the motion vector diagram can be stored by specifying a storage medium (such as memory, hard disk), the storage process is described in the subsequent embodiment, the specified storage medium can be referred to as frame pool, if the camera is in a motion state, the frame pool is emptied in the time period (i.e. the time period in the motion state) without turbulence detection.

[0091] Step 404, when the camera collects each frame of original image (taking one frame of original image as an example for description), the jitter information corresponding to the original image is obtained from the jitter detection sensor.

[0092] Exemplarily, the jitter detection sensor can collect jitter information of the camera in real time, and the collection process is not limited. When the camera collects an original image, jitter information corresponding to the original image can be obtained from the jitter detection sensor, that is, the collection time of the original image is the same as the collection time of the jitter information.

[0093] Step 405: performing anti-shake processing on the original image based on the jitter information to obtain an anti-shake image.

[0094] For example, after obtaining the jitter information corresponding to the original image, the original image can be processed by an image anti-shake algorithm based on the jitter information (the anti-shake processing is used to remove the jitter information in the original image, so as to avoid the interference of jitter on the original image), and an anti-shake image is obtained. In this embodiment, the image anti-shake algorithm is not limited. For example, an example of the image anti-shake algorithm can be seen from Figure 7 As shown in the figure, after obtaining the jitter information, the original image is cropped, enlarged, or the like based on the jitter information, so that the shooting target (i.e., the target object) is always at the center position of the image, and an anti-shake image is obtained.

[0095] Step 406: determining a motion vector map corresponding to the anti-shake image.

[0096] Exemplarily, after obtaining the anti-shake image, a motion vector map corresponding to the anti-shake image can be determined. The motion vector map can include a motion vector of each image block (an image block can be a pixel or can be composed of multiple pixels, and the image block is not limited) in the anti-shake image. The motion vector is the motion speed and motion direction of the image block, and the motion speed and motion direction of the image block are also called an optical flow field. The optical flow field is the instantaneous speed of the pixel motion of the spatial moving object on the observation imaging plane, and is a way of finding the corresponding relationship between the previous frame and the current frame by using the change of the pixel in the time domain and the correlation between adjacent frames, so as to calculate the motion vector of the object between adjacent frames.

[0097] Suppose that the position of point A is (x1, y1) at the tth frame (i.e., the previous anti-shake image), and the position of point A is (x2, y2) at the t+1th frame (i.e., the anti-shake image of the current frame), then the motion of point A can be determined as: (u x ,v y ) = (x2, y2) - (x1, y1), that is, the motion vector of point A. In order to obtain the position of point A at the t+1th frame, an optical flow algorithm such as Lucas-Kanade optical flow algorithm can be used. Based on the optical flow algorithm, the motion speed in the horizontal and vertical motion direction of the previous frame and the next frame is calculated as follows:

[0098]

[0099] u, v represent velocity components of a pixel in a certain block of an image in horizontal and vertical directions, I ix represents the gradient of the i-th pixel in the horizontal direction, I iy represents the gradient of the i-th pixel in the vertical direction, I t represents the difference between two frames. Given an image sequence I 1 , I 2 , I 3 ,..., I m , divide the image into M*N image blocks, calculate the motion vector in each image block, and obtain a set of motion vector maps Map 1 , Map 2 , Map 3 ,..., Map m-1 .

[0100] After obtaining the image after anti-shake, the image after anti-shake can be divided into multiple image blocks (an image block can be one pixel or composed of multiple pixels). For each image block, the motion vector corresponding to the image block can be determined based on the image after anti-shake of the previous frame and the image after anti-shake of the current frame, such as using an optical flow algorithm to determine the motion vector corresponding to the image block. After obtaining the motion vectors corresponding to all image blocks, the motion vectors corresponding to all image blocks can be combined to form a motion vector map.

[0101] Step 407, store the motion vector map into a designated storage medium.

[0102] For example, after obtaining the motion vector map corresponding to the image after anti-shake, the motion vector map can be directly stored into a designated storage medium, or the motion vector map can be stored into a designated storage medium in the following manner, or the motion vector map can be prohibited from being stored into a designated storage medium.

[0103] Method 1, determine the motion vector variance corresponding to the motion vector map. For example, the motion vector map includes motion vectors corresponding to multiple image blocks, and the variance of these motion vectors, i.e., the motion vector variance, is calculated. If the motion vector variance is less than a first threshold, the motion vector map is prohibited from being stored into a designated storage medium, and if the motion vector variance is not less than the first threshold, the motion vector map is stored into a designated storage medium.

[0104] For example, if the camera is shaking, the motion of each pixel in the image after the anti-shake should be consistent. The motion vector variance corresponding to the motion vector map is calculated, and the size relationship between the motion vector variance and the first threshold value is compared. If the motion vector variance is less than the first threshold value, it is considered that the camera is shaking, the motion vector map is discarded, the motion vector map is not stored in the designated storage medium, the turbulence reduction is not performed on the image after the anti-shake, and the image after the anti-shake is directly taken as the target image, and the target image is output.

[0105] Method 2, the length of the motion vector corresponding to the motion vector map is determined. For example, the motion vector map includes a plurality of motion vectors corresponding to a plurality of image blocks, the length value of each motion vector is calculated, the average of the length values of all motion vectors is taken as the length of the motion vector, or the maximum of the length values of all motion vectors is taken as the length of the motion vector, or the minimum of the length values of all motion vectors is taken as the length of the motion vector. If the length of the motion vector is greater than the second threshold value, the motion vector map is prohibited from being stored in the designated storage medium, and if the length of the motion vector is not greater than the second threshold value, the motion vector map is stored in the designated storage medium.

[0106] Method 3, the motion vector variance and the length of the motion vector corresponding to the motion vector map are determined. If the motion vector variance is less than the first threshold value and / or the length of the motion vector is greater than the second threshold value, the motion vector map is prohibited from being stored in the designated storage medium, and the turbulence reduction is not performed on the image after the anti-shake, and the image after the anti-shake can be directly taken as the target image. If the motion vector variance is not less than the first threshold value, and the length of the motion vector is not greater than the second threshold value, the motion vector map can be stored in the designated storage medium.

[0107] Method 4, the image after the anti-shake can include a plurality of image blocks, and the motion vector map includes a plurality of motion vectors corresponding to the plurality of image blocks, that is, each image block corresponds to a motion vector, and the image block and the motion vector correspond one by one. For each motion vector in the motion vector map, the length corresponding to the motion vector can be determined. Assuming that the motion vector is Map x =(u x ,v y ), the length corresponding to the motion vector is On this basis, if the length corresponding to the motion vector is greater than a third threshold, the motion vector can be removed from the motion vector map, and if the length corresponding to the motion vector is not greater than the third threshold, the motion vector can be retained in the motion vector map. After the above processing is performed on each motion vector, if at least one motion vector is retained, a processed motion vector map can be obtained, and the processed motion vector map is stored in a specified storage medium. The motion vector map can include all motion vectors or part of the motion vectors corresponding to a plurality of image blocks. After the above processing is performed on each motion vector, if all motion vectors are removed, the motion vector map is prohibited from being stored in the specified storage medium.

[0108] For example, if there is a large motion (i.e., the motion speed is greater than a threshold) in the de-shaken image, such as vehicle motion, the length corresponding to the motion vector of the image block where the large motion is located will exceed the third threshold. Therefore, the part of the large motion can be removed, i.e., the motion vector is removed from the motion vector map.

[0109] Mode 5, based on the variance of the motion vectors corresponding to the motion vector map and the length corresponding to the motion vector of each image block, it is decided whether to store the motion vector map in a specified storage medium or prohibit the motion vector map from being stored in the specified storage medium. Referring to Figure 8 The storage diagram of the motion vector map is shown.

[0110] Step 801, determine the variance of the motion vectors corresponding to the motion vector map.

[0111] Step 802, the motion vector map includes a plurality of motion vectors corresponding to a plurality of image blocks. For each motion vector in the motion vector map, determine the length corresponding to the motion vector.

[0112] Step 803, determine whether the variance of the motion vectors is less than a first threshold.

[0113] If yes, perform step 804, and if no, perform step 805.

[0114] Step 804, prohibit the motion vector map from being stored in a specified storage medium, and do not perform turbulence reduction on the de-shaken image. The de-shaken image is directly taken as a target image, and the target image is output.

[0115] Step 805, for each motion vector in the motion vector map, if the length corresponding to the motion vector is greater than a third threshold, the motion vector is removed from the motion vector map, and if the length corresponding to the motion vector is not greater than the third threshold, the motion vector is retained in the motion vector map to obtain a processed motion vector map.

[0116] Step 806, store the processed motion vector map in a specified storage medium.

[0117] At this point, the motion vector map is stored in the designated storage medium after step 407 is completed.

[0118] Step 408: Determine whether the number of motion vector maps in the designated storage medium reaches a preset number. If not, the turbulence reduction is not performed on the post-stabilization image, and the post-stabilization image is directly taken as the target image and output. If yes, the turbulence reduction is performed on the post-stabilization image, and step 409 is continued. The preset number can be configured according to experience, for example, the preset number can be greater than or equal to N.

[0119] Step 409: Determine the turbulence influence factor corresponding to the post-stabilization image based on N frames of motion vector maps.

[0120] For example, to determine the turbulence influence factor corresponding to the post-stabilization image, the following steps can be taken:

[0121] Step 4091: Obtain N frames of motion vector maps, which can include the motion vector map corresponding to the post-stabilization image and the motion vector maps corresponding to the N-1 frames of post-stabilization images before the post-stabilization image in the designated storage medium. For example, N-1 frames of motion vector maps are selected from the designated storage medium, and the N-1 frames of motion vector maps are combined with the current motion vector map to obtain N frames of motion vector maps.

[0122] Wherein, N is a positive integer, which can be configured according to experience, such as 20, 30, etc., and is not limited.

[0123] For example, assuming that the post-stabilization image (i.e., the post-stabilization image corresponding to the current frame of original image) is the 20th frame of post-stabilization image, the motion vector maps corresponding to the 1st-19th frames of post-stabilization image can be selected from the designated storage medium, and then the motion vector maps corresponding to the 1st-19th frames of post-stabilization image and the motion vector map corresponding to the 20th frame of post-stabilization image are combined to obtain N frames of motion vector maps.

[0124] Step 4092: Determine the target value of the motion vector length based on N frames of motion vector maps.

[0125] In a possible implementation, a length of a motion vector corresponding to each motion vector map can be determined, and a sum of lengths of motion vectors corresponding to N motion vector maps is determined as the motion vector length target value. The motion vector map can include motion vectors corresponding to a plurality of image blocks. To determine the length of the motion vector corresponding to the motion vector map, an average of length values of all motion vectors can be calculated as the length of the motion vector, or a maximum of length values of all motion vectors can be calculated as the length of the motion vector, or a minimum of length values of all motion vectors can be calculated as the length of the motion vector. Of course, the above are only examples, and the determination of the length of the motion vector is not limited in this way.

[0126] In another possible implementation, N motion vector maps can be summed to obtain a target motion vector map, and then a length of a motion vector corresponding to the target motion vector map is determined, and the length of the motion vector corresponding to the target motion vector map is determined as the motion vector length target value.

[0127] In this way, the length of the motion vector corresponding to the target motion vector map is determined as the motion vector length target value. In this case, considering that the motion caused by the turbulence phenomenon is regressive, the length of the motion vector is close to 0, and therefore, the N motion vector maps can be summed, for example, the motion vector map of the m-th frame and the motion vector maps of the m-N+1-th to m-1-th frames in the specified storage medium (for example, a DDR) are summed to obtain a summed motion vector map, and a length of the summed motion vector map is calculated, which participates in the calculation of the turbulence intensity.

[0128] In step 4093, a threshold interval in which the motion vector length target value is located is determined, and a turbulence intensity corresponding to the threshold interval is determined. The threshold interval and the turbulence intensity have a one-to-one correspondence.

[0129] For example, a correspondence between the threshold interval and the turbulence intensity can be preconfigured, and Table 1 shows an example of the correspondence, which is not limited in this regard and can be configured as needed. In Table 1, Thr1, Thr2, Thr3, and Thr4 increase in turn, and K1, K2, K3, and K4 decrease in turn, that is, the larger the threshold value, the smaller the turbulence intensity, and the smaller the threshold value, the larger the turbulence intensity.

[0130] Table 1

[0131] Turbulence intensity Threshold interval [K1] (0, Thr1) <![CDATA[K2]]> [Thr1, Thr2) [K3] [Thr2, Thr3) [K4] [Thr3, Thr4) … …

[0132] After the target value of the motion vector length is obtained, the target value of the motion vector length can be used to query the corresponding relationship shown in Table 1 to obtain the threshold interval in which the target value of the motion vector length is located, and then the threshold interval can be used to query the corresponding relationship shown in Table 1 to obtain the turbulence intensity corresponding to the threshold interval. For example, if the threshold interval is [Thr2, Thr3), the turbulence intensity corresponding to the threshold interval can be K3.

[0133] In step 4094, the turbulence influence factor is determined based on the turbulence intensity and the pixel size of the camera. For example, the quotient of the turbulence intensity and the pixel size can be used as the turbulence influence factor.

[0134] For example, it is assumed that the turbulence influence factor is K, the pixel size is d(um)*d(um), and the larger the pixel size, the smaller the influence of turbulence. Therefore, the turbulence influence factor K can be expressed as:

[0135]

[0136] wherein ‖Map x ‖<Thr1 indicates that the target value of the motion vector length is located in the threshold interval (0, Thr1), ‖Map x ‖<Thr2 indicates that the target value of the motion vector length is located in the threshold interval [Thr1, Thr2), and so on. The turbulence influence factor K is a factor positively related to turbulence and is an influence factor of the turbulence removal intensity.

[0137] At this point, step 409 is completed, and the turbulence influence factor corresponding to the image after the shake is removed is obtained. After the turbulence influence factor is obtained, the turbulence detection process is completed, and the turbulence mitigation operation is performed based on the turbulence influence factor.

[0138] Second, the turbulence mitigation operation.

[0139] After the image after the shake is removed and the turbulence influence factor are obtained, the turbulence mitigation operation can be performed on the image after the shake is removed based on the turbulence influence factor to obtain an image after the turbulence is mitigated, thereby mitigating the turbulence phenomenon. In the turbulence mitigation operation, the turbulence influence factor can be substituted into the residual image to obtain a motion coefficient with turbulence information, so that different inter-frame smoothing processing intensities are used. Inter-frame smoothing processing is based on the self-similarity of a video stream in the time dimension to distinguish motion information from background information and restore a real and clear image. For example, the residual image of the current frame (obtained by subtracting the current frame from the historical frame, containing motion information and noise information) can be obtained, the motion coefficient can be obtained by performing motion decision on the residual image, and the current frame and the historical frame can be synthesized (i.e., multi-frame smoothing) according to the motion coefficient. For positions with slight motion, the pixels in the historical frame are preferred, and for positions with intense motion, the pixels in the current frame are preferred.

[0140] Based on the above principle, the flowchart of the turbulence mitigation operation is shown in FIG. 10 Figure 9A The process includes:

[0141] Step 901, determine an initial residual image based on the de-shake image and a reference image, the reference image being a previous de-shake image of the de-shake image, and the difference between the de-shake image and the reference image being the initial residual image.

[0142] Step 902, perform large-scale filtering on the initial residual image to obtain a large-scale residual image, and perform small-scale filtering on the initial residual image to obtain a small-scale residual image.

[0143] For example, due to the large difference in sparsity between the motion information and the noise information, the motion information is more concentrated and has a larger area, while the noise information is more discrete. According to the above characteristics, small-scale filtering can remove a small part of the noise while leaving more real motion information. Large-scale filtering can remove most of the noise, but the motion information may be false or lost. Turbulence is more discrete on the image, and is suitable for removing part of the turbulence information using large-scale filtering. Therefore, the initial residual image can be filtered by large-scale filtering to obtain a large-scale residual image, and the initial residual image can be filtered by small-scale filtering to obtain a small-scale residual image.

[0144] Step 903, determine a target residual image corresponding to the original image based on the turbulence influence factor, the preset ratio factor, the large-scale residual image, and the small-scale residual image.

[0145] For example, the first weight coefficient corresponding to the large-scale residual image and the second weight coefficient corresponding to the small-scale residual image can be determined based on the turbulence influence factor and the preset ratio factor. If the turbulence influence factor is larger, the first weight coefficient is larger, and the second weight coefficient is smaller. The large-scale residual image, the first weight coefficient, the small-scale residual image, and the second weight coefficient are weighted to obtain the target residual image.

[0146] For example, after obtaining the large-scale residual image and the small-scale residual image, the large-scale residual image and the small-scale residual image can be fused. During image fusion, the large-scale residual image is used as much as possible in the still or turbulence area, i.e., the first weight coefficient corresponding to the large-scale residual image is larger.

[0147] For example, the target residual image corresponding to the original image can be determined by the following formula:

[0148] fusDiff=smallDiff*(1-K*ratio)+largeDiff*K*ratio

[0149] In the above formula, K represents a turbulence influence factor, ratio represents a preset ratio factor, which can be a value configured according to experience and can be between 0 and 1, smallDiff represents a small-scale residual image, (1-K*ratio) represents a second weight coefficient corresponding to the small-scale residual image, largeDiff represents a large-scale residual image, K*ratio represents a first weight coefficient corresponding to the large-scale residual image, and fusDiff represents the target residual image.

[0150] Step 904: determining a motion coefficient based on the average noise size of the target residual image and the image after the anti-shake.

[0151] For example, the motion coefficient can be determined by using the following formula: In the above formula, fusDiff represents the target residual image, math represents the average noise size of the image after the anti-shake, which can be obtained based on the image after the anti-shake, and tfs represents the motion coefficient.

[0152] Step 905: determining a third weight coefficient corresponding to the image after the anti-shake and a fourth weight coefficient corresponding to the reference image based on the motion coefficient, and performing a weighting operation based on the image after the anti-shake, the third weight coefficient, the reference image, and the fourth weight coefficient to obtain a turbulence-reduced image.

[0153] For example, the motion coefficient can guide a smoothing coefficient between the current frame (i.e., the image after the anti-shake) and the historical frame (i.e., the reference image, i.e., the previous frame of the image after the anti-shake), and output a smoothed result. For example, when the motion is relatively slight, the motion coefficient can guide the turbulence-reduced image to tend to use the reference image to reduce the noise jump amplitude of the background part and make the image background area subjectively feel relatively calm. When the motion is relatively severe, the motion coefficient can guide the turbulence-reduced image to tend to use the image after the anti-shake to retain the motion information. For example, the turbulence-reduced image can be determined by using the following formula:

[0154] tnrImg m = tfs*tnrImg m-1 + (1-tfs)*tnrImg m

[0155] In the above formula, tnrImg m-1 represents the reference image, tfs represents the fourth weight coefficient corresponding to the reference image, which is also the motion coefficient, tnrImg m represents the image after the anti-shake, (1-tfs) represents the third weight coefficient corresponding to the image after the anti-shake, and tnrImg m represents the turbulence-reduced image.

[0156] At this point, the turbulence mitigation operation is completed, and an image enhancement operation can be performed based on the turbulence mitigated image.

[0157] Third, the image enhancement operation.

[0158] To improve the image clarity, the image can also be enhanced for better results. The input of the image enhancement is the turbulence mitigated image and the turbulence impact factor. The image enhancement can be performed based on the turbulence impact factor and the turbulence mitigated image to obtain a target image corresponding to the original image, and the target image is output.

[0159] The flowchart of the image enhancement operation can be seen in Figure 9B The process can include:

[0160] Step 911, determine the image enhancement factor based on the turbulence impact factor and the target distance, the target distance being the distance between the target object in the original image and the camera. Wherein the greater the turbulence impact factor, the greater the image enhancement factor, the greater the target distance, the greater the image enhancement factor.

[0161] For example, the greater the magnification, the more obvious the reduction of image information caused by turbulence. Therefore, when enhancing the image, the effect of focal length can be considered, and the target distance between the target object and the camera can reflect the effect of focal length. As described above, the image enhancement factor can be determined by the following formula: λ = K * y, in the above formula, K represents the turbulence impact factor, y represents the target distance, and λ represents the image enhancement factor. Wherein the greater the turbulence impact factor, the stronger the image enhancement factor. At the same time, according to different target distances, the greater the magnification when the target distance is greater, and the greater the image enhancement factor.

[0162] Step 912, generate a high-frequency detail image corresponding to the original image based on the turbulence mitigated image.

[0163] For example, the turbulence mitigated image can be convolved with a Gaussian blur kernel to obtain a Gaussian blur image, and the turbulence mitigated image can be subtracted from the Gaussian blur image to obtain a high-frequency detail image. Of course, the above is only an example, as long as the image details can be extracted from the turbulence mitigated image.

[0164] Step 913, generate an enhanced image based on the high-frequency detail image and the image enhancement factor, and generate a target image based on the turbulence mitigated image and the enhanced image, the target image being the final output image.

[0165] For example, the target image can be generated by the following formula: proImg = Img + λ * detailImg, where detailImg represents a high-frequency detail image, λ represents an image enhancement factor, λ * detailImg represents an enhanced image, Img represents the image after turbulence mitigation, and proImg represents the target image. As can be seen from the above formula, the image enhancement factor can be used to control the degree of image enhancement, thereby improving the image effect.

[0166] At this point, the image enhancement operation is completed, and the target image corresponding to the original image is obtained.

[0167] In one possible implementation, during the turbulence detection operation, after the camera captures the original image, the original image can be subjected to anti-shake processing to obtain an anti-shake image, which can include a plurality of image blocks. After determining the motion vector map corresponding to the anti-shake image, the motion vector map can include a plurality of motion vectors corresponding to the plurality of image blocks, and each motion vector corresponds to one image block. Based on this, in step 409, for each image block in the anti-shake image, N motion vectors corresponding to the image block can be determined from the N motion vector maps, and the turbulence influence factor corresponding to the image block can be determined based on the N motion vectors, that is, the turbulence influence factor corresponding to each image block can be obtained.

[0168] During the turbulence mitigation operation, for each image block in the anti-shake image, the turbulence mitigation operation can be performed on the image block based on the turbulence influence factor corresponding to the image block, to obtain a turbulence-mitigated image block, that is, the turbulence-mitigated image block corresponding to each image block in the anti-shake image can be obtained.

[0169] During the image enhancement operation, based on each turbulence-mitigated image block, a target image block can be generated based on the turbulence-mitigated image block, so that a target image can be generated based on the target image block corresponding to each image block in the anti-shake image, that is, all target image blocks are combined to form the target image.

[0170] For the processing process of each image block, please refer to the turbulence detection operation, the turbulence mitigation operation, and the image enhancement operation. The image in these operations can be replaced by the image block, which will not be repeated here.

[0171] Based on the same application concept as the above method, an image processing device is proposed in the embodiments of the present application, which is applied to a PTZ camera. The device can include: a determination module configured to determine that the PTZ camera is not in a motion state when it is necessary to perform turbulent flow detection, wherein the motion state refers to any state of pitch motion, yaw motion, or zoom ratio switching; a processing module configured to perform anti-shake processing on a first image collected by the PTZ camera based on corresponding jitter information of the PTZ camera to obtain an anti-shake image corresponding to the first image, and determine a motion vector map corresponding to the anti-shake image, and store the motion vector map in a designated storage medium; wherein the jitter information is collected by a jitter detection sensor; an acquisition module configured to acquire motion vector maps corresponding to N-1 frames before the first image; and an image enhancement module configured to determine a turbulent flow influence factor corresponding to the anti-shake image based on the N motion vector maps, and perform a turbulent flow mitigation operation on the anti-shake image based on the turbulent flow influence factor to obtain a turbulent flow mitigated image corresponding to the first image, wherein the turbulent flow mitigation operation is used to mitigate turbulent flow effects in the anti-shake image.

[0172] For example, when at least one of a horizontal rotation P value, a vertical rotation T value, and a zoom ratio Z value corresponding to the PTZ camera changes, it is determined that the PTZ camera is in a motion state.

[0173] For example, when the image enhancement module determines the turbulent flow influence factor corresponding to the anti-shake image based on the N motion vector maps, it is specifically configured to: determine a motion vector length target value based on the N motion vector maps; wherein the motion vector length target value is the sum of motion vector lengths corresponding to the N motion vector maps; for each motion vector map, the motion vector map includes motion vectors corresponding to multiple image blocks, and the motion vector length of the motion vector map is determined based on the length value of the motion vector corresponding to each image block; determine a threshold interval in which the motion vector length target value is located, and determine a turbulent flow intensity corresponding to the threshold interval; wherein the threshold interval and the turbulent flow intensity have a one-to-one correspondence; and determine the quotient value of the turbulent flow intensity and the size of a pixel of the PTZ camera as the turbulent flow influence factor.

[0174] Illustratively, when the image enhancement module determines the turbulence mitigation image corresponding to the first image based on the turbulence influence factor, a preset scale factor, the large-scale residual image and the small-scale residual image, the image enhancement module is specifically configured to: determine a first weight coefficient corresponding to the large-scale residual image and a second weight coefficient corresponding to the small-scale residual image based on the turbulence influence factor and the preset scale factor; wherein, the larger the turbulence influence factor is, the larger the first weight coefficient is, and the smaller the second weight coefficient is; and perform a weighting operation based on the large-scale residual image, the first weight coefficient, the small-scale residual image and the second weight coefficient to obtain the target residual image.

[0175] Illustratively, when the image enhancement module determines the turbulence mitigation image corresponding to the first image based on the turbulence influence factor, a preset scale factor, the large-scale residual image and the small-scale residual image, the image enhancement module is specifically configured to: determine a first weight coefficient corresponding to the large-scale residual image and a second weight coefficient corresponding to the small-scale residual image based on the turbulence influence factor and the preset scale factor; wherein, the larger the turbulence influence factor is, the larger the first weight coefficient is, and the smaller the second weight coefficient is; and perform a weighting operation based on the large-scale residual image, the first weight coefficient, the small-scale residual image and the second weight coefficient to obtain the target residual image.

[0176] Illustratively, the first weight coefficient is a product value between the turbulence influence factor and the preset scale factor, and the second weight coefficient is a difference value between 1 and the first weight coefficient.

[0177] Illustratively, when the image enhancement module determines the turbulence mitigation image corresponding to the first image based on the turbulence influence factor, a preset scale factor, the large-scale residual image and the small-scale residual image, the image enhancement module is specifically configured to: determine the target residual image corresponding to the first image using the following formula: fusDiff = smallDiff * (1 - K * ratio) + largeDiff * K * ratio; wherein, K represents the turbulence influence factor, ratio represents the preset scale factor, smallDiff represents the small-scale residual image, largeDiff represents the large-scale residual image, and fusDiff represents the target residual image.

[0178] Illustratively, when determining the turbulence alleviated image based on the target residual image, the post-anti-shake image and the reference image, the image enhancement module is specifically configured to: determine a motion coefficient based on an average noise size of the target residual image and the post-anti-shake image; determine a third weight coefficient corresponding to the post-anti-shake image and a fourth weight coefficient corresponding to the reference image based on the motion coefficient; and perform a weighting operation based on the post-anti-shake image, the third weight coefficient, the reference image and the fourth weight coefficient to obtain the turbulence alleviated image.

[0179] Illustratively, after the image enhancement module performs the turbulence alleviation operation on the post-anti-shake image based on the turbulence influence factor to obtain the turbulence alleviated image corresponding to the first image, the image enhancement module is further configured to: determine an image enhancement factor based on the turbulence influence factor and a target distance, the target distance being a distance between a target object in the first image and the PTZ camera; wherein the greater the turbulence influence factor, the greater the image enhancement factor, and the greater the target distance, the greater the image enhancement factor; generate a high-frequency detail image corresponding to the first image based on the turbulence alleviated image; and generate an enhanced image based on the high-frequency detail image and the image enhancement factor; and generate a target image corresponding to the first image based on the turbulence alleviated image and the enhanced image.

[0180] Based on the same application concept as the above method, an electronic device (such as the camera in the above embodiment) is provided in the embodiments of the present application, and the electronic device comprises a processor and a machine readable storage medium, the machine readable storage medium stores machine executable instructions which can be executed by the processor; and the processor is configured to execute the machine executable instructions to implement the image processing method disclosed in the above examples of the present application.

[0181] Based on the same application concept as the above method, the embodiments of the present application further provide a machine readable storage medium, and the machine readable storage medium stores a plurality of computer instructions, and the computer instructions can implement the image processing method disclosed in the above examples of the present application when executed by a processor.

[0182] The machine readable storage medium can be any electronic, magnetic, optical or other physical storage device, and can contain or store information such as executable instructions, data, etc. For example, the machine readable storage medium can be a RAM (Random Access Memory), a volatile memory, a non-volatile memory, a flash memory, a storage drive (such as a hard disk drive), a solid state disk, any type of storage disk (such as an optical disk, a DVD, etc.), or similar storage medium, or a combination thereof.

[0183] The systems, apparatuses, modules, or units illustrated in the above embodiments can be implemented by computer entities or products with certain functions. A typical implementation device is a computer, and the specific form of the computer can be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0184] For the sake of description, the above apparatuses are described in various units by functions for description. Of course, the functions of the units can be implemented in one or more software and / or hardware in the implementation of the present application.

[0185] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device implemented in accordance with the flowcharts and / or block diagrams. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks.

[0187] Moreover, these computer program instructions can also be stored in a computer-readable memory capable of directing the computer or other programmable data processing apparatus to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including instruction devices that implement the flowcharts and / or block diagrams. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks. Figure 1 The device that implements the function specified in one flow or multiple flows and / or blocks.

[0188] These computer program instructions can also be loaded into computer or other programmable data processing devices, so that a series of operation steps are performed on the computer or other programmable data processing devices to generate computer-implemented processes, thus the instructions executed on the computer or other programmable data processing devices provide processes for implementing the functions specified in the flowcharts Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0189] The above only describes the embodiments of the present application and is not intended to limit the present application. The present application can have various modifications and changes for those skilled in the art. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. An image processing method, characterized in that, Applied to PTZ cameras, the method includes: When turbulence detection is required, it is determined that the PTZ camera is not in motion, wherein the motion state refers to any of the states of pitch, yaw or zoom switching. Based on the jitter information corresponding to the PTZ camera, the first image captured by the PTZ camera is subjected to jitter stabilization processing to obtain the jitter-stabilized image corresponding to the first image, and the motion vector diagram corresponding to the jitter-stabilized image is determined and stored in a specified storage medium; wherein, the jitter information is collected by a jitter detection sensor; Obtain the motion vector diagrams corresponding to the N-1 frames preceding the first image; Based on N frames of motion vector graphics, the turbulence influence factor corresponding to the image after image stabilization is determined. Based on the turbulence influence factor, a turbulence reduction operation is performed on the image after image stabilization to obtain the image after turbulence reduction corresponding to the first image. The turbulence reduction operation is used to reduce the turbulence effect in the image after image stabilization. The determination of the turbulence influence factor corresponding to the stabilized image based on N-frame motion vector maps includes: A target value for the motion vector length is determined based on the N frames of motion vector maps; wherein, the target value for the motion vector length is the sum of the motion vector lengths corresponding to the N frames of motion vector maps; for each motion vector map, the motion vector map includes motion vectors corresponding to multiple image blocks, and the motion vector length corresponding to the motion vector map is determined based on the length value of the motion vector corresponding to each image block; Determine the threshold interval in which the target value of the motion vector length is located, and determine the turbulence intensity corresponding to the threshold interval; wherein, there is a one-to-one correspondence between the threshold interval and the turbulence intensity; The quotient of the turbulence intensity and the pixel size of the PTZ camera is determined as the turbulence influence factor.

2. The method according to claim 1, characterized in that, When at least one of the horizontal rotation P value, vertical rotation T value, and magnification Z value corresponding to the PTZ camera changes, the PTZ camera is determined to be in motion.

3. The method according to claim 1, characterized in that, The step of performing turbulence reduction operation on the stabilized image based on the turbulence influence factor to obtain the turbulence-reduced image corresponding to the first image includes: An initial residual image is determined based on the stabilized image and the reference image, wherein the reference image is the stabilized image of the frame preceding the stabilized image. The initial residual image is subjected to large-scale filtering to obtain a large-scale residual image, and the initial residual image is subjected to small-scale filtering to obtain a small-scale residual image. Based on the turbulence influence factor, the preset scaling factor, the large-scale residual image, and the small-scale residual image, the target residual image corresponding to the first image is determined; The image after turbulence reduction is determined based on the target residual image, the image after stabilization, and the reference image.

4. The method according to claim 3, characterized in that, The step of determining the target residual image corresponding to the first image based on the turbulence influence factor, the preset scaling factor, the large-scale residual image, and the small-scale residual image includes: The first weighting coefficient corresponding to the large-scale residual image and the second weighting coefficient corresponding to the small-scale residual image are determined based on the turbulence influence factor and the preset scaling factor; wherein, if the turbulence influence factor is larger, the first weighting coefficient is larger and the second weighting coefficient is smaller; The target residual image is obtained by performing a weighted operation based on the large-scale residual image, the first weight coefficient, the small-scale residual image, and the second weight coefficient.

5. The method according to claim 4, characterized in that, The first weighting coefficient is the product of the turbulence influence factor and the preset scaling factor, and the second weighting coefficient is the difference between 1 and the first weighting coefficient.

6. The method according to claim 3, characterized in that, The step of determining the target residual image corresponding to the first image based on the turbulence influence factor, the preset scaling factor, the large-scale residual image, and the small-scale residual image includes: The target residual image corresponding to the first image is determined using the following formula: fusDiff=smallDiff*(1-K*ratio)+largeDiff*K*ratio Wherein, K represents the turbulence influence factor, ratio represents the preset scaling factor, smallDiff represents the small-scale residual image, largeDiff represents the large-scale residual image, and fusDiff represents the target residual image.

7. The method according to claim 3, characterized in that, The process of determining the turbulence-reduced image based on the target residual image, the stabilized image, and the reference image includes: Motion coefficients are determined based on the average noise level of the target residual image and the image after image stabilization. Based on the motion coefficients, a third weighting coefficient corresponding to the stabilized image and a fourth weighting coefficient corresponding to the reference image are determined. A weighted operation is then performed on the stabilized image, the third weighting coefficient, the reference image, and the fourth weighting coefficient to obtain the turbulence-reduced image.

8. The method according to claim 1, characterized in that, After performing turbulence reduction operation on the stabilized image based on the turbulence influence factor to obtain the turbulence-reduced image corresponding to the first image, the method further includes: The image enhancement factor is determined based on the turbulence influence factor and the target distance, wherein the target distance is the distance between the target object in the first image and the PTZ camera; wherein, the larger the turbulence influence factor, the larger the image enhancement factor, and the larger the target distance, the larger the image enhancement factor. A high-frequency detail image corresponding to the first image is generated based on the image after turbulence reduction, and an enhanced image is generated based on the high-frequency detail image and the image enhancement factor; A target image corresponding to the first image is generated based on the image after turbulence reduction and the enhanced image.

9. A PTZ camera with turbulence detection, including: A shake detection sensor is used to determine the shake information of the PTZ camera; The processor is used to execute the following methods: In response to an externally input turbulence detection command, determine whether the PTZ camera is in motion, wherein the motion state represents any one of pitch motion, yaw motion, or magnification switching: The turbulence detection command is not executed when the PTZ camera is in motion. as well as When the PTZ camera is not in motion, the turbulence detection command is executed, including: Acquire an N-frame image sequence, wherein the acquisition time of the last frame in the image sequence is no earlier than the time when the PTZ camera receives the turbulence detection command; Based on the jitter information, anti-shake processing is performed on the N-frame image sequence to obtain N-frame anti-shake images, and the N-frame motion vector diagrams corresponding to the N-frame anti-shake images are determined. The target value of the motion vector length is determined based on the N frames of motion vector maps; Determine the threshold interval in which the target value of the motion vector length is located, and determine the turbulence intensity corresponding to the threshold interval; wherein, there is a one-to-one correspondence between the threshold interval and the turbulence intensity; The turbulence influence factor is determined based on the turbulence intensity and the pixel size of the PTZ camera; and Based on the turbulence influence factor, the image of the last frame in the N-frame image sequence after image stabilization is processed.

Images