Image quality optimization method and device, electronic equipment and computer readable storage medium

Abstract

The invention relates to the technical field of image processing and video coding, and discloses an image quality optimization method, comprising: acquiring an image frame sequence and attitude data of a camera, the attitude data comprising a direction and an angle of the camera; performing time alignment on the image frame sequence and the attitude data, and establishing a time reference; detecting whether the image frame sequence is abnormal or not, and if the image frame sequence is abnormal, adjusting image frames in the image frame sequence according to a time reference and a preset frame rate to obtain a first image frame sequence; stabilizing the image frames of the first image frame sequence to obtain a second image frame sequence; and encoding the second image frame sequence to generate a target video. The invention further discloses a device, electronic equipment and a computer readable storage medium. According to the invention, the output video is smooth and continuous, and the jitter is obviously reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing and video coding, in particular to a picture quality optimization method and device, electronic equipment and computer readable storage medium. BACKGROUND

[0002] With the wide application of unmanned aerial vehicles and robots, the cameras carried by the unmanned aerial vehicles and robots need to realize stable and clear video acquisition and transmission in high-speed motion and complex environment. However, due to the high speed of the unmanned aerial vehicles and robots in flight or movement, the video acquired by the camera often has problems such as unstable frame rate, picture jitter and image distortion.

[0003] In this case, the video is prone to have picture tilt, motion blur and detail loss caused by rolling shutter effect, thereby seriously affecting the observation quality and subsequent processing effect of the video. SUMMARY

[0004] The embodiments of the present application provide a picture quality optimization method and device, electronic equipment and computer readable storage medium, which can solve the technical problem of poor video quality in video shooting.

[0005] The embodiments of the present application provide a picture quality optimization method, which comprises: acquiring an image frame sequence and attitude data of a camera, wherein the attitude data comprises a direction and an angle of the camera; time aligning the image frame sequence and the attitude data, and establishing a time reference; detecting whether the image frame sequence has an abnormality, and if the image frame sequence has an abnormality, adjusting image frames in the image frame sequence according to the time reference and a preset frame rate to obtain a first image frame sequence; stably processing the image frames of the first image frame sequence to obtain a second image frame sequence; encoding the second image frame sequence to generate a target video.

[0006] Further, the time stamps of each image frame in the image frame sequence are acquired; if the time stamp difference of any two continuous image frames in the image frame sequence is greater than a preset time threshold, it is determined that there is a missing abnormality in the image frame sequence; if the time stamp difference of any two continuous image frames in the image frame sequence is less than a preset time threshold, it is determined that there is a redundant abnormality in the image frame sequence.

[0007] Further, if the image frame sequence has an abnormality, the image frames in the image frame sequence are adjusted according to the time reference and a preset frame rate to obtain a first image frame sequence, which comprises: if the abnormality is that there is a missing image frame in the image frame sequence, obtaining the missing image frame; adjusting the image frame sequence according to the missing image frame, the time reference and a preset frame rate, to obtain a first image frame sequence.

[0008] Further, the adjusting the image frame sequence according to the time reference and the preset frame rate to obtain the first image frame sequence if the abnormality exists in the image frame sequence, comprises: if the abnormality is that there is a redundant image frame in the image frame sequence, obtaining the redundant image frame; adjusting the image frame sequence according to the redundant image frame, the time reference and a preset frame rate, to obtain a first image frame sequence.

[0009] Further, the stabilizing the image frame in the first image frame sequence to obtain a second image frame sequence, comprises: calculating a motion vector of each image frame in the first image frame sequence according to the attitude data; obtaining a pixel value of each image frame in the first image frame sequence, and screening a target image frame from the first image frame sequence according to the pixel value; adjusting the pixel value of the target image frame in the first image frame sequence according to the motion vector, to obtain a processed first image frame sequence; cutting a picture of the processed first image frame sequence to obtain the second image frame sequence.

[0010] Further, the encoding the second image frame sequence to generate a target video, comprises: filtering the second image frame sequence to obtain a target image frame sequence; encoding the target image frame sequence to generate a target video.

[0011] Further, the encoding the target image frame sequence to generate a target video, comprises: predicting a target motion direction and a target angle of the camera according to the attitude data; generating an encoding parameter according to the target motion direction and the target angle; encoding the target image frame sequence according to the encoding parameter to generate a target video.

[0012] Correspondingly, an embodiment of the present application provides a picture quality optimization device, comprising: an information collection module, configured to obtain an image frame sequence and attitude data of a camera, wherein the attitude data comprises a direction and an angle of the camera; a synchronization module, configured to time-align the image frame sequence and the attitude data, and establish a time reference; a frame rate adjustment module, configured to detect whether the image frame sequence is abnormal, and if the image frame sequence is abnormal, adjust the image frames in the image frame sequence according to the time reference and a preset frame rate, to obtain a first image frame sequence; a stabilization processing module, configured to perform stabilization processing on each image frame in the first image frame sequence, to obtain a second image frame sequence; an encoding module, configured to encode the second image frame sequence, to obtain a final video.

[0013] In addition, an electronic device is also provided in the embodiments of the present application, which includes a processor and a memory. The memory stores a computer program, and the processor is configured to run the computer program in the memory to implement the image quality optimization method provided in the embodiments of the present application.

[0014] In addition, a computer readable storage medium is also provided in the embodiments of the present application. The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor to execute any one of the image quality optimization methods provided in the embodiments of the present application.

[0015] In the embodiments of the present application, the image frame sequence and the attitude data of a camera are acquired, the attitude data including the direction and angle of the camera; the image frame sequence and the attitude data are time-aligned, to establish a time reference; whether the image frame sequence is abnormal is detected, and if the image frame sequence is abnormal, the image frames in the image frame sequence are adjusted according to the time reference and a preset frame rate, to obtain a first image frame sequence; the image frames in the first image frame sequence are subjected to stabilization processing, to obtain a second image frame sequence; and the second image frame sequence is encoded to generate a target video, thereby achieving the good effect of smooth and continuous output video and significantly reduced jitter. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative effort.

[0017] Figure 1 is a flowchart of the image quality optimization method provided in the present application; Figure 2 is a structural diagram of the image quality optimization device provided in the embodiments of the present application; Figure 3is a structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0018] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person skilled in the art without creative work fall within the scope of protection of the present application.

[0019] An image quality optimization method, device, electronic device and computer readable storage medium are provided in the embodiments of the present application. The image quality optimization device can be integrated in an electronic device, which can be a server or a terminal or the like.

[0020] The server can be a stand-alone physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery network (CDN) network acceleration services, and basic cloud computing services such as big data and artificial intelligence platforms.

[0021] The terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, a smart speaker, a smart watch, and the like, but is not limited thereto. The terminal and the server can be directly or indirectly connected through wired or wireless communication, which is not limited in the present application.

[0022] In addition, "multiple" in the embodiments of the present application refers to two or more. "First" and "second" and the like in the embodiments of the present application are used to distinguish the description, and cannot be understood as implying relative importance.

[0023] The following will be described in detail. It should be noted that the description order of the following embodiments is not limited as the preferred order of the embodiments.

[0024] Please refer to Figure 1 , Figure 1 is a flowchart of an image quality optimization method provided by an embodiment of the present application. The image quality optimization method can include: S101, acquiring an image frame sequence of a camera and attitude data, the attitude data including a direction and an angle of the camera; The image frame sequence refers to a collection of multiple frames of images collected by the camera, and the attitude data refers to the real-time motion direction and angle of the camera in space while collecting images.

[0025] The camera is initialized to make the camera enter a continuous image acquisition mode, a plurality of images are acquired at a preset sampling frequency, and the plurality of acquired images form an image frame sequence. The image frame sequence is acquired at the same time as the attitude data, including acquiring the real-time motion direction and angle of the camera.

[0026] In some embodiments, a three-dimensional coordinate system is established in the external space, the origin and reference direction are determined as the reference direction, a plurality of feature points of the camera are acquired, the initial space coordinates of the feature points are determined, and an initial attitude matrix is generated from the initial space coordinates. In the process of acquiring images by the camera, the real-time attitude matrix of the camera is acquired, the orientation and angle of the camera are calculated by comparing the real-time coordinate matrix with the initial coordinate matrix.

[0027] In some embodiments, the acceleration and angular velocity of the camera are acquired, the acceleration and angular velocity are calculated, and the acceleration and angular velocity can be converted into the pitch angle and yaw angle, so as to obtain the real-time direction and angle of the camera in the process of acquiring images.

[0028] S102, time alignment is performed on the image frame sequence and the attitude data, and a time reference is established; The time reference refers to that the image frame sequence and the attitude data adopt the same time reference system. Specifically, in some embodiments, the image frame sequence and the attitude data are time-aligned, a uniform reference starting time is set, and the time is taken as the common starting point and reference of the image frame sequence and the attitude data. For example, when the moment when the camera starts is set as the reference starting time t=0, after that, the moment when each image is acquired is acquired, and the attitude data of the camera corresponding to the moment is also acquired, so that the image frame and the attitude data at the same moment are one-to-one corresponding.

[0029] In some embodiments, the acquisition time stamp t1 of the image frame sequence and the measurement time stamp t2 of the attitude data are extracted during the time alignment of the image frame sequence and the attitude data. If there is a clock drift between the acquisition time stamp and the measurement time stamp, that is, the clock references of the two are different, the deviation value Δt=t1-t2 is calculated. In some implementations, a synchronization calibration point t1 is selected, and the measurement time stamps of all attitude data are uniformly added by the deviation value Δt, so that the time stamps of the image frame and the attitude data are based on the same reference. In some implementations, t2 can also be selected as the synchronization calibration point, and the acquisition time stamps of all image frames in the image frame sequence are subtracted by the deviation value Δt, so that the time stamps of the two are based on the same reference. In some implementations, an external reference time value t0 can be selected as the synchronization calibration point, and the offsets of the acquisition time stamp and the measurement time stamp relative to t0 are calculated respectively, and the acquisition time stamp t1 of the image frame sequence and the measurement time stamp t2 of the attitude data are corrected according to the offsets, so as to time-align the image frame sequence and the attitude data.

[0030] S103, detecting whether the image frame sequence is abnormal, if the image frame sequence is abnormal, adjusting the image frames in the image frame sequence according to the time reference and a preset frame rate to obtain a first image frame sequence; Specifically, whether the image frame sequence is abnormal is detected. In some embodiments, the time stamp of each frame in the image frame sequence needs to be acquired, and whether the time difference between any two adjacent frames is greater than a preset maximum allowed frame interval is compared. If it is greater than the threshold, it is judged that there is a missing frame abnormality in the image frame sequence. Based on the time reference and the preset frame rate, the missing frame is filled to ensure constant frame rate. If the time difference between any two adjacent frames is less than the preset minimum allowed frame interval, it is judged that there is a repeated frame in the image frame sequence. Based on the time reference and the preset frame rate, the redundant frame is discarded to ensure constant frame rate.

[0031] In some embodiments, whether the image frame sequence is abnormal can be detected by detecting whether there is an invalid frame in the image frame sequence. The brightness histogram and pixel variance of each image in the image frame sequence need to be extracted. If the brightness histogram of a certain image frame presents a single peak value concentrated in the low gray interval, and the pixel variance is less than a preset threshold, it is determined that the frame image is a black frame. If the histogram distribution is extremely discrete and there is an obvious abnormal peak, and the pixel noise level exceeds the preset threshold, it is determined that the frame image is a noise frame. When the above invalid image frame is detected, the following way can be taken to wake up the missing frame: in some implementations, the previous image frame can be used to replace the invalid frame to maintain the integrity of the sequence; in another implementation, the previous and subsequent two images can be interpolated to generate a new compensation frame to replace the invalid frame, thereby reducing the influence of abnormal frames on the continuity of the whole sequence.

[0032] S104, performing stable processing on the image frames of the first image frame sequence to obtain a second image frame sequence; The stable processing refers to smoothing, correcting or compensating the image frames in the first image frame sequence, so that the picture of the image frame sequence is more stable.

[0033] In some embodiments, the stable processing includes motion compensation of the image frames in the first image frame sequence. According to the attitude data, the motion vector of each image frame in the first image frame sequence relative to the reference frame is calculated, wherein the motion vector includes a translation vector and a rotation angle, and the rotation motion of each image in the image frame sequence is calculated. Each image frame corresponds to a motion vector, which represents the attitude change of the camera during the acquisition of the frame, the attitude change of the camera is obtained, and then the displacement and rotation amount between frames are calculated, and the image frames are corrected by corresponding geometric transformation, so as to eliminate the picture instability caused by the attitude change.

[0034] In some embodiments, the stabilization processing includes pixel value adjustment on the image frames in the first image frame sequence. The pixel values of each frame in the first image frame sequence are obtained; and the pixel values of each frame are adjusted by translation, rotation or scaling, etc. according to the need of image stabilization, so as to reduce the jitter caused by camera movement. The adjusted pixel values are interpolated to ensure image continuity and smoothness. After the adjustment, the stabilized image frame sequence is obtained. Specifically, for each frame in the first image frame sequence, the pixel matrix of the frame is read, which includes the gray value or RGB value of each pixel point. According to the attitude data of adjacent frames, the translation, rotation or scaling parameters of the current frame relative to the previous frame are calculated. The pixel matrix of the current frame is translated, rotated or scaled according to the parameters. The pixel blank area caused by translation or rotation is filled by interpolation method to ensure the continuity of the image frame.

[0035] In some embodiments, the stabilization processing includes cropping the image frames in the first image frame sequence. It is detected whether there is a blank in the edge area of each image frame in the first image frame sequence. If there is, the preset cropping ratio is adjusted, for example, the upper edge and the lower edge of the image frame are each cropped by 5%, thereby determining the cropping area, and the image frame with the blank is cropped.

[0036] S105, encoding the second image frame sequence to generate a target video.

[0037] The target video refers to the video output after the second image frame sequence is encoded.

[0038] In some embodiments, the second image frame sequence is encoded, and the encoding parameters can be generated according to the attitude data of the camera. The second image frame sequence is encoded according to the encoding parameters to generate a target video.

[0039] Specifically, the attitude data of the camera is obtained, and the attitude data is analyzed to calculate the angle difference and the average angular velocity of adjacent two frames of images, so as to determine the motion intensity of the camera, which can be divided into low intensity motion, medium intensity motion and high intensity motion. The encoding parameters are generated according to the motion intensity. The encoding parameters include quantization parameters, key frame interval and region of interest. For the image frames with low intensity motion, a higher quantization parameter, a smaller motion estimation search range, a longer key frame interval and a normal region of interest quality are set to reduce the code rate and improve the encoding efficiency. For the image frames with medium intensity motion, a medium quantization parameter, a medium search range, a standard key frame interval and a standard region of interest quality are set to balance the video definition and compression rate. For the image frames with high intensity motion, a lower quantization parameter, a larger search range, a shorter key frame interval and a high quality region of interest are set to ensure the image definition in high speed motion scene. The generated encoding parameters are input into a video encoder to encode the second image frame sequence and output a target video.

[0040] In some embodiments, the second image frame sequence is encoded, a format of video encoding is determined, such as H.264 / AVC, H.265 / HEVC, and a target video is output.

[0041] In the embodiments of the present application, by acquiring an image frame sequence and attitude data of a camera, the attitude data includes a direction and an angle of the camera; the image frame sequence and the attitude data are time-aligned to establish a time reference; whether the image frame sequence has an abnormality is detected, if the image frame sequence has an abnormality, image frames in the image frame sequence are adjusted according to the time reference and a preset frame rate to obtain a first image frame sequence; the image frames of the first image frame sequence are processed stably to obtain a second image frame sequence; the second image frame sequence is encoded to generate a target video, which realizes a good effect of smooth and continuous output video and obvious reduction of jitter.

[0042] In some embodiments, the detection of whether the image frame sequence has an abnormality includes: acquiring a time stamp of each image frame in the image frame sequence; if a time stamp difference of any two continuous image frames in the image frame sequence is greater than a preset time threshold, it is determined that there is a missing abnormality between the two image frames; if the time stamp difference of any two continuous image frames in the image frame sequence is less than the preset time threshold, it is determined that there is a redundant abnormality between the two image frames.

[0043] In the embodiments, the time stamp refers to time information marked for each image frame in the image frame sequence, and the time stamp can reflect the accurate time sequence of each image frame in the acquisition process. The time stamp is acquired, the time stamp difference of any two continuous image frames is calculated, and the obtained multiple time stamp differences are compared with the preset time threshold. If the time stamp difference is greater than the preset time threshold, it is determined that there is a missing image frame abnormality between the two image frames corresponding to the time stamps; if the time stamp is less than the preset time threshold, it is determined that there is a repeated image frame or an image frame with too high acquisition frequency, i.e., an image frame redundancy abnormality, between the two image frames corresponding to the time stamps.

[0044] Through the above detection, whether the image frame sequence has an abnormality can be effectively identified, and if there is an abnormality, the specific type and position of the abnormality can also be determined.

[0045] In some embodiments, if the image frame sequence has an abnormality, the image frames in the image frame sequence are adjusted according to the time reference and the preset frame rate to obtain a first image frame sequence, including: if the abnormality is that there is a missing image frame in the sequence of image frames, obtaining the missing image frame; adjusting the sequence of image frames according to the missing image frame, the time reference, and a preset frame rate, to obtain a first sequence of image frames.

[0046] In the embodiment, the preset frame rate is used to determine the position of the time stamp and the preset time threshold of the time stamps corresponding to two adjacent image frames. The preset time threshold is the inverse of the preset frame rate. For example, if the preset frame rate is set to 30 fps, i.e., 30 image frames need to be captured per second, then the preset time threshold is (1 / 30) second. If the abnormality is that there is a missing image frame in the sequence of image frames, the positions of the two adjacent image frames of the missing image frame, the time stamps, and the time reference are determined by the above steps, the missing image frame is inserted into the sequence of image frames, and a first sequence of image frames is obtained. Specifically, the preset time threshold is Δt. If the time stamps of two adjacent image frames are greater than Δt, a missing image frame is inserted between the two image frames. If the time stamp difference between the two adjacent image frames is nΔt, where n is an integer greater than 1, then (n-1) missing image frames need to be inserted into the time stamps corresponding to the two image frames. If the time stamp difference between the two adjacent image frames is between Δt and 2Δt, it can be ignored or a linear interpolation method can be used to insert a missing image frame after Δt of the time stamp of the previous image frame of the two adjacent image frames. The subsequent image frames are realigned according to the time reference and the preset frame rate, thereby obtaining the first sequence of image frames.

[0047] In some embodiments, the missing image frame can be generated by linear interpolation, i.e., the pixel value of each pixel position is interpolated according to the pixel values of the two adjacent image frames. Specifically, the average of the pixel values corresponding to the two adjacent image frames is set as the pixel value of the missing frame, and the new image frame is obtained as the missing frame inserted into the missing position, thereby obtaining the first sequence of image frames. The interpolation frame generated in this way has a smooth transition between the previous and subsequent frames in vision, reducing the sense of jumping frames caused by missing frames.

[0048] In some embodiments, the adjacent image frame can be directly copied and inserted into the missing position to obtain the first sequence of image frames.

[0049] Thus, by using the above interpolation method, the missing frames in the sequence of image frames can be effectively completed, so that the sequence of image frames remains at the preset frame rate and time reference, ensuring the continuity and smoothness of the subsequent video output.

[0050] In some embodiments, if the sequence of image frames is abnormal, the sequence of image frames is adjusted according to the time reference and the preset frame rate to obtain a first sequence of image frames, including: if the abnormality is that there is redundancy in the image frame sequence, obtaining a redundant image frame; adjusting the image frame sequence according to the redundant image frame, the time reference and a preset frame rate to obtain a first image frame sequence.

[0051] In the embodiment, if the abnormality is that there is redundancy in the image frame sequence, the position and timestamp of the redundant frame in the image frame sequence can be determined through the above steps. The timestamp and a preset time threshold are determined according to the preset frame rate. For example, if the timestamp difference between two adjacent image frames is less than the preset time threshold Δt, it is determined that there is a redundant image frame in the time period. The abnormality that the image frame sequence has redundancy can be divided into two cases.

[0052] Specifically, the preset time threshold is Δt, and the timestamp difference between two image frames is less than Δt. If the timestamp difference between an image frame and two adjacent image frames is less than or equal to 0.5Δt, it is determined that there is a redundant image frame. For example, if the timestamp difference between an image frame and two adjacent image frames is 0.5Δt, it is determined that the image frame is a redundant image frame. The image frame is deleted, and the timestamp difference between the remaining two adjacent image frames becomes Δt, thereby obtaining a first image frame sequence.

[0053] In some embodiments, if the timestamp difference between two image frames is between 0.5Δt and Δt, that is, the two image frames are sampled too quickly, one of the two image frames can be deleted or moved to a position corresponding to the preset time threshold based on the time reference, thereby obtaining a first image frame sequence.

[0054] Thus, through the above redundant frame processing method, the image frame sequence can be effectively adjusted to be consistent with the time reference and the preset frame rate. This not only makes the image frames in the image frame sequence more stable and smooth, but also makes the image frame sequence and the attitude data better synchronized.

[0055] In some embodiments, the image frames in the first image frame sequence are stably processed to obtain a second image frame sequence, including: According to the attitude data, the motion vector of each image frame in the first image frame sequence is calculated. The pixel value of each image frame in the first image frame sequence is obtained, and a target image frame is selected from the first image frame sequence according to the pixel value. The pixel value of the target image frame in the first image frame sequence is adjusted according to the motion vector to obtain a processed first image frame sequence. The picture of the processed first image frame sequence is cropped to obtain the second image frame sequence.

[0056] In the embodiment, the motion vector refers to a displacement or a rotation transformation amount between two adjacent image frames, and can reflect the motion of the image frames in space. Specifically, attitude data corresponding to different timestamps of the camera is obtained; the attitude data includes position parameters and rotation parameters; for any two adjacent image frames in the first image frame sequence, the attitude data at the corresponding timestamps is read respectively; based on the attitude data of the two adjacent timestamps, the displacement and the rotation of the camera in the time interval are calculated; the displacement is determined by the difference between the two adjacent position parameters, and the rotation can be determined by the difference between the two adjacent rotation parameters, thereby obtaining the motion vector of the latter image frame relative to the former image frame in the two adjacent image frames. Correspondingly, the motion vector of each image frame in the first image frame sequence can be calculated.

[0057] After obtaining the motion vector of each image frame in the first image frame sequence, the pixel value of each image frame in the first image frame sequence is obtained. The image frame with a pixel value less than or equal to a preset pixel value is determined as a target image frame. In some embodiments, the target image frame can also be all image frames in the first image frame sequence. According to the motion vector, the pixel position of the target image frame in the first image frame sequence is translated, rotated or scaled. For example, when the motion vector indicates that the picture exists a displacement to the right, the image is translated to the left by a corresponding number of pixels as a whole to compensate for the displacement; when the motion vector indicates that there is a rotation, the image is rotated by a corresponding angle to compensate. Thus, the pixel values of the adjusted image frames tend to be consistent. Further, in order to avoid the position of the original pixel point after the pixel value adjustment to produce a void, the adjusted pixel point can also be interpolated and compensated, and the nearest known pixel value is filled to obtain the processed first image frame sequence. According to the picture situation of the image frames in the processed first image frame sequence, the edges of the image frames are cropped to remove black or empty edge regions, and a second image frame sequence is obtained.

[0058] Thus, through the above steps, the image frames in the first image frame sequence are compensated and adjusted, thereby effectively eliminating the picture jitter phenomenon caused by camera jitter or slight motion, and through the adjustment of the pixel value and the cropping of the picture area, the picture transition in the second image frame sequence can be ensured to be more natural and stable.

[0059] In some embodiments, the encoding of the second image frame sequence to generate a target video includes: filtering the second image frame sequence to obtain a target image frame sequence; encoding the target image frame sequence to generate a target video.

[0060] In the embodiment, the second image frame sequence is filtered, including that noise detection and removal are performed on each image frame in the second image frame sequence, which can be achieved by means of mean filtering and median filtering.

[0061] Specifically, in some ways, the second image frame sequence is filtered by mean filtering, and each image frame in the second image frame sequence is obtained. For each image, a sliding window of a preset size is set, for example, the sliding window can include a 3x3 or 5x5 pixel neighborhood, each pixel point in the image is traversed, the average value of the pixel values of the pixel points in each sliding window is calculated, and the average value is taken as the new value of the pixel points in the window. Thus, the target image frame sequence is obtained; in some ways, the pixel values in the sliding window are sorted, and the middle value after sorting is taken as the new value of the pixel, replacing the original pixel value, to obtain the target image frame sequence. According to the target image frame sequence, encoding is performed, and a target video is output.

[0062] In this way, random noise in the image frame can be effectively removed, and the output video picture is clear and the visual interference caused by noise is reduced.

[0063] In some embodiments, the encoding of the target image frame sequence to generate a target video includes: According to the attitude data, a target motion direction and a target angle of the camera are predicted; According to the target motion direction and the target angle, an encoding parameter is generated; According to the encoding parameter, the target image frame sequence is encoded to generate a target video.

[0064] In the embodiment, according to the attitude data of the camera, the direction change and the angle change of the camera can be calculated, and based on the position change and the angle change of the camera, the attitude data of the camera at the next timestamp is predicted. Specifically, the attitude data of the camera corresponding to the current timestamp and the previous timestamp is obtained, the direction change and the angle change of the camera are calculated, and the target motion direction and the target angle of the camera can be calculated according to the direction change and the angle change. According to the target motion direction, the target angle, and a preset threshold, the image frame at the next timestamp is divided into low-intensity motion frames, medium-intensity motion frames, and high-intensity motion frames.

[0065] If the image frame of the next timestamp is a low-intensity motion frame, the encoding parameters corresponding to the low-intensity motion frame are generated, and low-code rate encoding is used when encoding the next image frame in the target image frame; if the image frame of the next timestamp is a medium-intensity motion frame, the encoding parameters corresponding to the medium-intensity motion frame are generated, and medium-code rate encoding is used when encoding the next image frame in the target image frame; if the image frame of the next timestamp is a high-intensity motion frame, the encoding parameters corresponding to the high-intensity motion frame are generated, and high-code rate encoding is used when encoding the next image frame in the target image frame, so as to ensure that the output video is still clear under high-speed movement.

[0066] Therefore, the balance between quality and efficiency can be achieved by dynamically adjusting the code rate allocation according to the predicted motion intensity of the image frame.

[0067] To better implement the image quality optimization method provided in the embodiments of the present application, the embodiments of the present application further provide a device based on the above image quality optimization method. The meanings of the terms are the same as those in the above image quality optimization method, and the specific implementation details can be referred to the description in the method embodiments.

[0068] For example, as shown in FIG. 1, the image quality optimization device can include: Figure 2 An information collection module is configured to acquire an image frame sequence and attitude data of a camera, wherein the attitude data includes the direction and angle of the camera. A synchronization module is configured to perform time alignment on the image frame sequence and the attitude data, and establish a time reference. A frame rate adjustment module is configured to detect whether there is an abnormality in the image frame sequence, and if there is an abnormality in the image frame sequence, adjust the image frames in the image frame sequence according to the time reference and a preset frame rate, to obtain a first image frame sequence. A stabilization processing module is configured to perform stabilization processing on the image frames in the first image frame sequence, to obtain a second image frame sequence. An encoding module is configured to encode the second image frame sequence, to generate a target video.

[0069] In an embodiment of the present application, the frame rate adjustment module is further configured to: acquire the timestamp of each image frame in the image frame sequence; if the timestamp difference between any two consecutive image frames in the image frame sequence is greater than a preset time threshold, it is determined that there is a missing abnormality in the image frame sequence; if the timestamp difference between any two consecutive image frames in the image frame sequence is less than a preset time threshold, it is determined that there is a redundant abnormality in the image frame sequence.

[0070] ​In an embodiment of the present application, the frame rate adjustment module is further configured to: if the abnormality is that there is a missing image frame in the image frame sequence, obtaining the missing image frame; adjusting the image frame sequence according to the missing image frame, the time reference and a preset frame rate to obtain a first image frame sequence.

[0071] In an embodiment of the present application, the frame rate adjustment module is further configured to: if the abnormality is that there is a redundant image frame in the image frame sequence, obtaining the redundant image frame; adjusting the image frame sequence according to the redundant image frame, the time reference and a preset frame rate to obtain a first image frame sequence.

[0072] In an embodiment of the present application, the stabilization processing module is further configured to: calculating a motion vector of each image frame in the first image frame sequence according to the attitude data; obtaining a pixel value of each image frame in the first image frame sequence, and selecting a target image frame from the first image frame sequence according to the pixel value; adjusting the pixel value of the target image frame in the first image frame sequence according to the motion vector to obtain a processed first image frame sequence; cropping a picture of the processed first image frame sequence to obtain the second image frame sequence.

[0073] In an embodiment of the present application, the encoding module is further configured to: performing filtering processing on the second image frame sequence to obtain a target image frame sequence; encoding the target image frame sequence to generate a target video.

[0074] In an embodiment of the present application, the encoding module is further configured to: predicting a target motion direction and a target angle of the camera according to the attitude data; generating an encoding parameter according to the target motion direction and the target angle; encoding the target image frame sequence according to the encoding parameter to generate a target video.

[0075] The image quality optimization device provided in the present application realizes time synchronization, frame rate stabilization, image stabilization and encoding enhancement processing of the image frame sequence, effectively solves the problems of jitter, lag and compression artifacts caused by the video under the condition of high-speed motion or uneven collection, and improves the stability and clarity of the video picture.

[0076] In practice, the above various modules can be implemented as independent entities, or combined as the same or several entities, and the specific implementation manners of the above various modules and the corresponding beneficial effects can be referred to the method embodiments, which will not be described here.

[0077] The embodiments of the present application also provide an electronic device, which can be a server or a terminal, etc. Figure 3 As shown in the figure, the structure of the electronic device related to the embodiments of the present application is shown, in particular: The electronic device can include a processor 601 with one or more processing cores, a memory 602 with one or more computer readable storage media, a power supply 603, an input unit 604, and the like. Those skilled in the art can understand that the structure of the electronic device shown in the figure does not constitute a limitation on the electronic device, and can include more or fewer components than shown, or combine certain components, or different component arrangements. Among them: Figure 3 The processor 601 is the control center of the electronic device, which connects various parts of the entire electronic device through various interfaces and lines, and performs various functions and processes data of the electronic device by running or executing computer programs and / or modules stored in the memory 602, and calling data stored in the memory 602. Optionally, the processor 601 can include one or more processing cores; preferably, the processor 601 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 601. The memory 602 can be used to store computer programs and modules, and the processor 601 executes various function applications and data processing by running the computer programs and modules stored in the memory 602. The memory 602 can mainly include a program storage area and a data storage area, wherein the program storage area can store the operating system, at least one computer program required by the function (such as sound playing function, image playing function, etc.), etc.; the data storage area can store data created according to the use of the electronic device, etc. In addition, the memory 602 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. Accordingly, the memory 602 can also include a memory controller to provide the processor 601 with access to the memory 602.

[0078]

[0079] ​The electronic device also includes a power supply 603 that supplies power to the various components. Preferably, the power supply 603 can be logically connected to the processor 601 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. The power supply 603 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0080] The electronic device may also include an input unit 604, which can be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0081] Although not shown, the electronic device may also include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 601 in the electronic device loads the executable files corresponding to the processes of one or more computer programs into the memory 602 according to the following instructions, and the processor 601 runs the computer programs stored in the memory 602 to realize various functions, such as: Acquire image frame sequences and pose data from the camera, wherein the pose data includes the orientation and angle of the camera; The image frame sequence and the pose data are time-aligned to establish a time reference; Detect whether there is an anomaly in the image frame sequence. If there is an anomaly in the image frame sequence, adjust the image frames in the image frame sequence according to the time base and the preset frame rate to obtain the first image frame sequence. The image frames of the first image frame sequence are stabilized to obtain the second image frame sequence; The second image frame sequence is encoded to generate the target video.

[0082] For details on the specific implementation methods and corresponding beneficial effects of each of the above operations, please refer to the detailed description of the image quality optimization methods above, which will not be repeated here.

[0083] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by a computer program, or by a computer program controlling related hardware. The computer program can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0084] Therefore, embodiments of this application provide a computer-readable storage medium storing a computer program that can be loaded by a processor to execute the steps of any of the image quality optimization methods provided in embodiments of this application. For example, the computer program can execute the following steps: obtain a sequence of image frames and pose data of a camera, the pose data comprising a direction and an angle of the camera; time-align the sequence of image frames and the pose data, and establish a time reference; detect whether the sequence of image frames is abnormal, and if the sequence of image frames is abnormal, adjust image frames in the sequence of image frames according to the time reference and a preset frame rate, to obtain a first sequence of image frames; perform stabilization processing on the image frames of the first sequence of image frames, to obtain a second sequence of image frames; encode the second sequence of image frames, to generate a target video.

[0085] The specific implementation and corresponding beneficial effects of each operation above can be found in the foregoing embodiments, which will not be repeated here.

[0086] The computer readable storage medium can include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, an optical disk, etc.

[0087] The computer program stored in the computer readable storage medium can execute the steps of any of the picture quality optimization methods provided by the embodiments of the present application, thus achieving the beneficial effects of any of the picture quality optimization methods provided by the embodiments of the present application. Details can be found in the foregoing embodiments, which will not be repeated here.

[0088] According to an aspect of the present application, a computer program product or computer program is provided, which includes computer instructions stored in a computer readable storage medium. A processor of a computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the picture quality optimization method described above.

[0089] The picture quality optimization method, device, electronic device and computer readable storage medium provided by the embodiments of the present application are described in detail above, and the principles and implementation manners of the present application are described by applying specific examples. The above description of the embodiments is only used to help understand the method and its core idea of the present application; meanwhile, for those skilled in the art, the specific implementation manner and application range can be changed according to the idea of the present application, and the content of the specification should not be understood as a limitation of the present application.

Claims

1. An image quality optimization method characterized by comprising: The method comprises the following steps: obtaining a sequence of image frames and attitude data of a camera, the attitude data comprising a direction and an angle of the camera; time aligning the sequence of image frames and the attitude data to establish a time reference; detecting whether the sequence of image frames has an abnormality, and if the sequence of image frames has an abnormality, adjusting the image frames in the sequence of image frames according to the time reference and a preset frame rate to obtain a first sequence of image frames; stabilizing the image frames of the first sequence of image frames to obtain a second sequence of image frames; encoding the second sequence of image frames to generate a target video.

2. The image quality optimization method of claim 1, wherein The detection of whether the sequence of image frames has an abnormality comprises: obtaining a timestamp of each image frame in the sequence of image frames; if a timestamp difference between any two continuous image frames in the sequence of image frames is greater than a preset time threshold, it is determined that there is a missing abnormality in the sequence of image frames; if a timestamp difference between any two continuous image frames in the sequence of image frames is less than a preset time threshold, it is determined that there is a redundant abnormality in the sequence of image frames.

3. The image quality optimization method of claim 2, wherein If the sequence of image frames has an abnormality, the image frames in the sequence of image frames are adjusted according to the time reference and a preset frame rate to obtain a first sequence of image frames, which comprises: if the abnormality is a missing abnormality in the sequence of image frames, a missing image frame is obtained; the sequence of image frames is adjusted according to the missing image frame, the time reference and a preset frame rate to obtain a first sequence of image frames.

4. The image quality optimization method of claim 2, wherein If the sequence of image frames has an abnormality, the image frames in the sequence of image frames are adjusted according to the time reference and a preset frame rate to obtain a first sequence of image frames, which comprises: if the abnormality is a redundant abnormality in the sequence of image frames, a redundant image frame is obtained; the sequence of image frames is adjusted according to the redundant image frame, the time reference and a preset frame rate to obtain a first sequence of image frames.

5. The image quality optimization method of claim 1, wherein The stabilization of the image frames of the first sequence of image frames to obtain a second sequence of image frames comprises: calculating a motion vector of each image frame in the first sequence of image frames according to the attitude data; obtaining a pixel value of each image frame in the first sequence of image frames, and screening a target image frame from the first sequence of image frames according to the pixel value; adjusting the pixel value of the target image frame in the first sequence of image frames according to the motion vector to obtain a processed first sequence of image frames; cropping a picture of the processed first sequence of image frames to obtain the second sequence of image frames.

6. The image quality optimization method of claim 1, wherein The encoding of the second sequence of image frames to generate a target video comprises: filtering the second sequence of image frames to obtain a target sequence of image frames; encoding the target sequence of image frames to generate a target video.

7. The image quality optimization method of claim 6, wherein The encoding of the target sequence of image frames to generate a target video comprises: predicting a target motion direction and a target angle of the camera according to the attitude data; generating an encoding parameter according to the target motion direction and the target angle; According to the encoding parameter, the target image frame sequence is encoded to generate a target video.

8. An image quality optimization apparatus, characterized by comprising: The method comprises the following steps: An information acquisition module is configured to acquire an image frame sequence and attitude data of a camera, wherein the attitude data comprises a direction and an angle of the camera. A synchronization module is configured to time-align the image frame sequence and the attitude data to establish a time reference. A frame rate adjustment module is configured to detect whether the image frame sequence is abnormal, and if the image frame sequence is abnormal, adjust image frames in the image frame sequence according to the time reference and a preset frame rate to obtain a first image frame sequence. A stabilization processing module is configured to perform stabilization processing on the image frames in the first image frame sequence to obtain a second image frame sequence. An encoding module is configured to encode the second image frame sequence to generate a target video.

9. An electronic device, comprising: The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by the processor to execute the image quality optimization method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by the processor to execute the image quality optimization method according to any one of claims 1 to 7.

Images