Video processing method and device and electronic equipment

Abstract

The invention discloses a video processing method and device and electronic equipment, and belongs to the technical field of videos. The method comprises the following steps: encoding a first image stream to obtain a code stream; decoding the code stream to obtain a second image stream; under the condition that the first decoded image exists in the second image stream, a first coded image before decoding of the first decoded image is determined in the code stream, and the first decoded image is a decoded image whose similarity with a corresponding source image in the first image stream is smaller than a similarity threshold; and repairing the first coded image based on the first image stream to obtain a coded video file.

Description

Technical Field

[0001] The present application belongs to the field of video technology, and specifically relates to a video processing method, device and electronic equipment. Background Art

[0002] Screen distortion refers to abnormal image damage during video playback or display, such as color distortion, mosaic, image loss or breakage, etc., which is usually caused by encoding compression errors or hardware failure.

[0003] The distorted screen seriously affects the user's viewing experience, causing the video content to be unable to be presented correctly, making the video content incomplete or unreadable. Therefore, it is necessary to repair the distorted screen to restore the normal video quality and ensure that the picture is coherent and clear. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a video processing method, device and electronic device, which can solve the technical problem that the existing screen distortion cannot be effectively repaired.

[0005] In a first aspect, an embodiment of the present application provides a video processing method, the method comprising:

[0006] Encoding the first image stream to obtain a code stream;

[0007] Decoding the code stream to obtain a second image stream;

[0008] In a case where there is a first decoded image in the second image stream, determining in the code stream a first coded image before decoding the first decoded image, wherein the first decoded image is a decoded image whose similarity to a corresponding source image in the first image stream is less than a similarity threshold;

[0009] The first encoded image is repaired based on the first image stream to obtain an encoded video file.

[0010] In a second aspect, an embodiment of the present application provides a video processing device, the device comprising:

[0011] An encoding module, used for encoding the first image stream to obtain a code stream;

[0012] A decoding module, used for decoding the code stream to obtain a second image stream;

[0013] a first determining module, configured to determine, in the bitstream, a first coded image before decoding the first decoded image, if there is a first decoded image in the second image stream, wherein the first decoded image is a decoded image whose similarity to a corresponding source image in the first image stream is less than a similarity threshold;

[0014] The first repair module is used to repair the first coded image based on the first image stream to obtain a coded video file.

[0015] In a third aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method provided in the first aspect are implemented.

[0016] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed by a processor, the steps of the method provided in the first aspect are implemented.

[0017] In a fifth aspect, an embodiment of the present application provides a chip, the chip including a processor and a communication interface, the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the method provided in the first aspect.

[0018] In a sixth aspect, an embodiment of the present application provides a computer program product, which is stored in a storage medium and is executed by at least one processor to implement the method provided in the first aspect.

[0019] In the video processing method, device and electronic device of the present application, by comparing the similarity between the image in the original first image stream and the decoded image, the first encoded image that has suffered quality loss during the encoding process can be accurately identified, and the second encoded image that needs to be encoded with reference to the first encoded image can be identified, and then the first encoded image can be repaired based on the first image stream. In this way, the similarity between the original image and the decoded image can be quantified as a reference for the image quality loss during the encoding process, so as to accurately infer the first encoded image with a large quality loss, and repair the first encoded image, which accurately identifies the encoded image in the code stream that may have a distorted screen, thereby effectively completing the video repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is one of the flowcharts of a video processing method provided by an embodiment of the present application;

[0021] Figure 2 This is one of the frame structure diagrams of a video processing method provided by an embodiment of the present application;

[0022] Figure 3 This is a second schematic diagram of a frame structure of a video processing method provided by an embodiment of the present application;

[0023] Figure 4 This is a third schematic diagram of a frame structure of a video processing method provided by an embodiment of the present application;

[0024] Figure 5 This is a fourth schematic diagram of a frame structure of a video processing method provided by an embodiment of the present application;

[0025] Figure 6 This is a second flow chart of a video processing method provided by an embodiment of the present application;

[0026] Figure 7 This is a third flow chart of a video processing method provided by an embodiment of the present application;

[0027] Figure 8 This is a fourth flow chart of a video processing method provided by an embodiment of the present application;

[0028] Fig. 9 is a structural schematic diagram of a video processing device provided by another embodiment of the present application;

[0029] Fig.10 is a structural schematic diagram of an electronic device provided by another embodiment of the present application;

[0030] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0031] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0032] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the terms used in this way are interchangeable where appropriate, so that the embodiments of the present application can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0033] In the related art, the repair of the flower screen in the video is mainly performed at the decoding end. First, it is possible to detect whether there is a flower screen frame in the video. If a flower screen frame is detected, the system will generate a prediction frame based on the flower screen frame and the image frame before the flower screen frame, and repair the current flower screen frame through the prediction frame. However, the accuracy of this repair method is low, especially when the screen switches quickly, the repair at the decoding end often cannot accurately predict the required picture, and cannot restore the user's real scene, so it is impossible to generate an accurate prediction frame. In addition, the flower screen detection technology in the above method is a reference-free detection method. Due to the lack of a reference image, the detection precision and accuracy of the flower screen frame are low. As a result, the flower screen repair effect is not ideal.

[0034] In order to solve the above technical problems, the present application provides a video processing method. The video processing method provided by the embodiment of the present application is described in detail below through specific embodiments and application scenarios in conjunction with the accompanying drawings.

[0035] like Figure 1 As shown, Figure 1 : is a flow chart of a video processing method provided by an embodiment of the present application. The embodiment of the present application provides a video processing method, which may include:

[0036] S11, encoding the first image stream to obtain a code stream;

[0037] In this embodiment, the image stream refers to a continuous data stream composed of uncompressed frames. The first image stream can be a sequence of image frames directly output from the camera during the recording process; the first image stream can also be a sequence of image frames obtained from an existing video file during video editing, video transcoding or video publishing. The code stream is the data stream after the first image stream is compressed and encoded, indicating the binary data of the encoded video.

[0038] Specifically, the acquired first image stream may be encoded by an encoder, and the encoder may be H264, H265, H266, or AV1.

[0039] S12, decoding the code stream to obtain a second image stream;

[0040] In this embodiment, after the first image stream is encoded to obtain a code stream, the code stream may be decoded by a decoder to obtain a second image stream.

[0041] Since the basic information of the image in the bitstream is known, such as the width, height and encoding format of the image, a corresponding decoder can be created based on the basic information, and then the bitstream can be passed to the decoder for decoding, that is, a second image stream similar to the first image stream can be restored. In addition, the decoder built into the encoder can also be used to decode the bitstream.

[0042] S13, when there is a first decoded image in the second image stream, determining in the code stream a first coded image before decoding the first decoded image, wherein the first decoded image is a decoded image whose similarity to the corresponding source image in the first image stream is less than a similarity threshold;

[0043] In this embodiment, since each decoded image in the second image stream is obtained by encoding and decoding a corresponding source image in the first image stream, and the encoding and decoding process will retain most of the key information of the image, each decoded image in the second image stream has a corresponding source image in the first image stream, and if there is no abnormality in the encoding process, the decoded image and the corresponding source image have a high degree of similarity.

[0044] The similarity of each decoded image in the second image stream and the corresponding source image in the first image stream can be compared. If the similarity between the decoded image and the corresponding source image is greater than or equal to a preset similarity threshold, it can be considered that the similarity between the decoded image and the corresponding source image is high, and it can be considered that there is no abnormality in the encoding process of this decoded image; if the similarity between the decoded image and the corresponding source image is less than the preset similarity threshold, it can be considered that the similarity between the decoded image and the corresponding source image is low, and it can be considered that this decoded image has a large image quality loss in the encoding process, so the decoded image is determined as the first decoded image.

[0045] For example, in the process of similarity comparison, the similarity of each decoded image and its corresponding source image can be compared by structural similarity index measurement, SSIM, peak signal-to-noise ratio (PSNR) or video multi-method assessment fusion (VMAF) method.

[0046] Taking the SSIM method as an example, SSIM can evaluate the overall similarity between the decoded image and the corresponding source image by comparing the brightness, contrast and structural information of the image. The similarity is between -1 and 1. A similarity of 1 means that the two images are exactly the same. The similarity threshold can be set to 0.7. If the similarity between the decoded image and the corresponding source image is less than 0.7, it means that the decoded image has serious image quality problems and needs to be repaired.

[0047] The decoded image and the corresponding source image can also be divided into 4 areas according to the same image division method, and the similarity of each corresponding area of ​​the two images is evaluated respectively. Only when the similarity of each area is greater than or equal to the similarity threshold, it is considered that the source image has no quality problems. As long as there is a region whose similarity is less than the similarity threshold, it is considered that the image quality of the decoded image has a large loss, and the decoded image is determined as the first decoded image.

[0048] Since the first decoded image has a large image quality loss during the encoding process, the first encoded image before the first decoded image is decoded will also have image quality problems such as screen distortion, so the problematic first encoded image in the bitstream needs to be repaired.

[0049] S14: repair the first encoded image based on the first image stream to obtain an encoded video file.

[0050] In this embodiment, if a first decoded image whose similarity with the source image is lower than a set similarity threshold is found in the decoded second image stream, it means that the quality of the first decoded image is poor or there is a picture quality problem, then the system will repair the first decoded image based on the original first image stream.

[0051] This means that the system re-encodes or adjusts the encoded images that do not meet the similarity requirements by referring to the original image data in the first image stream, and finally generates a higher quality, lossless or encoded video file that is closer to the original image. This repair process can improve the video quality and ensure that the image in the final video file is clearer and more similar to the original picture.

[0052] In the present application, by comparing the similarity between the image in the original first image stream and the decoded image, the first encoded image that has experienced quality loss during the encoding process can be accurately identified, and the second encoded image that needs to be encoded with reference to the first encoded image can be identified, and then the first encoded image can be repaired based on the first image stream. In this way, the similarity between the original image and the decoded image can be quantified as a reference for image quality loss during the encoding process, so as to accurately infer the first encoded image with a large quality loss, and repair the first encoded image, which accurately identifies the encoded image in the code stream that may have a distorted screen, thereby effectively completing the video repair.

[0053] In some embodiments, S14 includes:

[0054] The first coded image and the second coded image in the code stream are repaired based on the first image stream to obtain a coded video file, wherein the second coded image is a coded image that is encoded with reference to the first coded image.

[0055] In an embodiment of the present application, after determining a first encoded image with a large image quality loss from the code stream, a second encoded image that needs to be encoded with reference to the first encoded image can be further determined from the code stream. Since the first encoded image has image quality problems, the second encoded image encoded with reference to the first encoded image may also have image quality problems and also need to be repaired.

[0056] Then the source images of the first encoded image and the second encoded image in the first image stream can be re-encoded, and the re-encoded encoded images can replace the first encoded image and the second encoded image to repair the encoded images with quality loss in the code stream and obtain a higher-quality encoded video file.

[0057] In this way, the similarity between the original image and the decoded image can be quantified as a reference for the image quality loss during the encoding process, so as to accurately infer the first encoded image with a larger quality loss, and automatically identify the second encoded image encoded with reference to the first encoded image, and repair both the first encoded image and the second encoded image, which can ensure that the encoded images that may have problems in the bitstream are not missed, thereby effectively completing the video repair.

[0058] In some embodiments, the repairing the first coded image and the second coded image in the code stream based on the first image stream comprises:

[0059] Determine a first image corresponding to the first encoded image in the first image stream, and a second image corresponding to the second encoded image in the first image stream;

[0060] Re-encoding the first image and the second image respectively to obtain a third encoded image and a fourth encoded image;

[0061] The first coded image in the code stream is replaced by the third coded image, and the second coded image in the code stream is replaced by the fourth coded image.

[0062] In this embodiment, after the first coded image and the second coded image with degraded quality or distorted screen have been determined from the code stream, it is necessary to repair the first coded image and the second coded image in the code stream.

[0063] Specifically, the first image corresponding to the first encoded image and the second image corresponding to the second encoded image can be found in the initial first image stream, and then the first image is re-encoded to generate a new third encoded image, and the second image is re-encoded to generate a fourth encoded image.

[0064] After re-encoding, the generated third coded image may be used to replace the first coded image in the code stream, and the fourth coded image may be used to replace the second coded image in the code stream, thereby obtaining a repaired code stream.

[0065] In this embodiment, the source image corresponding to the coded image with serious quality loss is extracted from the original first image stream, and the source image is re-encoded to replace the coded image with problems. In this way, the problem of screen distortion or image quality loss caused by compression during the encoding process can be effectively repaired, and the video quality of the code stream can be restored.

[0066] In some embodiments, before respectively re-encoding the first image and the second image to obtain a third encoded image and a fourth encoded image, the method further includes:

[0067] Refresh data cached inside an encoder, where the encoder is used to re-encode the first image and the second image.

[0068] In this embodiment, before re-encoding the first image and the second image, data cached inside the encoder may be refreshed to ensure that the encoder does not use previous residual data when re-encoding the first image and the second image, thereby affecting encoding accuracy.

[0069] Specifically, during the video encoding process, the encoder will internally cache some reference data related to the previously encoded images, which are used to predict and compress the encoded images in the subsequent encoding process. If the encoder is not refreshed, these cached data may contain information that previously caused the screen to be distorted, thereby affecting the new encoding process and affecting the image quality of the re-encoded third and fourth encoded images.

[0070] In the above manner, the encoder can be refreshed to ensure that the re-encoded third coded image and the fourth coded image are based on clean reference data, thereby improving the quality of the restored video.

[0071] In some embodiments, the re-encoding the first image and the second image respectively to obtain a third encoded image and a fourth encoded image comprises:

[0072] Obtain a target coded image having a smallest corresponding timestamp between the first coded image and the second coded image;

[0073] Determining a third image corresponding to the target coded image in the first image stream;

[0074] The third image and images subsequent to the third image in the first image stream are encoded to obtain the third encoded image and the fourth encoded image.

[0075] In this embodiment, each coded image has a timestamp, which is a mark of the coded image on the time axis and is used to indicate the specific time position of the coded image in the bitstream. During the re-encoding process, a target coded image with the smallest timestamp can be determined from the first coded image and the second coded image, and the target coded image is the starting frame image that needs to be re-encoded.

[0076] After determining the target coded image, the third image corresponding to the target coded image in the first image stream can be searched according to the timestamp of the target coded image. Next, the system can remove the first code stream image and the second code stream image in the code stream, and start encoding again from the third image in the first image stream, and splice the encoded coded image to the end of the code stream after removing the first coded image and the second code stream image, thereby realizing code stream repair.

[0077] In this way, the earliest problematic coded image in the bitstream can be accurately determined by searching for the target coded image with the smallest timestamp, ensuring that repair is started from the earliest problematic coded image in the bitstream to avoid omissions in repair.

[0078] In some embodiments, before repairing the first coded image and the second coded image in the code stream based on the first image stream, the method further includes:

[0079] All coded images other than the first coded image in the image group where the first coded image is located are determined as the second coded images.

[0080] In this embodiment, a group of pictures (Group of Pictures, GOP) refers to a group of consecutive coded pictures, usually consisting of an I frame, multiple P frames and B frames. The code stream includes multiple groups of pictures. Figure 2 As shown, a group of pictures may include 30 coded pictures.

[0081] Since there is a strong dependency between the coded images in the same image group, if it is detected that the first coded image has a large quality loss, it is very likely that all the coded images in the image group where the first coded image is located will be affected. Therefore, all the coded images in the image group where the first coded image is located can be determined as the second coded images for repair.

[0082] By adopting the above method, after detecting the first coded image with quality problems, all coded images in the image group where the first coded image is located can be replaced to avoid the spread of quality problems such as screen distortion in the image group and ensure the quality of the code stream.

[0083] In some embodiments, before repairing the first coded image and the second coded image in the code stream based on the first image stream, the method further includes:

[0084] determining an image frame type of the first encoded image;

[0085] The second coded image is determined according to the image frame type.

[0086] In this embodiment, after determining the first coded image in the code stream, the image frame type of the first coded image can be identified, and the first coded image may be an I frame, a P frame, or a B frame. Among them, the I frame is an independently coded image frame, the P frame is coded with reference to the previous image frame in the image stream, and the B frame is coded with reference to the previous image frame and the next image frame in the image stream.

[0087] From the above, it can be seen that different types of encoded images have different dependencies in the encoding process. Therefore, the dependency between the first encoded image and other encoded images in the encoding process can be determined according to the image frame type of the first encoded image, and the second encoded image that needs to be repaired can be determined based on this dependency.

[0088] In the above manner, the dependency between the first coded image and other coded images can be determined based on the image frame type of the first coded image, and it can be deduced from this which other coded images also need to be repaired together, ensuring that all coded images associated with the first coded image are repaired, avoiding the impact of incomplete repair of problem frames on the code stream quality.

[0089] In some embodiments, determining the second coded image according to the image frame type comprises:

[0090] In the case that the first encoded image is a P frame or an I frame, the encoded image in the target image group after the first encoded image in the encoder is determined as the second encoded image, wherein the I frame is an independently encoded image frame, the P frame is an image frame encoded with reference to a previous image frame in a reference image stream, and wherein the target image group is the image group where the first encoded image is located.

[0091] In this embodiment, the structure of the picture group in the code stream generally includes the following: Figure 2The three types of GOP structures shown in the figure take each GOP including 30 coded images as an example. The first GOP S301 only includes I frames and P frames, and the structure of the GOP is IPPP..., that is, the first coded image of the GOP is an I frame, and the subsequent coded images are all P frames. The second GOP S302 includes I frames, P frames, and B frames, and the structure of the GOP is IPBPB..., that is, the first coded image of the GOP is an I frame, and the subsequent coded images are arranged alternately with P frames and B frames. The third GOP S303 includes I frames, P frames, and B frames, and the structure of the GOP is IPBBBP..., that is, the first coded image of the GOP is an I frame, and the subsequent coded images are arranged alternately with P frames and multiple B frames.

[0092] See also Figure 2 Among the three image group structures, the I frame is an independently encoded image frame, which can be encoded independently, while the P frame and B frame following the I frame usually need to rely on the I frame for encoding. Therefore, if the I frame has problems such as screen distortion, the subsequent B frame and P frame that rely on it for encoding cannot be encoded correctly, and there is a high possibility of quality problems.

[0093] See also Figure 2 There are three kinds of picture group structures in the image, and the P frame depends on the previous I frame or P frame for encoding. If the P frame has a distorted screen, the subsequent P frame or B frame that depends on this P frame cannot be encoded correctly.

[0094] like Figure 3 As shown, an I frame or P frame with a distorted screen is detected in the first image group S401, that is, the I frame or P frame, and the P frame after the I frame or P frame can be removed and replaced; an I frame or P frame with a distorted screen is detected in the third image group S402, that is, the I frame or P frame, and the B frame after the I frame or P frame can be removed and replaced.

[0095] Therefore, if the first encoded image with quality problems such as screen distortion is a P frame or an I frame, it is necessary to re-encode the subsequent encoded images of the first encoded image in the encoder to ensure that all image frames in the entire dependency chain in the bitstream can be processed correctly to prevent the encoding of subsequent frames from being affected by the damage of a single frame.

[0096] In some embodiments, determining the second coded image according to the image frame type comprises:

[0097] In the case that the first coded image is a B frame, the coded images in the target image group after the first coded image in the encoder and the N coded images before the first coded image are determined as the second coded images, the B frame is an image frame for encoding a previous image frame and a next image frame in a reference image stream, wherein N is 0 or a positive integer, and the N coded images are all coded images between the first coded image and the first P frame or I frame before the first coded image, wherein the target image group is the image group where the first coded image is located.

[0098] In this embodiment, a B frame is an image frame that is encoded with reference to both the previous image frame and the next image frame in the image stream. Figure 2 In the three image group structures, if the quality of the B frame is damaged, then the P frame or B frame that depends on the B frame after the B frame may not be correctly encoded, and if the encoded image before the B frame of the first encoded image is also a B frame, the B frame before the first encoded image may also not be correctly encoded.

[0099] like Figure 4 As shown, a B frame with a distorted screen is detected in the third image group S403, that is, the B frame, the coded images before the B frame until the first P frame or I frame, and the B frame after the B frame can be removed and replaced. Similarly, a B frame with a distorted screen is detected in the third image group S404, that is, the B frame, the coded images before the B frame until the first I frame, and the B frame after the B frame can be removed and replaced.

[0100] Therefore, if the first encoded image with quality problems such as screen distortion is a B frame, it is necessary to re-encode all the subsequent encoded images in the encoder, and also re-encode the B frame before the first encoded image, to ensure that all image frames in the entire dependency chain in the bitstream can be processed correctly, and to prevent the encoding of subsequent frames from being affected by the damage of a single frame.

[0101] Specifically, in some embodiments, when a distorted screen frame is detected in the bitstream, the distorted screen frame and the image frame associated with the distorted screen frame are discarded. Figure 6 The discarded distorted screen frame and the image frame associated with the distorted screen frame are selected in a manner.

[0102] S601, first determine the type of the distorted screen frame where the distorted screen appears, and select a solution for discarding the distorted screen frame and image frames associated with the distorted screen frame;

[0103] S602, selecting a general solution for frame loss;

[0104] S603 and S604, when the general solution is selected, no matter the distorted screen frame is an I frame, a P frame or a B frame, all GOP groups where the distorted screen frame is located are discarded.

[0105] S605, selecting a refined solution for frame loss;

[0106] S606 and S607, when the refined solution is selected, if the type of the distorted frame is an I frame or a P frame, the distorted frame and the image frames after the distorted frame may be discarded.

[0107] S608 and S609, when the refined solution is selected, if the type of the distorted screen frame is a B frame, then the distorted screen frame, the image frames after the distorted screen frame, and all image frames from the previous P frame to the distorted screen frame can be discarded.

[0108] S610, no matter whether the general solution or the detailed solution is selected, after discarding the distorted screen frame and the image frames associated with the distorted screen frame, it is necessary to search for the minimum timestamp in the discarded image frames.

[0109] S611, and output the timestamp. Specifically, each image frame has a corresponding timestamp. The starting frame to be repaired can be found from the cached original image stream through the minimum timestamp.

[0110] In some embodiments, the method further comprises:

[0111] Copying the first image stream to obtain a third image stream;

[0112] In a case where there is a second decoded image in the second image stream, determining in the code stream a fifth coded image before decoding the second decoded image, wherein the second decoded image is a decoded image whose similarity to a copy of a corresponding source image in the third image stream is less than a similarity threshold;

[0113] The fifth coded image and the sixth coded image in the code stream are repaired based on the third image stream to obtain a coded video file, wherein the sixth coded image is a coded image that is encoded with reference to the fifth coded image.

[0114] In this embodiment, in the process of repairing the code stream, the cached first image stream can be reused to repair the coded image with a distorted screen in the code stream, or when the original first image stream is obtained, the first image stream can be copied to obtain a copied third image stream, and the third image stream can be used to repair the coded image with a distorted screen in the code stream.

[0115] Specifically, in the process of repairing the distorted coded images in the code stream through the third image stream, if it is detected that there is a second decoded image in the second image stream after decoding the code stream, and the similarity with the copy image of the corresponding source image is less than a similarity threshold, then the fifth coded image before the decoding of the second decoded image can be determined, and then the fifth coded image and the sixth coded image in the code stream are determined as coded images that need to be repaired, and the copy images of the source images of the fifth coded image and the sixth coded image in the third image stream are encoded, and the re-encoded coded images replace the fifth coded image and the sixth coded image, so as to repair the coded images with severe image quality loss in the code stream and obtain a coded video file with higher image quality.

[0116] In addition to I-frames, P-frames and B-frames, the bitstream also includes header information, which includes key information of the subsequent video frames, such as encoding format, resolution, frame rate, color space, audio information, etc. Figure 5 As shown in S405, when the re-encoded frame is spliced ​​to the end of the original code stream, the spliced ​​part includes not only the re-encoded coded image, but also a new header information (Header) set before the coded image, and the new header information is used to describe the key information of the re-encoded coded image.

[0117] In some embodiments, the video processing method may be as follows Figure 7 As shown:

[0118] S101, the user can start the recording function, and the recorded video data starts to flow into the encoder.

[0119] S102, after the video starts recording, an original image stream is generated. The system can back up each frame of the original image stream, wherein for any image 1 in the original image stream, a backup image 2 is generated based on image 1, and the content of image 2 is completely consistent with the content of image 1.

[0120] S103: The original image stream may be sent to an encoder of an electronic device for encoding.

[0121] S104: The encoder encodes the received original image stream to generate a code stream.

[0122] S105, the system sends the encoded bitstream to the decoder. Specifically, there is no need to encapsulate the bitstream. Since the width, height and image information of the encoder and the like of the source image corresponding to the image in the bitstream can be obtained from the original image stream, a decoder can be directly created based on the image information, and the decoder can be applied to decode the bitstream.

[0123] S106, the electronic device includes a distorted screen detection module, and the decoded image can be sent to the distorted screen detection module. The distorted screen detection module can compare the decoded image with the image in the original image stream to identify the first encoded image with distorted screen.

[0124] S107, the electronic device includes a flower screen repair module. If the first coded image with a flower screen is detected, the first coded image can be sent to the flower screen repair module. The flower screen repair module can discard the first coded image of the flower screen and the second coded image associated with the first coded image, and obtain the image that needs to be re-encoded in the cached image by querying and calculating from the cached image 2. Send the corresponding cached image for encoding again. Before sending it for encoding, the encoder will be refreshed (flush operation) to ensure that the residual image data inside the encoder is refreshed. Append the encoded fragment to the end of the code stream.

[0125] S108, the electronic device includes an image cache module, and the system can back up each frame of the original image stream and store the backed-up image in the image cache module.

[0126] S109, after encoding the cached image, if the distorted screen is repaired successfully, the system saves the processed image stream as an MP4 format file.

[0127] In other embodiments, the video processing method may be as follows: Figure 8 As shown, the encoder contains a decoding process, and the decoding operation inside the encoder is reused to complete the image restoration. In this process, the original image stream can be cached in the cache module inside the encoder. There is no need to back up the original image stream, and the cached original image stream can be directly reused. Specifically:

[0128] S201, the user can start the recording function, and the recorded video data starts to flow into the encoder.

[0129] S202: After the video starts recording, an original image stream is generated.

[0130] S203, caching the images in the original image stream inside the encoder. Since the images in the original image stream are cached inside the encoder in this embodiment, the original image stream can be multiplexed.

[0131] S204: After the original image stream is cached, the original image stream may be sent to an encoder.

[0132] S205, before encoding the original image stream, refreshing the encoder.

[0133] S206 and S207, the encoder encodes the received original image stream to generate a code stream.

[0134] S208, the encoder includes a decoding module, sends the code stream to the decoding module, and uses the decoding module to decode the code stream.

[0135] S209, the electronic device includes a distorted screen detection module, and the decoded image can be sent to the distorted screen detection module. The distorted screen detection module can compare the decoded image with the image in the original image stream to identify the first encoded image with a distorted screen.

[0136] S210, the electronic device includes a distorted screen repair module. If a first coded image with a distorted screen is detected, the first coded image can be sent to the distorted screen repair module. The distorted screen repair module can discard the first coded image with a distorted screen and a second coded image associated with the first coded image, and reuse the image that needs to be re-encoded in the cached original image stream. The corresponding cached image is then sent for encoding.

[0137] S211 and S212, after encoding the cached image, if the distorted screen is repaired successfully, the system saves the processed image stream as an MP4 format file.

[0138] Fig. 9 is a schematic diagram of the structure of a video processing device provided by another embodiment of the present application. Fig. 9 As shown, the video processing device may include:

[0139] The encoding module 901 is used to encode the first image stream to obtain a code stream;

[0140] A decoding module 902, configured to decode the code stream to obtain a second image stream;

[0141] A first determining module 903 is configured to determine, in the bitstream, a first coded image before decoding the first decoded image, if there is a first decoded image in the second image stream, wherein the first decoded image is a decoded image whose similarity to the corresponding source image in the first image stream is less than a similarity threshold;

[0142] The first repairing module 904 is configured to repair the first coded image based on the first image stream to obtain a coded video file.

[0143] In the present application, by comparing the similarity between the image in the original first image stream and the decoded image, the first encoded image that has experienced quality loss during the encoding process can be accurately identified, and the second encoded image that needs to be encoded with reference to the first encoded image can be identified, and then the first encoded image can be repaired based on the first image stream. In this way, the similarity between the original image and the decoded image can be quantified as a reference for image quality loss during the encoding process, so as to accurately infer the first encoded image with a large quality loss, and repair the first encoded image, which accurately identifies the encoded image in the code stream that may have a distorted screen, thereby effectively completing the video repair.

[0144] In another optional example, the first repair module 904 includes:

[0145] A repairing unit is used to repair the first coded image and the second coded image in the code stream based on the first image stream to obtain a coded video file, wherein the second coded image is a coded image encoded with reference to the first coded image.

[0146] In another optional example, the repair unit includes:

[0147] A first determining subunit, configured to determine a first image corresponding to the first coded image in the first image stream, and a second image corresponding to the second coded image in the first image stream;

[0148] A first encoding subunit, configured to re-encode the first image and the second image respectively to obtain a third encoded image and a fourth encoded image;

[0149] The replacement subunit is used to replace the first coded image in the code stream with the third coded image, and to replace the second coded image in the code stream with the fourth coded image.

[0150] In another optional example, the video processing device includes:

[0151] A refresh module is used to refresh data cached inside an encoder, and the encoder is used to re-encode the first image and the second image.

[0152] In another optional example, the first encoding subunit is specifically used for:

[0153] Obtain a target coded image having a smallest corresponding timestamp between the first coded image and the second coded image;

[0154] Determining a third image corresponding to the target coded image in the first image stream;

[0155] The third image and images subsequent to the third image in the first image stream are encoded to obtain the third encoded image and the fourth encoded image.

[0156] In another optional example, the video processing device includes:

[0157] The second determining module is configured to determine all coded images in the image group where the first coded image is located, except the first coded image, as the second coded images.

[0158] In another optional example, the video processing device includes:

[0159] A third determination module, used to determine the image frame type of the first encoded image;

[0160] A fourth determination module is used to determine the second encoded image according to the image frame type.

[0161] In another optional example, the fourth determining module includes:

[0162] A first determination unit is used to determine a coded image in a target image group after the first coded image in the encoder as the second coded image when the first coded image is a P frame or an I frame, wherein the I frame is an independently coded image frame, the P frame is an image frame encoded with reference to a previous image frame in a reference image stream, and the target image group is the image group where the first coded image is located.

[0163] In another optional example, the fourth determining module includes:

[0164] A second determination unit is used to determine, when the first coded image is a B frame, a coded image in a target image group after the first coded image in the encoder and N coded images before the first coded image as the second coded image, the B frame being an image frame for encoding a previous image frame and a next image frame in a reference image stream, wherein N is 0 or a positive integer, and the N coded images are all coded images between the first coded image and the first P frame or I frame before the first coded image, wherein the target image group is the image group where the first coded image is located.

[0165] In another optional example, the video processing device includes:

[0166] a copy module, configured to copy the first image stream to obtain a third image stream;

[0167] a fifth determining module, configured to determine, in the bitstream, a fifth coded image before decoding the second decoded image, if there is a second decoded image in the second image stream, wherein the second decoded image is a decoded image whose similarity to a copy of a corresponding source image in the third image stream is less than a similarity threshold;

[0168] The second repair module is used to repair the fifth coded image and the sixth coded image in the code stream based on the third image stream to obtain a video file, wherein the sixth coded image is a coded image encoded with reference to the fifth coded image.

[0169] The video processing device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., and the embodiment of the present application is not specifically limited.

[0170] The video processing device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0171] The video processing device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0172] Alternatively, if Fig.10As shown, an embodiment of the present application also provides an electronic device 100, including a processor 110, a memory 119, and a program or instruction stored in the memory 119 and executable on the processor 110. When the program or instruction is executed by the processor 110, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0173] It should be noted that the electronic devices in the embodiments of the present application include the above-mentioned mobile electronic devices and non-mobile electronic devices.

[0174] Please refer to Fig.11 , Fig.11 The hardware structure diagram of an electronic device for implementing an embodiment of the present application is shown in FIG. The electronic device 100 includes, but is not limited to, a radio frequency unit 121, a network module 122, an audio output unit 123, an input unit 124, a sensor 125, a display unit 126, a user input unit 127, an interface unit 128, a memory 129, and a processor 120.

[0175] Those skilled in the art will appreciate that the electronic device 100 may also include a power source (such as a battery) for supplying power to various components, and the power source may be logically connected to the processor 120 through a power management system, thereby implementing functions such as managing charging, discharging, and power consumption management through the power management system. Fig.11 The electronic device structure shown in the figure does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently, which will not be described in detail here.

[0176] The processor 120 is used to encode the first image stream to obtain a code stream;

[0177] The processor 120 is configured to decode the code stream to obtain a second image stream;

[0178] The processor 120 is configured to determine, in the bitstream, a first coded image before decoding the first decoded image, if there is a first decoded image in the second image stream, wherein the first decoded image is a decoded image whose similarity to a corresponding source image in the first image stream is less than a similarity threshold;

[0179] The processor 120 is configured to repair the first encoded image based on the first image stream to obtain an encoded video file.

[0180] In the present application, by comparing the similarity between the image in the original first image stream and the decoded image, the first encoded image that has experienced quality loss during the encoding process can be accurately identified, and the second encoded image that needs to be encoded with reference to the first encoded image can be identified, and then the first encoded image can be repaired based on the first image stream. In this way, the similarity between the original image and the decoded image can be quantified as a reference for image quality loss during the encoding process, so as to accurately infer the first encoded image with a large quality loss, and repair the first encoded image, which accurately identifies the encoded image in the code stream that may have a distorted screen, thereby effectively completing the video repair.

[0181] In another optional example, the processor 120 is configured to:

[0182] The first coded image and the second coded image in the code stream are repaired based on the first image stream to obtain a coded video file, wherein the second coded image is a coded image that is encoded with reference to the first coded image.

[0183] In another optional example, the processor 120 is configured to:

[0184] Determine a first image corresponding to the first encoded image in the first image stream, and a second image corresponding to the second encoded image in the first image stream;

[0185] Re-encoding the first image and the second image respectively to obtain a third encoded image and a fourth encoded image;

[0186] The first coded image in the code stream is replaced by the third coded image, and the second coded image in the code stream is replaced by the fourth coded image.

[0187] In another optional example, the processor 120 is configured to:

[0188] Refresh data cached inside an encoder, where the encoder is used to re-encode the first image and the second image.

[0189] In another optional example, the processor 120 is further configured to:

[0190] Obtain a target coded image having a smallest corresponding timestamp between the first coded image and the second coded image;

[0191] Determining a third image corresponding to the target coded image in the first image stream;

[0192] The third image and images subsequent to the third image in the first image stream are encoded to obtain the third encoded image and the fourth encoded image.

[0193] In another optional example, the processor 120 is further configured to:

[0194] All coded images other than the first coded image in the image group where the first coded image is located are determined as the second coded images.

[0195] In another optional example, the processor 120 is further configured to:

[0196] determining an image frame type of the first encoded image;

[0197] The second coded image is determined according to the image frame type.

[0198] In another optional example, the processor 120 is further configured to:

[0199] In the case that the first encoded image is a P frame or an I frame, the encoded image in the target image group after the first encoded image in the encoder is determined as the second encoded image, wherein the I frame is an independently encoded image frame, the P frame is an image frame encoded with reference to a previous image frame in a reference image stream, and wherein the target image group is the image group where the first encoded image is located.

[0200] In another optional example, the processor 120 is further configured to:

[0201] In the case that the first coded image is a B frame, the coded images in the target image group after the first coded image in the encoder and the N coded images before the first coded image are determined as the second coded images, the B frame is an image frame for encoding a previous image frame and a next image frame in a reference image stream, wherein N is 0 or a positive integer, and the N coded images are all coded images between the first coded image and the first P frame or I frame before the first coded image, wherein the target image group is the image group where the first coded image is located.

[0202] In another optional example, the processor 120 is further configured to:

[0203] Copying the first image stream to obtain a third image stream;

[0204] In a case where there is a second decoded image in the second image stream, determining in the code stream a fifth coded image before decoding the second decoded image, wherein the second decoded image is a decoded image whose similarity to a copy of a corresponding source image in the third image stream is less than a similarity threshold;

[0205] The fifth coded image and the sixth coded image in the code stream are repaired based on the third image stream to obtain a video file, wherein the sixth coded image is a coded image encoded with reference to the fifth coded image.

[0206] It should be understood that in the embodiment of the present application, the input unit 124 may include a graphics processor (Graphics Processing Unit, GPU) 1241 and a microphone 1242, and the graphics processor 1241 processes the image data of a static picture or video obtained by an image capture device (such as a camera) in a video capture mode or an image capture mode. The display unit 126 may include a display panel 1261, and the display panel 1261 may be configured in the form of a liquid crystal display, an organic light emitting diode, etc. The user input unit 127 includes a touch panel 1271 and at least one of other input devices 1272. The touch panel 1271 is also called a touch screen. The touch panel 1271 may include two parts: a touch detection device and a touch controller. Other input devices 1272 may include, but are not limited to, a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which will not be repeated here.

[0207] The memory 129 can be used to store software programs and various data. The memory 129 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory 129 may include a volatile memory or a non-volatile memory, or the memory 129 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory 129 in the embodiment of the present application includes but is not limited to these and any other suitable types of memory.

[0208] The processor 120 may include one or more processing units; optionally, the processor 120 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor 120.

[0209] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, each process of the above-mentioned video processing method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0210] The processor is the processor in the electronic device in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0211] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface and the processor are coupled, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned video processing method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0212] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0213] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned video processing method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0214] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0215] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0216] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A video processing method, characterized in that: include: Encoding the first image stream to obtain a code stream; Decoding the code stream to obtain a second image stream; In a case where there is a first decoded image in the second image stream, determining in the code stream a first coded image before decoding the first decoded image, wherein the first decoded image is a decoded image whose similarity to a corresponding source image in the first image stream is less than a similarity threshold; The first encoded image is repaired based on the first image stream to obtain an encoded video file.

2. The method according to claim 1, characterized in that The repairing the first coded image based on the first image stream to obtain a coded video file comprises: The first coded image and the second coded image in the code stream are repaired based on the first image stream to obtain a coded video file, wherein the second coded image is a coded image encoded with reference to the first coded image.

3. The method according to claim 2, characterized in that The repairing the first coded image and the second coded image in the code stream based on the first image stream comprises: Determine a first image corresponding to the first encoded image in the first image stream, and a second image corresponding to the second encoded image in the first image stream; Re-encoding the first image and the second image respectively to obtain a third encoded image and a fourth encoded image; The first coded image in the code stream is replaced by the third coded image, and the second coded image in the code stream is replaced by the fourth coded image.

4. The method according to claim 3, characterized in that Before respectively re-encoding the first image and the second image to obtain a third encoded image and a fourth encoded image, the method further includes: Refresh data cached inside an encoder, where the encoder is used to re-encode the first image and the second image.

5. The method according to claim 3, characterized in that: The re-encoding the first image and the second image respectively to obtain a third encoded image and a fourth encoded image comprises: Obtain a target coded image having a smallest corresponding timestamp between the first coded image and the second coded image; Determining a third image corresponding to the target coded image in the first image stream; The third image and images subsequent to the third image in the first image stream are encoded to obtain the third encoded image and the fourth encoded image.

6. The method according to claim 2, characterized in that Before repairing the first coded image and the second coded image in the code stream based on the first image stream, the method further includes: All coded images other than the first coded image in the image group where the first coded image is located are determined as the second coded images.

7. The method according to claim 2, characterized in that Before repairing the first coded image and the second coded image in the code stream based on the first image stream, the method further includes: determining an image frame type of the first encoded image; The second coded image is determined according to the image frame type.

8. The method according to claim 7, characterized in that The determining the second coded image according to the image frame type comprises: In the case that the first encoded image is a P frame or an I frame, the encoded image in the target image group after the first encoded image in the encoder is determined as the second encoded image, wherein the I frame is an independently encoded image frame, the P frame is an image frame encoded with reference to a previous image frame in a reference image stream, and wherein the target image group is the image group where the first encoded image is located.

9. The method according to claim 7, characterized in that: The determining the second coded image according to the image frame type comprises: In the case that the first coded image is a B frame, the coded images in the target image group after the first coded image in the encoder and the N coded images before the first coded image are determined as the second coded images, the B frame is an image frame for encoding a previous image frame and a next image frame in a reference image stream, wherein N is 0 or a positive integer, and the N coded images are all coded images between the first coded image and the first P frame or I frame before the first coded image, wherein the target image group is the image group where the first coded image is located.

10. The method according to claim 1, characterized in that The method further comprises: Copying the first image stream to obtain a third image stream; In a case where there is a second decoded image in the second image stream, determining in the code stream a fifth coded image before decoding the second decoded image, wherein the second decoded image is a decoded image whose similarity to a copy of a corresponding source image in the third image stream is less than a similarity threshold; The fifth coded image and the sixth coded image in the code stream are repaired based on the third image stream to obtain a video file, wherein the sixth coded image is a coded image encoded with reference to the fifth coded image.

11. A video processing device, characterized in that: include: An encoding module, used for encoding the first image stream to obtain a code stream; A decoding module, used for decoding the code stream to obtain a second image stream; a first determining module, configured to determine, in the bitstream, a first coded image before decoding the first decoded image, if there is a first decoded image in the second image stream, wherein the first decoded image is a decoded image whose similarity to a corresponding source image in the first image stream is less than a similarity threshold; The first repair module is used to repair the first coded image based on the first image stream to obtain a coded video file.

12. The device according to claim 11, characterized in that The first repair module comprises: A repairing unit is used to repair the first coded image and the second coded image in the code stream based on the first image stream to obtain a coded video file, wherein the second coded image is a coded image encoded with reference to the first coded image.

13. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the video processing method according to any one of claims 1 to 10 are implemented.

Images