Image processing method and related devices

Abstract

Provided in the present application are an image processing method and related devices. The method comprises: obtaining a first video frame set, wherein a background in the first video frame set is in a static state; on the basis of a first video frame and a second video frame, determining a first image region in the second video frame, wherein the magnitude of a motion vector of the first image region in the second video frame is less than or equal to a first preset threshold value; filtering the first image region in the second video frame to obtain a filtered second video frame, wherein the magnitude of the motion vector of the first image region in the filtered second video frame is 0; and on the basis of the filtered second video frame, obtaining a second video frame set. By means of the method, micro-motion in a micro-motion image region (i.e., a first image region) in a second video frame is removed, thereby reducing the data volume after video encoding and further improving the compression ratio of video encoding.

Description

Image processing method and related device

[0001] The present application claims priority to the Russian patent application No. RU2024123679, filed on August 16, 2024, with the Russian Federal Service for Intellectual Property, the Russian patent application having the invention title of “Image processing method and related device”, the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of image processing, and more particularly, to an image processing method, an image processing device, a computing device cluster, a chip system, a computer program product and a computer readable storage medium. BACKGROUND

[0003] With the improvement of image or video acquisition devices, the current image or video data is large, and a lot of resources are needed for storage or transmission. In order to save the resource demand of video in storage or transmission, the video can be encoded to achieve the purpose of compressing data volume. The current video can be divided into steady video and non-steady video. Among them, the steady video is a video in which the scene does not switch and the background is in a static state. The non-steady video is a video in which the scene has switched and / or the background is in a motion state. The current video encoding method generally extracts the background and foreground in the video, and saves the motion vector and residual error of the foreground part in the video frame by calculation, reduces the redundant data in storage or transmission, and thus achieves the purpose of compressing data volume. However, the current video encoding method is not ideal in the micro-motion area. The micro-motion area is an area that slightly shakes in the video, such as an area including grass, leaves, flowers, animal fur, etc. Because there is a lot of residual error in the micro-motion area after motion prediction, the data volume after encoding is still large, which still needs a lot of resources in the storage or transmission process.

[0004] Therefore, how to process the micro-motion area in the video frame to improve the compression rate of video encoding has become a problem to be solved. SUMMARY

[0005] The present application provides an image processing method, an image processing device, a computing device cluster, a chip system, a computer program product and a computer readable storage medium, which can process the micro-motion area in the video frame, reduce the data volume after video encoding, and thus improve the compression rate of video encoding.

[0006] In a first aspect, an image processing method is provided. The method comprises: obtaining a first video frame set, the first video frame set comprising a plurality of video frames, and a background in the first video frame set being in a static state; determining, according to a first video frame and a second video frame, a first image region in the second video frame, the second video frame belonging to the first video frame set, a magnitude of a motion vector of the first image region in the second video frame being less than or equal to a first preset threshold, the first video frame being one frame before the second video frame in the first video frame set, or the first video frame being a frame obtained by filtering one frame before the second video frame in the first video frame set; filtering the first image region in the second video frame to obtain a filtered second video frame, a magnitude of a motion vector of the first image region in the filtered second video frame being 0; and obtaining a second video frame set according to the filtered second video frame.

[0007] The magnitude of the motion vector of the first image region in the second video frame is determined according to the first video frame and the second video frame, and the magnitude of the motion vector of the first image region in the filtered second video frame is determined according to the first video frame and the filtered second video frame.

[0008] In the embodiments of the present application, by filtering the micro-motion image region (i.e., the first image region) in the second video frame, the micro-motion in the micro-motion image region is removed, so that the image region becomes a static region after filtering, thereby reducing the data amount after video encoding, and further improving the compression rate of video encoding.

[0009] In combination with the first aspect, in some implementations, the magnitude of the motion vector of the first image region in the second video frame is greater than 0.

[0010] In the embodiments of the present application, the first image region is a micro-motion region before filtering, and the first image region is a static region after filtering, thereby the data amount after video encoding can be reduced, and further the compression rate of video encoding can be improved.

[0011] In combination with the first aspect, in some implementations, a texture irregularity degree or a motion irregularity degree of the first image region is greater than or equal to a second preset threshold.

[0012] In the embodiments of the present application, by the texture irregularity degree or the motion irregularity degree of the first image region, it is determined that the first image region is a texture irregular region. That is, the first image region is not only a micro-motion region, but also a texture irregular region. By removing the micro-motion in the first image region, the data amount of the first image region after video encoding is reduced, thereby the compression rate of video encoding is improved.

[0013] Exemplarily, the motion irregularity degree of the first image region is determined according to the direction of the motion vector of each pixel in the first image region in the second video frame. The motion irregularity degree of the first image region is used to represent the texture irregularity degree of the first image region.

[0014] With reference to the first aspect, in some implementations, the first video frame and the second video frame are adjacent video frames in the first video frame set, or the first video frame is a frame obtained by filtering one frame adjacent to the second video frame in the first video frame set.

[0015] In the embodiments of the present application, by processing two adjacent video frames in the video frame set, the micro-motions in each micro-motion region in the processed video frame set are removed, so that the data amount after video encoding can be reduced, and the compression rate of video encoding is improved.

[0016] With reference to the first aspect, in some implementations, at least one second image region in the second video frame is determined according to the first video frame and the second video frame, each second image region in the at least one second image region is similar to an image region in the first video frame; and each first image region is determined according to the amplitude of the motion vector of each second image region in the at least one second image region in the second video frame, the amplitude of the motion vector of each second image region in the second video frame being determined according to the first video frame and the second video frame.

[0017] In the embodiments of the present application, by determining the similar image regions in the first video frame and the second video frame, the amplitude of the motion vector of the image region in the second video frame is determined, and then the micro-motion region in the second video frame is determined.

[0018] With reference to the first aspect, in some implementations, at least one third image region is determined according to the amplitude of the motion vector of each second image region in the at least one second image region in the second video frame, each third image region in the at least one third image region has an amplitude of the motion vector in the second video frame less than or equal to a first preset threshold; and the first image region is determined from the at least one third image region, the texture irregularity degree or the motion irregularity degree of the first image region being greater than or equal to a second preset threshold.

[0019] In the embodiments of the present application, after the similar image regions in the first video frame and the second video frame are determined, the image region with micro-motions and texture irregularities in the second video frame is determined according to the amplitude of the motion vector of the image region in the second video frame and the texture irregularity degree of the image region, so that the image region with micro-motions and texture irregularities is filtered, and then the data amount after video encoding can be reduced, and the compression rate of video encoding is improved.

[0020] In some implementations of the first aspect, the pixels in the first image region in the second video frame are replaced with the pixels in the first image region in the reference frame to obtain a filtered second video frame.

[0021] In some implementations of the first aspect, the reference frame is any one of the following: a first video frame in the first video frame set, a first reconstructed frame corresponding to the first video frame set, a background frame corresponding to the first video frame set, or a first video frame obtained by filtering a frame before the second video frame in the first video frame set.

[0022] In the embodiments of the present application, by replacing the pixels in the first image region in the second video frame with the pixels in the first image region in the reference frame, the pixel values in the first image region in the second video frame and the first image region in the reference frame correspond to the same, thereby achieving the effect of removing micro-motions in the first image region.

[0023] In some implementations of the first aspect, when the reference frame is the first video frame in the first video frame set or the first reconstructed frame, and the first video frame is the first video frame in the first video frame set, the second video frame set includes the first video frame and the filtered second video frame.

[0024] In the embodiments of the present application, the first video frame in the first video frame set can be filtered or can not be filtered.

[0025] In some implementations of the first aspect, a third video frame set is obtained, the third video frame set includes a plurality of video frames, scenes in at least two video frames in the third video frame set are different, and / or a background in the third video frame set is in a motion state; scene switching detection is performed on the third video frame set to obtain a fourth video frame set, the fourth video frame set is a subset of the third video frame set, and the fourth video frame set includes a plurality of video frames, scenes in each video frame in the fourth video frame set are the same; the background in the fourth video frame set is analyzed to obtain the first video frame set, and the first video frame set is a subset of the fourth video frame set.

[0026] In some implementations of the first aspect, the first video frame set in the third video frame set is replaced with the second video frame set to obtain a fifth video frame set; the fifth video frame set is encoded to output a first code stream.

[0027] In the embodiments of the present application, when the first video frame set is a subset of the third video frame set, and scenes in other video frames in the third video frame set are different and / or backgrounds are in a motion state, only part or all of the video frames in the first video frame set can be filtered to obtain a second video frame set, and the first video frame set in the third video frame set is replaced by the second video frame set, thereby outputting.

[0028] In a second aspect, an image processing apparatus is provided. The apparatus includes modules for implementing the first aspect or any possible implementation of the first aspect.

[0029] In a third aspect, the present application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method in the first aspect or any possible implementation of the first aspect.

[0030] In a fourth aspect, the present application provides a computer program product including instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method in the first aspect or any possible implementation of the first aspect.

[0031] In a fifth aspect, the present application provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device cluster, cause the computing device cluster to perform the method in the first aspect or any possible implementation of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0032] FIG. 1 is a schematic structural diagram of an image processing system according to an embodiment of the present application.

[0033] FIG. 2 is a schematic flowchart of an image processing method according to an embodiment of the present application.

[0034] FIG. 3 is a schematic flowchart of an image processing method according to an embodiment of the present application.

[0035] FIG. 4 is a schematic flowchart of an image processing method according to an embodiment of the present application.

[0036] FIG. 5 is a schematic diagram of a video frame and an image region with micro-motions and texture irregularities in the video frame according to an embodiment of the present application.

[0037] FIG. 6 is a schematic structural block diagram of an image processing apparatus according to an embodiment of the present application.

[0038] FIG. 7 is a schematic structural diagram of a computing device according to an embodiment of the present application.

[0039] FIG. 8 is a schematic block diagram of a cluster of computing devices, according to an embodiment of the application.

[0040] FIG. 9 is a schematic diagram of a connection between computing devices 700A and 700B over a network, according to an embodiment of the application. DETAILED DESCRIPTION

[0041] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0042] Embodiments of the present application will present various aspects, embodiments or features around a system including a plurality of devices, components, modules, etc. It should be understood and appreciated that each system can include additional devices, components, modules, etc., and / or can not include all of the devices, components, modules, etc. discussed in connection with the figures. Furthermore, a combination of these aspects can also be used.

[0043] In addition, in the embodiments of the present application, the words "example" or "for example" are used to mean serving as an example, instance, or illustration. Any implementation described as "example" in the embodiments of the present application is not necessarily to be construed as preferred or advantageous over other implementations. Rather, use of the word example is intended to present concepts in a concrete manner.

[0044] The business scenarios described in the embodiments of the present application are to make the technical solutions of the embodiments of the present application more clearly, and do not constitute a limitation to the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, as the evolution of technology and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0045] In this specification, the reference "one embodiment" or "some embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" in various places in this specification are not necessarily all referring to the same embodiment, but can refer to one or more but not all embodiments, unless otherwise specified. The terms "including," "comprising," "having" and variations thereof mean "including but not limited to," unless expressly specified otherwise.

[0046] In the embodiments of the present application, "at least one" means one or more, and "multiple" means two or more. The "and / or" describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B can represent the following three cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the front and rear associated objects. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b, or c can represent a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0047] The technical solutions in the embodiments of the present application can be applied to computing devices, such as servers, hosts, personal computers, notebook computers, desktop computers, wearable devices, and other devices with computing capabilities. When the technical solutions in the embodiments of the present application are applied to servers, the servers can be local servers or cloud servers, and the embodiments of the present application are not limited thereto. The technical solutions in the embodiments of the present application can also be applied to video acquisition devices, such as cameras and video cameras, which are used to acquire videos and / or images. The technical solutions in the embodiments of the present application can also be applied to video storage devices, which are used to store videos and / or images. The technical solutions in the embodiments of the present application can be applied to video conferencing, video monitoring, video storage, and other scenarios.

[0048] FIG. 1 is a schematic structural diagram of an image processing system according to an embodiment of the present application. The image processing system 100 in FIG. 1 includes a sending device 110 and a receiving device 120. The sending device 110 includes a processor 112 and a memory 113. The receiving device 120 includes a processor 121 and a memory 122.

[0049] In some embodiments, the sending device 110 further includes an image sensor 111. Alternatively, the image sensor 111 is arranged outside the sending device 110, and the image sensor 111 is connected to the sending device 110. The image sensor 111 is used to acquire video images. The specific type of the image sensor 111 is not limited in the embodiments of the present application, for example, a charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

[0050] In some embodiments, the image processing system 100 further includes a receiving device 130.

[0051] The processor 112 in the sending device 110 is configured to process the data obtained by the image sensor 111. For example, the processor 112 is configured to perform the method provided in the embodiments of the present application. Alternatively, the processor 112 is configured to encode the obtained data, obtain a bitstream, and send the bitstream to the receiving device 120. The type of the processor 113 is not limited in the embodiments of the present application, and for example, includes at least one of a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), and the like. The memory 113 is configured to store at least one of the data collected by the image sensor 111, the data required by the processor 112 in the process of video encoding or performing the method in the embodiments of the present application, or the data generated by the processor 112. The type of the memory 113 is not limited in the embodiments of the present application, and for example, includes a volatile memory and / or a nonvolatile memory.

[0052] For example, the sending device 110 is a computing device or a video acquisition device.

[0053] The processor 121 in the receiving device 120 is configured to receive the bitstream from the sending device 110 and perform decoding. The processor 121 is also configured to perform the method provided in the embodiments of the present application on the decoded bitstream. Alternatively, the processor 121 is configured to re-encode the received bitstream and send the re-encoded bitstream to the receiving device 130. The memory 122 is configured to store at least one of the bitstream received by the receiving device 120, the data obtained after decoding the bitstream, the data required by the processor 121 in the process of performing the method in the embodiments of the present application, or the data generated by the processor 121.

[0054] For example, when the sending device 110 is a computing device, the receiving device 120 is also a computing device. When the sending device 110 is a video acquisition device, the receiving device 120 is a computing device or a video storage device.

[0055] In some embodiments, the receiving device 130 is configured to receive the bitstream from the receiving device 120, and the bitstream is obtained based on at least one video frame processed according to the method provided in the embodiments of the present application.

[0056] For example, the receiving device 130 is a computing device.

[0057] In some embodiments, the image processing system 100 is applied to the following scenarios:

[0058] (1) Scene one: the sending device 110 obtains the collected video images, and performs image processing on the video images according to the method in the embodiments of the present application, to obtain processed video images. The sending device 110 encodes the processed video images, to obtain a code stream, and sends the code stream to the receiving device 120. The receiving device 120 receives the code stream, re-encodes the code stream, and sends the re-encoded code stream to the receiving device 130. Alternatively, the receiving device 120 receives the code stream, and stores the code stream.

[0059] (2) Scene two: the sending device 110 obtains the collected video images, and encodes the video images, to obtain a code stream. The sending device 110 sends the code stream to the receiving device 120. The receiving device 120 decodes the received code stream, and performs image processing on the decoded video images according to the method provided in the embodiments of the present application, to obtain processed video images. The receiving device 120 encodes the processed video images, to obtain a code stream, and sends the code stream to the receiving device 130. Alternatively, the receiving device 120 encodes the processed video images, to obtain a code stream, and stores the code stream.

[0060] FIG. 2 is a schematic flowchart of the image processing method provided in the embodiments of the present application. The method in FIG. 2 is applied in an image processing device, for example, any device in FIG. 1. The method in FIG. 2 includes the following steps.

[0061] 210. Obtain a first video frame set.

[0062] The image processing device obtains a first video frame set, which includes a plurality of video frames. The background in the first video frame set is static. In other words, the scene in each video frame in the first video frame set is the same, and the background of a first video obtained according to the first video frame set is static.

[0063] Optionally, before step 210, the image processing device obtains a third video frame set. The third video frame set includes a plurality of video frames. In the case that the third video frame set is obtained by an image sensor with fixed shooting parameters and without behaviors such as panning, rotation, zooming, etc., the background in the third video frame set is static. In the case that the third video frame set is obtained by an image sensor with non-fixed shooting parameters or with at least one behavior such as panning, rotation, zooming, etc., the scene in at least two video frames in the third video frame set is different, and / or the background in the third video frame set is in motion.

[0064] In a case that the background in the third video frame set is in a static state, the third video frame set is the first video frame set. In a case that the background in the third video frame set is in a motion state, and / or the scenes in at least two video frames in the third video frame set are different, the first video frame set is a subset of the third video frame set. In a case that the first video frame set is a subset of the third video frame set, the first video frame set is determined in a manner as shown in steps 320-330 in FIG. 3.

[0065] In some embodiments, the video frames in the first video frame set are arranged in a normal playing order or a capturing order.

[0066] Optionally, the image processing device captures the third video frame set by itself. Alternatively, the image processing device receives the third video frame set from another device connected to the image processing device, such as an image capturing device, a computing device, a storage device, etc.

[0067] 220, determining a first image region in the second video frame according to the first video frame and the second video frame.

[0068] After the image processing device obtains the first video frame set, the image processing device determines a first video frame and a second video frame in the first video frame set, and determines a first image region in the second video frame. The first image region includes at least one pixel. The magnitude of the motion vector of the first image region in the second video frame is less than or equal to a first preset threshold. The value of the first preset threshold is not limited in the embodiments of the present application. The magnitude of the motion vector of the first image region in the second video frame is determined according to the first video frame and the second video frame.

[0069] Optionally, the image processing device determines the magnitude of the motion vector of at least one image block in the second video frame according to the first video frame and the second video frame. Each of the at least one image block includes at least one pixel. The image processing device determines the first image region according to the magnitude of the motion vector of each image block in the second video frame. The first image region includes one or more image blocks. The magnitude of the motion vector of the first image region in the second video frame is less than or equal to the first preset threshold includes: the magnitude of the motion vector of each image block included in the first image region in the second video frame is less than or equal to the first preset threshold; or, the number of image blocks whose magnitude of the motion vector in the second video frame is less than or equal to the first preset threshold is greater than or equal to a third preset threshold. The method for determining the motion vector of the image block is not limited in the embodiments of the present application, such as a block-level motion estimation method. The value of the third preset threshold is not limited in the embodiments of the present application.

[0070] In some embodiments, the magnitude of the motion vector of the first image block in the second video frame is used to indicate the displacement between the first image block in the second video frame and the second image block in the first video frame. The first image block belongs to the second video frame. The first image block has the same number of pixels as the second image block. The second image block in the first video frame is the image block in the first video frame that is most similar to the first image block. Alternatively, the similarity between the first image block and the second image block is greater than or equal to the similarity between the first image block and other image blocks in the first video frame except the second image block. Alternatively, the similarity between the second image block in the first video frame and the first image block in the second video frame is greater than or equal to a fourth preset threshold. The size of the fourth preset threshold is not limited in the embodiments of the present application.

[0071] For example, the magnitude of the motion vector of the first image block in the second video frame is determined according to the horizontal and vertical displacements of the first image block and the second image block in the video frames.

[0072] For example, assuming that the position of a pixel (for example, the top-left corner vertex pixel) in the first image block in the second video frame is (x1, y1), and the position of the corresponding pixel (for example, the top-left corner vertex pixel) in the second image block in the first video frame is (x2, y2), the horizontal displacement of the first image block and the second image block in the video frames is x3 = |x1-x2|, and the vertical displacement of the first image block and the second image block in the video frames is y3 = |y1-y2|, then the magnitude of the motion vector of the first image block in the second video frame is the L2 norm of x3 and y3, that is, the magnitude of the motion vector of the first image block in the second video frame is Alternatively, the magnitude of the motion vector of the first image block in the second video frame is the L1 norm of x3 and y3, that is, the magnitude of the motion vector of the first image block in the second video frame is |x3|+|y3|.

[0073] For example, the magnitude of the motion vector of each pixel in the same image block in the second video frame is the same.

[0074] Optionally, the image processing device determines a motion vector of each pixel in the second video frame according to the first video frame and the second video frame. The image processing device determines the first image region according to the motion vector of each pixel in the second video frame. The first image region includes at least one pixel. The motion vector of each pixel in the first image region in the second video frame is less than or equal to a first preset threshold, or the number of pixels in the first image region whose motion vector in the second video frame is less than or equal to the first preset threshold is greater than or equal to a fifth preset threshold. The embodiments of the present application are not limited to the method of determining the motion vector of the pixel, for example, the method of optical flow estimation. The embodiments of the present application are not limited to the specific value of the fifth preset threshold.

[0075] In some embodiments, the magnitude of the motion vector of the first pixel in the second video frame is used to indicate the displacement between the first pixel in the second video frame and the second pixel in the first video frame. The first pixel belongs to the second video frame, and the first pixel corresponds to the second pixel.

[0076] For example, the first pixel corresponds to the second pixel includes that the similarity between a third image block centered at the first pixel in the second video frame and a fourth image block centered at the second pixel in the first video frame is greater than or equal to a fourth preset threshold, or the fourth image block is the most similar image block to the third image block in the first video frame, or the similarity between the fourth image block and the third image block is greater than or equal to the similarity between the third image block and the image blocks in the first video frame except the fourth image block. The third image block and the fourth image block include the same number of pixels, and the third image block includes a plurality of pixels.

[0077] For example, the magnitude of the motion vector of the first pixel in the second video frame is determined according to the horizontal and vertical displacement of the first pixel and the second pixel in the video frame.

[0078] For example, assuming that the position of the first pixel in the second video frame is (x4, y4), and the position of the second pixel in the first video frame is (x5, y5), the horizontal displacement of the first pixel and the second pixel in the video frame is x6 = |x4-x5|, and the vertical displacement of the first pixel and the second pixel in the video frame is y6 = |y4-y5|, then the magnitude of the motion vector of the first pixel in the second video frame is the L2 norm of x6 and y6, that is, the magnitude of the motion vector of the first pixel in the second video frame is Or, the magnitude of the motion vector of the first pixel in the second video frame is the L1 norm of x6 and y6, that is, the magnitude of the motion vector of the first pixel in the second video frame is |x6|+|y6|.

[0079] In some embodiments, the first video frame is one frame before the second video frame in the first video frame set, or the first video frame is obtained by filtering one frame before the second video frame in the first video frame set. Alternatively, the first video frame is one frame after the second video frame in the first video frame set, or the first video frame is obtained by filtering one frame after the second video frame in the first video frame set.

[0080] Exemplarily, the first video frame and the second video frame are two adjacent frames in the first video frame set. For example, the first video frame is one frame before the second video frame in normal play order or in capture order, and the second video frame is one frame after the first video frame in normal play order or in capture order. Alternatively, the first video frame is one frame after the second video frame in normal play order or in capture order, and the second video frame is one frame before the first video frame in normal play order or in capture order. Alternatively, the first video frame is obtained by filtering one frame adjacent to the second video frame in the first video frame set. For example, the first video frame is obtained by filtering one frame before the second video frame in the first video frame set, or the first video frame is obtained by filtering one frame after the second video frame in the first video frame set.

[0081] Exemplarily, the first video frame and the second video frame are spaced by at least one frame in the first video frame set. For example, in normal play order, the first video frame is the 2nd frame before the second video frame, or the first video frame is the 2nd frame after the second video frame, i.e. the first video frame and the second video frame are spaced by 1 frame. Alternatively, the first video frame is obtained by filtering one frame spaced by at least one frame from the second video frame in the first video frame set. For example, the first video frame is obtained by filtering the 2nd frame before the second video frame in the first video frame set, or the first video frame is obtained by filtering the 2nd frame after the second video frame in the first video frame set.

[0082] In some embodiments, in the case that the first video frame is obtained by filtering one frame before the second video frame in the first video frame set, the image processing device filters the one frame before the second video frame in the first video frame set to obtain the first video frame before step 220. The specific implementation of obtaining the first video frame is similar to the way of obtaining the filtered second video frame in step 230, which is not described here.

[0083] In some embodiments, the first image region has a motion vector with a magnitude greater than 0 in the second video frame. In other words, the first image region is a micro-motion region.

[0084] Exemplarily, the magnitude of the motion vector of the first image region in the second video frame is greater than 0 includes that: the magnitude of the motion vector of each image block in the first image region in the second video frame is greater than 0; or, the number of image blocks whose magnitude of the motion vector in the second video frame is greater than 0 in the image blocks included in the first image region is greater than or equal to a sixth preset threshold; or, the magnitude of the motion vector of the image block other than the edge image block in the first image region in the second video frame is greater than 0, the edge image block including at least one image block close to the edge in the first image region; or, the magnitude of the motion vector of each pixel included in the first image region in the second video frame is greater than 0; or, the number of pixels whose magnitude of the motion vector in the second video frame is greater than 0 in the pixels included in the first image region is greater than or equal to a seventh preset threshold; or, the magnitude of the motion vector of the pixel other than the edge pixel in the first image region in the second video frame is greater than 0, the edge pixel including at least one pixel close to the edge in the first image region. The embodiments of the present application do not limit the specific values of the sixth preset threshold and the seventh preset threshold.

[0085] In some embodiments, the texture irregularity degree or the motion irregularity degree of the first image region in the second video frame is greater than or equal to a second preset threshold. The embodiments of the present application do not limit the specific value of the second preset threshold. For example, when the first image region includes images with irregular textures such as grass, leaves, flowers, animal fur, etc., the texture irregularity degree of the first image region is greater than or equal to the second preset threshold. That is, the first image region is an irregular texture region. The method for determining the texture irregularity degree and the motion irregularity degree is described in step 430.

[0086] Exemplarily, the motion irregularity degree of the first image region is used to represent the texture irregularity degree of the first image region. The greater the motion irregularity degree of the first image region is, the higher the texture irregularity degree of the first image region is. The smaller the motion irregularity degree of the first image region is, the lower the texture irregularity degree of the first image region is.

[0087] In some embodiments, the value of the first preset threshold can be the same or different according to different application scenarios. For example, for the video frame set after removing the micro-motions in the micro-motion image region, in the case that the tolerance degree of the visual effect when playing the video obtained according to the video frame set is higher, the first preset threshold can be set to a larger value; in the case that the tolerance degree of the visual effect when playing the video obtained according to the video frame set is lower, the first preset threshold can be set to a smaller value.

[0088] Optionally, the image processing device determines at least one second image region in the second video frame according to the first video frame and the second video frame. Each of the at least one second image region is similar to one image region in the first video frame. The image processing device further determines at least one first image region according to a magnitude of a motion vector of each of the at least one second image region in the second video frame. The magnitude of the motion vector of the second image region in the second video frame is determined according to the first video frame and the second video frame. Each of the at least one first image region has a magnitude of a motion vector in the second video frame less than or equal to a first preset threshold, i.e., each of the at least one first image region is a micro-motion region. The image processing device performs step 230 on each of the at least one first image region.

[0089] Exemplarily, the second image region being similar to one image region in the first video frame includes that a similarity between the second image region and the one image region in the first video frame is greater than or equal to a similarity between the second image region and other image regions in the first video frame except the one image region, or a similarity between the second image region and the one image region in the first video frame is greater than or equal to a fourth preset threshold.

[0090] In some embodiments, the image processing device determines at least one second image region in the second video frame and at least one fourth image region in the first video frame according to the first video frame and the second video frame. The at least one second image region corresponds to the at least one fourth image region one by one. Each of the at least one fourth image region is the most similar image region to the second image region corresponding to the fourth image region in the first video frame. Alternatively, a similarity between each second image region and the corresponding fourth image region is greater than or equal to a similarity between the second image region and other image regions in the first video frame except the corresponding fourth image region. Alternatively, the similarity between the second image region and the corresponding fourth image region is greater than or equal to a fourth preset threshold.

[0091] Optionally, after the image processing device determines the at least one second image region in the second video frame, the image processing device determines at least one third image region from the at least one second image region according to a magnitude of a motion vector of each of the at least one second image region in the second video frame. Each of the at least one third image region has a magnitude of a motion vector in the second video frame less than or equal to a first preset threshold. Alternatively, each of the at least one third image region has a magnitude of a motion vector in the second video frame less than or equal to the first preset threshold and a magnitude of a motion vector in the second video frame greater than 0. That is, each of the at least one third image region is a micro-motion region. The image processing device determines at least one first image region from the at least one third image region. Each of the at least one first image region has a texture irregularity or a motion irregularity greater than or equal to a second preset threshold. That is, each of the at least one first image region is a micro-motion region and an irregular texture region. The image processing device performs step 230 on each of the at least one first image region.

[0092] Optionally, the image processing device determines at least one fifth image region in the second video frame. Each of the at least one fifth image region has a texture irregularity or a motion irregularity greater than or equal to the second preset threshold. The image processing device determines the at least one first image region according to the at least one fifth image region and the at least one third image region. Each of the at least one first image region has a magnitude of a motion vector in the second video frame less than or equal to the first preset threshold and a texture irregularity or a motion irregularity greater than or equal to the second preset threshold. The image processing device performs step 230 on each of the at least one first image region.

[0093] In some embodiments, an edge pixel in the first image region has a magnitude of a motion vector in the second video frame of 0. That is, the edge pixel in the first image region is in a stationary state in the first video frame and the second video frame. The edge pixel in the first image region includes at least one pixel in the first image region close to an edge. Alternatively, an edge image block in the first image region has a magnitude of a motion vector in the second video frame of 0. That is, the edge image block in the first image region is in a stationary state in the first video frame and the second video frame. The edge image block in the first image region includes at least one image block in the first image region close to an edge.

[0094] Optionally, step 220 can be replaced by determining the first image region in the second video frame according to the plurality of video frames and the second video frame. At least one video frame in the plurality of video frames belongs to the first video frame set, and / or at least one video frame in the plurality of video frames is a video frame obtained by filtering a video frame in the first video frame set. The plurality of video frames includes the first video frame.

[0095] In some embodiments, the image processing device determines at least one fifth image block in the first video frame and at least one sixth image block in the second video frame according to the first video frame and the second video frame. Each sixth image block in the at least one sixth image block is similar to one fifth image block in the at least one fifth image block. The magnitude of the motion vector of each sixth image block in the at least one sixth image block in the second video frame is less than or equal to a first preset threshold. The image processing device determines the first image region in the second video frame according to the magnitude of the motion vector of the at least one fifth image block in the first video frame. Specifically, in the case that the magnitude of the motion vector of a fifth image block in the first video frame is less than or equal to the first preset threshold, the image processing device determines that the sixth image block similar to the fifth image block belongs to the first image region. In the case that the magnitude of the motion vector of the fifth image block in the first video frame is greater than the first preset threshold, the image processing device determines that the sixth image block similar to the fifth image block does not belong to the first image region. The magnitude of the motion vector of the fifth image block in the first video frame is determined according to the first video frame and a third video frame. The third video frame belongs to the first video frame set, or the third video frame is a frame obtained by filtering a frame before or after the first video frame in the first video frame set. The determination manner of the magnitude of the motion vector of the fifth image block in the first video frame is similar to the determination manner of the magnitude of the motion vector of the first image block in the second video frame, which will not be described herein.

[0096] In some embodiments, the image processing device determines, according to the first video frame and the second video frame, at least one third pixel in the first video frame and at least one fourth pixel in the second video frame. Each fourth pixel in the at least one fourth pixel corresponds to one third pixel in the at least one third pixel. The magnitude of the motion vector of each fourth pixel in the at least one fourth pixel in the second video frame is less than or equal to a first preset threshold. The image processing device determines a first image region in the second video frame according to the magnitude of the motion vector of the at least one third pixel in the first video frame. Specifically, in the case that the magnitude of the motion vector of the third pixel in the first video frame is less than or equal to the first preset threshold, the image processing device determines that the fourth pixel corresponding to the third pixel belongs to the first image region. In the case that the magnitude of the motion vector of the third pixel in the first video frame is greater than the first preset threshold, the image processing device determines that the fourth pixel corresponding to the third pixel does not belong to the first image region. The magnitude of the motion vector of the third pixel in the first video frame is determined according to the first video frame and a third video frame. The third video frame belongs to the first video frame set, or the third video frame is obtained by filtering one frame before or after the first video frame in the first video frame set. The determination of the magnitude of the motion vector of the third pixel in the first video frame is similar to the determination of the magnitude of the motion vector of the first pixel in the second video frame, which will not be described here.

[0097] 230, filtering the first image region in the second video frame to obtain a filtered second video frame.

[0098] The image processing device filters the first image region in the second video frame in the time domain to obtain a filtered second video frame. The magnitude of the motion vector of the first image region in the filtered second video frame is 0. The motion vector of the first image region in the filtered second video frame is determined according to the first video frame and the filtered second video frame. The first video frame is described in step 220. In other words, after filtering the first image region in the second video frame, the micro-motion in the first image region is removed, and the first image region is filtered as a static region.

[0099] Exemplarily, the determination of the motion vector of the first image region in the filtered second video frame is similar to the determination of the motion vector of the first image region in the second video frame, which will not be described here.

[0100] In some embodiments, the magnitude of the motion vector of the first image region in the filtered second video frame is 0 includes that the magnitude of the motion vector of each pixel in the first image region in the filtered second video frame is 0, or the magnitude of the motion vector of each image block in the first image region in the filtered second video frame is 0.

[0101] Optionally, the image processing device replaces the pixels in the first image region in the second video frame with the pixels in the first image region in the reference frame to obtain a filtered second video frame. The reference frame includes the first image region.

[0102] For example, assuming that the first image region in the second video frame and the reference frame are both 5*5 image blocks, the image processing device directly replaces the pixel values of the pixels in the 5*5 image block in the second video frame with the pixel values of the corresponding pixels in the 5*5 image block in the reference frame to obtain the filtered second video frame.

[0103] In some embodiments, the reference frame is any one of the following: a first video frame in the first video frame set, a first reconstructed frame corresponding to the first video frame set, a background frame corresponding to the first video frame set, or a first video frame. The first video frame is a frame obtained by filtering a frame before the second video frame in the first video frame set. The first reconstructed frame is a first key frame reconstructed in a decoding process of the first video frame set. The background frame is a frame including a background extracted from part or all of the video frames in the first video frame set. Illustratively, the background frame does not include foreground in the first video frame set.

[0104] When the reference frame is the first video frame in the first video frame set or the first reconstructed frame, if the first video frame is the first video frame in the first video frame set, the image processing device does not need to filter the first video frame. When the reference frame is the background frame, if the first video frame is the first video frame in the first video frame set, the image processing device performs step 230 on a sixth image region in the first video frame to obtain a filtered first video frame. The sixth image region has a motion vector in the first video frame with a magnitude less than or equal to a first preset threshold; or the sixth image region has a motion vector in the first video frame with a magnitude less than or equal to the first preset threshold, and the motion vector of the sixth image region in the first video frame has a magnitude greater than 0. The magnitude of the motion vector of the sixth image region in the first video frame is determined according to the first video frame and the background frame. Alternatively, when the reference frame is the background frame, if the first video frame is the first video frame in the first video frame set, the image processing device does not need to filter the first video frame.

[0105] Illustratively, the sixth image region has a texture irregularity or a motion irregularity greater than or equal to a second preset threshold.

[0106] Exemplarily, the magnitude of the motion vector of the sixth image region in the first video frame being less than or equal to a first preset threshold comprises: the magnitude of the motion vector of each image block included in the sixth image region in the first video frame being less than or equal to the first preset threshold; or, the number of image blocks in the image blocks included in the sixth image region, in which the magnitude of the motion vector in the first video frame is less than or equal to the first preset threshold, being greater than or equal to a third preset threshold; or, the magnitude of the motion vector of each pixel in the sixth image region in the first video frame being less than or equal to the first preset threshold; or, the number of pixels in the pixels included in the sixth image region, in which the magnitude of the motion vector in the first video frame is less than or equal to the first preset threshold, being greater than or equal to a fifth preset threshold. The specific values of the third preset threshold and the fifth preset threshold are not limited in the embodiments of the present application.

[0107] Exemplarily, the magnitude of the motion vector of the sixth image region in the first video frame being greater than 0 comprises: the magnitude of the motion vector of each image block in the sixth image region in the first video frame being greater than 0; or, the number of image blocks in the image blocks included in the sixth image region, in which the magnitude of the motion vector in the first video frame is greater than 0, being greater than or equal to a sixth preset threshold; or, the magnitude of the motion vector of the image block other than the edge image block in the sixth image region in the first video frame being greater than 0, the edge image block including at least one image block close to the edge in the sixth image region; or, the magnitude of the motion vector of each pixel included in the sixth image region in the first video frame being greater than 0; or, the number of pixels in the pixels included in the sixth image region, in which the magnitude of the motion vector in the first video frame is greater than 0, being greater than or equal to a seventh preset threshold, or, the magnitude of the motion vector of the pixel other than the edge pixel in the sixth image region in the first video frame being greater than 0, the edge pixel including at least one pixel close to the edge in the sixth image region. The specific values of the sixth preset threshold and the seventh preset threshold are not limited in the embodiments of the present application.

[0108] In some embodiments, in the case that the reference frame is a background frame, the image processing device determines the background frame according to all the video frames in the first video frame set, and thus performs step 230 on each of the video frames in the first video frame set according to the background frame. Alternatively, in the case that the reference frame is a background frame, the image processing device determines the background frame according to the first n video frames in the first video frame set, and thus performs step 230 on each of the (n+1)th video frame to the Nth video frame in the first video frame set according to the background frame. Wherein, the first video frame set includes N video frames, N is a positive integer greater than 1, and n = 1, 2,..., N-1.

[0109] In some embodiments, when the plurality of first image regions are included in the second video frame, the image processing device replaces the pixel values of the pixels in each of the first image regions in the second video frame with the pixel values of the pixels in the corresponding first image regions in the reference frame, to obtain a filtered second video frame.

[0110] 240. The image processing device obtains a second video frame set according to the filtered second video frame.

[0111] The image processing device replaces the second video frame in the first video frame set with the filtered second video frame, to obtain a second video frame set. That is, the filtered second video frame is included in the second video frame set. In other words, the image processing device filters at least one video frame in the first video frame set, and replaces the at least one video frame in the first video frame set with the corresponding filtered at least one video frame, to obtain the second video frame set.

[0112] When the first video frame is the first video frame in the first video frame set, and the first video frame does not need to be filtered, the second video frame set includes the first video frame and the filtered second video frame. When the first video frame is the first video frame in the first video frame set, and the image processing device filters the first video frame, the second video frame set includes the filtered first video frame and the filtered second video frame.

[0113] When the first video frame set is a subset of the third video frame set, the image processing device replaces the first video frame set in the third video frame set with the second video frame set, to obtain a fifth video frame set. The image processing device encodes the fifth video frame set, and outputs a first bitstream. The image processing device stores or transmits the first bitstream. Since the micro-motions in the micro-motion regions in at least one video frame in the second video frame set are removed, the amount of data of the residual after motion prediction can be reduced in the process of encoding the fifth video frame set, thereby reducing the amount of data after encoding, and further improving the compression rate of encoding.

[0114] When the third video frame set is the same as the first video frame set, the image processing device encodes the second video frame set, and outputs a second bitstream. The image processing device stores or transmits the second bitstream. Since the micro-motions in the micro-motion regions in at least one video frame in the second video frame set are removed, the amount of data of the residual after motion prediction can be reduced in the process of encoding the second video frame set, thereby reducing the amount of data after encoding, and further improving the compression rate of encoding.

[0115] In the method of FIG. 2, the image processing device filters the micro-motion image region in the second video frame to remove the micro-motion in the micro-motion image region, so that the image region becomes a static region after the filtering, thereby reducing the data amount after the video encoding and improving the compression rate of the video encoding.

[0116] Exemplarily, in a case where the first video frame set is a subset of the third video frame set, the image processing device processes the third video frame set by the method in FIG. 3, thereby outputting the first code stream.

[0117] FIG. 3 is a schematic flowchart of an image processing method according to an embodiment of the present application. The method in FIG. 3 is applied in an image processing device, for example, any device in FIG. 1. The method in FIG. 3 includes the following steps.

[0118] 310, obtaining a third video frame set.

[0119] The third video frame set includes a plurality of video frames. In a case where the third video frame set is obtained by an image sensor with fixed shooting parameters and without behaviors such as translation, rotation, zooming, etc., the scenes in each video frame in the third video frame set are the same, and the background in the third video frame set is in a static state. In a case where the third video frame set is obtained by an image sensor with non-fixed shooting parameters or with at least one behavior such as translation, rotation, zooming, etc., the scenes in at least two video frames in the third video frame set are different, and / or the background in the third video frame set is in a motion state.

[0120] Optionally, the image processing device obtains the third video frame set by itself. Alternatively, the image processing device receives the third video frame set from another device connected to the image processing device, for example, an image capturing device, a computing device, a storage device, etc.

[0121] 320, performing scene switching detection on the third video frame set to obtain a fourth video frame set.

[0122] The image processing device performs scene switching detection on the third video frame set to obtain at least one fourth video frame set. Each fourth video frame set in the at least one fourth video frame set includes at least one video frame, and the scenes in the video frames included in each fourth video frame set are the same. That is, each fourth video frame set is a subset of the third video frame set.

[0123] Optionally, the embodiments of the present application do not limit the specific type of the scene change detection algorithm. For example, the scene change detection algorithm includes an algorithm in an open source encoder x264 / x265, a gray histogram detection algorithm, a deep learning algorithm, etc. The open source encoder x264 / x265 determines whether there is a scene change between two video frames by calculating the dissimilarity of the two video frames. The gray histogram detection algorithm determines whether there is a scene change between two video frames by comparing the difference between the gray histograms of the two video frames. The deep learning algorithm determines whether there is a scene change between two video frames by extracting features in the two video frames respectively through a deep neural network and comparing the features.

[0124] In some embodiments, in a case where it is determined that there is no scene change between two adjacent video frames in the third video frame set, the image processing device divides the two adjacent video frames into the same fourth video frame set; in a case where it is determined that there is a scene change between two adjacent video frames in the third video frame set, the image processing device divides the two adjacent video frames into different fourth video frame sets, thereby obtaining the at least one fourth video frame set.

[0125] 330, analyzing the background in the fourth video frame set to obtain the first video frame set.

[0126] The image processing device analyzes the background in the fourth video frame set to obtain at least one first video frame set. Each first video frame set in the at least one first video frame set includes a plurality of video frames, and the background in each first video frame set is in a static state. That is, each first video frame set is a subset of the fourth video frame set.

[0127] Optionally, the image processing device performs motion analysis on the fourth video frame set to obtain the at least one first video frame set. The specific type of the motion analysis method is not limited in the embodiments of the present application. For example, the motion analysis method includes: a method based on optical flow estimation, a method based on block-level motion estimation, a method based on original video code stream, a deep learning algorithm, etc. The method based on optical flow estimation determines the motion direction and speed of a corresponding pixel in two video frames by determining the position change of the corresponding pixel in the two video frames, thereby extracting the motion vector of the corresponding pixel in the two video frames, and further determining whether the background in the two video frames is in a static state. The method based on block-level motion estimation determines the motion vector of a corresponding image block in two video frames by calculating the displacement between the image block, thereby determining whether the background in the two video frames is in a static state. The method based on original video code stream determines whether the background in the two video frames is in a static state by extracting the motion vector information included in the code stream after encoding of the original video. The deep learning algorithm extracts features in two video frames respectively through a deep neural network and compares them, thereby determining whether the background in the two video frames is in a static state.

[0128] In some embodiments, in a case where it is determined that the background in two adjacent video frames in the fourth video frame set is in a static state, the image processing device divides the two adjacent video frames into the same first video frame set; in a case where it is determined that the background in two adjacent video frames in the fourth video frame set is not in a static state (i.e., in a motion state), the image processing device divides the two adjacent video frames into different first video frame sets, thereby obtaining the at least one first video frame set.

[0129] Optionally, the image processing device performs background modeling on the fourth video frame set to obtain the at least one first video frame set. The specific type of the background modeling method is not limited in the embodiments of the present application. For example, the background modeling method includes a Gaussian mixture model (GMM) based background modeling method, a deep learning based background modeling method, an inter-frame difference method, etc. The GMM based background modeling method determines whether each pixel belongs to the background or the foreground according to the Gaussian distribution parameters of the pixel values of the pixels in the video frame, thereby extracting the background frame corresponding to the video frame set, based on the principle that the gray value of the background pixel point usually has a lower change, while the gray value of the foreground pixel point changes greatly. The deep learning based background modeling method extracts the background frame corresponding to the video frame set by using a deep neural network model to perform feature extraction and background modeling on the image. The deep neural network model includes a convolutional neural network (CNN), a generative adversarial network (GAN), etc. The inter-frame difference method determines the foreground region through the binarization image of the difference image of two video frames, and determines the image outside the foreground region as the background region, thereby extracting the background frame corresponding to the video frame set.

[0130] In some embodiments, the image processing device determines m1 video frames (e.g., 1st frame to m1th frame) in the fourth video frame set according to the sliding window, and performs background modeling on the m1 video frames. m1 is a positive integer, and m1≤M, where M is the number of video frames included in the fourth video frame set, and M is a positive integer. In a case where the m1 video frames can model a background frame, the image processing device determines that the m1 video frames belong to a same first video frame set. The image processing device determines, according to the background frame modeled by the m1 video frames and the remaining video frames in the fourth video frame set other than the m1 video frames, that a video frame in the remaining video frames in the fourth video frame set other than the m1 video frames having a same background as the background frame belongs to the same first video frame set as the m1 video frames, and a video frame in the remaining video frames in the fourth video frame set other than the m1 video frames having a different background as the background frame belongs to a different first video frame set as the m1 video frames. In a case where there is a video frame in the remaining video frames in the fourth video frame set other than the m1 video frames having a different background as the background frame, the image processing device determines another group of m1 video frames from the video frame having a different background as the background frame according to the sliding window, performs background modeling on the m1 video frames, and repeats the above steps, so as to determine at least one first video frame set. In a case where the m1 video frames cannot model a background frame, the image processing device determines a next group of m1 video frames (e.g., m2th frame to m2+m1-1th frame, where m2 is a positive integer, and m2+m1-1≤M) in the fourth video frame set according to the sliding window, and repeats the above steps, so as to determine at least one first video frame set.

[0131] In some embodiments, in a case where it is determined that M video frames in the fourth video frame set can model a background frame, the image processing device divides the M video frames into a same first video frame set. The fourth video frame set includes M video frames, and M is a positive integer. In a case where it is determined that the M video frames in the fourth video frame set cannot model a background frame, the image processing device determines m1 video frames in the fourth video frame set according to the sliding window, performs background modeling on the m1 video frames, and repeats the above steps, so as to determine at least one first video frame set. m1 is a positive integer, and m1≤M.

[0132] For example, m1 is greater than or equal to a minimum number of video frames required by a background modeling method.

[0133] 340, filtering the image region in at least one video frame in the first video frame set to obtain a second video frame set.

[0134] The image processing device filters the image region in each of the at least one video frame in each first video frame set, to obtain a second video frame set corresponding to each first video frame set. The specific filtering manner can refer to the description in FIG. 2 or FIG. 4, which is not described herein again.

[0135] 350, replacing the first video frame set in the third video frame set with the second video frame set to obtain a fifth video frame set.

[0136] The image processing device replaces at least one first video frame set in the third video frame set with a corresponding second video frame set, to obtain a fifth video frame set. In other words, after filtering the micro-motion image region in the steady video frame in the third video frame set, the image processing device combines the steady video frame with the non-steady video frame in the third video frame set according to the normal playing order or the collecting order, to obtain the fifth video frame set. The steady video frame refers to the video frame in the at least one first video frame set, and the non-steady video frame refers to the video frame in the third video frame set except the at least one first video frame set.

[0137] Optionally, the image processing device encodes the fifth video frame set and outputs a first bitstream. The image processing device stores or transmits the first bitstream. Since the micro-motion in the micro-motion region in the steady video frame in the fifth video frame set is removed, the data amount of the residual after motion prediction can be reduced in the process of encoding the fifth video frame set, so that the data amount after encoding is reduced, and the compression rate of encoding is improved.

[0138] In the method in FIG. 3, whether the video frame set received by the image processing device is obtained in a steady scene or not, the image processing device can extract the steady video frame therefrom and filter the same, so that the data amount of the video frame set after filtering and encoding is reduced, and the compression rate of encoding is improved. The steady scene refers to a shooting scene in which the shooting parameter of the image collecting device is fixed and there is no translation, rotation, zooming or the like. The non-steady scene refers to a shooting scene in which the shooting parameter of the image collecting device is not fixed or at least one of translation, rotation, zooming or the like exists.

[0139] FIG. 4 is a schematic flowchart of an image processing method according to an embodiment of the present application. The method in FIG. 4 is applied to an image processing device, for example, any device in FIG. 1. The method in FIG. 4 includes the following steps.

[0140] 410, determining at least one second image region in the second video frame according to the first video frame and the second video frame.

[0141] The image processing device determines, according to the first video frame and the second video frame, at least one image region in the first video frame and the second video frame that is most similar. That is, the image processing device determines at least one second image region in the second video frame and at least one fourth image region in the first video frame. Each second image region in the at least one second image region is most similar to one fourth image region in the first video frame. The at least one second image region and the at least one fourth image region refer to the description in step 220.

[0142] Optionally, the embodiments of the present application do not limit the specific type of the method for determining the similar image regions in the two video frames. For example, the method for determining the similar image regions in the two video frames includes a matching method based on grayscale values, a matching method based on features, a matching method based on deep learning, etc. The matching method based on grayscale values converts the two video frames into grayscale images, moves a sliding window on one of the video frames, and calculates the similarity between the image region in the sliding window and each image region of the same size in the other video frame, thereby determining at least one image region that matches in the two video frames. The similarity of the image regions can be measured by the pixel difference between the two image regions, the correlation coefficient, etc. The matching method based on features extracts key points (such as corner points, edges, etc.) and descriptors (vectors for describing the image region information around the key points) in the two video frames, and compares the similarity between the descriptors, thereby determining at least one image region that matches in the two video frames. The matching method based on deep learning extracts feature vectors of the image regions in the two video frames, and compares the similarity between the feature vectors of different image regions in the two video frames, thereby determining at least one image region that matches in the two video frames.

[0143] Optionally, step 410 can be replaced by: determining, according to the plurality of video frames and the second video frame, a first image region in the second video frame. At least one video frame in the plurality of video frames belongs to the first video frame set, and / or at least one video frame in the plurality of video frames is a video frame obtained by filtering the video frames in the first video frame set. The plurality of video frames includes the first video frame. The specific implementation manner refers to the description in FIG. 2.

[0144] 420, determining at least one third image region according to the magnitude of the motion vector of each second image region in the at least one second image region in the second video frame.

[0145] The image processing device determines at least one third image region according to a magnitude of a motion vector of each second image region in the second video frame. The magnitude of the motion vector of the second image region in the second video frame is determined according to the first video frame and the second video frame. The magnitude of the motion vector of each third image region in the at least one third image region in the second video frame is less than or equal to a first preset threshold. In other words, each third image region is a micro-motion image region. Embodiments of the present application do not limit the specific value of the first preset threshold.

[0146] In some embodiments, the magnitude of the motion vector of the third image region in the second video frame being less than or equal to the first preset threshold includes: the magnitude of the motion vector of each image block included in the third image region in the second video frame being less than or equal to the first preset threshold; or, the number of image blocks in the image blocks included in the third image region, the magnitude of the motion vector of which in the second video frame is less than or equal to the first preset threshold, being greater than or equal to a third preset threshold; or, the magnitude of the motion vector of each pixel in the third image region in the second video frame being less than or equal to the first preset threshold; or, the number of pixels in the pixels included in the third image region, the magnitude of the motion vector of which in the second video frame is less than or equal to the first preset threshold, being greater than or equal to a fifth preset threshold. Embodiments of the present application do not limit the specific value of the third preset threshold and the fifth preset threshold.

[0147] Optionally, embodiments of the present application do not limit the specific type of the method of determining the motion vector. For example, the method of determining the motion vector includes: a method based on optical flow estimation, a method based on block-level motion estimation, a method based on original video code stream, etc., which is described in detail in step 330.

[0148] For example, the calculation method of the magnitude of the motion vector of the second image region in the second video frame is described in step 220, which is not described herein again.

[0149] In some embodiments, the magnitude of the motion vector of each third image region in the second video frame is greater than 0.

[0150] Exemplarily, the magnitude of the motion vector of each image block in the third image region in the second video frame is greater than 0; or, the number of image blocks in the third image region whose motion vector in the second video frame is greater than 0 is greater than or equal to a sixth preset threshold; or, the magnitude of the motion vector of each image block in the third image region except for edge image blocks is greater than 0, the edge image blocks including at least one image block near the edge in the third image region; or, the magnitude of the motion vector of each pixel in the third image region in the second video frame is greater than 0; or, the number of pixels in the third image region whose motion vector in the second video frame is greater than 0 is greater than or equal to a seventh preset threshold, or, the magnitude of the motion vector of each pixel in the third image region except for edge pixels is greater than 0, the edge pixels including at least one pixel near the edge in the third image region. The specific values of the sixth preset threshold and the seventh preset threshold are not limited in the embodiments of the present application.

[0151] In some embodiments, the magnitude of the motion vector of the edge pixels in the third image region in the second video frame is 0. That is, the edge pixels in the third image region are in a static state in the first video frame and the second video frame. The edge pixels in the third image region include at least one pixel near the edge in the third image region. Alternatively, the magnitude of the motion vector of the edge image blocks in the third image region in the second video frame is 0. That is, the edge image blocks in the third image region are in a static state in the first video frame and the second video frame. The edge image blocks in the third image region include at least one image block near the edge in the third image region.

[0152] 430, determining the first image region from the at least one third image region.

[0153] The magnitude of the motion vector of each first image region in the at least one first image region in the second video frame is less than or equal to a first preset threshold, and the texture irregularity or the motion irregularity of each first image region is greater than or equal to a second preset threshold. In other words, the first image region is a micro-motion and texture-irregular image region. The specific value of the second preset threshold is not limited in the embodiments of the present application. The texture irregularity or the motion irregularity of the first image region is described in step 220.

[0154] Optionally, the embodiments of the present application do not limit the specific type of the method for determining the texture irregularity degree. For example, the method for determining the texture irregularity degree includes an error energy-based method, a deep learning-based method, etc. The error energy-based method determines an image region in a video frame, and another image region around the image region that is most similar to the image region, and calculates the error energy between the two image regions (i.e., the sum of absolute values of the difference between pixel values of corresponding pixels in the two image regions), thereby taking the error energy as the texture irregularity degree. The deep learning-based method directly outputs the texture irregularity degree of an image by using a deep neural network model that has been trained to extract and analyze features of the image.

[0155] Optionally, the motion irregularity degree of the first image region is determined according to the motion vector of each pixel in the first image region in the second video frame. For example, the motion irregularity degree of the first image region is determined according to the irregularity degree of the distribution of the direction of the motion vector of each pixel in the first image region in the second video frame. For example, the image processing device determines the distribution of the direction of the motion vector of the pixels in the first image region in P direction intervals according to the direction of the motion vector of each pixel in the first image region in the second video frame, where P is a positive integer greater than 1. The P direction intervals are obtained by equally dividing 360 degrees (°) into P parts. The image processing device determines the probability corresponding to each direction interval in the P direction intervals according to the distribution of the direction of the motion vector of the pixels in the first image region in the P direction intervals, and determines a first entropy value according to the probabilities corresponding to the P direction intervals. The first entropy value is used to represent the motion irregularity degree of the first image region. When the first entropy value is large, the motion irregularity degree of the first image region is large; when the first entropy value is small, the motion irregularity degree of the first image region is small.

[0156] Optionally, step 430 can be replaced by the following steps: determining at least one fifth image region in the second video frame; and determining the at least one first image region according to the at least one fifth image region and the at least one third image region. Wherein the texture irregularity degree or the motion irregularity degree of each fifth image region in the at least one fifth image region is greater than or equal to a second preset threshold.

[0157] In some embodiments, the determination of the at least one first image region according to the at least one fifth image region and the at least one third image region includes: determining the intersection of the at least one fifth image region and the at least one third image region, and the at least one first image region is included in the intersection.

[0158] 440, replacing the pixels in the first image region in the second video frame with the pixels in the first image region in the reference frame to obtain a filtered second video frame.

[0159] The image processing device replaces the pixel values of the pixels in the first image region in the second video frame with the pixel values of the corresponding pixels in the first image region in the reference frame to obtain a filtered second video frame. The magnitude of the motion vector of the first image region in the filtered second video frame is 0. The magnitude of the motion vector of the first image region in the filtered second video frame is determined according to the first video frame and the filtered second video frame. The first video frame is described in step 220.

[0160] In some embodiments, the magnitude of the motion vector of the first image region in the filtered second video frame is 0 includes that the magnitude of the motion vector of each pixel in the first image region in the filtered second video frame is 0, or the magnitude of the motion vector of each image block in the first image region in the filtered second video frame is 0.

[0161] In some embodiments, the reference frame is any one of the following: the first video frame in the first video frame set, the first reconstructed frame corresponding to the first video frame set, the background frame corresponding to the first video frame set, or the first video frame. The reference frame is described in step 230.

[0162] When the reference frame is the first video frame in the first video frame set or the first reconstructed frame, if the first video frame is the first video frame in the first video frame set, the image processing device does not need to filter the first video frame. When the reference frame is the background frame, if the first video frame is the first video frame in the first video frame set, the image processing device can filter the first video frame, or can not filter the first video frame.

[0163] When the second video frame includes a plurality of first image regions, the image processing device replaces the pixels in each first image region in the second video frame with the pixels in the corresponding first image region in the reference frame to obtain a filtered second video frame.

[0164] Optionally, the image processing device can continue to perform the method in FIG. 4 on the next video frame in the first video frame set after performing the method in FIG. 4 on the second video frame. Alternatively, the image processing device can perform steps 440 on all video frames in the first video frame set after performing steps 410-430 on all video frames in the first video frame set.

[0165] Optionally, after step 440, the image processing device performs smoothing processing on the filtered video frame to avoid discontinuous visual effects of the edge of the replaced image region and the surrounding region due to pixel replacement.

[0166] In the method of FIG. 4, the image processing device filters the micro-motion and texture-irregular image region in the second video frame to remove the micro-motion in the micro-motion and texture-irregular image region, so that the micro-motion and texture-irregular image region becomes a static region after filtering, thereby reducing the data amount after video encoding and improving the compression rate of video encoding.

[0167] FIG. 5 is a schematic diagram of a micro-motion and texture-irregular image region obtained according to the method in the embodiments of the present application. FIG. 5(a) is a video frame, and FIG. 5(b) is an image obtained by identifying the micro-motion and texture-irregular image region in FIG. 5(a). In FIG. 5(b), the white region is a mask of the micro-motion and texture-irregular image region in FIG. 5(a).

[0168] For example, when the method in the embodiments of the present application is used to process and encode five different videos, the code rate saving is shown in Table 1.

[0169] Table 1

[0170] As shown in Table 1, in Video 1, the average proportion of the micro-motion and texture-irregular image region in the plurality of video frames included in Video 1 is 7.3%, and the code stream obtained by encoding the processed Video 1 obtained by processing the plurality of video frames included in Video 1 according to the method in the embodiments of the present application is 1.41% less than the code stream obtained by encoding the original Video 1. As can be seen from Table 1, by removing the micro-motion in the micro-motion and texture-irregular image region in the video, the code rate can be saved by 21.2% on average without affecting the visual effect. As can be seen from Table 1, the more the area of the micro-motion and texture-irregular image region in the video frame, the more the code rate saved.

[0171] FIG. 6 is a schematic structural diagram of an image processing device provided in the embodiments of the present application. The image processing device 600 in FIG. 6 includes an acquisition module 610 and a processing module 620. The image processing device 600 in FIG. 6 can be used to execute the method in FIG. 2 to FIG. 4. The image processing device 600 in FIG. 6 can be applied to any device in FIG. 1.

[0172] When the image processing apparatus 600 is configured to perform the method in FIG. 2, the obtaining module 610 is configured to obtain the first video frame set. The obtaining module 610 is configured to perform step 210 in FIG. 2. The processing module 620 is configured to: determine the first image region in the second video frame according to the first video frame and the second video frame; filter the first image region in the second video frame to obtain a filtered second video frame; and obtain the second video frame set according to the filtered second video frame. The processing module 620 is configured to perform steps 220-240 in FIG. 2.

[0173] When the image processing apparatus 600 is configured to perform the method in FIG. 3, the obtaining module 610 is configured to obtain the third video frame set. The obtaining module 610 is configured to perform step 310 in FIG. 3. The processing module 620 is configured to: perform scene cut detection on the third video frame set to obtain a fourth video frame set; analyze the background in the fourth video frame set to obtain the first video frame set; filter the image region in at least one video frame in the first video frame set to obtain a second video frame set; and replace the first video frame set in the third video frame set with the second video frame set to obtain a fifth video frame set. The processing module 620 is configured to perform steps 320-350 in FIG. 3.

[0174] When the image processing apparatus 600 is configured to perform the method in FIG. 4, the processing module 620 is configured to: determine at least one second image region in the second video frame according to the first video frame and the second video frame; determine at least one third image region according to the magnitude of the motion vector of each second image region in the at least one second image region in the second video frame; determine the first image region from the at least one third image region; and replace the pixels in the first image region in the second video frame with the pixels in the first image region in the reference frame to obtain a filtered second video frame. The processing module 620 is configured to perform steps 410-440 in FIG. 4.

[0175] The obtaining module 610 and the processing module 620 can be implemented by software or by hardware. For example, the implementation of the processing module 620 is described below. The implementation of the obtaining module 610 can be similar to the implementation of the processing module 620.

[0176] As an example of a software functional unit, the processing module 620 can include code running on a compute instance. The compute instance can include at least one of a physical host (computing device), a virtual machine, a container. Further, the compute instance can be one or more. For example, the processing module 620 can include code running on multiple hosts / virtual machines / containers. It is noted that the multiple hosts / virtual machines / containers running the code can be distributed in the same region, or in different regions. Further, the multiple hosts / virtual machines / containers running the code can be distributed in the same availability zone (AZ), or in different AZs, each of which includes one data center or multiple data centers in close geographical proximity. Typically, a region can include multiple AZs.

[0177] Similarly, the multiple hosts / virtual machines / containers running the code can be distributed in the same VPC, or in multiple VPCs. Typically, a VPC is set up within a region, and communication between two VPCs in the same region, or between VPCs in different regions, requires a communication gateway in each VPC to enable interconnection between VPCs.

[0178] As an example of a hardware functional unit, the processing module 620 can include at least one computing device, such as a server, etc. Alternatively, the processing module 620 can also be a device implemented using an application-specific integrated circuit (ASIC), or a programmable logic device (PLD), etc. The PLD can be implemented as a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0179] The multiple computing devices included in the processing module 620 can be distributed in the same region or in different regions. The multiple computing devices included in the processing module 620 can be distributed in the same AZ or in different AZs. Similarly, the multiple computing devices included in the processing module 620 can be distributed in the same VPC or in multiple VPCs. The multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0180] Therefore, the modules of the examples described in the embodiments of the present application can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0181] It should be noted that: the device provided by the above-mentioned embodiments is in executing the above-mentioned method, only above-mentioned each functional module is divided and carries out example explanation, actually applies, can according to need to complete by different functional module to distribute above-mentioned function, namely divides the internal structure of device into different functional module, to complete above description all or partial function. For example, the acquisition module 610 can be used to execute any step in the above method, and the processing module 620 can be used to execute any step in the above method. The steps responsible for the acquisition module 610 and the processing module 620 can be specified as needed, and the acquisition module 610 and the processing module 620 respectively implement different steps in the above method to realize the entire function of the above device.

[0182] In addition, the device and method embodiments provided by the above-mentioned embodiments belong to the same concept, and the specific implementation process is described in the method embodiments above, which will not be repeated here.

[0183] The method provided by the embodiments of the present application can be executed by a computing device, which can also be referred to as a computer system. The computer system includes a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as a processing unit, a memory, and a memory control unit, and the functions and structures of the hardware are described in detail later. The operating system is any one or more computer operating systems that implement business processing through processes, such as a Linux operating system, a Unix operating system, an Android operating system, an iOS operating system, or a Windows operating system. The application layer includes application programs such as a browser, an address book, word processing software, and instant messaging software. Optionally, the computer system is a handheld device such as a smartphone or a terminal device such as a personal computer, and the present application is not particularly limited as long as the method provided by the embodiments of the present application can be executed. The execution subject of the method provided by the embodiments of the present application can be a computing device, or a functional module in the computing device that can call and execute a program.

[0184] FIG. 7 is a schematic structural block diagram of a computing device 700 provided by an embodiment of the present application. The computing device 700 can be a server or a computer or other device with computing capability. The computing device 700 shown in FIG. 7 includes at least one processor 710 and a memory 720.

[0185] It should be understood that the present application does not limit the number of processors and memories in the computing device 700.

[0186] The processor 710 executes instructions in the memory 720, so that the computing device 700 implements the method provided by the present application. Alternatively, the processor 710 executes instructions in the memory 720, so that the computing device 700 implements the functional modules provided by the present application, thereby implementing the method provided by the present application.

[0187] Optionally, the computing device 700 further includes a communication interface 730. The communication interface 730 uses a transceiving module such as, but not limited to, a network interface card and a transceiver to implement communication between the computing device 700 and other devices or communication networks.

[0188] Optionally, the computing device 700 further includes a system bus 740, wherein the processor 710, the memory 720 and the communication interface 730 are connected with the system bus 740 respectively. The processor 710 can access the memory 720 through the system bus 740, for example, the processor 710 can read and write data in the memory 720 or execute code in the memory 720 through the system bus 740. The system bus 740 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 740 is divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, only one thick line is shown in FIG. 7, but it does not mean that there is only one bus or only one type of bus.

[0189] In one possible implementation, the function of the processor 710 is mainly to interpret the instructions (or code) of the computer program and process the data in the computer software. The instructions of the computer program and the data in the computer software can be saved in the memory 720 or the cache of the processor 710.

[0190] Optionally, the processor 710 can be an integrated circuit chip having a processing capability of signals. As an example but not limitation, the processor 710 is a general purpose processor, a digital signal processor (DSP), an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general purpose processor is a microprocessor, etc. For example, the processor 710 is a central processing unit (CPU).

[0191] The memory 720 can provide a running space for a process in the computing device 700, for example, the memory 720 saves the computer program (specifically, the code of the program) used to generate the process. After the computer program is run by the processor to generate the process, the processor allocates a corresponding storage space for the process in the memory 720. Further, the above-mentioned storage space further includes a text segment, an initialized data segment, a bit initialized data segment, a stack segment, a heap segment, etc. The memory 720 saves the data generated during the running of the process in the above-mentioned storage space of the process, for example, intermediate data, process data, etc.

[0192] Optionally, memory, also called a main memory, is used to store data and instructions for processing by processor 710. A bus is used for moving data to and from the memory to processor 710 when such movement is desired.

[0193] By way of example, and not limitation, memory 720 is volatile memory or nonvolatile memory, or can include both volatile and nonvolatile memory. By way of example, and not limitation, nonvolatile memory, other than RAM, can be used for storage of persistent information and instructions, such as a read-only memory (ROM) or a programmable ROM (PROM). Nonvolatile memory can also be used for storage of persistent information and / or instructions that are authoritative to the processing system 700, and nonvolatile memory can be used for storage of instructions and data that are not authoritative, and that can only be used when the processing system 700 is not in a full operating state or when the processing system 700 is in an operational state. By way of example, and not limitation, nonvolatile memory can include a ROM, a programmable ROM (PROM), an erasable programmable ROM (EPROM), an electrically erasable programmable ROM (EEPROM), a flash memory, or a solid state memory such as a class of memory known as "Omnipath," for example. Volatile memory is random access memory (RAM) used for storage of instructions and / or data that a processing system 700 is actively using at a given instance of time. By way of example, and not limitation, volatile memory can be used for storage of instructions and data that the processing system 700 is actively using at a given instance of time. Exemplary volatile memories include a dynamic RAM (DRAM), such as a synchronous DRAM (SDRAM), a double data rate SDRAM (DDR SDRAM), a synchronous link DRAM (SLDRAM), a RAMBUS DRAM (DRAM), etc. Note that the system and method described herein is not limited to a particular memory system and / or memory architecture. For example, although not shown, a bus bridge can be used as the bus in some implementations, and processor 710 can be implemented using a set of processors rather than a single processor.

[0194] The structure of the computing device 700 listed above is only an example, and the application is not limited thereto. The computing device 700 of the embodiments of the application includes various hardware in the prior art computer system, for example, the computing device 700 also includes other memories in addition to the memory 720, such as a disk memory, etc. Those skilled in the art should understand that the computing device 700 can also include other devices necessary for normal operation. Meanwhile, according to specific needs, those skilled in the art should understand that the above computing device 700 can also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the above computing device 700 can also only include devices necessary for the embodiments of the application, and does not necessarily include all the devices shown in FIG. 7.

[0195] The embodiments of the application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server. In some embodiments, the computing device can also be a desktop computer, a notebook computer, or a terminal device such as a smart phone.

[0196] As shown in FIG. 8, the computing device cluster includes at least one computing device 700. The memory 720 in one or more computing devices 700 in the computing device cluster can store the same instructions for executing the above method.

[0197] In some possible implementations, the memory 720 in one or more computing devices 700 in the computing device cluster can also respectively store partial instructions for executing the above method. In other words, the combination of one or more computing devices 700 can collectively execute the instructions of the above method.

[0198] It should be noted that the memories 720 in different computing devices 700 in the computing device cluster can store different instructions, respectively for executing partial functions of the above apparatus. That is, the instructions stored in the memories 720 in different computing devices 700 can implement the functions of one or more modules in the above apparatus.

[0199] In some possible implementations, one or more computing devices in the computing device cluster can be connected through a network. The network can be a wide area network or a local area network, etc. FIG. 9 shows one possible implementation. As shown in FIG. 9, two computing devices 700A and 800B are connected through a network. Specifically, the communication interface in each computing device is connected to the network.

[0200] It should be understood that the functions of the computing device 700A shown in FIG. 9 can also be completed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be completed by multiple computing devices 700.

[0201] In the embodiments of the present application, a computer program product containing instructions is also provided. The computer program product can be software or a program product containing instructions, which can run on a computing device cluster or be stored in any available medium. When it is run by the computing device cluster, it causes the computing device cluster to perform the method provided above or causes the computing device cluster to realize the functions of the apparatus provided above.

[0202] In the embodiments of the present application, a computer readable storage medium is also provided. The computer readable storage medium can be any available medium that the computing device can store or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium contains instructions, which, when executed by the computing device cluster, cause the computing device cluster to perform the method provided above.

[0203] Those skilled in the art can clearly understand that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0204] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, apparatus and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be described here.

[0205] In several embodiments provided in the present application, it should be understood that the disclosed system, apparatus and method can be implemented in other ways. For example, the apparatus embodiments described above are only schematic. For example, the division of the units is only a logical function division, and actual implementation can have another division manner. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0206] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0207] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0208] The functions, if realized in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0209] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining a first video frame set, the first video frame set comprising a plurality of video frames, and a background of the first video frame set being static; determining a first image region in a second video frame according to a first video frame and the second video frame, the second video frame belonging to the first video frame set, a magnitude of a motion vector of the first image region in the second video frame being less than or equal to a first preset threshold, the first video frame being one frame before the second video frame in the first video frame set, or the first video frame being a frame obtained by filtering one frame before the second video frame in the first video frame set; filtering the first image region in the second video frame to obtain a filtered second video frame, a magnitude of a motion vector of the first image region in the filtered second video frame being 0; obtaining a second video frame set according to the filtered second video frame; wherein the magnitude of the motion vector of the first image region in the second video frame is determined according to the first video frame and the second video frame, and the magnitude of the motion vector of the first image region in the filtered second video frame is determined according to the first video frame and the filtered second video frame.

2. The method of claim 1, wherein, The magnitude of the motion vector of the first image region in the second video frame is greater than 0.

3. The method according to claim 1 or 2, characterized in that, A texture irregularity degree or a motion irregularity degree of the first image region is greater than or equal to a second preset threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The first video frame and the second video frame are adjacent video frames in the first video frame set, or the first video frame is a frame obtained by filtering one frame adjacent to the second video frame in the first video frame set.

5. The method according to any one of claims 1 to 4, characterized in that, The determining of the first image region in the second video frame according to the first video frame and the second video frame comprises: determining at least one second image region in the second video frame according to the first video frame and the second video frame, each second image region in the at least one second image region being similar to one image region in the first video frame; determining the first image region according to a magnitude of a motion vector of each second image region in the at least one second image region in the second video frame, the magnitude of the motion vector of each second image region in the second video frame being determined according to the first video frame and the second video frame.

6. The method of claim 5, wherein, The determining of the first image region according to the magnitude of the motion vector of each second image region in the at least one second image region in the second video frame comprises: determining at least one third image region according to the magnitude of the motion vector of each second image region in the at least one second image region in the second video frame, each third image region in the at least one third image region having a magnitude of a motion vector in the second video frame less than or equal to the first preset threshold; determining the first image region from the at least one third image region, a texture irregularity degree or a motion irregularity degree of the first image region being greater than or equal to a second preset threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The filtering on the first image region in the second video frame obtains a filtered second video frame. The filtering on the first image region in the second video frame obtains a filtered second video frame.

8. The method of claim 7, wherein, The reference frame is any one of the following: a first video frame in the first video frame set, a first reconstructed frame corresponding to the first video frame set, a background frame corresponding to the first video frame set, or the first video frame, wherein the first video frame is a frame obtained by filtering a frame before the second video frame in the first video frame set.

9. The method of claim 8, wherein, In a case where the reference frame is the first video frame in the first video frame set or the first reconstructed frame, and the first video frame is the first video frame in the first video frame set, the second video frame set includes the first video frame and the filtered second video frame.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: obtaining a third video frame set, the third video frame set including a plurality of video frames, scenes in at least two video frames in the third video frame set being different, and / or a background in the third video frame set being in a motion state; performing scene switching detection on the third video frame set to obtain a fourth video frame set, the fourth video frame set being a subset of the third video frame set, the fourth video frame set including a plurality of video frames, scenes in each video frame in the fourth video frame set being the same; performing analysis on a background in the fourth video frame set to obtain the first video frame set, the first video frame set being a subset of the fourth video frame set.

11. The method of claim 10, wherein, The method further includes: replacing the first video frame set in the third video frame set with the second video frame set to obtain a fifth video frame set; encoding the fifth video frame set to output a first code stream.

12. An image processing apparatus characterized by comprising: The method further includes: an obtaining module configured to obtain a first video frame set, the first video frame set including a plurality of video frames, a background in the first video frame set being in a static state; a processing module configured to determine a first image region in a second video frame according to the first video frame and the second video frame, the second video frame belonging to the first video frame set, a magnitude of a motion vector of the first image region in the second video frame being less than or equal to a first preset threshold, the first video frame being a frame before the second video frame in the first video frame set, or the first video frame being a frame obtained by filtering a frame before the second video frame in the first video frame set; the processing module is further configured to filter the first image region in the second video frame to obtain a filtered second video frame, a magnitude of a motion vector of the first image region in the filtered second video frame being 0; the processing module is further configured to obtain a second video frame set according to the filtered second video frame; and the processing module is further configured to filter the first image region in the second video frame to obtain a filtered second video frame, a magnitude of a motion vector of the first image region in the filtered second video frame being 0. The motion vector of the first image region in the second video frame is determined according to the first video frame and the second video frame, and the motion vector of the first image region in the filtered second video frame is determined according to the first video frame and the filtered second video frame.

13. The apparatus of claim 12, wherein, The amplitude of the motion vector of the first image region in the second video frame is greater than 0.

14. The apparatus of claim 12 or 13, wherein, The texture irregularity or the motion irregularity of the first image region is greater than or equal to a second preset threshold.

15. The apparatus of any one of claims 12-14, wherein, The first video frame and the second video frame are adjacent video frames in the first video frame set, or the first video frame is a frame obtained by filtering a frame adjacent to the second video frame in the first video frame set.

16. The apparatus of any one of claims 12-15, wherein, The processing module is specifically configured to: determine at least one second image region in the second video frame according to the first video frame and the second video frame, each second image region in the at least one second image region being similar to an image region in the first video frame; determine the first image region according to the amplitude of the motion vector of each second image region in the at least one second image region in the second video frame, the amplitude of the motion vector of each second image region in the second video frame being determined according to the first video frame and the second video frame.

17. The apparatus of claim 16, wherein, The processing module is specifically configured to: determine at least one third image region according to the amplitude of the motion vector of each second image region in the at least one second image region in the second video frame, each third image region in the at least one third image region having an amplitude of the motion vector in the second video frame less than or equal to the first preset threshold; determine the first image region from the at least one third image region, the texture irregularity or the motion irregularity of the first image region being greater than or equal to a second preset threshold.

18. The apparatus of any one of claims 12-17, wherein, The processing module is specifically configured to: replace pixels in the first image region in the second video frame with pixels in the first image region in a reference frame to obtain the filtered second video frame.

19. The apparatus of claim 18, wherein, The reference frame is any one of the following: a first video frame in the first video frame set, a first reconstructed frame corresponding to the first video frame set, a background frame corresponding to the first video frame set, or a first video frame in the first video frame set, wherein the first video frame is a frame obtained by filtering a frame before the second video frame in the first video frame set.

20. The apparatus of claim 19, wherein, In a case where the reference frame is the first video frame in the first video frame set or the first reconstructed frame, and the first video frame is the first video frame in the first video frame set, the second video frame set includes the first video frame and the filtered second video frame.

21. The apparatus of any one of claims 12-20, wherein, The acquisition module is specifically configured to: obtain a third video frame set, the third video frame set including a plurality of video frames, scenes in at least two video frames in the third video frame set being different, and / or a background in the third video frame set being in a motion state; perform scene cut detection on the third video frame set to obtain a fourth video frame set, the fourth video frame set being a subset of the third video frame set, the fourth video frame set including a plurality of video frames, a scene in each video frame in the fourth video frame set being the same; perform analysis on a background in the fourth video frame set to obtain the first video frame set, the first video frame set being a subset of the fourth video frame set.

22. The apparatus of claim 21, wherein, The processing module is further configured to: replace the first video frame set in the third video frame set with the second video frame set to obtain a fifth video frame set; encode the fifth video frame set to output a first code stream.

23. A cluster of computing devices, characterized in that, include at least one computing device, each computing device including a processor and a memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to perform the method of any one of claims 1-11.

24. A chip system, characterized by include: a logic circuit, the logic circuit being configured to be coupled with an input / output interface through which data is transmitted to perform the method of any one of claims 1-11.

25. A computer program product comprising instructions, wherein: The instructions, when executed by the computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-11.

26. A computer-readable storage medium, characterized in that, include computer program instructions that, when executed by a computing device cluster, cause the computing device cluster to perform the method of any one of claims 1-11.

Images