Image processing method and related equipment

Abstract

The invention provides an image processing method and related equipment. The method comprises the following steps: obtaining a first video frame set, wherein the background in the first video frame set is in a static state; according to the first video frame and the second video frame, a first image area in the second video frame is determined, and the amplitude of a motion vector of the first image area in the second video frame is smaller than or equal to a first preset threshold value; the first image area in the second video frame is filtered, the filtered second video frame is obtained, and the amplitude of the motion vector of the first image area in the filtered second video frame is 0; and obtaining a second video frame set according to the filtered second video frame. According to the method, the micro-motion in the micro-motion image region (namely the first image region) in the second video frame is removed, so that the data volume after video coding is reduced, and the compression ratio of video coding is improved.

Description

[0001] This application claims priority to Russian Patent Application No. RU2024123679, filed with the Russian Federal Intellectual Property Office on August 16, 2024, entitled "An Image Processing Method and Related Apparatus", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of image processing, and more specifically, to an image processing method, an image processing apparatus, a computing device cluster, a chip system, a computer program product, and a computer-readable storage medium. Background Technology

[0003] With improvements in image and video acquisition equipment, current image and video data volumes are substantial, requiring significant resources for storage and transmission. To conserve resources during storage and transmission, video encoding can be used to compress the data. Videos can be categorized into steady-state and non-steady-state videos. Steady-state videos are those where the scene remains static and the background is static. Non-steady-state videos are those where the scene changes and / or the background is in motion. Current video encoding methods typically extract the background and foreground from the video and calculate and store the motion vectors and residuals of the foreground portion of the video frames to reduce redundant data during storage and transmission, thereby compressing the data volume. However, current video encoding methods are not ideal for micro-motion regions. These micro-motion regions are areas in the video where slight jitter occurs, such as areas including grass, leaves, flowers, and animal fur. Because these micro-motion regions have a large amount of residuals after motion prediction, the encoded data volume remains large, resulting in continued resource consumption during storage and transmission.

[0004] Therefore, how to process the micro-motion regions in video frames to improve the compression rate of video encoding has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method, an image processing apparatus, a computing device cluster, a chip system, a computer program product, and a computer-readable storage medium, which can process micro-motion regions in video frames, reduce the amount of data after video encoding, and thereby improve the compression rate of video encoding.

[0006] In a first aspect, an image processing method is provided. The method includes: obtaining a first video frame set, the first video frame set including multiple video frames, wherein the background of the first video frame set is static; determining a first image region in a second video frame based on the first video frame and a second video frame, wherein the second video frame belongs to the first video frame set, the amplitude of the motion vector of the first image region in the second video frame is less than or equal to the first preset threshold, and the first video frame is a frame preceding the second video frame in the first video frame set, or the first video frame is a frame obtained by filtering the frame preceding the second video frame in the first video frame set; filtering the first image region in the second video frame to obtain a filtered second video frame, wherein the amplitude of the motion vector of the first image region in the filtered second video frame is 0; and obtaining a second video frame set based on the filtered second video frame.

[0007] The magnitude of the motion vector of the first image region in the second video frame is determined based on the first video frame and the second video frame, and the magnitude of the motion vector of the first image region in the filtered second video frame is determined based on the first video frame and the filtered second video frame.

[0008] In this embodiment of the application, by filtering the micro-motion image region (i.e. the first image region) in the second video frame, the micro-motion in the micro-motion image region is removed, so that the image region becomes a stationary region after filtering, thereby reducing the amount of data after video encoding and thus improving the compression rate of video encoding.

[0009] In conjunction with the first aspect, in some implementations, the magnitude of the motion vector of the first image region in the second video frame is greater than 0.

[0010] In this embodiment, the first image region is a micro-motion region before filtering and a stationary region after filtering, thereby reducing the amount of data after video encoding and improving the compression rate of video encoding.

[0011] In conjunction with the first aspect, in some implementations, the degree of texture irregularity or motion irregularity of the first image region is greater than or equal to a second preset threshold.

[0012] In this embodiment, the first image region is determined to be a region with irregular texture by the degree of texture irregularity or motion irregularity. That is, the first image region is not only a region with micro-motion but also a region with irregular texture. By removing the micro-motion in the first image region, the amount of data in the first image region after video encoding is reduced, thereby improving the compression rate of video encoding.

[0013] For example, the degree of motion irregularity of the first image region is determined based on the direction of the motion vector of each pixel in the first image region in the second video frame. The degree of motion irregularity of the first image region is used to characterize the degree of texture irregularity of the first image region.

[0014] In conjunction with the first aspect, in some implementations, the first video frame and the second video frame are adjacent video frames in the first video frame set, or the first video frame is a frame obtained by filtering a frame in the first video frame set that is adjacent to the second video frame.

[0015] In this embodiment of the application, by processing two adjacent video frames in the video frame set, the micro-motion in each micro-motion region of the processed video frame set is removed, thereby reducing the amount of data after video encoding and thus improving the compression rate of video encoding.

[0016] In conjunction with the first aspect, in some implementations, at least one second image region in the second video frame is determined based on the first video frame and the second video frame, and each second image region in the at least one second image region is similar to an image region in the first video frame; a first image region is determined based on the magnitude of the motion vector of each second image region in the second video frame, and the magnitude of the motion vector of each second image region in the second video frame is determined based on the first video frame and the second video frame.

[0017] In this embodiment of the application, by determining similar image regions in the first video frame and the second video frame, the magnitude of the motion vector of the image region in the second video frame is determined, thereby determining the micro-motion region in the second video frame.

[0018] In conjunction with the first aspect, in some implementations, at least one third image region is determined based on the magnitude of the motion vector of each of the at least one second image regions in the second video frame, wherein the magnitude of the motion vector of each of the at least one third image regions in the second video frame is less than or equal to a first preset threshold; a first image region is determined from the at least one third image region, wherein the degree of texture irregularity or motion irregularity of the first image region is greater than or equal to a second preset threshold.

[0019] In this embodiment, after determining similar image regions in the first and second video frames, the image regions with micro-motion and irregular textures in the second video frame are determined based on the magnitude of the motion vector of the image region in the second video frame and the degree of texture irregularity of the image region. The image regions with micro-motion and irregular textures are then filtered, thereby reducing the amount of data after video encoding and improving the compression rate of video encoding.

[0020] In conjunction with the first aspect, in some implementations, pixels in the first image region of the second video frame are replaced with pixels in the first image region of the reference frame to obtain the filtered second video frame.

[0021] In conjunction with the first aspect, in some implementations, the reference frame is any one of the following: the first video frame in the first video frame set, the first reconstructed frame corresponding to the first video frame set, the background frame corresponding to the first video frame set, or the first video frame, wherein the first video frame is a frame obtained by filtering a frame preceding the second video frame in the first video frame set.

[0022] In this embodiment, by replacing the pixels in the first image region of the second video frame with the pixels in the first image region of the reference frame, the pixel values ​​of the first image region in the second video frame and the first image region in the reference frame correspond to the same value, thereby achieving the effect of removing micro-motion in the first image region.

[0023] In conjunction with the first aspect, in some implementations, when the reference frame is the first video frame or the first reconstructed frame in the first video frame set, and the first video frame is the first video frame in the first video frame set, the second video frame set includes the first video frame and the filtered second video frame.

[0024] In this embodiment of the application, the first video frame in the first video frame set may or may not be filtered.

[0025] In conjunction with the first aspect, in some implementations, a third video frame set is obtained, which includes multiple video frames, at least two of which have different scenes, and / or the background of the third video frame set is in motion; scene switching detection is performed on the third video frame set to obtain a fourth video frame set, which is a subset of the third video frame set, and includes multiple video frames, each of which has the same scene; the background of the fourth video frame set is analyzed to obtain a first video frame set, which is a subset of the fourth video frame set.

[0026] In conjunction with the first aspect, in some implementations, the first video frame set in the third video frame set is replaced with the second video frame set to obtain the fifth video frame set; the fifth video frame set is then encoded to output the first bitstream.

[0027] In this embodiment of the application, when the first video frame set is a subset of the third video frame set, and the scenes in other video frames in the third video frame set are different and / or the background is in motion, the second video frame set can be obtained by filtering only some or all of the video frames in the first video frame set, and the first video frame set in the third video frame set can be replaced with the second video frame set for output.

[0028] Secondly, an image processing apparatus is provided. This apparatus includes modules for implementing the first aspect or any possible implementation thereof.

[0029] Thirdly, this application provides a computing device cluster, including at least one computing device, each computing device including a processor and a memory; the processor of the at least one computing device is used to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster performs the method in the first aspect or any possible implementation of the first aspect.

[0030] Fourthly, this application provides a computer program product containing instructions that, when executed by a cluster of computer devices, cause the cluster of computer devices to perform the method described in the first aspect or any possible implementation thereof.

[0031] Fifthly, this application provides a computer-readable storage medium including computer program instructions that, when executed by a cluster of computing devices, perform the method described in the first aspect or any possible implementation thereof. Attached Figure Description

[0032] Figure 1 This is a schematic structural diagram of an image processing system according to an embodiment of this application.

[0033] Figure 2 This is a schematic flowchart of an image processing method according to an embodiment of this application.

[0034] Figure 3 This is a schematic flowchart of an image processing method according to an embodiment of this application.

[0035] Figure 4 This is a schematic flowchart of an image processing method according to an embodiment of this application.

[0036] Figure 5 This is a schematic diagram of a video frame and an image region with micro-motion and irregular texture in the video frame, according to an embodiment of this application.

[0037] Figure 6 This is a schematic structural block diagram of an image processing apparatus according to an embodiment of the present application.

[0038] Figure 7 This is a schematic structural diagram of a computing device according to an embodiment of this application.

[0039] Figure 8 This is a schematic structural diagram of a computing device cluster according to an embodiment of this application.

[0040] Figure 9 This is a schematic diagram showing the connection between computing devices 700A and 700B via a network according to an embodiment of this application. Detailed Implementation

[0041] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0042] This application will present various aspects, embodiments, or features relating to a system comprising multiple devices, components, modules, etc. It should be understood and appreciated that individual systems may include additional devices, components, modules, etc., and / or may not include all the devices, components, modules, etc. discussed in conjunction with the accompanying drawings. Furthermore, combinations of these approaches are also possible.

[0043] Furthermore, in the embodiments of this application, the words "exemplary," "for example," etc., are used to indicate that they are examples, illustrations, or descriptions. Any embodiment or design scheme described as "exemplary" in the embodiments of this application should not be construed as being better or more advantageous than other embodiments or design schemes. Specifically, the use of the term "exemplary" is intended to present the concept in a concrete manner.

[0044] The business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0045] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0046] In this application embodiment, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, and B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0047] The technical solutions in this application embodiment can be applied to computing devices, such as servers, hosts, personal computers, laptops, desktops, wearable devices, and other devices with computing capabilities. When the technical solutions in this application embodiment are applied to a server, the server can be a local server or a cloud server, etc., and this application embodiment is not limited in this regard. The technical solutions in this application embodiment can also be applied to video acquisition devices, such as cameras, camcorders, and other devices used to acquire video and / or images. The technical solutions in this application embodiment can also be applied to video storage devices used to store video and / or images. The technical solutions in this application embodiment can be applied to scenarios such as video conferencing, video surveillance, and video storage.

[0048] Figure 1 This is a schematic structural diagram of the image processing system provided in the embodiments of this application. Figure 1 The image processing system 100 includes a transmitting device 110 and a receiving device 120. The transmitting device 110 includes a processor 112 and a memory 113. The receiving device 120 includes a processor 121 and a memory 122.

[0049] In some embodiments, the transmitting device 110 further includes an image sensor 111. Alternatively, the image sensor 111 may be located outside the transmitting device 110, but connected to the transmitting device 110. The image sensor 111 is used to acquire video images. This application embodiment does not limit the specific type of the image sensor 111, such as a charge-coupled device (CCD) or complementary metal-oxide-semiconductor (CMOS).

[0050] In some embodiments, the image processing system 100 further includes a receiving device 130.

[0051] The processor 112 in the transmitting device 110 is used to process the data acquired by the image sensor 111. For example, the processor 112 is used to execute the methods provided in the embodiments of this application. Alternatively, the processor 112 is used to encode the acquired data to obtain a bitstream and send the bitstream to the receiving device 120. The type of processor 113 is not limited in the embodiments of this application, and may include at least one of the following: a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), etc. The memory 113 is used to store at least one of the following: data acquired by the image sensor 111, data required or generated by the processor 112 during video encoding or execution of the methods in the embodiments of this application. The type of memory 113 is not limited in the embodiments of this application, and may include volatile memory and / or non-volatile memory.

[0052] For example, the transmitting device 110 is a computing device or a video acquisition device.

[0053] The processor 121 in the receiving device 120 is used to receive and decode the bitstream from the transmitting device 110. The processor 121 is also used to execute the methods provided in the embodiments of this application on the decoded bitstream. Alternatively, the processor 121 is used to re-encode the received bitstream and send it to the receiving device 130. The memory 122 is used to store at least one of the following: the bitstream received by the receiving device 120, the decoded data of the bitstream, and data required or generated by the processor 121 during the execution of the methods in the embodiments of this application.

[0054] For example, when the transmitting device 110 is a computing device, the receiving device 120 is a computing device. When the transmitting device 110 is a video acquisition device, the receiving device 120 is a computing device or a video storage device.

[0055] In some embodiments, receiving device 130 is used to receive a bitstream from receiving device 120, the bitstream being obtained based on at least one video frame after image processing according to the method provided in the embodiments of this application.

[0056] For example, the receiving device 130 is a computing device.

[0057] In some embodiments, the image processing system 100 is applied to the following scenarios:

[0058] (1) Scenario 1: The transmitting device 110 acquires a captured video image and performs image processing according to the method in this embodiment to obtain a processed video image. The transmitting device 110 encodes the processed video image to obtain a bitstream and sends the bitstream to the receiving device 120. After receiving the bitstream, the receiving device 120 re-encodes the bitstream and sends the re-encoded bitstream to the receiving device 130. Alternatively, the receiving device 120 receives the bitstream and stores it.

[0059] (2) Scenario 2: The transmitting device 110 acquires the captured video image and encodes it to obtain a bitstream. The transmitting device 110 sends the bitstream to the receiving device 120. The receiving device 120 decodes the received bitstream and performs image processing on the decoded video image according to the method provided in the embodiments of this application to obtain a processed video image. The receiving device 120 encodes the processed video image to obtain a bitstream and sends the bitstream to the receiving device 130. Alternatively, the receiving device 120 encodes the processed video image to obtain a bitstream and stores the bitstream.

[0060] Figure 2 This is a schematic flowchart of the image processing method provided in the embodiments of this application. Figure 2 The method described above is applied in an image processing device, such as... Figure 1 Any device in the system. Figure 2 The method includes the following steps.

[0061] 210, obtain the first set of video frames.

[0062] An image processing device obtains a first set of video frames, which includes multiple video frames. The background of the first set of video frames is static. In other words, the scene in each video frame of the first set of video frames is the same, and the background of the first video obtained from the first set of video frames is static.

[0063] Optionally, before step 210, the image processing device obtains a third video frame set. This third video frame set includes multiple video frames. When the third video frame set is acquired by an image sensor with fixed shooting parameters and no translation, rotation, zoom, or other similar actions, the background in the third video frame set is static. When the third video frame set is acquired by an image sensor with variable shooting parameters or with at least one of translation, rotation, zoom, or other similar actions, at least two video frames in the third video frame set depict different scenes, and / or, the background in the third video frame set is in motion.

[0064] If the background in the third video frame set is static, then the third video frame set is considered part of the first video frame set. If at least two video frames in the third video frame set have different scenes, and / or the background in the third video frame set is in motion, then the first video frame set is a subset of the third video frame set. For details on how to determine the first video frame set when it is a subset of the third video frame set, please refer to [link to relevant documentation]. Figure 3 Steps 320-330 in the process.

[0065] In some embodiments, the video frames in the first video frame set are arranged in either the normal playback order or the acquisition order.

[0066] Optionally, the image processing device acquires the third set of video frames independently. Alternatively, the image processing device receives a third set of video frames from other devices connected to it, such as image acquisition devices, computing devices, storage devices, etc.

[0067] 220. Based on the first video frame and the second video frame, determine the first image region in the second video frame.

[0068] After obtaining a first set of video frames, the image processing device determines a first video frame and a second video frame within the first set of video frames, and determines a first image region within the second video frame. The first image region includes at least one pixel. The amplitude of the motion vector of the first image region in the second video frame is less than or equal to a first preset threshold. This application embodiment does not limit the value of the first preset threshold. The amplitude of the motion vector of the first image region in the second video frame is determined based on the first and second video frames.

[0069] Optionally, the image processing device determines the magnitude of the motion vector of at least one image block in the second video frame based on the first video frame and the second video frame. Each image block in the at least one image block includes at least one pixel. The image processing device determines the first image region based on the magnitude of the motion vector of each image block in the second video frame. The first image region includes one or more image blocks. The magnitude of the motion vector of the first image region in the second video frame being less than or equal to a first preset threshold includes: the magnitude of the motion vector of each image block included in the first image region in the second video frame being less than or equal to the first preset threshold; or, among the image blocks included in the first image region, the number of image blocks whose motion vector magnitude in the second video frame is less than or equal to the first preset threshold is greater than or equal to a third preset threshold. The embodiments of this application do not limit the method for determining the motion vector of the image block, and may include methods such as block-level motion estimation. The embodiments of this application do not limit the specific value of the third preset threshold.

[0070] In some embodiments, the magnitude of the motion vector of the first image block in the second video frame is used to indicate the displacement between the first image block and the second image block in the second video frame. The first image block belongs to the second video frame. The first image block and the second image block include the same number of pixels. The second image block in the first video frame is the image block in the first video frame that is most similar to the first image block. Alternatively, the similarity between the first image block and the second image block is greater than or equal to the similarity between the first image block and other image blocks in the first video frame besides the second image block. Alternatively, the similarity between the second image block in the first video frame and the first image block in the second video frame is greater than or equal to a fourth preset threshold. The embodiments of this application do not limit the size of this fourth preset threshold.

[0071] For example, the magnitude of the motion vector of the first image block in the second video frame is determined based on the horizontal and vertical displacements of the first image block and the second image block in the first video frame.

[0072] For example, suppose a pixel in the first image block (e.g., the bottom left vertex pixel) is positioned (x1, y1) in the second video frame, and the corresponding pixel in the second image block (e.g., the bottom left vertex pixel) is positioned (x2, y2) in the first video frame. Then, the horizontal displacement of the first and second image blocks in the video frame is x3 = |x1 - x2|, and the vertical displacement of the first and second image blocks in the video frame is y3 = |y1 - y2|. Therefore, the magnitude of the motion vector of the first image block in the second video frame is the L2 norm of x3 and y3, that is, the magnitude of the motion vector of the first image block in the second video frame is... Alternatively, the magnitude of the motion vector of the first image block in the second video frame is the L1 norm of x3 and y3, that is, the magnitude of the motion vector of the first image block in the second video frame is |x3|+|y3|.

[0073] For example, the magnitude of the motion vector of each pixel in the same image block is the same in the second video frame.

[0074] Optionally, the image processing device determines the motion vector of each pixel in the second video frame based on the first video frame and the second video frame. The image processing device then determines the first image region based on the motion vector of each pixel in the second video frame. The first image region includes at least one pixel. The amplitude of the motion vector of the first image region in the second video frame being less than or equal to a first preset threshold includes: the amplitude of the motion vector of each pixel in the first image region in the second video frame being less than or equal to the first preset threshold; or, among the pixels included in the first image region, the number of pixels whose motion vector amplitude in the second video frame is less than or equal to the first preset threshold is greater than or equal to a fifth preset threshold. This application embodiment does not limit the method for determining the motion vector of a pixel, and may include methods such as optical flow estimation. This application embodiment does not limit the specific value of the fifth preset threshold.

[0075] In some embodiments, the magnitude of the motion vector of the first pixel in the second video frame is used to indicate the displacement between the first pixel in the second video frame and the second pixel in the first video frame. The first pixel belongs to the second video frame, and the first pixel corresponds to the second pixel.

[0076] For example, the correspondence between the first pixel and the second pixel includes: the similarity between a third image block centered on the first pixel in the second video frame and a fourth image block centered on the second pixel in the first video frame is greater than or equal to a fourth preset threshold; or, the fourth image block is the image block in the first video frame most similar to the third image block; or, the similarity between the fourth image block and the third image block is greater than or equal to the similarity between the third image block and image blocks other than the fourth image block in the first video frame. The third image block and the fourth image block include the same number of pixels, and the third image block includes multiple pixels.

[0077] For example, the magnitude of the motion vector of the first pixel in the second video frame is determined based on the horizontal and vertical displacements of the first pixel and the second pixel in the first video frame.

[0078] For example, assuming the position of the first pixel in the second video frame is (x4, y4) and the position of the second pixel in the first video frame is (x5, y5), then the horizontal displacement of the first and second pixels in the video frame is x6 = |x4 - x5|, and the vertical displacement of the first and second pixels in the video frame is y6 = |y4 - y5|. Therefore, the magnitude of the motion vector of the first pixel in the second video frame is the L2 norm of x6 and y6, that is, the magnitude of the motion vector of the first pixel in the second video frame is... Alternatively, the magnitude of the motion vector of the first pixel in the second video frame is the L1 norm of x6 and y6, that is, the magnitude of the motion vector of the first pixel in the second video frame is |x6|+|y6|.

[0079] In some embodiments, the first video frame is a frame preceding the second video frame in the first video frame set, or the first video frame is a frame obtained by filtering the frame preceding the second video frame in the first video frame set. Alternatively, the first video frame is a frame following the second video frame in the first video frame set, or the first video frame is a frame obtained by filtering the frame following the second video frame in the first video frame set.

[0080] For example, the first video frame and the second video frame are two adjacent frames in the first video frame set. For instance, the first video frame is the frame preceding the second video frame in the normal playback order or capture order, and the second video frame is the frame following the first video frame in the normal playback order or capture order. Alternatively, the first video frame is the frame following the second video frame in the normal playback order or capture order, and the second video frame is the frame preceding the first video frame in the normal playback order or capture order. Or, the first video frame is a frame obtained by filtering the frames adjacent to the second video frame in the first video frame set. For example, the first video frame is a frame obtained by filtering the frame preceding the second video frame in the first video frame set, or the first video frame is a frame obtained by filtering the frame following the second video frame in the first video frame set.

[0081] For example, the first video frame and the second video frame are separated by at least one frame in the first video frame set. For instance, in normal playback order, the first video frame is the second frame before the second video frame, or the first video frame is the second frame after the second video frame, meaning there is a one-frame interval between the first and second video frames. Alternatively, the first video frame can be a frame obtained by filtering frames in the first video frame set that are separated from the second video frame by at least one frame. For example, the first video frame can be a frame obtained by filtering the second frame in the first video frame set before the second video frame, or the first video frame can be a frame obtained by filtering the second frame in the first video frame set after the second video frame.

[0082] In some embodiments, where the first video frame is a frame obtained by filtering a frame preceding the second video frame in the first video frame set, the image processing device filters the frame preceding the second video frame in the first video frame set before step 220 to obtain the first video frame. The specific implementation of obtaining the first video frame is similar to the method of obtaining the filtered second video frame in step 230, and will not be described again here.

[0083] In some embodiments, the magnitude of the motion vector of the first image region in the second video frame is greater than 0. In other words, the first image region is a micro-motion region.

[0084] For example, the motion vector amplitude of the first image region in the second video frame being greater than 0 includes: the motion vector amplitude of each image block in the first image region in the second video frame being greater than 0; or, among the image blocks included in the first image region, the number of image blocks with motion vector amplitudes greater than 0 in the second video frame being greater than or equal to a sixth preset threshold; or, the motion vector amplitude of image blocks in the first image region excluding edge image blocks in the second video frame being greater than 0, wherein the edge image blocks include at least one image block near the edge in the first image region; or, the motion vector amplitude of each pixel included in the first image region in the second video frame being greater than 0; or, among the pixels included in the first image region, the number of pixels with motion vector amplitudes greater than 0 in the second video frame being greater than or equal to a seventh preset threshold; or, the motion vector amplitude of pixels in the first image region excluding edge pixels in the second video frame being greater than 0, wherein the edge pixels include at least one pixel near the edge in the first image region. The specific values ​​of the sixth and seventh preset thresholds are not limited in this application embodiment.

[0085] In some embodiments, the degree of texture irregularity or motion irregularity of the first image region in the second video frame is greater than or equal to a second preset threshold. The specific value of this second preset threshold is not limited in the embodiments of this application. For example, when the first image region includes images with irregular textures such as grass, leaves, flowers, and animal fur, the degree of texture irregularity of the first image region is greater than or equal to the second preset threshold. That is, the first image region is an irregular texture region. The method for determining the degree of texture irregularity and motion irregularity is described in step 430.

[0086] For example, the degree of motion irregularity of the first image region is used to represent the degree of texture irregularity of the first image region. A greater degree of motion irregularity in the first image region indicates a higher degree of texture irregularity. Conversely, a smaller degree of motion irregularity in the first image region indicates a lower degree of texture irregularity.

[0087] In some embodiments, the value of the first preset threshold may be the same or different depending on the application scenario. For example, for a set of video frames after removing micro-motion in the micro-motion image region, if the tolerance for visual effects when playing the video obtained from the set of video frames is high, the first preset threshold may be set to a larger value; if the tolerance for visual effects when playing the video obtained from the set of video frames is low, the first preset threshold may be set to a smaller value.

[0088] Optionally, the image processing device determines at least one second image region in the second video frame based on the first video frame and the second video frame. Each of the at least one second image region is similar to an image region in the first video frame. The image processing device further determines at least one first image region based on the magnitude of the motion vector of each of the at least one second image regions in the second video frame. The magnitude of the motion vector of the second image region in the second video frame is determined based on the first video frame and the second video frame. The magnitude of the motion vector of each of the at least one first image regions in the second video frame is less than or equal to a first preset threshold, i.e., each first image region is a micro-motion region. The image processing device performs step 230 on each of the at least one first image regions.

[0089] For example, the similarity between the second image region and an image region in the first video frame includes: the similarity between the second image region and an image region in the first video frame is greater than or equal to the similarity between the second image region and other image regions in the first video frame besides that image region; or, the similarity between the second image region and an image region in the first video frame is greater than or equal to a fourth preset threshold.

[0090] In some embodiments, the image processing device determines at least one second image region in the second video frame and at least one fourth image region in the first video frame based on a first video frame and a second video frame. The at least one second image region corresponds one-to-one with the at least one fourth image region. Each of the at least one fourth image region is the image region in the first video frame most similar to the second image region corresponding to that fourth image region. Alternatively, the similarity between each second image region and its corresponding fourth image region is greater than or equal to the similarity between the second image region and other image regions in the first video frame besides the corresponding fourth image region. Alternatively, the similarity between the second image region and its corresponding fourth image region is greater than or equal to a fourth preset threshold.

[0091] Optionally, after determining at least one second image region in the second video frame, the image processing device determines at least one third image region from the at least one second image region based on the magnitude of the motion vector of each second image region in the second video frame. The magnitude of the motion vector of each third image region in the at least one third image region in the second video frame is less than or equal to a first preset threshold. Alternatively, the magnitude of the motion vector of each third image region in the second video frame is less than or equal to the first preset threshold, and the magnitude of the motion vector of each third image region in the second video frame is greater than 0. That is, each third image region is a micro-motion region. The image processing device determines at least one first image region from the at least one third image region. The degree of texture irregularity or motion irregularity of each first image region in the at least one first image region is greater than or equal to a second preset threshold. That is, each first image region is a micro-motion region, and each first image region is an irregular texture region. The image processing device performs step 230 on each first image region in the at least one first image region.

[0092] Optionally, the image processing device determines at least one fifth image region in the second video frame. The texture irregularity or motion irregularity of each of the at least one fifth image regions is greater than or equal to a second preset threshold. Based on the at least one fifth image region and at least one third image region, the image processing device determines at least one first image region. The magnitude of the motion vector of each of the at least one first image region in the second video frame is less than or equal to a first preset threshold, and the texture irregularity or motion irregularity of the first image region is greater than or equal to a second preset threshold. The image processing device performs step 230 for each of the at least one first image regions.

[0093] In some embodiments, the magnitude of the motion vector of edge pixels in the first image region in the second video frame is 0. That is, the edge pixels in the first image region are stationary in both the first and second video frames. The edge pixels in the first image region include at least one pixel near the edge of the first image region. Alternatively, the magnitude of the motion vector of edge image blocks in the first image region in the second video frame is 0. That is, the edge image blocks in the first image region are stationary in both the first and second video frames. The edge image blocks in the first image region include at least one image block near the edge of the first image region.

[0094] Optionally, step 220 can be replaced by: determining a first image region in the second video frame based on a plurality of video frames and a second video frame. At least one of the plurality of video frames belongs to a first video frame set, and / or, at least one of the plurality of video frames is a video frame obtained by filtering video frames in the first video frame set. The plurality of video frames includes the first video frame.

[0095] In some embodiments, the image processing device determines at least one fifth image block in the first video frame and at least one sixth image block in the second video frame based on a first video frame and a second video frame. Each of the at least one sixth image block is similar to one of the at least one fifth image blocks. The magnitude of the motion vector of each of the at least one sixth image block in the second video frame is less than or equal to a first preset threshold. The image processing device determines a first image region in the second video frame based on the magnitude of the motion vector of the at least one fifth image block in the first video frame. Specifically, if the magnitude of the motion vector of a fifth image block in the first video frame is less than or equal to the first preset threshold, the image processing device determines that a sixth image block similar to the fifth image block belongs to the first image region. If the magnitude of the motion vector of a fifth image block in the first video frame is greater than the first preset threshold, the image processing device determines that a sixth image block similar to the fifth image block does not belong to the first image region. The magnitude of the motion vector of the fifth image block in the first video frame is determined based on the first video frame and a third video frame. The third video frame belongs to the set of first video frames, or the third video frame is a frame obtained by filtering a frame before or after the first video frame in the set of first video frames. The method for determining the magnitude of the motion vector of the fifth image block in the first video frame is similar to the method for determining the magnitude of the motion vector of the first image block in the second video frame, and will not be repeated here.

[0096] In some embodiments, the image processing device determines at least one third pixel in the first video frame and at least one fourth pixel in the second video frame based on a first video frame and a second video frame. Each of the at least one fourth pixel corresponds to one of the at least one third pixels. The magnitude of the motion vector of each of the at least one fourth pixel in the second video frame is less than or equal to a first preset threshold. The image processing device determines a first image region in the second video frame based on the magnitude of the motion vector of the at least one third pixel in the first video frame. Specifically, if the magnitude of the motion vector of the third pixel in the first video frame is less than or equal to the first preset threshold, the image processing device determines that the fourth pixel corresponding to the third pixel belongs to the first image region. If the magnitude of the motion vector of the third pixel in the first video frame is greater than the first preset threshold, the image processing device determines that the fourth pixel corresponding to the third pixel does not belong to the first image region. The magnitude of the motion vector of the third pixel in the first video frame is determined based on the first video frame and the third video frame. The third video frame belongs to the set of first video frames, or the third video frame is a frame obtained by filtering a frame before or after the first video frame in the set of first video frames. The method for determining the magnitude of the motion vector of the three pixels in the first video frame is similar to the method for determining the magnitude of the motion vector of the first pixel in the second video frame, and will not be repeated here.

[0097] 230. Filter the first image region in the second video frame to obtain the filtered second video frame.

[0098] The image processing device performs temporal filtering on the first image region in the second video frame to obtain a filtered second video frame. The magnitude of the motion vector of the first image region in the filtered second video frame is 0. The motion vector of the first image region in the filtered second video frame is determined based on the first video frame and the filtered second video frame. The first video frame is described in step 220. In other words, after filtering the first image region in the second video frame, the micro-motion in the first image region is removed, and the first image region is filtered into a stationary region.

[0099] For example, the method for determining the motion vector of the first image region in the filtered second video frame is similar to the method for determining the motion vector of the first image region in the second video frame, and will not be repeated here.

[0100] In some embodiments, the magnitude of the motion vector of the first image region in the filtered second video frame is 0, which includes: the magnitude of the motion vector of each pixel in the first image region in the filtered second video frame is 0; or, the magnitude of the motion vector of each image block in the first image region in the filtered second video frame is 0.

[0101] Optionally, the image processing device replaces pixels in the first image region of the second video frame with pixels in the first image region of the reference frame to obtain a filtered second video frame. The reference frame includes the first image region.

[0102] For example, assuming that the first image region in both the second video frame and the reference frame is a 5*5 image block, the image processing device directly replaces the pixel value of the pixel in the 5*5 image block in the second video frame with the corresponding pixel value in the 5*5 image block in the reference frame, thereby obtaining the filtered second video frame.

[0103] In some embodiments, the reference frame is any one of the following: the first video frame in the first video frame set, the first reconstructed frame corresponding to the first video frame set, the background frame corresponding to the first video frame set, or the first video frame. The first video frame is a frame obtained by filtering a frame preceding the second video frame in the first video frame set. The first reconstructed frame is the first keyframe reconstructed during the decoding process of the first video frame set. The background frame is a frame including the background extracted from some or all of the video frames in the first video frame set. For example, the background frame does not include the foreground from the first video frame set.

[0104] When the reference frame is the first video frame or the first reconstructed frame in the first video frame set, if the first video frame is the first video frame in the first video frame set, the image processing device does not need to filter the first video frame. When the reference frame is a background frame, if the first video frame is the first video frame in the first video frame set, the image processing device performs step 230 on the sixth image region in the first video frame to obtain the filtered first video frame. The amplitude of the motion vector of the sixth image region in the first video frame is less than or equal to a first preset threshold; or, the amplitude of the motion vector of the sixth image region in the first video frame is less than or equal to the first preset threshold, and the amplitude of the motion vector of the sixth image region in the first video frame is greater than 0. The amplitude of the motion vector of the sixth image region in the first video frame is determined based on the first video frame and the background frame. Alternatively, when the reference frame is a background frame, if the first video frame is the first video frame in the first video frame set, the image processing device does not need to filter the first video frame.

[0105] For example, the degree of texture irregularity or motion irregularity in the sixth image region is greater than or equal to the second preset threshold.

[0106] For example, the motion vector amplitude of the sixth image region in the first video frame being less than or equal to a first preset threshold includes: the motion vector amplitude of each image block included in the sixth image region in the first video frame being less than or equal to the first preset threshold; or, among the image blocks included in the sixth image region, the number of image blocks whose motion vector amplitude in the first video frame is less than or equal to the first preset threshold is greater than or equal to a third preset threshold; or, the motion vector amplitude of each pixel in the sixth image region in the first video frame being less than or equal to the first preset threshold; or, among the pixels included in the sixth image region, the number of pixels whose motion vector amplitude in the first video frame is less than or equal to the first preset threshold is greater than or equal to a fifth preset threshold. The specific values ​​of the third and fifth preset thresholds are not limited in the embodiments of this application.

[0107] For example, the motion vector amplitude of the sixth image region in the first video frame being greater than 0 includes: the motion vector amplitude of each image block in the sixth image region in the first video frame being greater than 0; or, among the image blocks included in the sixth image region, the number of image blocks with motion vector amplitudes greater than 0 in the first video frame being greater than or equal to a sixth preset threshold; or, the motion vector amplitude of image blocks in the sixth image region excluding edge image blocks in the first video frame being greater than 0, wherein the edge image blocks include at least one image block near the edge in the sixth image region; or, the motion vector amplitude of each pixel included in the sixth image region in the first video frame being greater than 0; or, among the pixels included in the sixth image region, the number of pixels with motion vector amplitudes greater than 0 in the first video frame being greater than or equal to a seventh preset threshold; or, the motion vector amplitude of pixels in the sixth image region excluding edge pixels in the first video frame being greater than 0. The edge pixels include at least one pixel near the edge in the sixth image region. The specific values ​​of the sixth and seventh preset thresholds are not limited in this application embodiment.

[0108] In some embodiments, when the reference frame is a background frame, the image processing device determines the background frame based on all video frames in the first video frame set, and then performs step 230 on each video frame in the first video frame set based on the background frame. Alternatively, when the reference frame is a background frame, the image processing device determines the background frame based on the first n video frames in the first video frame set, and then performs step 230 on each video frame from the (n+1)th to the Nth video frame in the first video frame set based on the background frame. The first video frame set includes N video frames, where N is a positive integer greater than 1, and n = 1, 2, ..., N-1.

[0109] In some embodiments, when the second video frame includes multiple first image regions, the image processing device replaces the pixel value of a pixel in each first image region in the second video frame with the pixel value of a corresponding pixel in each first image region in the reference frame to obtain a filtered second video frame.

[0110] 240. Based on the filtered second video frame, obtain the second video frame set.

[0111] The image processing device replaces a second video frame in the first video frame set with a filtered second video frame, thereby obtaining a second video frame set. That is, the second video frame set includes the filtered second video frame. In other words, the image processing device filters at least one video frame in the first video frame set, then replaces at least one video frame in the first video frame set with a corresponding filtered at least one video frame, thereby obtaining a second video frame set.

[0112] If the first video frame is the first video frame in the first video frame set, and the first video frame does not require filtering, then the second video frame set includes the first video frame and the filtered second video frame. If the first video frame is the first video frame in the first video frame set, and the image processing device has filtered the first video frame, then the second video frame set includes the filtered first video frame and the filtered second video frame.

[0113] When the first video frame set is a subset of the third video frame set, the image processing device replaces the first video frame set in the third video frame set with the second video frame set to obtain a fifth video frame set. The image processing device encodes the fifth video frame set and outputs a first bitstream. The image processing device stores or transmits the first bitstream. Since micro-motion in the micro-motion region of at least one video frame in the second video frame set is removed, the amount of residual data after motion prediction can be reduced during the encoding of the fifth video frame set, thereby reducing the amount of encoded data and improving the compression ratio.

[0114] When the third video frame set is the same as the first video frame set, the image processing device encodes the second video frame set and outputs a second bitstream. The image processing device stores or transmits this second bitstream. Since micro-motion in the micro-motion region of at least one video frame in the second video frame set is removed, the amount of residual data after motion prediction can be reduced during the encoding of the second video frame set, thereby reducing the amount of encoded data and improving the compression ratio.

[0115] exist Figure 2In this method, the image processing device filters the micro-motion image region in the second video frame to remove the micro-motion in the micro-motion image region, making the image region a stationary region after filtering, thereby reducing the amount of data after video encoding and improving the compression rate of video encoding.

[0116] For example, when the first set of video frames is a subset of the third set of video frames, the image processing device... Figure 3 The method described in the text processes the third set of video frames to output the first bitstream.

[0117] Figure 3 This is a schematic flowchart of the image processing method provided in the embodiments of this application. Figure 3 The method described above is applied in an image processing device, such as... Figure 1 Any device in the system. Figure 3 The method includes the following steps.

[0118] 310, obtain the third set of video frames.

[0119] The third video frame set includes multiple video frames. When the third video frame set is acquired by an image sensor with fixed shooting parameters and no translation, rotation, zoom, or other similar actions, the scene in each video frame of the third video frame set is the same, and the background of the third video frame set is static. When the third video frame set is acquired by an image sensor with variable shooting parameters or with at least one of the actions such as translation, rotation, zoom, etc., the scenes in at least two video frames of the third video frame set are different, and / or, the background of the third video frame set is in motion.

[0120] Optionally, the image processing device acquires the third set of video frames independently. Alternatively, the image processing device receives a third set of video frames from other devices connected to it, such as image acquisition devices, computing devices, storage devices, etc.

[0121] 320. Perform scene switching detection on the third video frame set to obtain the fourth video frame set.

[0122] The image processing device performs scene transition detection on the third video frame set to obtain at least one fourth video frame set. Each of the at least one fourth video frame set includes at least one video frame, and the scenes in the video frames included in each fourth video frame set are the same. That is, each fourth video frame set is a subset of the third video frame set.

[0123] Optionally, the specific type of scene transition detection algorithm is not limited in the embodiments of this application. For example, the scene transition detection algorithm includes: algorithms in the open-source encoder x264 / x265, grayscale histogram detection algorithms, deep learning algorithms, etc. The open-source encoder x264 / x265 determines whether there is a scene transition between two video frames by calculating the degree of dissimilarity between them. The grayscale histogram detection algorithm determines whether there is a scene transition between two video frames by comparing the differences between their grayscale histograms. The deep learning algorithm extracts features from two video frames using a deep neural network and compares them to determine whether there is a scene transition between them.

[0124] In some embodiments, if it is determined that there is no scene switching between two adjacent video frames in the third video frame set, the image processing device assigns the two adjacent video frames to the same fourth video frame set; if it is determined that there is a scene switching between two adjacent video frames in the third video frame set, the image processing device assigns the two adjacent video frames to different fourth video frame sets, thereby obtaining the at least one fourth video frame set.

[0125] 330. Analyze the background in the fourth video frame set to obtain the first video frame set.

[0126] The image processing device analyzes the background in the fourth video frame set to obtain at least one first video frame set. Each of the at least one first video frame set includes multiple video frames, and the background in each first video frame set is static. That is, each first video frame set is a subset of the fourth video frame set.

[0127] Optionally, the image processing device performs motion analysis on the fourth video frame set to obtain the at least one first video frame set. This application embodiment does not limit the specific type of motion analysis method. For example, the motion analysis method includes: a method based on optical flow estimation, a method based on block-level motion estimation, a method based on the original video bitstream, a deep learning algorithm, etc. The optical flow estimation method determines the motion direction and velocity of corresponding pixels in two video frames by determining the position changes of corresponding pixels in the two video frames, thereby extracting the motion vectors of corresponding pixels in the two video frames, and thus determining whether the background in the two video frames is stationary. The block-level motion estimation method determines the motion vector of the image block by calculating the displacement between corresponding image blocks in two video frames, thereby determining whether the background in the two video frames is stationary. The original video bitstream-based method determines whether the background in the two video frames is stationary by extracting the motion vector information included in the encoded bitstream of the original video. The deep learning algorithm extracts features from the two video frames using a deep neural network and compares them to determine whether the background in the two video frames is stationary.

[0128] In some embodiments, if it is determined that the background of two adjacent video frames in the fourth video frame set is stationary, the image processing device assigns the two adjacent video frames to the same first video frame set; if it is determined that the background of two adjacent video frames in the fourth video frame set is not stationary (i.e., in motion), the image processing device assigns the two adjacent video frames to different first video frame sets, thereby obtaining the at least one first video frame set.

[0129] Optionally, the image processing device performs background modeling on the fourth video frame set to obtain the at least one first video frame set. This application embodiment does not limit the specific type of the background modeling method. For example, the background modeling method includes: a background modeling method based on a Gaussian mixture model (GMM), a background modeling method based on deep learning, inter-frame difference methods, etc. The GMM-based background modeling method utilizes the principle that background pixels typically have lower grayscale value variations while foreground pixels have higher grayscale value variations. Based on the Gaussian distribution parameters of the pixel values ​​in the video frames, it determines whether each pixel belongs to the background or foreground, thereby extracting the background frame corresponding to the video frame set. The deep learning-based background modeling method uses a deep neural network model to perform feature extraction and background modeling on the image, thereby extracting the background frame corresponding to the video frame set. This deep neural network model includes: a convolutional neural network (CNN), a generative adversarial network (GAN), etc. The inter-frame difference method determines the foreground region by binarizing the difference images of two video frames and defines the image outside the foreground region as the background region, thereby extracting the background frame corresponding to the set of video frames.

[0130] In some embodiments, the image processing device determines m1 video frames (e.g., frames 1 to m1) in a fourth video frame set based on a sliding window, and performs background modeling on these m1 video frames. Here, m1 is a positive integer, m1 ≤ M, and M is the number of video frames included in the fourth video frame set, where M is a positive integer. If a background frame can be modeled from these m1 video frames, the image processing device determines that these m1 video frames belong to the same first video frame set. Based on the background frame modeled from these m1 video frames and the remaining video frames in the fourth video frame set excluding the m1 video frames, the image processing device determines that video frames in the remaining video frames in the fourth video frame set excluding the m1 video frames have the same background as the background frame and belong to the same first video frame set as the m1 video frames; and video frames in the remaining video frames in the fourth video frame set excluding the m1 video frames have a different background than the background frame and belong to different first video frame sets. If, in the fourth video frame set, there are video frames whose background differs from the specified m1 video frames, the image processing device determines another set of m1 video frames from these frames using a sliding window to perform background modeling, and repeats the above steps to determine at least one first video frame set. If the specified m1 video frames cannot be used to model a background frame, the image processing device determines the next set of m1 video frames in the fourth video frame set (e.g., frames m2 to m2+m1-1, where m2 is a positive integer and m2+m1-1 ≤ M) using a sliding window, and repeats the above steps to determine at least one first video frame set.

[0131] In some embodiments, if it is determined that M video frames in the fourth video frame set can be used to model background frames, the image processing device assigns these M video frames to the same first video frame set. The fourth video frame set includes M video frames, where M is a positive integer. If it is determined that M video frames in the fourth video frame set cannot be used to model background frames, the image processing device determines m1 video frames in the fourth video frame set according to a sliding window, performs background modeling on these m1 video frames, and repeats the above steps to determine at least one first video frame set. m1 is a positive integer, and m1 ≤ M.

[0132] For example, m1 is greater than or equal to the minimum number of video frames required by the background modeling method.

[0133] 340. Filter the image region in at least one video frame in the first video frame set to obtain the second video frame set.

[0134] The image processing device filters image regions in at least one video frame within each first video frame set to obtain a second video frame set corresponding to each first video frame set. For details on the filtering method, please refer to [link / reference needed]. Figure 2or Figure 4 The description in the text will not be repeated here.

[0135] 350, replace the first video frame set in the third video frame set with the second video frame set to obtain the fifth video frame set.

[0136] The image processing device replaces at least one set of first video frames in the third video frame set with a corresponding set of second video frames to obtain a fifth video frame set. In other words, the image processing device filters the micro-motion image regions in the steady-state video frames of the third video frame set and combines them with the non-steady-state video frames in the third video frame set according to the normal playback order or the acquisition order to obtain the fifth video frame set. The steady-state video frames refer to video frames in at least one set of first video frames, and the non-steady-state video frames refer to video frames in the third video frame set other than those in the at least one set of first video frames.

[0137] Optionally, the image processing device encodes the fifth video frame set and outputs a first bitstream. The image processing device stores or transmits the first bitstream. Since micro-motion in the micro-motion regions of the steady-state video frames in the fifth video frame set is removed, the amount of residual data after motion prediction can be reduced during the encoding of the fifth video frame set, thereby reducing the amount of encoded data and improving the compression ratio.

[0138] exist Figure 3 In this method, regardless of whether the video frame set received by the image processing device was captured in a steady-state scene, the image processing device can extract the steady-state video frames and filter them, thereby reducing the amount of data in the encoded video frame set and improving the compression ratio. A steady-state scene refers to a shooting scene where the image acquisition device's shooting parameters are fixed and there is no translation, rotation, or zoom. A non-steady-state scene refers to a shooting scene where the image acquisition device's shooting parameters are not fixed or where there is at least one of translation, rotation, or zoom.

[0139] Figure 4 This is a schematic flowchart of the image processing method provided in the embodiments of this application. Figure 4 The method described above is applied in an image processing device, such as... Figure 1 Any device in the system. Figure 4 The method includes the following steps.

[0140] 410. Based on the first video frame and the second video frame, determine at least one second image region in the second video frame.

[0141] The image processing device determines at least one image region that is most similar to the first video frame and the second video frame based on the first video frame and the second video frame. That is, the image processing device determines at least one second image region in the second video frame and at least one fourth image region in the first video frame. Each of the at least one second image region is most similar to a fourth image region in the first video frame. The at least one second image region and the at least one fourth image region are described in step 220.

[0142] Optionally, the embodiments of this application do not limit the specific type of method for determining similar image regions in two video frames. For example, methods for determining similar image regions in two video frames include: grayscale-based matching methods, feature-based matching methods, deep learning-based matching methods, etc. The grayscale-based matching method converts two video frames into grayscale images, moves a sliding window across one video frame, and calculates the similarity between the image region within the sliding window and each image region of the same size in the other video frame, thereby determining at least one matching image region in the two video frames. The similarity of image regions can be measured by the pixel difference, correlation coefficient, etc., between the two image regions. The feature-based matching method extracts key points (e.g., corners, edges) and descriptors (vectors used to describe information about the image regions around the key points) from the two video frames and compares the similarity between the descriptors, thereby determining at least one matching image region in the two video frames. The deep learning-based matching method extracts feature vectors of image regions from the two video frames and compares the similarity between feature vectors of different image regions in the two video frames, thereby determining at least one matching image region in the two video frames.

[0143] Optionally, step 410 can be replaced by: determining a first image region in the second video frame based on multiple video frames and the second video frame. At least one of the multiple video frames belongs to a first video frame set, and / or, at least one of the multiple video frames is a video frame obtained by filtering video frames in the first video frame set. The multiple video frames include the first video frame. See [link to specific implementation details] Figure 2 The description in the text.

[0144] 420, determine at least one third image region based on the magnitude of the motion vector of each second image region in the second video frame.

[0145] The image processing device determines at least one third image region based on the amplitude of the motion vector of each second image region in a second video frame. The amplitude of the motion vector of the second image region in the second video frame is determined based on the first and second video frames. The amplitude of the motion vector of each of the at least one third image region in the second video frame is less than or equal to a first preset threshold. In other words, each third image region is a micro-motion image region. The specific value of the first preset threshold is not limited in the embodiments of this application.

[0146] In some embodiments, the fact that the amplitude of the motion vector of the third image region in the second video frame is less than or equal to a first preset threshold includes: the amplitude of the motion vector of each image block included in the third image region in the second video frame is less than or equal to the first preset threshold; or, among the image blocks included in the third image region, the number of image blocks whose motion vector amplitude in the second video frame is less than or equal to the first preset threshold is greater than or equal to a third preset threshold; or, the amplitude of the motion vector of each pixel in the third image region in the second video frame is less than or equal to the first preset threshold; or, among the pixels included in the third image region, the number of pixels whose motion vector amplitude in the second video frame is less than or equal to the first preset threshold is greater than or equal to a fifth preset threshold. The specific values ​​of the third preset threshold and the fifth preset threshold are not limited in the embodiments of this application.

[0147] Optionally, the embodiments of this application do not limit the specific type of method for determining motion vectors. For example, the method for determining motion vectors includes: a method based on optical flow estimation, a method based on block-level motion estimation, a method based on the original video bitstream, etc., as described in step 330.

[0148] For example, the calculation method of the magnitude of the motion vector of the second image region in the second video frame is described in step 220, and will not be repeated here.

[0149] In some embodiments, the magnitude of the motion vector of each third image region in the second video frame is greater than 0.

[0150] For example, the motion vector amplitude of the third image region in the second video frame being greater than 0 includes: the motion vector amplitude of each image block in the third image region in the second video frame being greater than 0; or, among the image blocks included in the third image region, the number of image blocks with motion vector amplitudes greater than 0 in the second video frame being greater than or equal to a sixth preset threshold; or, the motion vector amplitude of image blocks in the third image region excluding edge image blocks in the second video frame being greater than 0, wherein the edge image blocks include at least one image block near the edge in the third image region; or, the motion vector amplitude of each pixel included in the third image region in the second video frame being greater than 0; or, among the pixels included in the third image region, the number of pixels with motion vector amplitudes greater than 0 in the second video frame being greater than or equal to a seventh preset threshold; or, the motion vector amplitude of pixels in the third image region excluding edge pixels in the second video frame being greater than 0. The edge pixels include at least one pixel near the edge in the third image region. The specific values ​​of the sixth and seventh preset thresholds are not limited in this application embodiment.

[0151] In some embodiments, the magnitude of the motion vector of the edge pixels of the third image region in the second video frame is 0. That is, the edge pixels of the third image region are stationary in both the first and second video frames. The edge pixels in the third image region include at least one pixel near the edge of the third image region. Alternatively, the magnitude of the motion vector of the edge image block in the third image region in the second video frame is 0. That is, the edge image block in the third image region is stationary in both the first and second video frames. The edge image block in the third image region includes at least one image block near the edge of the third image region.

[0152] 430, determine the first image region from at least one third image region.

[0153] The amplitude of the motion vector of each of the at least one first image region in the second video frame is less than or equal to a first preset threshold, and the degree of texture irregularity or motion irregularity of each first image region is greater than or equal to a second preset threshold. In other words, the first image region is an image region with micro-motion and irregular texture. The specific value of the second preset threshold is not limited in the embodiments of this application.

[0154] The degree of texture irregularity or motion irregularity in the first image region is described in step 220.

[0155] Optionally, the embodiments of this application do not limit the specific type of method for determining the degree of texture irregularity. For example, the method for determining the degree of texture irregularity includes: an error energy-based method, a deep learning-based method, etc. The error energy-based method identifies an image region in a video frame and another image region around that image region that is most similar to it, and calculates the error energy between the two image regions (i.e., the sum of the absolute values ​​of the differences between the pixel values ​​of corresponding pixels in the two image regions), thereby using the error energy as the degree of texture irregularity. The deep learning-based method uses a pre-trained deep neural network model to extract and analyze features from the image, thereby directly outputting the degree of texture irregularity of the image.

[0156] Optionally, the degree of motion irregularity of the first image region is determined based on the motion vector of each pixel in the first image region in the second video frame. For example, the degree of motion irregularity of the first image region is determined based on the irregularity of the distribution of the directions of the motion vectors of each pixel in the first image region in the second video frame. For instance, the image processing device determines the distribution of the directions of the motion vectors of the pixels in the first image region across P directional intervals, where P is a positive integer greater than 1, based on the direction of the motion vectors of each pixel in the first image region in the second video frame. These P directional intervals are obtained by dividing 360 degrees (°) into P equal parts. The image processing device determines the probability corresponding to each of the P directional intervals based on the distribution of the directions of the motion vectors of the pixels in the first image region across the P directional intervals, and determines a first entropy value based on the probabilities corresponding to the P directional intervals. This first entropy value represents the degree of motion irregularity of the first image region. A larger first entropy value indicates a larger degree of motion irregularity in the first image region; a smaller first entropy value indicates a smaller degree of motion irregularity in the first image region.

[0157] Optionally, step 430 can be replaced by the following steps: determining at least one fifth image region in the second video frame; determining at least one first image region based on the at least one fifth image region and at least one third image region. Wherein, the degree of texture irregularity or motion irregularity of each fifth image region within the at least one fifth image region is greater than or equal to a second preset threshold.

[0158] In some embodiments, determining at least one first image region based on at least one fifth image region and at least one third image region includes: determining the intersection of at least one fifth image region and at least one third image region, the intersection including the at least one first image region.

[0159] 440, Replace the pixels in the first image region of the second video frame with the pixels in the first image region of the reference frame to obtain the filtered second video frame.

[0160] The image processing device replaces the pixel values ​​of pixels in the first image region of the second video frame with the corresponding pixel values ​​in the first image region of the reference frame, thereby obtaining a filtered second video frame. The magnitude of the motion vector of the first image region in the filtered second video frame is 0. The magnitude of the motion vector of the first image region in the filtered second video frame is determined based on the first video frame and the filtered second video frame. The first video frame is described in step 220.

[0161] In some embodiments, the amplitude of the motion vector of the first image region in the filtered second video frame being 0 includes: the amplitude of the motion vector of each pixel in the first image region in the filtered second video frame being 0; or, the amplitude of the motion vector of each image block in the first image region in the filtered second video frame being 0.

[0162] In some embodiments, the reference frame is any one of the following: the first video frame in the first video frame set, the first reconstructed frame corresponding to the first video frame set, the background frame corresponding to the first video frame set, or the first video frame. Refer to the description in step 230 for the reference frame.

[0163] When the reference frame is the first video frame or the first reconstructed frame in the first video frame set, if the first video frame is the first video frame in the first video frame set, the image processing device does not need to filter the first video frame. When the reference frame is a background frame, if the first video frame is the first video frame in the first video frame set, the image processing device may or may not filter the first video frame.

[0164] When the second video frame includes multiple first image regions, the image processing device replaces the pixels in each first image region of the second video frame with the pixels of the corresponding first image region in the reference frame, thereby obtaining the filtered second video frame.

[0165] Optionally, the image processing device can perform processing on the second video frame. Figure 4 After the method in the first video frame set, continue to execute the next video frame in the first video frame set. Figure 4 Alternatively, the image processing device may perform step 440 on all video frames in the first video frame set after performing steps 410-430 on all video frames in the first video frame set.

[0166] Optionally, after step 440, the image processing device smooths the filtered video frame to avoid visual discontinuities between the edges of the replaced image area and the surrounding area due to pixel replacement.

[0167] exist Figure 4 In this method, the image processing device filters the image region with micro-motion and irregular texture in the second video frame to remove the micro-motion in the image region with micro-motion and irregular texture, so that the image region becomes a stationary region after filtering, thereby reducing the amount of data after video encoding and thus improving the compression rate of video encoding.

[0168] Figure 5 This is a schematic diagram of an image region with micro-motion and irregular texture obtained according to the method in the embodiments of this application. Figure 5 (a) in the image represents a video frame. Figure 5 (b) in the text is a pair Figure 5 The image obtained by identifying the micro-motion and irregular texture region in (a) of the image. Figure 5 The white area in (b) is Figure 5 The mask for the image region with micro-motion and irregular texture in (a) of the image.

[0169] For example, the bitrate savings when processing five different videos according to the method in the embodiments of this application before encoding are shown in Table 1 below.

[0170] Table 1

[0171]

[0172] As shown in Table 1, in Video 1, the average proportion of image regions with slight motion and irregular texture in the multiple video frames included in Video 1 is 7.3%. After processing the multiple video frames included in Video 1 according to the method in this embodiment, the processed Video 1 is obtained. The bitstream obtained by encoding the processed Video 1 saves 1.41% of the bitrate compared to the bitstream obtained by encoding the original Video 1. It can be seen from Table 1 that by removing the slight motion in the image regions with slight motion and irregular texture in the video, an average of 21.2% of the bitrate can be saved without affecting the visual effect. Furthermore, it can also be seen from Table 1 that the bitrate saved is greater when the area of ​​the image regions with slight motion and irregular texture in the video frame is larger.

[0173] Figure 6 This is a schematic structural diagram of the image processing apparatus provided in the embodiments of this application. Figure 6 The image processing device 600 includes an acquisition module 610 and a processing module 620. Figure 6The image processing device 600 in the middle can be used to perform Figures 2 to 4 The method in the middle. Figure 6 The image processing device 600 in the middle can be applied to Figure 1 Any one of the devices in it.

[0174] The image processing device 600 is used to perform Figure 2 In the method described above, the acquisition module 610 is used to obtain the first video frame set. The acquisition module 610 is used to perform... Figure 2 Step 210. Processing module 620 is used to: determine a first image region in the second video frame based on the first video frame and the second video frame; filter the first image region in the second video frame to obtain a filtered second video frame; and obtain a set of second video frames based on the filtered second video frames. Processing module 620 is used to perform... Figure 2 Steps 220-240 in the text.

[0175] The image processing device 600 is used to perform Figure 3 In the method described above, the acquisition module 610 is used to obtain the third video frame set. The acquisition module 610 is used to perform... Figure 3 Step 310. Processing module 620 is used to: perform scene switching detection on the third video frame set to obtain a fourth video frame set; analyze the background in the fourth video frame set to obtain a first video frame set; filter image regions in at least one video frame in the first video frame set to obtain a second video frame set; replace the first video frame set in the third video frame set with the second video frame set to obtain a fifth video frame set. Processing module 620 is used to execute... Figure 3 Steps 320-350 in the process.

[0176] The image processing device 600 is used to perform Figure 4 In the method described above, the processing module 620 is configured to: determine at least one second image region in the second video frame based on a first video frame and a second video frame; determine at least one third image region based on the magnitude of the motion vector of each second image region in the second video frame; determine a first image region from the at least one third image region; and replace pixels in the first image region in the second video frame with pixels in the first image region in a reference frame to obtain a filtered second video frame. The processing module 620 is configured to perform... Figure 4 Steps 410-440 in the text.

[0177] Both the acquisition module 610 and the processing module 620 can be implemented in software or in hardware. For example, the implementation of the processing module 620 will be described below. Similarly, the implementation of the acquisition module 610 can be referenced from the implementation of the processing module 620.

[0178] As an example of a software functional unit, processing module 620 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, processing module 620 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0179] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same VPC or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0180] As an example of a hardware functional unit, the processing module 620 may include at least one computing device, such as a server. Alternatively, the processing module 620 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0181] The processing module 620 includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the processing module 620 includes multiple computing devices that can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the processing module 620 includes multiple computing devices that can be distributed within the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0182] Therefore, the modules of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] It should be noted that the above embodiments of the device, when executing the above methods, are only illustrative examples of the division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. For example, the acquisition module 610 can be used to execute any step in the above methods, and the processing module 620 can be used to execute any step in the above methods. The steps implemented by the acquisition module 610 and the processing module 620 can be specified as needed, and all the functions of the above device can be realized by implementing different steps in the above methods through the acquisition module 610 and the processing module 620 respectively.

[0184] Furthermore, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments above, which will not be repeated here.

[0185] The method provided in this application can be executed by a computing device, which can also be referred to as a computer system. It includes a hardware layer, an operating system layer running on top of the hardware layer, and an application layer running on the operating system layer. The hardware layer includes hardware such as processing units, memory, and memory control units; the functions and structure of this hardware will be described in detail later. The operating system can be any one or more computer operating systems that implement business processing through processes, such as Linux, Unix, Android, iOS, or Windows. The application layer includes applications such as browsers, address books, word processing software, and instant messaging software. Optionally, the computer system can be a handheld device such as a smartphone, or a terminal device such as a personal computer; this application does not particularly limit this, as long as the method provided in this application can be used. The executing entity of the method provided in this application can be a computing device, or a functional module within the computing device capable of calling and executing programs.

[0186] Figure 7 This is a schematic structural block diagram of a computing device 700 provided in an embodiment of this application. The computing device 700 may be a server, a computer, or other device with computing capabilities. Figure 7 The computing device 700 shown includes at least one processor 710 and a memory 720.

[0187] It should be understood that this application does not limit the number of processors and memories in the computing device 700.

[0188] The processor 710 executes instructions in the memory 720, causing the computing device 700 to implement the method provided in this application. Alternatively, the processor 710 executes instructions in the memory 720, causing the computing device 700 to implement the various functional modules provided in this application, thereby implementing the method provided in this application.

[0189] Optionally, the computing device 700 also includes a communication interface 730. The communication interface 730 uses a transceiver module, such as, but not limited to, a network interface card or a transceiver, to enable communication between the computing device 700 and other devices or communication networks.

[0190] Optionally, the computing device 700 also includes a system bus 740, wherein the processor 710, memory 720, and communication interface 730 are respectively connected to the system bus 740. The processor 710 can access the memory 720 through the system bus 740; for example, the processor 710 can perform data read / write or code execution in the memory 720 through the system bus 740. The system bus 740 is a peripheral component interconnect express (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus 740 is divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0191] In one possible implementation, the processor 710 primarily functions to interpret the instructions (or code) of a computer program and process data within the computer software. The instructions of the computer program and the data within the computer software can be stored in the memory 720 or the cache of the processor 710.

[0192] Optionally, the processor 710 may be an integrated circuit chip with signal processing capabilities. By way of example and not limitation, the processor 710 may be a general-purpose processor, a digital signal processor (DSP), an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. Among these, a general-purpose processor is a microprocessor, etc. For example, the processor 710 may be a central processing unit (CPU).

[0193] The memory 720 provides runtime space for processes in the computing device 700. For example, the memory 720 stores the computer program (specifically, the program code) used to generate the process. After the computer program is run by the processor to generate a process, the processor allocates corresponding storage space for the process in the memory 720. Furthermore, the aforementioned storage space further includes text segments, initialized data segments, bit initialized data segments, stack segments, heap segments, etc. The memory 720 stores data generated during the process's execution, such as intermediate data or process data, in the aforementioned process-specific storage space.

[0194] Optionally, the memory, also known as RAM, is used to temporarily store the data processed by the processor 710, as well as data exchanged with external storage devices such as hard disks. As long as the computer is running, the processor 710 will load the data required for processing into RAM for computation, and then transfer the result back out after the computation is complete.

[0195] By way of example and not limitation, memory 720 may be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile storage medium may be, for example, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory is random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct rambus DRAM (DRDRAM). It should be noted that the memory 720 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0196] The structures of the computing device 700 listed above are merely illustrative and are not limited thereto. The computing device 700 in this application includes various hardware components in existing computer systems. For example, the computing device 700 also includes other memories besides the memory 720, such as disk storage. Those skilled in the art should understand that the computing device 700 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the computing device 700 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the computing device 700 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 7All the devices shown.

[0197] This application also provides a computing device cluster. The computing device cluster includes at least one computing device. The computing device may be a server. In some embodiments, the computing device may also be a desktop computer, a laptop computer, or a smartphone, or other terminal device.

[0198] like Figure 8 As shown, the computing device cluster includes at least one computing device 700. The memory 720 of one or more computing devices 700 in the computing device cluster may store the same instructions for performing the methods described above.

[0199] In some possible implementations, the memory 720 of one or more computing devices 700 in the computing device cluster may also each store a portion of the instructions for executing the above-described method. In other words, a combination of one or more computing devices 700 can jointly execute the instructions of the above-described method.

[0200] It should be noted that the memories 720 in different computing devices 700 within the computing device cluster can store different instructions, each used to execute a portion of the functions of the aforementioned apparatus. That is, the instructions stored in the memories 720 of different computing devices 700 can implement the functions of one or more modules within the aforementioned apparatus.

[0201] In some possible implementations, one or more computing devices in a computing device cluster can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 One possible implementation is shown. For example... Figure 9 As shown, the two computing devices 700A and 800B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0202] It should be understood that Figure 9 The functions of the computing device 700A shown can also be performed by multiple computing devices 700. Similarly, the functions of the computing device 700B can also be performed by multiple computing devices 700.

[0203] In this embodiment of the application, a computer program product containing instructions is also provided. The computer program product may be software or a program product containing instructions that can run on a computing device cluster or be stored on any available medium. When run by the computing device cluster, it causes the computing device cluster to perform the methods provided above, or causes the computing device cluster to perform the functions of the apparatus provided above.

[0204] This application embodiment also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that, when executed by a cluster of computing devices, cause the cluster of computing devices to perform the method provided above.

[0205] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0206] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0207] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0208] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0209] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0210] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0211] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Obtain a first set of video frames, which includes multiple video frames, and the background of the first set of video frames is static. Based on the first video frame and the second video frame, a first image region in the second video frame is determined. The second video frame belongs to the first video frame set. The magnitude of the motion vector of the first image region in the second video frame is less than or equal to a first preset threshold. The first video frame is a frame preceding the second video frame in the first video frame set. Alternatively, the first video frame is a frame obtained by filtering a frame preceding the second video frame in the first video frame set. The first image region in the second video frame is filtered to obtain a filtered second video frame, wherein the magnitude of the motion vector of the first image region in the filtered second video frame is 0. Based on the filtered second video frame, a second video frame set is obtained; The magnitude of the motion vector of the first image region in the second video frame is determined based on the first video frame and the second video frame, and the magnitude of the motion vector of the first image region in the filtered second video frame is determined based on the first video frame and the filtered second video frame.

2. The method according to claim 1, characterized in that, The magnitude of the motion vector of the first image region in the second video frame is greater than 0.

3. The method according to claim 1 or 2, characterized in that, The degree of texture irregularity or motion irregularity in the first image region is greater than or equal to the second preset threshold.

4. The method according to any one of claims 1 to 3, characterized in that, The first video frame and the second video frame are adjacent video frames in the first video frame set, or the first video frame is a frame obtained by filtering a frame in the first video frame set that is adjacent to the second video frame.

5. The method according to any one of claims 1 to 4, characterized in that, The step of determining the first image region in the second video frame based on the first video frame and the second video frame includes: Based on the first video frame and the second video frame, at least one second image region in the second video frame is determined, and each second image region in the at least one second image region is similar to an image region in the first video frame; The first image region is determined based on the magnitude of the motion vector of each of the at least one second image regions in the second video frame, wherein the magnitude of the motion vector of each second image region in the second video frame is determined based on the first video frame and the second video frame.

6. The method according to claim 5, characterized in that, Determining the first image region based on the magnitude of the motion vector of each of the at least one second image region in the second video frame includes: At least one third image region is determined based on the magnitude of the motion vector of each of the at least one second image regions in the second video frame, wherein the magnitude of the motion vector of each of the at least one third image regions in the second video frame is less than or equal to the first preset threshold. The first image region is determined from the at least one third image region, wherein the degree of texture irregularity or motion irregularity of the first image region is greater than or equal to a second preset threshold.

7. The method according to any one of claims 1 to 6, characterized in that, The step of filtering the first image region in the second video frame to obtain the filtered second video frame includes: The pixels in the first image region of the second video frame are replaced with the pixels in the first image region of the reference frame to obtain the filtered second video frame.

8. The method according to claim 7, characterized in that, The reference frame is any one of the following: the first video frame in the first video frame set, the first reconstructed frame corresponding to the first video frame set, the background frame corresponding to the first video frame set, or the first video frame, wherein the first video frame is a frame obtained by filtering a frame preceding the second video frame in the first video frame set.

9. The method according to claim 8, characterized in that, When the reference frame is the first video frame in the first video frame set or the first reconstructed frame, and the first video frame is the first video frame in the first video frame set, the second video frame set includes the first video frame and the filtered second video frame.

10. The method according to any one of claims 1 to 9, characterized in that, Obtaining the first set of video frames includes: Obtain a third video frame set, which includes multiple video frames, wherein at least two video frames in the third video frame set have different scenes, and / or the background in the third video frame set is in motion; Scene switching detection is performed on the third video frame set to obtain a fourth video frame set, which is a subset of the third video frame set. The fourth video frame set includes multiple video frames, and each video frame in the fourth video frame set has the same scene. The background in the fourth video frame set is analyzed to obtain the first video frame set, which is a subset of the fourth video frame set.

11. The method according to claim 10, characterized in that, The method further includes: The first video frame set in the third video frame set is replaced with the second video frame set to obtain the fifth video frame set. The fifth set of video frames is encoded to output the first bitstream.

12. An image processing apparatus, characterized in that, include: The acquisition module is used to acquire a first video frame set, which includes multiple video frames, and the background of the first video frame set is static. The processing module is used to determine a first image region in the second video frame based on the first video frame and the second video frame. The second video frame belongs to the first video frame set. The amplitude of the motion vector of the first image region in the second video frame is less than or equal to a first preset threshold. The first video frame is a frame preceding the second video frame in the first video frame set. Alternatively, the first video frame is a frame obtained by filtering the frame preceding the second video frame in the first video frame set. The processing module is further configured to filter the first image region in the second video frame to obtain a filtered second video frame, wherein the magnitude of the motion vector of the first image region in the filtered second video frame is 0. The processing module is further configured to obtain a second video frame set based on the filtered second video frame; Wherein, the motion vector of the first image region in the second video frame is determined based on the first video frame and the second video frame, and the motion vector of the first image region in the filtered second video frame is determined based on the first video frame and the filtered second video frame.

13. The apparatus according to claim 12, characterized in that, The magnitude of the motion vector of the first image region in the second video frame is greater than 0.

14. The apparatus according to claim 12 or 13, characterized in that, The degree of texture irregularity or motion irregularity in the first image region is greater than or equal to the second preset threshold.

15. The apparatus according to any one of claims 12 to 14, characterized in that, The first video frame and the second video frame are adjacent video frames in the first video frame set, or the first video frame is a frame obtained by filtering a frame in the first video frame set that is adjacent to the second video frame.

16. The apparatus according to any one of claims 12 to 15, characterized in that, The processing module is specifically used for: Based on the first video frame and the second video frame, at least one second image region in the second video frame is determined, and each second image region in the at least one second image region is similar to an image region in the first video frame; The first image region is determined based on the magnitude of the motion vector of each of the at least one second image regions in the second video frame, wherein the magnitude of the motion vector of each second image region in the second video frame is determined based on the first video frame and the second video frame.

17. The apparatus according to claim 16, characterized in that, The processing module is specifically used for: At least one third image region is determined based on the magnitude of the motion vector of each of the at least one second image regions in the second video frame, wherein the magnitude of the motion vector of each of the at least one third image regions in the second video frame is less than or equal to the first preset threshold. The first image region is determined from the at least one third image region, wherein the degree of texture irregularity or motion irregularity of the first image region is greater than or equal to a second preset threshold.

18. The apparatus according to any one of claims 12 to 17, characterized in that, The processing module is specifically used for: The pixels in the first image region of the second video frame are replaced with the pixels in the first image region of the reference frame to obtain the filtered second video frame.

19. The apparatus according to claim 18, characterized in that, The reference frame is any one of the following: the first video frame in the first video frame set, the first reconstructed frame corresponding to the first video frame set, the background frame corresponding to the first video frame set, or the first video frame, wherein the first video frame is a frame obtained by filtering a frame preceding the second video frame in the first video frame set.

20. The apparatus according to claim 19, characterized in that, When the reference frame is the first video frame in the first video frame set or the first reconstructed frame, and the first video frame is the first video frame in the first video frame set, the second video frame set includes the first video frame and the filtered second video frame.

21. The apparatus according to any one of claims 12 to 20, characterized in that, The acquisition module is specifically used for: Obtain a third video frame set, which includes multiple video frames, wherein at least two video frames in the third video frame set have different scenes, and / or the background in the third video frame set is in motion; Scene switching detection is performed on the third video frame set to obtain a fourth video frame set, which is a subset of the third video frame set. The fourth video frame set includes multiple video frames, and each video frame in the fourth video frame set has the same scene. The background in the fourth video frame set is analyzed to obtain the first video frame set, which is a subset of the fourth video frame set.

22. The apparatus according to claim 21, characterized in that, The processing module is further configured to: The first video frame set in the third video frame set is replaced with the second video frame set to obtain the fifth video frame set. The fifth set of video frames is encoded to output the first bitstream.

23. A computing device cluster, characterized in that, It includes at least one computing device, each computing device including a processor and memory; The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device to cause the cluster of computing devices to perform the method as described in any one of claims 1 to 11.

24. A chip system, characterized in that, include: A logic circuit for coupling with an input / output interface, through which data is transmitted to perform the method as described in any one of claims 1 to 11.

25. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device cluster, the computing device cluster causes the computing device cluster to perform the method as described in any one of claims 1 to 11.

26. A computer-readable storage medium, characterized in that, Includes computer program instructions, which, when executed by a cluster of computing devices, perform the method as described in any one of claims 1 to 11.

Images