Image fusion method, video image processing device and computer readable storage medium

Abstract

According to the image fusion method, video image processing device and computer-readable storage medium provided by the embodiments of the present invention, a current frame image is obtained from a video file; image segmentation processing is performed on the current frame image to obtain a current frame segmented image; the current frame segmented image is cached in a video segmentation queue, and fused with the historical frame segmented images in the video segmentation queue to obtain a current frame fused image; in certain implementation processes, the current frame segmented image is fused with the historical frame segmented images to obtain a fused current frame image, thereby reducing holes and edge jitters in the current frame image, avoiding jitter problems and delays caused by inconsistencies between previous and next frames of the video image, and affecting the quality of image segmentation, thereby improving the stability and fluency of the video.

Description

Technical Field

[0001] The embodiments of the present invention relate to, but are not limited to, the field of image processing technology, and specifically, relate to, but are not limited to, an image fusion method, a video image processing device, and a computer-readable storage medium. Background Art

[0002] With the popularization and development of various video software applications, various video processing algorithms are widely used in the processing of various video images. Among them, video segmentation technology, as a basic video processing method, has also been widely used. Video segmentation technology usually uses the image segmentation method, which often has holes and edge jitters. The existence of hole flickering and edge jitter seriously affects the quality of image segmentation, resulting in poor stability and fluency of the video. Summary of the invention

[0003] The image fusion method, video image processing device and computer-readable storage medium provided by the embodiments of the present invention mainly solve the technical problem that holes and edge jitters exist during video image segmentation, which affect the quality of image segmentation and lead to poor stability and fluency of the video.

[0004] In order to solve the above technical problems, an embodiment of the present invention provides an image fusion method, including:

[0005] The current frame image is obtained from the video file; the current frame image is subjected to image segmentation processing to obtain the current frame segmentation image; the current frame segmentation image is cached in the video segmentation queue, and is fused with the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image.

[0006] An embodiment of the present invention also provides a video image processing device, which includes a processor, a memory and a communication bus; the communication bus is used to realize the connection and communication between the processor and the memory; the processor is used to execute one or more computer programs stored in the memory to implement the steps of the image fusion method as described above.

[0007] An embodiment of the present invention further provides a computer storage medium, wherein the computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps of the image fusion method as described above.

[0008] According to the image fusion method, video image processing device and computer-readable storage medium provided by the embodiments of the present invention, a current frame image is obtained from a video file; image segmentation processing is performed on the current frame image to obtain a current frame segmented image; the current frame segmented image is cached in a video segmentation queue, and fused with the historical frame segmented images in the video segmentation queue to obtain a current frame fused image; in certain implementation processes, the current frame segmented image is fused with the historical frame segmented images to obtain a fused current frame image, thereby reducing holes and edge jitters in the current frame image, avoiding jitter problems and delays caused by inconsistencies between previous and next frames of the video image, and affecting the quality of image segmentation, thereby improving the stability and fluency of the video.

[0009] Other features and corresponding beneficial effects of the present invention are described in the latter part of the specification, and it should be understood that at least part of the beneficial effects become obvious from the description in the specification of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 A schematic diagram of a basic flow chart of an image fusion method according to a first embodiment of the present invention;

[0011] Figure 2 A schematic diagram of the basic flow of the image fusion method according to the second embodiment of the present invention;

[0012] Figure 3 This is a schematic diagram of the basic structure of an image fusion terminal according to Embodiment 3 of the present invention;

[0013] Figure 4 FIG. 4 is a schematic diagram of the basic structure of a video image processing device according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0014] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the following is a further detailed description of the embodiments of the present invention through specific implementation methods combined with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0015] Embodiment 1:

[0016] In order to solve the problems of holes and edge jitters in video image segmentation, which affect the quality of image segmentation and lead to poor stability and fluency of the video.

[0017] See also Figure 1 , Figure 1 The following is a basic flow chart of the image fusion method, which includes but is not limited to:

[0018] S101, extracting the current frame image from the video file;

[0019] In some embodiments, extracting the current frame image from the video file includes: extracting the current frame image from the currently shot video file; that is, the current frame image may be the video file currently shot and recorded by the terminal, and during the video shooting process, the terminal may obtain the current frame image of the currently shot video file in real time; in some embodiments, the current frame image may also be the current frame of the video file being played. Among them, the video file may be an offline video file stored in a memory on the terminal, or an online video file extracted by the terminal from a server. For example, the video file may be an online video file extracted by the terminal (such as a mobile phone) from a server. At this time, when the user requests to play a specified video file through the video software on the mobile phone, the terminal may transmit the video playback request to the server through the network, and the server may return the playback address of the specified video file, etc., so that the specified video file can be played on the mobile phone. During the playback of the video file, the terminal may extract the current frame image of the current specified video file in real time. .

[0020] S102, performing image segmentation processing on the current frame image to obtain a current frame segmented image;

[0021] In some embodiments, performing image segmentation processing on the current frame image to obtain the current frame segmentation image includes: performing image segmentation processing on the current frame image using a binary segmentation processing method to obtain the current frame segmentation image. In one embodiment, the current frame image obtained can be subjected to binary segmentation processing, and the image binary segmentation processing is to read the gray value of the point on the current frame image as 0 or 255, that is, the entire current frame image presents an obvious black and white effect, that is, the gray image of 256 brightness levels is selected through an appropriate threshold to obtain a binary image that can still reflect the overall and local features of the current frame image, and then the pixel areas belonging to different objects in the current frame image are separated, and the semantics of each area is classified; for example, the foreground and the background are separated, and the dog area and the cat area are separated from the background, and all objects in the picture are pointed out to their respective categories.

[0022] It should be understood that, in some embodiments, a convolutional neural network can also be used to perform image segmentation processing on the current frame image. For example, in the embodiment of the present application, the convolutional neural network can adopt a traditional CNN (Convolutional Neural Networks), that is, in order to classify a pixel, an image block around the pixel is used as the input of the CNN for training and prediction. In one embodiment, the process of image segmentation processing on the current frame image using a convolutional neural network can be, first, the convolutional neural network is used to identify each object in the current image, and the identified objects are classified, and finally, the edge of the target object is detected and pooled, and a series of operations are performed to separate the pixel areas belonging to different objects in the current frame image, and the semantics of each area is classified, so as to distinguish the target object from other objects in the current frame video image, and obtain the current frame segmented image. In other embodiments, the convolutional neural network in the embodiment of the present application can also adopt FCN (Fully Convolutional Networks, Fully Convolutional Neural Networks).

[0023] It should be understood that video image segmentation methods such as time domain-based video object segmentation, motion-based video object segmentation, and interactive video object segmentation may also be used to perform image segmentation processing on the previous frame image.

[0024] S103: Cache the current frame segmentation image into the video segmentation queue, and perform fusion processing with the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image.

[0025] It should be understood that the video segmentation queue is used to store historical frame segmentation images and current frame segmentation images, wherein the historical frame images stored in the video segmentation queue have an upper limit value, and when the stored historical frame images exceed the upper limit, the historical frame image farthest from the current frame image will be automatically overwritten or deleted, wherein the farthest from the current frame image is also the longest time from the current frame image; in some embodiments, the image fusion method further includes: determining whether the number of historical frame images in the video segmentation queue exceeds the storage upper limit; when the historical frame images in the video segmentation queue exceed the storage upper limit, deleting the historical frame image with the longest time from the current frame image. For example, when the upper limit of the number of historical frame images stored in the video segmentation queue is 5, when the current frame image is cached in the video segmentation queue, when it is determined that the number of historical frame images stored in the video segmentation queue is 6, the historical frame image with the longest time from the current frame image is automatically deleted, so that the historical segmentation images stored in the video segmentation queue are several frames of images that are relatively close to the current frame image in time.

[0026] In some embodiments, before caching the current frame segmentation image into the video segmentation queue, it includes: obtaining the previous frame image, calculating the change between the current frame image and the previous frame image, so as to determine whether the change between the current frame image and the previous frame image exceeds the first threshold; caching the current frame segmentation image into the video segmentation queue includes: when the change between the current frame image and the previous frame image does not exceed the first preset threshold, directly caching the current frame segmentation image into the video segmentation queue, and then the change between the two adjacent frames of the video segmentation queue does not exceed the first preset threshold. In some embodiments, when the change between the current frame image and the previous frame image exceeds the first preset threshold, the video segmentation queue is cleared, and the current frame segmentation image is cached into the cleared video segmentation queue; it should be understood that the so-called image fusion is to fuse the information of two or more images into one image, so that the fused image contains more information and can be more convenient for people to observe or computer processing. The goal of image fusion is to reduce the uncertainty and redundancy of the output on the basis of maximizing the merging of relevant information under the actual application goal. Therefore, when the change between the current frame image and the previous frame image exceeds the first preset threshold, the video segmentation queue is cleared and the current frame segmentation image is cached in the video segmentation queue. At this time, there is no historical frame segmentation image in the video segmentation queue. At this time, the image fusion method only needs to add the current frame image to the video segmentation queue.

[0027] In some embodiments, determining whether the change between the current frame image and the previous frame image exceeds a first preset threshold includes: dividing the current frame image into blocks to obtain multiple current frame block images, dividing the previous frame image into blocks to obtain multiple previous frame block images that correspond one to one to each current frame block image; comparing each current frame block image with the corresponding previous frame block image to determine whether each current frame block image has changed; when the number of changes in the current frame block image reaches a second preset threshold, determining that the change between the current frame image and the previous frame image exceeds the first preset threshold. For example, the current frame image is divided into blocks using a preset block parameter to obtain a current frame block image with a preset number of blocks. At the same time, the previous frame image is divided into blocks using a preset block parameter to obtain a previous frame block image with a preset number of blocks. Then, the current frame block image and the corresponding previous frame block image are differentially calculated, that is, the current frame block image and the corresponding previous frame block image are differentially calculated for each pixel point. When the pixel difference cumulative value of each block of the current frame image and the previous frame image exceeds the pixel difference change threshold, it is considered that the block image of the current frame image has changed. When the ratio of the number of all changed block images reaches the second preset threshold or a specific ratio threshold, it is determined that the change between the current frame image and the previous frame image exceeds the first preset threshold, and it is considered that the current frame image has a drastic change compared with the previous frame image. It should be understood that the preset block parameters can be adjusted according to user feedback to achieve the most accurate effect of determining whether the change between the current frame image and the previous frame image exceeds the first preset threshold.

[0028] It should be understood that determining whether the change between the current frame image and the previous frame image exceeds a threshold may also include: directly performing feature comparison between the current frame image and the previous frame image, or performing differential calculation, and directly performing comparison without image segmentation to determine whether the change between the current frame image and the previous frame image exceeds a first preset threshold.

[0029] In some embodiments, fusing the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image includes: obtaining the fusion weights of each historical frame segmentation image according to the chronological order; fusing the current frame image according to the fusion weights of each historical frame segmentation image to obtain the current frame fusion image, the historical frame images have different fusion weights corresponding to different time lengths from the current frame image; in some implementations, different fusion weights are assigned to each historical frame image according to the time length from the current frame image, the closer the historical frame image is to the current frame image, the higher the fusion weight, and the farther the historical frame image is from the current frame image, the lower the fusion weight.

[0030] In some embodiments, performing fusion processing with historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image further includes: directly performing OR operations according to corresponding pixel points based on each historical frame segmentation image in the video segmentation queue.

[0031] In some embodiments, after caching the current frame segmentation image in the video segmentation queue and fusing it with the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image, the process may further include: optimizing the current frame fusion image and outputting the optimized current frame fusion image. For example, performing one or more optimizations such as Gaussian blurring / smoothing and sharpening on the current frame fusion image and outputting the optimized current frame fusion image.

[0032] The image fusion method provided by the embodiment of the present invention obtains the current frame image from the video file; performs image segmentation processing on the current frame image to obtain the current frame segmentation image; caches the current frame segmentation image in a video segmentation queue, and performs fusion processing with the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image; obtains the fused current frame image by fusing the current frame segmentation image with the historical frame segmentation images, thereby avoiding the problem of holes and edge jitters caused by inconsistency between previous and next frame images, which affects the quality of image segmentation, and improves the stability and fluency of the video.

[0033] Embodiment 2:

[0034] In order to better understand the present invention, this embodiment provides a more specific example to illustrate the present invention. Figure 2 , the image fusion method includes but is not limited to:

[0035] S201, extracting a current frame image from a currently shot video file;

[0036] In some embodiments, during the video shooting process, the terminal extracts the current frame image of the currently shot video file in real time.

[0037] S202, performing image segmentation processing on the current frame image to obtain a current frame segmented image;

[0038] In some embodiments, a binary segmentation processing method is used to perform image segmentation processing on the current frame image to obtain a current frame segmentation image; specifically, the obtained current frame image is subjected to a binary segmentation processing, and the image binary segmentation processing is to read the grayscale value of the point on the current frame image as 0 or 255, that is, the entire current frame image presents an obvious black and white effect, that is, the grayscale image of 256 brightness levels is obtained through appropriate threshold selection to obtain a binary image that can still reflect the overall and local features of the current frame image, and then the pixel areas belonging to different objects in the current frame image are separated, and the semantics of each area is classified; for example, the foreground and the background are separated, and the dog area and the cat area are separated from the background, and all objects in the picture are pointed out to their respective categories.

[0039] S203, obtaining the previous frame image, and determining whether the change between the current frame image and the previous frame image exceeds a first preset threshold, if not, proceed to S204, otherwise, proceed to S205.

[0040] In some embodiments, before caching the current frame segmented image into the video segmentation queue, the process includes: obtaining the previous frame image, and determining whether the change between the current frame image and the previous frame image exceeds a first preset threshold. Determining whether the change between the current frame image and the previous frame image exceeds the first preset threshold includes: dividing the current frame image into blocks to obtain a plurality of current frame block images, dividing the previous frame image into blocks to obtain a plurality of previous frame block images corresponding to each current frame block image; comparing each current frame block image with the corresponding previous frame block image to determine whether each current frame block image has changed; when the number of changes in the current frame block image reaches a second preset threshold, determining that the change between the current frame image and the previous frame image exceeds the first preset threshold. For example, the current frame image is divided into blocks using a preset block parameter to obtain a current frame block image with a preset number of blocks. At the same time, the previous frame image is divided into blocks using a preset block parameter to obtain a previous frame block image with a preset number of blocks. Then, the current frame block image and the corresponding previous frame block image are differentially calculated, that is, the current frame block image and the corresponding previous frame block image are differentially calculated for each pixel point. When the pixel difference cumulative value of each block of the current frame image and the previous frame image exceeds the pixel difference change threshold, it is considered that the block image of the current frame image has changed. When the ratio of the number of all changed block images reaches the second preset threshold or a specific ratio threshold, it is determined that the change between the current frame image and the previous frame image exceeds the first preset threshold, and it is considered that the current frame image has a drastic change compared with the previous frame image. It should be understood that the preset block parameters can be adjusted according to user feedback to achieve the most accurate effect of determining whether the change between the current frame image and the previous frame image exceeds the threshold.

[0041] It should be understood that determining whether the change between the current frame image and the previous frame image exceeds the first preset threshold may also include: directly comparing the features of the current frame image with the previous frame image, or performing differential calculation, and directly comparing the features without image segmentation to determine whether the change between the current frame image and the previous frame image exceeds the first preset threshold.

[0042] S204: Cache the current frame segmentation image into a video segmentation queue, and perform fusion processing with the historical frame segmentation images in the video segmentation queue to obtain a current frame fusion image.

[0043] In some embodiments, caching the current frame segmentation image into the video segmentation queue, and fusing it with the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image includes: obtaining the fusion weights of each historical frame segmentation image, and fusing according to the fusion weights; for example, assigning different fusion weights to each historical frame image according to the time length from the current frame image, the closer the historical frame image is to the current frame image, the higher the fusion weight, and the farther the historical frame image is from the current frame image, the lower the fusion weight. Alternatively, the OR operation can be performed directly according to the corresponding pixel points of each historical frame segmentation image in the video segmentation queue.

[0044] In some embodiments, after caching the current frame segmentation image into the video segmentation queue and fusing it with the historical frame segmentation images in the video segmentation queue to obtain the current frame fusion image, the process further includes: optimizing the current frame fusion image and outputting the optimized current frame fusion image. For example, Gaussian blurring, sharpening, etc. are performed on the current frame fusion image, and the optimized current frame fusion image is output.

[0045] S205, after the video segmentation queue is cleared, the current frame segmentation image is cached in the video segmentation queue;

[0046] In some embodiments, the current frame segmented image is directly optimized and the optimized current frame image is output, for example, the current frame image is optimized by Gaussian blurring, sharpening, etc., and the optimized current frame image is output.

[0047] The image fusion method provided by the embodiment of the present invention obtains a current frame image from a currently shot video file, performs image segmentation processing on the current frame image to obtain a current frame segmented image, obtains a previous frame image, and when a change between the current frame image and the previous frame image does not exceed a threshold, caches the current frame segmented image into a video segmentation queue, and performs fusion processing with historical frame segmented images in the video segmentation queue to obtain a current frame fused image; by fusing the current frame segmented image with the historical frame segmented images to obtain a fused current frame image, the problem of holes and edge jitters caused by inconsistencies between previous and next frame images, which affects the quality of image segmentation, is avoided, thereby improving the stability and fluency of the video.

[0048] Embodiment three:

[0049] This embodiment also provides an image fusion terminal. Figure 3 As shown, the image fusion terminal includes:

[0050] An acquisition module is used to obtain the current frame image from the video file;

[0051] A segmentation module, used for performing image segmentation processing on the current frame image to obtain a current frame segmented image;

[0052] A fusion module, used for caching the current frame segmentation image into a video segmentation queue, and fusing it with the historical frame segmentation images in the video segmentation queue to obtain a current frame fusion image;

[0053] It should be understood that the image fusion terminal also includes a judgment module for judging whether the change between the current frame image and the previous frame image exceeds a threshold; the combination of the various modules of the image fusion terminal can achieve the same effect as the various steps in the image fusion method in the above-mentioned embodiment one and / or embodiment two.

[0054] This embodiment also provides a video image processing device, see Figure 4 As shown, it includes a processor 401, a memory 402 and a communication bus 403, wherein:

[0055] The communication bus 403 is used to realize the connection and communication between the processor 401 and the memory 402;

[0056] The processor 401 is used to execute one or more computer programs stored in the memory 402 to implement at least one step in the image fusion method in the above-mentioned embodiment 1 and embodiment 2.

[0057] The present embodiment also provides a computer-readable storage medium, which includes volatile or non-volatile, removable or non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, computer program modules or other data). Computer-readable storage media include, but are not limited to, RAM (Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable read only memory), flash memory or other memory technology, CD-ROM (Compact Disc Read-Only Memory), digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer.

[0058] The computer-readable storage medium in this embodiment can be used to store one or more computer programs, and the one or more computer programs stored therein can be executed by a processor to implement at least one step of the image fusion method in the above-mentioned embodiment 1 and embodiment 2.

[0059] It can be seen that those skilled in the art should understand that all or some of the steps, systems, and functional modules / units in the above disclosed methods can be implemented as software (which can be implemented with computer program code executable by a computing device), firmware, hardware, and appropriate combinations thereof. In hardware implementations, the division between the functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component can have multiple functions, or a function or step can be performed by several physical components in cooperation. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor, or a microprocessor, or implemented as hardware, or implemented as an integrated circuit, such as an application-specific integrated circuit.

[0060] In addition, it is well known to those skilled in the art that communication media generally contain computer readable instructions, data structures, computer program modules or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery media. Therefore, the present invention is not limited to any specific hardware and software combination.

[0061] The above contents are further detailed descriptions of the embodiments of the present invention in combination with specific implementation methods, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. An image fusion method, comprising: Extract the current frame image from the video file; Performing image segmentation processing on the current frame image to obtain a current frame segmented image; Acquire a previous frame image of the video file, divide the current frame image into blocks to obtain a plurality of current frame block images, divide the previous frame image into blocks to obtain a plurality of previous frame block images corresponding to each of the current frame block images; Comparing each of the current frame block images with the corresponding previous frame block images to determine whether each of the current frame block images has changed; When the number of changes in the current frame segmented image reaches a second preset threshold, determining that the change between the current frame image and the previous frame image exceeds a first preset threshold; caching the current frame segmented image into a video segmentation queue, and fusing it with the historical frame segmented images in the video segmentation queue to obtain a current frame fused image; The step of caching the current frame segmented image into a video segmentation queue comprises: When the difference between the current frame image and the previous frame image does not exceed a first preset threshold, the current frame segmented image is cached in the video segmentation queue.

2. The image fusion method according to claim 1, characterized in that: Extracting the current frame image from a video file includes: Extract the current frame image from the currently shot video file; or The current frame image is extracted from the video file obtained from the server.

3. The image fusion method according to claim 2, characterized in that: The step of caching the current frame segmented image into the video segmentation queue further comprises: When the difference between the current frame image and the previous frame image exceeds the first preset threshold, the video segmentation queue is cleared, and the current frame segmented image is added to the cleared video segmentation queue.

4. The image fusion method according to claim 3, characterized in that: After caching the current frame segmented image into the video segmentation queue, the following step is further included: Determining whether the number of historical frame images in the video segmentation queue exceeds a preset storage upper limit; When the number of historical frame images in the video segmentation queue exceeds the preset storage upper limit, the historical frame image with the longest time distance from the current frame image is deleted.

5. The image fusion method according to claim 1, characterized in that: The fusing process with the historical frame segmentation images in the video segmentation queue to obtain the current frame fused image comprises: Obtaining the fusion weight of each of the historical frame segmented images according to the chronological order; The current frame image is fused according to the fusion weights of the historical frame segmented images to obtain the current frame fused image.

6. The image fusion method according to claim 1, characterized in that: The method further comprises: Optimizing the current frame fused image and outputting the optimized current frame fused image; The optimization process includes one or more of the following: Gaussian smoothing and sharpening.

7. A video image processing device, comprising a processor, a memory and a communication bus; The communication bus is used to realize the connection and communication between the processor and the memory; The processor is used to execute one or more computer programs stored in the memory to implement the steps of the image fusion method according to any one of claims 1 to 6.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores one or more computer programs, and the one or more computer programs can be executed by one or more processors to implement the steps of the image fusion method according to any one of claims 1 to 6.

Images