A Binarized Video Recording Method Based on Object Region Motion Sequence

Abstract

The present invention provides a binary video recording method based on object region motion sequence, which relates to the technical field of image processing, and comprises the following steps: traversing each image frame in a mask sequence of a human body motion image after matting, performing image segmentation based on the pixel value of each image frame to obtain a target region; traversing each target region, dividing the target region according to preset conditions one and two, and obtaining an object region list describing the target region; traversing each object region list in time sequence, matching the object regions in the preceding and following frames in turn, and constructing a motion sequence description of each object region. The present invention analyzes the object region in each frame image, matches each object region in a continuous video, and utilizes the similarity of the same object region to greatly reduce the amount of data required to describe the video data, thereby greatly reducing the amount of data for recording the binary video data, and thus the binary video data can be stored with much smaller data than the original video.

Description

Technical Field

[0001] The invention relates to the technical field of image processing, and in particular to a binary video recording method based on object region motion sequence. Background Art

[0002] With the development of digital media technology, video editing has become a widely used tool, not only in the field of professional film and television production, but also increasingly popular among individual users. Especially in the fields of short video platforms, live broadcasts, online education, etc., users' requirements for video quality are constantly increasing, and video editing technology is also constantly innovating and developing. Among them, character video matting is an important technical link, which allows users to separate the characters in the video from the original background in order to replace it with a new background or perform other creative processing. This process usually involves accurately marking the area where the characters are located and transmitting these marked image data sequences to the server for further processing.

[0003] At present, the methods for realizing character video matting mainly include manual matting, semi-automatic matting and fully automatic matting. Among them, fully automatic matting is favored by users due to its high efficiency and convenience. However, in practical applications, fully automatic matting technology faces a significant problem - low efficiency of data transmission and storage. Specifically, in order to ensure the fineness of the matting effect, the system usually saves the marked character image in a lossless format (such as PNG) to ensure that the details of each frame of the image are preserved. Although this approach can provide high-quality image output, it results in an abnormally large amount of data in a single file, especially when processing high-definition or ultra-high-definition videos, the amount of data grows exponentially. This not only increases the cost and time of network transmission, but also puts higher requirements on the storage resources on the server side.

[0004] In addition, large-scale data transmission will also occupy a lot of bandwidth resources, affecting the normal operation of other services, especially in poor network conditions, where transmission delays or even failures may occur. Therefore, how to effectively reduce the amount of data transmission and storage space requirements while ensuring image quality has become one of the key issues that need to be urgently addressed in current character video matting technology.

[0005] Therefore, the present application provides a binary video recording method based on object region motion sequence, which can solve the above technical problems. Summary of the invention

[0006] To solve the above technical problems, the present invention provides a binary video recording method based on object region motion sequence, which records data by utilizing the spatiotemporal region change characteristics of marked data after matting, and can significantly reduce the amount of data that needs to be stored and transmitted.

[0007] The present invention provides a binary video recording method based on an object region motion sequence, the recording method comprising the following steps:

[0008] Traversing each image frame in the mask sequence of the matted human motion image, and performing image segmentation based on the pixel value of each image frame to obtain the target area;

[0009] Traversing each of the target areas, dividing the target areas according to preset conditions 1 and 2, and obtaining an object area list describing the target areas, wherein the object area list includes a plurality of rectangular areas of different sizes, the preset condition 1 being that the number of black pixel points contained in all the divided rectangular areas is the least, and the preset condition 2 being that the area of ​​the smallest rectangular area is greater than a preset minimum area value;

[0010] Each object region list is traversed in time order, the object regions in the previous and next frames are matched in sequence, and a motion sequence description of each object region is constructed to losslessly represent the binary video in a list form.

[0011] Preferably, the traversing each image frame in the matted human motion image mask sequence and performing image segmentation based on the pixel value of each image frame to obtain the target area includes:

[0012] Obtaining a mask sequence of the human motion image after matting, wherein the mask sequence is a black and white image and is used to identify the target area and the background area;

[0013] For each image frame in the mask sequence, perform the following operations:

[0014] Traverse all pixels of the image frame;

[0015] The pixel value of each pixel is binarized. Specifically, if the pixel value is black, it is marked as bit0; if the pixel value is white, it is marked as bit1;

[0016] Based on the pixel labels after the binarization process, image segmentation is performed on each image frame, wherein the image segmentation specifically includes: classifying all pixel points with white pixel values ​​as the target area, and classifying all pixel points with black pixel values ​​as the background area.

[0017] Preferably, traversing each of the target areas, dividing the target area according to preset condition 1 and preset condition 2, and obtaining an object area list describing the target area includes:

[0018] For a single target area, the pixel point centered in the target area is used as the starting point for the area division, and the length and width of the rectangular area are increased in a scattered manner according to a preset angle to the boundary of the target area;

[0019] In the exhaustive process, all rectangular regions satisfying the area condition are exhaustively enumerated, and the object region is constructed based on all the rectangular regions, wherein the area condition is that the area of ​​the rectangular region is greater than a given area threshold;

[0020] The remaining target areas are marked to obtain an object area list describing the target areas, wherein the object area list is marked with the center position, length, width of each target area, and whether it contains pixels of non-target areas and their positions.

[0021] Preferably, in the exhaustive process, if the area of ​​the determined rectangular region is not greater than a given area threshold, the following operations are performed:

[0022] Find all rectangular regions whose area is equal to the given area threshold, calculate the area ratio of the non-target area of ​​each rectangular region whose area is equal to the given area threshold, select the rectangular region with the smallest area ratio and construct a new object region.

[0023] Preferably, for a new object area, it is fused with an existing object area based on a preset fusion condition, wherein the fusion condition is that the number of pixels in the non-target area of ​​the fused object area is less than or equal to the number of pixels in the non-target area before fusion.

[0024] Preferably, the preset angle is 15°.

[0025] Preferably, the area division further includes:

[0026] Exclude the portion of the current target area that is already occupied by the existing object area.

[0027] Preferably, traversing each object region list in time order, sequentially matching the object regions in the previous and next frames, and constructing a motion sequence description of each object region includes:

[0028] Obtain an object region list for each image frame, traverse each object region in the object region list to perform region screening, and obtain matching regions until the motion sequence construction of all object regions is completed.

[0029] Preferably, for each object region, the region screening includes the following sub-steps:

[0030] Screen out all candidate regions whose area difference with the current object region is within 20% from the object region list of the next image frame to form a first list;

[0031] Filter out all candidate regions whose aspect ratio errors with the current object region are within 20% in the first list to form a second list;

[0032] Select the candidate area with the smallest product of the center point offset and the angle change in the second list as the matching area of ​​the current object area in the next image frame;

[0033] Recording the center point offset, length and width changes and corresponding image numbers of the matching area into the motion sequence of the current object area;

[0034] If there is no matching region that meets the conditions in the next image frame, the motion sequence for marking the current object region is finished.

[0035] Preferably, if there is an object region in any intermediate image frame that does not match any object region in the previous frame, the object region is used as a starting point to perform region screening and construct a new motion sequence.

[0036] Compared with the related art, the binary video recording method based on the object region motion sequence provided by the present invention has the following beneficial effects:

[0037] The present invention can obtain a higher compression rate by analyzing the object area and its changes in each frame and continuously considering the motion matching of the object area within the entire video time length, using less data to describe the entire binary video data. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A flowchart of a binary video recording method based on an object region motion sequence provided by the present invention;

[0039] Figure 2 is a schematic diagram of a mask sequence of the present invention;

[0040] Figure 3 It is a schematic diagram of the scattering increasing direction of the present invention;

[0041] Figure 4 It is a schematic diagram after the area division of the present invention;

[0042] Figure 5 It is a schematic diagram of the motion sequence of the present invention. DETAILED DESCRIPTION

[0043] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only the parts related to the present invention, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments of the present invention and the features in the embodiments may be combined with each other without conflict.

[0044] It should also be noted that, for ease of description, only the parts related to the present invention, but not all of the contents, are shown in the accompanying drawings. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow charts describe the operations (or steps) as sequential processes, many of the operations therein can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operation is completed, but it can also have additional steps not included in the accompanying drawings. The process can correspond to methods, functions, procedures, subroutines, subprograms, etc.

[0045] The present invention provides a method for recording binarized video based on object region motion sequence, referring to Figure 1 As shown, the recording method comprises the following steps:

[0046] S1: traverse each image frame in the mask sequence of the matted human motion image, and perform image segmentation based on the pixel value of each image frame to obtain a target area.

[0047] Specifically, refer to Figure 2 As shown, it shows a schematic diagram of a mask sequence, step S1 includes the following steps:

[0048] S11: Obtaining a mask sequence of the matted human motion image, wherein the mask sequence is a black and white image and is used to identify a target area and a background area.

[0049] In this embodiment, a black and white image sequence generated by the matting process is obtained, that is, a mask sequence. Each image in this sequence is black and white, where white represents the target area and black represents the background area. These images are used to identify the difference between the target area and the background area in the video.

[0050] S12: Perform the following operations for each image frame in the mask sequence:

[0051] a. Traverse all pixels of the image frame.

[0052] In this embodiment, for each frame of image in the mask sequence, each pixel in the image is accessed one by one to prepare for binarization processing.

[0053] b. Binarize the pixel value of each pixel. Specifically, if the pixel value is black, it is marked as bit0; if the pixel value is white, it is marked as bit1.

[0054] In this embodiment, for each pixel accessed, its color value is checked. If the color of the pixel is black, it is marked as bit0; if the color of the pixel is white, it is marked as bit1. This process converts the image into a data form with only two values ​​(bit0 and bit1), which is convenient for subsequent processing.

[0055] S13: Based on the pixel labels after the binarization process, image segmentation is performed on each image frame, wherein the image segmentation specifically includes: classifying all pixel points with white pixel values ​​as the target area, and classifying all pixel points with black pixel values ​​as the background area.

[0056] In this embodiment, according to the result of the binarization process, all pixels marked as bit1 are classified as the target area, and pixels marked as bit0 are regarded as the background area. The purpose of this is to clearly distinguish the target area from the background area, thereby providing a basis for subsequent object area segmentation and motion sequence construction.

[0057] S2: traverse each of the target areas, divide the target areas according to preset conditions one and two, and obtain an object area list describing the target areas, wherein the object area list includes multiple rectangular areas of different sizes, the preset condition one is that the number of black pixel points contained in all the divided rectangular areas is the least, and the preset condition two is that the area of ​​the smallest rectangular area is larger than the preset minimum area value.

[0058] Specifically, step S2 includes the following steps:

[0059] S21: For a single target area, a pixel point centered in the target area is used as a starting point for dividing the area, and the length and width of the rectangular area are increased in a scattered manner according to a preset angle to the boundary of the target area.

[0060] In this embodiment, reference Figure 3 As shown, the preset angle is 15°.

[0061] For a single target area, a central point in the target area (the white part of the image, representing the object of interest) is selected as the starting point. From this point, try to expand the rectangle in 12 different directions (each direction is 15 degrees apart, from 0 degrees to 165 degrees) until the edge of the rectangle touches the boundary of the target area. The purpose of this is to try to find a rectangle that can best cover the target area.

[0062] S22: In the exhaustive process, all rectangular regions satisfying the area condition are exhaustively enumerated, and the object region is constructed based on all the rectangular regions, wherein the area condition is that the area of ​​the rectangular region is greater than a given area threshold.

[0063] In this embodiment, this process will encounter many rectangles of different sizes. However, only those rectangles whose areas are larger than the set minimum area standard are of concern. This minimum area standard is to prevent the selection of too small an area, because too small an area may not effectively represent a part of the object, and may also result in a large number of small areas being generated during the processing, increasing the complexity of the processing.

[0064] S23: Mark the remaining target areas to obtain an object area list describing the target areas, wherein the object area list is marked with the center position, length, width of each target area, and whether it contains pixels of non-target areas and their positions.

[0065] In this embodiment, reference Figure 4 As shown in the figure, the red rectangle is the divided object area. Once a rectangle that meets the requirements is found, its center position, length, width and other information will be recorded. Merging the newly found rectangular area with the existing area: If the area of ​​the non-target area in the rectangle obtained by merging the newly found rectangular area with one or more previously recorded rectangular areas does not exceed the sum of the areas of the non-target areas in these rectangles before merging, these rectangular areas should be merged. Doing so can reduce the number of areas and simplify subsequent processing.

[0066] In addition, during the exhaustive process, if the area of ​​the determined rectangular region is not greater than the given area threshold, the following operations are performed:

[0067] Find all rectangular regions whose area is equal to the given area threshold, calculate the area ratio of the non-target area of ​​each rectangular region whose area is equal to the given area threshold, select the rectangular region with the smallest area ratio and construct a new object region.

[0068] In this embodiment, if no rectangle with an area larger than the minimum area standard is found during the expansion process, all rectangles with an area exactly equal to the minimum area standard are selected, and the rectangle with the smallest proportion of non-target area (i.e., black part) is selected from them. This is done to reduce unnecessary background information as much as possible.

[0069] In addition, the regional divisions include:

[0070] Exclude the portion of the current target area that is already occupied by the existing object area.

[0071] In this embodiment, each time a suitable rectangle is found, this part of the area is deleted from the target area to ensure that the processed part will not be considered repeatedly in the next processing. This process will continue until there is no remaining unprocessed part in the target area.

[0072] Merge the newly found rectangular area with the existing area: If the area of ​​the non-target area in the rectangle obtained by merging the newly found rectangular area with one or more previously recorded rectangular areas does not exceed the sum of the areas of the non-target areas in these rectangles before merging, these rectangular areas should be merged. The merged rectangular area can overlap with other existing areas. Overlapping areas do not increase the amount of data description, and the amount of description data can be further reduced after the area is merged.

[0073] S3: traverse each object region list in time order, match the object regions in the previous and next frames in turn, and construct a motion sequence description of each object region to losslessly represent the binary video in a list form.

[0074] Wherein, step S3 is specifically as follows:

[0075] Obtain an object region list for each image frame, traverse each object region in the object region list to perform region screening, and obtain matching regions until the motion sequence construction of all object regions is completed.

[0076] In this embodiment, an object region list is obtained for each image frame. These lists contain detailed information of all identified object regions in the image, such as the center position, length and width, whether it contains non-target area pixels and their position information, etc. Then, each object region in the object region list is traversed to perform region screening to find the corresponding region of the current object region in the subsequent frame, and construct its motion sequence until the motion sequence construction of all object regions is completed. Figure 5 As shown, the figure contains three object area motion sequences, the red box is the head area motion sequence, the green box is the body trunk motion sequence, and the blue box is the arm area motion sequence.

[0077] Wherein, for each object area, the area screening includes the following sub-steps:

[0078] a. In the object region list of the next image frame, select all candidate regions whose area difference with the current object region is within 20% to form a first list.

[0079] In this embodiment, the area difference is calculated by comparing the area of ​​the current object area with the area of ​​each candidate area in the next frame. If the area of ​​a candidate area differs from the area of ​​the current object area by no more than 20%, it is added to the first list. The purpose of this step is to initially narrow the scope of the candidate areas and improve the efficiency of subsequent screening.

[0080] b. Filter out all candidate regions whose aspect ratio error with the current object region is within 20% in the first list to form a second list.

[0081] In this embodiment, based on the first list, further screening is performed based on the aspect ratio error. The aspect ratio error is calculated by comparing the aspect ratio of the current object area with the aspect ratio of the candidate area. If the aspect ratio of a candidate area differs from the aspect ratio of the current object area by no more than 20%, it is added to the second list. The purpose of this step is to further narrow the scope of the candidate area and improve the accuracy of matching.

[0082] c. Select the candidate area with the smallest product of the center point offset and the angle change in the second list as the matching area of ​​the current object area in the next image frame.

[0083] In this embodiment, in the second list, the center point offset and angle change of each candidate region and the current object region are calculated. The center point offset refers to the distance between the center of the candidate region and the center of the current object region, and the angle change refers to the rotation angle difference between the candidate region and the current object region. Then, the candidate region with the smallest product of the center point offset and the angle change is selected as the matching region. The purpose of this step is to find the candidate region that is most similar to the current object region in position and shape.

[0084] d. Record the center point offset, length and width changes and corresponding image numbers of the matching area into the motion sequence of the current object area.

[0085] In this embodiment, once a matching area is found, its center point offset, length and width change, and corresponding image number are recorded as part of the current object area motion sequence. The purpose of this step is to construct the motion trajectory of the object area for subsequent analysis and processing.

[0086] e. If there is no matching area that meets the conditions in the next image frame, the motion sequence of the current object area is marked as finished.

[0087] In this embodiment, if no candidate region matching the current object region can be found in the next image frame, it is considered that the motion sequence of the current object region has ended. The purpose of this step is to handle the situation where the object region disappears in the video.

[0088] In addition, if there is an object region in any intermediate image frame that does not match any object region in the previous frame, the object region is used as a starting point to perform region screening and construct a new motion sequence.

[0089] In this embodiment, the purpose of this step is to process newly appeared object regions and ensure that each object region has a corresponding motion sequence description.

[0090] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0091] A person skilled in the art may understand that all or part of the steps in the various methods of the above embodiments may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, the storage medium including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically-erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, magnetic disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0092] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

Claims

1. A method for recording binarized video based on object region motion sequence, characterized in that: The recording method comprises the following steps: Traversing each image frame in the mask sequence of the matted human motion image, and performing image segmentation based on the pixel value of each image frame to obtain the target area; Traversing each of the target areas, dividing the target areas according to preset conditions 1 and 2, and obtaining an object area list describing the target areas, wherein the object area list includes a plurality of rectangular areas of different sizes, the preset condition 1 being that the number of black pixel points contained in all the divided rectangular areas is the least, and the preset condition 2 being that the area of ​​the smallest rectangular area is greater than a preset minimum area value; Each object region list is traversed in time order, the object regions in the previous and next frames are matched in sequence, and a motion sequence description of each object region is constructed to losslessly represent the binary video in a list form.

2. A method for recording binarized video based on object region motion sequence according to claim 1, characterized in that: The step of traversing each image frame in the matted human motion image mask sequence and performing image segmentation based on the pixel value of each image frame to obtain the target area includes: Obtaining a mask sequence of the human motion image after matting, wherein the mask sequence is a black and white image and is used to identify the target area and the background area; For each image frame in the mask sequence, perform the following operations: Traverse all pixels of the image frame; The pixel value of each pixel is binarized. Specifically, if the pixel value is black, it is marked as bit0; if the pixel value is white, it is marked as bit1; Based on the pixel labels after the binarization process, image segmentation is performed on each image frame, wherein the image segmentation specifically includes: classifying all pixel points with white pixel values ​​as the target area, and classifying all pixel points with black pixel values ​​as the background area.

3. The method for recording a binary video based on an object region motion sequence according to claim 2, characterized in that: The traversing each of the target areas, dividing the target area according to the preset condition 1 and the preset condition 2, and obtaining an object area list describing the target area includes: For a single target area, the pixel point centered in the target area is used as the starting point for the area division, and the length and width of the rectangular area are increased in a scattered manner according to a preset angle to the boundary of the target area; In the exhaustive process, all rectangular regions satisfying the area condition are exhaustively enumerated, and the object region is constructed based on all the rectangular regions, wherein the area condition is that the area of ​​the rectangular region is greater than a given area threshold; The remaining target areas are marked to obtain an object area list describing the target areas, wherein the object area list is marked with the center position, length, width of each target area, and whether it contains pixels of non-target areas and their positions.

4. The method for recording a binary video based on an object region motion sequence according to claim 3, characterized in that: In the exhaustive process, if the area of ​​the determined rectangular region is not greater than the given area threshold, the following operations are performed: Find all rectangular regions whose area is equal to the given area threshold, calculate the area ratio of the non-target area of ​​each rectangular region whose area is equal to the given area threshold, select the rectangular region with the smallest area ratio and construct a new object region.

5. The method for recording a binary video based on an object region motion sequence according to claim 4, characterized in that: For the new object area, it is fused with the existing object area based on a preset fusion condition, wherein the fusion condition is that the number of pixels in the non-target area of ​​the fused object area is less than or equal to the number of pixels in the non-target area before fusion.

6. The method for recording a binary video based on an object region motion sequence according to claim 3, characterized in that: The preset angle is 15°.

7. The method for recording a binary video based on an object region motion sequence according to claim 5, characterized in that: The regional division also includes: Exclude the portion of the current target area that is already occupied by the existing object area.

8. The method for recording a binary video based on an object region motion sequence according to claim 7, characterized in that: The step of traversing each object region list in time order, sequentially matching the object regions in the previous and next frames, and constructing a motion sequence description of each object region includes: Obtain an object region list for each image frame, traverse each object region in the object region list to perform region screening, and obtain matching regions until the motion sequence construction of all object regions is completed.

9. The method for recording a binary video based on an object region motion sequence according to claim 8, characterized in that: For each object region, the region screening includes the following sub-steps: Screen out all candidate regions whose area difference with the current object region is within 20% from the object region list of the next image frame to form a first list; Filter out all candidate regions whose aspect ratio errors with the current object region are within 20% in the first list to form a second list; Select the candidate area with the smallest product of the center point offset and the angle change in the second list as the matching area of ​​the current object area in the next image frame; Recording the center point offset, length and width changes and corresponding image numbers of the matching area into the motion sequence of the current object area; If there is no matching region that meets the conditions in the next image frame, the motion sequence for marking the current object region is finished.

10. The method for recording a binary video based on an object region motion sequence according to claim 9, characterized in that: If there is an object region in any of the intermediate image frames that does not match any object region in the previous frame, the object region is used as a starting point to perform region screening and construct a new motion sequence.

Images