Video processing device, method and program

Abstract

To provide a video processing device that can suppress processing delays while preventing an image not to be detected from being mistakenly determined to be an image of a target object.SOLUTION: A video processing device 10 comprise an input unit 11 to which an input image is input, a database 13 for storing specified images to be stored, and a determination processing unit 12 for comparing each pixel in an arbitrary area containing an arbitrary pixel of the input image with a pixel associated with each pixel in the arbitrary area in the image to be stored, and determining whether or not the arbitrary pixel is a detection target pixel based on the comparison result.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to a video processing method for extracting or removing a specified area from an image. [Background technology]

[0002] For real-time video and audio communication, video processing to extract and cut out people and other objects from video is important. Therefore, in real-time video and audio communication tools used in web conferences and other similar applications, technology has been used to cut out the video of the target object from the background and composite it with another background. By combining the image with an appropriate background, such communication tools enable smoother communication.

[0003] Furthermore, in video and audio communication, which requires real-time performance, it is necessary to execute video and audio processing within a processing time that satisfies the real-time requirements of the video and audio communication. For example, in a remote ensemble performance, if a 240 BPM (beats per minute) song is allowed with a delay of about 1 / 10 of a beat, the duration of one beat is 60 seconds / 240 BPM = 0.25 seconds, and the allowable delay is 1 / 10 of that, or 0.025 seconds, or about 25 milliseconds.

[0004] This processing time includes all of the following: the camera's shooting time, the processing time within the camera, the transmission time over the network, the video and audio processing time of the communication system itself, etc. Therefore, the time available for object extraction and clipping processing is limited to a few milliseconds or less.

[0005] For example, Non-Patent Document 1 discloses an image segmentation method for dividing an image based on area and boundary information as a method for recognizing an image. Also, Non-Patent Document 2 discloses methods that mimic human thinking, such as fuzzy logic, deep learning, and genetic algorithms, as part of image segmentation.

[0006] Furthermore, Non-Patent Document 3 discloses a highly accurate cutout method using MODNet, which extracts and deletes the background of a portrait photograph and then synthesizes another background. Specifically, Non-Patent Document 3 cuts out the foreground image (image of the target object) using a mask image used to separate the foreground image from the background image. Furthermore, Non-Patent Document 4 discloses a background subtraction method that compares pixel values ​​of an input image with pixel values ​​of a prepared background image to extract pixels having a specific pixel value. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Freixenet, Jordi, et al. “Yet Another Survey on Image Segmentation: Region and Boundary Information Integration.”European conference on computer vision. Springer,Berlin, Heidelberg, 2002. [Non-patent document 2] Chouhan, Siddharth Singh, Ajay Kaul, and Uday Pratap Singh. “Soft computing approaches for image segmentation: a survey” Multimedia Tools and Applications 77.21 (2018): 28483-28537 [Non-patent document 3] Z. Ke et al., “Modnet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition,” vol. 36, pp. 1140-1147, Jun. 2022. Internet<https: / / ojs.aaai.org / index.php / AAAI / article / view / 19999> [Non-patent document 4] J. Morita, Y. Iwai, and M. Yachida, “Separation of background and object in an indoor scene” IPSJ SIG Computer Vision and Image Media, vol. 34(2002-CVIM-133), pp. 9-16, May 2002. Summary of the Invention [Problem to be solved by the invention]

[0008] However, when cutting out a foreground image using MODNet as described in Non-Patent Document 3, it is necessary to first input the input image into a learning model and generate a mask image. The mask image is then used to extract the foreground image, and the original image, foreground image, and mask image are then connected. As described above, the technology described in Non-Patent Document 3 requires multiple processes, as well as connection processes between each process (pre-processing, post-processing, and connection processes between before and after processing). Therefore, while the time available for object extraction and cutting out is a few milliseconds or less, the method described in Non-Patent Document 3 results in a delay of at least three frames (50 ms or more at 60 fps).

[0009] On the other hand, the background subtraction method described in Non-Patent Document 4 can reduce processing delays relatively, but since it compares pixel values, it is affected by light such as shadows, lighting, and sunlight, and there is a possibility that the extracted range will be incorrect.

[0010] Therefore, in the image processing for extracting a target object from an image, the processing delay can be suppressed, It has been desired to prevent an image that is not the detection target from being mistakenly determined to be an image of the target object.

[0011] In order to solve the above problems, the present disclosure aims to provide an image processing device and an image processing method that can suppress processing delays while preventing an image that is not the target of detection from being mistakenly determined to be an image of the desired object. [Means for solving the problem]

[0012] To achieve the above object, the video processing device and video processing method of the present disclosure employ a technique of comparing each pixel in an arbitrary region including a given pixel of an input image with each pixel associated with each pixel in the arbitrary region in a specified stored image, and determining whether the given pixel in the input image is a detection target pixel based on the comparison result. For example, the stored image may be a background image or a foreground image. Furthermore, for example, the detection target pixel may be a background pixel or a foreground pixel.

[0013] Specifically, the video processing device of the present disclosure includes: an input unit to which an input image is input; a storage unit for storing the designated storage image; a determination processing unit that compares each pixel in an arbitrary region including the arbitrary pixel of the input image with a pixel associated with each pixel in the arbitrary region in the stored image, and determines whether the arbitrary pixel is a detection target pixel based on a result of the comparison; Equipped with.

[0014] In addition, the video processing method of the present disclosure includes: A video processing method executed by a video processing device including an input unit to which an input image is input, a storage unit to store a specified storage image, and a determination processing unit, The determination processing unit: comparing each of the pixels of a given region including the given pixel of the input image with a pixel associated with each of the pixels of the given region in the stored image; determining whether the arbitrary pixel is a detection target pixel based on a result of the comparison; Includes.

[0015] This allows for determination of whether a pixel to be determined is a pixel to be detected based not only on the arbitrary pixel itself, but also on the determination results for the surrounding pixels of the arbitrary pixel included in the area containing the arbitrary pixel, thereby suppressing processing delays while also suppressing erroneous determinations.

[0016] The determination processing unit may determine whether or not the given pixel is a detection target pixel based on the ratio of the number of pixels in the given region that satisfy a predetermined condition.

[0017] Moreover, the determination processing unit Calculating a difference between the pixel value of each pixel in the arbitrary region and the pixel value of a pixel associated with each pixel in the arbitrary region in the stored image; comparing the difference with a predetermined threshold; It may be determined whether or not the given pixel is a detection target pixel based on the ratio of the number of pixels in the given region that satisfy the given condition as a result of comparing the difference with the given threshold value.

[0018] For example, the following (i) to (iv) are possible relationships between the difference and the predetermined threshold value. (i): Difference > Threshold (Difference is greater than the threshold) (ii): Difference ≧ threshold (difference is greater than or equal to threshold) (iii): Difference = Threshold (the difference is equal to the threshold) (iv): Difference≦threshold (difference is less than or equal to threshold) (v): Difference < threshold (difference is smaller than threshold)

[0019] The determination processing unit may determine whether the arbitrary pixel is a detection target pixel based on whether the number of pixels that satisfy the predetermined condition is more or less than half of the number of pixels in the arbitrary region.

[0020] The arbitrary region may also be a square region on the frame in which the arbitrary pixel is located.

[0021] The program of the present disclosure is a program for causing a computer to realize each functional unit of the video processing device of the present disclosure, and is a program for causing a computer to execute each step of the communication method performed by the video processing device of the present disclosure.

[0022] The above disclosures can be combined as much as possible. [Effects of the Invention]

[0023] According to the present disclosure, it is possible to suppress processing delays while suppressing the erroneous determination that an image that is not the detection target is an image of the target object. [Brief explanation of the drawings]

[0024] [Figure 1] 1 is a diagram illustrating a configuration of a video processing device according to an embodiment of the present disclosure. [Figure 2] 10 is a flowchart illustrating a processing flow of a determination processing unit according to an embodiment of the present disclosure. [Figure 3] FIG. 1 is a diagram illustrating a background determination method according to an embodiment of the present disclosure. [Figure 4] FIG. 1 is a diagram illustrating a background determination method according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0025] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0026] [Configuration of video processing device] An image processing device 10 according to an embodiment of the present disclosure will be described with reference to Figures 1 and 2. The image processing device 10 according to the embodiment determines the range of the background image in a frame at a certain time based on pixel values ​​of an input image in the frame and pixel values ​​of a background image stored in advance. However, the range of this embodiment is not limited to this, and it may also be determined whether a pixel to be determined is a pixel related to the foreground (hereinafter referred to as a "foreground pixel") based on pixel values ​​of a foreground image stored in advance.

[0027] Specifically, in this embodiment, in addition to the determination result based on the pixel value of a pixel currently being input in real time (hereinafter referred to as the "target pixel"), it is determined whether the target pixel is a pixel related to the background (hereinafter referred to as the "background pixel") based on the determination result based on the pixel values ​​of pixels neighboring the target pixel in the same frame. However, the scope of this embodiment is not limited to this, and instead of the determination result based on the pixel values ​​of pixels neighboring the target pixel in the same frame, it may be determined whether the target pixel is a background pixel using the determination result based on the pixel values ​​of pixels related to the target pixel in a previous frame or their neighboring pixels. Alternatively, it may be determined whether the target pixel is a background pixel by combining the determination result based on the pixel values ​​of pixels neighboring the target pixel in the same frame and the determination result based on the pixel values ​​of pixels related to the target pixel in a previous frame or their neighboring pixels.

[0028] In this way, the video processing device 10 determines whether or not the determination target pixel is a detection target pixel that is a target for detection. This makes it possible to reduce processing delay while suppressing erroneous determinations. That is, it is possible to correctly determine whether or not a pixel is a background pixel (in other words, whether or not it is a pixel other than a background pixel), and to appropriately cut out the image of the target object from the background image. Note that a pixel determined to be a detection target pixel may be cut out and used as is (extracted), or may be removed from the image.

[0029] FIG. 1 is a diagram illustrating the configuration of a video processing device 10. The video processing device 10 includes an input unit 11, a determination processing unit 12, and a database 13. The database 13 functions as a "storage unit." A video captured by an external camera or the like is input to the input unit 11 as an input image. A background image is stored in advance in the database 13. The determination processing unit 12 determines the range of the background image in a certain frame based on the background image stored in the database 13. The database 13 may be configured to store a background image as a template in advance, or to allow a user to input an arbitrary background image from outside. Alternatively, the database 13 may be omitted, in which case the user may directly input an arbitrary background image into the determination processing unit 12. Note that "in advance" in the present disclosure includes cases where a background image is already stored in the database 13, as well as cases where a user inputs a background image into the determination processing unit 12 or the database 13 at an arbitrary timing. Furthermore, the image stored in the database 13 is not limited to a background image, but may be an arbitrarily created image. For example, the images stored in database 13 may be foreground images (i.e., images of the object to be cut out). These images stored in the database are an example of the "specified stored image" in the present disclosure. Database 13 may also store a "threshold value A," which will be described later.

[0030] In this way, the video processing device 10 an input unit 11 to which an input image is input; a database 13 for storing the designated memory image; a determination processing unit 12 that compares each pixel in an arbitrary region including a given pixel of the input image with a pixel associated with each pixel in the arbitrary region in the stored image, and determines whether or not the given pixel is a detection target pixel based on the comparison result; Equipped with.

[0031] [flowchart] 2 is a flowchart illustrating the processing flow of the determination processing unit 12. The determination processing unit 12 compares a background image prepared in advance with the input image, and determines whether or not each pixel in a specific frame of the input image is a background pixel multiple times. This will be described in detail below.

[0032] However, the object of comparison with the input image is not limited to a background image. Various images arbitrarily designated and stored in database 13 are included in the objects of comparison with the input image. For example, a configuration may be adopted in which a foreground image is compared with the input image. Furthermore, although the following description will discuss a case in which it is determined whether an arbitrary pixel is a foreground pixel (an example of a detection target pixel), the scope of the present disclosure also includes a case in which it is determined whether an arbitrary pixel is a foreground pixel (an example of a detection target pixel).

[0033] First, in step S1, the determination processing unit 12 calculates the difference (diff) between the pixel value of a pixel in the input image and the pixel value of a pixel in the background image associated with the pixel. Specifically, the determination processing unit 12 calculates the difference between the pixel value of a pixel to be determined and the pixel value of a pixel associated with the pixel to be determined in the background image. Here, being associated with the pixel to be determined means that the positions of the pixels in each frame correspond. However, the scope of the present disclosure is not limited to this, and the determination processing unit 12 may also calculate the difference between a neighboring pixel or a combination of a pixel associated with a neighboring pixel.

[0034] In step S1, the determination processing unit 12 calculates the difference (diff) between the pixel value of each of n-1 pixels (peripheral pixels) in an arbitrary peripheral region centered on the determination target pixel and the pixel value of a pixel in the background image associated with the pixel. Here, the number of pixels in an arbitrary region including the determination target pixel is defined as n, and the number of pixels in the peripheral region of the determination target pixel within that region is defined as n-1.

[0035] Next, in step S2, the determination processing unit 12 determines for each pixel whether the difference calculated in step S1 exceeds a predetermined threshold A (first determination). If the determination processing unit 12 determines that the difference is smaller than threshold A, it determines that the determination target pixel and surrounding pixels are background pixels. On the other hand, if the determination processing unit 12 determines that the difference is larger than threshold A, it determines that the determination target pixel and surrounding pixels are not background pixels. Threshold A is an example of a "first threshold."

[0036] The relationship between the difference and the threshold value A is not limited to the above. In other words, how to set the threshold value A is not limited to the above. For example, the relationship between the difference and the threshold value A may be one of the following (i) to (v). (i): Difference > Threshold A (The difference is greater than Threshold A) (ii): Difference ≧ threshold A (difference is greater than or equal to threshold A) (iii): Difference = Threshold A (the difference is equal to threshold A) (iv): Difference≦threshold A (difference is less than or equal to threshold A) (v): Difference < Threshold A (The difference is smaller than Threshold A) Then, if the relationship between the difference and the threshold A is set to any one of (i) to (v) above and satisfies the set condition, the determination processing unit 12 may determine that the determination target pixel is a background pixel.

[0037] The determination processing unit 12 temporarily stores the determination result no matter what the determination result is in step S2. Specifically, the determination processing unit 12 temporarily stores these determination results even when the difference is smaller than threshold A and the determination target pixel and the surrounding pixels are determined to be background pixels, or when the difference is larger than threshold A and the determination target pixel and the surrounding pixels are determined not to be background pixels. These determination results may be stored in the database 13.

[0038] In step S3, the determination processing unit 12 counts the number of pixels (diff2) having pixel values ​​greater than the threshold A based on the determination results for the n pixels including the target determination pixel. In other words, the determination processing unit 12 counts the number of pixels determined not to be background pixels. However, the scope of the present disclosure is not limited to this, and the determination processing unit 12 may count the number of pixels determined to be background pixels.

[0039] Next, in step S4, the determination processing unit 12 determines whether the number of pixels (diff2) determined to be background pixels in the determination results for n pixels included in an arbitrary region including the determination target pixel exceeds a predetermined ratio (second determination). That is, the determination processing unit 12 determines whether the determination target pixel is a background pixel based on the determination results for the determination target pixel as well as the determination results for surrounding pixels. Specifically, the determination processing unit 12 compares the number of pixels (diff2) having pixel values ​​greater than threshold A with n / k, which is the number of pixels n included in the region divided by a predetermined value k. Here, 1 / k represents an arbitrary ratio.

[0040] In this embodiment, if the number of pixels having pixel values ​​greater than threshold A is less than half of the number of pixels included in the region, determination processing unit 12 determines that the determination target pixel is a background pixel (step S5). On the other hand, if the number of pixels having pixel values ​​greater than threshold A is more than half of the number of pixels included in the region, determination processing unit 12 determines that the determination target pixel is not a background pixel (step S6). That is, in this embodiment, k is set to 2.

[0041] However, the scope of the present disclosure is not limited to the case where k=2, and k can be set to any natural number. For example, since multiple types of pixels are mixed near the boundary between the background portion and other portions (e.g., human portions) in an image, it is possible to use the result of determining by changing the proportion near the boundary. For example, near the boundary, the result obtained when the value of k is set to a value greater than 2 may be used. In other words, near the boundary, the determination processing unit 12 may use a method of determining that the portion is a human portion even if the proportion of the number of pixels exceeding threshold A is relatively small. In this case, the determination processing unit 12 may use the determination result obtained when the value of k is changed so as to complement the determination result obtained when k=2. The set value of k may also be stored in the database 13.

[0042] In this way, the determination processing unit 12 determines whether or not a given pixel is a detection target pixel based on the ratio of the number of pixels that satisfy a predetermined condition in a given region.

[0043] In addition, the determination processing unit 12 Calculating a difference between the pixel value of each pixel in the arbitrary region and the pixel value of a pixel associated with each pixel in the arbitrary region in the stored image; The difference is compared with a predetermined threshold A; In a given region, it is determined whether or not a given pixel is a detection target pixel based on the ratio of the number of pixels for which the result of comparing the difference with threshold A satisfies a predetermined condition.

[0044] Furthermore, the determination processing unit 12 determines whether or not a given pixel is a detection target pixel based on whether the number of pixels that satisfy a predetermined condition is more or less than half the number of pixels in the given region.

[0045] According to this embodiment, whether or not a pixel to be determined is a background pixel is determined based on the determination results of surrounding pixels in addition to the determination result of the pixel to be determined, so that if the surrounding pixels are close to the background, it is easy to determine that the pixel is the background, and if the surrounding pixels are different from the background, it is easy to determine that the pixel is something other than the background (for example, a person, foreground). Furthermore, according to this embodiment, whether or not a pixel to be determined is a background pixel can be determined based on the determination results of the surrounding pixels, so that it is less likely to make an erroneous determination even if a change occurs in the pixel.

[0046] Furthermore, according to this embodiment, in ultra-low latency object extraction and cut-out processing to realize communication using real-time video in scenes with strict latency requirements, such as remote ensembles, it is possible to suppress processing delays while also preventing images that are not the target of detection from being mistakenly determined to be the desired image.

[0047] Furthermore, according to this embodiment, it is possible to suppress erroneous cutout processing of backgrounds and objects due to the influence of light such as shadows, lighting, and sunlight, or when the cutout target has a similar color.

[0048] In this embodiment, the first determination is performed using one threshold value A, but any number of threshold values ​​may be used to perform multi-stage determination. Similarly, the second determination may also be performed using any number of ratios to perform multi-stage determination.

[0049] [Background determination method] Next, a background determination method according to an embodiment of the present disclosure will be described with reference to Figs. 3 and 4. Figs. 3 and 4 are diagrams for explaining the background determination method. (A) of each figure shows the determination result using the conventional background subtraction method, and (B) of each figure shows the determination result using the determination method according to this embodiment. That is, (B) of each figure shows the determination result when the determination processing unit 12 performs the process shown in Fig. 2.

[0050] The numbers in the diagram correspond to the difference between the pixel value of the pixel to be determined and the pixel value of a pixel associated with the pixel to be determined in the background image. Pixels with underlines under the numbers are non-background pixels (e.g., people, foreground), while pixels without underlines under the numbers are background pixels. In other words, the underlined parts of the numbers are parts that should not be determined as background pixels, while the non-underlined parts are parts that should be positively determined as background pixels. Note that Figures 3 and 4 show the case where threshold A=25.

[0051] In the conventional background subtraction method shown in (A) of each figure, the difference between the pixel value of the pixel to be determined and the pixel value of a pixel associated with the target pixel in the background image is compared with a threshold value A to determine whether the target pixel is a background pixel, without using the determination results of surrounding pixels. As a result, there is a possibility that pixels that are not background pixels may be determined to be background pixels. In Figure 3 (A), some pixels with differences less than the threshold value A (=25) are determined to be background pixels even though they are not background pixels.

[0052] On the other hand, in this embodiment, the determination processing unit 12 determines whether the number of pixels (diff2) determined to be background pixels exceeds a predetermined ratio based on the determination results for n pixels included in an arbitrary region including the target pixel (second determination). (B) of each figure shows a case where the region considered in the second determination is a 5 × 5 square centered on the target pixel. That is, in this embodiment, the arbitrary region including the target pixel is a square region on the frame in which the target pixel is located. As described above, in this embodiment, whether the target pixel is a background pixel is determined based on the determination results of the target pixel itself as well as the determination results of its surrounding pixels. Therefore, a pixel that should be determined to be a non-background pixel even if its pixel value is less than threshold A can be correctly determined to be a non-background pixel. Furthermore, because the determination can be made based on the surrounding pixels, it is less likely to make an incorrect determination even if there is a change in the pixel.

[0053] As shown in FIG. 4B, even when the determination method according to this embodiment is used, there is a possibility that pixels located on the boundary between the background portion and other portions will not be correctly determined as background pixels (or not as background pixels). In this case, it is possible to address this by combining other determination methods. For example, as described above, it is possible to complementarily use the method of determining that an area is a person portion even if the proportion of pixels exceeding threshold A is relatively small. Also, it is possible to complementarily use the determination results of several surrounding pixels determined before or after in time.

[0054] Furthermore, the area considered in the second determination is not limited to a square area. The area may have any shape, such as a cross, a circle, or a rectangle. The number of pixels included in these areas may also be arbitrary. The area considered in the second determination may also be a temporally wide area, that is, an area spanning multiple frames.

[0055] (Other Examples) The above-described embodiments can be combined as appropriate, thereby providing a video processing device that can reduce processing delays and suppress erroneous determinations.

[0056] Furthermore, the video processing device of the present disclosure may be an integrated circuit such as an FPGA (Field-Programmable Gate Array).

[0057] The device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program of the present disclosure is a program for causing a computer to realize each function of the device according to the present disclosure, and a program for causing a computer to execute each procedure of the method executed by the device according to the present disclosure. [Explanation of symbols]

[0058] 10: Image processing device 11: Input section 12: Judgment processing unit 13: Database

Claims

1. an input unit to which an input image is input; a storage unit for storing the designated storage image; a determination processing unit that compares each pixel in an arbitrary region including the arbitrary pixel of the input image with a pixel associated with each pixel in the arbitrary region in the stored image, and determines whether the arbitrary pixel is a detection target pixel based on a result of the comparison; A video processing device comprising:

2. the determination processing unit determines whether the arbitrary pixel is a detection target pixel based on a ratio of a number of pixels that satisfy a predetermined condition in the arbitrary region. The video processing device according to claim 1 .

3. The determination processing unit Calculating a difference between the pixel value of each pixel in the arbitrary region and the pixel value of a pixel associated with each pixel in the arbitrary region in the stored image; comparing the difference with a predetermined threshold; determining whether the arbitrary pixel is a detection target pixel based on the ratio of the number of pixels in the arbitrary region where the result of comparing the difference with the predetermined threshold satisfies the predetermined condition; The video processing device according to claim 2 .

4. the determination processing unit determines whether the arbitrary pixel is a detection target pixel based on whether the number of pixels that satisfy the predetermined condition is more than or less than half of the number of pixels in the arbitrary region. The video processing device according to claim 3 .

5. the arbitrary region is a square region on the frame in which the arbitrary pixel is located; The video processing device according to claim 4 .

6. A video processing method executed by a video processing device including an input unit to which an input image is input, a storage unit to store a specified storage image, and a determination processing unit, The determination processing unit: comparing each of the pixels of a given region including the given pixel of the input image with a pixel associated with each of the pixels of the given region in the stored image; determining whether the arbitrary pixel is a detection target pixel based on a result of the comparison; A video processing method including:

7. A program for causing a computer to implement each of the functional units of the video processing device according to any one of claims 1 to 5.

Images