Video processing apparatus, method, and program

Abstract

To provide a video processing apparatus capable of suppressing processing delays and preventing an image that is not a detection target from being mistakenly determined to be an image of a target object.SOLUTION: A video processing apparatus 10 comprises: an input unit 11 to which an input image is input; a database 13 for storing a designated stored image; and a determination processing unit 12 for comparing an arbitrary pixel of the input image with a pixel associated with the arbitrary pixel in the stored image, and determining that the arbitrary pixel having a pixel value within a predetermined range is a detection target pixel based on the comparison result and pixel values of pixels surrounding the arbitrary pixel.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to a video processing method for cutting out a desired portion from an image. [Background technology]

[0002] For real-time video and audio communication, video processing to extract and cut out people and other objects from video is important. Therefore, in real-time video and audio communication tools used in web conferences and other similar applications, technology has been used to cut out the video of the target object from the background and composite it with another background. By combining the image with an appropriate background, such communication tools enable smoother communication.

[0003] Furthermore, in video and audio communication, which requires real-time performance, it is necessary to execute video and audio processing within a processing time that satisfies the real-time requirements of the video and audio communication. For example, in a remote ensemble performance, if a 240 BPM (beats per minute) song is allowed with a delay of about 1 / 10 of a beat, the duration of one beat is 60 seconds / 240 BPM = 0.25 seconds, and the allowable delay is 1 / 10 of that, or 0.025 seconds, or about 25 milliseconds.

[0004] This processing time includes all of the following: the camera's shooting time, the processing time within the camera, the transmission time over the network, the video and audio processing time of the communication system itself, etc. Therefore, the time available for object extraction and clipping processing is limited to a few milliseconds or less.

[0005] For example, Non-Patent Document 1 discloses an image segmentation method for dividing an image based on area and boundary information as a method for recognizing an image. Also, Non-Patent Document 2 discloses methods that mimic human thinking, such as fuzzy logic, deep learning, and genetic algorithms, as part of image segmentation.

[0006] Furthermore, Non-Patent Document 3 discloses a highly accurate cutout method using MODNet, which extracts and deletes the background of a portrait photograph and then synthesizes another background. Specifically, Non-Patent Document 3 cuts out the foreground image (image of the target object) using a mask image used to separate the foreground image from the background image. Furthermore, Non-Patent Document 4 discloses a background subtraction method that compares pixel values ​​of an input image with pixel values ​​of a prepared background image to extract pixels having a specific pixel value. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Freixenet, Jordi, et al. “Yet Another Survey on Image Segmentation: Region and Boundary Information Integration.”European conference on computer vision. Springer,Berlin, Heidelberg, 2002. [Non-patent document 2] Chouhan, Siddharth Singh, Ajay Kaul, and Uday Pratap Singh. “Soft computing approaches for image segmentation: a survey” Multimedia Tools and Applications 77.21 (2018): 28483-28537 [Non-patent document 3] Z. Ke et al., “Modnet: Real-Time Trimap-Free Portrait Matting via Objective Decomposition,” vol. 36, pp. 1140-1147, Jun. 2022. Internet<https: / / ojs.aaai.org / index.php / AAAI / article / view / 19999> [Non-patent document 4] J. Morita, Y. Iwai, and M. Yachida, “Separation of background and object in an indoor scene” IPSJ SIG Computer Vision and Image Media, vol. 34(2002-CVIM-133), pp. 9-16, May 2002. Summary of the Invention [Problem to be solved by the invention]

[0008] However, when cutting out a foreground image using MODNet as described in Non-Patent Document 3, it is necessary to first input the input image into a learning model and generate a mask image. The mask image is then used to extract the foreground image, and the original image, foreground image, and mask image are then connected. As described above, the technology described in Non-Patent Document 3 requires multiple processes, as well as connection processes between each process (pre-processing, post-processing, and connection processes between before and after processing). Therefore, while the time available for object extraction and cutting out is a few milliseconds or less, the method described in Non-Patent Document 3 results in a delay of at least three frames (50 ms or more at 60 fps).

[0009] On the other hand, the background subtraction method described in Non-Patent Document 4 can reduce processing delays relatively, but since it compares pixel values, it is affected by light such as shadows, lighting, and sunlight, and there is a possibility that the extracted range will be incorrect.

[0010] Therefore, in the image processing for extracting a target object from an image, the processing delay can be suppressed, It has been desired to prevent an image that is not the detection target from being mistakenly determined to be an image of the target object.

[0011] In order to solve the above problems, the present disclosure aims to provide an image processing device and an image processing method that can suppress processing delays while preventing an image that is not the target of detection from being mistakenly determined to be an image of the desired object. [Means for solving the problem]

[0012] To achieve the above object, the video processing device and video processing method of the present disclosure employ a technique in which an arbitrary pixel of an input image is compared with a pixel associated with an arbitrary pixel in a stored image, and an arbitrary pixel having a pixel value within a predetermined range is determined to be a detection target pixel based on the result of the comparison and the pixel values ​​of pixels adjacent to the arbitrary pixel. For example, the stored image may be a background image or a foreground image. Furthermore, for example, the detection target pixel may be a background pixel or a foreground pixel.

[0013] Specifically, the video processing device of the present disclosure includes: an input unit to which an input image is input; a storage unit for storing the designated storage image; a determination processing unit that compares an arbitrary pixel of the input image with a pixel associated with the arbitrary pixel in the stored image, and determines, based on a result of the comparison and pixel values ​​of pixels surrounding the arbitrary pixel, that the arbitrary pixel has a pixel value within a predetermined range as a detection target pixel; Equipped with.

[0014] In addition, the video processing method of the present disclosure includes: A video processing method executed by a video processing device including an input unit to which an input image is input, a storage unit to store a specified storage image, and a determination processing unit, The determination processing unit: comparing a given pixel of the input image with a pixel associated with the given pixel in the stored image; determining, based on the result of the comparison and pixel values ​​of pixels adjacent to the arbitrary pixel, that the arbitrary pixel has a pixel value within a predetermined range as a detection target pixel; Includes.

[0015] According to these methods, by using pixel values ​​of adjacent pixels, it is possible to reduce processing delay and suppress erroneous determination.

[0016] Moreover, the determination processing unit Calculating a difference between a pixel value of an arbitrary pixel of the input image and a pixel value of a pixel associated with the arbitrary pixel of the stored image; The arbitrary pixel having a pixel value within a predetermined range may be determined to be a detection target pixel based on the difference, the pixel value of a pixel adjacent to the arbitrary pixel in an input image including the arbitrary pixel, the pixel value of a pixel located at a position related to the arbitrary pixel in an input image different from the input image including the arbitrary pixel, or the pixel value of a pixel located adjacent to a position related to the arbitrary pixel in the different input image.

[0017] According to this, by using the pixel values ​​of adjacent pixels, it is possible to reduce the processing delay and suppress erroneous determinations.

[0018] Furthermore, when determining whether the arbitrary pixel is a detection target pixel, the determination processing unit: comparing the difference with a first threshold; If the difference satisfies a predetermined condition in relation to the first threshold value, the pixel is determined to be a detection target pixel or not a detection target pixel; When the difference does not satisfy the predetermined condition in relation to the first threshold value, When it is determined that the adjacent pixel satisfies either one of being a detection target pixel and not being a detection target pixel, a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value is determined to satisfy either one of being a detection target pixel and not being a detection target pixel in accordance with the determination of the adjacent pixel, When it is determined that an adjacent pixel satisfies either the condition that it is a pixel to be detected or that it is not a pixel to be detected, the difference may be compared with a second threshold value different from the first threshold value to determine whether a pixel for which the difference does not satisfy the predetermined condition in relation to the first threshold value is a pixel to be detected.

[0019] For example, the relationship between the difference and the first threshold value may be one of the following (i) to (v). (i): Difference > First Threshold (The difference is greater than the first threshold) (ii): Difference ≧ First Threshold (The difference is equal to or greater than the first threshold) (iii): Difference = First Threshold (The difference is equal to the first threshold) (iv): Difference≦First Threshold (The difference is equal to or less than the first threshold) (v): Difference > First Threshold (The difference is smaller than the first threshold) The relationship between the difference and the first threshold value may be set to any one of (i) to (v) above, and the judgment processing unit 12 may be configured to perform subsequent processing if the set condition is not satisfied.

[0020] Furthermore, the following (i)' to (v)' are possible relationships between the difference and the second threshold. (i)´: Difference > Second Threshold (The difference is greater than the second threshold) (ii)´: Difference ≧ second threshold (the difference is equal to or greater than the second threshold) (iii)´: difference = second threshold (the difference is equal to the second threshold) (iv)´: Difference≦second threshold (the difference is equal to or less than the second threshold) (v)´: Difference > Second Threshold (The difference is smaller than the second threshold) The relationship between the difference and the second threshold value may be set to any one of (i) to (v) above, and if the set condition is not satisfied, the judgment processing unit 12 may be configured to perform subsequent processing.

[0021] According to this, by using two thresholds, it is possible to reduce processing delay and suppress erroneous determination.

[0022] Moreover, the determination processing unit When the difference does not satisfy the predetermined condition in relation to the first threshold value, If it is determined that a neighboring pixel satisfies either one of being a detection target pixel or not being a detection target pixel before determining whether or not the arbitrary pixel is a detection target pixel, determine that a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value satisfies either one of being a detection target pixel or not being a detection target pixel in accordance with the determination of the neighboring pixel, If, before determining whether or not the arbitrary pixel is a pixel to be detected, it is determined that a neighboring pixel satisfies either the condition of being a pixel to be detected or the condition of not being a pixel to be detected, it may be determined whether or not a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold is a pixel to be detected by comparing the difference with the second threshold, which is different from the first threshold.

[0023] This makes it possible to reduce processing delay and suppress erroneous determinations based on pixels that have previously been determined as being detection target pixels.

[0024] Moreover, the determination processing unit When the difference does not satisfy the predetermined condition in relation to the first threshold value, if a plurality of adjacent pixels have been consecutively determined to be either a detection target pixel or a detection target pixel before determining whether the arbitrary pixel is a detection target pixel, determining a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value as either a detection target pixel or a detection target pixel in accordance with the determination of the plurality of adjacent pixels, If, before determining whether or not the arbitrary pixel is a detection target pixel, a plurality of adjacent pixels are consecutively determined to satisfy either the condition of being a detection target pixel or not being a detection target pixel, the difference may be compared with the second threshold value, which is different from the first threshold value, to determine whether or not a pixel for which the difference does not satisfy the predetermined condition in relation to the first threshold value is a detection target pixel.

[0025] This makes it possible to reduce processing delay and suppress erroneous determinations based on whether or not there are multiple pixels that have previously been determined to be detection target pixels in succession.

[0026] Moreover, the determination processing unit When the difference does not satisfy the predetermined condition in relation to the first threshold value, When a plurality of pixels above the arbitrary pixel are consecutively determined to be either a detection target pixel or a detection target pixel, a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold is determined to be either a detection target pixel or a detection target pixel so as to match the determination of the plurality of pixels above, When a plurality of consecutive pixels above the given pixel are determined to be either a target pixel or not a target pixel, the difference may be compared with the second threshold value, which is different from the first threshold value, to determine whether a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value is a target pixel.

[0027] This makes it possible to reduce processing delays while suppressing erroneous determinations based on whether there are multiple pixels in the upper part of the input image that are consecutively determined to be detection target pixels.

[0028] Furthermore, the program of the present disclosure is a program for realizing each functional unit of the video processing device of the present disclosure in a computer, and is a program for causing a computer to execute each step of the communication method performed by the video processing device of the present disclosure.

[0029] The above disclosures can be combined as much as possible. [Effects of the Invention]

[0030] According to the present disclosure, it is possible to suppress processing delays while suppressing the erroneous determination that an image that is not the detection target is an image of the target object. [Brief explanation of the drawings]

[0031] [Figure 1] 1 is a diagram illustrating a configuration of a video processing device according to a first embodiment of the present disclosure. [Figure 2] 5 is a flowchart illustrating a processing flow of a determination processing unit according to the first embodiment of the present disclosure. [Figure 3] 10 is a flowchart illustrating a processing flow of a determination processing unit according to a modified example of the first embodiment of the present disclosure. [Figure 4] 10 is a flowchart illustrating a processing flow of a determination processing unit according to a second embodiment of the present disclosure. [Figure 5] FIG. 10 is a diagram illustrating a background determination method according to the second embodiment. [Figure 6] 10 is a flowchart illustrating a processing flow of a video processing device according to a third embodiment of the present disclosure. [Figure 7] FIG. 10 is a diagram illustrating a background determination method according to the third embodiment. [Figure 8]10 is a flowchart illustrating a processing flow of a determination processing unit according to a fourth embodiment of the present disclosure. [Figure 9] FIG. 10 is a diagram illustrating a background determination method according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0032] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. Note that the present disclosure is not limited to the embodiments shown below. These implementation examples are merely illustrative, and the present disclosure can be implemented in various forms with various modifications and improvements based on the knowledge of those skilled in the art. Note that components with the same reference numerals in this specification and drawings indicate the same components.

[0033] (First embodiment) A video processing device 10 according to a first embodiment of the present disclosure will be described with reference to FIGS. 1 to 3. The video processing device 10 according to the first embodiment determines the range of a background image in a frame based on pixel values ​​of an input image in the frame at a certain time and pixel values ​​of a background image stored in advance. In addition, in this embodiment, the video processing device 10 uses a determination result for a previously input frame to determine whether a pixel currently being input in real time (hereinafter referred to as a "target pixel") is a pixel related to the background (hereinafter referred to as a "background pixel"). However, the scope of this embodiment is not limited to this, and the video processing device 10 may determine whether a target pixel is a pixel related to the foreground (hereinafter referred to as a "foreground pixel") based on pixel values ​​of a foreground image stored in advance. In other words, the video processing device 10 determines whether the target pixel is a detection target pixel to be detected (extracted). This reduces processing delay and prevents erroneous determinations. In other words, it is possible to correctly determine whether a pixel is a background pixel (in other words, whether it is a pixel other than a background pixel) and appropriately cut out the image of the target object from the background image. However, the scope of the present disclosure is not limited to this, and it may be possible to determine whether or not a target pixel is a background pixel by comparing it with the pixel values ​​of neighboring pixels within the same frame. This will be described in detail below.

[0034] FIG. 1 is a diagram illustrating the configuration of a video processing device 10. The video processing device 10 includes an input unit 11, a determination processing unit 12, and a database 13. The database 13 functions as a "storage unit." A video captured by an external camera or the like is input to the input unit 11 as an input image. A background image is stored in advance in the database 13. The determination processing unit 12 determines the range of the background image in a certain frame based on the background image stored in the database 13. The database 13 may store a background image as a template in advance, or may be configured to allow a user to input an arbitrary background image from outside. Alternatively, the database 13 may be omitted, in which case the user may directly input an arbitrary background image into the determination processing unit 12. Note that "in advance" in the present disclosure includes cases where a background image is already stored in the database 13, as well as cases where a user inputs a background image into the determination processing unit 12 or the database 13 at an arbitrary timing. Furthermore, the image stored in the database 13 is not limited to a background image, but may be an arbitrarily specified image. For example, the images stored in database 13 may be foreground images (i.e., images of the object to be cut out). These images stored in the database are examples of "specified stored images" in the present disclosure. Database 13 may also store "threshold A" and "threshold B," which will be described later.

[0035] In this way, the video processing device 10 an input unit 11 to which an input image is input; a database 13 for storing the designated memory image; a determination processing unit 12 that compares an arbitrary pixel of the input image with a pixel associated with the arbitrary pixel in the stored image, and determines an arbitrary pixel having a pixel value within a predetermined range as a detection target pixel based on the comparison result and pixel values ​​of pixels surrounding the arbitrary pixel; Equipped with.

[0036] 2 is a flowchart illustrating the processing flow of the determination processing unit 12. The determination processing unit 12 compares a background image prepared in advance with the input image, and determines whether or not each pixel in a specific frame of the input image is a background pixel multiple times. This will be described in detail below.

[0037] Specifically, the determination processing unit 12: Calculating a difference between a pixel value of an arbitrary pixel of the input image and a pixel value of a pixel associated with the arbitrary pixel of the stored image; Based on the difference, and the pixel values ​​of pixels adjacent to the given pixel in an input image including the given pixel, the pixel values ​​of pixels located at positions related to the given pixel in an input image different from the input image including the given pixel, or the pixel values ​​of pixels located adjacent to positions related to the given pixel in the different input image, any pixel having a pixel value within a predetermined range is determined to be the pixel to be detected.

[0038] More specifically, when determining whether a pixel is a detection target pixel, the determination processing unit 12 performs the following: comparing the difference with a first threshold; If the difference satisfies a predetermined condition in relation to a first threshold value, the pixel is determined to be a detection target pixel or not a detection target pixel; When the difference does not satisfy a predetermined condition in relation to the first threshold value, If it is determined that a neighboring pixel satisfies either one of the conditions of being a detection target pixel or not being a detection target pixel before determining whether or not a given pixel is a detection target pixel, a pixel whose difference does not satisfy a predetermined condition in relation to a first threshold value is determined to satisfy either one of the conditions of being a detection target pixel or not being a detection target pixel in accordance with the determination of the neighboring pixel, If, before determining whether a given pixel is a target pixel for detection, it is determined that a neighboring pixel satisfies either the condition that it is a target pixel for detection or that it is not a target pixel for detection, the difference is compared with a second threshold value different from the first threshold value to determine whether a pixel whose difference does not satisfy a predetermined condition in relation to the first threshold value is a target pixel for detection.

[0039] However, the object of comparison with the input image is not limited to a background image. Various images arbitrarily designated and stored in database 13 are included in the objects of comparison with the input image. For example, a configuration may be adopted in which a foreground image is compared with the input image. Furthermore, although the following description will discuss a case in which it is determined whether an arbitrary pixel is a background pixel (an example of a detection target pixel), the scope of the present disclosure also includes a case in which it is determined whether an arbitrary pixel is a foreground pixel (an example of a detection target pixel).

[0040] First, in step S10, the determination processing unit 12 calculates the difference between the pixel value of a pixel in the input image and the pixel value of a pixel in the background image associated with the pixel. Specifically, the determination processing unit 12 calculates the difference between the pixel value of the pixel to be determined and the pixel value of a pixel associated with the pixel to be determined in the background image. Here, being associated with the pixel to be determined means that the positions of the pixels in each frame correspond. However, the scope of the present disclosure is not limited to this, and the determination processing unit 12 may also calculate the difference between a neighboring pixel or a combination of a pixel associated with a neighboring pixel.

[0041] Next, in step S11, the determination processing unit 12 determines whether or not the difference calculated in step S10 exceeds a predetermined threshold A (first determination). If the determination processing unit 12 determines that the difference is greater than threshold A, it determines that the pixel to be determined is not a background pixel (step S12). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold A, it proceeds to the next step S13. Threshold A is an example of a "first threshold."

[0042] Furthermore, the relationship between the difference and the threshold value A is not limited to the above-described relationship. For example, the relationship between the difference and the threshold value A may be any of the following (i) to (v). (i): Difference > Threshold A (The difference is greater than Threshold A) (ii): Difference ≧ threshold A (difference is greater than or equal to threshold A) (iii): Difference = Threshold A (the difference is equal to threshold A) (iv): Difference≦threshold A (difference is less than or equal to threshold A) (v): Difference > Threshold A (Difference is smaller than Threshold A) The relationship between the difference and the threshold value A may be set to any one of (i) to (v) above, and if the set condition is not satisfied, the judgment processing unit 12 may be configured to perform subsequent processing.

[0043] In step S13, the determination processing unit 12 refers to the previous (immediately preceding) determination result. Specifically, the determination processing unit 12 determines whether the target pixel is a background pixel or not based on the determination result for a pixel associated with the target pixel in the frame input immediately before the frame containing the target pixel (second determination). Here, being associated with the target pixel means that the pixel's position in each frame corresponds. However, the scope of the present disclosure is not limited to this, and the determination may be made based on neighboring pixels, or may be made by combining pixels associated with the target pixel and neighboring pixels. If the determination processing unit 12 determined that the target pixel is a background pixel in the immediately preceding frame, the determination processing unit 12 also determines that the target pixel is a background pixel (step S14). On the other hand, if the determination processing unit 12 determined that the pixel associated with the target pixel is not a background pixel in the immediately preceding frame, the process proceeds to the next step S15.

[0044] In step S15, the determination processing unit 12 determines whether the difference calculated in step S10 exceeds a predetermined threshold B (third determination). Here, threshold B is smaller than threshold A. In other words, in the third determination, the pixel is determined to be a background pixel more strictly than in the first determination. In other words, the pixel is determined to be an image of the target object more leniently. If the determination processing unit 12 determines that the difference is larger than threshold B, it determines that the pixel to be determined is not a background pixel (step S16). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold B, it determines that the pixel to be determined is a background pixel (step S17). Threshold B is an example of a "second threshold."

[0045] Furthermore, the relationship between the difference and the threshold value B is not limited to the above-described content. For example, the following (i)' to (v)' are possible relationships between the difference and the threshold value B. (i)´: Difference > Threshold B (The difference is greater than Threshold B) (ii)´: Difference ≧ threshold B (difference is greater than or equal to threshold B) (iii)´: Difference = Threshold B (difference is equal to threshold B) (iv)´: Difference≦threshold B (difference is less than or equal to threshold B) (v)´: Difference > Threshold B (Difference is smaller than Threshold B) The relationship between the difference and the threshold value B may be set to any one of (i) to (v) above, and if the set condition is not satisfied, the judgment processing unit 12 may be configured to perform subsequent processing.

[0046] This type of processing can prevent a pixel to be determined to be a background pixel when it is associated with a pixel that was determined not to be a background pixel in the previous frame (for example, a pixel that was determined to be a person).

[0047] In the above, in step S13, when the frame including the pixel to be determined is n, the determination is made using pixels in the immediately preceding (n-1)th frame. However, the scope of the present disclosure is not limited to this. The determination may also be made using pixels in the (n-1)th to nkth frames.

[0048] Furthermore, in this embodiment, the determination is made using two thresholds A and B, but the determination may be made in multiple stages using any number of thresholds.

[0049] (Modification of the first embodiment) Next, a modified example of the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart illustrating the processing flow of the determination processing unit 12 according to the modified example of the first embodiment. The processing of this modified example makes it easier to determine a determination target pixel as a background pixel than in the first embodiment.

[0050] First, in step S10', similarly to step S10, the determination processing unit 12 calculates the difference between the pixel value of the determination target pixel and the pixel value of a pixel associated with the determination target pixel in the background image.

[0051] Next, in step S11', the determination processing unit 12 determines whether or not the difference calculated in step S10' is smaller than a predetermined threshold A (first determination). If the determination processing unit 12 determines that the difference is smaller than the threshold A, it determines that the determination target pixel is a background pixel (step S12'). On the other hand, if the determination processing unit 12 determines that the difference is larger than the threshold A, it proceeds to the next step S13'.

[0052] In step S13', similar to step S13 described above, the determination processing unit 12 determines whether the target pixel is a background pixel or not based on the determination result for the pixel associated with the target pixel in the frame input immediately before the frame including the target pixel (second determination). If the determination processing unit 12 determined that the pixel associated with the target pixel in the immediately input frame was not a background pixel, the determination processing unit 12 also determines that the target pixel is not a background pixel (step S14'). On the other hand, if the determination processing unit 12 determined that the pixel associated with the target pixel in the immediately input frame was a background pixel, the process proceeds to the next step S15'.

[0053] In step S14', the determination processing unit 12 determines whether the difference calculated in step S10' is smaller than a predetermined threshold B (third determination). Here, threshold B is larger than threshold A. That is, in the third determination, the pixel is determined to be a background pixel more gradually than in the first determination. If the determination processing unit 12 determines that the difference is smaller than threshold B, it determines that the pixel to be determined is a background pixel (step S16'). On the other hand, if the determination processing unit 12 determines that the difference is larger than threshold B, it determines that the pixel to be determined is not a background pixel (step S17').

[0054] This process can prevent a pixel to be determined that is associated with a pixel that was determined to be a background pixel in the immediately preceding frame from being erroneously determined to be a non-background pixel.

[0055] In this modification, the determination may be made using pixels in the (n-1)th to nk-th frames, or may be made in multiple stages using any number of thresholds.

[0056] (Second embodiment) Next, processing by the image processing device 10 according to a second embodiment of the present disclosure will be described with reference to FIGS. 4 and 5. The image processing device 10 according to the second embodiment has a configuration similar to that of the image processing device 10 according to the first embodiment, but differs in the processing performed by the determination processing unit 12. In this embodiment, a determination is made as to whether a pixel to be determined is a background pixel or not using the determination results for the immediately preceding few pixels in the same frame. In other words, in this embodiment, horizontal correction is performed by using the determination results for the immediately preceding few pixels. This makes it possible to reduce processing delay while suppressing erroneous determinations. In other words, it is possible to correctly determine whether a pixel is a background pixel (in other words, whether a pixel is other than a background pixel) and appropriately cut out the image of the target object from the background image. The processing by the determination processing unit 12 will be described in detail below.

[0057] Specifically, the determination processing unit 12: When the difference does not satisfy a predetermined condition in relation to the first threshold value, When a plurality of adjacent pixels have been consecutively determined to be either a detection target pixel or a detection target pixel before determining whether or not a given pixel is a detection target pixel, a pixel whose difference is either larger or smaller than the first threshold value is determined to be either a detection target pixel or a detection target pixel in accordance with the determination of the plurality of adjacent pixels; If, before determining whether a given pixel is a target pixel for detection, a plurality of adjacent pixels are consecutively determined to be either a target pixel for detection or not a target pixel for detection, the difference is compared with a second threshold value different from the first threshold value to determine whether a pixel whose difference does not satisfy a predetermined condition in relation to the first threshold value is a target pixel for detection.

[0058] 4 is a flowchart illustrating the processing flow of the determination processing unit 12. First, in step S20, the determination processing unit 12 calculates the difference between the pixel value of the determination target pixel and the pixel value of a pixel associated with the determination target pixel in the background image.

[0059] Next, in step S21, the determination processing unit 12 determines whether or not the difference calculated in step S20 exceeds a predetermined threshold A (first determination). If the determination processing unit 12 determines that the difference is greater than threshold A, it determines that the pixel to be determined is not a background pixel (step S22). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold A, it proceeds to the next step S23.

[0060] In step S23, the determination processing unit 12 determines whether the target pixel is a background pixel based on the determination results of several pixels immediately preceding the target pixel in the frame including the target pixel (second determination). Specifically, if the immediately preceding pixel is determined to be a background pixel multiple times in succession, the determination processing unit 12 determines the target pixel to be a background pixel (step S24). On the other hand, if the immediately preceding pixel is not determined to be a background pixel multiple times in succession, that is, if the immediately preceding pixel is determined to be a non-background pixel multiple times in succession, the determination processing unit 12 proceeds to the next step S25. Here, the number of pixels immediately preceding the target pixel for determining the target pixel to be a background pixel is any multiple number, and may be two, three, or more. Furthermore, the term "immediately preceding" here refers to a concept with a time range, and the multiple determination results do not necessarily include determination results for pixels adjacent to the target pixel. The determination processing unit 12 may determine whether the target pixel is a background pixel based on the determination results of several pixels immediately preceding the target pixel in a frame other than the frame including the target pixel. Specifically, the determination processing unit 12 may determine whether or not the determination target pixel is a background pixel based on the determination result in a past frame.

[0061] In step S25, the determination processing unit 12 determines whether the difference calculated in step S20 exceeds a predetermined threshold B (third determination). Here, threshold B is smaller than threshold A. That is, in the third determination, the pixel is determined to be a background pixel more strictly than in the first determination. That is, the pixel is determined to be an image of the target object more leniently. If the determination processing unit 12 determines that the difference is larger than threshold B, it determines that the pixel to be determined is not a background pixel (step S26). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold B, it determines that the pixel to be determined is a background pixel (step S27).

[0062] According to this processing, it is possible to correctly determine and cut out portions in the horizontal direction of the frame that could not be cut out in the past, thereby improving accuracy. Details will be described later.

[0063] Furthermore, by combining the processing procedures of the modified example of the first embodiment, it may be possible to easily determine that the target pixel is a background pixel. Furthermore, in this embodiment, the determination is made using two thresholds A and B, but the determination may be made in multiple stages using any number of thresholds.

[0064] The number of pixels referenced in step S23 can be set arbitrarily. The target pixel may be determined to be a background pixel if all the referenced pixels are background pixels, or if a set number or more of the referenced pixels are background pixels.

[0065] Fig. 5 is a diagram for explaining the background determination method according to the second embodiment. Fig. 5(A) shows the determination result by the conventional background subtraction method, and Fig. 5(B) shows the determination result by the determination method according to the second embodiment. That is, Fig. 5(B) shows the determination result obtained when the determination processing unit 12 performs the process of Fig. 4.

[0066] The numbers in the figure correspond to the difference between the pixel value of the pixel to be determined and the pixel value of the pixel associated with the determined gymnastics pixel in the background image. Pixels with underlines under the numbers are parts that are not background pixels (e.g., people), and pixels without underlines under the numbers are parts that are background pixels. In other words, parts with underlines under the numbers are parts that should be determined not to be background pixels, and parts without underlines under the numbers are parts that should be actively determined to be background pixels. Note that Figure 5 shows the case where threshold A = 25 and threshold B = 10.

[0067] In the conventional background subtraction method shown in Figure 5(A), whether a target pixel is a background pixel or not is determined based on only one threshold value A. Therefore, there is a possibility that a pixel that is not a background pixel may be determined to be a background pixel. In Figure 5(A), some pixels with differences less than the threshold value A (=25) are determined to be background pixels even though they are not background pixels.

[0068] On the other hand, in this embodiment, if the previous pixel is not determined to be a background pixel multiple times in a row, the pixel to be determined is again determined to be a background pixel using another threshold value B (=10). Therefore, as shown in Figure 5(B), a pixel whose difference is between threshold values ​​A and B is correctly determined not to be a background pixel.

[0069] Generally, video data is input from the left edge to the right edge at the top of each frame. Therefore, when processing a certain pixel, it is normal for the determination of pixels located above or to the left of the pixel in question to have already been completed. Therefore, in order to achieve low-latency processing, it is effective to determine whether the pixel to be determined is a background pixel using the determination results of the immediately preceding few pixels, as in this embodiment.

[0070] Unlike the above, in the conventional background subtraction method, there is a possibility that a pixel that is a background pixel may not be correctly determined to be a background pixel. In this case, by combining the processing procedure of the modified example of the first embodiment with this embodiment, it is possible to correctly determine that the pixel is a background pixel.

[0071] (Third embodiment) Next, processing by the video processing device 10 according to a third embodiment of the present disclosure will be described with reference to FIGS. 6 and 7. The video processing device 10 according to the third embodiment has a configuration similar to that of the video processing device 10 according to the first embodiment, but differs in the processing by the determination processing unit 12. In this embodiment, a determination result for the top few pixels in the same frame is used to determine whether the pixel to be determined is a background pixel. In other words, in this embodiment, vertical correction is performed by using the determination result for the top few pixels. This makes it possible to reduce processing delay while suppressing erroneous determinations. In other words, it is possible to correctly determine whether a pixel is a background pixel (in other words, whether a pixel is other than a background pixel) and appropriately cut out the image of the target object from the background image. The processing by the determination processing unit 12 will be described in detail below.

[0072] Specifically, the determination processing unit 12: When the difference does not satisfy a predetermined condition in relation to the first threshold value, When a plurality of pixels above a given pixel are consecutively determined to be either detection target pixels or not detection target pixels, a pixel whose difference does not satisfy a predetermined condition in relation to a first threshold is determined to be either detection target pixel or not detection target pixel so as to match the determination of the plurality of pixels above, When a plurality of consecutive pixels above any given pixel are determined to be either the target pixel or not the target pixel, the difference is compared with a second threshold value different from the first threshold value to determine whether the pixel whose difference does not satisfy a predetermined condition in relation to the first threshold value is the target pixel or not.

[0073] 6 is a flowchart illustrating the processing flow of the determination processing unit 12. First, in step S30, the determination processing unit 12 calculates the difference between the pixel value of the determination target pixel and the pixel value of a pixel associated with the determination target pixel in the background image.

[0074] Next, in step S31, the determination processing unit 12 determines whether or not the difference calculated in step S30 exceeds a predetermined threshold value A (first determination). If the determination processing unit 12 determines that the difference is greater than threshold value A, it determines that the pixel to be determined is not a background pixel (step S32). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold value A, the process proceeds to the next step S33.

[0075] In step S33, the determination processing unit 12 determines whether the target pixel is a background pixel based on the determination results of several upper pixels in a frame including the target pixel (second determination). Specifically, if the determination results of the pixels above the target pixel are background pixels multiple times in succession, the determination processing unit 12 determines the target pixel to be a background pixel (step S34). On the other hand, if the determination results of the pixels above the target pixel are not background pixels multiple times in succession, that is, if the determination results of the pixels above the target pixel are not background pixels multiple times in succession, the determination processing unit 12 proceeds to the next step S35. Here, the number of pixels above the target pixel to determine the target pixel as a background pixel is any plural number, and may be two, three, or more. Furthermore, the term "above" here refers to a concept with a spatial range, and the multiple determination results do not necessarily include the determination results of pixels adjacent to the target pixel above. The determination processing unit 12 may determine whether the target pixel is a background pixel based on the determination results of several upper pixels in a frame other than the frame including the target pixel. Specifically, the determination processing unit 12 may determine whether or not the determination target pixel is a background pixel based on the determination result in a past frame.

[0076] In step S35, the determination processing unit 12 determines whether the difference calculated in step S30 exceeds a predetermined threshold B (third determination). Here, threshold B is smaller than threshold A. That is, in the third determination, the pixel is determined to be a background pixel more strictly than in the first determination. That is, the pixel is determined to be an image of the target object more gently. If the determination processing unit 12 determines that the difference is larger than threshold B, it determines that the pixel to be determined is not a background pixel (step S36). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold B, it determines that the pixel to be determined is a background pixel (step S37).

[0077] According to this processing, it is possible to correctly determine and cut out portions in the vertical direction of the frame that could not be cut out in the past, thereby improving accuracy. Details will be described later.

[0078] Furthermore, by combining the processing procedures of the modified example of the first embodiment, it may be possible to easily determine that the target pixel is a background pixel. Furthermore, in this embodiment, the determination is made using two thresholds A and B, but the determination may be made in multiple stages using any number of thresholds.

[0079] The number of pixels to be referenced in step S33 can be set arbitrarily. The target pixel may be determined to be a background pixel if all the referenced pixels are background pixels, or if a set number or more of the referenced pixels are background pixels.

[0080] Fig. 7 is a diagram for explaining the background determination method according to the third embodiment. Fig. 7(A) shows the determination result by the conventional background subtraction method, and Fig. 5(B) shows the determination result by the determination method according to the third embodiment. That is, Fig. 5(B) shows the determination result obtained when the determination processing unit 12 performs the process of Fig. 6.

[0081] In the conventional background subtraction method shown in Figure 7(A), whether a target pixel is a background pixel or not is determined based on only one threshold value A. Therefore, there is a possibility that a pixel that is not a background pixel may be determined to be a background pixel. In Figure 7(A), some pixels with differences less than the threshold value A (=25) are determined to be background pixels even though they are not background pixels.

[0082] On the other hand, in this embodiment, if the upper pixel is not determined to be a background pixel multiple times in a row, the pixel to be determined is again determined to be a background pixel using another threshold value B (=10). Therefore, as shown in Figure 7(B), pixels whose difference is between threshold values ​​A and B are correctly determined not to be background pixels.

[0083] Generally, video data is input from the upper left of each frame. Therefore, when processing a certain pixel, it is normal for the determination of pixels located above and to the left of the pixel to have already been completed. Therefore, in order to achieve low-latency processing, it is effective to determine whether the pixel to be determined is a background pixel using the determination results of the upper few pixels, as in this embodiment.

[0084] Unlike the above, in the conventional background subtraction method, there is a possibility that a pixel that is a background pixel may not be correctly determined as a background pixel. In this case, by combining the processing procedure of the modified example of the first embodiment with this embodiment, it is possible to correctly determine that the pixel is a background pixel.

[0085] (Fourth embodiment) Next, processing by the image processing device 10 according to a fourth embodiment of the present disclosure will be described with reference to FIGS. 8 and 9. The image processing device 10 according to the fourth embodiment has a configuration similar to that of the image processing device 10 according to the first embodiment, but differs in the processing performed by the determination processing unit 12. In this embodiment, a determination is made as to whether a pixel to be determined is a background pixel using a determination result for the immediately preceding few pixels, the upper few pixels, or both in the same frame. In other words, this embodiment combines the horizontal correction in the second embodiment with the vertical correction in the third embodiment. This makes it possible to reduce processing delay while suppressing erroneous determinations. In other words, it is possible to correctly determine whether a pixel is a background pixel (in other words, whether a pixel is other than a background pixel), and appropriately cut out the image of the target object from the background image. The processing by the determination processing unit 12 will be described in detail below.

[0086] 8 is a flowchart illustrating the processing flow of the determination processing unit 12. First, in step S40, the determination processing unit 12 calculates the difference between the pixel value of the determination target pixel and the pixel value of a pixel associated with the determination target pixel in the background image.

[0087] Next, in step S41, the determination processing unit 12 determines whether or not the difference calculated in step S40 exceeds a predetermined threshold A (first determination). If the determination processing unit 12 determines that the difference is greater than threshold A, it determines that the pixel to be determined is not a background pixel (step S42). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold A, it proceeds to the next step S43.

[0088] In step S43, the determination processing unit 12 determines whether the target pixel is a background pixel based on the determination results of the linear few pixels, the upper few pixels, or both in the frame including the target pixel (second determination). In particular, the determination processing unit 12 determines the target pixel to be a background pixel if the immediately preceding pixel has been determined to be a background pixel multiple times in succession and the upper pixel has been determined to be a background pixel multiple times in succession (step S44). On the other hand, if the immediately preceding pixel has not been determined to be a background pixel multiple times in succession or the upper pixel has not been determined to be a background pixel multiple times in succession, the determination processing unit 12 proceeds to the next step S45.

[0089] In step S45, the determination processing unit 12 determines whether the difference calculated in step S40 exceeds a predetermined threshold B (third determination). Here, threshold B is smaller than threshold A. That is, in the third determination, the pixel is determined to be a background pixel more strictly than in the first determination. That is, the pixel is determined to be an image of the target object more leniently. If the determination processing unit 12 determines that the difference is larger than threshold B, it determines that the pixel to be determined is not a background pixel (step S46). On the other hand, if the determination processing unit 12 determines that the difference is smaller than threshold B, it determines that the pixel to be determined is a background pixel (step S47).

[0090] According to this processing, it is possible to correctly determine and cut out portions of the frame that should be cut out in the horizontal and vertical directions, which were previously not possible, thereby improving accuracy. Details will be described later.

[0091] Furthermore, by combining the processing procedures of the modified example of the first embodiment, it may be possible to easily determine that the target pixel is a background pixel. Furthermore, in this embodiment, the determination is made using two thresholds A and B, but the determination may be made in multiple stages using any number of thresholds.

[0092] The number of pixels referenced in step S43 can be set arbitrarily. The target pixel may be determined to be a background pixel if all the referenced pixels are background pixels, or if a set number or more of the referenced pixels are background pixels.

[0093] Fig. 9 is a diagram for explaining the background determination method according to the fourth embodiment. Fig. 9(A) shows the determination result by the conventional background subtraction method, and Fig. 9(B) shows the determination result by the determination method according to the fourth embodiment. That is, Fig. 9(B) shows the determination result obtained when the determination processing unit 12 performs the process of Fig. 8.

[0094] In the conventional background subtraction method shown in Figure 9(A), whether a target pixel is a background pixel or not is determined based on only one threshold value A. Therefore, there is a possibility that a pixel that is not a background pixel may be determined to be a background pixel. In Figure 9(A), some pixels with differences less than the threshold value A (=25) are determined to be background pixels even though they are not background pixels.

[0095] On the other hand, in this embodiment, if the immediately preceding pixel has not been determined to be a background pixel multiple times in succession, or if the upper pixel has not been determined to be a background pixel multiple times in succession, the pixel is again determined to be a background pixel using another threshold value B (=10). Therefore, as shown in Figure 9(B), a pixel whose difference is between threshold values ​​A and B is correctly determined not to be a background pixel.

[0096] Generally, video data is input from the upper left of each frame. Therefore, when processing a certain pixel, it is normal that judgments have already been made on pixels located above and to the left of the pixel. Therefore, in order to achieve low-latency processing, it is effective to determine whether the pixel to be judged is a background pixel by combining the judgment results of the last few pixels and the upper few pixels, as in this embodiment.

[0097] Unlike the above, in the conventional background subtraction method, there is a possibility that a pixel that is a background pixel may not be correctly determined to be a background pixel. In this case, by combining the processing procedure of the modified example of the first embodiment with this embodiment, it is possible to correctly determine that the pixel is a background pixel.

[0098] (Other Examples) The contents of the above-described first to fifth embodiments can be combined as appropriate, thereby providing a video processing device that can reduce processing delays and suppress erroneous determinations.

[0099] Furthermore, the video processing device 10 of the present disclosure may be an integrated circuit such as an FPGA (Field-Programmable Gate Array).

[0100] The device of the present invention can also be realized by a computer and a program, and the program can be recorded on a recording medium or provided via a network. The program of the present disclosure is a program for causing a computer to realize each function of the device according to the present disclosure, and a program for causing a computer to execute each procedure of the method executed by the device according to the present disclosure. [Explanation of symbols]

[0101] 10: Image processing device 11: Input section 12: Judgment processing unit 13: Database

Claims

1. an input unit to which an input image is input; a storage unit for storing the designated storage image; a determination processing unit that compares an arbitrary pixel of the input image with a pixel associated with the arbitrary pixel in the stored image, and determines, based on a result of the comparison and pixel values ​​of pixels surrounding the arbitrary pixel, that the arbitrary pixel has a pixel value within a predetermined range as a detection target pixel; A video processing device comprising:

2. The determination processing unit Calculating a difference between a pixel value of an arbitrary pixel of the input image and a pixel value of a pixel associated with the arbitrary pixel of the stored image; determining that the arbitrary pixel having a pixel value within a predetermined range is a detection target pixel based on the difference, and a pixel value of a pixel adjacent to the arbitrary pixel in an input image including the arbitrary pixel, a pixel value of a pixel located at a position related to the arbitrary pixel in an input image different from the input image including the arbitrary pixel, or a pixel value of a pixel located adjacent to a position related to the arbitrary pixel in the different input image; The video processing device according to claim 1 .

3. When determining whether the arbitrary pixel is a detection target pixel, the determination processing unit comparing the difference with a first threshold; If the difference satisfies a predetermined condition in relation to the first threshold value, the pixel is determined to be a detection target pixel or not a detection target pixel; When the difference does not satisfy the predetermined condition in relation to the first threshold value, When it is determined that the adjacent pixel satisfies either one of being a detection target pixel and not being a detection target pixel, a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value is determined to satisfy either one of being a detection target pixel and not being a detection target pixel in accordance with the determination of the adjacent pixel; when it is determined that the adjacent pixel satisfies either the condition that it is a detection target pixel or that it is not a detection target pixel, by comparing the difference with a second threshold value different from the first threshold value, it is determined whether or not the pixel for which the difference does not satisfy the predetermined condition in relation to the first threshold value is a detection target pixel; The video processing device according to claim 2 .

4. The determination processing unit When the difference does not satisfy the predetermined condition in relation to the first threshold value, if it is determined that a neighboring pixel satisfies either one of being a detection target pixel or not being a detection target pixel before determining whether the arbitrary pixel is a detection target pixel or not, determine that a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value satisfies either one of being a detection target pixel or not being a detection target pixel in accordance with the determination of the neighboring pixel, if it is determined that a neighboring pixel satisfies either the other of being a detection target pixel or not being a detection target pixel before determining whether or not the arbitrary pixel is a detection target pixel, by comparing the difference with the second threshold value different from the first threshold value, the difference is determined to be a pixel that does not satisfy the predetermined condition in relation to the first threshold value. The video processing device according to claim 3 .

5. The determination processing unit When the difference does not satisfy the predetermined condition in relation to the first threshold value, if a plurality of adjacent pixels have been consecutively determined to be either a detection target pixel or a detection target pixel before determining whether the arbitrary pixel is a detection target pixel, determining a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value as either a detection target pixel or a detection target pixel in accordance with the determination of the plurality of adjacent pixels, when a plurality of adjacent pixels have been consecutively determined to satisfy either one of being a detection target pixel or not being a detection target pixel before determining whether the arbitrary pixel is a detection target pixel, by comparing the difference with a second threshold value different from the first threshold value, the difference is used to determine whether a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value is a detection target pixel; The video processing device according to claim 3 .

6. The determination processing unit When the difference does not satisfy the predetermined condition in relation to the first threshold value, When a plurality of pixels above the arbitrary pixel are consecutively determined to be either a detection target pixel or a detection target pixel, a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold is determined to be either a detection target pixel or a detection target pixel so as to match the determination of the plurality of pixels above, when a plurality of consecutive pixels above the given pixel are determined to be either a detection target pixel or a detection target pixel, the difference is compared with the second threshold value, which is different from the first threshold value, to determine whether or not a pixel whose difference does not satisfy the predetermined condition in relation to the first threshold value is a detection target pixel; The video processing device according to claim 3 .

7. A video processing method executed by a video processing device including an input unit to which an input image is input, a storage unit to store a specified storage image, and a determination processing unit, The determination processing unit: comparing a given pixel of the input image with a pixel associated with the given pixel in the stored image; determining, based on the result of the comparison and pixel values ​​of pixels adjacent to the arbitrary pixel, that the arbitrary pixel has a pixel value within a predetermined range as a detection target pixel; A video processing method including:

8. A program for causing a computer to realize each functional unit of the video processing device according to any one of claims 1 to 6.

Images