Determination method, determination program, and information processing apparatus

Abstract

A computer acquires a captured image including an image region of a person captured by a camera. The computer determines an image region other than the image region of the person from the acquired captured image. The computer determines whether the captured image is an image in which a display object of the person is captured, based on a distribution of motions of a plurality of positions included in the determined image region.

Description

Technical Field

[0001] This invention relates to a technique for image determination. Background Technology

[0002] Biometric authentication technology uses biometric features such as fingerprints, facial features, and vein patterns to verify an individual's identity. In biometric authentication, when verification is required, the acquired biometric features are compared with pre-registered biometric features to determine if they match, thus confirming the individual's identity.

[0003] Facial recognition technology, as a type of biometric authentication technology, has attracted much attention as a method for contactless identity verification. It can be used in various applications, such as access management on personal terminals like PCs or smartphones, access control, and identity verification at airport gates.

[0004] The facial image information used as a biometric feature in this facial authentication technology differs from that used in other biometric authentication technologies such as fingerprint or palm vein authentication. It does not require special sensors and can be obtained through ordinary camera photography. Furthermore, facial images are frequently published on the Internet via social networking services (SNS). Therefore, there are concerns that others may impersonate individuals by displaying publicly available photos of their faces on cameras or smartphones showing such images. Therefore, several techniques are proposed to determine whether a camera-captured image is an image of a person (the person actually located at the shooting location) or a photograph of a person, a display showing a person, or other similar image.

[0005] At first glance, it is difficult to distinguish between a photograph or a display image showing one's face and a pre-registered facial image used as authentication information. Therefore, a method has been proposed to capture the characteristics of the object being photographed using infrared images obtained by an infrared camera or three-dimensional information obtained by a depth camera, etc. (for example, see Patent Documents 1 to 3).

[0006] Furthermore, when the image captured is an image of a person displayed on a device, such a display cannot respond to requests on the spot. Taking advantage of this, techniques have been proposed for instructing the person being authenticated to input a prescribed action, for observing the person's response to the device's display, and even for determining whether a person is a living being by detecting the actions of a natural person (such as blinking) (see, for example, Patent Documents 4 to 9).

[0007] Furthermore, several techniques have been proposed to determine whether an image is of a person or an object by utilizing features of the image region of a person in the captured image and features of the image region outside the person's image region (the image region of the background). More specifically, for example, a technique has been proposed to identify an object as a non-living object when the feature quantity of the region outside the person's image, i.e., the background region, changes by a predetermined value or more. In addition, for example, a technique has also been proposed to determine whether the subject is a photograph or a person by using the similarity of the motion feature quantities of the facial region and the background region in the captured image (for example, see Patent Documents 10 to 12).

[0008] In addition, several techniques for image determination were proposed.

[0009] For example, techniques for detecting image regions of objects and image regions of human faces from captured images have been proposed (see, for example, Non-Patent Literature 1 to Non-Patent Literature 4).

[0010] In addition, for example, a technique has been proposed to extract motion from an image by using optical flow obtained from the changes in the brightness gradient of each pixel of the image constituting the time series (see, for example, Non-Patent Document 5).

[0011] Patent Document 1: International Publication No. 2009 / 107237

[0012] Patent Document 2: Japanese Patent Application Publication No. 2005-259049

[0013] Patent Document 3: International Publication No. 2009 / 110323

[0014] Patent Document 4: Japanese Patent Application Publication No. 2016-152029

[0015] Patent Document 5: International Publication No. 2019 / 151368

[0016] Patent Document 6: Japanese Patent Application Publication No. 2008-000464

[0017] Patent Document 7: Japanese Patent Application Publication No. 2001-126091

[0018] Patent Document 8: Japanese Patent Application Publication No. 2008-090452

[0019] Patent Document 9: Japanese Patent Application Publication No. 2006-330936

[0020] Patent Document 10: Japanese Patent Application Publication No. 2010-225118

[0021] Patent Document 11: Japanese Patent Application Publication No. 2006-099614

[0022] Patent Document 12: Japanese Patent Application Publication No. 2016-173813

[0023] Non-patent literature 1: Hengshuang Zhao et al., “Pyramid Scene Parsing Network”, Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 2881-2890

[0024] Non-patent literature 2: Wei Liu et al., “SSD: Single Shot MultiBox Detector”, European Conference on Computer Vision (ECCV) 2016, Springer International Publishing, 2016, pp. 21-37

[0025] Non-patent literature 3: Joseph Redmon et al., "You Only Look Once: Unified, Real-Time Object Detection", 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779-788.

[0026] Non-patent document 4: Kaipeng Zhang et al., "Joint Face Detection and Alignmentusing Multi-task Cascaded Convolutional Networks", IEEE Signal Processing Letters (SPL), Volume 23, Issue 10, Oct. 2016, p. 1499-1503

[0027] Non-patent document 5: Gunnar Farneback, "Two-Frame Motion Estimation Based on Polynomial Expansion" In Proceedings of the 13th Scandinavian Conference onImage Analysis (SCIA 2003), 2003, p.363-370

[0028] There are instances of image shakiness during facial recognition implementation. This can occur, for example, when a laptop is used on one's lap inside a vehicle such as a train, or when the camera is not securely fixed and shakes due to surrounding vibrations. If such camera shake exists in the captured image, the accuracy of determining whether the image depicts a person or an object may be reduced.

[0029] As described above, a technique has been proposed to identify an object as a non-living object when the feature quantity in the background region (excluding the human figure area) of an image changes by a predetermined value or more. This technique focuses on detecting changes in the feature quantity of the background region when the image depicts a real object containing a human figure, ensuring that the feature quantity remains almost unchanged. However, this technique also detects changes in the feature quantity of the background region in images with shaky footage, as described above. Therefore, in cases where the image is shaky, there is a possibility that this technique might misidentify the object as a non-living object even if it is a living being.

[0030] Furthermore, as mentioned above, a technique has been proposed to determine whether a photograph or a person is the subject by using the similarity of motion features between the facial region and the background region in the captured image. This technique focuses on the motion linkage between the facial region and the background region in an image obtained from a photograph containing a person, and performs the aforementioned determination by detecting this linkage. However, motion linkage between the facial region and the background region can occur in images with shaky footage, as described above. Therefore, in cases where the image is shaky, there is a possibility that this technique might mistakenly identify an image of a person as a photograph even if the image is of a real object. Summary of the Invention

[0031] In one respect, the purpose of this invention is to improve the accuracy of determining whether a captured image is an image of a display object that has captured a person.

[0032] In one approach, a computer acquires a photographic image of an image region containing a person, captured by a camera. The computer then identifies image regions outside the person's image region from the acquired image. Based on the distribution of motion at multiple locations within the identified image region, the computer determines whether the photographic image is indeed an image of a display containing a person.

[0033] The accuracy of determining whether an image captured from one side is a display of a person has been improved. Attached Figure Description

[0034] Figure 1 It is a diagram illustrating the various image regions of a captured image.

[0035] Figure 2A This is a diagram illustrating the synchronous / asynchronous movement of different image regions in a captured image when camera shake occurs during shooting (one of the diagrams).

[0036] Figure 2B This is a diagram illustrating the synchronous / asynchronous movement of different image regions in a captured image when camera shake occurs during shooting (Part Two).

[0037] Figure 3 This is a diagram illustrating the configuration of an example information processing device.

[0038] Figure 4 This is a diagram illustrating an example of computer hardware configuration.

[0039] Figure 5 This is a flowchart representing the processing steps involved in determining and handling captured images.

[0040] Figure 6 This is a flowchart representing the processing steps involved in determining an image region.

[0041] Figure 7A This is an example diagram illustrating the method for determining the area of ​​a person (one of the examples).

[0042] Figure 7B This is an example diagram illustrating the method for determining the area of ​​a person (Part Two).

[0043] Figure 8 This is a diagram illustrating an example of a method for determining the background area.

[0044] Figure 9 This is a flowchart representing the processing content of motion extraction.

[0045] Figure 10 This is a flowchart representing the processing content of the judgment and handling.

[0046] Figure 11This is a diagram illustrating an example of using multiple sets of captured image pairs to obtain the motion vectors of an image. Detailed Implementation

[0047] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings.

[0048] In this embodiment, it is determined whether the captured image is an image of a display object that has captured a person, based on the distribution of motion at multiple positions within the image area outside the image area of ​​the person in the captured image. This method will be described.

[0049] In this embodiment, firstly, each image region is detected from the captured image obtained by the camera.

[0050] Figure 1 This is a diagram illustrating the various image regions of the captured image 10. In this embodiment, the surrounding area 11, the person area 12, and the background area 13 are detected from the captured image 10.

[0051] The peripheral region 11 is the area surrounding the captured image 10, a ring-shaped area with the edge of the captured image 10 as its outer perimeter. The person region 12 and the background region 13 are both areas surrounded by the inner perimeter of the peripheral region 11. The person region 12 is the image area showing a person. The background region 13, on the other hand, is the area outside the person region 12, showing objects other than the person.

[0052] When image 10 is a photograph of a person, the person is displayed in the person area 12, and the actual background of the person at the time image 10 was taken is displayed in both the background area 13 and the surrounding area 11. The surrounding area 11 displays the scenery surrounding the background shown in the background area 13.

[0053] On the other hand, when the captured image 10 is an image of a person on the display, the image displayed on the display at the time the captured image 10 is taken is displayed on both the person area 12 and the background area 13, and the surrounding scenery of the display at the time the captured image 10 is taken is displayed in the surrounding area 11. Specifically, the person area 12 displays the image of the person shown on the display, and the background area 13 displays the image of the background shown on the display along with the person.

[0054] When camera shake occurs during the photograph of a person, the movement of the images in the surrounding area 11 and background area 13, which both display the actual background of the person, is synchronized. On the other hand, the movement of the images in the person area 12 and background area 13 is not synchronized. Conversely, when camera shake occurs during the photograph of a displayed object, the movement of the images in the person area 12 and background area 13, which both display the content of the displayed object, is synchronized. On the other hand, the movement of the images in the surrounding area 11, which displays the scenery around the displayed object, is not synchronized with the movement of the images in the background area 13. Figure 2A as well as Figure 2B The synchronous / asynchronous motion of each image region in the captured image 10 under such circumstances of camera shake will be explained.

[0055] exist Figure 2A as well as Figure 2B In the graph, the solid line represents the movement of the difference vector of the captured image 10 obtained by photographing the person, and the dashed line represents the movement of the difference vector of the captured image 10 obtained by photographing the display object.

[0056] Figure 2A as well as Figure 2B The horizontal axis in each chart represents the time when image 10 was captured. Figure 2A In the chart, the difference vector between the motion vector representing the movement of the character area 12 and the motion vector representing the movement of the background area 13 is shown on the vertical axis. On the other hand, in Figure 2B In the graph, the magnitude of the difference vector between the motion vector representing the motion of the peripheral region 11 and the motion vector representing the motion of the background region 13 is shown on the vertical axis.

[0057] When the motion of the two regions in the captured image 10 is synchronized, the magnitude of the difference vector of the motion of the two regions decreases; when the motion of the two regions is not synchronized, the magnitude of the difference vector of the motion of the two regions increases.

[0058] exist Figure 2A In the diagram, the difference vector of the image 10 showing the display object is smaller, while the difference vector of the image 10 showing the person is larger. Therefore, it can be seen that the movement of the person area 12 and the background area 13 in the image 10 showing the display object is almost synchronized, while the movement of the person area 12 and the background area 13 in the image 10 showing the person is not synchronized.

[0059] In contrast, Figure 2BIn the diagram, the difference vector of the photographed image 10 showing a person is smaller, while the difference vector of the photographed image 10 showing a display object is larger. Therefore, it can be seen that the movement of the peripheral region 11 of the photographed image 10 showing a person is almost synchronized with the movement of the background region 13, while the movement of the peripheral region 11 of the photographed image 10 showing a display object is not synchronized with the movement of the background region 13.

[0060] In this embodiment, focusing on the synchronous / asynchronous relationship of the motion of each image region in such a jittery captured image 10, it is determined whether the captured image 10 is an image of the display object based on the distribution of motion of each position contained in each image region.

[0061] Next, the configuration of the device for determining whether the captured image 10 is an image of a display object that has captured a person will be explained. Figure 3 The configuration of the illustrated information processing device 20 is shown.

[0062] A camera 30 is connected to the information processing device 20. The camera 30 captures images of the subject and outputs captured images 10. The subject of the camera 30 is a person; for example, in the case of facial authentication, the camera 30 captures the face of the person being authenticated. Furthermore, the camera 30 repeatedly captures images of the subject and outputs a time-series of captured images 10. The time-series captured images 10 are used to extract motion in different regions of the captured images 10.

[0063] The information processing device 20 includes an image acquisition unit 21, a region determination unit 22, a motion extraction unit 23, and a determination unit 24 as its constituent elements.

[0064] The image acquisition unit 21 acquires and stores the captured images 10 taken by the camera 30 in advance.

[0065] The region determination unit 22 determines the region to be used based on the captured image 10 acquired by the image acquisition unit 21. Figure 1 The image regions to be described, more specifically, define the figure region 12 and the regions outside the figure region 12 (surrounding region 11 and background region 13).

[0066] The motion extraction unit 23 extracts the motion of each image region determined by the region determination unit 22 from the captured image 10, and obtains the distribution of motion at each position contained in each image region.

[0067] The determination unit 24 determines whether the captured image 10 is an image of a person or a display object based on the distribution of motion at each position in each image region acquired by the motion extraction unit 23.

[0068] In addition, it can also be constructed through a combination of computers and software. Figure 3 Information processing device 20.

[0069] Figure 4 An example of the hardware configuration of computer 40 is shown.

[0070] The computer 40 includes, for example, a processor 41, a memory 42, a storage device 43, a reading device 44, a communication interface 46, and an input / output interface 47 as its constituent hardware components. These constituent components are connected via a bus 48, enabling data exchange between them.

[0071] Processor 41 can be a single processor, a multi-processor, or a multi-core processor. Processor 41 uses memory 42 to execute, for example, an image determination processing program that describes the sequence of image determination processing described later.

[0072] The memory 42 is, for example, a semiconductor memory, which may include RAM and ROM regions. The storage device 43 is, for example, a hard disk, flash memory, or other semiconductor memory, or an external storage device. Furthermore, RAM is short for Random Access Memory. ROM is short for Read Only Memory.

[0073] The reading device 44 accesses the removable storage medium 45 according to the instructions of the processor 41. For example, the removable storage medium 45 can be implemented using semiconductor devices (USB memory, etc.), media that input and output information via magnetic means (disk, etc.), or media that input and output information via optical means (CD-ROM, DVD, etc.). Furthermore, USB stands for Universal Serial Bus; CD stands for Compact Disc; and DVD stands for Digital Versatile Disk.

[0074] Communication interface 46 sends and receives data via a communication network (not shown), for example, according to the instructions of processor 41.

[0075] The input / output interface 47 acquires various data, such as image data of the captured image 10 sent from the camera 30. Additionally, the input / output interface 47 outputs the results of the captured image determination processing, described later, from the processor 41.

[0076] For example, a program that can be executed by the processor 41 of the computer 40 can be provided in the following manner.

[0077] (1) Pre-installed in storage device 43.

[0078] (2) Provided via removable storage medium 45.

[0079] (3) Provided to the communication interface 46 from the program server and other servers via the communication network.

[0080] Furthermore, the hardware configuration of computer 40 is exemplified, and the implementation method is not limited to this. For example, some or all of the functions of the above-mentioned functional units can also be implemented as hardware based on FPGA and SoC. In addition, FPGA is short for Field Programmable Gate Array. SoC is short for System-on-a-chip.

[0081] Next, the image processing and judgment procedures will be explained. Figure 5 This is a flowchart illustrating the processing steps involved in determining and resolving the captured image. Figure 4 The combination of computer 40 and software constitutes Figure 3 In the case of the information processing device 20, the processor 41 is made to execute the image determination program that describes the image determination process.

[0082] exist Figure 5 First, in S101, image acquisition processing is performed. This processing involves acquiring the time-series captured images 10 sent from the camera 30 via the input / output interface 47 and storing them in the memory 42. Furthermore, in this embodiment, it is assumed that the outer perimeter of the captured image 10 is a horizontally elongated rectangle. In the following description, the direction of the long side of this rectangle is defined as the horizontal direction of the captured image 10. Additionally, the direction of the short side of the rectangle (the direction orthogonal to the horizontal direction of the captured image 10) is defined as the vertical direction of the captured image 10, the direction of the head of the person shown in the captured image 10 is defined as the vertical direction of the captured image 10, and the direction of the person's torso is defined as the vertical direction of the captured image 10.

[0083] Processor 41 provides by executing the processing of S101. Figure 3 The function of the image acquisition unit 21.

[0084] Next, image region determination processing is performed in S102. This processing involves determining the person region 12 and the regions outside the person region 12 (surrounding region 11 and background region 13) from the captured image 10 obtained through the processing in S101. The details of this processing will be described later.

[0085] Next, motion extraction processing is performed in S103. This process extracts the motion of each image region determined by the processing in S102 from the captured image 10, and obtains the distribution of motion at each location within each image region. The details of this process will be described later.

[0086] Next, a determination process is performed in S104. This process extracts the motion of each image region determined by the process in S102 from the captured image 10, and obtains the distribution of motion at each position within each image region. The details of this process will be described later.

[0087] If the processing in S104 ends, the image capture determination process ends.

[0088] Next, for Figure 5 The processing of S102, namely the image region determination process, will be explained in detail. Figure 6 This is a flowchart illustrating the processing content of the image region determination process. Processor 41 provides [the following information] by executing this image region determination process. Figure 3 The function of the region determination section 22.

[0089] exist Figure 6 First, in S201, a process is performed to determine the peripheral region 11 in each of the time-series captured images 10 stored in the memory 42. In this process, the region of the outer periphery of the captured image 10, and specifically the annular region having the edge of the rectangular captured image 10 as the outer periphery and having the inner periphery of the rectangle as the inner periphery, is determined as the peripheral region 11.

[0090] Furthermore, if the width of the ring forming the surrounding area 11 is too wide, other areas will become narrower, and the accuracy of the judgment of the captured image 10 will decrease. Therefore, it is preferable to determine the value that can sufficiently obtain the required judgment accuracy beforehand through experiments and set this width. In this embodiment, the value of this width is set to five percent of the length of the horizontal width of the captured image 10.

[0091] Next, in S202, the process of determining the person region 12 in each of the time-series captured images 10 stored in the memory 42 is performed. Many techniques are known as methods for determining the person region from an image, and any of these known techniques can be used for the process in S202.

[0092] For example, techniques for semantic segmentation, such as extracting pixels corresponding to a person within an image, are known. Methods for implementing semantic segmentation, such as those using Convolutional Neural Networks (CNNs), are known. The "Pyramid Scene Parsing Network" (PSPNet) proposed in the aforementioned Non-Patent Document 1 is an example of a method for semantic segmentation using CNNs. As part of the processing in S202, PSPNet can also be used to determine the person region 12 from the region surrounded by the inner periphery of the surrounding region 11 in the captured image 10.

[0093] Additionally, techniques for detecting rectangular regions (also known as bounding boxes) of objects from images are known, for example. Methods using CNNs are also known for implementing this rectangular region detection. For instance, "Single Shot MultiBox Detector" (SSD) proposed in Non-Patent Document 2 and "You Only Look Once" (YOLO) proposed in Non-Patent Document 3 are examples of methods using CNNs to detect such rectangular regions. Furthermore, "Multi-task Cascaded Convolutional Networks" (MTCNN) proposed in Non-Patent Document 4 is also an example of a method for detecting such rectangular regions, but this MTCNN is specifically designed for detecting facial regions. As part of the processing in S202, any of these techniques for detecting rectangular regions can be used to determine the person region 12 from the region surrounded by the inner periphery of the surrounding region 11 in the captured image 10.

[0094] Furthermore, when using semantic segmentation methods such as PSPNet for determination, such as Figure 7A The region shown, which is enclosed by the inner perimeter of the surrounding region 11 and contains the body parts of a person, including the head and torso, is designated as the person region 12. On the other hand, when detecting rectangular regions using methods such as SSD, YOLO, or MTCNN, the rectangular region containing the head of the person, which is enclosed by the inner perimeter of the surrounding region 11, is designated as the face region 14. In this case, as... Figure 7B As shown, the area encompassed by the rectangle that extends the rectangle of the face region 14 downwards in the captured image 10 until it connects with the inner periphery of the surrounding region 11 can also be defined as the person region 12, so that a part of the person's body is also included in the person region 12.

[0095] continue Figure 6The flowchart is explained below. Following S202, in S203, a process is performed to determine the background region 13 in each of the time-series captured images 10 stored in the memory 42. In this process, the remaining regions in the captured images 10, excluding the peripheral region 11 determined by the process in S201 and the person region 12 determined by the process in S202, are defined as the background region 13.

[0096] Furthermore, in the processing of S202, when the rectangle of the face region 14 is extended downwards towards the captured image 10 to determine the person region 12, if the remaining region is entirely determined as the background region 13 as described above, then a part of the person's body (shoulders, etc.) will be included in the background region 13. Therefore, in this case, if... Figure 8 As shown, the rectangular area in the captured image 10 that connects horizontally to the inner perimeter of the peripheral area 11 and the rectangle of the person area 12 can be defined as the background area 13. Furthermore, the lower end of this background area 13 in the captured image 10 can be aligned with the lower side of the rectangle of the face area 14. If the background area 13 is defined in this way, the area of ​​the person's body included in the background area 13 is reduced.

[0097] If the processing in S203 ends, the image region determination process ends, and the processor 41 returns the processing to... Figure 5 Image capture and processing.

[0098] The above processing steps are for determining the image region.

[0099] Next, for Figure 5 The processing of S103, i.e., motion extraction processing, will be explained in detail. Figure 9 This is a flowchart illustrating the processing content of motion extraction. Processor 41 provides [the following information] by executing this motion extraction process. Figure 3 The function of motion extraction unit 23.

[0100] exist Figure 9 First, in S301, a process is performed to acquire the motion vectors of each pixel constituting the captured image 10. In this process, a process based on... Figure 5 The S101 process extracts the motion vector of the brightness gradient change of two captured images in the time series captured images 10 stored in the memory 42.

[0101] Many techniques are known for extracting motion vectors from images, and any of these known techniques can be used for processing S301. For example, optical flow techniques are widely known. Various methods for calculating optical flow are known, including correspondence establishment based on correlation (block matching), correspondence establishment based on gradient methods, and correspondence establishment utilizing feature point tracking. The method proposed in Non-Patent Document 5 is also an example of an optical flow calculation method. As part of processing S301, the optical flow calculated using the method proposed in Non-Patent Document 5 can be used to obtain a two-dimensional motion vector for each pixel of the captured image 10.

[0102] Next, in S302, the average vector of the surrounding region 11 is calculated. This process calculates the average of all pixels of the motion vectors acquired from each pixel of the captured image 10 within the surrounding region 11, as processed in S301. The average vector vp calculated by this process is an example of a motion vector representing the motion of the location within the surrounding region 11.

[0103] The average vector vp of the surrounding area 11 of the captured image 10 is a two-dimensional vector. In this embodiment, the horizontal (x-direction) average vector vp component vpx and the vertical (y-direction) component vpy in the captured image 10 are calculated by performing the following [Symbol 1] calculation.

[0104] [Number 1]

[0105]

[0106]

[0107] Furthermore, in Equation [1], vx(i,j) and vy(i,j) are the x-component and y-component values ​​of the motion vector of the pixel (pixels contained in the peripheral region 11) determined at its position (i,j) in the two-dimensional coordinates defined by the x and y directions of the captured image 10, respectively. Additionally, np is the number of pixels contained in the peripheral region 11. In other words, Equation [1] represents the calculation of the components vpx and vpy of the average vector vp by dividing the sum of each component of the x-component and y-component of the motion vector of each pixel contained in the peripheral region 11 by the number of pixels in the peripheral region 11, respectively.

[0108] Next, in S303, the average vector of the person region 12 is calculated. This process calculates the average of all pixels of the motion vectors acquired from each pixel of the captured image 10 contained within the person region 12, as described in S301. The average vector vf calculated by this process is an example of a motion vector representing the movement of the position contained within the person region 12. Furthermore, the method for calculating the average vector vf of the person region 12 can be the same as the method for calculating the average vector vp of the surrounding region 11, as explained in S302.

[0109] Next, in S304, the average vector of the background region 13 is calculated. This process calculates the average of all pixels of the motion vectors acquired from each pixel of the captured image 10 contained within the background region 13, as described in S301. The average vector vb calculated by this process is an example of a motion vector representing the motion of the location contained within the background region 13. Furthermore, the method for calculating the average vector vb of the background region 13 can be the same as the method for calculating the average vector vp of the surrounding region 11, as described in S302.

[0110] If the processing of S304 ends, the motion extraction process ends, and the processor 41 returns the processing to... Figure 5 Image capture and processing.

[0111] The above processing steps are motion extraction processing.

[0112] In addition, Figure 9 In the calculation of the average vector in each of the processes S302, S303, and S304, there are cases where the region to which the average vector is calculated contains pixels with only minimal motion (the magnitude of the motion vector is close to zero). For example, pixels in a region with the same brightness have little difference in brightness from their surrounding pixels, so the brightness gradient is not visible. Therefore, there are cases where, although the motion is relatively large, only a tiny motion is detected. The accuracy of the average vector calculated using the motion vectors of such pixels as a vector representing the motion of the region to be calculated is reduced. Therefore, pixels whose motion vector magnitude obtained in the S301 process is smaller than a specified value can be removed from the pixels used in the calculation of the average vector.

[0113] In addition, Figure 9In the flowchart, the motion vectors of the images in each pixel constituting the captured image 10 are obtained through processing in S301, and the average vector of the pixels contained in each region is calculated through subsequent processing in S302, S303, and S304. Alternatively, the captured image 10 can be divided into regions, and then the motion vectors of the images in each pixel contained in the segmented captured image 10 are obtained, and the average vector of each region is calculated.

[0114] Next, for Figure 5 The processing of S104, i.e. the judgment processing, will be explained in detail. Figure 10 This is a flowchart illustrating the processing content of the decision-making process. Processor 41 provides [services / equipment] by executing this decision-making process. Figure 3 The function of the determination unit 24.

[0115] exist Figure 10 First, in S401, the first difference vector is calculated. The first difference vector vdiff1 is the difference between the motion vector representing the movement of the position contained in the character area 12 and the motion vector representing the movement of the position contained in the background area 13. In this embodiment, the difference vector is calculated by performing the following calculation [Equation 2].

[0116] [Number 2]

[0117] vdiff1=vf-vb=(vfx-vbx,vfy-vby)

[0118] Furthermore, in equation [2], vf and vb are the average vectors of the character region 12 and the background region 13, respectively. Additionally, vfx and vfy are the x and y components of the average vector vf of the character region 12, respectively, and vbx and vby are the x and y components of the average vector vb of the background region 13, respectively.

[0119] The first difference vector vdiff1 calculated in this way is an example of an indicator representing the difference between the motion of the position contained in the background region 13 and the motion of the position contained in the character region 12, and is an example of information representing the distribution of motion of these two positions.

[0120] Next, in S402, the second difference vector is calculated. The second difference vector vdiff2 is the difference between the motion vector representing the motion of the position contained in the background region 13 and the motion vector representing the motion of the position contained in the surrounding region 11. In this embodiment, it is calculated by performing the following calculation using the formula [3].

[0121] [Number 3]

[0122] vdiff2=vb-vp=(vbx-vpx,vby-vpy)

[0123] Furthermore, in Equation [3], vb and vp are the average vectors of the background region 13 and the surrounding region 11, respectively. Additionally, vbx and vby are the x and y components of the average vector vb of the background region 13, respectively, and vpx and vpy are the x and y components of the average vector vp of the surrounding region 11, respectively.

[0124] The second difference vector vdiff2 calculated in this way is an example of an indicator representing the difference between the motion of the location contained in the background region 13 and the motion of the location contained in the surrounding region 11, and is an example of information representing the distribution of motion of these two locations.

[0125] Next, in S403, it is determined whether the magnitude of the first difference vector vdiff1 calculated by the processing in S401 is above the first threshold.

[0126] The magnitude of the first difference vector vdiff1 is calculated by taking the square root of the sum of the squares of the x-components and y-components of the first difference vector vdiff1.

[0127] The first threshold is a pre-set value. For example, the average vector VB of the background region 13 in the shaken image 10 of the display containing people is estimated in advance through multiple experiments, and about half of the estimated value is set as the first threshold.

[0128] In the processing of S403, if it is determined that the magnitude of the first difference vector vdiff1 is above the first threshold (when the determination result is yes), it is considered that the movement of the background region 13 is asynchronous with the movement of the character region 12, and the processing proceeds to S404.

[0129] In S404, it is performed as Figure 10 The result of the judgment processing is used to determine that the captured image 10 is an image of a real person.

[0130] On the other hand, in the processing of S403, if it is determined that the size of the first difference vector vdiff1 is smaller than the first threshold (if the determination result is no), the processing will proceed to S405.

[0131] In S405, a process is performed to determine whether the magnitude of the second difference vector vdiff2 calculated through the processing in S402 is above the second threshold.

[0132] The magnitude of the second difference vector vdiff2 is calculated by taking the square root of the sum of the squares of the x-components and y-components of the second difference vector vdiff2.

[0133] The second threshold is a pre-set value. For example, the average vector VB of the background region 13 in the shaken image 10 of the display containing people is estimated in advance through multiple experiments, and about half of the estimated value is set as the second threshold.

[0134] In the processing of S405, if it is determined that the magnitude of the second difference vector vdiff2 is above the second threshold (when the determination result is yes), it is considered that the movement of the background region 13 is asynchronous with the movement of the surrounding region 11, and the processing proceeds to S406.

[0135] In S406, it is performed as Figure 10 The result of the judgment processing is used to determine that the captured image 10 is an image of a person on a display.

[0136] On the other hand, in the processing of S405, if it is determined that the magnitude of the second difference vector vdiff2 is smaller than the second threshold (when the determination result is negative), it is considered that the movement of the background region 13 is synchronized with the movement of the surrounding region 11, and the processing proceeds to S404. Therefore, in S404, the process is performed as... Figure 10 The result of the judgment processing is used to determine that the captured image 10 is an image of a real person.

[0137] If the processing in S404 or S406 ends, the process proceeds to S407. In S407, the result of the decision made through the processing in S404 or S406 is used as... Figure 5 The processing result of the captured image is determined and output from the input / output interface 47.

[0138] If the processing of S407 ends, the processing is determined to be complete, and the processor 41 returns the processing to... Figure 5 Image processing and judgment.

[0139] The above processing steps constitute a judgment process.

[0140] The above image determination process is performed by processor 41. Figure 4 Computer 40 as Figure 3 The information processing device 20 operates and can accurately determine whether the captured image 10 is an image of a display object that has captured a person.

[0141] The disclosed embodiments and their advantages have been described in detail above, but those skilled in the art can certainly make various changes, additions, and omissions without departing from the scope of the invention as expressly stated in the claims.

[0142] For example, in Figure 9 In the motion detection processing S301, two time-series captured images 10 are used to acquire the motion vectors of each pixel constituting the captured image 10. Alternatively, this could be replaced by... Figure 11 As illustrated, multiple pairs of captured images 10, consisting of two images from a time-series image 10, are used. For each pair, the motion vector of each pixel is acquired, and the average of the resulting multiple motion vectors is taken as the motion vector of each pixel. Furthermore, Figure 11 This example illustrates how, for each pair of four captured images 10, the motion vectors of each pixel are acquired, and the average of the four motion vectors calculated for each pixel is used to obtain the motion vectors of the image constituting each pixel of the captured image 10. This improves the accuracy of the acquired image motion vectors.

[0143] In addition, a moving average can also be calculated when the average of the motion vectors obtained for each pair of time-series captured images 10 is calculated, as described above, since each pixel is a motion vector of the image.

[0144] Furthermore, when calculating the average of the motion vectors obtained for each pair of captured images 10 in the time series, since the area of ​​each region is different in each frame of captured image 10, a weighted average corresponding to the area of ​​each region can also be calculated.

[0145] In addition, Figure 11 In the example, four image pairs are formed by taking five consecutive frames of images 10 in a time series and having two adjacent frames as a pair. Alternatively, the two frames forming an image pair could be separated by several frames instead of being adjacent. This increases the difference between the two frames forming an image pair, so even if the camera 30 captures images at a very high frame rate, motion stability in the detected images may still be achieved.

[0146] Furthermore, in the above embodiments, it is assumed that a general camera is used, as a counterpart to... Figure 3 The information processing device 20 is connected to the camera 30. However, even if the captured image 10 is a grayscale image, the motion vector of the image can be obtained. Therefore, an infrared camera or a depth camera capable of outputting grayscale images can also be used as the camera 30.

[0147] Explanation of reference numerals in the attached figures

[0148] 10…Image capture, 11…Surrounding area, 12…Person area, 13…Background area, 14…Face area, 20…Information processing device, 21…Image acquisition unit, 22…Area determination unit, 23…Motion extraction unit, 24…Determination unit, 30…Camera, 40…Computer, 41…Processor, 42…Memory, 43…Storage device, 44…Reading device, 45…Removable storage medium, 46…Communication interface, 47…Input / output interface, 48…Bus.

Claims

1. A determination method, characterized in that, Computer execution: Acquire images of the area containing people captured by a camera; From the acquired captured image, an image region including a peripheral region and a background region is determined as the image region other than the image region of the person. The peripheral region is a ring-shaped region with the edge of the captured image as its outer perimeter, and the background region is the region other than the image region of the person, which is surrounded by the inner perimeter of the peripheral region. A first determination is made, which is based on the distribution of movement between a first position contained in the background area and a third position contained in the image area of ​​the person, to determine whether the captured image is an image of the actual person; and If, in the first determination above, it is determined that the captured image is not an image of the actual person, a second determination is made. The second determination is based on the distribution of movement between the first position and the second position included in the surrounding area to determine whether the captured image is an image of the displayed object that captured the person.

2. The determination method according to claim 1, characterized in that, The second determination is made based on the difference between the movement at the first position and the movement at the second position.

3. The determination method according to claim 2, characterized in that, The aforementioned computer also performs: Calculate the average of the motion vectors of each pixel in the captured image contained within the aforementioned background region, and use this average as the first motion vector; and The average motion vector of each pixel in the captured image within the aforementioned surrounding area is calculated and used as the second motion vector. The second determination is based on the magnitude of the difference vector between the first motion vector and the second motion vector.

4. The determination method according to claim 3, characterized in that, In the second determination mentioned above, if the magnitude of the difference vector is greater than a predetermined threshold, it is determined that the captured image is an image of the display object that captured the person.

5. The determination method according to claim 1, characterized in that, The aforementioned first determination is based on the difference between the movement at the first position and the movement at the third position.

6. The determination method according to claim 5, characterized in that, The aforementioned computer also performs: Calculate the average of the motion vectors of each pixel in the captured image contained within the aforementioned background region, and use this average as the first motion vector; and The average of the motion vectors of each pixel in the aforementioned captured image, contained within the image region of the aforementioned person, is calculated as the third motion vector. The first determination mentioned above is based on the magnitude of the difference vector between the first motion vector and the third motion vector.

7. A non-transitory computer-readable storage medium storing a judgment program, characterized in that, Used to enable the computer to perform the following processes: Acquire images of the area containing people captured by a camera; From the acquired captured image, an image region including a peripheral region and a background region is determined as the image region other than the image region of the person. The peripheral region is a ring-shaped region with the edge of the captured image as its outer perimeter, and the background region is the region other than the image region of the person, which is surrounded by the inner perimeter of the peripheral region. A first determination is made, which is based on the distribution of movement between a first position contained in the background area and a third position contained in the image area of ​​the person, to determine whether the captured image is an image of the actual person; and If, in the first determination above, it is determined that the captured image is not an image of the actual person, a second determination is made. The second determination is based on the distribution of movement between the first position and the second position included in the surrounding area to determine whether the captured image is an image of the displayed object that captured the person.

8. An information processing device, characterized in that, have: The image acquisition unit acquires images of the image area containing people captured by the camera; The region determination unit determines an image region, including a peripheral region and a background region, from the acquired captured image as an image region other than the image region of the person. The peripheral region is a ring-shaped region with the edge of the captured image as its outer perimeter, and the background region is a region other than the image region of the person that is surrounded by the inner perimeter of the peripheral region. as well as The determination unit performs a first determination, which is based on the distribution of movement between a first position contained in the background area and a third position contained in the image area of ​​the person, to determine whether the captured image is an image of the actual person. If the first determination determines that the captured image is not an image of the actual person, a second determination is performed, which is based on the distribution of movement between the first position and a second position contained in the surrounding area, to determine whether the captured image is an image of the displayed object of the person.

Images