Information processing apparatus, information processing method, and program

Abstract

To make it easy for the administrator to set appropriate questions in advance. An information processing apparatus extracts a person region determined to exist a person from an image captured by an imaging device, acquires a representative value of first characteristic values calculated by inputting colors of pixels included in a part of the person region into a transformation function respectively, corrects a skin range based on the representative value, the skin range indicating that a color represented by a characteristic value acquired by the transformation function is a color of a person's skin, and determines whether or not there is a person in a part of an image captured by the imaging device based on whether or not second characteristic values calculated by the transformation function for each of a plurality of pixels constituting the part of the image are included in the corrected skin range.

Description

Technical Field

[0001] This invention relates to information processing apparatus, information processing methods, and programs. Background Technology

[0002] There are techniques for determining the presence of a person based on images captured by a camera device. One method for determining the presence of a person is based on whether the color of pixels located in a certain region of the image converges to a predetermined range.

[0003] Non-Patent Document 1 discloses the following technique: by inputting the color of a pixel into a specified function to obtain a value, the pixel is determined to be skin if the value is within a specified range.

[0004] Existing technical documents

[0005] Patent documents

[0006] Non-patent literature 1: ANEW COLOUR SPACE FOR SKIN TONE DETECTION, Abbas Cheddad, Joan Condell, Kevin Curran and Paul Mc Kevitt, 2009 IEEE International Conference on Image Processing Summary of the Invention

[0007] The problem that the invention aims to solve

[0008] Due to factors such as lighting, the skin color in sometimes captured images does not converge to a predetermined color range. In such cases, it may be impossible to properly determine the presence of a person.

[0009] The present invention was made in view of the above-mentioned problems, and its object is to provide a technique for determining the presence of people with higher accuracy based on the colors in an image.

[0010] Methods for solving problems

[0011] To address the aforementioned issues, the information processing apparatus of the present invention includes: an extraction unit that extracts a human region determined to contain a person from an image captured by a camera device; a representative acquisition unit that acquires a representative value of a first feature value calculated by inputting the colors of pixels contained in a portion of the human region to a transformation function; a correction unit that corrects a skin region based on the representative value, the skin region representing the color represented by the feature value obtained by the transformation function being the color of human skin; and a determination unit that determines whether a person is present in a portion of the image based on whether a second feature value calculated by the transformation function for each pixel among a plurality of pixels constituting a portion of the image captured by the camera device is included in the corrected skin region.

[0012] Furthermore, the information processing method of the present invention includes the following steps: extracting a human region determined to contain a person from an image captured by a camera device; obtaining a representative value of a first feature value calculated by inputting the colors of pixels contained in a portion of the human region into a transformation function; correcting a skin range based on the representative value, the skin range representing the color represented by the feature value obtained by the transformation function being the color of human skin; and determining whether a person is in a portion of the image based on whether a second feature value calculated by the transformation function for each pixel among a plurality of pixels constituting a portion of the image captured by the camera device is included in the corrected skin range.

[0013] Furthermore, the program of the present invention enables a computer to function as the following units: an extraction unit that extracts a human region determined to contain a person from an image captured by a camera device; a representative acquisition unit that acquires a representative value of a first feature value calculated by inputting the colors of pixels contained in a portion of the human region into a transformation function; a correction unit that corrects a skin range based on the representative value, the skin range representing the color represented by the feature value obtained by the transformation function being the color of human skin; and a determination unit that determines whether a person is in a portion of the image based on whether a second feature value calculated by the transformation function for each pixel among a plurality of pixels constituting a portion of the image captured by the camera device is contained within the corrected skin range.

[0014] In one embodiment of the invention, the human region may be a facial region, and a portion of the human region may be an area that includes the area near the eyes but excludes the area near the mouth and the upper part of the forehead.

[0015] In one embodiment of the invention, the correction unit may correct the skin area in such a way that the less pixels the first feature value is not included in the skin area, the less correction is required.

[0016] In one embodiment of the invention, the representative value may be the average value of the feature values.

[0017] In one aspect of the present invention, when the red component of a pixel is set to R, the green component to G, the blue component to B, and Cr, Cg, and Cb are set to predetermined coefficients, the output e of the transformation function can be calculated by the formula e = Cr × R + Cg × G + Cb × B - max(G, B).

[0018] In one aspect of the invention, when the average value of the feature values ​​used in the calculation of the skin area before correction is set to μr, the average value of the first feature value is set to μ1, the number of pixels whose first feature value is not within the skin area is set to An, the number of pixels constituting a part of the human region is set to Ap, and n is set to an integer greater than or less than 1 and less than 3, the result can be based on the formula Δμ=(μ1-μr)×(An / Ap). n The calculated Δμ is used to correct the upper and lower limits of the skin area.

[0019] In one aspect of the invention, when the average value of the feature values ​​used in calculating the skin area before correction is set to μr, the average value of the first feature value is set to μ1, the number of pixels whose first feature value is not within the skin area is set to An, the number of pixels constituting a part of the human region is set to Ap, the number of pixels in the eye region included in the part is set to Ae, and n is set to an integer greater than or equal to 1 and less than or equal to 3, the result can be based on the formula Δμ=(μ1-μr)×(An / (Ap-Ae)). n The calculated Δμ is used to correct the upper and lower limits of the skin area.

[0020] In one embodiment of the present invention, the extraction unit may extract multiple face regions as multiple person regions, and the representative acquisition unit may input the colors of the pixels contained in a portion of the largest face region among the multiple face regions into a transformation function to obtain representative values ​​of the first feature values ​​of each pixel calculated by the transformation function. The determination unit may determine whether the face region contains a person's face based on whether the second feature values ​​calculated by the transformation function for the pixels contained in the face regions among the multiple face regions that were not used to obtain the representative values ​​are included within the corrected skin range.

[0021] Invention Effects

[0022] According to the present invention, it is possible to determine whether there is a person based on an image with higher precision. Attached Figure Description

[0023] Figure 1 This is a diagram illustrating an example of an information processing system according to an embodiment of the present invention.

[0024] Figure 2 It is a block diagram representing the functions implemented by an information processing system.

[0025] Figure 3 This is a diagram illustrating an example of facial regions identified from an image.

[0026] Figure 4 This is a flowchart representing an example of a process performed by a client device.

[0027] Figure 5 This is a flowchart representing an example of a process performed by a client device.

[0028] Figure 6 This is a diagram illustrating the reference areas contained within the facial region.

[0029] Figure 7 It is a diagram illustrating the distribution of skin color.

[0030] Figure 8 It is a diagram that roughly illustrates the effects of lighting. Detailed Implementation

[0031] Hereinafter, embodiments of the present invention will be described based on the accompanying drawings. For structures marked with the same symbols, repeated descriptions will be omitted. In this embodiment, an information processing system will be described that detects whether someone other than the worker is spying on them based on captured images in an environment without strict access control, such as remote work.

[0032] Figure 1 This diagram illustrates an example of an information processing system according to an embodiment of the present invention. The information processing system includes an information processing server 1 and client devices 2. The information processing server 1 is connected to one or more client devices 2 via a network.

[0033] Information processing server 1 includes a processor 11, a storage unit 12, a communication unit 13, and an input / output unit 14. Furthermore, information processing server 1 is a server computer. The processing of information processing server 1 can also be implemented using multiple server computers. Client device 2 includes a processor 11, a storage unit 12, a communication unit 13, an input / output unit 14, and a camera 25. Client device 2 is a personal computer or a tablet terminal.

[0034] Processors 11 and 21 operate according to programs stored in storage units 12 and 22. Furthermore, processors 11 and 21 control communication units 13 and 23, and input / output units 14 and 24. Processor 21 also controls camera 25 according to the program. The aforementioned program can be provided via the Internet or other means, or it can be provided by storing it in computer-readable storage media such as flash memory or DVD-ROM.

[0035] Storage units 12 and 22 are composed of storage elements such as RAM and flash memory, as well as external storage devices such as hard disk drives. Storage units 12 and 22 store the aforementioned program. In addition, storage units 12 and 22 store information input from processors 11 and 21, communication units 13 and 23, and input / output units 14 and 24, as well as calculation results.

[0036] The communication units 13 and 23 enable communication with other devices, and are composed of integrated circuits that implement wireless LAN and wired LAN. Based on the control of the processors 11 and 21, the communication units 13 and 23 input information received from other devices to the processors 11 and 21, the storage units 12 and 22, and send information to other devices.

[0037] Input / output units 14 and 24 consist of a video controller that controls the display output device and a controller that acquires data from the input device. Input devices include keyboards, mice, and touch panels. Based on the control of processors 11 and 21, input / output units 14 and 24 output display data to the display output device and acquire data input by the user through the input device. The display output device is, for example, a monitor connected to an external device.

[0038] The camera 25 captures images based on the control of the processor 21. The camera 25 is, for example, built into a personal computer or tablet terminal, and is positioned such that the person operating the personal computer or tablet terminal is captured in a roughly centered position.

[0039] Next, the functions provided by the information processing system will be explained. Figure 2 This is a block diagram illustrating the functions implemented by the information processing system. Functionally, the information processing system includes an image acquisition unit 51, a face extraction unit 52, a representative value acquisition unit 53, a skin range correction unit 54, a person determination unit 55, and a result output unit 56. These functions are implemented by the processor 21 included in the client device 2 executing a program stored in the storage unit 22 and controlling the communication unit 23, camera 25, etc. Furthermore, these functions can also be implemented by the processor 11 included in the information processing server 1 executing a program stored in the storage unit 12 and controlling the communication unit 13, etc.

[0040] The image acquisition unit 51 acquires the image captured by the camera 25, which is a camera device. When the image acquisition unit 51 is implemented by the information processing server 1, the image can be acquired by receiving the captured image sent from the communication unit 23 of the client device 2 via the communication unit 13.

[0041] The face extraction unit 52 extracts the face region 70 determined to contain a face from the acquired image. The face extraction unit 52 converts the acquired image to grayscale and extracts the region containing the face identified by a known face detection technique based on pattern recognition as the face region 70. The extracted face region 70 can be a rectangular region. Alternatively, instead of the face region 70, the face extraction unit 52 can detect human body parts other than the face, such as hands, or human regions with human contours, using a known pattern recognition technique; these human regions are used instead of the face region 70 in the processing described later.

[0042] Figure 3 This is a diagram illustrating an example of facial region 70 identified from an image. Figure 3 In this example, face regions 71 and 72 are extracted from the image, as one example of face region 70. Face region 71 is obtained by extracting an image of the face of the person operating the client device 2, and face region 72 is extracted by misidentifying patterns such as furniture in the background as faces.

[0043] The representative value acquisition unit 53 inputs the colors of the pixels contained in a portion of the face region into the transformation function and calculates the feature values ​​of each pixel. The representative value acquisition unit 53 acquires a representative value of the calculated feature values, such as the average value of the feature values. The face region 70 that is processed by the representative value acquisition unit 53 is either the largest face region 70 extracted by the face extraction unit 52 or the central face region 70. Details regarding the transformation function and feature values ​​will be described later.

[0044] The skin range correction unit 54 corrects the skin range based on the obtained representative value, which represents the color of human skin as indicated by the eigenvalues ​​obtained through the transformation function. Details regarding the skin range will be described later.

[0045] The person determination unit 55 determines whether a person is present in a portion of the image based on whether the feature values ​​calculated by a transformation function for each pixel among a plurality of pixels constituting a portion of the image captured by the camera 25 are included within the corrected skin area. The portion of the image processed by the person determination unit 55 may be a face region 70 extracted by the face extraction unit 52 that is not used for obtaining representative values ​​in the representative value acquisition unit 53. The processing by the person determination unit 55 can be performed when multiple face regions 70 are extracted by the face extraction unit 52.

[0046] The result output unit 56 outputs the determination result of the person determination unit 55. More specifically, in the processing of the person determination unit 55, if it is determined that there is a person in a part of the image captured by the camera 25, or more specifically, if it is determined that there is a person in any one of the one or more face regions 70 that were not used to obtain the representative value, the result output unit 56 outputs a warning message indicating that a person is spying. The output destination of the warning message can be the display or speaker of the client device 2.

[0047] Next, we will explain the details of the information processing system. Figure 4 , 5 This is a flowchart illustrating an example of the processing performed by client device 2. Figure 4 , 5 The process shown can be repeated at certain time intervals.

[0048] First, the image acquisition unit 51 acquires the image captured by the camera 25 (step S101). Next, the face extraction unit 52 extracts one or more face regions 70 from the acquired image using known face recognition technology (step S102).

[0049] When a face region 70 is acquired, the representative value acquisition unit 53 selects one face region 70 from one or more face regions 70, and acquires a reference region 75 located around the eyes from the selected face region 70 (step S103). The representative value acquisition unit 53 may acquire a region located in a relatively position where there is a high probability of the area around the eyes as the reference region 75 from the selected face region 70. Alternatively, the representative value acquisition unit 53 may acquire a region that is determined to be around the eyes by image recognition as the reference region 75. Hereinafter, the reference region 75 will be described, and the method of the former will be further described.

[0050] Figure 6This diagram illustrates the reference region 75 included in the facial region 70. The facial region 70 includes the reference region 75, the upper region 76, and the lower region 77. The reference region 75 encompasses the area near the eyes. The upper region 76, located above the reference region 75, encompasses the upper part of the forehead. The lower region 77, located below the reference region 75, encompasses the mouth. The upper part of the forehead in the upper region 76 is highly likely to be obscured by hair, and the area around the mouth in the lower region 77 is highly likely to be obscured by a mask. Therefore, the reference region 75, located around the eyes, is the area where the color of a person's skin is most easily visually discernible.

[0051] The representative value acquisition unit 53 determines the reference region 75 based on the relative position within the face region 70. For example, the face region 70 is a rectangular region with sides extending in the horizontal and vertical directions of the captured image. Let the upper left coordinate of this rectangular face region 70 be (tx, ty), the horizontal dimension be th, and the vertical dimension be tv. In this case, the representative value acquisition unit 53 can acquire the rectangular region with the upper left coordinate of (tx + 0.05 × th, ty + 0.1tv) and the lower right coordinate of (tx + 0.9 × th, ty + 0.45tv) as the reference region 75. Alternatively, the reference region 75 can also be other regions with a high probability of skin exposure.

[0052] The representative value acquisition unit 53 acquires feature values ​​for each pixel located in the reference region 75 using the transformation function e (step S104). More specifically, the representative value acquisition unit 53 inputs the color of each pixel located in the reference region 75 into the transformation function e and acquires the output of the transformation function e as the feature value. The transformation function e is the following formula.

[0053] e=Cr×R+Cg×G+Cb×B-max(G,B)

[0054] Here, R, G, and B represent the red, green, and blue components of the pixel's color, respectively. max(G,B) is a function that outputs the larger of G and B. Cr, Cg, and Cb are predetermined coefficients. Cr, Cg, and Cb are calculated by observing under reference lighting conditions, and for example, are the following values. These values ​​can also be adjusted based on environmental factors, etc.

[0055] Cr = 0.2989

[0056] Cg = 0.5870

[0057] Cb = 0.1402

[0058] When feature values ​​are calculated for each pixel located in the reference region 75, the representative value acquisition unit 53 obtains the average value as the representative value of the feature values ​​(step S105). The representative value acquisition unit 53 may also calculate the representative value using other methods such as the most frequent value.

[0059] In addition, the skin range correction unit 54 calculates the amount of pixels whose feature values ​​are not included in the skin range R (step S106).

[0060] Figure 7 It is a diagram illustrating the distribution of skin color. Figure 7 This diagram illustrates the normal distribution of characteristic values ​​for skin color in individuals, reflecting individual differences based on environmental and genetic variations. If we define the mean of the normal distribution as μ and the variance as σ, then the lower limit of the skin range R is set as (μ - σ), and the upper limit as (μ + 3σ). The values ​​of μ and σ are determined experimentally; for example, μ is 12.3057 and σ is 5.09026.

[0061] The processing is described in detail. The skin range correction unit 54 determines whether the feature value of each pixel is included in the skin range R, and counts the number of pixels whose feature values ​​are not included in the skin range R.

[0062] Then, the skin range correction unit 54 corrects the skin range R based on the calculated representative value and the calculated amount of pixels contained in the skin range R (step S107).

[0063] The lower and upper limits of the skin area R are calculated based on Δμ, and the skin area R is then corrected accordingly. As a specific example, the corrected lower limit of the skin area R is (μ + Δμ - σ), and the upper limit is (μ + Δμ + 3σ). Here, Δμ is calculated using the following formula.

[0064] Δμ=(μ1-μr)×(An / Ap) n

[0065] Figure 8 This is a diagram that roughly illustrates the effect of lighting, showing the changes in the distribution of feature values ​​caused by different lighting conditions, in other words, the generation of Δμ. μ1 is the value of μ calculated based on the distribution before correction. μr is a representative value (in this case, the average value) of the feature values ​​calculated for each pixel located in the reference region 75.

[0066] An is the amount of pixels in the reference region 75 whose feature values ​​are not included in the skin area R, and Ap is the amount of all pixels in the reference region 75. An is equivalent to the area of ​​the portion of the reference region 75 whose feature values ​​are not included in the skin area R, and Ap is equivalent to the area of ​​the reference region 75. n is an integer greater than or equal to 1, preferably 3, but can also be 1.

[0067] By minimizing the number of pixels whose feature values ​​are not included in the skin region R, the correction becomes weaker, reducing the likelihood of overcorrection due to factors other than illumination being reflected in the correction. Furthermore, by increasing n, even when the skin region R contains a large number of pixels, the possibility of overcorrection in the skin region R can be reduced.

[0068] In addition, Δμ can also be calculated using the following formula.

[0069] Δμ=(μ1-μr)×(An / (Ap-Ae)) n

[0070] In this formula, Ae is the amount of pixels in reference region 75 that contain an eye, which can be simply a specified proportion (e.g., 5%) of the total number of pixels in reference region 75. In this case, Ap-Ae can be obtained simply by multiplying the amount of pixels in reference region 75 by a constant less than 1 (e.g., 0.95).

[0071] If the skin area R is corrected, the person determination unit 55 determines, based on the image captured by the camera 25, whether a person's face is contained in the face area 70 other than the face area 70 for which a representative value was calculated to correct the skin area R. More specifically, in this process, the person determination unit 55 determines whether there is a face area 70 other than the face area 70 extracted by the face extraction unit 52 and for which a representative value was calculated (step S111). If no other face area 70 exists ("No" in step S111), it is determined that there is no person different from the operator of the client device 2, and the process ends.

[0072] On the other hand, if other face regions 70 exist ("Yes" in step S111), the person determination unit 55 selects one of the other face regions 70 (step S112). Then, the person determination unit 55 obtains feature values ​​for each pixel located in the selected face region 70 through a transformation function (step S113). The person determination unit 55 calculates the ratio of the number of pixels with feature values ​​in the corrected skin range R to the number of all pixels in the reference region 75 (step S114).

[0073] If the calculated proportion exceeds a predetermined threshold ("Yes" in step S115), the result output unit 56 outputs a warning message to the user operating the client device 2, indicating that they may be being spied on (step S116), and sends this message to the information processing server 1. If the number of times or the duration during which the calculated proportion exceeds the predetermined threshold exceeds a specified threshold, the result output unit 56 forcibly shuts down the client device 2. On the other hand, if the calculated proportion does not exceed the threshold ("No" in step S115), the processing in step S116 is skipped.

[0074] Then, the human determination unit 55 determines whether there are any unselected face regions among the other face regions 70 (step S117). If there are unselected face regions 70 ("Yes" in step S117), one of the unselected face regions 70 is selected (step S118), and the processing after step S113 is repeated. On the other hand, if there are no unselected face regions 70 ("No" in step S117), the processing ends.

[0075] also, Figure 4 , 5 The processing shown can also be performed via the information processing server 1 instead of the client device 2. In this case, in step S101, the image acquisition unit 51 acquires the image sent from the client device 2. Furthermore, in step S116, information causing the client device 2 to output a warning message is sent to the client device 2.

[0076] As described above, by correcting the skin range R using areas where skin color is easily identifiable, and using the corrected skin range to determine the presence of skin in the image, the decrease in determination accuracy caused by changes in the color of the photographed skin due to lighting can be reduced.

[0077] Symbol Explanation

[0078] 1 Information processing server, 2 Client device, 11, 21 Processor, 12, 22 Storage unit, 13, 23 Communication unit, 14, 24 Input / output unit, 25 Camera, 51 Image acquisition unit, 52 Face extraction unit, 53 Representative value acquisition unit, 54 Skin range correction unit, 55 Person determination unit, 56 Result output unit, 70, 71, 72 Face region, 75 Reference region, 76 Upper region, 77 Lower region.

Claims

1.An information processing apparatus comprising: an extraction unit that extracts a person region in which a person is determined to be present, from an image captured by an imaging device; a representative value acquisition unit that acquires a representative value of first characteristic values calculated by inputting colors of pixels included in a part of the person region to a transformation function respectively; a correction unit that corrects a skin range based on the representative value, the skin range indicating that a color represented by a characteristic value acquired by the transformation function is a color of a person's skin; and a determination unit that determines whether or not a person is present in a part of an image acquired by the imaging device, according to whether or not second characteristic values calculated by the transformation function with respect to each of a plurality of pixels constituting the part of the image are included in the skin range after correction, the correction unit correcting the skin range in such a manner that the amount of correction is smaller the smaller the amount of pixels in which the first characteristic value is not included in the skin range, in a case where an average value of characteristic values used in calculation of the skin range before correction is set to μr, an average value of the first characteristic values is set to μ1, the amount of pixels in which the first characteristic value is not included in the skin range is set to An, the amount of pixels constituting a part of the person region is set to Ap, and n is set to an integer of 1 or more and 3 or less, the upper limit value and the lower limit value of the skin range are corrected based on Δμ calculated by the following equation: 。 2.The information processing apparatus according to claim 1, wherein the person region is a face region, the part of the person region is a region in the person region that includes a vicinity of an eye and does not include a vicinity of a mouth and an upper portion of a forehead. 3.The information processing apparatus according to claim 1, wherein the representative value is an average value of the characteristic values. 4.The information processing apparatus according to claim 1, wherein in a case where a red component of a pixel is set to R, a green component is set to G, a blue component is set to B, Cr, Cg, and Cb are set to predetermined coefficients, an output e of the transformation function is calculated by the following equation: 。 5.The information processing apparatus according to claim 1, wherein in a case where the amount of pixels of a region of an eye included in the part is set to Ae, the upper limit value and the lower limit value of the skin range are corrected based on Δμ calculated by the following equation: 。 6.The information processing apparatus according to claim 1, wherein the extraction unit extracts a plurality of face regions as a plurality of person regions, the representative value acquisition unit inputs colors of pixels included in a part of a largest face region among the plurality of face regions to a transformation function respectively, acquires a representative value of first characteristic values of the respective pixels calculated by the transformation function, the determination unit determines whether or not a face region included in the plurality of face regions includes a person's face, according to whether or not second characteristic values calculated by the transformation function with respect to pixels included in the face region are included in the skin range after correction. 7.An information processing method comprising the steps of: extracting a person region in which a person is determined to exist from an image captured by an imaging device; obtaining a representative value of first characteristic values calculated by inputting colors of pixels included in a part of the person region to a transformation function respectively; correcting a skin range indicating that a color represented by a characteristic value obtained by the transformation function is a color of a person's skin based on the representative value; and determining whether or not there is a person in a part of the image based on whether or not second characteristic values calculated by the transformation function for each of a plurality of pixels constituting the part of the image obtained by the imaging device are included in the corrected skin range, in the step of correcting, the skin range is corrected in such a manner that the smaller the amount of pixels in which the first characteristic value is not included in the skin range, the smaller the amount of correction, in a case where an average value of characteristic values used in calculation of the skin range before correction is set as μr, an average value of the first characteristic values is set as μ1, the amount of pixels in which the first characteristic value is not in the skin range is set as An, the amount of pixels constituting a part of the person region is set as Ap, and n is set as an integer of 1 or more and 3 or less, the upper limit value and the lower limit value of the skin range are corrected based on Δμ calculated by the following formula: 。

Images