Information processing apparatus, method, and program

Abstract

To prevent fluctuations of exposure along with subject detection processing.SOLUTION: An information processing apparatus has: image acquisition means that acquires an image picked up by an imaging unit; first detection means that detects a first feature area from the image acquired by the image acquisition means; second detection means that detects a second feature area different from the first feature area from the image acquired by the image acquisition means; photometric means that determines a photometric value for at least one of the first feature area detected by the first detection means and the second feature area detected by the second detection means; and exposure determination means that determines an exposure value for a plurality of images continuously picked up by the imaging unit on the basis of the photometric value determined by the photometric means.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, method, program, and storage medium.

Background Art

[0002] Conventionally, in a backlight environment, a technique for determining an exposure amount using not only face detection but also human body detection is known (see Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The problem to be solved by the present invention is to suppress fluctuations in exposure associated with subject detection processing.

Means for Solving the Problems

[0005] Image acquisition means for acquiring an image captured by an imaging unit, face detection processing for detecting a face region, and human body detection processing for detecting a human body region, which can be executed on the image captured by the imaging unit; storage means for storing a first luminance value based on the face region in the first image when a face region is detected by the detection means in the first image captured by the imaging unit, and for storing a second luminance value based on the human body region in the second image when a face region is not detected by the detection means and a human body region is detected in the second image different from the first image captured by the imaging unit; and determination means for determining an exposure correction value for an image captured by the imaging unit based on an average value of a plurality of luminance values associated with a plurality of images captured by the imaging unit, the plurality of luminance values including at least the first and second luminance values.

Brief Description of the Drawings

[0006]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Mode for Carrying Out the Invention

[0007] (First Embodiment) Hereinafter, with reference to FIGS. 1 to 8, embodiments of the information processing apparatus according to the present embodiment will be described. Note that one or more of the functional blocks shown in the figures described later may be realized by hardware such as an ASIC or a programmable logic array (PLA), or may be realized by a programmable processor such as a CPU or an MPU executing software. Further, it may be realized by a combination of software and hardware. Therefore, in the following description, even if different functional blocks are described as the operating entity, the same hardware may be realized as the main body.

[0008] <Basic Configuration> FIG. 1 is a block diagram exemplarily illustrating the configuration of an imaging control system according to the present embodiment. The imaging control system shown in FIG. 1 includes a surveillance camera 101, a network 102, a client device 103, an input device 104, and a display device 105. The surveillance camera 101 is a device capable of imaging a subject and performing image processing for acquiring a moving image. The surveillance camera 101 and the client device 103 are connected in a state where they can communicate with each other via the network 102.

[0009] FIG. 2 is a block diagram exemplarily illustrating the internal configuration of the surveillance camera 101 according to the present embodiment. The imaging optical system 201 is composed of a zoom lens, a focus lens, an image stabilization lens, an aperture, a shutter, etc., and is a group of optical members that collect the light information of the subject.

[0010] The imaging device 202 is a charge accumulation type solid-state imaging device such as a CMOS or a CCD that converts the light beam collected by the imaging optical system 201 into a current value (signal value), and is an imaging unit that acquires color information by combining with a color filter, etc.

[0011] The camera CPU 203 is a control unit that comprehensively controls the operation of the surveillance camera 101. The camera CPU 203 reads instructions stored in a ROM (Read Only Memory) 204 and a RAM (Random Access Memory) 205, and executes processing according to the results. Further, the imaging system control unit 206 controls each part of the surveillance camera 101, such as focus control, shutter control, and aperture adjustment, for the imaging optical system 201 (based on instructions from the camera CPU 203). The communication control unit 207 performs control for transmitting the control related to each part of the surveillance camera 101 to the camera CPU 203 through communication with the client device 103.

[0012] The A / D conversion unit 208 converts the amount of light of the subject detected by the imaging device 202 into a digital signal value. The image processing unit 209 is image processing means for performing image processing on the image data of the digital signal output from the imaging device 202. The encoder unit 210 is conversion means for performing conversion processing on the image data processed by the image processing unit 209 into a file format such as Motion Jpeg, H264, or H265. The network I / F 211 is an interface used for communication via the network 102 with an external device such as the client device 103, and is controlled by the communication control unit 207.

[0013] The network 102 is an IP network that connects the surveillance camera 101 and the client device 103. The network is composed of, for example, a plurality of routers, switches, cables, etc. that satisfy a communication standard such as Ethernet (registered trademark). In the present embodiment, the network 102 only needs to be able to communicate between the surveillance camera 101 and the client device 103, regardless of its communication standard, scale, configuration, etc. For example, the network 102 may be composed of the Internet, a wired LAN (Local Area Network), a wireless LAN (Wireless LAN), a WAN (Wide Area Network), etc.

[0014] FIG. 3 is a block diagram exemplarily explaining the internal configuration of the client device 103 which is an information processing device according to the present embodiment. The client device 103 includes a client CPU 301, a main storage device 302, an auxiliary storage device 303, an input I / F 304, an output I / F 305, and a network I / F 306. Each element is communicably connected to each other via a system bus.

[0015] The client CPU 301 is a central processing unit that comprehensively controls the operation of the client device 103. Note that the client CPU 301 may be configured to execute overall control of the surveillance camera 101 via the network 102. The main storage device 302 is a storage device such as a RAM that functions as a temporary storage location for data of the client CPU 301. The auxiliary storage device 303 is a storage device such as an HDD, ROM, or SSD that stores various programs, various setting data, and the like. The input I / F 304 is an interface used when receiving inputs from the input device 104 and the like. The output I / F 305 is an interface used for outputting information to the display device 105 and the like. The network I / F 306 is an interface used for communication with external devices such as the surveillance camera 101 via the network 102.

[0016] By the client CPU 301 executing processing based on the program stored in the auxiliary storage device 303, the functions and processing of the client device 103 shown in FIG. 4 are realized. Details thereof will be described later.

[0017] As shown in FIG. 1, the input device 104 is an input device composed of a mouse, a keyboard, and the like. The display device 105 is a display device such as a monitor that displays the image output by the client device 103. In the present embodiment, the client device 103, the input device 104, and the display device 105 each have an independent configuration, but are not limited thereto. For example, the client device 103 and the display device 105 may be integrated, or the input device 104 and the display device 105 may be integrated. Further, the client device 103, the input device 104, and the display device 105 may be integrated.

[0018] FIG. 4 is a diagram exemplarily explaining the functions and configurations executed by the client device 103 according to the present embodiment. In other words, each part illustrated in FIG. 4 is a function and configuration that can be executed by the client CPU 301, and each of these parts is synonymous with the client CPU 301. That is, the client CPU 301 of the client device 103 includes an input information acquisition unit 401, a communication control unit 402, an input image acquisition unit 403, a subject detection unit 404, a photometric value calculation unit 405, an exposure determination unit 406, a detection result storage unit 407, and a display control unit 408. Note that the client device 103 may be configured to include each part illustrated in FIG. 4 as a configuration different from the client CPU 301.

[0019] The input signal acquisition unit 401 is an input means for receiving an input by a user via the input device 104.

[0020] The communication control unit 402 executes control for receiving the image transmitted from the surveillance camera 101 via the network 102. Further, the communication control unit 402 executes control for transmitting a control command to the surveillance camera 101 via the network 102.

[0021] The input image acquisition unit 403 is an image acquisition means for acquiring, as an image to be subjected to the subject detection process, the image captured by the surveillance camera 101 via the communication control unit 402. Details of the detection process will be described later.

[0022] The subject detection unit 404 is a subject detection means that performs various detections including face area detection (face detection) and human body area detection (human body detection) on the image acquired by the image acquisition unit 403. Here, the subject detection unit 404 in the present embodiment is configured to set an arbitrary method from face detection and human body detection, but is not limited thereto. For example, a configuration for detecting a feature area of a part of a person, such as an upper body of a person, or a partial area such as eyes, pupils, nose, mouth, etc. of the face, may be selectable. Also, in the present embodiment, the subject to be detected is described as a person, but a configuration capable of detecting a specific area related to a predetermined subject other than a person may also be used. For example, a configuration capable of detecting a predetermined subject preset in the client device 103, such as the face of an animal or a car, may be used.

[0023] The photometric value calculation unit 405 is a photometric value calculation means that calculates the photometric value of the current frame based on the detection result of the current frame obtained from the subject detection unit 404 and the detection result of the past frame obtained from the detection result storage unit 406.

[0024] The exposure determination unit 406 is an exposure determination means that determines the exposure when imaging a subject and acquiring an image based on the photometric value calculated by the subject detection unit 405 and the target value. Note that the exposure determined by the exposure determination unit 406 includes, in addition to the exposure value according to the exposure control program diagram pre-recorded in the client device 103, an exposure correction value for correcting this exposure. Information regarding the exposure determined by the exposure determination unit 406 is transmitted by the communication control unit 402 to the surveillance camera 101, and exposure control inside the surveillance camera 101 is executed. Details of the processing related to the operations of the subject detection unit 404, the photometric value calculation unit 405, and the exposure determination unit 406 will be described later with reference to the flowchart of FIG. 5. The detection result storage unit 407 is an exposure storage means that stores the exposure amount determined by the exposure determination unit 405. The display control unit 408 is a display control means that outputs an image in which the exposure determined by the exposure determination unit is reflected to the display device 105 according to an instruction from the client CPU 301.

[0025] <Subject Detection Process · Exposure Determination Process> Hereinafter, with reference to the flowchart illustrated in FIG. 5, the subject detection process and exposure determination process according to the present embodiment will be described. FIG. 5 is a flowchart exemplarily explaining the detection process and exposure determination process according to the present embodiment. Note that, in the imaging system illustrated in FIG. 1, it is assumed that the power of each device is turned on and the connection (communication) between the surveillance camera 101 and the client device 103 is established. And in this state, it is assumed that imaging of a subject, transmission of image data, and image display on the display device are repeated at a predetermined update cycle in the imaging system. Then, when an image obtained by imaging a subject is input to the client CPU 301 of the client device 103 from the surveillance camera 101 via the network 102, the flowchart illustrated in FIG. 5 is started.

[0026] First, in step S501, the subject detection unit 406 performs face detection on the image acquired by the input signal acquisition unit 401. As a face detection method, a pattern (discriminator) created using statistical learning as a pattern matching method may be used, or as a method other than pattern matching, a configuration may be adopted in which subject detection is performed using the luminance gradient within a local area. That is, the detection method is not limited, and various methods such as detection based on machine learning and detection based on distance information can be adopted.

[0027] Next, in step S502, it is determined whether a face has been detected in the face detection executed in step S501. If no face is detected, the process proceeds to step S507, and if one or more faces are detected, the process proceeds to step S503.

[0028] In step S503, the photometric value calculation unit 405 calculates the average luminance value of the average luminance value Iface of the face region determined to have detected a face in step S502 based on the detection result obtained from the subject detection unit 406. Specifically, the exposure determination unit 407 applies information regarding the number of detections in which a face has been detected, the detection position, and the detection size obtained from the subject detection unit 406 to the following formula (1).

[0029] [Number]

[0030] Here, I(x, y) represents the luminance value of the two-dimensional coordinate position (x, y) in the horizontal direction (x-axis direction) and the vertical direction (y-axis direction) within the image. Also, f represents the number of detected faces, (v, h) represents the center coordinates where the subject is detected, k and l respectively represent the detection sizes of the subject in the horizontal and vertical directions, and t represents the detected frame time. Note that in step S509 as well, similar to step S503, by applying the information regarding the number of detections, detection positions, and detection sizes of the detected human body to the following formula (2), the average luminance value of the human body region is calculated.

[0031] [Number]

[0032] Here, g represents the number of detections of the detected human body, and the other symbols are the same as in formula (1). Subsequently, in step S504, the photometry value calculation unit 405 calculates the photometry value at the current frame (time t) based on the average luminance value of the face region calculated in step S503 and the detection results of the past frames. For example, by applying formulas (3) and (4), the weighted average of the face average luminance value in the current frame and the average luminance values of the face and human body in the past frames consecutive to this current frame is calculated to obtain the photometry value E(t).

[0033] [Number]

[0034] [Number]

[0035] Here, n represents the number of frames used for weighted average, and in the proposed method, n ≥ 1 is set. For example, when n = 10, the weighted average of 10 consecutive frames is performed. Also, α and β represent the weighting parameters used when performing weighted average on the average luminance values of the face and the human body, and the values can be changed according to the environment where the subject is detected, the purpose of the authentication process executed later, and the accuracy of the detection process provided in the subject detection unit 404. The specific setting method and effects of the parameters α and β will be described later with reference to FIGS. 6 - 7. Note that the method for calculating the photometric value based on the detection results of the current frame and the past frames is not limited to the methods of formulas (3) and (4). For example, it may be a method of obtaining a statistical value that takes into account the influence of a plurality of frames occurring in the time direction, such as arithmetic mean, harmonic mean, geometric mean, etc.

[0036] Subsequently, in step S505, as shown in formula (5), the difference value ΔDiff between the target value Itarget of the subject area and the photometric value E(t) calculated in step S504 is calculated.

[0037]

Equation

[0038] Here, the target value Itarget of the subject area may be a target value preset by the user, or a fixed value preset on the hardware.

[0039] Next, in step S506, based on the difference value ΔDiff calculated in step S505, a predetermined threshold Th, and the current exposure value EVcurrent, the exposure correction amount EVcorrection is determined. For example, the correction amount EVcorrection is determined as shown in formula (6).

[0040]

Equation

[0041] Here, the parameter γ represents an exposure correction value that affects the correction for distributing to the under side or over side centered on the current exposure value EVcurrent when the difference ΔDiff calculated in step S507 is equal to or greater than a set threshold Th. For example, as shown in the lower branch of Equation (6), when the photometric value is greater than the threshold Th with respect to the target exposure value (Th < ΔDiff), it is determined that the average luminance of the subject area is on the under side in that state. Then, with respect to the current exposure value EVcurrent, by correcting to the positive side (+γ), the brightness of the subject area is controlled to approach the target value Itarget. Therefore, in order to stably execute the exposure correction in the time direction, it is important that the difference value calculated by Equation (5) changes gently in the time direction.

[0042] Here, the specific setting methods and effects of the parameters α and β shown in Equations (3) and (4) will be described. As shown in FIG. 6, generally, compared with the detection result of the face area, the detection result of the human body area originally contains many more pixels (hatched area) that should be judged as the background area by mistake. For example, when it is assumed that a subject is detected at the entrance or exit of a building or stadium as shown in FIG. 6, it is conceivable that the average luminance of the human body area is calculated to be higher than expected due to the influence of the pixels of the background area that are erroneously included. FIG. 7 is a diagram exemplarily showing the time course of the average luminance value of the subject and the difference value ΔDiff when a subject is detected in a backlight scene as shown in FIG. 6. During the time course, in many frames, face detection is performed, and exposure correction is executed on the positive side so as to approach the target value. However, for example, when the orientation of the face changes or the face area is shaded, there are moments when a human body instead of a face is detected, and as shown at times T1, T2, T3, and T4, the average luminance of the subject temporarily becomes a high value.

[0043] In such a case, in the conventional method that does not consider the detection results in the past frames, the photometric value E(t) temporarily becomes large with respect to the target value Itarget, and it is not possible to stably calculate the exposure correction amount with respect to the time direction. The conventional method referred to here is a method in which the parameter n = 0 is substituted in equations (3) and (4). As a result, depending on the timing, the brightness of the subject area is likely to suddenly become bright or dark. On the other hand, in the proposed method of the present embodiment, the photometric value E(t) is calculated based on the detection results of the subjects in a plurality of past frames so as to be smooth in the time direction (the dotted line part in FIG. 7). Therefore, the fluctuation of the exposure occurring in the time direction is stabilized. The proposed method referred to here is a method in which the parameter n = 1 or more is substituted in equations (3) and (4). Further, as described above, considering that the average luminance value of the human body area is generally more susceptible to the influence of the background area than the average luminance value of the face area, in equation (4), weighting is performed such that α>β, thereby realizing exposure control that places importance on more accurate detection processing. Here, the weighting relationship is not limited to the accuracy of the detection processing. For example, when the number of people is being counted in the subsequent authentication process, weighting such as α<β that places importance on the human body detection process may be used. Also, if the detection accuracy is equal and there is no inferiority, weighting such as α = β may be performed. Further, in the description using FIG. 7, the fluctuation of the exposure caused by the type of detection process has been described as a problem, but it is not limited to this. For example, by applying the proposed method to the problems caused by false detection, stabilization of the exposure is realized.

[0044] Returning to FIG. 5, in step S507, the average luminance value calculated in steps S503 and S509 and the information regarding the subject detection method are held in the detection result holding unit 407. The above is the processing in the present embodiment when at least one or more face areas are detected.

[0045] Next, the processing when the face area according to this embodiment is not detected will be described. When the face area is not detected in the processing of step S501, in step S508, the subject detection unit 404 performs human body detection on the image acquired by the input signal acquisition unit 401.

[0046] Next, in step S509, based on the result of the human body detection executed in step S508, it is determined whether a human body area is detected in the image. If at least one or more human body areas are detected, the process proceeds to step S510, and if no human body area is detected, the process proceeds to step S913. When proceeding to the processing of step S514 (that is, when neither the face area nor the human body area is detected), exposure correction based on the subject detection result is not performed. Note that the processing of steps S510 to S513 is executed based on substantially the same arithmetic expressions as steps S503 to S507 described above, except for calculating the average luminance value of the human body area and determining the exposure, and thus detailed description is omitted.

[0047] As described above, in the imaging system of this embodiment, in addition to the subject detection result of the current frame, the photometric value is calculated based on the subject detection result of the past frame, and the optimal exposure is set for the subject existing in the image. Therefore, in the imaging system of this embodiment, it is possible to reduce the exposure fluctuation caused by the switching of the detection process, and stably execute the optimal exposure control for the subject. In addition, since the weight can be variably controlled for each detection process, the influence of the process with low detection accuracy is not received, and robust exposure control is realized even for randomly occurring false detections.

[0048] (Second Embodiment) As a modified example described above, the case of calculating a photometric value based on a detection score calculated by a detection means will be described with reference to a figure and a mathematical formula. Here, the detection score is an evaluation value indicating the degree of reliability with respect to the detection result by the detection means. The larger the value of the detection score, the higher the probability that the detection target exists in the set detection method (area), and the smaller the value, the higher the possibility that the detection target does not exist (i.e., false detection). Note that the detection score described in this modified example is described using a value normalized in a value range with a minimum value of 0 and a maximum value of 100 for convenience, but it is not limited to this.

[0049] FIG. 8 is a diagram exemplarily showing the time course of the average luminance value of a subject, the detection score, and the difference value ΔDiff. During the time course, subject detection is performed in many frames, but false detections with low detection scores occur at random timings as shown at times T1, T2, T3, T4, T5, and T6, for example. Many of the false detections refer to areas where the subject does not originally exist, so the average luminance value calculated based on the false detection is a value far from the average luminance value of the subject. As a result, when false detections occur frequently, in the conventional method, exposure fluctuations occur, and even in the proposed weighted average method described above, there is a somewhat affected result (see FIG. 8(a)). Therefore, in this modified example, for example, as shown in Equation (7), the photometric value E(t) is obtained using the detection score.

[0050]

Equation

[0051] Here, Score(t) represents the value of the detection score at time t. The dotted line in FIG. 8 is a graph exemplarily showing the average luminance value of the subject calculated when the formula (7) is applied. By considering the detection score, the photometric value E(t) can be calculated without being affected by the average luminance value of frames that are likely to be false detections. As a result, as can be seen from the comparison in FIG. 8, in the proposed method that considers the detection score, it becomes possible to photograph the subject at a brightness closer to the target value for a longer time compared to the proposed method that does not consider the detection score. Also, it is possible to vary the number of frames used when performing weighted averaging according to the value of the detection score. For example, if the detection score remains low for a certain period of time, the value of the number of frames n used for weighted averaging is increased to reduce the influence on the photometric value. Conversely, if the detection score remains high, the value of the number of frames n is decreased to calculate a highly accurate photometric value while suppressing the calculation amount. Note that the parameter of Score(t) used in the formula (7) is not limited to the detection score, and it is also possible to substitute information related to detection accuracy such as the amount of noise included in the video, the Q value that determines the video quality, and the camera setting conditions.

[0052] With the configuration described above, stable and optimal exposure control can be achieved for the main subject that the user intends to image.

[0053] In the above-described embodiment, a so-called lens-integrated imaging device in which the imaging optical system 201 is integrally formed with the surveillance camera 101 as an example of the imaging device has been described, but it is not limited thereto. For example, a so-called lens-exchangeable imaging device in which a lens unit including the surveillance camera 101 and the imaging optical system 201 are provided separately may be used as the imaging device.

[0054] Further, part or all of the control in the present invention may be supplied to an imaging device or an information processing device via a network or various storage media as a computer program (software) that realizes the functions of the above-described embodiments. Then, a computer (or a CPU, MPU, etc.) in the imaging device or the information processing device may read and execute the program. In that case, the program and the storage medium storing the program will constitute the present invention.

Claims

1. image acquisition means for acquiring an image captured by an imaging unit; detection means capable of performing face detection processing for detecting a face region and human body detection processing for detecting a human body region on an image captured by the imaging unit; when a face region is detected by the detection means in a first image captured by the imaging unit, storing a first luminance value based on the face region in the first image; storage means for storing a second luminance value based on the human body region in a second image when a face region is not detected by the detection means and a human body region is detected in the second image different from the first image captured by the imaging unit; determination means for determining an exposure correction value for an image captured by the imaging unit based on an average value of a plurality of luminance values associated with a plurality of images captured by the imaging unit, the plurality of luminance values including at least the first and second luminance values; An information processing apparatus comprising the same.

2. The information processing apparatus according to claim 1, wherein the average value of the plurality of luminance values is an average value obtained by arithmetically averaging the plurality of luminance values.

3. The information processing apparatus according to claim 1 or 2, wherein the plurality of images are images continuously captured by the imaging unit.

4. The information processing apparatus according to any one of claims 1 to 3, wherein the determination means determines the exposure correction value based on a score indicating the reliability of at least one of the face detection processing and the human body detection processing.

5. an image acquisition step of acquiring an image captured by an imaging unit; a detection step capable of performing face detection processing for detecting a face region and human body detection processing for detecting a human body region on an image captured by the imaging unit; when a face region is detected in the detection step in a first image captured by the imaging unit, storing the first luminance value based on the face region in the first image; a storage step of storing the second luminance value based on the human body region in the second image when a face region is not detected in the detection step and a human body region is detected in the second image different from the first image captured by the imaging unit; A determining step of determining an exposure correction value for an image captured by the imaging unit based on an average value of a plurality of luminance values associated with a plurality of images captured by the imaging unit, the plurality of luminance values including at least the first and second luminance values A method characterized by comprising the above. [

6. ] On a computer,[[]] An image acquisition step of acquiring an image captured by the imaging unit, and A detection step capable of executing a face detection process for detecting a face region and a human body detection process for detecting a human body region on the image captured by the imaging unit, and When a face region is detected in the detection step in a first image captured by the imaging unit, storing a first luminance value based on the face region in the first image; A storage step of storing a second luminance value based on the human body region in the second image when a face region is not detected and a human body region is detected in the detection step in a second image different from the first image captured by the imaging unit; A determining step of determining an exposure correction value for an image captured by the imaging unit based on an average value of a plurality of luminance values associated with a plurality of images captured by the imaging unit, the plurality of luminance values including at least the first and second luminance values A program characterized by causing the above to be executed.

Images