Information Processing System, Image Processing Method, and Image Processing Program

Abstract

To improve user experience.SOLUTION: An information processing system includes a camera, a face detection unit 21 that detects faces included in an image captured by the camera, a region of interest setting unit 22 that sets a region of interest on the basis of the feature amounts of the detected face, a frame setting unit 26 that has a plurality of frame sets each consisting of a combination of an enlarged frame and a reduced frame having a size smaller than the enlarged frame, and sets a frame set according to the number of faces included in the image, and a framing unit 24b that performs framing when any region of interest exceeds the set enlarged frame or when the entire area of any region of interest is included in the set reduced frame.SELECTED DRAWING: Figure 10

Description

Technical Field

[0001] The present disclosure relates to an information processing system, an image processing method, and an image processing program.

Background Art

[0002] In recent years, video conferences have been frequently held, and accordingly, camera modules having functions suitable for video conferences have been proposed. For example, conventionally, a camera module has been proposed that incorporates a VPU (Visual Processing Unit) and automatically performs framing (adjustment of the viewing angle) to realize a more natural video conference. In such a camera module, the position and area of a person in the image are detected, a region of interest (ROI) is set based on the detection result, and automatic adjustment of the image is performed based on the region of interest and the resolution of the output image.

Summary of the Invention

Problems to be Solved by the Invention

[0003] By the way, when the auto-framing of an image is frequently performed, a phenomenon occurs in which the image sways in the vertical, horizontal, and depth directions (zoom in / zoom out). Such swaying tends to appear more prominently as the number of people in the frame increases, and there is a possibility of degrading the user experience.

[0004] The present disclosure has been made in view of such circumstances, and an object thereof is to provide an information processing system, an image processing method, and an image processing program capable of improving the user experience.

Means for Solving the Problems

[0005] The first aspect of the present disclosure includes a camera, a face detection unit that detects faces included in an image acquired by the camera, an interest area setting unit that sets an interest area based on feature amounts of the detected faces, an enlarged frame, and a reduced frame having a size smaller than the enlarged frame, and a frame setting unit that sets the frame set corresponding to the number of faces included in the image from among a plurality of frame sets associated with the number of faces, and a framing unit that performs framing when any of the interest areas exceeds the set enlarged frame or when the entire area of any of the interest areas is included in the set reduced frame. Moreover, in a plurality of the frame sets, the enlarged frame when the image contains one face is set to a size larger than the enlarged frame when the image contains a plurality of faces. It is an information processing system.

[0006] In the second aspect of the present disclosure, a computer detects faces included in an image acquired by a camera, sets an interest area based on feature amounts of the detected faces, includes an enlarged frame and a reduced frame having a size smaller than the enlarged frame, sets the frame set corresponding to the number of faces included in the image from among a plurality of frame sets associated with the number of faces, and performs framing when any of the interest areas exceeds the set enlarged frame or when the entire area of any of the interest areas is included in the set reduced frame. Moreover, in a plurality of the frame sets, the enlarged frame when the image contains one face is set to a size larger than the enlarged frame when the image contains a plurality of faces. It is an image processing method.

[0007] The third aspect of the present disclosure is an image processing program for causing a computer to execute the above image processing method.

[0008] One aspect as a reference example of the present disclosure is an information processing system including a camera, a face detection unit that detects faces included in an image acquired by the camera, a parameter setting unit that sets reaction parameters for the framing according to the number of detected faces, and a framing unit that performs framing of the image using the set reaction parameters.

[0009] One aspect as a reference example of the present disclosure is an image processing method in which a computer detects a face included in an image acquired by a camera, sets reaction parameters for the framing according to the number of detected faces, and performs framing of the image using the set reaction parameters.

Advantages of the Invention

[0010] According to the information processing system, image processing method, and image processing program of the present disclosure, there is an effect that the user experience can be improved.

Brief Description of the Drawings

[0011]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0012] 〔First Embodiment〕 Hereinafter, an embodiment of an information processing system, an image processing method, and an image processing program according to the first embodiment of the present disclosure will be described with reference to the drawings. Hereinafter, as an example of the information processing system, the camera module 1 will be exemplified and described.

[0013] FIG. 1 is an external view as an example of the camera module 1 according to the present embodiment. As shown in FIG. 1, the camera module 1 includes a camera 2 and a housing 3 that supports the camera 2 and houses various electronic components that constitute an image processing unit 10 described later.

[0014] The camera 2 includes, for example, a lens, a lens driving unit, and an image sensor. The lens captures light from the subject and forms a subject image on the image sensor. The image sensor converts the light captured by the lens into signal charges and captures the subject image. In the image sensor, for example, analog image signals are generated by capturing signal values of red (R), green (G), and blue (B) in an order corresponding to the Bayer array, and the obtained image signals are converted from an analog format to a digital format and the image data (RAW data) is output to the image processing unit 10. The camera 2 can employ a known camera.

[0015] The image processing unit 10 (see FIG. 3) performs predetermined signal processing (image processing) on the image data (RAW data) output from the camera 2. The image processing unit 10 performs various processes such as automatic exposure adjustment, automatic white balance adjustment, matrix processing, edge enhancement, luminance compression, and gamma processing on the RAW data, for example. Further, the image processing unit 10 analyzes the image data and performs autoframing. Details of autoframing will be described later. The image data processed by the image processing unit 10 is output to, for example, an information processing device having a display via a communication medium and is displayed on the display.

[0016] FIG. 2 is a diagram showing an example of a hardware configuration included in the camera module 1 according to the present embodiment. As shown in FIG. 2, the image processing unit 10 includes, for example, an NPU (Neural Network Processing Unit) 11, a main memory device (Main Memory) 12, a secondary storage device (Secondary storage: memory) 13, a communication interface 14, and the like. These units are directly or indirectly connected to each other via a bus and cooperate with each other to execute various processes.

[0017] The NPU 11 reads various programs (for example, an image processing program) stored in the secondary storage device 13 connected via a bus into the main memory device 12 and executes information processing and arithmetic processing to realize various functions described later. The main memory device 12 is composed of, for example, a writable memory such as a cache memory and a RAM (Random Access Memory), and is used as a work area for reading the execution program of the NPU 11, writing processing data by the execution program, and the like.

[0018] The secondary storage device 13 is a non-transitory computer readable storage medium. Examples of the secondary storage device 13 include a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, etc. Examples of the secondary storage device 13 include a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive) flash memory, etc. The secondary storage device 13 stores various programs (firmware) for realizing functions described later and various data necessary for realizing various processes. A plurality of secondary storage devices 13 may be provided, and the programs and data as described above may be divided and stored in each secondary storage device 13. Various programs (e.g., firmware), etc. may be applied in a form pre-installed in the secondary storage device 13 at the time of manufacture, or in a form distributed via communication means by wire or wirelessly.

[0019] The communication interface 14 functions as an interface for connecting to a network, communicating with other devices, and transmitting and receiving information. For example, the communication interface 14 communicates with other devices by wire or wirelessly. Examples of wireless communication include communication through lines such as Bluetooth (registered trademark), Wi-Fi, a mobile communication system (3G, 4G, 5G, 6G, LTE, etc.), and a wireless LAN. An example of wired communication includes communication through a line such as a wired LAN (Local Area Network).

[0020] FIG. 3 is a functional configuration diagram showing an example of the image processing function provided in the camera module 1 according to the present embodiment. As shown in FIG. 3, the image processing unit 10 includes a face detection unit 21, a region of interest setting unit 22, a parameter setting unit 23, and a framing unit 24. The face detection unit 21 detects a face included in the image acquired by the camera 2. For example, the face detection unit 21 extracts feature points of the face included in the image and detects the face using the extracted feature points. Since numerous known techniques have been proposed for face detection, these known techniques may be appropriately adopted.

[0021] The region of interest setting unit 22 sets a region of interest (ROI: Region of Interest) based on the feature points of the face detected by the face detection unit 21. The parameter setting unit 23 sets the reaction parameters for framing according to the number of detected faces, that is, the number of people included in the image. For example, the parameter setting unit 23 has parameter information in which the number of faces included in the image is associated with the reaction parameters for framing and registered, and acquires and sets the reaction parameters corresponding to the number of faces included in the image from the parameter information.

[0022] FIG. 4 shows an example of the parameter information. In the parameter information, the first parameter "3" is set as the reaction parameter corresponding to the number of faces being 1 or more and less than 3, the second parameter "2" is set as the reaction parameter corresponding to the number of faces being 3 or more and less than 6, and the third parameter "1" is set as the reaction parameter corresponding to the number of faces being 6 or more. The smaller the value of the parameter, the slower the reaction of the framing. Here, in the parameter information illustrated in FIG. 4, the number of faces is divided into three levels, and the reaction parameters corresponding to each level are set, but the division of the number of faces into levels is not limited to this example. For example, it may be divided into two, or may be divided into three or more levels.

[0023] As an example of the reaction parameters, a sensitivity parameter regarding the sensitivity of the framing can be mentioned. The sensitivity parameter is, for example, a parameter regarding the response time from when the execution conditions for executing the framing are satisfied until the framing is executed, and the smaller the value of the sensitivity parameter, the longer the response time. In the present embodiment, it is set such that the value of the sensitivity parameter becomes smaller as the number of faces increases.

[0024] Also, as an example of the reaction parameter, a speed parameter related to the speed of framing can be mentioned. The speed parameter is the execution speed of framing, and the smaller the value of the speed parameter, the slower the speed of the movement of the frame in the vertical, horizontal, and depth directions (zooming). In the present embodiment, when the number of faces increases, the value of the speed parameter is set to decrease.

[0025] Also, as shown in FIG. 5, the reaction parameter may be configured by a combination of a sensitivity parameter and a speed parameter. Also, the threshold value of the number of faces for changing the speed parameter and the threshold value of the number of faces for changing the sensitivity parameter may be different. For example, in the parameter information illustrated in FIG. 5, the number of faces "3" and "6" are set as common threshold values, but it is not limited to this. For example, for the sensitivity parameter, the number of faces "3" and "6" may be set as threshold values, and for the speed parameter, the number of faces "2", "4", and "6" may be set as threshold values.

[0026] The framing unit 24 performs framing of the image acquired by the camera 2 using the set reaction parameter. Specifically, position adjustment of the image in the vertical, horizontal, and depth directions is performed using the set reaction parameter. In other words, framing is an image process including vertical and horizontal position adjustment and zoom adjustment. For example, as shown in FIG. 6, the framing unit 24 has a reference frame Fre. Then, when a part of the region of interest ROI exceeds the reference frame Fre, it is determined that the framing condition is satisfied, and framing is performed. Also, thereafter, when the entire region of interest ROI is included in the reference frame Fre, it is determined that the framing condition is satisfied, and framing is performed.

[0027] Also, as shown in FIG. 7, when a plurality of faces are included in an image and a plurality of regions of interest ROI_1 to ROI_3 are set in the image, if a part of any of the regions of interest ROI_1 to ROI_3 exceeds the reference frame Fre, it is determined that the framing condition is satisfied and framing is performed. Also, when the entire area of the region of interest ROI is included in the reference frame Fre, it is determined that the framing condition is satisfied and framing is performed. Note that since the framing process itself is a known technique, it is not limited to the above method, and other known methods may be appropriately adopted.

[0028] Next, an image processing method executed by the camera module 1 will be described with reference to FIG. 8. FIG. 8 is a flowchart showing an example of the processing procedure of the image processing method according to the present embodiment. For example, the following processing is repeatedly executed at a predetermined time interval or for each predetermined number of image frames input from the camera 2.

[0029] As shown in FIG. 8, first, when image data is acquired from the camera 2 (SA1), faces included in the image are detected (SA2), and further, a region of interest is set based on the feature points of the detected faces (SA3). Subsequently, reaction parameters corresponding to the number of detected faces are acquired from the parameter information (SA4). Subsequently, it is determined whether the acquired reaction parameters are the same as the currently set reaction parameters (SA5). If they are the same (SA5: YES), the process ends. On the other hand, if the acquired reaction parameters are different from the currently set reaction parameters (SA5: NO), the reaction parameters used for framing are updated (SA6), and the process ends.

[0030] As described above, according to the camera module (information processing system) 1, the image processing method, and the image processing program according to the present embodiment, the reaction parameters for framing are set according to the number of faces included in the image, and the image is framed using the set reaction parameters. Therefore, it is possible to set appropriate reaction parameters according to the number of faces. Specifically, as the number of faces included in the image increases, the reaction parameters are set so that the reaction of framing becomes slower. Thus, it is possible to make the framing reaction more sluggish as the number of faces included in the image increases. As a result, it is possible to reduce the shaking of the image, which has been a conventional problem, and to improve the user experience.

[0031] 〔Second Embodiment〕 Next, an information processing system, an image processing method, and an image processing program according to the second embodiment of the present disclosure will be described. The camera module (information processing system) according to the present embodiment is different from the first embodiment described above in that, as shown in FIG. 9, the image processing unit 10a further includes a determination unit 25, and the parameter setting unit 23 sets reaction parameters according to the determination result of the determination unit 25. Hereinafter, the components common to the first embodiment will be denoted by the same reference numerals and the description thereof will be omitted, and the different points will be mainly described.

[0032] FIG. 9 is a functional configuration diagram showing an example of the image processing functions provided in the camera module according to the present embodiment. As shown in FIG. 9, the image processing unit 10a according to the present embodiment further includes a determination unit 25 that determines whether or not the number of faces detected by the face detection unit 21 has increased compared to the image processing unit 10 according to the first embodiment. When the parameter setting unit 23 determines that the number of faces has increased by the determination unit 25, it determines whether or not the period until the number of faces increases, in other words, the period during which the same number of faces has been maintained, is equal to or longer than a predetermined period. Then, when the period until the number of faces increases is equal to or longer than the predetermined period, it sets reaction parameters that slow down the framing the most.

[0033] For example, if a participant arrives late for a meeting after a predetermined period has elapsed since the meeting started, it may be undesirable as it may interfere with the meeting if framing is performed according to the movement of the late participant.

[0034] Therefore, in the present embodiment, when it is determined that the number of faces included in the image has increased and, furthermore, the period until the number of faces increases is equal to or longer than a predetermined period, reaction parameters that slow down the framing the most are set. As a result, when there is a person who arrives late for a meeting, it becomes possible to reduce the influence of framing according to the movement of that person.

[0035] Alternatively, instead of the above aspect, when the period until the number of faces changes is equal to or longer than a predetermined period, the framing unit 24 may stop the framing for a predetermined period. According to this aspect, since the framing itself is stopped for a predetermined period, when there is a person who arrives late for a meeting, it becomes possible to further reduce the influence of framing according to the movement of that person.

[0036] 〔Third Embodiment〕 Next, an information processing system, an image processing method, and an image processing program according to the third embodiment of the present disclosure will be described. The camera module (information processing system) according to the present embodiment has a framing method by the framing unit 24b different from that of the first embodiment described above. Hereinafter, components common to the first embodiment will be denoted by the same reference numerals and the description thereof will be omitted, and different points will be mainly described.

[0037] FIG. 10 is a functional configuration diagram showing an example of an image processing function provided in the camera module according to the third embodiment of the present disclosure. The image processing unit 10b according to the present embodiment further includes a frame setting unit 26. The frame setting unit 26 has, for example, a plurality of frame sets each consisting of a combination of an enlarged frame Fla and a reduced frame Fsm.

[0038] The enlarged frame Fla is a frame that includes the reference frame Fre described in the first embodiment and has a size larger than that of the reference frame Fre. Also, the reduced frame Fsm is a frame that is the same as the reference frame Fre or is included in the reference frame Fre and has a size smaller than that of the reference frame Fre. And the frame setting unit 26 sets a frame set according to the number of faces included in the image.

[0039] For example, FIG. 11 shows an example of a frame set when one face is included in the image. As shown in FIG. 11, when one face is included in the image, the reduced frame Fsm is the same frame as the reference frame Fre. Also, the enlarged frame Fla has, for example, a size obtained by enlarging the vertical length of the reference frame Fre by +15% and the horizontal length by +10%.

[0040] Also, FIG. 12 shows an example of a frame set when a plurality of faces are included in the image. As shown in FIG. 12, when a plurality of faces are included in the image, the reduced frame Fsm is a frame having a size smaller than that of the reference frame Fre. For example, the reduced frame Fsm has a size obtained by reducing the vertical length of the reference frame Fre by -5% and the horizontal length by -10%. Also, the enlarged frame Fla has, for example, a size smaller than that when one face is included in the image. As an example, the enlarged frame Fla has a size obtained by enlarging the vertical length of the reference frame Fre by +10% and the horizontal length by +5%.

[0041] In this way, the enlarged frame Fla when one face is included in the image is set to be larger than the enlarged frame Fla when a plurality of faces are included in the image. This is because, for example, when one face is included in the image, the zoom is larger than when a plurality of faces are included. If the enlarged frame Fla is not set to an appropriate size, the framing will be sensitive to a slight movement of the subject, and there is a possibility that the framing will be repeated frequently. According to this embodiment, since the enlarged frame Fla when the image contains one face is set larger than the enlarged frame Fla when the image contains multiple faces, frequent repetition of framing can be suppressed, and shaking of the image can be reduced.

[0042] When a part of any region of interest set in the image exceeds the enlarged frame Fla set by the frame setting unit 26, or when the entire region of interest ROI is included in the reduced frame Fsm set by the frame setting unit 26, the framing unit 24 determines that the framing condition is satisfied and performs framing.

[0043] According to such a configuration, a frame set corresponding to the number of faces included in the image is set by the frame setting unit 26 from among a plurality of frame sets. Then, framing is executed based on the set frame set, that is, the enlarged frame Fla and the reduced frame Fsm. In this way, since the frame used for framing has hysteresis, the frequency of executing framing can be reduced. Further, at the time of executing framing, as described in the first embodiment above, framing based on reaction parameters corresponding to the number of faces included in the image is executed, so that the reaction of framing can be made slower as the number of faces included in the image is larger. From the above, it becomes possible to effectively reduce the shaking of the image, which has been a conventional problem.

[0044] As described above, the present disclosure has been described using embodiments, but the technical scope of the present disclosure is not limited to the scope described in the above embodiments. Various settings or improvements can be added to the above embodiments without departing from the gist of the disclosure, and the forms with such settings or improvements are also included in the technical scope of the present disclosure. Also, the above embodiments may be combined as appropriate. Also, the processing flow described in the above embodiments is also an example, and unnecessary steps may be deleted, new steps may be added, or the processing order may be changed within the range not departing from the gist of the present disclosure.

[0045] In the above-described embodiment, the case where the camera module 1 includes the image processing unit 10 and the camera module 1 functions as an information processing system has been exemplified and described, but the present invention is not limited thereto. For example, the image processing unit 10 may be provided in an information processing device (not shown) connected to the camera 2, and the information processing system may be realized by the camera 2 and the information processing device. In this case, the camera 2 may be integrally provided in the information processing device, or may be provided in a manner of being electrically connected to the information processing device as an external device. Examples of the information processing device include a notebook PC, a desktop PC, a tablet terminal, a smartphone, and the like.

Explanation of Signs

[0046] 1: Camera module 2: Camera 3: Housing 10: Image processing unit 10a: Image processing unit 10b: Image processing unit 12: Main storage device 13: Secondary storage device 14: Communication interface 21: Face detection unit 22: Region of interest setting unit 23: Parameter setting unit 24: Framing unit 24b: Framing unit 25: Determination unit 26: Frame setting unit Fla: Enlarged frame Fre: Reference frame Fsm: Reduced frame

Claims

1. A camera, a face detection unit that detects a face included in an image acquired by the camera, a region of interest setting unit that sets a region of interest based on the feature amount of the detected face, a frame setting unit that sets the frame set corresponding to the number of faces included in the image from among a plurality of frame sets each consisting of a combination of an enlarged frame and a reduced frame smaller in size than the enlarged frame, a framing unit that executes framing when any of the regions of interest exceeds the set enlarged frame or when the entire area of any of the regions of interest is included in the set reduced frame, comprising: An information processing system in which, in a plurality of the frame sets, the enlarged frame when one face is included in the image is set to a size larger than the enlarged frame when a plurality of faces are included in the image.

2. The information processing system according to claim 1, further comprising a parameter setting unit that sets a reaction parameter for framing according to the number of faces detected in the image, wherein the framing unit performs framing of the image using the set reaction parameter.

3. The information processing system according to claim 2, wherein the parameter setting unit sets a reaction parameter so that the reaction of framing becomes slower as the number of faces included in the image increases.

4. A computer, detects a face included in an image acquired by a camera, sets a region of interest based on the feature amount of the detected face, sets the frame set corresponding to the number of faces included in the image from among a plurality of frame sets each consisting of a combination of an enlarged frame and a reduced frame smaller in size than the enlarged frame, executes framing when any of the regions of interest exceeds the set enlarged frame or when the entire area of any of the regions of interest is included in the set reduced frame, An image processing method in which, in a plurality of the frame sets, the enlarged frame when one face is included in the image is set to a size larger than the enlarged frame when a plurality of faces are included in the image.

5. An image processing program for causing a computer to execute the image processing method according to claim 4.

Images