Image capture device, image capture device control method, program, and storage medium

Abstract

To improve the visibility of an image displayed on a display unit.SOLUTION: An imaging apparatus has: an imaging unit 201 that picks up an image of a subject to create an image; a control unit 404 that controls the exposure time or gain for every pixel group of an imaging surface in the imaging unit 201; an acquisition unit 401 that acquires information on a specific area in the image; a determination unit 402 that determines a specific pixel area on the imaging surface based on the information on the specific area acquired by the acquisition unit 401; and a changing unit 403 that changes at least one of the number, size, and shape of the pixel groups in the specific pixel area.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an imaging device, a control method for an imaging device, a program, and a storage medium. [Background technology]

[0002] Conventionally, there is known a technique for generating a captured image suitable for display by controlling the shooting conditions based on information from a display unit such as an electronic viewfinder or a live view monitor. Patent Document 1 discloses a technique for changing the exposure conditions based on information about the resolution, refresh rate, and contrast ratio of the display unit. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-121815 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to improve the visibility of an image displayed on a display unit. [Means for solving the problem]

[0005] In order to solve the above problem, an imaging device according to one aspect of the present invention includes an imaging unit that captures an image of a subject and generates an image, a control unit that controls an exposure time or a gain for each pixel group on an imaging surface of the imaging unit, and Display area an acquisition unit that acquires information about the above; a determination unit that determines a specific pixel region on the imaging surface based on the information acquired by the acquisition unit; and a modification unit that modifies at least one of the number, size, and shape of the pixel groups in the specific pixel region. the information includes information related to an electronic zoom operation in the image generated by the imaging unit, and the change unit changes the number of the pixel groups based on a zoom magnification of the electronic zoom operation. It is characterized by: [Effects of the Invention]

[0006] According to the present invention, it is possible to improve the visibility of an image displayed on a display unit. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram showing an example of a system configuration according to a first embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the device configuration of an imaging device according to a first embodiment. [Figure 3] FIG. 1 is a diagram showing an example of the device configuration of an information processing device according to a first embodiment. [Figure 4] FIG. 1 is a diagram showing an example of the functional configuration of an imaging apparatus according to a first embodiment. [Figure 5] FIG. 1 is a diagram showing an example of the functional configuration of an information processing apparatus according to a first embodiment. [Figure 6] 2 is a diagram showing an example of an image captured by the imaging device according to the first embodiment and a display screen of a display device (input device). FIG. [Figure 7] FIG. 2 is a view showing an example of a screen displayed on the display device according to the first embodiment. [Figure 8] FIG. 2 is a diagram showing an example of the operation of the imaging device according to the first embodiment. [Figure 9] 5A to 5C are diagrams showing examples of changes to the number of pixel groups, size, and shape of a pixel region corresponding to a specific region in the imaging device according to the first embodiment. [Figure 10] 4A and 4B are diagrams showing a state in which the number of pixel groups, size, and shape of a pixel region corresponding to a specific region are not changed in the imaging device according to the first embodiment. [Figure 11] 3A and 3B are diagrams showing examples of combining pixel groups corresponding to display areas and non-display areas in the imaging device according to the first embodiment. [Figure 12] 5A and 5B are diagrams showing examples of changing the number, size, and shape of pixel groups in a pixel region corresponding to a specific region based on the upper limit of the number of pixel groups in the imaging device according to the first embodiment. [Figure 13] 10 shows an example of combining pixel groups in a pixel region corresponding to a specific region in the imaging device according to the first embodiment. [Figure 14] 5A to 5C are diagrams showing examples of changes in the number, size, and shape of pixel groups when multiple images are displayed in the imaging device according to the first embodiment. [Figure 15] 10A and 10B are diagrams showing an example of an image captured by an imaging device according to a second embodiment and a display screen of a display device (input device). DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments described below are examples of means for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions. The present invention is not limited to the following embodiments. Furthermore, a configuration may be made by appropriately combining parts of each embodiment described below.

[0009] <Embodiment 1> (System Configuration) 1 is a diagram showing an example of the configuration of an imaging system including an imaging device according to this embodiment. The imaging system 100 includes a surveillance camera 101, a network 102, a client device 103, a display device 104 (display unit), and an input device (input unit) 105.

[0010] The surveillance camera 101 can communicate with the client device 103 via the network 102. The surveillance camera 104 captures an image of a subject, generates an image, and transmits the captured image to the client device 103 via the network 102. A display device 104 and an input device 105 are connected to the client device 103, and images sent from the surveillance camera 101 are displayed on the display device 104. The input device 105 is a keyboard, mouse, or the like, and is used to operate the client device 103. Note that operations on the client device 103 include setting the imaging conditions of the surveillance camera 101 via the network 102 and operating PTZ (Pan Tilt Zoom).

[0011] In this embodiment, the client device 103, the display device 104, and the input device 105 are separate entities, but the client device 103, the display device 104, and the input device 105 may be integrated into one unit, as in a notebook PC with a touch panel display. Also, they do not need to be connected via the network 102, and the surveillance camera 101 and the client device 103 may be directly connected. Furthermore, the camera (surveillance camera 101), the client device 103, the display device 104, and the input device 105 may all be integrated into one unit, as in a consumer camera with a touch panel display.

[0012] (Device configuration) 2 is a diagram showing an example of the configuration of an imaging device according to this embodiment. The surveillance camera 101 has an imaging unit 201, an encoder unit 202, a network I / F 203, a CPU (Central Processing Unit) 204, a RAM (Random Access Memory) 205, and a ROM (Read Only Memory) 206. The imaging optical system 200 is detachably provided on the surveillance camera 101.

[0013] The imaging optical system 200 is a lens that focuses light from a subject onto an imaging surface of an imaging element 201a (described later), and is composed of, for example, a zoom lens, a focus lens, and a blur correction lens.

[0014] In this embodiment, the surveillance camera 101 and the imaging optical system 200 are provided as separate and detachable bodies, but the surveillance camera 101 may have the imaging optical system 200, as in a camera with an integrated lens.

[0015] The imaging unit 201 captures an image of a subject using the imaging optical system 200 and generates an image. The imaging unit 201 includes an imaging element 201a, an amplifier 201b, and an image processing unit 201c.

[0016] The image sensor 201a converts light from a subject focused on the imaging surface by the imaging optical system 200 into an electrical signal for each pixel and outputs the signal. The image sensor 201a can set and change the exposure time (charge accumulation time) for each pixel or pixel group on the imaging surface. The image sensor 101b is an IC chip on which pixels made of photoelectric conversion elements such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal Oxide Semiconductor) sensor are arranged in a matrix. The image sensor 101a is sensitive mainly to visible light, with each pixel highly sensitive to one of red (R), green (G), and blue (B), but also has some sensitivity to infrared light. This allows it to clearly capture bright infrared light subjects, such as those in times of sunlight and in places illuminated by infrared lighting.

[0017] The amplifier 201b amplifies and outputs the electrical signal output from the image sensor 201a. An amplifier 201b is provided for each pixel, and the signal amplification factor (analog gain) can be set or changed for each pixel or pixel group of the image sensor 201a.

[0018] The image processing unit 201c performs A / D conversion on the analog electrical signal output from the amplifier 201b to a digital signal, and generates a digital image by performing image processing including demosaicing, white balance processing, gamma processing, etc. The image processing unit 201c amplifies or attenuates the digital value of the image signal output from each pixel or pixel group for each pixel or pixel group, thereby correcting the brightness of the image signal.

[0019] The encoder unit 202 encodes the image data output from the imaging unit 201 (image processing unit 201c) into a predetermined file format such as Motion JPEG, H264, or H265.

[0020] The network I / F 203 transmits the image data that has been encoded by the encoder unit 202 to the client device 103 via the network 102. The encoded image data may be stored in an internal storage device such as a RAM 205 or a ROM 206 (described later) or in a removable storage medium (not shown) such as an SD card. The image data may also be stored in these storage devices after being output by the image processing unit 201c. In this case, the image data is saved as RAW data before encoding.

[0021] The CPU 204 is a central processing unit that controls the surveillance camera 101 .

[0022] The RAM 205 temporarily stores computer programs executed by the CPU 204. The RAM 205 also provides a work area used when the CPU 204 executes processing. The RAM 205 also functions as a frame memory and a buffer memory.

[0023] The ROM 206 stores a program for the CPU 204 to control the surveillance camera 101 and the like.

[0024] 3 is a diagram showing an example of the configuration of a client device 103 connected to the imaging device according to this embodiment via the Internet. The client device 103 is an information processing device having a CPU 301, a RAM 302, a ROM 303, an input I / F 304, an output I / F 305, and a network I / F 406.

[0025] The CPU 301 is a central processing unit that controls the client device 103 .

[0026] The RAM 302 stores programs and the like that are used by the CPU 301 to control the client device 103 .

[0027] The ROM 303 stores a program for the CPU 301 to control the client device 103 and the like.

[0028] The input I / F 304 is an interface that is connected to the input device 105 and that accepts operations for the client device 103 input by the user via the input device 105 .

[0029] The output I / F 306 is an interface that connects to the display device 104 to display the image output from the surveillance camera 101 on the display device 104 .

[0030] The network I / F 306 is an interface that connects to the surveillance camera 101 via the network 102 , and is used to input operation information for the surveillance camera 101 and to receive images output from the surveillance camera 101 .

[0031] (Functional configuration) FIG. 4 is a diagram illustrating an example of the functional configuration of an imaging device according to this embodiment. For functions implemented by software, among the functional blocks shown in FIG. 4, a program for providing the function of each functional block is stored in a memory such as the ROM 206. The program is then loaded into the RAM 205 and executed by the CPU 204, thereby realizing the function. For functions implemented by hardware, a dedicated circuit may be automatically generated on an FPGA from a program for realizing the function of each functional block, for example, using a predetermined compiler. FPGA stands for Field Programmable Gate Array. Alternatively, a gate array circuit may be formed in a similar manner to an FPGA, and implemented as hardware. Alternatively, the function may be implemented using an ASIC (Application Specific Integrated Circuit). The functional block configuration shown in FIG. 1 is merely an example; multiple functional blocks may constitute a single functional block, or any functional block may be divided into blocks performing multiple functions.

[0032] The surveillance camera 101 includes an acquisition unit 401 , a determination unit 402 , a change unit 403 , an exposure control unit 404 , and a communication control unit 405 .

[0033] The acquisition unit 401 acquires image data captured by the imaging unit 201 and operation information input from the client device 103 via the network 102. The operation information input from the client device 103 is, for example, information about an operation related to the PTZ of the surveillance camera 101 or an operation to set the exposure time or gain for each pixel or pixel group of the imaging unit 201. The client device 103 receives an input of an image captured by the surveillance camera 101 and displays the image on the display device 104, and the image displayed on the display device 104 is the entire image captured by the surveillance camera 101 or a part of it. The client device 103 outputs to the surveillance camera 101 information about a display area, which is an area of ​​the image captured by the surveillance camera 101 to be displayed on the display device 104. The acquisition unit 401 acquires the display area information output from the client device 103 as operation information.

[0034] The determination unit 402 determines the pixel area (also referred to as pixel area) of the imaging unit 201 (image sensor 201a) based on the operation information acquired by the acquisition unit 401. More specifically, the determination unit 402 determines the pixel area corresponding to the display area information acquired by the acquisition unit 401. In other words, the determination unit 402 determines the pixel area of ​​the imaging unit 201 that is currently capturing (or will capture) the display area.

[0035] The change unit 403 changes at least one of the number and size of pixel groups in the pixel region determined by the determination unit 402. More specifically, the change is made based on user operation information and display area information acquired by the acquisition unit 401. The change may also be made based on the number of pixel groups in the display region before the change. For example, if there were nine pixel groups in the display region before the change, the change is made so that the number of pixel groups in the display region is maintained. The change is not limited to the number and size of pixel groups, and the shape of the pixel groups may also be changed. For example, pixel groups may be set to a shape that follows the main subject. In this embodiment, a case will be described in which the change unit 403 changes the number, size, and shape of pixel groups that have already been set in the pixel region determined by the determination unit 402. The change unit 403 is also a setting unit that sets the number, size, and shape of pixel groups on the imaging plane of the imaging unit.

[0036] The exposure control unit 404 determines and controls the exposure time or gain for each pixel group set on the imaging surface of the imaging unit 201. For example, the exposure control unit 404 determines and controls the exposure time or gain so that the average luminance value of the image for each pixel group becomes the median value of the data gradation that can be output.

[0037] Each function of the surveillance camera 101 may be configured as a function of the client device 103 .

[0038] FIG. 5 is a diagram illustrating an example of the functional configuration of a client device connected to the imaging device according to this embodiment via the Internet. For software-implemented functions of the functional blocks illustrated in FIG. 5, a program for providing the function of each functional block is stored in a memory such as the ROM 303. The program is then loaded into the RAM 302 and executed by the CPU 301, thereby implementing the function. For hardware-implemented functions, a dedicated circuit may be automatically generated on an FPGA from a program for implementing the function of each functional block, for example, using a predetermined compiler. FPGA stands for Field Programmable Gate Array. Alternatively, a gate array circuit may be formed in a similar manner to an FPGA, and implemented as hardware. Alternatively, an ASIC (Application Specific Integrated Circuit) may be used. The functional block configuration illustrated in FIG. 1 is merely an example; multiple functional blocks may constitute a single functional block, or any functional block may be divided into blocks performing multiple functions.

[0039] The client device 103 includes an input signal acquisition unit 501 , an input image acquisition unit 502 , a camera information acquisition unit 503 , a display control unit 504 , an operation information output unit 505 , and a communication control unit 506 .

[0040] The input signal acquisition unit 501 receives an input from the user via the input device 105 .

[0041] The input image acquisition unit 502 acquires an image received from the surveillance camera 101 via the network 102 as an input image.

[0042] The camera information acquisition unit 503 acquires camera information via the network 102 when the surveillance camera 101 captures an image of a subject or the like. The camera information is various camera setting information and image processing information when capturing an image of a subject or the like to acquire the image. Specifically, the camera information includes, for example, exposure parameters (camera setting information related to exposure) such as aperture, shutter speed, and gain, and information related to image processing related to brightness such as gamma correction, edge enhancement, and white balance. In addition, the camera information also includes the coordinates of the PTZ of the surveillance camera 101.

[0043] In accordance with instructions from CPU 301, display control unit 504 outputs an image captured by surveillance camera 101 to display device 104. The image output to display device 104 is the entire image or a part of the image captured by surveillance camera 101. For example, when an electronic zoom instruction or a PinP (Picture in Picture) instruction is received from a user and acquired by input signal acquisition unit 501, display control unit 504 performs electronic zoom processing or trimming processing on the image acquired by input image acquisition unit 502 and outputs the image to display device 104.

[0044] The operation information output unit 505 outputs the operation performed by the user on the surveillance camera 101, which is acquired by the input signal acquisition unit 501. For example, it outputs the PTZ operation of the surveillance camera 101, the position of the electronic zoom controlled by the display control unit 504, the area to be trimmed, and the like, to the surveillance camera 101 via the network 102. More specifically, it outputs the PT control value and position information indicating the area to be electronically zoomed (trimmed).

[0045] The communication control unit 506 comprehensively controls transmission and reception between the surveillance camera 101 and the client device 103 via the network.

[0046] The display control unit 504 may be provided in the surveillance camera 101, in which case electronic zoom processing (trimming processing) is performed in the image processing unit 201c of the surveillance camera 101, and the image after electronic zoom is sent to the client device 103 via the network 102. Also, if the surveillance camera 101 and the client device 103 are integrated (for example, a consumer camera), the functional configuration has all or some of the functions of the surveillance camera 101 and each function of the client device 103.

[0047] (Operation description) The operation of the imaging device according to this embodiment will be described with reference to FIGS. 6, 7, 8, 9 and 10. FIG.

[0048] 6 shows an image captured by the imaging unit 201 and an image output to the display device 104. In FIG. 6(a), an imaging area 601 shows an example of the area of ​​an image captured with the effective number of pixels of the imaging unit 201 (image sensor 201a). The grid lines in FIG. 6(a) indicate the correspondence between the imaging area 601 and pixel groups on the imaging surface of the imaging unit 201. In the case of FIG. 6(a), the area is shown divided into nine pixel groups. Pixel group 602 is one of the nine pixel groups and is the area indicated by the shaded area.

[0049] The subject 603 is an object imaged by the imaging unit 201, and is imaged using one pixel group. The exposure control unit 404 controls all pixels in the pixel group including the subject 603 to have the same exposure time or gain.

[0050] FIG. 6(b) shows an example of a display screen of an image sent to the client device 103 via the network 102 and output to the display device 104. The display screen 604 has an image display area 605 and an operation button display area 606. The image display area 605 is an area for displaying images captured by the imaging unit 201, i.e., still images, videos, and live view images. In this embodiment, the image display area 605 and the operation button display area 606 are provided within a single display screen 604. The grid lines drawn in the image display area 605 indicate nine pixel groups, as in FIG. 6(a). In this embodiment, the input device 106 and the display device 104 are described as being integrated. Therefore, the display screen 604 is a touch panel display, and the user can select each pixel group by touching or dragging. If the input device 106 and the display device 104 are separate, the user can select each pixel group by clicking (touching) or dragging the input device 106, such as a mouse, keyboard, or touchpad.

[0051] The imaging conditions, including exposure time and gain, of the pixel group selected by the user can be set by touching (clicking) a setting button 607 displayed in the operation button display area 606. The operation buttons include an ISO sensitivity button 608 (also called a gain button), a shutter speed button 609 (also called an exposure time button), a frame rate button 610, and a group setting button 611. For example, the pixel group is selected by the user touching (clicking) in the image display area 605 a pixel group for which the user wants to change the exposure time. After that, by touching (clicking) the setting button 607 and then touching the shutter speed button, a button for setting the exposure time, which will be described later, is displayed.

[0052] 7 is a diagram showing an example of a display screen after pressing the shutter speed button 609. A list of shutter speeds is displayed next to the shutter speed button 609, and the user can set the exposure time of a pixel group by touching any shutter speed from the list.

[0053] 8 is an example of a flowchart showing the operation of the imaging device according to this embodiment. When the imaging unit 201 captures an image, the CPU 204 temporarily stores a program stored in the ROM 206 in the RAM 205, and starts operation by reading the program stored in the RAM 205 line by line.

[0054] In step S801, information about the display area (specific area) of the display device 104 is first obtained from the client device 103 via the Internet 102. The display area is the area of ​​the image displayed in the image display area 605, and refers to the entire area or a portion of the image captured by the imaging unit 201. The display area varies depending on various operations, such as a user's electronic zoom operation. Note that the acquisition unit 401 identifies which area of ​​the image captured by the imaging unit 201 is the display area by acquiring information about the display area, such as coordinate information about the display area in the captured image and the aspect ratio of the display area. In addition, the acquisition unit 401 also acquires operation information input by the user to the client device 103 as information about the specific area, so the display area may be identified in advance based on the user's operation information. For example, when a user inputs an electronic zoom operation for an image to the client device 103, the acquisition unit 401 acquires information about the electronic zoom operation instructed by the user and identifies the display area. The information about the electronic zoom is the coordinate information of the image input by the user to the client device 103. For example, in an electronic zoom operation, the user selects a desired area (specific area) on the image to be zoomed in as a rectangle by dragging the image display area 605 using a mouse or a finger. In this case, the acquisition unit 401 acquires coordinate information of two points, the start point and end point of the drag operation (or coordinate information of either one of the points and the size of the rectangle).

[0055] In step S802, the determination unit 402 determines the pixel area (hereinafter referred to as pixel area) of the imaging unit 201 corresponding to the display area based on the information related to the display area or the user operation information acquired in step S801. Since the acquisition unit 401 has acquired the coordinate information, aspect ratio, and user operation information of the display area, the determination unit 402 determines the pixel area on the imaging surface of the imaging unit 201 (image sensor 201a) corresponding to the display area based on this information.

[0056] In step S803, the modification unit 403 modifies at least one of the number or size of pixel groups in the pixel region determined by the determination unit 402. The extent to which the number or size of pixel groups is modified is based on the ratio of the size of the display region to the imaging region of the imaging unit 201, the upper or lower limit of the number of blocks that can be set by the user or designer, and instructions from the user regarding the number and size of pixel groups (for example, modifying the number, size, and shape of pixel groups in the pixel region so that the number of pixel groups is smaller than the upper limit (predetermined value)). The modification unit 403 may also determine whether to modify the number of pixel groups, etc., by determining whether the size of the specific region (display region) acquired by the acquisition unit 401 is larger than the image captured by the imaging unit 201. For example, if the size of the specific region is smaller than the size of the image captured by the imaging unit 201, the modification unit 403 modifies at least one of the number, size, and shape of pixel groups. The region in which the number or size of pixel groups is modified is not limited to the pixel region determined by the determination unit 402. Therefore, the modification unit 403 can appropriately modify the number and size of pixel groups in pixel areas other than the pixel area determined by the determination unit 402 based on the above information. That is, the number and size of pixel groups on the entire imaging surface may be appropriately modified in accordance with the modification of the number and size of pixel groups in a pixel area. Also, the number and size of pixel groups that have already been set may not need to be modified. Alternatively, they may be modified based on the number of pixel groups in the display area before the modification.

[0057] In step S804, the exposure control unit 304 controls the exposure time or gain for each pixel group changed by the change unit 403.

[0058] 9 shows an example of an image captured by applying the operation of the imaging device according to this embodiment and a displayed image. Also, FIG. 9 is a diagram transitioning from FIG. 6, and it will be explained that the total number of pixel groups is set to an upper limit of 10. FIG. 9(a) shows the entire image captured by the imaging unit 201, which is the same region as the imaging region 601 in FIG. 6. Also, the subject 603 is the same subject as in FIG. 6.

[0059] 9(b), unlike FIG. 6(b), the subject 603 is displayed in the entire image display area 605. In other words, a part of the image captured by the image capturing unit 201 is the display area. This can occur due to the user's operation of the electronic zoom, etc.

[0060] 9(b) from the client device 103, and in step S802 the determination unit 402 determines a pixel area corresponding to the display area. In FIG. 9(a), the display area 901 is the area displayed in the image display area 605 in the imaging area 601. The non-display area 902 is the area not displayed in the image display area 605 in the imaging area 601, and is the area indicated by the diagonal lines. Furthermore, the grid lines in FIG. 9(a) indicate pixel group boundaries after the number and size of pixel groups have been changed by the change unit 403.

[0061] In step S803, the change unit 403 changes the number and size of pixel groups in the imaging area 601 compared to those in FIG. 6(a). There are nine pixel groups in the display area 901, and one pixel group in the non-display area 902. In this embodiment, the number of pixel groups in the display area 901 is changed to maintain the number of pixel groups (nine) in the display area of ​​FIG. 6. Furthermore, because the upper limit of pixel groups that can be set is set to 10, the number of pixel groups in the non-display area 901 is set to one. Furthermore, all pixel groups in the display area 901 are of uniform size. A more specific example will be shown separately. Since the ratio of the size of the imaging area 601 to the size of the display area 901 is approximately 9:1, the change unit 403 changes the number of pixel groups in the pixel area to nine. The size is determined by equally dividing the pixel area, and the shape is similar to the shape of the pixel area.

[0062] Fig. 10 shows an example of an image captured when the operation of the imaging device according to this embodiment is not applied. As with Fig. 9, the transition from Fig. 6 will be explained. In Fig. 10(a), the number and size of pixel groups in the display area 901 and non-display area 902 are unchanged. Therefore, the number of pixel groups for the display area 901 and non-display area 902 differs between Fig. 9 and Fig. 10.

[0063] If there is one pixel group in the display area 901, the exposure control unit 304 can control only one exposure time or gain for the display area 901. Therefore, when there is a large difference in brightness within the display area 901, bright areas will be overexposed or dark areas will be crushed to black, reducing the visibility of the image within the display area 901. On the other hand, in this embodiment, the number of pixel groups within the display area 901 is changed to nine as shown in FIG. 9( a), so the exposure control unit 304 can control the display area 901 with different exposure times or gains for each pixel group. This allows dark areas within the display area 901 to be captured with a longer exposure time and bright areas with a shorter exposure time. Therefore, the dynamic range of the display area 901 is improved, and images with high visibility can be displayed.

[0064] A supplementary explanation of this embodiment will be provided. In Figure 9, the number and size of pixel groups in the display area 901 are changed so that the number of pixel groups is the same as that of the pixel groups in the display area in Figure 6, and the number of pixel groups in the non-display area 902 is set to 1 in accordance with the upper limit of the settable number of pixel groups. As a result, the total number of pixel groups in the imaging area 601 is 10. In this case, if the upper limit for the number of pixel groups is 9, the number of exposure areas must be kept within the upper limit of 9. There are four methods for keeping the number within the upper limit:

[0065] The first is that the image processing calculations of the image processing unit 201c are not performed in the non-display area 902. Because the non-display area 902 is an internal area that is visually recognized by the user, omitting calculations related to exposure conditions in the image processing unit 201c prevents an increase in the data processing load. By not performing image processing, the processing performed for each pixel group can be reduced by one, thereby virtually suppressing an increase in the number of pixel groups to be processed.

[0066] The second method is to combine one of the pixel groups in the display area 901 that is adjacent to the non-display area 902 with a pixel group in the non-display area. A specific method for combining exposure areas will be described with reference to FIG. 11. FIG. 11 is a diagram illustrating the combination of pixel groups in the display area and pixel groups in the non-display area. The imaging area 601 in FIG. 11(a) has the same imaging range as the imaging area 601 in FIG. 6(a). In this case, the pixel group located in the upper right corner of the display area 901 is combined with a pixel group in the non-display area 902. The combined boundary is shown by a dotted line. By combining pixel groups in this manner, an increase in the number of pixel groups can be suppressed. In this case, it is desirable to limit the calculation range of the exposure time or gain of the combined pixel group to only the image display area 605. In other words, the image information of the non-display area 902 is not used to calculate the exposure time or gain of the combined pixel group, but rather the calculation is based on image information (e.g., luminance information) corresponding to the upper right corner of the image display area 605. This makes it possible to set an appropriate exposure time or gain by referring only to the luminance information of the image display area 605 that is visible to the user, without referring to the luminance of the non-display area 902. Note that the operation of combining pixel groups is performed by the change unit 403 in step S803.

[0067] The third method is to divide the pixel groups in the display area 901 evenly by the number of pixel groups, minus one in the non-display area 902. A specific method for changing the size and shape of pixel groups will be described with reference to FIG. 12. FIG. 12 is a diagram showing the number of pixel groups in the display area reduced by the number of pixel groups in the non-display area. The imaging area 601 in FIG. 12(a) has the same imaging range as the imaging area 601 in FIG. 6(a). The size of the pixel groups is changed so that the number of pixel groups in the display area 901 in FIG. 12(a) is reduced from nine to eight, the number corresponding to one pixel group in the non-display area 902. This makes it possible to suppress an increase in the number of pixel groups, even if one pixel group is allocated to the non-display area 902. Furthermore, the image display area 605 in FIG. 12(b) has eight pixel groups, which is one fewer than the image display area 605 in FIG. 8(b). However, the number of pixel groups is greater in the image display area 605 in FIG. 12(b) than in the image display area 605 in FIG. 10(b).

[0068] The fourth method is to combine adjacent pixel groups within a display area. A specific method for combining adjacent pixel groups within a display area will be described with reference to FIG. 13. FIG. 13 is a diagram illustrating the combination of adjacent pixel groups within a display area. The imaging area 601 in FIG. 13(a) has the same imaging range as the imaging area 601 in FIG. 6(a). In this case, adjacent pixel groups in the display area 901 in FIG. 13(a) are combined. The combined pixel groups are illustrated as a combined area 1301 with diagonal lines. This makes it possible to suppress an increase in the number of pixel groups even if one pixel group is allocated in the non-display area. Furthermore, looking at the image display area 605 in FIG. 13(b), it can be seen that there are eight pixel groups. This is one fewer than the display image 402 in FIG. 6(b). However, the number of pixel groups has increased when comparing the image display area 605 in FIG. 13(b) with the image display area 605 in FIG. 10(b). The pixel groups to be combined may be any pixel groups within the display area, and the pixel groups are combined by the change unit 403 in step S803.

[0069] Although the number of pixel groups has been exemplified as 9 or 10, the number is not limited to this, and the lower limit of the number of pixel groups may be any number greater than or equal to 2. Similarly, the upper limit of the number of pixel groups is not limited.

[0070] It is desirable for the designer to set the upper limit on the number of pixel groups based on the frame rate and power consumption. Furthermore, the upper limit on the number of pixel groups may be changed based on the load on the CPU data processing in the surveillance camera 101 or the client device 103, or the usage of the RAM calculation area. In particular, a table may be prepared that lists the drive conditions (gain, frame rate, power consumption, number of effective pixels, etc.) of the imaging unit 201 (imaging element 201a) and the upper limit on the number of pixel groups, and the upper limit on the number of pixel groups may be set based on the table.

[0071] It is desirable that the number or range of pixel groups be set arbitrarily by the designer or user, as long as it does not exceed the upper limit of the number of pixel groups. In this embodiment, a case where the number of pixel groups is set to 9 has been mainly described as an example. To set the number or range of pixel groups, the user touches (clicks) the group setting button 611 in the operation button display area 606, for example. This causes the display device 104 to transition to a setting screen where the number or range of pixel groups can be set, or a setting button is displayed. This allows the user to capture an image with the number of pixel groups they desire, thereby further improving the visibility of the image displayed in the display area 605.

[0072] The number of pixel groups may be linked to user operation information acquired by the acquisition unit 401. For example, it may be linked to operation information (zoom magnification and zoom position) of the user's electronic zoom operation. In this case, the change unit 403 changes the number of pixel groups so that it increases as the zoom magnification increases. In this way, by increasing the number of pixel groups in the display area as the display area becomes narrower, the dynamic range is improved even in finer areas within the imaging area, making it possible to display an image with high visibility.

[0073] In this embodiment, a method for changing the size of pixel groups so that they are uniform in size relative to the display area has been described, but the sizes of pixel groups do not have to be uniform. Regarding the shape, while a rectangular shape has been described, the pixel group shape may also be trapezoidal or match the shape of the main subject (e.g., a person, animal, building, etc.). However, because a pixel group is made up of one or more pixels on the imaging surface of the image sensor 201a, it cannot be changed to a shape that deviates from the pixel array or to a size smaller than the size of a pixel.

[0074] As explained above, the image display area 605 is an area for displaying an image captured by the image capturing unit 201. The captured image refers to an image captured using the effective pixels of the image capturing element 201a. It does not include an image captured using surplus pixels (pixels used only for image processing such as filtering and black level correction).

[0075] In this embodiment, the case where an image is displayed on the display device in one area has been described, but there may be multiple areas. Referring to FIG. 14, a case where an image is displayed on the display device in multiple areas will be described. In particular, a case where different areas of an image captured by the surveillance camera 101 are displayed on the display device 104 will be described. A case where images captured by multiple surveillance cameras are displayed on the same screen will be omitted, as the operation is the same as described above. Furthermore, in this embodiment, a case where an image captured by the surveillance camera 101 and an image of a partial area of ​​that image are displayed will be described. FIG. 14 is a diagram schematically illustrating how multiple areas of an image captured by the imaging device according to this embodiment are displayed on the display device.

[0076] 14(a), the imaging area 601 is the same as the imaging range in Fig. 6. The display area 1401 is a display area different from the display area 901 displayed within the display screen 604. The display area 1401 is the area indicated by the hatched portion, and is the area indicated by the imaging area 601 other than the display area 901.

[0077] 14(b) is a diagram showing two image display areas within the display screen 604. An image from the imaging area 601 is displayed in the image display area 1402, and an image from the display area 901 is displayed in the image display area 1403.

[0078] In this case, since both display area 1401 and display area 901 are displayed on display screen 604, modification unit 403 modifies at least one of the number, size, and shape of pixel groups corresponding to each display area. In this embodiment, the number of pixel groups corresponding to each display area is modified so as to maintain the number of pixel groups in Fig. 6. Therefore, the number of pixel groups corresponding to display area 901 and the number of pixel groups corresponding to display area 1401 are both modified to nine.

[0079] As a result, even when images of multiple regions in an image captured by the imaging unit are displayed, at least one of the number, size, and shape of pixel groups in the pixel regions corresponding to each region can be changed. Therefore, both display images have improved dynamic ranges and high visibility. Note that at least one of the number, size, and shape of pixel groups may be changed for only one of the multiple display images. In this case, it is desirable that the image to be displayed be the image on which the user focuses. For example, it is desirable that the image on which the user focuses is selected using the display unit 106, that the image last operated on is determined to be the image of interest, or that the image over which the cursor operated using the operation unit is positioned is determined to be the image of interest.

[0080] In this embodiment, when both an image of the entire imaging area and an image obtained by cropping and enlarging a portion of the imaging area are displayed, the number of pixel groups for each pixel area is nine. The number, size, shape, and ratio of pixel groups in each pixel area can be changed as desired. For example, the number and size ratio of pixel groups may be changed based on the size ratio of each pixel area.

[0081] The same applies when multiple client devices are connected to the network 102. An acquisition unit 401 acquires the image area to be displayed on the display device from each client device, and a determination unit 402 determines the pixel area corresponding to each area. A modification unit 403 modifies the number, size, and shape of pixel groups in the pixel area.

[0082] There are also cases where the number, size, and shape of pixel groups in pixel regions corresponding to areas not displayed on the display device 104 are changed. For example, when the recording mode of the client device is on, the number, size, and shape of pixel groups in pixel regions corresponding to areas not displayed on the display device 104 are changed as well. In this case, in step S801, the acquisition unit 401 acquires information regarding whether the recording mode of the client device 103 is on or off. Next, in step S803, if the recording mode is on, the change unit 403 changes at least one of the number, size, and shape of pixel groups in pixel regions corresponding to the non-display area 902. This makes it possible to display an image with improved dynamic range and high visibility even when recording an image of an area not displayed on the display device 104 and then displaying the recorded image.

[0083] In this embodiment, an area displayed on the display device 104 has been described as an example of a specific area. Other examples of a specific area include a face recognition area and a human body recognition area. For example, when the client device 103 performs face recognition or human body recognition, the acquisition unit 401 acquires information on the specific area, which is the area for face recognition (face recognition area) or the area for human body recognition (human body recognition area). Then, at least one of the number, size, and shape of pixel groups in the pixel area corresponding to the face recognition area or the human body recognition area is changed. This improves the dynamic range of the image in the face recognition area or the human body recognition area, resulting in an image with high visibility. This improves the recognition accuracy of the client device 103.

[0084] <Embodiment 2> In this embodiment, a case where display images are superimposed, as typified by PIP (Picture In Picture), will be described. In particular, in this embodiment, a case where an image captured by a surveillance camera 101 and an image of a partial area of ​​that image are superimposed and displayed will be described. Note that the device configurations and functional configurations of the imaging device and information processing device (client device) according to this embodiment are the same as those in the first embodiment, and therefore description thereof will be omitted.

[0085] The imaging device according to this embodiment differs from the first embodiment in that a pixel group in an area where an image of a specific area is superimposed is combined with an area other than the specific area.

[0086] A method for changing pixel groups when display images are superimposed (PIP) will be described with reference to Fig. 15. Fig. 15 is a diagram showing a PIP display area and its pixel groups.

[0087] 15(a) is the same as the imaging range in FIG. 6(a), and a selected area 1501, which is a portion of the subject 603, is an area that the user specified by dragging the image display area 605. On the other hand, a non-selected area 1502 is an area that the user did not select. A superimposed area 1503 is an area where an image obtained by enlarging the image in the selected area 1501 using electronic zoom processing is displayed, and is indicated by the hatched area.

[0088] In FIG. 15B, the image display area 605 displays a PIP display image 1504 in which the image in the selection area 1501 is enlarged to the superimposition area 1501 .

[0089] Here, a method for changing pixel groups in pixel regions corresponding to the selected region 1501, the non-selected region 1502, and the overlap region 1503 will be described. Note that the description will be given assuming that the upper limit of the number of pixel groups is 10. Also, since the operation of the imaging device in this embodiment is almost the same as in embodiment 1, the operation of this embodiment in each step in FIG. 8 will be described. Note that step S804 is the same as in embodiment 1, so its description will be omitted.

[0090] In step S801, information about a specific region is acquired. Here, information about the selected region 1501, the non-selected region 1502, and the overlapping region 1503 is acquired as the information about the specific region. The selected region 1501 and the overlapping region 1503 are determined by the user selecting each region by a drag operation or the like in the image display region 605, so the acquisition unit 401 acquires operation information input by the user to the client device 103. More specifically, the coordinates and sizes of the selected region 1501 and the overlapping region 1503 set by the user are acquired. In this embodiment, a case will be described in which the user sets the selected region 1501 and the overlapping region 1503, respectively. However, the user may select only the selected region 1501. In that case, the display control unit 504 selects an area that does not overlap the selected region 1501 as the overlapping region 1503, thereby setting information about the overlapping region 1503. In this way, the acquisition unit 401 acquires the selected region 1501 set by the user and the overlapping region 1503 set by the display control unit 504. The non-selected region 1502 is a region other than the selected region 1501, and is therefore uniquely determined from information about the selected region 1501. Therefore, the information about the non-selected region 1502 is determined and acquired by the acquisition unit 401 based on information about the selected region 1501.

[0091] In step S802, the pixel regions corresponding to the specific region are determined. In this embodiment, the determining unit 402 determines the pixel regions corresponding to the selected region 1501, the non-selected region 1502, and the overlap region 1503.

[0092] In step S803, the modification unit 403 modifies at least one of the number, size, and shape of pixel groups in the determined pixel region. In this embodiment, the upper limit of pixel groups is set to nine, so the modification is performed so that the total number of pixel groups in each pixel region is equal to or less than the upper limit. Here, the ratio of the number of pixel groups in the pixel regions corresponding to the selected region 1501 and the non-selected region 1502 is determined to be 4:6. The modification is also performed so that the size and shape of the pixel groups in each pixel region are the same. The pixel groups in the pixel region corresponding to the overlap region 1503 are combined with the pixel groups in the non-selected region 1502.

[0093] In the image display area 605, there is an enlarged PIP display image 1504, a non-selected area 1502 that is not enlarged, and all displayed areas of the selected area 1501. By changing at least one of the size, number, and shape of the pixel groups in each area, it is possible to display an image with an improved dynamic range and high recognition accuracy.

[0094] In addition, in this embodiment, the area of ​​the overlapping area 1503 is acquired, and image processing is not performed on the image captured by the pixel group of the overlapping area, thereby reducing the load on the CPU in the surveillance camera 101 and the amount of memory used for calculations.

[0095] <Other embodiments> The present invention can be realized by a process of reading and executing a program that realizes one or more functions of the above-described first embodiment. This program is supplied to a system or device via a network or a storage medium, and is read and executed by one or more processors in the computer of the system or device. The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0096] 101 Surveillance Camera 201 Imaging unit 401 Acquisition Department 402 Decision Section 403 Changes 404 Exposure control unit

Claims

1. an imaging unit that captures an image of a subject and generates an image; a control unit that controls an exposure time or a gain for each pixel group on an imaging surface of the imaging unit; an acquisition unit that acquires information about a display area in the image; a determination unit that determines a specific pixel region on the imaging surface based on the information acquired by the acquisition unit; a change unit that changes the number of the pixel groups in the specific pixel region, the information includes information regarding an electronic zoom operation on the image generated by the imaging unit, the change unit changes the number of pixel groups based on a zoom magnification of the electronic zoom operation. An imaging device characterized by:

2. The imaging device according to claim 1 , wherein the change unit changes the number of pixel groups in the pixel area when the size of the display area is smaller than the size of the image.

3. 3. The imaging device according to claim 1, wherein the change unit changes the number of pixel groups in the pixel region so that the number of pixel groups is smaller than a predetermined value.

4. 4. The imaging device according to claim 1, wherein, when the number of pixel groups is greater than a predetermined value, the modification unit modifies the number of pixel groups so as to combine the pixel groups in the pixel region corresponding to the display area with the pixel groups in the pixel region corresponding to an area other than the display area.

5. further comprising an image processing unit that performs image processing on the image captured by the imaging unit, 5. The imaging device according to claim 1, wherein the image processing unit does not perform image processing on the pixel groups in the pixel region corresponding to a region other than the display region.

6. 6. The imaging device according to claim 1, wherein the change unit combines pixel groups in the specific pixel region with pixel groups in regions other than the specific pixel region.

7. 7. The imaging device according to claim 6, wherein the control unit controls an exposure time or a gain of the pixel groups combined by the change unit based on an image captured by the pixel groups in the display area.

8. 8. The imaging device according to claim 1, wherein the change unit changes the number of pixel groups on the entire imaging surface so that the sum of the number of pixel groups in the specific pixel region and the number of pixel groups in pixel regions other than the specific pixel region is equal to or less than a predetermined upper limit value.

9. 9. The imaging device according to claim 1, wherein the change unit changes the ratio between the number of pixel groups in the specific pixel area and the number of pixel groups in pixel areas other than the specific pixel area based on the ratio between the size of the image captured by the imaging unit and the size of the image in the display area.

10. The imaging device described in Claim 1, characterized in that the higher the zoom magnification, the greater the number of pixel groups in the specific pixel area.

11. an imaging step of capturing an image of a subject to generate an image; a control step of controlling an exposure time or a gain for each pixel group on the imaging surface; an acquisition step of acquiring information about a display area in the image; a determination step of determining a specific pixel region on the imaging surface based on the information acquired in the acquisition step; a modifying step of modifying the number of pixel groups in the specific pixel region, the information includes information regarding an electronic zoom operation on the image generated in the imaging step, the changing step changes the number of pixel groups based on the zoom magnification of the electronic zoom operation.

10. A method for controlling an imaging device, comprising:

12. The method for controlling an imaging device according to claim 11, wherein the higher the zoom magnification, the greater the number of pixel groups in the specific pixel region.

13. A program for causing a computer to execute the control method according to claim 11 or 12.

14. A computer-readable storage medium storing the program according to claim 13.

Images