Electronic apparatus, method for controlling electronic apparatus, program, and storage medium

Abstract

To provide a technique that can acquire an image including an effective region and an ineffective region and having less noise.SOLUTION: An electronic apparatus of the present invention has: acquisition means that acquires a picked-up image including an effective region where a video is present and an ineffective region where a video is not present; and processing means that performs compression processing for reducing the amount of data on the picked-up image. The processing means performs dither processing in the compression processing, and for pixels included in the ineffective region, determines whether to perform the dither processing on the basis of an estimated value of the display luminance of the pixels before the compression processing is performed.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to an electronic device, a control method for an electronic device, a program, and a storage medium.

Background Art

[0002] In recent years, a technique has been known in which two parallax circular fisheye images (parallax images) are captured using two cameras, and the two parallax images are reproduced as a stereoscopic virtual reality (VR) image. Further, a technique is known in which two parallax images can be captured at once using a single camera equipped with two optical systems (a binocular lens, a VR lens) facing the same direction. In a camera equipped with a binocular lens, an image including an area of two circular fisheye images (effective area, effective video area, area used for VR reproduction) where an image exists and an area of a black image (invalid area, area not used for VR reproduction) where no image exists is captured.

[0003] The display devices included in a camera are an EVF (Electronic View Finder) and a display panel. In recent years, the luminance range that can be displayed on these display devices has been increasing. However, the bit width of the image data that can be input to these display devices is less than 10 bits, and there may be a large luminance difference between 1 bit. When the luminance difference between 1 bit is large, the gradation of the image is low. Therefore, various image processes are performed in the camera in order to enhance the gradation of the image. One of these image processes is dither processing, which is a process for pseudo-enhancing the gradation of an image.

[0004] By the way, in the invalid area, dark noise often occurs. When dither processing is performed on this dark noise, the dark noise is emphasized. Further, when a parallax image captured using a binocular lens is displayed on a display device with a high brightness setting value, the dark noise emits light unnaturally.

[0005] In order to reduce such noise, for example, the following methods can be considered. Patent Document 1 discloses a method of changing the gradation value for each pixel of an input image based on the average value of brightness in a predetermined area centered on the pixel. Patent Document 2 discloses a method of not performing development processing on the pixels included in the invalid area.

Prior Art Documents

Patent Documents

[0006]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0007] However, in the technique disclosed in Patent Document 1, in an area where a predetermined area includes a valid area and an invalid area, the average value of brightness may become brighter due to the influence of the valid area, and dark noise may not be reduced. Further, in the technique disclosed in Patent Document 2, even when the noise is not noticeable at all, it is necessary to perform different processes for the pixels included in the valid area and the pixels included in the invalid area.

[0008] Therefore, an object of the present invention is to provide a technique capable of obtaining an image with less noise in an image including a valid area and an invalid area.

Means for Solving the Problems

[0009] A first aspect of the present invention includes acquisition means for acquiring a captured image including a valid region where an image exists and an invalid region where no image exists, and processing means for performing compression processing for reducing the data amount on the captured image. The processing means performs dithering processing in the compression processing, and for pixels included in the invalid region, determines whether to perform the dithering processing based on an estimated value of the display luminance of the pixel before the compression processing is performed. This is an electronic device characterized by this.

[0010] A second aspect of the present invention includes acquisition means for acquiring a captured image including a valid region where an image exists and an invalid region where no image exists, and processing means for performing replacement processing for replacing the pixel values of pixels included in the invalid region with a predetermined value. The processing means determines whether to perform the replacement processing based on an estimated value of the display luminance of the pixels included in the invalid region before the replacement processing is performed. This is an electronic device characterized by this.

Advantages of the Invention

[0011] According to the present invention, in an image including a valid region and an invalid region, an image with less noise can be acquired.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Embodiments for Carrying Out the Invention

[0013] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0014] <First Embodiment> FIG. 1 is a block diagram showing an example of the configuration of an image display system according to the first embodiment of the present invention. In the first embodiment, the case where the image display system is a camera (imaging device, electronic device) 100 will be described as an example.

[0015] The camera 100 includes an acquisition unit 101, a memory 102, an OSD generation unit 103, a system control unit 104, an image processing unit 105, and a display unit 106.

[0016] Based on the control signal received from the system control unit 104, the acquisition unit 101 performs imaging processing and focus adjustment processing, and acquires an imaging image (RGB image data). Further, the acquisition unit 101 acquires information (lens information) regarding the optical system of the lens unit attached to the camera 100. The lens information will be described later with reference to FIG. 4(A). The acquisition unit 101 outputs the imaging image to the memory 102 and transmits the lens information to the system control unit 104.

[0017] The memory 102 stores (stores) the imaging image acquired from the acquisition unit 101. The image data stored in the memory 102 is transmitted to the display unit 106 via the image processing unit 105. Further, the memory 102 stores the OSD (On Screen Display) image generated by the OSD generation unit 103, and based on the control signal received from the system control unit 104, transmits the OSD image to the image processing unit 105.

[0018] The OSD generation unit 103 generates an OSD image based on the control signal received from the system control unit 104. Further, the OSD generation unit 103 generates information (area information) indicating whether the area containing each pixel is a valid area or an invalid area based on the lens information received from the system control unit 104. Details of the valid area, invalid area, and area information will be described later.

[0019] The system control unit 104 is a control unit composed of at least one processor and / or at least one circuit. That is, the system control unit 104 may be a processor, a circuit, or a combination of a processor and a circuit. The system control unit 104 controls the entire camera 100. The system control unit 104 receives lens information from the acquisition unit 101. Further, the system control unit 104 transmits the brightness setting information of the display unit 106 to the display unit 106. The system control unit 104 transmits the area information and the brightness setting information of the display unit 106 to the image processing unit 105. The system control unit 104 determines the processing performed by the image processing unit 105 and transmits the information of the processing to the image processing unit 105.

[0020] The image processing unit 105 performs image processing on the image data received from the memory 102 based on the information received from the system control unit 104. The image processing performed by the image processing unit 105 includes, for example, gamma conversion, color conversion, pixel interpolation, resizing processing, dithering processing, OSD processing, white balance adjustment, sharpness processing, and RGB - YUV conversion. The image processing unit 105 transmits the captured image after the image processing to the display unit 106.

[0021] The display unit 106 is an EVF (Electronic View Finder) and a display panel composed of a display such as an LCD or an organic EL. One pixel of the display unit 106 is further composed of RGB (red, green, blue) sub - pixels. The display unit 106 displays the captured image received from the image processing unit 105 based on the brightness setting information received from the system control unit 104.

[0022] FIG. 2 is a schematic diagram showing an example of the configuration of the lens unit 200.

[0023] The lens unit 200 is a type of interchangeable lens unit that can be attached to and detached from the camera 100. The lens unit 200 is a binocular lens (VR lens) unit capable of imaging right and left images with parallax. The lens unit 200 has two optical systems, and each of the two optical systems can image a subject within a wide viewing angle range of approximately 180 degrees. Specifically, with each of the two optical systems of the lens unit 200, a subject with a viewing angle (angle of view) of 180 degrees in the horizontal direction (horizontal angle, azimuth angle, yaw angle) and 180 degrees in the vertical direction (vertical angle, pitch angle) can be imaged. That is, with each of the two optical systems, a range of the front hemisphere can be imaged.

[0024] The lens unit 200 includes a right-eye optical system 201 having a plurality of lenses and a reflection mirror, etc., a left-eye optical system 202 having a plurality of lenses and a reflection mirror, etc., and a lens system control circuit 203. The right-eye optical system 201 has a lens disposed on the subject side, and the left-eye optical system 202 has a lens disposed on the subject side. The lenses of the right-eye optical system 201 and the lenses of the left-eye optical system 202 face the same direction, and their optical axes are substantially parallel.

[0025] The lens unit 200 is a binocular lens unit (VR180 lens unit) for obtaining a VR180 image, which is one of the formats of VR (Virtual Reality) images enabling binocular stereoscopic vision. The lens unit 200 has a fisheye lens capable of capturing a range of approximately 180 degrees in each of the right-eye optical system 201 and the left-eye optical system 202. Note that the lenses of each of the right-eye optical system 201 and the left-eye optical system 202 capture The possible range may be about 160 degrees, which is narrower than the 180-degree range. The lens unit 200 can form a right image formed through the right optical system 201 and a left image formed through the left optical system 202 on one or two image sensors of the camera 100 to which the lens unit 200 is attached. In the camera 100, the right image and the left image are formed on one image sensor (imaging sensor), and one image (stereoscopic image) in which the right image area (area of the right image) and the left image area (area of the left image) are arranged side by side is generated.

[0026] The lens unit 200 is attached to the camera 100 via the lens mount portion 204 and the camera mount portion 205 of the camera 100. By doing so, the system control unit 104 of the camera 100 and the lens system control circuit 203 of the lens unit 200 are electrically connected via the communication terminal 206 of the camera 100 and the communication terminal 207 of the lens unit 200. The lens system control circuit 203 transmits lens information and receives a control signal related to the focus adjustment of the lens.

[0027] In FIG. 2, the right image formed through the right optical system 201 and the left image formed through the left optical system 202 are formed side by side on the imaging unit of the camera 100. That is, two optical images (subject images) are respectively formed on two areas of one image sensor (imaging sensor) by the right optical system 201 and the left optical system 202. The imaging unit converts the formed optical image (optical signal) into an analog electrical signal. By using the lens unit 200 in this way, one image including two image areas with parallax can be acquired from two locations (optical systems) of the right optical system 201 and the left optical system 202. By dividing the acquired image into an image for the left eye and an image for the right eye and performing VR display, the user can view a stereoscopic VR image in a range of approximately 180 degrees. That is, the user can view the VR180 image in stereoscopy.

[0028] Here, the VR image is an image that can be VR - displayed as described later. The VR image includes an omnidirectional image (equirectangular image) captured by an omnidirectional camera (full - sphere camera), a panoramic image having a video range (effective video range) wider than the display range that can be displayed at once on the display unit, and the like. Also, the VR image is not limited to still images, but also includes moving images and live images (images acquired almost in real - time from a camera). The VR image has a video range (effective video range) of up to 360 degrees in the horizontal direction and 360 degrees in the vertical direction. Also, the VR image includes images having a wider angle of view than the angle of view that can be captured by a normal camera, or a video range wider than the display range that can be displayed at once on the display unit, even if it is less than 360 degrees in the horizontal direction and less than 360 degrees in the vertical direction. The image captured by the camera 100 using the above - described lens unit 200 is a type of VR image. The VR image can be VR - displayed, for example, by setting the display mode of a display device (a display device capable of displaying a VR image) to "VR view". By displaying a partial range of a VR image having a 360 - degree angle of view and changing the posture of the display device in the horizontal direction (horizontal rotation direction), the displayed range can be moved, and an omnidirectional video without seams in the horizontal direction can be viewed.

[0029] VR display (VR view) is a display method (display mode) with a variable display range that displays a video of a visual field range corresponding to the posture of a display device among VR images. VR display includes "monocular VR display (monocular VR view)" in which a VR image is mapped onto a virtual sphere (distortion correction) to display one image. Also, VR display includes "binocular VR display (binocular VR view)" in which a VR image for the left eye and a VR image for the right eye are each mapped onto a virtual sphere and arranged and displayed in the left and right regions. By performing "binocular VR display" using a VR image for the left eye and a VR image for the right eye with a parallax from each other, it is possible to view these VR images stereoscopically. In any VR display, for example, when a user wears a display device such as an HMD (head - mounted display), a video of a visual field range corresponding to the direction of the user's face is displayed. For example, among VR images, at a certain point in time, 0 degrees in the horizontal direction (specific Suppose that the video within the visual field range centered at 90 degrees in the vertical direction (90 degrees from the zenith, i.e., horizontal) is being displayed in a certain orientation, for example, north. If the posture of the display device is reversed from this state (for example, changing the display surface from south-facing to north-facing), among the same VR images, the display range is changed to the video within the visual field range centered at 180 degrees in the left-right direction (the opposite orientation, for example, south) and 90 degrees in the vertical direction. That is, when the user wears the HMD and turns the face from north to south (i.e., turns to the back), the video displayed on the HMD is also changed from the north video to the south video. Note that the VR image captured using the lens unit 200 is an image (180° image) that captures the range of approximately 180 degrees in the front, and there is no video in the range of approximately 180 degrees in the back. When such an image is VR-displayed and the posture of the display device is changed to the side where there is no video, a blank area is displayed.

[0030] By VR-displaying the VR image in this way, the user can visually obtain a feeling (sense of immersion) as if they were inside the VR image (within the VR space). Note that the method of displaying the VR image is not limited to the method of changing the posture of the display device. For example, the display range may be moved (scrolled) in response to user operations via a touch panel, direction buttons, etc. Also, during VR display (when in the display mode "VR view"), in addition to the change in the display range due to the posture change, the display range may be changed in response to a touch move on the touch panel, a drag operation with a mouse, etc., or the pressing of a direction button. Note that a smartphone mounted on a VR goggle (head-mounted adapter) is a type of HMD.

[0031] Referring to FIGS. 3(A) and 3(B), the captured image of the camera 100 with the lens unit 200 attached will be described. FIGS. 3(A) and 3(B) are schematic diagrams showing an example of the captured image displayed as a live view image (LV image) or a recorded image.

[0032] As shown in FIG. 3(A), the camera 100 with the lens unit 200 attached thereto acquires a captured image including a valid region where an image exists and an invalid region where no image exists. The image 310 is a captured image including two valid regions 301 captured through two optical systems respectively. Each of the valid regions 301 is a region of a circumferential fisheye image. There is a parallax between the two valid regions 301, and the imaging ranges differ by the amount of the parallax. There is an invalid region 302 outside the valid region 301.

[0033] Here, the lens unit 200 is designed to correspond to a specific value (assuming a specific value) with respect to the size of the image sensor and the recorded angle of view. For example, in the case of the lens unit 200 corresponding to DCI 8K, the corresponding angle of view is (8192×4320). On the other hand, the camera 100 can record at an angle of view other than the angle of view corresponding to the lens unit 200 according to the shooting settings. For example, when the recording size setting is UHD 8K, the recorded angle of view is (7680×4320). Depending on the attached lens unit and the recording size setting, shooting is performed at an angle of view using a part of the image sensor.

[0034] The image 310 in FIG. 3(A) is an example of an image recorded (shot) at an angle of view using the entire image sensor of the camera 100 having an image sensor sized to correspond to the lens unit 200. For example, the size of the image sensor corresponding to the lens unit 200 is DCI 8K (8192×4320) and the recording size is DCI 8K. In this case, as in the image 310, two circumferential fisheye images arranged side by side are completely recorded.

[0035] The image 320 in FIG. 3(B) is an example of an image shot at an angle of view using a part of the image sensor of the camera 100 having an image sensor sized to correspond to the lens unit 200. For example, the size of the image sensor corresponding to the lens unit 200 is DCI 8K (8192×4320) and the recording size is UHD 8K (7680×4320). In this case, two circumferential fisheye images arranged side by side are completely formed on the image sensor, but It is recorded in a state where a part is missing. As in the image 320, among the two circumferential fisheye images arranged side by side, for the effective area (left image area) 303, a part on the left side is missing, and for the effective area (right image area) 304, a part on the right side is missing and it is recorded. The live view image is also displayed in a state where a part is missing.

[0036] Referring to FIGS. 4(A) and 4(B), the lens information will be described. FIG. 4(A) is a schematic diagram showing an example of the lens information acquired by the acquisition unit 101 from the lens unit 200. The lens information includes information such as lens design values, lens individual values (manufacturing error values), and lens flags.

[0037] The lens design value is a design value for performing aberration correction. In the manufacturing process of the lens unit 200, errors such as decentration and inclination of the lens occur in each of the two optical systems (right-eye optical system 201 and left-eye optical system 202). If processes such as left-right interchange processing and positive-distance cylindrical conversion processing are performed without considering the errors, the quality of the binocular VR display will deteriorate and it will be difficult to obtain good stereoscopic vision. The lens individual value is a measurement result of the deviation (error) from the design value detected during the manufacturing process of the binocular lens. Details of the lens design value and the lens individual value will be described later with reference to FIG. 4(B).

[0038] The lens flag is a flag indicating that it is a binocular lens unit and can be used to determine whether a binocular lens unit has been used.

[0039] FIG. 4(B) is a schematic diagram showing details of the lens design value and the lens individual value. The lens design value and the lens individual value are used for left-right interchange processing and positive-distance cylindrical conversion processing. In the first embodiment, the lens design value and the lens individual value are also used for the determination process of whether the lens is for imaging an image including an effective area and an invalid area.

[0040] The lens design value includes information such as image circle position, image circle diameter, angle of view, and distortion correction coefficient.

[0041] The image circle position is the optical axis center coordinates in the image to be captured (the image including the right image area and the left image area), and is prepared for each of the two optical systems (the right optical system 201 and the left optical system 202) of the binocular lens unit. That is, the image circle position is the center coordinates of the image circle (circumferential fisheye image) formed on the imaging element, and is prepared for each of the right image and the left image. The origin of the coordinates is, for example, the center of the imaging element (the center of the image to be captured). The image circle position includes the horizontal coordinate and the vertical coordinate. Note that various information regarding the optical axis center in the image to be captured can be used as the image circle position. For example, the distance from a predetermined position (such as the center or the upper left corner) in the image to be captured to the optical axis center can be used.

[0042] The image circle diameter is the diameter of the image circle (circumferential fisheye image) formed on the imaging element. The angle of view is the angle of view of the image circle (circumferential fisheye image) formed on the imaging element. The distortion correction coefficient is the ratio of the designed image height to the ideal image height of the lens. The distortion correction coefficient may be set for each image height, and for an image height for which the distortion correction coefficient is not set, the distortion correction coefficient may be calculated by interpolation using a plurality of distortion correction coefficients. A polynomial approximating the relationship between the image height and the distortion correction coefficient may be set. The image circle diameter, the angle of view, and the distortion correction coefficient may or may not be common parameters for the two optical systems (the right optical system 201 and the left optical system 202) of the binocular lens unit.

[0043] The lens individual value does not include information such as image circle position deviation, optical axis tilt, and image magnification deviation.

[0044] The image circle position deviation is the deviation of the center coordinates of the image circle (circumferential fisheye image) formed on the imaging element from the design values. For example, the image circle position deviation includes a horizontal deviation and a vertical deviation. Taking the coordinates of the design values (two-dimensional coordinates including the horizontal coordinate and the vertical coordinate) as the origin, the horizontal deviation is indicated by the horizontal coordinate, and the vertical deviation is indicated by the vertical coordinate. The optical axis tilt is the deviation of the direction of the optical axis on the subject side from the design value. For example, the optical axis tilt includes a horizontal deviation and a vertical deviation. The deviation in each direction is indicated by an angle. The image magnification deviation is the deviation of the size of the image circle (circumferential fisheye image) formed on the imaging element from the design value. This deviation is indicated by, for example, a ratio to the design value. These lens individual values are measured and recorded for each of the two optical systems.

[0045] Referring to FIG. 5, the dither process will be described. FIG. 5 is a schematic diagram showing an example of the time dither process. The time dither process is a process of pseudo-increasing the gradation of an image by switching data according to the value of the bits to be truncated. Here, an example will be described in which the time dither process is performed on 10-bit captured image (RGB image data) to generate 8-bit image data.

[0046] In the 10-bit RGB image data, the R data at pixels 500 to 504 are "0110010000", "0110010001", "0110010010", "0110010011", and "0110010100" in binary notation, respectively.

[0047] When outputting 10-bit data in 8 bits, 2 bits are truncated. If the lower 2 bits of the data of pixels 500 to 503 are truncated and output in 8 bits, they all become "01100100". That is, pixels 500 to 503 are treated as the same pixel 510. Pixel 504 becomes "01100101" and takes a value 1 bit larger than pixel 510. That is, pixel 504 is treated as pixel 511. The values of the truncated bits are "00" for pixel 500 and pixel 504, "01" for pixel 501, "10" for pixel 502, and "11" for pixel 503.

[0048] Here, when time dithering is performed, each pixel outputs the following data. Pixel 500 outputs pixel 510-1-00 in the first frame, pixel 510-2-00 in the second frame, pixel 510-3-00 in the third frame, and pixel 510-4-00 in the fourth frame. The data of pixel 510-1-00, pixel 510-2-00, pixel 510-3-00, and pixel 510-4-00 is the same as that of pixel 510, which is "01100100". That is, pixel 500 outputs pixel 510 four times from the first frame to the fourth frame.

[0049] Pixel 501 outputs pixel 510-1-01 in the first frame, pixel 511-2-01 in the second frame, pixel 510-3-01 in the third frame, and pixel 510-4-01 in the fourth frame. The data of pixel 510-1-01, pixel 510-3-01, and pixel 510-4-01 is the same as that of pixel 510, which is "01100100". Also, the data of pixel 511-2-01 is the same value as pixel 511, which is "01100101". That is, pixel 501 outputs pixel 510 three times in 4 frames and outputs pixel 511 once in 4 frames.

[0050] Pixel 502 outputs pixel 511-1-10 in the first frame, pixel 510-2-10 in the second frame, pixel 510-3-10 in the third frame, and pixel 511-4-10 in the fourth frame. The data of pixel 510-2-10 and pixel 510-3-10 is the same "01100100" as that of pixel 510. Also, the data of pixel 511-1-10 and pixel 511-4-10 is "01100101" which is the same value as that of pixel 511. That is, pixel 502 outputs pixel 510 twice in 4 frames and outputs pixel 511 twice in 4 frames.

[0051] Pixel 503 outputs pixel 511-1-11 in the first frame, pixel 510-2-11 in the second frame, pixel 511-3-11 in the third frame, and pixel 511-4-11 in the fourth frame. The data of pixel 510-2-11 is the same "01100100" as that of pixel 510. Also, the data of pixel 511-1-11, pixel 511-3-11, and pixel 511-4-11 is "01100101" which is the same as that of pixel 511. That is, pixel 503 outputs pixel 510 once in 4 frames and outputs pixel 511 three times in 4 frames.

[0052] Pixel 504 outputs pixel 511 four times from the first frame to the fourth frame.

[0053] Here, when images of multiple types of luminance are quickly displayed, the luminance of the image recognized by a person is close to the average luminance of all the displayed images. Therefore, as shown in pixels 520 to 524, the original pixels 500 to 504 can be pseudo-expressed by time dithering processing. It is perceived with a smoother gradation than when time dithering processing is not performed, and the visual recognition of false contours is suppressed.

[0054] Although an example of performing temporal dithering on the G data among the RGB data has been described, temporal dithering is similarly performed on the R data and the B data. Note that the timing of outputting a pixel (pixel 511) having a value one bit larger than the original pixel (pixel 510) output in 8 bits is not limited to the above-described example. As long as the required number of times is satisfied within 4 frames, it may be output at any timing.

[0055] Here, as shown in FIGS. 3(A) and 3(B), although the invalid region 302 appears to be a black image at first glance, since pixel values are being acquired from the imaging sensor, it has slightly pixel values due to dark noise. The pixel values due to dark noise are, for example, R:4, G:4, B:12 in RGB 10-bit data. As the sensitivity of the imaging sensor increases, the pixel values due to dark noise increase. Also, when dithering is performed on pixels having pixel values due to dark noise, the noise is emphasized, making it easier to emit light or causing flicker. Therefore, it is desirable not to perform dithering on such pixels. Thus, in the first embodiment, it is determined whether or not to perform dithering on the pixels included in the invalid region.

[0056] FIG. 6 is a flowchart showing an example of information processing of the camera 100. This processing is realized by the system control unit 104 expanding and executing a program recorded in the non-volatile memory in the system memory. For example, when the user performs an operation to turn on the shooting mode, the processing of FIG. 6 starts.

[0057] In step S601, the system control unit 104 acquires information (lens information) regarding the optical system of the lens unit attached to the camera 100.

[0058] In step S602, the system control unit 104 determines whether the camera 100 corresponds to a binocular lens unit (for example, the lens unit 200). For example, the system control unit 104 determines whether the version of the firmware of the system control unit 104 is a version corresponding to the binocular lens unit. If it is determined that it corresponds to the binocular lens unit, the process proceeds to step S603; otherwise, the process proceeds to step S606. In the case of a binocular lens unit, different from the case of a normal single-lens unit, it is necessary to acquire and record lens information for post-processing. Therefore, the processes of steps S601 and S602 are required.

[0059] In step S603, the system control unit 104 determines whether the lens unit from which the captured image including the valid area and the invalid area is acquired is attached. Note that the valid area is the area where the video exists, and the invalid area is the area where the video does not exist. For example, the system control unit 104 determines whether the binocular lens unit is attached to the camera 100. If it is determined that the lens unit from which the captured image including the valid area and the invalid area is acquired is attached, the process proceeds to step S604; otherwise, the process proceeds to step S606.

[0060] In step S604, the system control unit 104 determines whether the set value of the brightness of the display unit 106 is greater than a predetermined value (threshold value). If it is determined that the set value of the brightness of the display unit 106 is greater than the predetermined value, the process proceeds to step S605; otherwise, the process proceeds to step S606. Note that when the set value of the brightness of the display unit 106 is equal to the predetermined value, the process may proceed to step S605 or step S606. The predetermined value in step S604 is, for example, a value such that if the set value of the brightness of the display unit 106 is smaller than that value, emission and flickering due to dark noise in the invalid area do not occur.

[0061] In step S605, the system control unit 104 transmits the lens information acquired in step S601 to the OSD generation unit 103. The OSD generation unit 103 generates area information based on the received lens information and stores it in the memory 102. The area information is information indicating whether the area containing each pixel is a valid area or an invalid area. The area information is indicated by 1 for a valid area and 0 for an invalid area for each coordinate.

[0062] In step S606, the system control unit 104 transmits processing information including the brightness setting value of the display unit 106 and information on various image processes to the image processing unit 105, and ends this process.

[0063] FIG. 7 is a flowchart showing an example of the image processing of the camera 100. When the system control unit 104 (image processing unit 105) acquires the processing information, the area information, and the captured image, the processing in FIG. 7 starts.

[0064] In step S701, the system control unit 104 (image processing unit 105) performs (executes) preprocessing on the captured image. The preprocessing here refers to image processing other than compression processing (processing for reducing the data amount). For example, the system control unit 104 performs OSD processing for superimposing an OSD image on a partial area of the captured image.

[0065] In step S702, the system control unit 104 reads out (acquires) the area information stored in the memory 102.

[0066] In step S703, the system control unit 104 determines whether the pixel currently held (processed) is a pixel included in the invalid area from the area information. If it is determined that the pixel currently held is a pixel included in the invalid area, the process proceeds to step S704; otherwise, the process proceeds to step S706.

[0067] In step S704, the system control unit 104 calculates the luminance value (nit) from the pixel values currently held. For example, when each of RGB is 10 bits and the display unit 106 has the characteristics of a gamma curve of 2.2, the luminance value of G can be calculated by the following calculation method. Luminance value of G = (maximum luminance of G) × ((pixel value of G) / 1023) 2.2

[0068] The luminance values of R and B are also calculated by this calculation method. In step S704, the system control unit 104 determines the value obtained by summing the luminance values of RGB as the estimated value of the display luminance.

[0069] In step S705, the system control unit 104 determines whether the estimated value of the display luminance calculated in step S704 is greater (higher) than a predetermined value (threshold value). If the estimated value of the display luminance is greater than the predetermined value, the process proceeds to step S707; otherwise, the process proceeds to step S706. Note that when the estimated value of the display luminance is equal to the predetermined value, the process may proceed to step S706 or step S707. The predetermined value in step S705 is, for example, a value such that if the estimated value of the display luminance is smaller than that value, no light emission or flicker occurs even when dithering is performed.

[0070] In step S706, the system control unit 104 performs compression processing. Also, the system control unit 104 performs dithering processing in the compression processing.

[0071] According to the processes of steps S705 and S706, the system control unit 104 determines whether to perform dithering processing on the pixels included in the invalid area based on the estimated value of the display luminance of the pixel before the compression process. When the estimated value of the display luminance is greater than a predetermined value, the system control unit 104 does not perform dithering processing. When the estimated value of the display luminance is less than a predetermined value, the system control unit 104 does not perform dithering processing. This can avoid emphasizing noise by dithering processing. Note that if it is not determined in step S604 of FIG. 6 that the brightness setting value of the display unit 106 is greater than a predetermined value, the system control unit 104 performs dithering processing on the pixels included in the invalid area. Also, for the pixels included in the valid area, dithering processing is performed regardless of the estimated value of the display luminance of the pixel.

[0072] In step S707, the system control unit 104 outputs (controls to display) the captured image after various image processes including the compression process to the display unit 106, and ends this process.

[0073] In the example of FIG. 7, the system control unit 104 performs image processes other than dithering processing (for example, OSD processing) in step S701 before performing dithering processing in step S706, but the order of performing the processes is not limited to this. For example, the system control unit 104 may perform OSD processing after performing dithering processing.

[0074] Although the case where the lens attached to the camera 100 is a binocular lens has been described, the attached lens is not limited to this. As long as it is a lens that acquires a captured image including a valid area and an invalid area, a lens other than a binocular lens may be used. In this case, the process of step S602 may not be performed.

[0075] The system control unit 104 determines whether to perform dithering processing on the pixels included in the invalid area in consideration of the estimated value of the display luminance of the pixel, and other conditions may be added.

[0076] For example, for pixels included in both the invalid area and the area where OSD processing is performed, the system control unit 104 may perform dithering regardless of the estimated display luminance value. This is because even for pixels where noise is prominent without dithering, the noise will not be visible if the OSD image is superimposed. For example, for pixels included in both the invalid area and the area where OSD processing is performed, the system control unit 104 may omit the calculation process of the estimated display luminance value (step S704) and perform dithering. Thereby, the processing load on the camera 100 can be reduced.

[0077] Also, for example, the system control unit 104 may determine whether to perform dithering in consideration of the number of pixels included in the invalid area. When the number of pixels included in the invalid area is larger than a predetermined number (for example, half of the total number of pixels in the entire image), the system control unit 104 may not perform dithering on the captured image regardless of the estimated display luminance value. For example, even if each estimated display luminance value is small, if the area occupied by the invalid area is large, there will be many areas where dark noise is emphasized by dithering, and there is a possibility of flicker. Therefore, when the number of pixels included in the invalid area is larger than the predetermined number, the occurrence of flicker can be suppressed by not performing dithering.

[0078] The system control unit 104 calculates the estimated display luminance value based on the gamma curve and the maximum luminance of the display unit 106 (display device), but the calculation method of the estimated value is not limited to this. For example, the system control unit 104 may calculate the estimated display luminance value based on information such as the temperature of the display unit 106 and the ambient brightness. Thereby, the influence of noise in the invalid area, such as the emphasis of noise in the invalid area or the occurrence of flicker, can be more effectively suppressed.

[0079] The system control unit 104 performs dithering in the time direction, but the type of dithering is not limited to this. For example, the system control unit 104 may perform dithering in the spatial direction or dithering using random numbers.

[0080] As described above, according to the first embodiment, for the pixels included in the invalid area, it is determined whether or not to perform dithering processing based on the estimated value of the display luminance of the pixel (the brightness setting value of the display unit 106). Thereby, an image with the influence of noise in the invalid area suppressed can be obtained.

[0081] <Second Embodiment> In the first embodiment, for the pixels included in the invalid area, it is determined whether or not to perform dithering processing (the presence or absence of dithering processing). However, when the pixel value is large due to the dark noise in the invalid area, or when the brightness of the display unit 106 is bright, depending on the presence or absence of dithering processing, the influence of the noise in the invalid area may not be suppressed. Therefore, in the second embodiment, for the pixels included in the invalid area, processing for replacing the pixel value with a predetermined value is performed. Note that the same configurations and flows as those in the first embodiment will be appropriately omitted from the description.

[0082] FIG. 8 is a block diagram showing an example of the configuration of an image display system (camera) 800 according to the second embodiment of the present invention.

[0083] In addition to the components (acquisition unit 101, memory 102, OSD generation unit 103, system control unit 104, image processing unit 105, and display unit 106) that the camera 100 has, the camera 800 has a recording medium I / F 801 and a communication unit 802.

[0084] The recording medium I / F 801 is an interface with a recording medium. The recording medium is, for example, an SD card or a CFexpress card, and stores metadata and image data including lens information. The recording medium I / F 801 transmits the lens information read from the recording medium to the system control unit 104. The recording medium I / F 801 outputs image data to the memory 102 based on a control signal from the system control unit 104. The recording medium I / F 801 records the captured image stored in the memory 102 on the recording medium based on a control signal from the system control unit 104.

[0085] The communication unit 802 transmits and receives video signals and audio signals to and from external devices connected by wireless or wired cables. The communication unit 802 can also be connected to a wireless LAN (Local Area Network) or the Internet. The communication unit 802 transmits the lens information acquired from the external device to the system control unit 104. The communication unit 802 outputs the image data acquired from the external device to the memory 102.

[0086] FIG. 9 is a flowchart showing an example of information processing of the camera 800. This processing is realized by the system control unit 104 expanding and executing a program recorded in the non-volatile memory in the system memory. For example, when the user performs an operation to display an image stored in the recording medium, the processing of FIG. 9 starts.

[0087] In step S901, the system control unit 104 acquires the metadata stored in the recording medium via the recording medium I / F 801. The system control unit 104 acquires the lens information from the metadata.

[0088] The processing of steps S902 to S906 is the same as the processing of steps S602 to S606 in FIG. 6.

[0089] FIG. 10 is a flowchart showing an example of image processing of the camera 800. When the system control unit 104 (image processing unit 105) acquires processing information (information including the brightness setting value of the display unit 106 and information on various image processes), region information, and the captured image, the processing of FIG. 10 starts. In the example of FIG. 10, the following processing is performed on the image acquired from the recording medium.

[0090] In step S1001, the system control unit 104 (image processing unit 105) determines whether the pixel currently held (processed) is a pixel included in the OSD region (region where OSD processing is performed). If it is determined that the currently held pixel is a pixel included in the OSD region, the process proceeds to step S1003; otherwise, the process proceeds to step S1002.

[0091] In step S1002, the system control unit 104 reads (acquires) the area information stored in the memory 102.

[0092] In step S1003, the system control unit 104 performs OSD processing to superimpose an OSD image on a partial area of the captured image.

[0093] In step S1004, the system control unit 104 determines whether the pixel currently held is a pixel included in the invalid area from the area information. If it is determined that the pixel currently held is a pixel included in the invalid area, the process proceeds to step S1005; otherwise, the process proceeds to step S1006.

[0094] In step S1005, the system control unit 104 performs replacement processing to replace the currently held pixel value with a predetermined value (for example, 0). That is, the system control unit 104 performs replacement processing on the pixels included in both the invalid area and the area where OSD processing is not performed.

[0095] In step S1006, the system control unit 104 performs image processing other than replacement processing and OSD processing (other image processing).

[0096] In step S1007, the system control unit 104 outputs the captured image after the image processing to the display unit 106 and ends this process.

[0097] In FIGS. 9 and 10, the system control unit 104 performs each process on the image acquired from the recording medium, but the image on which the process is performed is not limited to this. For example, the system control unit 104 may perform the same process on the image acquired from the communication unit 802 or the image acquired from the acquisition unit 101 (the image captured by the camera 800).

[0098] The system control unit 104 determined whether to perform replacement processing on the pixels included in the invalid area based on the brightness setting value of the display unit 106. Note that the system control unit 104 may determine whether to perform replacement processing based on the estimated value of the display brightness of the pixels included in the invalid area before the replacement processing is performed. When the estimated value of the display brightness is greater than a predetermined value, the system control unit 104 may perform replacement processing, and when the estimated value of the display brightness is less than the predetermined value, replacement processing may not be performed.

[0099] Also, the system control unit 104 may perform a process combining both the replacement process and the dither process. For example, the system control unit 104 may perform a replacement process of replacing the pixel value with a predetermined value (for example, 0) for the pixels included in both the invalid area and the area where the dither process is not performed. Thereby, an image with less noise can be obtained.

[0100] As described above, according to the second embodiment, for the pixels included in the invalid area, it is determined whether to perform replacement processing based on the estimated value of the display brightness of the pixel (the brightness setting value of the display unit 106). Thereby, an image with reduced noise in the invalid area can be obtained.

[0101] Also, although the embodiments of the present invention have been described in detail, the present invention is not limited to these specific embodiments, and various forms within the scope not departing from the gist of the present invention are also included in the present invention. Furthermore, each of the above-described embodiments merely shows one embodiment of the present invention, and it is also possible to appropriately combine the embodiments.

[0102] For example, although obtaining one image in which two image areas with parallax are arranged side by side has been described, the number of image areas, that is, the number of optical systems, may be more than two, and the arrangement of the plurality of image areas is not particularly limited.

[0103] Further, the present invention is applicable not only to cameras and PCs, but also to any electronic device that can handle an image having a plurality of image regions respectively corresponding to a plurality of optical systems. For example, the present invention is applicable to PDAs, mobile phone terminals, portable image viewers, printer devices, digital photo frames, music players, game machines, electronic book readers, cloud servers, and the like. Further, the present invention is applicable to video players, display devices (including projection devices), tablet terminals, smartphones, AI speakers, household appliances, in-vehicle devices, and the like. The present invention is also applicable to multi-eye smartphones having a plurality of optical systems of different types, such as standard lenses, wide-angle lenses, and zoom lenses.

[0104] <Other Embodiments> The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and causing one or more processors in a computer of the system or device to read and execute the program. Further, it can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0105] The disclosure of the present embodiment includes the following configurations, methods, programs, and media. (Configuration 1) An acquisition unit that acquires a captured image including a valid region where an image exists and an invalid region where no image exists; A processing unit that performs a compression process for reducing the data amount on the captured image; and In the compression process, the processing unit performs dither processing, and for pixels included in the invalid region, determines whether to perform the dither processing based on an estimated value of the display luminance of the pixel before the compression process is performed. An electronic device characterized by the above. (Configuration 2) The electronic device according to Configuration 1, further comprising a control unit that controls to display the captured image after the compression process on a display device. Characterized by the above. (Configuration 3) The processing means obtains the estimated value based on the gamma curve and the maximum luminance of the display device. The electronic device according to Configuration 2, characterized in that. (Configuration 4) The processing means further performs OSD processing for superimposing an OSD (On Screen Display) image on a partial area of the captured image. For pixels included in both the invalid area and the area where the OSD processing is performed, the dithering process is performed regardless of the estimated value. The electronic device according to Configuration 2 or 3, characterized in that. (Configuration 5) For pixels included in the invalid area, when the estimated value is greater than a predetermined value, the dithering process is not performed, and when the estimated value is less than the predetermined value, the dithering process is performed. The electronic device according to any one of Configurations 1 to 4, characterized in that. (Configuration 6) When the number of pixels included in the invalid area is greater than a predetermined number, the dithering process is not performed regardless of the estimated value. The electronic device according to any one of Configurations 1 to 5, characterized in that. (Configuration 7) For pixels included in both the invalid area and the area where the dithering process is not performed, a replacement process for replacing the pixel value with a predetermined value is performed. The electronic device according to any one of Configurations 1 to 6, characterized in that. (Configuration 8) The captured image is an image including a plurality of effective areas captured through a plurality of optical systems respectively. The electronic device according to any one of Configurations 1 to 7, characterized in that. (Configuration 9) Each of the plurality of optical systems includes a fish-eye lens. Each of the plurality of effective areas is an area of a circumferential fish-eye image. The electronic device according to Configuration 8, characterized in that. (Configuration 10) An acquisition unit that acquires a captured image including a valid region where an image exists and an invalid region where no image exists; A processing unit that performs a replacement process of replacing a pixel value of a pixel included in the invalid region with a predetermined value, and the processing unit determines whether to perform the replacement process based on an estimated value of the display luminance of a pixel included in the invalid region before the replacement process is performed. An electronic device characterized by the above. (Method 1) An acquisition step of acquiring a captured image including a valid region where an image exists and an invalid region where no image exists; A processing step of performing a compression process for reducing the data amount on the captured image, and wherein in the processing step, dithering processing is performed in the compression process, and for a pixel included in the invalid region, based on an estimated value of the display luminance of the pixel before the compression process is performed, it is determined whether to perform the dithering process. A control method for an electronic device characterized by the above. (Method 2) (Method 2) An acquisition step of acquiring a captured image including a valid region where an image exists and an invalid region where no image exists; A processing step of performing a replacement process of replacing a pixel value of a pixel included in the invalid region with a predetermined value, and wherein in the processing step, based on an estimated value of the display luminance of a pixel included in the invalid region before the replacement process is performed, it is determined whether to perform the replacement process. A control method for an electronic device characterized by the above. (Program) A program for causing a computer to function as each unit of the electronic device according to any one of Configurations 1 to 10. (Medium) A computer-readable storage medium storing a program for causing a computer to function as each unit of the electronic device according to any one of Configurations 1 to 10.

Description of Signs

[0106] 100: Camera 101: Acquisition unit 104: System control unit

Claims

1. An acquisition means for acquiring an imaging image including a valid area where an image exists and an invalid area where no image exists; A processing means for performing a compression process for reducing the data amount on the imaging image and having, In the compression process, the processing means performs dithering processing, and for pixels included in the invalid area, based on an estimated value of the display luminance of the pixel before the compression process is performed, determines whether to perform the dithering processing An electronic device characterized by this.

2. The electronic device according to claim 1, further comprising a control means for controlling to display the imaging image after the compression process on a display device characterized by this.

3. The processing means acquires the estimated value based on the gamma curve and the maximum luminance of the display device An electronic device characterized by this according to claim 2.

4. The processing means further performs OSD processing for superimposing an OSD (On Screen Display) image on a partial area of the imaging image, For pixels included in both the invalid area and the area where the OSD processing is performed, the processing means performs the dithering processing regardless of the estimated value An electronic device characterized by this according to claim 2.

5. For pixels included in the invalid area, when the estimated value is greater than a predetermined value, the processing means does not perform the dithering processing, and when the estimated value is less than the predetermined value, the processing means performs the dithering processing An electronic device characterized by this according to claim 1.

6. When the number of pixels included in the invalid area is greater than a predetermined number, the processing means does not perform the dithering processing regardless of the estimated value An electronic device characterized by this according to claim 1.

7. For pixels included in both the invalid area and the area where the dithering processing is not performed, the processing means performs a replacement process of replacing the pixel value with a predetermined value An electronic device characterized by this according to claim 1.

8. The imaging image is an image including a plurality of valid areas imaged through a plurality of optical systems respectively An electronic device characterized by this according to claim 1.

9. Each of the plurality of optical systems includes a fish-eye lens, Each of the plurality of valid areas is an area of a circumferential fish-eye image An electronic device characterized by this according to claim 8.

10. An acquisition means for acquiring an imaging image including a valid area where an image exists and an invalid area where no image exists; processing means for performing a replacement process of replacing the pixel value of a pixel included in the invalid area with a predetermined value; the processing means determines whether to perform the replacement process based on an estimated value of the display luminance of a pixel included in the invalid area before the replacement process is performed. An electronic device characterized by the above.

11. an acquisition step of acquiring a captured image including a valid area where an image exists and an invalid area where no image exists; a processing step of performing a compression process for reducing the data amount on the captured image; and in the processing step, dithering is performed in the compression process, and for a pixel included in the invalid area, it is determined whether to perform the dithering based on an estimated value of the display luminance of the pixel before the compression process is performed. A control method for an electronic device characterized by the above.

12. an acquisition step of acquiring a captured image including a valid area where an image exists and an invalid area where no image exists; a processing step of performing a replacement process of replacing the pixel value of a pixel included in the invalid area with a predetermined value; and in the processing step, it is determined whether to perform the replacement process based on an estimated value of the display luminance of a pixel included in the invalid area before the replacement process is performed. A control method for an electronic device characterized by the above.

13. A program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 10.

14. A computer-readable storage medium storing a program for causing a computer to function as each means of the electronic device according to any one of claims 1 to 10.

Images