Display processing device, display processing method, and program

Abstract

The present disclosure provides a display processing device, a display processing method, and a program capable of simultaneously confirming the focus state at a plurality of positions of a subject area. An enlarged image generation unit generates a plurality of enlarged images obtained by enlarging a plurality of enlargement positions including at least one of a focus position and a peripheral position away from the focus position in an input image acquired from an image sensor or a preview image for confirming the input image, and a display control unit controls a display unit so as to display the plurality of enlarged images simultaneously. The present disclosure can be applied to an imaging device having a focus assistance function.

Description

Display processing device, display processing method, and program

[0001] The present disclosure relates to a display processing device, a display processing method, and a program, and more particularly to a display processing device, a display processing method, and a program that enable the focus state of a subject area to be checked simultaneously at multiple positions.

[0002] Many techniques have been proposed for checking the focus of an imaging device. For example, Patent Literature 1 discloses a technique for detecting the most in-focus area of ​​a captured image as a display center position based on a focus score indicating the degree of focus of each area of ​​the captured image, and then cropping and enlarging an area of ​​a predetermined size including the display center position to display it.

[0003] Japanese Patent Application Laid-Open No. 2015-105996

[0004] In recent years, multi-viewpoint imaging, in which a subject is photographed from multiple viewpoints, has been used to create 3D content. When performing multi-viewpoint imaging, it is required that not only the focus position but also peripheral positions in the captured image from each viewpoint are simultaneously in focus.

[0005] The present disclosure has been made in consideration of such circumstances, and makes it possible to simultaneously check the focus state at multiple positions in a subject area.

[0006] The display processing device of the present disclosure is a display processing device that includes an enlarged image generation unit that generates multiple enlarged images by enlarging multiple enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image obtained from an image sensor or a preview image for confirming the input image, and a display control unit that controls a display unit to display the multiple enlarged images simultaneously.

[0007] The display processing method disclosed herein is a display processing device that includes generating multiple enlarged images by enlarging multiple enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image obtained from an image sensor or a preview image for confirming the input image, and controlling a display unit to display the multiple enlarged images simultaneously.

[0008] The program disclosed herein is a program that causes a processor to execute a process of generating multiple enlarged images by enlarging multiple enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image obtained from an image sensor or a preview image for confirming the input image, and controlling a display unit to display the multiple enlarged images simultaneously.

[0009] In the present disclosure, a plurality of enlarged images are generated by enlarging a plurality of enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image obtained from an image sensor or a preview image for confirming the input image, and a display unit is controlled to simultaneously display the plurality of enlarged images.

[0010] 10 is a diagram illustrating the relationship between F-number and blur. FIG. 10 is a diagram illustrating focus error in deep focus shooting. FIG. 10 is a block diagram illustrating an example functional configuration of a display processing device according to the present disclosure. FIG. 10 is a flowchart illustrating the operation of a display processing device. FIG. 10 is a flowchart illustrating details of a multiple region enlargement preparation process. FIG. 10 is a diagram illustrating an example of an enlargement position confirmation image. FIG. 10 is a flowchart illustrating details of a multiple region enlargement display process. FIG. 10 is a diagram illustrating an example of the positional relationship of enlargement positions. FIG. 10 is a diagram illustrating an example of arrangement of enlarged images according to the positional relationship of enlargement positions. FIG. 10 is a diagram illustrating an example of a depth image. FIG. 10 is a diagram illustrating another example of arrangement of enlarged images according to the positional relationship of enlargement positions. FIG. 10 is a block diagram illustrating another example functional configuration of a display processing device. FIG. 10 is a flowchart illustrating automatic enlargement position setting processing. FIG. 10 is a diagram illustrating an example of a normalized cumulative frequency distribution. FIG. 10 is a diagram illustrating an example of setting of an enlargement position. FIG. 10 is a block diagram illustrating an example configuration of a free viewpoint video presentation system. FIG. 10 is a block diagram illustrating an example configuration of a computer.

[0011] Modes for carrying out the present disclosure (hereinafter referred to as embodiments) will be described below in the following order.

[0012] 1. Background art and problems therein 2. Configuration and operation of a display processing device according to the present disclosure 3. Details of preparation processing for enlarging multiple regions 4. Details of processing for enlarging multiple regions display 5. Automatic setting of enlargement position using depth information 6. Configuration of a free viewpoint video presentation system 7. Example configuration of a computer

[0013] 1. Background Art and Issues There has been a surge in technologies for capturing three-dimensional space and creating free-viewpoint video, such as photogrammetry, Neural Radiance Fields (NeRF), and 3D Gaussian Splatting. These technologies share the commonality of using images of a subject captured from multiple viewpoints, i.e., multi-viewpoint images, as input. In this disclosure, capturing multi-viewpoint images is sometimes referred to as multi-viewpoint imaging. Consistency between the input multi-viewpoint images is important for generating high-quality three-dimensional models. Therefore, it is common to manually fix white balance and exposure (F-number (aperture), exposure time, ISO sensitivity), etc., while performing pan-focus imaging with the aperture narrowed as much as possible. That is, because the focus range is narrowed when capturing images at a small F-number, as shown on the left side of Figure 1, it is necessary to expand the focus range by performing pan-focus imaging with a large F-number, as shown on the right side of Figure 1.

[0014] However, there are situations where this shooting technique does not guarantee stable shooting. Because the depth of field is deep when shooting with a narrow aperture, even if the focus position is slightly shifted forward or backward, the appearance of the subject in the on-focus area remains unchanged, making focus adjustment difficult. Specifically, as shown in Figure 2, in deep-focus shooting, even if the main subject is in focus without any focus shift, the user will not notice the shift in focus because the main subject is within the acceptable focus range, even if there is a slight shift in focus. As a result, the blur in front of and behind the main subject increases, resulting in variations in the degree of blur from image to image. This is exacerbated by factors such as significant changes in composition and changes in the direction of the lens optical axis when shooting from multiple viewpoints.

[0015] For shooting with more time, manual focus adjustment is also possible. Focus support functions for manual focus include a function to highlight the on-focus area and a function to zoom in on a specified area. However, it was difficult to obtain information not only about the on-focus area but also about the entire subject area, especially information about areas in front of and behind the focus position that are more susceptible to blurring. Furthermore, with the zoom in on a specified area function, you can move the specified area within the subject area by pressing a button after zooming in, but this movement is time-consuming and only one area can be viewed at a time. This makes it difficult to grasp the degree of blurring of the entire subject area while adjusting the focus.

[0016] In contrast, the technology disclosed herein provides a focus assist function by simultaneously displaying enlarged images of multiple areas within the subject area, allowing the focus state of multiple positions in the subject area to be checked simultaneously.

[0017] 2. Configuration and Operation of a Display Processing Device According to the Present Disclosure (Functional Configuration of a Display Processing Device) FIG. 3 is a block diagram showing an example of the functional configuration of a display processing device according to the present disclosure.

[0018] The display processing device 1 shown in FIG. 3 is realized by a processor having a function of displaying an input image acquired from an image sensor provided in an imaging device such as a single-lens reflex camera or a smartphone as a monitoring image on a display unit provided in the imaging device. The image displayed on the display unit may be generally referred to as a display image, regardless of the type of image. The input image is an RGB image obtained by processing the output of the image sensor and may be considered a so-called full-resolution image. The monitoring image is a reduced image for display, reduced in size according to the display resolution of the display unit, for checking the input image. In the present disclosure, the monitoring image is also referred to as a preview image, as appropriate. The display processing device 1 is configured to include the following functional blocks: a monitoring image generation unit 10, an enlargement position setting unit 11, an instruction image superimposition unit 12, an enlarged image generation unit 13, an enlarged image integration unit 14, and a display control unit 15.

[0019] The monitoring image generating unit 10 acquires an input image from the image sensor and performs a reduction process according to the display resolution of the display unit to generate a monitoring image (preview image) for checking the input image. The monitoring image generated by the monitoring image generating unit 10 is supplied to the instruction image superimposing unit 12 and the enlarged image generating unit 13.

[0020] The enlargement position setting unit 11 sets the number of enlargement positions indicated by the enlargement setting information at positions indicated by the designation information in the monitoring image (subject area). In the present disclosure, the enlargement positions include at least one of a default focus position in the monitoring image or the input image and a peripheral position away from the focus position. The designation information represents a position designated by the user in the monitoring image using coordinate information. The enlargement position designated by the user in the monitoring image may also be set to a corresponding position in the input image. The enlargement setting information is information representing a predetermined number of enlarged images to be displayed (number of displays). The number of enlarged images represented by the enlargement setting information is usually two or more to confirm the degree of blurring of the entire subject area, and each enlarged image corresponds to a different position. The designated enlargement positions may be multiple peripheral positions away from the focus position, or may be a combination of the focus position and peripheral positions. The enlargement position information representing the enlargement positions set by the enlargement position setting unit 11 is supplied to the designated image superimposition unit 12, the enlarged image generation unit 13, and the enlarged image integration unit 14.

[0021] The instruction image superimposing unit 12 superimposes an instruction image indicating the enlargement position at the enlargement position on the monitoring image based on the enlargement position information from the enlargement position setting unit 11. The monitoring image on which the instruction image is superimposed is supplied to the display control unit 15 as an enlargement position confirmation image.

[0022] The enlarged image generating unit 13 generates enlarged images by enlarging the multiple enlargement positions set in the monitoring image based on the enlargement position information from the enlargement position setting unit 11. The multiple enlarged images generated by the enlarged image generating unit 13 are supplied to the enlarged image integrating unit 14.

[0023] The enlarged image integrating unit 14 generates an integrated image by integrating multiple enlarged images from the enlarged image generating unit 13 in accordance with the number indicated in the display setting information. The enlarged image integrating unit 14 also functions as an arrangement determining unit that determines the arrangement in which the multiple enlarged images from the enlarged image generating unit 13 are displayed in accordance with the positional relationship of each enlargement position, based on the enlargement position information from the enlargement position setting unit 11. That is, the enlarged image integrating unit 14 generates an integrated image in which the multiple enlarged images are integrated in an arrangement according to the positional relationship of each enlargement position. The integrated image generated by the enlarged image integrating unit 14 is supplied to the display control unit 15.

[0024] The display control unit 15 controls the display of the display image on a display unit (not shown). Specifically, the display control unit 15 controls the display unit to display the enlargement position confirmation image from the instruction image superimposition unit 12 as the display image. The display control unit 15 also controls the display unit to display the integrated image from the enlarged image integration unit 14, thereby causing the display unit to simultaneously display multiple enlarged images generated by the enlarged image generation unit 13. The integrated image from the enlarged image integration unit 14 is displayed as a focus confirmation image.

[0025] (Operation of display processing device) When a user operates the imaging device, the operating mode of the display processing device 1 transitions from a normal mode, in which the input image captured in the imaging device is displayed as a monitoring image that has been reduced in size according to the display resolution of the display unit, to a multiple area enlargement mode, in which enlarged images of multiple areas within the subject area are displayed.

[0026] FIG. 4 is a flowchart illustrating the operation of the display processing device 1 in the multi-region enlargement mode.

[0027] When the operation mode transitions to the multiple-area enlargement mode, the display processing device 1 executes a multiple-area enlargement preparation process in step S1. In the multiple-area enlargement preparation process, the position (enlargement position) of the area to be displayed as an enlarged image is set in the input image captured by the imaging device, i.e., the input image acquired from the image sensor. The multiple-area enlargement preparation process will be described in detail later.

[0028] When the enlargement positions are set by the multiple region enlargement preparation process, in step S2, the display processing device 1 determines whether an enlargement start signal for starting enlargement of the enlargement position in the input image has been input. The enlargement start signal may be input in response to, for example, a user's button operation or rotation of a focus ring on the imaging device, or voice input instructing the start of enlargement.

[0029] Until an enlargement start signal is input, the display processing device 1 will be in standby mode in normal mode, displaying a monitoring image generated from input images input from the imaging device at any time, and if it is determined that an enlargement start signal has been input, it proceeds to step S3.

[0030] When the enlargement start signal is input, in step S3, the display processing device 1 executes a multi-area enlarged display process. In the multi-area enlarged display process, an enlarged image obtained by enlarging the enlargement position set in the monitoring image is displayed as a focus confirmation image. The multi-area enlarged display process will be described in detail later.

[0031] After the enlarged image is displayed by the multiple-area enlargement display process, in step S4, the display processing device 1 determines whether an enlargement end signal for ending the multiple-area enlargement mode has been input. The enlargement end signal may be input in response to, for example, a user's button operation on the imaging device or voice input instructing the user to end the enlargement. Alternatively, the enlargement end signal may be input in response to the elapse of a certain time period after the start or end of the user's rotation of the focus ring to adjust the focus while checking the enlarged image.

[0032] The multiple region enlarged display process in step S3 is repeated until an enlargement end signal is input, and when it is determined that an enlargement end signal has been input, the multiple region enlargement mode is terminated.

[0033] 3. Details of the Multiple Region Expansion Preparation Process> The multiple region expansion preparation process executed in step S1 of FIG. 4 will be described in detail with reference to the flowchart of FIG.

[0034] In step S11, the enlargement position setting unit 11 sets an enlargement position in the monitoring image. Specifically, the enlargement position setting unit 11 sets an enlargement position in the monitoring image based on enlargement setting information, which is a preset parameter, and designation information input by the user. The designation information for designating the enlargement position may be input by touching a touch panel, operating a cursor using a button, selecting a language, or by voice input for designating the enlargement position. The range designated by the designation information may be a point or rectangular area on the monitoring image, or may be defined in object units using semantic segmentation or the like. The designation information is set in areas of the input image or in object units within the input image, and may be referenced again when the same object or area is recognized at a different angle of view.

[0035] In step S12, the instruction image superimposing unit 12 superimposes an instruction image at the enlargement position of the monitoring image set by the enlargement position setting unit 11. The instruction image is sequentially superimposed on the monitoring image.

[0036] Then, in step S13, the display control unit 15 displays the monitoring image on which the instruction image has been superimposed by the instruction image superimposing unit 12 as an enlarged position confirmation image.

[0037] 6A, 6B, and 6C are diagrams showing examples of enlargement position check images, in which an enlargement position check image is superimposed on a monitoring image including four objects sb1, sb2, sb3, and sb4, with the position (area) of object sb4 being the enlargement position.

[0038] That is, Fig. 6A shows an enlarged position check image in which a dot (cross)-shaped instruction image EP1 is superimposed on the position of object sb4 in the monitoring image. Fig. 6B shows an enlarged position check image in which a rectangular instruction image EP2 is superimposed on the position of object sb4 in the monitoring image. Fig. 6C shows an enlarged position check image in which an instruction image EP3 is superimposed on the entire area of ​​object sb4 in the monitoring image.

[0039] In this way, while the enlargement position confirmation image in which the instruction image is superimposed on the monitoring image is displayed, the display processing device 1 enters a standby state for input of an enlargement start signal.

[0040] 4. Details of the Multiple Region Enlarged Display Processing> Details of the multiple region enlarged display processing executed in step S3 of FIG. 4 will be described with reference to the flowchart of FIG.

[0041] In step S31, the enlarged image generation unit 13 generates an enlarged image for each enlargement position set in the monitoring image through the multiple-area enlargement preparation process. The monitoring image is generally generated by thinning out and reading from an image sensor. The enlarged image generation unit 13 generates the enlarged image by cutting out a rectangular area at the enlargement position from an input image, which is a high-resolution RGB image. The enlarged image generation unit 13 can also generate the enlarged image by reading pixels corresponding to the rectangular area at the enlargement position from an image sensor included in the imaging device. More specifically, the enlarged image generation unit 13 generates the enlarged image by reading all pixels within the rectangular area. The enlarged image generation unit 13 may also generate the enlarged image by enlarging or reducing the image of the area to be enlarged acquired from the image sensor through signal processing.

[0042] In step S32, the enlarged image integration unit 14 integrates the multiple enlarged images generated by the enlarged image generation unit 13. In integrating the enlarged images, the multiple enlarged images may be aggregated into a single image and integrated, or the multiple enlarged images may be divided into two or more images and integrated.

[0043] Then, in step S33, the display control unit 15 displays the integrated image obtained by integrating the multiple enlarged images by the enlarged image integration unit 14 as a focus-confirmation image.

[0044] When the magnification ratio of each enlarged image in the integrated image is large, it may be difficult to identify the position of each enlarged image in the subject area. Therefore, random arrangement of the enlarged images in the integrated image may cause confusion during focus adjustment. In response to this, for example, the integrated image may be generated by setting the frame color of each enlarged image in the same color as the indication image indicating the enlargement position in the enlargement position confirmation image. Furthermore, the layout of the enlarged images in the integrated image may be determined according to the positional relationship of the enlargement positions of each enlarged image.

[0045] FIG. 8 is a diagram showing an example of the positional relationship of the enlargement positions.

[0046] In the example of Figure 8, three enlargement positions are set in the monitoring image IMG by an instruction image EP10 superimposed on the upper right of object sb3, an instruction image EP11 superimposed on the lower left of object sb1, and an instruction image EP12 superimposed on the top edge of object sb4. Here, object sb3 is in the on-focus area, and object sb3 is in focus in the subject area of ​​the monitoring image IMG. In other words, objects sb1, sb2, and sb4 are out of focus. In the present disclosure, the position in focus in the monitoring image (preview image) is appropriately referred to as the focus position of the monitoring image. The focus position of the monitoring image can also be considered as the focus position of the input image corresponding to the monitoring image.

[0047] In this case, the arrangement of the enlarged image in the integrated image is determined, for example, according to the coordinate information of each enlargement position on the two-dimensional coordinates (XY coordinates) of the monitoring image IMG. In the present disclosure, the arrangement of the enlarged image in the integrated image can also be determined according to the coordinate information of each enlargement position on the two-dimensional coordinates of the input image.

[0048] Specifically, when the arrangement of enlarged images is determined based on the X-axis coordinate of the monitoring image IMG, an integrated image ITG1 such as that shown in Figure 9A is displayed. In the integrated image ITG1, an enlarged image EM11 corresponding to the instruction image EP11, an enlarged image EM12 corresponding to the instruction image EP12, and an enlarged image EM10 corresponding to the instruction image EP10 are arranged in this order from left to right. That is, in the integrated image ITG1, the enlarged images EM11, EM12, and EM10 are arranged in order along the X-axis coordinate of the monitoring image IMG. Note that the enlarged image EM10 corresponding to the instruction image EP10 is a focus area image obtained by enlarging the focus position (object sb3) in the monitoring image IMG or the input image. In contrast, the instruction images EP11 and EP12 are peripheral area images obtained by enlarging peripheral positions away from the focus position.

[0049] Furthermore, when the arrangement of the enlarged images is determined based on the Y-axis coordinate of the monitoring image IMG, an integrated image ITG2 such as that shown in Figure 9B is displayed. In the integrated image ITG2, an enlarged image EM12 corresponding to the instruction image EP12, an enlarged image EM10 corresponding to the instruction image EP10, and an enlarged image EM11 corresponding to the instruction image EP11 are arranged in this order from left to right. That is, in the integrated image ITG2, the enlarged images EM12, EM10, and EM11 are arranged in the order along the Y-axis coordinate of the monitoring image IMG.

[0050] 10 is a diagram showing a depth image DPT corresponding to an input image. The depth image DPT can be obtained by providing an imaging device with an image sensor having image plane phase difference pixels. By using such a depth image DPT, the arrangement of the enlarged image in the integrated image may be determined according to the depth information of each of the enlargement positions (pointer images EP10, EP11, EP12) in the depth image DPT corresponding to the input image.

[0051] That is, when the arrangement of the enlarged images is determined based on the Z-axis coordinate of the depth image DPT (depth of the input image), an integrated image ITG3 such as that shown in Figure 9C is displayed. In the integrated image ITG3, an enlarged image EM11 corresponding to the instruction image EP11, an enlarged image EM10 corresponding to the instruction image EP10, and an enlarged image EM12 corresponding to the instruction image EP12 are arranged in that order from left to right. That is, in the integrated image ITG3, the enlarged images EM11, EM10, and EM11 are arranged in order along the depth of the input image.

[0052] Furthermore, by using the depth image DPT, the arrangement of the enlarged images in the integrated image may be determined based on the absolute value of the distance from the on-focus area. As shown on the left side of FIG. 11 , when the enlargement position indicated by the instruction image EP10 is in the on-focus area, the enlargement position indicated by the instruction image EP12 is closer to the on-focus area in the Z-axis direction (depth direction) than the enlargement position indicated by the instruction image EP11. In this case, an integrated image ITG4 such as that shown on the right side of FIG. 11 is displayed. In the integrated image ITG4, the enlarged image EM10 corresponding to the instruction image EP10, the enlarged image EM12 corresponding to the instruction image EP12, and the enlarged image EM11 corresponding to the instruction image EP11 are arranged in order from left to right. That is, in the integrated image ITG4, the enlarged images EM10, EM12, and EM11 are arranged in order of proximity from the on-focus area.

[0053] In the above example, an image is divided left and right (horizontally) into regions in which enlarged images are arranged and displayed simultaneously, but enlarged images may also be arranged and displayed simultaneously in regions divided up and down (vertically) or in regions divided in an m x n two-dimensional array. When an integrated image is displayed as two or more images, multiple tabs representing two or more integrated images obtained by integrating multiple enlarged images may be displayed, or two or more integrated images obtained by integrating multiple enlarged images may be displayed on multiple display units, respectively.

[0054] In the above example, the display unit displays only the integrated image as a focus confirmation image, but the integrated image may be displayed so that it can be compared with the monitoring image. For example, the display control unit 15 may control the display unit to switch between displaying the monitoring image and the integrated image (multiple enlarged images) based on a user's operation input. Furthermore, the display control unit 15 may control the display unit to simultaneously display the monitoring image and the integrated image (multiple enlarged images).

[0055] According to the above process, enlarged images of multiple areas within the subject area are simultaneously displayed as focus confirmation images, allowing the user to simultaneously check the focus state at multiple positions within the subject area. Furthermore, in the integrated image, the layout of the enlarged images is determined according to the positional relationship of each enlarged position, making the correspondence between the enlarged images clear and enabling efficient focus adjustment.

[0056] In the above description, a position specified by a user in a monitoring image is set as the enlargement position. However, the enlargement position can also be set automatically by the imaging device. For example, the enlargement position may be automatically set based on the results of continuous autofocus (AF), which is capable of tracking and focusing on a specific subject (object or area). In this case, the enlargement position in the input image is automatically set using the cursor position indicating the on-focus area and the results of object detection such as face detection, pupil detection, and moving object detection. Furthermore, in this case, the number of enlargement positions may also be set automatically. This makes it possible to realize a function such as simultaneous enlargement of face areas according to the number of people in the subject area.

[0057] The enlargement position setting unit 11 may also track an object or region for which an enlargement position has been set at a specific viewpoint, thereby automatically setting an enlargement position for the object or region at the current viewpoint. By three-dimensionally tracking an enlargement position corresponding to an object or region in real space in this manner as the viewpoint moves, multi-viewpoint imaging with a moving camera can be efficiently performed. If an object or region for which an enlargement position has been set as the viewpoint moves is occluded by another object, the display of an enlarged image of the occluded object or region may be omitted at the viewpoint after the movement. That is, the number of enlargement positions / enlarged images set at a specific viewpoint may be reduced at the viewpoint after the movement. To indicate that the occluded object or region is not displayed on the monitoring image, the display of the enlarged image may be omitted, and a corresponding indication image may be superimposed on the occluded position. In the present disclosure, the specific viewpoint for which an enlargement position has been set is referred to as a first viewpoint, and the viewpoint after the movement is referred to as a second viewpoint. Furthermore, input images acquired at the first viewpoint and the second viewpoint are referred to as a first input image and a second input image, respectively.

[0058] Furthermore, the enlargement position may be automatically set based on the depth information in the depth image DPT described above.

[0059] 5. Automatic Setting of Enlargement Position Using Depth Information FIG. 12 is a block diagram showing another example of the functional configuration of the display processing device 1 according to the present disclosure.

[0060] In the display processing device 1 of Fig. 12, components having the same functions as those of the functional blocks of the display processing device 1 of Fig. 3 are denoted by the same reference numerals, and descriptions thereof will be omitted. That is, the display processing device 1 of Fig. 12 differs from the display processing device 1 of Fig. 3 in that an enlargement position setting unit 111 is provided instead of the enlargement position setting unit 11.

[0061] That is, the enlargement position setting unit 111 sets an enlargement position corresponding to the depth indicated by the enlargement setting information based on the depth information of the input image displayed as the monitoring image. The depth information is set as depth information constituting a depth image corresponding to the input image. Note that the depth information in the monitoring image may be considered to correspond to the depth information constituting the depth image corresponding to the input image. The enlargement setting information is set as a plurality of depth information that are set in advance as enlargement positions (targets). Note that, in the following, the cumulative frequency of normalized depth information is set as the target depth information, but any information indicating the degree of depth in the entire subject area may be used.

[0062] The automatic enlargement position setting process executed by the enlargement position setting unit 111 will be described with reference to the flowchart of FIG.

[0063] In step S111, the enlargement position setting unit 111 calculates a cumulative frequency distribution of normalized depth information based on the depth information corresponding to the input image that is input as depth information.

[0064] In step S112, the enlargement position setting unit 111 calculates the depth range to be enlarged in accordance with the target cumulative frequency, which is set by the enlargement setting information.

[0065] For example, suppose that a cumulative frequency distribution of normalized depth information is calculated as shown in Fig. 14. When the target cumulative frequencies are set to 0.1, 0.5, and 0.9 according to the expansion setting information, the depth ranges to be expanded are the depth information included in the ranges d1, d2, and d3, respectively.

[0066] In step S113, the enlargement position setting unit 111 generates an edge image based on the monitoring image. To generate the edge image, a general edge detection method using, for example, a Sobel filter is used. The edge image may be generated based on the input image instead of the monitoring image.

[0067] Then, in step S114, the enlargement position setting unit 111 sets the area with the largest edge among the areas in each depth range corresponding to each target cumulative frequency as the enlargement position based on the edge image. That is, the enlargement position setting unit 111 automatically sets the enlargement position in the monitoring image to a position corresponding to the area with the largest edge among the areas corresponding to each of the multiple pieces of depth information (target cumulative frequencies) set in the depth image corresponding to the input image.

[0068] 15, if there are three regions r1, r2, and r3 in the depth range corresponding to a cumulative frequency of 0.5, the position corresponding to the region r1 with the largest edge is set as the enlargement position corresponding to the cumulative frequency of 0.5. Similarly, for the depth range corresponding to a cumulative frequency of 0.1 and the depth range corresponding to a cumulative frequency of 0.9, the position corresponding to the region with the largest edge is set as the enlargement position.

[0069] According to the above process, the enlargement position can be automatically set in a region of the input image corresponding to the preset depth information in accordance with the depth information (cumulative frequency) set as the enlargement setting information. This eliminates the need for the user to specify the enlargement position, enabling multi-viewpoint image capture for creating a free viewpoint video to be completed in a short time. Furthermore, by selecting and enlarging a position with a large edge among subjects at the same depth, the user can easily determine the focus situation. Note that the number of target depth information (cumulative frequency) may be automatically set according to the distribution of depth information in the depth image and the number of subjects.

[0070] 6. Configuration of Free Viewpoint Video Presentation System FIG. 16 is a block diagram showing an example of the configuration of a free viewpoint video presentation system.

[0071] The free viewpoint video presentation system 200 shown in Figure 16 creates a free viewpoint video using multiple viewpoint images captured while the user checks the focus state of the entire subject area, using multiple enlarged images displayed by the display processing device 1 of the present disclosure, and presents the video to the user.

[0072] The free viewpoint video presentation system 200 receives input of RGB images as multi-viewpoint images captured by a single-lens reflex camera or a smartphone, and may also receive input of depth images obtained by a depth sensor, a stereo camera, LiDAR (Light Detection and Ranging), etc.

[0073] The free viewpoint video presentation system 200 is composed of an information processing unit 210, a storage unit 220, a display unit 230, and a physics calculation unit 240. The information processing unit 210 uses the storage area of ​​the storage unit 220 to perform preprocessing to estimate camera parameters of RGB images and to learn luminance fields using multiple RGB images. The image from the arbitrary viewpoint as the learning result is displayed on the display unit 230, which may be a spatial reproduction display or a head-mounted display. The image from the arbitrary viewpoint as the learning result is also used for graphic rendering in the physics calculation unit 240, which may be implemented as a game engine or the like.

[0074] According to the technology disclosed herein, blur-free multi-viewpoint images can be obtained, and therefore the free viewpoint video presentation system 200 can maintain consistency between the input multi-viewpoint images, making it possible to generate high-quality three-dimensional models.

[0075] 7. Example Computer Configuration The above-described series of processes can be executed by hardware or software. When the series of processes is executed by software, a program constituting the software is installed on a computer. Here, the computer includes a computer built into dedicated hardware, and a general-purpose personal computer, for example, that can execute various functions by installing various programs.

[0076] FIG. 17 is a block diagram showing an example of the hardware configuration of a computer that executes the above-described series of processes by a program.

[0077] In the computer 300 , a CPU (Central Processing Unit) 301 , a ROM (Read Only Memory) 302 , and a RAM (Random Access Memory) 303 are interconnected by a bus 304 .

[0078] An input / output interface 305 is also connected to the bus 304. An input unit 306, an output unit 307, a storage unit 308, a communication unit 309, and a drive 310 are connected to the input / output interface 305.

[0079] The input unit 306 includes buttons, a touch panel, etc. The output unit 307 includes a display, a speaker, etc. The storage unit 308 includes a hard disk, a non-volatile memory, etc. The communication unit 309 includes a network interface, etc. The drive 310 drives removable media 311 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory.

[0080] In the computer 300 configured as described above, the CPU 301 performs the above-described series of processes by, for example, loading a program stored in the storage unit 308 into the RAM 303 via the input / output interface 305 and the bus 304 and executing the program.

[0081] The program executed by the computer 300 (CPU 301) can be provided by being recorded on a removable medium 311 such as a package medium, for example. The program can also be provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

[0082] In a computer, the program can be installed in the storage unit 308 via the input / output interface 305 by inserting the removable medium 311 into the drive 310. The program can also be received by the communication unit 309 via a wired or wireless transmission medium and installed in the storage unit 308. Alternatively, the program can be installed in the ROM 302 or the storage unit 308 in advance.

[0083] The program executed by the computer may be a program that processes in chronological order according to the order described in this specification, or may be a program that processes in parallel or at the required timing, such as when called.

[0084] The embodiments of the present disclosure are not limited to the above-described embodiments, and various modifications are possible within the scope of the gist of the present disclosure.

[0085] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0086] Furthermore, the present disclosure may have the following configurations. (1) A display processing device comprising: an enlarged image generation unit that generates a plurality of enlarged images by enlarging a plurality of enlargement positions, including at least one of a focus position and a peripheral position distant from the focus position, in an input image acquired from an image sensor or a preview image for confirming the input image; and a display control unit that controls a display unit to simultaneously display the plurality of enlarged images. (2) The display processing device according to (1), further comprising: an arrangement determination unit that determines an arrangement in which the plurality of enlarged images are displayed in the preview image according to a positional relationship of the plurality of enlargement positions. (3) The display processing device according to (2), wherein the positional relationship includes coordinate information for each of the plurality of enlargement positions on a two-dimensional coordinate system of the input image or the preview image. (4) The display processing device according to (2), wherein the positional relationship includes depth information for each of the plurality of enlargement positions in a depth image corresponding to the input image. (5) The display processing device according to any of (1) to (4), further comprising: an enlargement position setting unit that sets the plurality of enlargement positions according to a predetermined number of the plurality of enlarged images to be displayed. (6) The display processing device according to (5), wherein the plurality of enlargement positions include a position specified by a user in the preview image. (7) The display processing device according to (5), wherein the input image includes a first input image acquired from a first viewpoint in real space and a second input image acquired from a second viewpoint different from the first viewpoint, the preview image includes a first preview image corresponding to the first input image and a second preview image corresponding to the second input image, and the enlargement position setting unit automatically sets the enlargement position in the second preview image to a position corresponding to at least one of the enlargement positions set in the first preview image. (8) The display processing device according to (5), wherein the enlargement position setting unit automatically sets the enlargement position in the preview image to a position corresponding to the region with the largest edge among regions corresponding to each of a plurality of pieces of depth information set in a depth image corresponding to the input image.(9) The display processing device according to any one of (5) to (8), further comprising an instruction image superimposing unit that superimposes an instruction image indicating the enlargement position at the enlargement position set in the input image or the preview image. (10) The display processing device according to any one of (1) to (9), wherein the enlarged image generation unit generates the enlarged images by cutting out a plurality of rectangular areas from the input image or the preview image that respectively correspond to the plurality of enlargement positions. (11) The display processing device according to (10), wherein the enlarged image generation unit generates the enlarged images by reading out pixels corresponding to the plurality of rectangular areas from the image sensor. (12) The display processing device according to any one of (1) to (11), wherein the plurality of enlarged images include a focus area image obtained by enlarging the focus position in the preview image and a peripheral area image obtained by enlarging the peripheral position in the preview image, and the display control unit controls the display unit to simultaneously display the focus area image and the peripheral area image. (13) The display processing device according to any one of (1) to (12), wherein the display control unit controls the display unit to display a plurality of tabs representing the plurality of enlarged images. (14) The display processing device according to any one of (1) to (13), wherein the display control unit controls the display unit to switch between displaying the preview image and the plurality of enlarged images based on a user's operation input. (15) The display processing device according to any one of (1) to (13), wherein the display control unit controls the display unit to simultaneously display the preview image and the plurality of enlarged images. (16) A display processing method comprising: generating a plurality of enlarged images by enlarging a plurality of enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image acquired from an image sensor or a preview image for checking the input image, and controlling a display unit to simultaneously display the plurality of enlarged images.(17) A program for causing a processor to execute a process of generating a plurality of enlarged images by enlarging a plurality of enlargement positions including at least one of a focus position and a peripheral position away from the focus position in an input image acquired from an image sensor or a preview image for confirming the input image, and controlling a display unit to simultaneously display the plurality of enlarged images.

[0087] REFERENCE SIGNS LIST 1 Information processing device, 10 Monitoring image generation unit, 11 Enlargement position setting unit, 12 Instruction image superimposition unit, 13 Enlarged image generation unit, 14 Enlarged image integration unit, 15 Display control unit, 111 Enlargement position setting unit

Claims

1. A display processing device comprising: an enlarged image generation unit that generates a plurality of enlarged images by enlarging a plurality of enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image obtained from an image sensor or a preview image for confirming the input image; and a display control unit that controls a display unit to simultaneously display the plurality of enlarged images.

2. The display processing device according to claim 1, further comprising an arrangement determination unit that determines an arrangement in which the plurality of enlarged images are displayed according to the positional relationship of the plurality of enlargement positions.

3. The display processing device according to claim 2, wherein the positional relationship includes coordinate information of each of the plurality of enlargement positions on the two-dimensional coordinate system of the input image or the preview image.

4. The display processing device according to claim 2, wherein the positional relationship includes depth information for each of the plurality of enlarged positions in a depth image corresponding to the input image.

5. The display processing device according to claim 1, further comprising an enlargement position setting unit that sets the plurality of enlargement positions in accordance with a preset number of the plurality of enlarged images to be displayed.

6. The display processing device according to claim 5, wherein the plurality of enlargement positions include a position in the preview image designated by the user.

7. The display processing device according to claim 5, wherein the input image includes a first input image acquired from a first viewpoint in real space and a second input image acquired from a second viewpoint different from the first viewpoint, the preview image includes a first preview image corresponding to the first input image and a second preview image corresponding to the second input image, and the enlargement position setting unit automatically sets the enlargement position in the second preview image to a position corresponding to at least one of the enlargement positions set in the first preview image.

8. The display processing device according to claim 5, wherein the enlargement position setting unit automatically sets the enlargement position in the preview image to a position corresponding to the area with the largest edge among the areas corresponding to each of multiple depth information set in a depth image corresponding to the input image.

9. The display processing device according to claim 5, further comprising an indication image superimposing unit that superimposes an indication image indicating the enlargement position at the enlargement position set in the input image or the preview image.

10. The display processing device according to claim 1, wherein the enlarged image generating unit generates the enlarged images by cutting out a plurality of rectangular areas from the input image or the preview image, each of which corresponds to one of the plurality of enlargement positions.

11. The display processing device according to claim 10, wherein the enlarged image generating unit generates the plurality of enlarged images by reading pixels corresponding to the plurality of rectangular areas from the image sensor.

12. The display processing device according to claim 1, wherein the plurality of enlarged images include a focus area image obtained by enlarging the focus position in the preview image and a peripheral area image obtained by enlarging the peripheral position in the preview image, and the display control unit controls the display unit to simultaneously display the focus area image and the peripheral area image.

13. The display processing device according to claim 1, wherein the display control unit controls the display unit to display a plurality of tabs representing the plurality of enlarged images.

14. The display processing device according to claim 1, wherein the display control unit controls the display unit to switch between displaying the preview image and the plurality of enlarged images based on a user's operation input.

15. The display processing device according to claim 1, wherein the display control unit controls the display unit to simultaneously display the preview image and the plurality of enlarged images.

16. A display processing method comprising: generating a plurality of enlarged images by enlarging a plurality of enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image acquired from an image sensor or a preview image for confirming the input image; and controlling a display unit to simultaneously display the plurality of enlarged images.

17. A program for causing a processor to execute a process of generating a plurality of enlarged images by enlarging a plurality of enlargement positions, including at least one of a focus position and a peripheral position away from the focus position, in an input image obtained from an image sensor or a preview image for confirming the input image, and controlling a display unit to simultaneously display the plurality of enlarged images.

Images