Image processing device and image processing method

Abstract

The image processing device according to the present technology comprises: a near-far separation unit for separating an input image into a foreground image portion, which is a portion in which a subject in the foreground is captured, and a background image portion, which is a portion in which a subject further toward the background than the foreground image portion is captured; a foreground correction unit for making a correction to the foreground image portion; and an inconsistency correction unit for making a correction to an inconsistent portion produced due to the correction of the foreground image portion by the foreground correction unit.

Description

[Technical Field]

[0001] The present technology relates to an image processing device and a method thereof. to The present invention relates to an image processing technique suitable for dividing an input image into a foreground image portion and a background image portion and correcting the foreground image portion. [Background technology]

[0002] For example, there is a growing demand for public viewing and group viewing, which allow people to enjoy events such as sporting matches and concerts at remote venues. In these viewing styles, images of the venue where the event is actually taking place are displayed on a display device located at the remote venue. In this case, the images at the remote venue must be displayed on a large screen to enhance the sense of unity with the event.

[0003] As a related prior art, the following Patent Document 1 can be cited. Patent Document 1 discloses a technology relating to an image processing device that includes a correction unit that adaptively corrects the image quality of at least one of a foreground and background image in accordance with the difference in image quality between the foreground and background images, and a synthesis unit that synthesizes the corrected foreground and background images to generate a synthesized image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2020-27409 Summary of the Invention [Problem to be solved by the invention]

[0005] In a remote venue, if an image captured from the audience seats is displayed on a large screen as an image of an event, the audience members closest to the camera will be displayed enlarged on the screen. For example, depending on the screen size, the audience members closest to the camera may appear larger than life-size, which can cause a strong sense of discomfort to the user and significantly reduce the sense of immersion.

[0006] The present technology has been made in view of the above circumstances, and aims to alleviate the sense of discomfort felt by users when a foreground subject is displayed, and to improve the user's sense of immersion in image content. [Means for solving the problem]

[0007] The image processing device according to the present technology includes a near-distance dividing section that divides an input image into a near-distance image portion that is a portion capturing a near subject and a far-distance image portion that is a portion capturing a subject that is farther away than the near-distance image portion, a near-distance correction section that performs correction on the near-distance image portion, and an inconsistency correction section that corrects inconsistencies that arise as a result of the correction made to the near-distance image portion by the near-distance correction section. Here, the inconsistent portion refers to a portion where an inconsistency occurs in visual elements such as the shape, size, color, etc. of the subject. For example, if a correction is performed to make the entire foreground image portion transparent as foreground correction, the foreground image portion becomes a non-image portion in the input image, and the entire foreground image portion becomes an inconsistent portion. According to the above configuration, the correction by the foreground correction unit makes it possible to make a correction that makes a subject that is shown close up in the foreground less noticeable, and the inconsistency correction unit corrects the inconsistency portion caused by the correction by the foreground correction unit, making it possible to alleviate the discomfort felt by the user by the inconsistency portion.

[0008] In the image processing device according to the present technology described above, the near / far division unit can be configured to divide the near image portion and the far image portion based on distance measurement information of a subject in the input image. By using distance measurement information of the subject, it becomes possible to appropriately identify the near view image portion or the far view image portion.

[0009] In the image processing device according to the present technology described above, the near / far division unit can be configured to divide the near image portion and the far image portion based on the difference in distance between subjects in the input image. By dividing the image based on the difference in distance, it is possible to identify as a close-up image part parts that are likely to cause discomfort to the user, parts that have a large difference in distance from specific parts in the distance, such as the part of a hand covering the field of a stadium.

[0010] In the image processing device according to the present technology described above, the near / far division unit can be configured to divide the input image into the near image portion and the far image portion based on an image analysis result for the input image. This eliminates the need to use a distance measuring sensor to separate the image into a near view image portion and a far view image portion.

[0011] In the image processing device according to the present technology described above, the near / far division unit can be configured to divide the input image into the near image portion and the far image portion based on the result of object recognition processing performed on the input image. This makes it possible to properly recognize in which part of the image an object that should be in the foreground or background is located.

[0012] In the image processing device according to the present technology described above, the near-distance dividing unit may be configured to perform the object recognition process using an image recognizer based on artificial intelligence. By using an image recognizer based on artificial intelligence, it is possible to appropriately set elements of object recognition, such as the types of objects that can be recognized and the accuracy of recognition, depending on the learning method.

[0013] In the image processing device according to the present technology described above, the near / far division unit may be configured to divide the near image portion and the far image portion by performing image analysis using template matching. This eliminates the need for artificial intelligence training to distinguish between foreground and background image portions.

[0014] In the image processing device according to the present technology described above, the foreground correction section can be configured to make at least a part of the foreground image portion transparent or reduce the size of the foreground image portion as the correction to the foreground image portion. By making the image transparent or reducing it in size as described above, it is possible to make the foreground subject less noticeable.

[0015] In the image processing device according to the present technology described above, the inconsistency correction unit can be configured to correct the current inconsistency part, which is the inconsistency part in the current frame, by using an image of the overlapping part with the current inconsistency part in the background image part of the past frame. For example, if the foreground subject is a spectator in the stands, the subject may move, and a background portion that is not visible in the current mismatch portion in the current frame may be visible as the overlapping portion in the past frame. According to the above configuration, if there is an overlapping portion with the current mismatch portion in the background image portion of the past frame, the current mismatch portion is corrected using the image of the overlapping portion.

[0016] In the image processing device according to the present technology described above, the inconsistency correction unit can be configured to generate a last value holding image, which is an image that holds the pixel value for each pixel of the input image when it was last divided into a background image portion at least within a period prior to the current frame, and to correct the current inconsistency portion using the last value holding image. As a result, if there is an overlapping portion between the past background image portion and the current mismatch portion, the pixel values ​​of the pixels in the overlapping portion when it was last classified as a background image portion are used to correct the current mismatch portion.

[0017] In the image processing device according to the present technology described above, the inconsistency correction unit can be configured to generate the last value-retaining image by performing a process over multiple frames in which, for pixels that are divided into distant image portions, the pixel values ​​of the pixels are retained, and for pixels that are divided into near image portions, the existing retained values ​​are maintained. This makes it possible to appropriately generate the latest value holding image.

[0018] In the image processing device according to the present technology described above, the inconsistency correction unit can be configured to interpolate pixel values ​​for parts of the current inconsistency portion where no overlapping portion exists using an image of the overlapping portion. This makes it possible to appropriately interpolate pixels outside the overlapping portion using the image of the overlapping portion, even if the overlapping portion with respect to the current mismatched portion is only partial.

[0019] In the image processing device according to the present technology described above, the near / far division unit can be configured to change the size of the distance range divided as the near image portion so as to have a negative correlation with the distance between the display surface on which the input image is displayed and the user viewing the display image on the display surface. When the distance between the display surface and the user is short, the subject in the image appears larger to the user, and the range of distances in the foreground that the user finds uncomfortable also becomes wider. Therefore, as described above, the size of the distance range classified as the foreground image portion is changed so that it has a negative correlation with the distance between the display surface and the user; in other words, the shorter the distance, the larger the distance range classified as the foreground image portion.

[0020] In the image processing device according to the present technology described above, the near / far division unit can be configured to change the size of the distance range divided as the near image portion so that it has a positive correlation with the size of the display area of ​​the input image. When the display area for displaying the input image is large, the subject in the image appears larger to the user, and the distance range of the foreground that the user finds uncomfortable also becomes wider. Therefore, as described above, the size of the distance range classified as the foreground image portion is changed so that it has a positive correlation with the size of the display area; in other words, the larger the display area, the larger the distance range classified as the foreground image portion.

[0021] The image processing method according to the present technology divides an input image into a close-up image portion that captures a close-up subject and a distant image portion that captures a subject that is further away than the close-up image portion, corrects the close-up image portion, and corrects inconsistencies that arise from the correction to the close-up image portion. This image processing method also provides the same effects as the image processing device according to the present technology described above. [Brief explanation of the drawings]

[0023] [Figure 1] 1 is a diagram illustrating an example of the configuration of an image display system according to an embodiment of the present technology. [Figure 2] 1 is a block diagram illustrating an example of the internal configuration of a projector device including an image processing device according to an embodiment. [Figure 3] 10A and 10B are explanatory diagrams of foreground correction in the embodiment. [Figure 4] FIG. 2 is a functional block diagram showing the functional configuration of an image processing unit in the first embodiment. [Figure 5] FIG. 2 is an explanatory diagram of functions of a mismatch correction unit in the embodiment. [Figure 6] FIG. 2 is a diagram illustrating an example of image content of a current frame and a past frame. [Figure 7] 10A and 10B are explanatory diagrams illustrating correction of a mismatched portion in the embodiment. [Figure 8] FIG. 10 is a diagram illustrating an example of image content of a mismatched portion after correction. [Figure 9]FIG. 2 is a block diagram illustrating an example of the internal configuration of a latest value holding image generating unit in the embodiment. [Figure 10] 10 is a flowchart illustrating an example of a specific processing procedure for realizing image correction according to the first embodiment. [Figure 11] FIG. 10 is a diagram showing functional blocks of an image processing unit as a modified example relating to near and far divisions. [Figure 12] 10A and 10B are diagrams showing examples of foreground section information and background section information. [Figure 13] FIG. 10 is a functional block diagram showing the functional configuration of an image processing unit according to a second embodiment. [Figure 14] FIG. 10 is an explanatory diagram of a first example of the third embodiment. [Figure 15] FIG. 10 is a block diagram illustrating a configuration of a projector device as a first example of a third embodiment. [Figure 16] FIG. 10 is an explanatory diagram of a second example of the third embodiment. [Figure 17] FIG. 13 is a block diagram illustrating a configuration of a projector device as a second example of the third embodiment. [Figure 18] FIG. 10 is an explanatory diagram of an example of life-size processing. [Figure 19] FIG. 10 is an explanatory diagram of another example of life-size processing. [Figure 20] 10A and 10B are explanatory diagrams of a modified example in which near and far division is performed based on the difference in distance between subjects. DETAILED DESCRIPTION OF THE INVENTION

[0024] The embodiments will be described below in the following order. <1. First embodiment> (1-1. System Configuration) (1-2. Image Processing Device as an Embodiment) (1-3. Image Correction as an Embodiment) (1-4. Processing Procedure) (1-5. Modifications related to near and far divisions) 2. Second Embodiment 3. Third Embodiment (3-1. First example) (3-2. Second example) <4. Modifications> <5. Summary of embodiments> <6. This technology>

[0025] <1. First embodiment> (1-1. System Configuration) FIG. 1 shows an example of the configuration of an image display system 100 as an embodiment according to the present technology. In this example, the image display system 100 is a system that is applied to the public viewing and group viewing described above. In these viewing styles using public viewing and group viewing, captured images of an event such as a competitive sport such as soccer, baseball, or basketball, or a concert are displayed to users as spectators at a remote venue. The "imaging environment" in the figure is an environment in which the state of the event is captured, and the imaging device 2 is placed as shown. The imaging device 2 is configured as a digital camera device having a solid-state imaging element such as a CCD (Charge Coupled Device) sensor or a CMOS (Complementary Metal-Oxide-Semiconductor) sensor, and captures captured images as digital data. In this example, the imaging device 2 captures captured images as moving images. The number of imaging devices 2 is not limited to one, and a plurality of imaging devices may be provided.

[0026] Furthermore, the "display environment" is an environment in which captured images of an event obtained in the imaging environment are displayed, and is located in a place remote from the venue where the event is held, such as a stadium or concert hall. A device for displaying the captured images is disposed in the display environment. Specifically, in this example, a projector device 1 is disposed as the display device, and users in the display environment (i.e., spectators at the remote venue) can view the remote event venue from the images projected by the projector device 1 onto a projection surface S, such as the surface of a screen.

[0027] In this example, an information processing device 3 for transmitting an image captured by the imaging device 2 to an external device (particularly, the projector device 1 in this example) via a network 4 is disposed in the imaging environment. Here, the network 4 is a communication network such as the Internet or a LAN (Local Area Network), etc. The information processing device 3 is configured by a computer device having a communication function, such as a personal computer.

[0028] As will be described later, in the image display system 100 of this example, the projector device 1 divides the image captured by the imaging device 2 into a foreground image portion and a background image portion. Here, the foreground image portion is a portion of the input image that captures a foreground subject, and the background image portion is a portion that captures a subject that is further away than the foreground image portion.

[0029] To enable such a distinction between the foreground image portion and the background image portion, the imaging environment of this example is provided with a distance measuring device 5. The distance measuring device 5 is configured as a device that measures distance using a known distance measuring method such as a ToF (Time of Flight) method, a structured light method, or a stereo method, and in this example is configured to acquire distance image data (depth image data) as two-dimensional distance measurement data. In this example, the field of view (angle of view) of the distance measurement by the distance measuring device 5 is set to cover the field of view of the imaging device 2, and the distance measuring device 5 is capable of measuring the distance to each subject present within the field of view of the imaging device 2.

[0030] In this example, the imaging device 2 and the distance measuring device 5 assign time codes to the captured images and distance images on a frame-by-frame basis, respectively, making it possible to achieve time synchronization (frame synchronization) between the two images. The information processing device 3 acquires the captured image and distance image with the time code added thereto from the imaging device 2 and the distance measuring device 5, respectively, and transmits them to the projector device 1 via the network 4.

[0031] In addition, the function of the information processing device 3, specifically, the function of acquiring an image captured by the imaging device 2 and distance measurement data (distance image) by the distance measuring device 5 and transmitting them to a predetermined external device (in this example, the projector device 1), can be realized not only by a single computer device but also by the cooperation of multiple computer devices.

[0032] (1-2. Image Processing Device as an Embodiment) 2 is a block diagram illustrating an example of the internal configuration of a projector device 1 including an image processing device according to an embodiment. In addition, FIG. 2 illustrates the projection surface S shown in FIG. 1 together with the example of the internal configuration of the projector device 1.

[0033] As shown in the figure, the projector device 1 includes an image processing unit 10, a communication unit 11, a projection unit 12, and a control unit 13. The communication unit 11 performs communication processing via the network 4 shown in Fig. 1. The communication unit 11 may be configured to be capable of wired or wireless communication with peripheral devices.

[0034] Image processing unit 10 is one embodiment of the image processing device according to the present technology, and performs the above-mentioned division of foreground image portions and background image portions, correction of the foreground image portions (image correction), and correction of inconsistent portions caused by the correction of the foreground image portions for an image (input image: specifically, an image captured by imaging device 2 in this example) input via communication unit 11. Note that the division of foreground image portions and background image portions, the correction of the foreground image portions, and the correction of inconsistent portions will be described again later.

[0035] The projection unit 12 projects a reproduced image of the captured image that has been subjected to image processing such as the correction by the image processing unit 10 onto the projection surface S. Specifically, the projection unit 12 has a spatial light modulator 12a such as a liquid crystal panel, a light source (not shown), and a projection optical system including a projection lens, etc., and the spatial light modulator 12a generates a reproduced image of the captured image by performing spatial light modulation such as spatial light intensity modulation on incident light from the light source based on the captured image input from the image processing unit 10. The reproduced image thus generated is then projected onto the projection surface S via the projection optical system.

[0036] The control unit 13 is configured with a microcomputer having, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc., and performs overall control of the projector device 1 by executing processing according to a program stored in, for example, the ROM. For example, the control unit 13 controls data communication by the communication unit 11, instructs the image processing unit 10 to execute and terminate image processing, and performs various controls on the projection unit 12 (such as instructing the above-mentioned light source to be on / off and controls for various distortion corrections of the projected image).

[0037] (1-3. Image Correction as an Embodiment) 3A shows an example of an image projected onto the projection surface S, i.e., an example of a display image on the projection surface S. Specifically, the example shows a display image in the case where an event such as a soccer match is captured by an imaging device 2 placed in the spectator seats. In this case, the captured image will have spectators at the event venue in the foreground (the side closer to the imaging device 2) and the competition field and players on the competition field in the background, as shown in the figure.

[0038] At this time, the subject captured in the foreground in the captured image, especially the subject located in the frontmost row, will be the subject closest to the imaging device 2, and depending on its positional relationship with the imaging device 2, will be displayed huge in the captured image (display image). Displaying an audience is effective in enhancing the sense of realism, but as mentioned above, if the subject is displayed too large, it may give the user a sense of discomfort and reduce the sense of immersion in the image content.

[0039] Therefore, in this embodiment, as shown by the transition from Fig. 3A to Fig. 3B, correction (foreground correction) is performed to make foreground subjects that may appear large less noticeable. For example, a process of making foreground subjects transparent or reducing their size is performed.

[0040] However, when correction processes such as transparency and reduction are performed, inconsistencies occur in the image due to the correction. The inconsistent portion referred to here means a portion where an inconsistency occurs in visual elements such as the shape, size, color, etc. of the subject. For example, if correction is performed to make the entire foreground image portion transparent as foreground correction, the foreground image portion becomes a non-image portion in the captured image, and the entire foreground image portion becomes an inconsistent portion. Alternatively, if foreground correction is performed to reduce the foreground subject, the difference between the subject before and after reduction becomes a non-image portion, and this non-image portion becomes an inconsistent portion.

[0041] Therefore, in this embodiment, the above-described foreground correction is performed, and also the inconsistent portion caused by the foreground correction is corrected. In the projector device 1, the image processing unit 10 performs correction processing for the foreground and correction processing for the inconsistent portion.

[0042] FIG. 4 is a functional block diagram showing the functional configuration of the image processing unit 10. As shown in the figure, the image processing unit 10 includes a near / far division unit 15, a near view correction unit 16, a mismatch correction unit 17, and a synthesis unit 18. The near-far division section 15 receives the captured image from the imaging device 2 and distance measurement data (distance image data in this example) from the distance measuring device 5 via the communication section 11 shown in Fig. 2. As mentioned above, the captured image and distance image are assigned time codes, and the near-far division section 15 can achieve frame synchronization between the captured image and distance image.

[0043] The near / far division unit 15 divides the input image as the captured image into a near image portion and a far image portion. Specifically, in this example, the near / far division unit 15 classifies the captured image into parts where the distance value is below a predetermined value as near-distance image parts and other parts as distant-distance image parts based on the distance image data input as ranging data. The near / far division unit 15 outputs an image of a portion divided as a near image portion in the input captured image as a near image, and outputs an image of a portion divided as a distant image portion as a distant image.

[0044] In the above example, the near / far division is performed by determining the near image portion and then designating the portion other than the near image portion as the distant image portion. However, it is also possible to determine the distant image portion and then designate the portion other than the distant image portion as the near image portion. In addition, although the above example shows that the near / far division is based on distance measurement data, it is not essential to use distance measurement data for the near / far division, as will be explained later.

[0045] The foreground correction unit 16 applies correction to the foreground image output by the near / far division unit 15 to make the subject less noticeable. As described above, foreground correction, transparency and reduction are possible. In the figure, the transparency function is shown as transparency processing unit 16a, and the reduction function is shown as reduction processing unit 16b. In the following description, as an example, the process for making foreground subjects less noticeable is assumed to be a transparency process. Here, transparency is assumed to be a process for maximizing the transparency of the target image part (i.e., completely erasing it), but the transparency setting is arbitrary. The process of reducing the image will be explained later.

[0046] Foreground correction unit 16 outputs the image of the inconsistent portion resulting from the correction as an "inconsistent portion image" to inconsistency correction unit 17. Since the inconsistent portion image here is subjected to the transparency processing as described above as part of the foreground correction, it becomes a foreground image subjected to transparency processing.

[0047] The mismatch correction unit 17 corrects the mismatch portion that occurs due to the correction made to the foreground image portion by the foreground correction unit 16 . In this example, the mismatch correction unit 17 corrects the mismatched partial image (that is, the transparentized foreground image in this example) output by the foreground correction unit 16 based on the background image output by the near / far division unit 15.

[0048] FIG. 5 is an explanatory diagram of the functions of the mismatch correction unit 17, showing a functional block diagram illustrating the functional configuration of the mismatch correction unit 17, together with the foreground correction unit 16. As shown in the figure, the mismatch correction unit 17 includes a latest value holding image generation unit 17a, a mismatch pixel correction unit 17b, and a blur processing unit 17c.

[0049] Here, the mismatch correction unit 17 corrects mismatched portions for each frame of the input image (captured image) as a moving image. Hereinafter, the frame that is the subject of processing for correcting the foreground image and the inconsistent portion that occurs as a result of correcting the foreground image will be referred to as the "current frame." Furthermore, the inconsistent portion in the current frame will be referred to as the "current inconsistent portion."

[0050] In this example, the mismatch correction unit 17 corrects the current mismatch portion by using an image of the overlapping portion with the current mismatch portion in the distant view image of the past frame. For example, if the foreground subject is a spectator in the stands, the subject may move, and the background part that is not visible in the current mismatched part in the current frame may be visible as the overlapped part in the past frame. Therefore, the accuracy of correcting the mismatched part is improved by using an image of the overlapped part in the past frame (i.e., an image of the part that was actually visible in the past).

[0051] In this example, in order to retain the image of the overlapping portion of the past background image with the current mismatched portion as described above, the latest value retained image generating unit 17a generates a latest value retained image. The latest value retained image is an image that retains the pixel values ​​when the image was last divided into a background image portion at least within a period of time prior to the current frame.

[0052] An example of a method for correcting mismatched portions using such a latest value holding image will be described with reference to FIGS. Figure 6 shows an example of the image content of the current frame and past frames. Figure 6A shows an example of the image content of the current frame (assumed to be frame N), Figure 6B shows an example of the image content of frame Nx, which is the frame x frames before the current frame, and Figure 6C shows an example of the image content of frame Ny, which is the frame y frames before the current frame (where y>x).

[0053] In each of these Figures 6A to 6C, the black portions represent the portions of the respective frames that are classified as foreground images. 6B and 6C, the circled portions are portions of the background image that are not visible in the current frame because they are obscured by the subject in the foreground image. In other words, they are portions of the background image in the previous frame that overlap with the current mismatched portion.

[0054] By using such an overlapping portion of the distant view image of the past frame with the current mismatched portion, it is possible to compensate for the distant view portion in the current frame that is blocked by the current mismatched portion (that is, the entire foreground image in this case).

[0055] In Figure 7A, when the current frame and past frame illustrated in Figure 6 are obtained, the dotted lines indicate the current mismatch portion that can be compensated for (i.e., filled in) by the background image in the past frame (Figures 6B and 6C).

[0056] On the other hand, in FIG. 7B, the white area surrounded by a solid line (the same as the black area in FIG. 7A) represents an area that cannot be compensated for with the past distant images of FIGS. 6B and 6C. In this example, for portions that cannot be compensated for using the past distant view image, that is, portions of the current mismatched portion where the above-mentioned overlapping portion does not exist, pixel values ​​are interpolated using the image of the overlapping portion. A specific example of interpolation is a technique of copying the pixel value of an adjacent pixel, for example, copying the pixel value of an adjacent pixel on the left or right.

[0057] The interpolation method is not particularly limited, and examples include a method of interpolating the pixel value of a pixel to be interpolated using pixel values ​​of four pixels adjacent to the pixel to be interpolated vertically and horizontally, or eight pixels adjacent to the pixel to be interpolated vertically, horizontally, and diagonally.

[0058] Figure 8 shows an example of the image content of the current mismatch portion obtained by performing the above-described correction process for the mismatch portion, i.e., pixel filling using the overlapping portion of the past background image with the current mismatch portion, and interpolation processing for the portion that could not be filled using the overlapping portion. In the figure, the portion indicated by the dotted line represents the current mismatch portion after correction (in this example, the entire foreground image is made transparent, so it has the same shape and size as the foreground image). The remaining portion is the background image of the current frame. Note that FIG. 8 illustrates, as an example of the image content of the current mismatched portion after the correction process, the image content when the process of copying the pixel value of the pixel adjacent to the left side in the above-mentioned interpolation process is performed.

[0059] As can be seen by referring to FIG. 8, by performing the above-described correction process for the inconsistent parts, the parts that have become inconsistent due to the foreground correction can be made less noticeable, thereby alleviating the discomfort felt by the user due to the inconsistent parts, that is, the discomfort felt by the user due to the foreground correction can be alleviated.

[0060] FIG. 9 is a block diagram showing an example of the internal configuration of the latest value holding image generating unit 17a. As shown in the figure, the latest value holding image generating unit 17a includes a frame memory 21, a write control unit 22, and a read control unit . A background image is input to write control unit 22 for each frame. Write control unit 22 performs the following process for each frame. That is, for pixels that constitute one frame and that are classified as background images, the pixel values ​​of those pixels are held in frame memory 21, and for pixels that are classified as foreground images (image portions other than the background image), the existing held values ​​(i.e., pixel values ​​already held) are maintained in frame memory 21. Specifically, for pixels that are classified as background images, the pixel values ​​of the corresponding pixels in frame memory 21 are overwritten with the pixel values ​​of the pixels classified as background images, and for pixels that are classified as foreground images, the pixel values ​​of the corresponding pixels in frame memory 21 are maintained as they are without being overwritten. By performing such writing control over a plurality of frames, the frame memory 21 stores the last value holding image.

[0061] The read control unit 23 reads out the last value holding image held in the frame memory 21 for each frame and outputs it to the mismatch pixel correction unit 17b shown in FIG.

[0062] 5, mismatch pixel corrector 17b performs correction processing of the mismatch portion by the method described with reference to FIGS. 6 to 8 based on the mismatch portion image input from foreground corrector 16, the latest value held image input from latest value held image generator 17a, and the background image of the current frame output by near-far section 15 (FIG. 4), i.e., performs pixel filling using the overlapping portion with the current mismatch portion in the past background image, and performs interpolation processing for the portion that could not be filled using the overlapping portion. In this interpolation processing, mismatch pixel corrector 17b uses the background image from near-far section 15 as necessary. Hereinafter, the image of the mismatched portion (current mismatched portion) that has been subjected to the correction process by the mismatched pixel corrector 17b will be referred to as the "corrected mismatched portion image."

[0063] The blurring processing unit 17c performs blurring on the corrected mismatched portion image input from the mismatched pixel correction unit 17b. The blurring process is a process of adding visual blur to the target image. Specifically, examples of this include blurring of edges (shape blurring) and color blurring. By performing this blurring process, the mismatched portion caused by the foreground correction can be made less noticeable, further reducing the sense of incongruity.

[0064] The blurring processing unit 17c outputs the blurred mismatched portion image after correction to the foreground correction unit 16.

[0065] Foreground correction unit 16 replaces the image of the portion of the foreground image that corresponds to the inconsistent portion with the corrected inconsistent portion image. The foreground image obtained by replacing the image of the portion that corresponds to the inconsistent portion with the corrected inconsistent portion image in this way is hereinafter referred to as the "corrected foreground image." When the foreground image is entirely made transparent as part of the foreground correction as in this example, the inconsistent portion becomes the same as the foreground image, and the corrected foreground image becomes the corrected inconsistent portion image itself.

[0066] 4, composition unit 18 combines the corrected foreground image obtained by foreground correction unit 16 with the background image input from near / far division unit 15. That is, the two images are combined to obtain one frame of image. 2 is driven based on one frame of images obtained by the synthesis process of the synthesis unit 18. As a result, a reconstructed image of the synthesized image is projected onto the projection surface S, and the reconstructed image is viewed by a user in the display environment.

[0067] The functions of the image processing unit 10 described above can be realized by software processing, or alternatively, the image processing unit 10 can be configured with hardware that realizes each function. It is also possible to realize part of the functions of the image processing unit 10 by software processing and the other part by hardware.

[0068] (1-4. Processing Procedure) Fig. 10 is a flowchart showing a specific example of a processing procedure for realizing the image correction according to the first embodiment. The processing shown in Fig. 10 is executed for each frame of an input image (in this example, a captured image input from an imaging environment).

[0069] First, in step S101, image processing unit 10 executes near / far classification processing. This processing is performed by near / far classification unit 15, and in this example, based on the distance image data obtained by distance measuring device 5, the processing classifies the captured image into a portion where the distance value is equal to or less than a predetermined value as a near image portion, and the other portion as a far image portion.

[0070] In step S102 following step S101, the image processing unit 10 executes a foreground correction process. That is, a predetermined correction process is performed on the foreground image, which is an image of a portion of the input captured image that has been classified as a foreground image portion. Specifically, in this example, a process is performed to make the entire foreground image transparent.

[0071] In step S103 following step S102, the image processing unit 10 acquires the latest value holding image, that is, the latest value holding image obtained by the configuration described with reference to FIG.

[0072] In step S104 following step S103, image processing unit 10 executes a mismatch pixel filling process. That is, based on the last value held image acquired in step S103, pixels of the current mismatch portion are filled in using an overlapping portion of the past background image with the current mismatch portion. That is, if there is a portion of the current mismatch portion that was previously visible in the background image, the pixel values ​​of that portion are applied as is.

[0073] In step S105 following step S104, the image processing unit 10 determines whether or not there is an unfillable portion. That is, it determines whether or not there is a portion of the current mismatch that could not be filled with the past background image in the filling process of step S104.

[0074] If there is an unfillable portion, the image processing unit 10 proceeds to step S106 to perform interpolation processing for the unfillable portion, that is, interpolation processing based on the pixel values ​​of adjacent pixels as described above. After the interpolation process in step S106 is completed, the image processing unit 10 advances the process to step S107. On the other hand, if the image processing unit 10 determines in step S105 that there is no portion that cannot be filled, it skips the interpolation process in step S106 and proceeds to step S107.

[0075] In step S107, the image processing unit 10 performs blurring of the mismatched portion. As described above, the blurring may be any process that provides a visual blurring effect.

[0076] In step S108 following step S107, image processing unit 10 generates a corrected foreground image based on the corrected inconsistent portion image. That is, the corrected foreground image is generated based on the corrected inconsistent portion image obtained by applying the filling process in step S104, the interpolation process in step S106 which is executed as needed, and the blurring process in step S107. As can be understood from the above explanation, in this example where the foreground image correction process involves making the entire foreground image transparent, the process in step S108 treats the post-correction inconsistent partial image itself as the corrected foreground image.

[0077] In step S109 following step S108, image processing unit 10 combines the corrected foreground image and the background image. That is, as described above as combining unit 18, the background image of the current frame and the corrected foreground image are combined to obtain an image for one frame.

[0078] After executing the process of step S109, the image processing unit 10 ends the series of processes shown in FIG.

[0079] While the above example illustrates making the entire foreground image transparent as foreground correction, it is also possible to make only a portion of the foreground image transparent, such as the boundary with the background image. In this case, of the divided foreground image portions, only the transparent portion becomes the inconsistent portion. When only a portion of the foreground image is made inconsistent in this way, the corrected inconsistent portion becomes a part of the foreground image. Therefore, in this case, foreground correction unit 16 combines the corrected inconsistent portion image made in this way with the portion of the foreground image excluding the inconsistent portion (i.e., the portion not made transparent as foreground correction) to generate a corrected foreground image.

[0080] For foreground correction, it is also possible to perform reduction. In this case, the difference between the foreground image after reduction and the foreground image before reduction becomes an inconsistent part (specifically, a non-image part). In this case too, the post-correction inconsistent part becomes a part of the divided foreground image part, and therefore, in the foreground correction unit 16, the post-correction inconsistent part image thus made a part of the foreground image part is synthesized with the part of the foreground image part excluding the inconsistent part (i.e., in this case, the part of the foreground image after reduction) to generate a corrected foreground image.

[0081] (1-5. Modifications related to near and far divisions) Here, in the above example, the near-distance division unit 15 not only divides the image into a near-distance image portion and a distant image portion, but also generates a near-distance image and a distant image according to these divisions, but it is not essential that the near-distance division unit 15 generates a near-distance image and a distant image.

[0082] FIG. 11 is a diagram showing functional blocks of an image processing unit 10' as a modified example relating to near and far division. In the following description, parts that are similar to parts that have already been described will be given the same reference numerals and description thereof will be omitted.

[0083] As shown in the figure, in the image processing unit 10', a near-distance division unit 15' is provided instead of the near-distance division unit 15, a near-distance correction unit 16' is provided instead of the near-distance correction unit 16, and an inconsistency correction unit 17' is provided instead of the inconsistency correction unit 17. The near-distance division section 15' outputs near-distance division information and far-distance division information based on the results of near-distance division.

[0084] FIG. 12 is a diagram showing an example of foreground section information and background section information. The foreground classification information is information indicating, for each pixel in a frame, whether or not the pixel is classified as a foreground image portion, and the background classification information is information indicating, for each pixel in a frame, whether or not the pixel is classified as a background image portion. 12A illustrates an example of foreground division information in which the value of pixels classified as foreground image portions is set to "1" and the value of other pixels is set to "0." FIG. 12B illustrates an example of background division information in which the value of pixels classified as foreground image portions is set to "1" and the value of other pixels is set to "0."

[0085] 11, foreground correction section 16' receives foreground division information from near / far division section 15' and an input image. Foreground correction section 16' obtains a foreground image based on the input image and foreground division information, and performs correction processing on the foreground image, such as making it transparent.

[0086] The mismatch correction unit 17' receives the background division information from the near / far division unit 15' and the input image. The mismatch correction unit 17' obtains a background image of the current frame based on the input image and the background division information, and performs the above-mentioned filling and interpolation processing for the mismatched portion based on the background image.

[0087] 2. Second Embodiment Next, a second embodiment will be described. In the first embodiment, an example has been given in which distance measurement data from the distance measuring device 5 is used in the near / far division. However, the near / far division may also be performed by image analysis of the image captured by the imaging device 2.

[0088] FIG. 13 is a functional block diagram showing the functional configuration of an image processing unit 10A according to the second embodiment. Image processing unit 10A differs from image processing unit 10 in that a near-far division unit 15A is provided instead of near-far division unit 15. Near-far division unit 15A performs near-far division by performing image analysis on the captured image from imaging device 2 as an input image.

[0089] Here, the image analysis for near / far classification by the near / far classification unit 15A may involve object recognition processing. Specifically, the object recognition processing may involve, for example, object recognition processing using an image recognizer based on AI (Artificial Intelligence). One example would be to use AI that has been trained to recognize the distance of a subject from an input image using machine learning (e.g., deep learning) with distance data as the correct answer data and images as the learning input data. As another example, it is possible to use AI that has been trained to recognize the foreground and background image portions from input images by using identification information for the foreground and background image portions as correct answer data and learning input data as images. Alternatively, it is possible to use AI trained to enable semantic segmentation, which recognizes the types of general objects. In this case, it is possible to predetermine the types of objects expected to be in the foreground, and to classify the parts of the objects recognized in the input image that fall into the predetermine types as foreground image parts.

[0090] Another method for dividing images into near and far scenes using image analysis is to use template matching. For example, an image template of the back view of an audience member is used to match the input image, and the matching portion is then classified as the near-distance image portion.

[0091] 3. Third Embodiment (3-1. First example) In the third embodiment, the size of the distance range that is divided into the foreground image portion is changed based on a predetermined condition. Below, we will explain two specific examples: one in which the size of the distance range classified as the close-up image portion is changed depending on the user distance based on the image display surface (first example), and one in which the size is changed depending on the size of the image display area (second example). In the following description, the distance range classified as the foreground image portion will be referred to as the "distance range RD."

[0092] FIG. 14 is an explanatory diagram of the first example. In a first example, the size of the distance range RD is changed so as to have a negative correlation with the user distance, which here means the distance between the user and the display surface on which the image is displayed. Specifically, Fig. 14A illustrates an example in which the user is close to the display surface (when the user distance is short), and Fig. 14B illustrates an example in which the user is far from the display surface (when the user distance is long), and the distance range RD is set to be larger when the user distance is short than when the user distance is long. In the example in Fig. 14, when the user distance is short, the distance range RD is set to the first row (the row closest to the user) to the second row, and when the user distance is far, the distance range RD is set to only the first row. The size of the distance range RD can be set by setting the distance threshold used for the near / far division.

[0093] FIG. 15 is a block diagram illustrating a configuration of a projector device 1B as a first example of the third embodiment. First, in this display environment, a distance measuring device 6 is provided to measure the distance to the user. The distance measuring device 6 is configured to obtain a distance image, for example, in the same way as the distance measuring device 5 described above. The distance measuring device 6 is positioned so as to measure the distance from the projection surface S (i.e., the image display surface) side in the direction where the user is located, thereby making it possible to measure the distance to the user. Note that it is not necessary to strictly measure the distance between the display surface and the user as the user distance, but it is sufficient if distance information that correlates with the distance between the display surface and the user can be acquired.

[0094] Projector device 1B differs from projector device 1 in that image processing unit 10B is provided instead of image processing unit 10. Image processing unit 10B differs from image processing unit 10 in that image processing unit 10B has near-far division unit 15B instead of near-far division unit 15.

[0095] The near / far division unit 15B estimates the user distance based on distance measurement data (in this example, distance image data) obtained by the distance measurement device 6. When multiple users (objects as people) are recognized within the field of view of distance measurement, the shortest user distance is selected. Alternatively, the average value of the distances of the multiple users may be used.

[0096] The near / far division unit 15B compares the estimated user distance value with a predetermined threshold, and if the user distance is equal to or less than the threshold, sets a first distance range (for example, a distance range from the first row to the second row) as the distance range RD, and if the user distance is not equal to or less than the threshold, sets a second distance range (for example, a distance range to the first row) that is narrower than the first distance range as the distance range RD.

[0097] (3-2. Second example) FIG. 16 is an explanatory diagram of the second example. In the second example, the size of the distance range RD is changed so as to have a positive correlation with the size of the display area of ​​the image. Specifically, Fig. 16A illustrates an example where the image display area is small, and Fig. 16B illustrates an example where the image display area is large, and the distance range RD is set to be larger when the display area is large than when the display area is small. In the example of Fig. 16, when the display area is small, the distance range RD is set to the first column, and when the display area is large, the distance range RD is set to the first to second columns.

[0098] FIG. 17 is a block diagram illustrating a configuration of a projector device 1C as a second example of the third embodiment. As shown in the figure, in projector device 1C, control unit 13 has display area control unit 13a. Although not shown in the figure, projection unit 12 is provided with an adjustment mechanism for optically adjusting the size of the display area of ​​the image on projection surface S, and display area control unit 13a controls the adjustment mechanism to change the size of the display area.

[0099] In the projector device 1C, an image processing unit 10C is provided instead of the image processing unit 10, and the image processing unit 10C differs from the image processing unit 10 in that a near-far division unit 15C is provided instead of the near-far division unit 15.

[0100] Here, the display area control unit 13a outputs information indicating the size of the display area of ​​the image on the projection surface S (hereinafter referred to as "display area size information") to the near / far division unit 15C. Based on the display area size information output by the display area control unit 13a in this manner, the near / far division unit 15C sets a first distance range (for example, a distance range from the first column to the second column) as the distance range RD if the display area size is equal to or greater than a predetermined threshold, and sets a second distance range (for example, a distance range up to the first column) that is narrower than the first distance range as the distance range RD if the display area size is not equal to or greater than the threshold.

[0101] In the above, it is assumed that the size of the image display area dynamically changes due to optical adjustments in the projection unit 12. However, cases in which the size of the display area differs can also be cited, for example, when the distance from the projection unit 12 to the projection surface S differs. For example, the display environment can be classified into two types: one with ample space and a sufficient distance from the projection unit 12 to the projection surface S, and another with limited space and a sufficient distance from the projection unit 12 to the projection surface S. In the former environment, the display area size will be larger, and in the latter environment, the display area size will be smaller. It is also possible to change the size of the distance range RD to accommodate changes in the display area size due to such environmental differences. In this case, it is possible to estimate information indicating the size of the display area from, for example, an image captured by a camera capturing an image of the projection surface S. Alternatively, it is also possible to measure the distance from the projection unit 12 to the projection surface S and estimate the size of the display area from that distance.

[0102] In the first and second examples described above, the distance range RD is adjusted in two stages, but it is of course possible to adjust it in three or more stages.

[0103] Furthermore, the first and second examples described above can be combined, that is, the distance range RD can be adjusted based on both the user distance and the size of the display area.

[0104] <4. Modifications> The embodiment is not limited to the specific example described above, and various modified configurations can be adopted. For example, in the above example, foreground correction is performed to make foreground subjects less noticeable, but foreground correction can also be performed as a process of correcting the display size of foreground subjects to life-size, as exemplified in Figures 18 and 19. Specifically, Figures 18 and 19 each show an example of a user (represented by diagonal lines) in a display environment, a projection surface S located behind the user, and an image displayed on the projection surface S. Note that Figure 18 imagines a virtual experience of a shopping mall or the like, and Figure 19 imagines a remote dance lesson. 18A and 19A, in the imaging environment, a person as a subject is shown close up in the imaging device 2. For the close-up subject part in this way, correction is performed to reduce it to life-size as foreground correction.

[0105] In this case, the life-size process can involve not only reducing the size of the target subject as described above, but also offsetting the position of the target subject in the image to the rear, which allows for a more appropriate life-size representation than when only resizing is performed.

[0106] In the explanation so far, an example has been given in which the near / far division is performed as a division of the distance range RD, but the near / far division can also be performed based on the difference in distance between subjects. For example, Figure 20 shows a scene in which a spectator in the first row is raising his hand. If the hand overlaps the playing field at the back, as shown in the figure, the difference in distance between the hand and the playing field becomes large. Therefore, for example, it is conceivable to calculate the distance difference between the surrounding pixels and an object within a certain distance range on the near side, and classify the image portion of the object where the distance difference is large as a near-distance object portion. As can be seen from the example in Figure 20, areas with large distance differences from the surroundings are areas that are likely to cause discomfort to the user. For this reason, by classifying such areas as foreground image areas and applying foreground correction, i.e., correction such as transparency to make them less noticeable, it is possible to improve the effectiveness of alleviating the discomfort felt by the user.

[0107] Although the above examples of foreground correction include transparency and reduction, foreground correction can also be performed using a virtual image. For example, a virtual image can be replaced with a pre-prepared image of a person.

[0108] Further, the foreground correction can also be a correction related to color. For example, the saturation or brightness of at least a portion of the color of the portion classified as the foreground image portion is reduced to make it less noticeable. In this case, too, the inconsistent portion resulting from the correction is compensated for as much as possible using the previous background image. Furthermore, for the portion that cannot be compensated for, it is also possible to perform interpolation using the values ​​of adjacent pixels. Furthermore, it is also possible to perform a blurring process on the corrected inconsistent portion image obtained by performing these compensation and interpolation processes.

[0109] Furthermore, although the above examples show that the near / far division is performed using ranging data and image analysis, it is also possible to perform the near / far division based on, for example, a designation operation (manual operation by the user) of at least one of the near image portion or the far image portion.

[0110] Furthermore, although the above provides an example in which the image processing device according to the present technology is applied to a projector device, the image processing device according to the present technology can also be applied to self-luminous display devices such as LED (Light Emitting Diode) displays, LCD (Liquid Crystal Displays), and organic EL (Electro-Luminescence) displays.

[0111] Furthermore, the image processing device according to the present technology can be configured as a device separate from the display device. For example, in the system shown in FIG. 1, an image processing device is provided that receives captured images and distance measurement data from the imaging environment via a network 4, and the image processing device performs image processing according to an embodiment, such as near-distance division, near-distance correction, and correction of mismatched portions. The image processing device can then transmit the processed captured images to a projector device. This configuration has the advantage of allowing existing devices that do not have the image processing function according to the embodiment to be used as display devices, such as projector devices. The present technology is also suitable for application to a multi-projection system in which multiple projector devices share the task of projecting a reconstructed image based on an input image.

[0112] <5. Summary of embodiments> As described above, the image processing device of the embodiment (image processing units 10, 10', 10A, 10B, 10C) comprises a near-distance division unit (15, 15', 15A, 15B, 15C) that divides an input image into a near-distance image portion that is a portion capturing a near subject and a far-distance image portion that is a portion capturing a subject that is farther away than the near-distance image portion, a near-distance correction unit (16, 16') that makes corrections to the near-distance image portion, and an inconsistency correction unit (17, 17') that corrects inconsistencies that arise from the correction made to the near-distance image portion by the near-distance correction unit. According to the above configuration, the correction by the foreground correction section makes it possible to make corrections to make subjects that are shown close up in the foreground less noticeable, and the mismatch correction section corrects the mismatch parts that arise from the correction by the foreground correction section, making it possible to alleviate the discomfort felt by the user by the mismatch parts. Therefore, it is possible to alleviate the sense of discomfort felt by the user when a subject in the foreground is displayed, and it is possible to improve the user's sense of immersion in the image content. Here, if the subject in the foreground is a person, leaving the image in a close-up state can pose a privacy issue, but privacy can be protected by performing foreground correction such as making the image transparent or reducing the size as exemplified above.

[0113] In the image processing device of the embodiment, the near / far division units (15, 15', 15B, 15C) divide the input image into a near image portion and a far image portion based on distance measurement information of the subject in the input image. By using distance measurement information of the subject, it becomes possible to appropriately identify the near view image portion or the far view image portion. Therefore, it is possible to improve the accuracy of dividing the foreground image portion and the background image portion.

[0114] Furthermore, in the image processing device of the embodiment, the near / far division unit divides the input image into a near image portion and a far image portion based on the difference in distance between subjects (see FIG. 20). By dividing the image based on the difference in distance, it is possible to identify as a close-up image part parts that are likely to cause discomfort to the user, parts that have a large difference in distance from specific parts in the distance, such as the part of a hand covering the field of a stadium. Therefore, the effect of reducing the sense of discomfort can be improved.

[0115] Furthermore, in the image processing device of the embodiment, the near / far division unit (15A) divides the input image into a near image portion and a far image portion based on the image analysis result of the input image (see the second embodiment). This eliminates the need to use a distance measuring sensor to separate the image into a near view image portion and a far view image portion. Therefore, it is possible to reduce the number of sensors required to alleviate the sense of discomfort felt by the user due to the display of a close-up subject, and it is possible to reduce the number of components required to build a system and reduce costs.

[0116] In the image processing device according to the embodiment, the near / far division unit divides the input image into a near image portion and a far image portion based on the result of object recognition processing performed on the input image. This makes it possible to properly recognize in which part of the image an object that should be in the foreground or background is located. Therefore, the image can be appropriately divided into a foreground image portion and a background image portion.

[0117] Furthermore, in the image processing device of the embodiment, the near / far division unit performs object recognition processing using an image recognizer based on artificial intelligence. By using an image recognizer based on artificial intelligence, it is possible to appropriately set elements of object recognition, such as the types of objects that can be recognized and the accuracy of recognition, depending on the learning method. Therefore, the image can be appropriately divided into a near view image portion and a far view image portion based on an appropriate object recognition result.

[0118] Furthermore, in the image processing device of the embodiment, the near / far division section performs image analysis using template matching to divide the image into near image portions and far image portions. This eliminates the need for artificial intelligence training to distinguish between foreground and background image portions. Therefore, the cost required for learning can be reduced.

[0119] In the image processing device according to the embodiment, the foreground correction section corrects the foreground image portion by making at least a part of the foreground image portion transparent or reducing the size of the foreground image portion. By making the image transparent or reducing it in size as described above, it is possible to make the foreground subject less noticeable. Therefore, it is possible to alleviate the sense of discomfort felt by the user regarding the display of a close-up subject.

[0120] Furthermore, in the image processing device of the embodiment, the inconsistency correction unit corrects the current inconsistency part, which is the inconsistency part in the current frame, using an image of the overlapping part with the current inconsistency part in the background image part of the past frame. For example, if the foreground subject is a spectator in the stands, the subject may move, and a background portion that is not visible in the current mismatch portion in the current frame may be visible as the overlapping portion in the past frame. According to the above configuration, if there is an overlapping portion with the current mismatch portion in the background image portion of the past frame, the current mismatch portion is corrected using the image of the overlapping portion. Therefore, the current mismatch portion can be appropriately corrected using the image portion of the subject that was actually seen in the distance.

[0121] Furthermore, in the image processing device of the embodiment, the inconsistency correction unit generates a last value holding image, which is an image that holds the pixel value for each pixel of the input image when it was last divided into a background image portion at least within a period prior to the current frame, and corrects the current inconsistency portion using the last value holding image. As a result, if there is an overlapping portion between the past background image portion and the current mismatch portion, the pixel values ​​of the pixels in the overlapping portion when it was last classified as a background image portion are used to correct the current mismatch portion. Therefore, the mismatched parts of the current frame can be corrected using the latest image that was actually seen in the distance, improving the accuracy of correcting the mismatched parts and reducing the sense of incongruity caused by the correction.

[0122] In addition, in the image processing device of the embodiment, the inconsistency correction unit generates a last value retention image by performing a process over multiple frames in which, for pixels classified as distant image portions, the pixel values ​​of the pixels are retained, and for pixels classified as nearby image portions, the existing retained values ​​are maintained. This allows the latest value holding image to be generated appropriately.

[0123] Furthermore, in the image processing apparatus according to the embodiment, the mismatch correction unit interpolates pixel values ​​for a portion of the current mismatch portion where no overlapping portion exists, using the image of the overlapping portion. This makes it possible to appropriately interpolate pixels outside the overlapping portion using the image of the overlapping portion, even if the overlapping portion with respect to the current mismatched portion is only partial. Therefore, the accuracy of correcting the mismatched portion can be improved, and the sense of incongruity caused by the correction can be alleviated.

[0124] Furthermore, in the image processing device of the embodiment, the near / far division section (same 15B) changes the size of the distance range divided as the near image portion so as to have a negative correlation with the distance between the display surface on which the input image is displayed and the user viewing the displayed image on the display surface. When the distance between the display surface and the user is short, the subject in the image appears larger to the user, and the range of distances in the foreground that the user finds uncomfortable also becomes wider. Therefore, as described above, the size of the distance range classified as the foreground image portion is changed so that it has a negative correlation with the distance between the display surface and the user; in other words, the shorter the distance, the larger the distance range classified as the foreground image portion. This allows the range of the foreground image portion to be corrected to alleviate the sense of discomfort to be appropriately divided even if the distance between the display surface and the user changes. In other words, appropriate corrections can be made to alleviate the sense of discomfort according to the distance between the display surface and the user.

[0125] In the image processing device of the embodiment, the near / far division unit (15C) changes the size of the distance range divided as the near image portion so that it has a positive correlation with the size of the display area of ​​the input image. When the display area for displaying the input image is large, the subject in the image appears larger to the user, and the distance range of the foreground that the user finds uncomfortable also becomes wider. Therefore, as described above, the size of the distance range classified as the foreground image portion is changed so that it has a positive correlation with the size of the display area; in other words, the larger the display area, the larger the distance range classified as the foreground image portion. This allows the range of the foreground image portion to be corrected to alleviate the sense of incongruity to be appropriately divided even if the size of the display area changes. In other words, appropriate corrections can be made to alleviate the sense of incongruity according to the size of the display area.

[0126] The image processing method of the embodiment is an image processing method that divides an input image into a close-up image portion, which is a portion capturing a subject in the foreground, and a distant image portion, which is a portion capturing a subject that is further away than the close-up image portion, corrects the close-up image portion, and corrects inconsistencies that arise from the correction to the close-up image portion. According to this image processing method, it is possible to obtain the same functions and effects as those of the image processing device according to the above embodiment.

[0127] In addition, the projector device of the embodiment (same as 1, 1B, 1C) is equipped with a near-distance division section that divides an input image into a near-distance image section that is a section capturing a near subject and a far-distance image section that is a section capturing a subject that is farther away than the near-distance image section, a near-distance correction section that makes corrections to the near-distance image section, an inconsistency correction section that makes corrections to inconsistencies that arise from the correction to the near-distance image section by the near-distance correction section, and a projection section (same as 12) that generates a reproduced image based on the image of the inconsistency section corrected by the inconsistency correction section and projects the reproduced image onto a projection surface. Such a projector device can also provide the same functions and effects as the image processing device according to the above-described embodiment.

[0128] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0129] <6. This technology> The present technology can also be configured as follows. (1) a near-distance dividing section for dividing the input image into a near image portion which is a portion capturing a near subject and a far image portion which is a portion capturing a far subject relative to the near image portion; a foreground correction unit that performs correction on the foreground image portion; an inconsistency correction unit that corrects an inconsistency portion that occurs due to the correction of the foreground image portion by the foreground correction unit; Image processing device. (2) The near and far section is The near view image portion and the far view image portion are separated based on distance measurement information of the subject in the input image. The image processing device according to (1) above. (3) The near and far section is The foreground image portion and the background image portion are separated based on the difference in distance between the objects in the input image. The image processing device according to (2) above. (4) The near and far section is The foreground image portion and the background image portion are separated based on the image analysis result of the input image. The image processing device according to any one of (1) to (3). (5) The near and far section is The input image is divided into the foreground image portion and the background image portion based on the result of object recognition processing performed on the input image. The image processing device according to (4) above. (6) The near and far section is The object recognition process is performed using an image recognizer based on artificial intelligence. The image processing device according to (5) above. (7) The near and far section is Image analysis is performed using template matching to separate the foreground image portion from the background image portion. The image processing device according to (4) above. (8) The foreground correction unit As a correction to the foreground image portion, at least a part of the foreground image portion is made transparent or reduced. The image processing device according to any one of (1) to (7). (9) The mismatch correction unit As a correction for the current mismatch portion, which is the mismatch portion in the current frame, a correction is performed using an image of a portion of the past frame that overlaps with the current mismatch portion in the distant image portion. The image processing device according to any one of (1) to (8). (10) The mismatch correction unit A last value holding image is generated, which is an image that holds the pixel value when the input image was last divided into a distant image portion within at least a period of time prior to the current frame, for each pixel of the input image, and the current mismatch portion is corrected using the last value holding image. The image processing device according to (9) above. (11) The mismatch correction unit For the input image, the pixel values ​​of the pixels classified as distant image portions are retained, and for the pixels classified as near image portions, the existing retained values ​​are maintained, and by performing this process over multiple frames, the latest value retained image is generated. The image processing device according to (10) above. (12) The mismatch correction unit For a portion of the current mismatched portion where the overlapping portion does not exist, pixel values ​​are interpolated using the image of the overlapping portion. The image processing device according to any one of (9) to (11) above. (13) The near and far section is The size of the distance range divided as the near-distance image portion is changed so as to have a negative correlation with the distance between a display surface on which the input image is displayed and a user viewing the display image on the display surface. The image processing device according to any one of (1) to (12). (14) The near and far section is The size of the distance range divided into the near view image portion is changed so as to have a positive correlation with the size of the display area of ​​the input image. The image processing device according to any one of (1) to (13). (15) The input image is divided into a foreground image portion that is a portion capturing a foreground subject and a background image portion that is a portion capturing a subject that is further away than the foreground image portion; Correcting the foreground image portion; Correcting the inconsistent portion caused by the correction to the foreground image portion Image processing methods. (16) a near-distance dividing section for dividing the input image into a near image portion which is a portion capturing a near subject and a far image portion which is a portion capturing a far subject relative to the near image portion; a foreground correction unit that performs correction on the foreground image portion; an inconsistency correction unit that corrects an inconsistency portion caused by the correction of the foreground image portion by the foreground correction unit; a projection unit that generates a reconstructed image based on the image of the mismatch portion corrected by the mismatch correction unit and projects the reconstructed image onto a projection surface. Projector equipment. [Explanation of symbols]

[0130] 100 Image Display System 1, 1B, 1C Projector equipment 2. Imaging device 5,6 Ranging device 10, 10', 10A, 10B, 10C Image processing unit 12 Projection section 12a Spatial Light Modulator 13 Control Unit 13a Display area control section 15,15',15A,15B,15C Near-far section 16,16' Foreground correction section 16a Transparency processing section 16b Downsizing processing section 17,17' Mismatch correction section 17a Last value holding image generation unit 17b Mismatched pixel correction unit 17c Blur processing section 18 Synthesis section 21 Frame Memory 22 Write control section 23 Readout control unit S projection surface

Claims

1. a near-distance dividing section for dividing the input image into a near image portion which is a portion capturing a near subject and a far image portion which is a portion capturing a far subject relative to the near image portion; a foreground correction unit that performs correction on the foreground image portion; an inconsistency correction unit that corrects an inconsistency portion caused by the correction of the foreground image portion by the foreground correction unit, The mismatch correction unit When performing correction of the current mismatch portion, which is the mismatch portion in the current frame, using an image of an overlapping portion with the current mismatch portion in the distant image portion of the past frame, A last value holding image is generated, which is an image that holds the pixel value when the input image was last divided into a distant image portion within at least a period of time prior to the current frame, for each pixel of the input image, and the current mismatch portion is corrected using the last value holding image. Image processing device.

2. The mismatch correction unit For the input image, the pixel values ​​of the pixels classified as distant image portions are retained, and for the pixels classified as near image portions, the existing retained values ​​are maintained, and by performing this process over multiple frames, the latest value retained image is generated. The image processing device according to claim 1 .

3. The near and far section is The near view image portion and the far view image portion are separated based on distance measurement information of the subject in the input image. The image processing device according to claim 1 .

4. The near and far section is The foreground image portion and the background image portion are separated based on the difference in distance between the objects in the input image. The image processing device according to claim 3 .

5. The near and far section is The foreground image portion and the background image portion are separated based on the image analysis result of the input image. The image processing device according to claim 1 .

6. The near and far section is The input image is divided into the foreground image portion and the background image portion based on the result of object recognition processing performed on the input image. The image processing device according to claim 5 .

7. The near and far section is The object recognition process is performed using an image recognizer based on artificial intelligence. The image processing device according to claim 6 .

8. The near and far section is Image analysis is performed using template matching to separate the foreground image portion from the background image portion. The image processing device according to claim 5 .

9. The foreground correction unit As a correction to the foreground image portion, at least a part of the foreground image portion is made transparent or reduced. The image processing device according to claim 1 .

10. The input image is divided into a foreground image portion that is a portion capturing a foreground subject and a background image portion that is a portion capturing a subject that is further away than the foreground image portion; Correcting the foreground image portion; Correcting an inconsistency caused by the correction to the foreground image portion, and When performing correction of the current mismatch portion, which is the mismatch portion in the current frame, using an image of an overlapping portion with the current mismatch portion in the distant image portion of the past frame, A last value holding image is generated, which is an image that holds the pixel value when the input image was last divided into a distant image portion within at least a period of time prior to the current frame, for each pixel of the input image, and the current mismatch portion is corrected using the last value holding image. Image processing methods.

11. A near-distance dividing section for dividing an input image into a near-distance image portion which is a portion capturing a near subject and a far-distance image portion which is a portion capturing a far-distance subject relative to the near-distance image portion; a foreground correction unit that performs correction on the foreground image portion; an inconsistency correction unit that corrects an inconsistency portion caused by the correction of the foreground image portion by the foreground correction unit, The mismatch correction unit As a correction for the current mismatch portion, which is the mismatch portion in the current frame, a correction is performed using an image of a portion that overlaps with the current mismatch portion in the distant image portion of the past frame, and For a portion of the current mismatched portion where the overlapping portion does not exist, pixel values ​​are interpolated using the image of the overlapping portion. Image processing device.

12. An input image is divided into a foreground image portion that is a portion capturing a foreground subject and a background image portion that is a portion capturing a subject that is further away than the foreground image portion, Correcting the foreground image portion; correcting an inconsistency caused by the correction to the foreground image portion; In correcting the mismatched portion, As a correction for the current mismatch portion, which is the mismatch portion in the current frame, a correction is performed using an image of a portion that overlaps with the current mismatch portion in the distant image portion of the past frame, and For a portion of the current mismatched portion where the overlapping portion does not exist, pixel values ​​are interpolated using the image of the overlapping portion. Image processing methods.

13. A near-distance dividing section for dividing an input image into a near-distance image portion which is a portion capturing a near subject and a far-distance image portion which is a portion capturing a far-distance subject relative to the near-distance image portion; a foreground correction unit that performs correction on the foreground image portion; an inconsistency correction unit that corrects an inconsistency portion caused by the correction of the foreground image portion by the foreground correction unit, The near and far section is The size of the distance range divided as the near-distance image portion is changed so as to have a negative correlation with the distance between a display surface on which the input image is displayed and a user viewing the display image on the display surface. Image processing device.

14. An input image is divided into a foreground image portion that is a portion capturing a foreground subject and a background image portion that is a portion capturing a background subject that is further away than the foreground image portion, Correcting the foreground image portion; Correcting an inconsistency caused by the correction to the foreground image portion, and When dividing the foreground image portion and the background image portion, The size of the distance range divided as the near-distance image portion is changed so as to have a negative correlation with the distance between a display surface on which the input image is displayed and a user viewing the display image on the display surface. Image processing methods.

15. A near-distance dividing section for dividing an input image into a near-distance image portion which is a portion capturing a near subject and a far-distance image portion which is a portion capturing a far-distance subject relative to the near-distance image portion; a foreground correction unit that performs correction on the foreground image portion; an inconsistency correction unit that corrects an inconsistency portion caused by the correction of the foreground image portion by the foreground correction unit, The near and far section is The size of the distance range divided into the near view image portion is changed so as to have a positive correlation with the size of the display area of ​​the input image. Image processing device.

16. An input image is divided into a foreground image portion that is a portion capturing a foreground subject and a background image portion that is a portion capturing a subject that is further away than the foreground image portion, Correcting the foreground image portion; Correcting an inconsistency caused by the correction to the foreground image portion, and When dividing the foreground image portion and the background image portion, The size of the distance range divided into the near view image portion is changed so as to have a positive correlation with the size of the display area of ​​the input image. Image processing methods.

Images