Information processing device

Abstract

To provide a technique that enables processing based on a feature amount of a real space image (for example, exposure adjustment processing of a real space image) to be carried out appropriately under various situations (for example, a situation where the real space image is an image of a backlight scene).SOLUTION: An information processing device includes first acquisition means for acquiring an image of a real space, second acquisition means for acquiring information on a composite area which is an area to be combined with graphics in the image of the real space, and third acquisition means for acquiring a feature amount from the image of the real space. Based on the information, the third acquisition means acquires, from the image of the real space, the feature amount for which the influence of the composite area is reduced.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device. [Background technology]

[0002] A technology for adjusting the exposure of a real space image (real space image) has been proposed as a technology related to a mixed reality (MR) technology using an HMD (Head Mounted Display). Patent Document 1 discloses a technology for uniformly correcting the exposure (image processing) of a real space image and correcting the exposure of a computer graphic (CG) to be synthesized (superimposed) on the real space image. Patent Document 2 discloses a technology for extracting an area of ​​a main subject from a real space image and synthesizing it with a background image (another real space image) prepared in advance. Patent Document 2 also discloses a technology for correcting the exposure so that the exposure of the main subject and the exposure of the background image match. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2005-135052 A [Patent Document 2] JP 2005-79776 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, under certain circumstances, processing based on the feature amount of the real space image (for example, exposure adjustment of the real space image) cannot be suitably performed. In a backlit scene, a dark real space image is obtained overall, and when a CG with appropriate brightness is combined with the dark real space image overall, an unnatural composite image is obtained. In the technology disclosed in Patent Document 1, the exposure correction of the real space image and the exposure correction of the CG are performed uniformly, so the composite image after the exposure correction is also unnatural. In addition, blackout may occur in a backlit scene, and the exposure of the blackout area cannot be suitably corrected by image processing (exposure correction). Therefore, even with the technology disclosed in Patent Document 2, an incomplete composite image is obtained. Furthermore, if the area of ​​the light source that causes the backlight is hidden by the CG, the user cannot understand why the background image (real space image) is dark, and is likely to feel that the composite image is unnatural.

[0005] The present invention aims to provide a technology that enables processing based on the features of a real-space image (e.g., exposure adjustment of a real-space image) to be performed suitably under various circumstances (e.g., a situation in which the real-space image is an image of a backlit scene). [Means for solving the problem]

[0006] A first aspect of the present invention is an information processing device comprising a first acquisition means for acquiring an image of real space, a second acquisition means for acquiring information relating to a synthesis area in the image of real space where a graphic is synthesized, and a third acquisition means for acquiring a feature from the image of real space, wherein the third acquisition means acquires the feature from the image of real space based on the information, with the influence of the synthesis area reduced.

[0007] A second aspect of the present invention is a head-mounted display device comprising the above-mentioned information processing device and a display means for displaying a composite image in which the graphic is synthesized with an image of the real space.

[0008] A third aspect of the present invention provides a method for obtaining an image of a real space, The information processing method includes a step of acquiring information about a synthesis area, which is an area where graphics are synthesized, and a step of acquiring features from the image in real space, and is characterized in that the feature values ​​in which the influence of the synthesis area is reduced are acquired from the image in real space based on the information.

[0009] A fourth aspect of the present invention is a program for causing a computer to function as each of the means of the image processing device described above. Effect of the Invention

[0010] According to the present invention, it becomes possible to suitably perform processing based on the features of a real space image (e.g., exposure adjustment of a real space image) under various circumstances (e.g., a situation in which the real space image is an image of a backlit scene). [Brief description of the drawings]

[0011] [Figure 1] FIG. 2 is a block diagram showing the configuration of an HMD. [Diagram 2] FIG. 13 is a schematic diagram showing how a composite image is generated. [Diagram 3] 4 is a schematic diagram showing the operation according to the first embodiment. FIG. [Figure 4] 4 is a flowchart showing an operation according to the first embodiment. [Diagram 5] 10 is a flowchart showing an operation according to the second embodiment. [Figure 6] 11 is a flowchart showing an operation according to the third embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] (Embodiment 1) Hereinafter, a first embodiment of the present invention will be described. Fig. 1 is a block diagram showing the configuration of an HMD (Head Mounted Display) 100 which is a head mounted display device according to the first embodiment.

[0013] The lens 101, the timing generator 102, the image sensor 103, the analog signal processing unit 104, the camera control unit 105, and the digital gain unit 106 are used to obtain a real space image (image of real space) for the left eye. The lens 101', the timing generator 102', the image sensor 103', the analog signal processing unit 104', the camera control unit 105', and the digital gain unit 106' are used to obtain a real space image for the right eye. The image display unit 107 is used to display an image for the right eye, and the image display unit 107' is used to display an image for the left eye. The lens 101' has a similar function to the lens 101. The timing generator 102' has a similar function to the timing generator 102. The image sensor 103' has a similar function to the image sensor 103. The analog signal processing unit 104' has a similar function to the analog signal processing unit 104. The camera control unit 105' has a similar function to the camera control unit 105. The digital gain unit 106' has a similar function to the digital gain unit 106. The image display unit 107 ′ has the same function as the image display unit 107 .

[0014] The lens 101 guides light (reflected light) from a subject to the image sensor 103. The image sensor 103 is, for example, a CMOS sensor or a CCD sensor, and captures an image of real space. The image sensor 103 obtains an analog signal by performing photoelectric conversion of the incident light (optical image of the subject). The timing generator 102 outputs a timing signal to the image sensor 103. The image sensor 103 is driven based on the timing signal. The camera control unit 105 controls the time (accumulation time) that the image sensor 103 accumulates light, and controls the gain value used by the digital gain unit 106. The analog signal processing unit 104 obtains a digital signal by performing sampling (sample and hold), gain adjustment, and A / D conversion of the analog signal obtained by the image sensor 103. The digital gain unit 106 is an analog signal processing unit. The digital signal obtained by 104 is subjected to a predetermined signal processing (for example, gain adjustment), and the processed digital signal (real space image) is output to an image processing unit 110.

[0015] The image processing unit 110 acquires real space images from the analog signal processing units 104 and 104' and performs various processes using the real space images. For example, the image processing unit 110 performs a process of synthesizing (superimposing) graphics (computer graphics: CG) on the real space images. The image processing unit 110 may perform image processing to cancel the aberration of the lens 101 and the aberration of the optical system of the image display unit 107. The image processing unit 110 may perform distance measurement to acquire distance information of the real space in which the user is located (the real space in which the HMD 100 is located). For distance measurement, the image sensor 103 and the image sensor 103' may be used, or another stereo camera may be used. The image processing unit 110 may acquire posture information of the user (HMD 100) from a posture sensor (not shown). The image processing unit 110 may change the position, direction, and size of the CG using the distance information and posture information. By doing so, an image in which the CG is synthesized with the image of the real space (synthetic image) can be generated as if the CG object exists in the real space. CG (CG information (data)) is stored in the content DB 111, and it is possible to switch the CG to be read out.

[0016] The image processing unit 110 stores the composite image in the memory 113 via the memory control unit 112, or outputs the composite image to the image display unit 107, 107'. For example, the image processing unit 110 acquires a real space image for the left eye from the analog signal processing unit 104, and generates a composite image for the left eye by synthesizing the real space image for the left eye with CG. Then, the image processing unit 110 outputs the composite image for the left eye to the image display unit 107, and the image display unit 107 displays the image for the left eye. Similarly, the image processing unit 110 acquires a real space image for the right eye from the analog signal processing unit 104', and generates a composite image for the right eye by synthesizing the real space image for the right eye with CG. Then, the image processing unit 110 outputs the composite image for the right eye to the image display unit 107', and the image display unit 107' displays the image for the right eye.

[0017] Fig. 2 is a schematic diagram showing how a composite image is generated. In Fig. 2, a composite image 203 is generated by combining a real space image 201 including a sun area with a CG image 202 including a CG building area.

[0018] FIG. 3 is a schematic diagram showing the operation according to the first embodiment, and FIG. 4 is a flowchart showing the operation according to the first embodiment.

[0019] In S401, the image processing unit 110 sets a plurality of partial regions for a CG image. For example, a plurality of partial regions 302 are set for a CG image 301. Note that the plurality of partial regions may or may not be a plurality of divided regions constituting the entire region of the CG image (or the real space image). The plurality of partial regions may include a partial region separated from another partial region, or may include a partial region at least a part of which is included in another partial region.

[0020] In S402, the image processing unit 110 sets information on whether each of the multiple partial regions of the CG image is a CG region (CG region). For example, a weight 303 is set for each of the multiple partial regions. A weight 303 of "0" is set for a CG region (a partial region that is a CG region), and a weight 303 of "1" is set for a non-CG region (a partial region that is not a CG region). A partial region that is partly a CG region may or may not be treated as a CG region.

[0021] In S403, the image processing unit 110 sets a plurality of partial regions in the real space image, similarly to S401. For example, a plurality of partial regions 305 (= a plurality of partial regions 302) are set in the real space image 304.

[0022] In S404, the image processing unit 110 sets information on whether each of the partial regions of the real space image is an area where CG is to be synthesized (synthesized area). For example, a plurality of weights 303 are mapped to the partial regions 305. A partial region in the real space image with a weight of "0" is a synthesis area.

[0023] By the processing of S401 to S404, information (synthesis area information) related to the synthesis area (area where CG is synthesized) in the real space image is acquired. Note that the method of acquiring the synthesis area information is not particularly limited. The synthesis area information is not limited to weights. For example, the synthesis area information may be the coordinates of multiple vertices of the synthesis area, or multiple coordinates on the outline of the synthesis area.

[0024] In S405, the image processing unit 110 acquires a feature amount from the real space image. In the first embodiment, the image processing unit 110 acquires a feature amount in which the influence of the synthesis area is reduced from the real space image based on the synthesis area information. For example, the image processing unit 110 acquires a plurality of feature amounts (a plurality of partial feature amounts) corresponding to a plurality of partial areas in the real space image. Then, the image processing unit 110 acquires a feature amount (a weighted average of a plurality of partial feature amounts) of the real space image by performing weighted synthesis of the plurality of partial feature amounts using the weights set in S404. For example, the feature amount is a feature amount related to brightness, and the partial feature amount is an average luminance value of the partial areas. In this case, the feature amount of the real space image may be regarded as a photometric calculation result. In FIG. 3, since the weight of the synthesis area is "0", the synthesis area is excluded from the real space image, and the feature amount of the real space image is acquired.

[0025] In S406, the image processing unit 110 uses the camera control units 105 and 105' to control the imaging of the real space based on the feature amount (feature amount of the real space image) acquired in S405. For example, the image processing unit 110 performs exposure adjustment (exposure control) of the real space image by controlling the accumulation time of the image sensor 103 or the gain value of the digital gain unit 106.

[0026] As described above, according to the first embodiment, synthesis area information is acquired, and a feature amount in which the influence of the synthesis area is reduced is acquired from the real space image based on the synthesis area information. This makes it possible to suitably perform processing based on the feature amount of the real space image (for example, exposure adjustment of the real space image) under various circumstances (for example, a situation in which the real space image is an image of a backlit scene). For example, a feature amount in which the influence of the sun area (high luminance area) is reduced can be obtained from the real space image 201 in FIG. 2. As a result, the exposure adjustment of the real space image can be performed so as to obtain a synthesis image 203 that matches the brightness of the surroundings of the CG (building) and the brightness of the CG.

[0027] Although an example has been described in which the synthesis area is excluded when acquiring the feature amount of the real space image, the synthesis area does not have to be excluded. Although an example has been described in which "0" is used as the weight of the partial feature amount corresponding to the synthesis area, a different weight lower than the weight of the partial feature amount not corresponding to the synthesis area may be used.

[0028] Also, although an example of performing exposure adjustment based on the feature amount of the real space image has been described, the present invention is not limited to this. For example, the gradation characteristics of the real space image may be adjusted based on the feature amount of the real space image. Gamma correction or dynamic range adjustment may be performed based on the feature amount of the real space image. Although an example of controlling imaging of the real space based on the feature amount of the real space image has been described, the present invention is not limited to this. For example, image processing may be performed based on the feature amount of the real space image. Image processing (exposure correction) for adjusting (correcting) the exposure of the real space image may be performed.

[0029] (Embodiment 2) Hereinafter, a description will be given of the second embodiment of the present invention. Note that in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0030] FIG. 5 is a flowchart showing the operation according to the second embodiment.

[0031] In S501, the image processing unit 110 determines whether to perform exposure control for still image shooting (to acquire feature amounts) or for live view display. This determination may be interpreted as a determination of whether to record a composite image as a still image in the memory 113 or to display the composite image in real time on the image display units 107, 107'. This determination may be interpreted as a determination of whether to perform exposure control for a composite image recorded as a still image in the memory 113 (to acquire feature amounts) or to perform exposure control for a composite image displayed in real time on the image display units 107, 107'. If exposure control for still image shooting is to be performed, the process proceeds to S402, and if exposure control for live view display is to be performed, the process proceeds to S502.

[0032] As described in the first embodiment, in S402, the image processing unit 110 sets a weight of "0" for CG regions (partial regions that are CG regions) and a weight of "1" for non-CG regions (partial regions that are not CG regions).

[0033] In S502, the image processing unit 110 sets a weight of "0.5" for the CG region, and a weight of "1" for the non-CG region. Note that the weight set for the CG region in S502 may be another weight that is greater than "0" and less than "1".

[0034] As described above, according to the second embodiment, when a composite image is displayed on the image display units 107 and 107' in real time, a weight for the CG region (composite region) is set to a larger weight than when the composite image is recorded in the memory 113 as a still image. The inverse of the weight set for the CG region (composite region) may be interpreted as a reduction degree of the influence of the composite region on the feature amount. Therefore, in the second embodiment, when a composite image is displayed on the image display units 107 and 107' in real time, the influence of the composite region on the feature amount is reduced to a lower degree than when the composite image is recorded in the memory 113 as a still image. This makes it possible to reduce flicker (change in image quality (e.g., exposure)) caused by a slight change in composition, such as a change in composition caused by shaking of the HMD 100 (shaking of the head) during live view display.

[0035] (Embodiment 3) Hereinafter, a third embodiment of the present invention will be described. Note that in the following, a description of the same points as in the first embodiment (for example, the same configuration and processing as in the first embodiment) will be omitted, and only points different from the first embodiment will be described.

[0036] FIG. 6 is a flowchart showing the operation according to the third embodiment.

[0037] In S601, the image processing unit 110 predicts the movement of the CG based on the output signal of an acceleration sensor (not shown) of the HMD 100. Then, based on the result of predicting the movement of the CG, the image processing unit 110 updates the weight of the partial area, which is the area where the CG moves, to "0.5". The movement area is an area that becomes a CG area (synthetic area) either before or after the CG movement. For example, the movement area is an area that changes from a non-CG area to a CG area due to the movement of the CG, an area that changes from a CG area to a non-CG area due to the movement of the CG, or both. The weight of an area that becomes a CG area (synthetic area) both before and after the movement of the CG is "0", and The weight of the area that is a non-CG area both before and after the moving object is "1." Note that the weight set for the moving object may be a different weight that is greater than "0" and less than "1."

[0038] As described above, according to the third embodiment, a weight is set for a moving area that is greater than the weight of an area that becomes a CG area (synthetic area) both before and after the CG movement and smaller than the weight of an area that becomes a non-CG area both before and after the CG movement. The inverse of the weight set for an area may be interpreted as the degree of reduction in the influence of the area on the feature amount. Therefore, in the third embodiment, a feature amount is obtained in which the influence of an area that becomes a CG area (synthetic area) both before and after the CG movement and the influence of an area that becomes a CG area either before or after the CG movement are reduced. The degree of reduction in the influence of an area that becomes a CG area either before or after the CG movement is lower than the degree of reduction in the influence of an area that becomes a CG area (synthetic area) both before and after the CG movement. This makes it possible to reduce flicker (change in image quality (e.g., exposure)) caused by a slight change in composition even when the timing of capturing the image in real space and the timing of compositing the CG are not synchronized.

[0039] The above-described embodiment (including the modified examples) is merely an example, and the present invention also includes configurations obtained by appropriately modifying or changing the configurations of the above-described embodiment within the scope of the gist of the present invention. The present invention also includes configurations obtained by appropriately combining the configurations of the above-described embodiment.

[0040] For example, although an example in which the present invention is applied to an HMD has been described, the present invention can also be applied to devices other than HMDs. The present invention can be applied to various information processing devices such as normal cameras, smartphones, tablet terminals, personal computers, and servers. A part of the above-mentioned multiple processes may be performed by an external device. For example, at least a part of the imaging of real space, CG synthesis, image display, exposure adjustment (exposure control, exposure correction), and gamma correction may be performed by an external device. The device to which the present invention is applied may acquire a real space image from the outside, acquire feature amounts from the real space image, and output the feature amounts to the outside.

[0041] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0042] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A first acquisition means for acquiring an image of a real space; A second acquisition means for acquiring information regarding a synthesis area, which is an area in the image in real space where a graphic is synthesized; a third acquisition means for acquiring a feature amount from the image in real space; having The third acquisition means acquires the feature amount in which the influence of the synthesis area is reduced from the image in the real space based on the information. 23. An information processing apparatus comprising: (Configuration 2) The imaging device further includes a control unit for controlling imaging of the real space based on the feature amount. 2. The information processing device according to configuration 1. (Configuration 3) The third acquisition means is acquiring a plurality of partial feature amounts corresponding to a plurality of partial regions in the image in real space, The weight of the partial feature corresponding to the synthesis region is lower than the weight of the partial feature not corresponding to the synthesis region, and the feature is obtained by performing weighted synthesis of the plurality of partial feature amounts. 3. The information processing device according to configuration 1 or 2. (Configuration 4) The third acquisition means acquires the feature amount by excluding the synthesis area from the image in the real space. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The third acquisition means acquires the feature amount in which an influence of the synthesis area is reduced without excluding the synthesis area from the image in the real space. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 6) The third acquisition means is When a composite image, which is an image obtained by combining the image in real space with the graphic, is recorded as a still image, a feature amount in which an influence of the composite area is reduced by a first reduction degree is acquired as the feature amount, When the composite image is displayed in real time, a feature amount in which the influence of the composite area is reduced by a second reduction degree that is lower than the first reduction degree is acquired as the feature amount. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) A prediction means for predicting a movement of the graphic is further provided, The third acquisition means acquires, based on the information on the synthesis area and a prediction result of the movement of the graphic, the feature amount in which an influence of the area to become the synthesis area is reduced by a third reduction degree both before and after the movement of the graphic, and the influence of the area to become the synthesis area is reduced by a fourth reduction degree lower than the third reduction degree either before or after the movement of the graphic. 7. The information processing device according to any one of configurations 1 to 6. (Configuration 8) The feature amount is used for adjusting the exposure of the image in the real space. 8. The information processing device according to any one of configurations 1 to 7. (Configuration 9) The feature amount is used to adjust the gradation characteristics of the image in the real space. 9. The information processing device according to any one of configurations 1 to 8. (Configuration 10) The present invention further includes an imaging means for imaging the real space. 10. The information processing device according to any one of configurations 1 to 9. (Configuration 11) The image processing device further includes a synthesizing unit for synthesizing the graphic with the image in the real space. 11. The information processing device according to any one of configurations 1 to 10. (Configuration 12) An information processing device according to any one of configurations 1 to 11, a display means for displaying a composite image obtained by combining the image in real space with the graphic; A head-mounted display device comprising: (method) acquiring an image of a real space; acquiring information about a synthesis area, which is an area in the image in real space where a graphic is synthesized; acquiring a feature amount from the image in real space; having The feature amount in which the influence of the synthesis area is reduced is acquired from the image in the real space based on the information. 23. An information processing method comprising: (program) 12. A program for causing a computer to function as each of the means of the image processing device according to any one of configurations 1 to 11. [Explanation of symbols]

[0043] 100: HMD 110: Image processing unit

Claims

1. A first acquisition means for acquiring an image of a real space; a second acquisition means for acquiring information regarding a synthesis area, which is an area in the image in real space where a graphic is synthesized; a third acquisition means for acquiring a feature amount from the image in real space; having The third acquisition means acquires the feature amount in which an influence of the synthesis area is reduced from the image in the real space based on the information.

23. An information processing apparatus comprising:

2. The imaging device further includes a control unit for controlling imaging of the real space based on the feature amount.

2. The information processing apparatus according to claim 1,

3. The third acquisition means is acquiring a plurality of partial feature amounts corresponding to a plurality of partial regions in the image in real space, The weight of the partial feature corresponding to the synthesis region is lower than the weight of the partial feature not corresponding to the synthesis region, and the feature is obtained by performing weighted synthesis of the plurality of partial feature amounts.

2. The information processing apparatus according to claim 1,

4. The third acquisition means acquires the feature amount by excluding the synthesis area from the image in the real space.

2. The information processing apparatus according to claim 1,

5. The third acquisition means acquires the feature amount in which an influence of the synthesis area is reduced without excluding the synthesis area from the image in the real space.

2. The information processing apparatus according to claim 1,

6. The third acquisition means is When a composite image, which is an image obtained by combining the image in the real space with the graphic, is recorded as a still image, a feature amount in which an influence of the composite area is reduced by a first reduction degree is acquired as the feature amount, When the composite image is displayed in real time, a feature amount in which the influence of the composite area is reduced by a second reduction degree that is lower than the first reduction degree is acquired as the feature amount.

2. The information processing apparatus according to claim 1,

7. A prediction means for predicting a movement of the graphic is further provided, The third acquisition means acquires, based on the information on the combination area and a prediction result of the movement of the graphic, the feature amount in which an influence of the area to become the combination area is reduced by a third reduction degree both before and after the movement of the graphic, and the influence of the area to become the combination area is reduced by a fourth reduction degree lower than the third reduction degree either before or after the movement of the graphic.

2. The information processing apparatus according to claim 1,

8. The feature amount is used for adjusting the exposure of the image in the real space.

2. The information processing apparatus according to claim 1,

9. The feature amount is used to adjust the gradation characteristics of the image in the real space.

2. The information processing apparatus according to claim 1,

10. The present invention further includes an imaging means for imaging the real space.

2. The information processing apparatus according to claim 1,

11. The image processing device further includes a synthesizing unit for synthesizing the graphic with the image in the real space.

2. The information processing apparatus according to claim 1,

12. An information processing device according to any one of claims 1 to 11, a display means for displaying a composite image obtained by combining the image in real space with the graphic; A head-mounted display device comprising:

13. acquiring an image of a real space; acquiring information about a synthesis area, which is an area in the image in real space where a graphic is synthesized; acquiring a feature amount from the image in real space; having The feature amount in which the influence of the synthesis area is reduced is acquired from the image in the real space based on the information.

23. An information processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 11.

Images