Information processor and information processing method

Abstract

To provide a technique for suppressing an increase in a calculation amount in an information processor for compositing an image of real space and an image of virtual space.SOLUTION: An information processor includes: acquisition means that acquires a real image obtained by imaging real space; generation means that generates a virtual image which is an image of virtual space aligned with the real space; image processing means that can perform image processing on the real image; and composition means that generates a composite image obtained by compositing the real image and the virtual image. The image processing means changes the contents of the image processing to the real image according to the result of comparing the proportion of the virtual image to the composite image with a threshold.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an information processing device and an information processing method. [Background technology]

[0002] In recent years, mixed reality (MVR), which seamlessly combines real and virtual spaces, has become popular. Research into mixed reality (MR) technology is currently active. Among MR technologies, Augmented Reality (AR), which presents a virtual space overlaid on the real space, is attracting particular attention. AR technology can be applied to, for example, a head-mounted display (HMD) worn by the user.

[0003] AR image presentation is mainly realized by a video see-through method or an optical see-through method. In the case of the video see-through method, a virtual space generated by aligning an image of real space captured by an imaging device such as a video camera according to the position and orientation of the imaging device is synthesized and displayed. The image of the virtual space is specifically composed of virtual objects and text information drawn by computer graphics (CG). In the case of the optical see-through method, an image of the virtual space generated according to the position and orientation of the display device is displayed on a transparent display. This allows the real space and the virtual space to be synthesized on the observer's retina.

[0004] One of the most important issues in AR technology is how to perform accurate alignment between real space and virtual space in real time. When using a video see-through method, the problem of alignment in AR is a problem of determining the position and orientation of an imaging device in a scene (i.e., in a reference coordinate system defined in the scene). A representative example of an alignment method in the video see-through method is a method in which an artificial index (marker) whose shape information is known is placed in a scene, and the marker in a captured real image is image-recognized to determine the position and orientation of the imaging device in the reference coordinate system. In this case, the position and orientation of the imaging device in the reference coordinate system are determined from the correspondence between the projection position of the index in the real image and the 3D position and orientation of the index in the reference coordinate system, which is known information.

[0005] In addition, there is a registration method for the video see-through method that uses features that exist in the scene (hereinafter, natural features) rather than artificial markers. Conventionally, this method calculates the position and orientation by equally using all artificial markers or natural features that appear in the image, but this leads to an increase in the amount of calculations.

[0006] Therefore, in order to suppress the increase in the amount of calculation, foveated rendering, which is a type of optimization method for image rendering based on the area where the user is gazing, has been proposed. Foveated rendering is a method in which only the area around the area where the user is gazing in an image is rendered at a standard resolution, and the other areas are rendered at a resolution lower than the standard. Here, human vision has a characteristic that the amount of visual information that can be recognized decreases the farther away from the gaze area. Therefore, even if the resolution of an area far from the gaze area is reduced, the quality of experience is not affected. Foveated rendering is a technology that utilizes this characteristic to optimize both the image quality experienced by the user and the calculation cost of rendering by rendering the user's gaze area at high quality and lowering the rendering quality as it gets farther away from the gaze area.

[0007] In a video see-through HMD, rendering may be performed by an information processing device outside the HMD in order to reduce the size and weight of the HMD body and reduce power consumption. In this case, an image captured by an imaging device mounted on the HMD is transmitted to the information processing device, and the information is After rendering and superimposing the virtual image in the information processing device, the generated composite image is transferred to the HMD again. Here, Patent Document 1 (JP 2018-088604 A) discloses a technology that detects the user's gaze area in the same way as foveated rendering, reduces the resolution of areas away from the gaze area, and then transfers the image to the information processing device. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] JP 2018-088604 A Summary of the Invention [Problem to be solved by the invention]

[0009] In particular, in recent years, there has been an increasing demand for making the HMD portable and expanding the user's range of movement by making the connection between the HMD and an external information processing device wireless. In this case, the HMD is battery-powered, so there is a demand for further reduction in power consumption. However, in Patent Document 1, it is necessary to perform calculation processing on the area the user is gazing at, which consumes power. In addition, regardless of the display form of the virtual image in the composite image, a processing method is adopted in which the resolution is uniformly reduced as the image moves away from the area being gazed at. Therefore, there is a possibility that image processing is performed on the real image that is not necessarily required depending on the display form of the virtual image in the composite image, which may lead to an increase in the amount of calculation and may result in increased power consumption.

[0010] The present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for suppressing an increase in the amount of calculations in an information processing device for synthesizing an image in a real space with an image in a virtual space. [Means for solving the problem]

[0011] The present invention employs the following configuration. An acquisition means for acquiring a real image obtained by capturing an image of a real space; A generating means for generating a virtual image which is an image of a virtual space; an image processing means capable of performing image processing on the real image; a synthesis means for generating a synthetic image by synthesizing the real image and the virtual image; having The image processing means changes the content of image processing for the real image in accordance with a result of comparing a ratio of the virtual image in the composite image with a threshold value. The present invention is an information processing device characterized by the above. The present invention also employs the following configuration. an acquisition step of acquiring a real image obtained by capturing an image of a real space; A generating step of generating a virtual image which is an image of a virtual space; an image processing step of performing image processing on the real image; a synthesis step of generating a synthetic image by synthesizing the real image and the virtual image; having In the image processing step, the content of image processing for the real image is changed according to a result of comparing a ratio of the virtual image in the composite image with a threshold value. The information processing method is characterized in that: Effect of the Invention

[0012] According to the present invention, it is possible to provide a technique for suppressing an increase in the amount of calculation in an information processing device for synthesizing an image in real space with an image in virtual space. [Brief description of the drawings]

[0013] [Figure 1] Block diagram showing an example of the system's functional configuration [Diagram 2] Flowchart of processing performed by the HMD and the external computing device [Diagram 3] Flowchart of processing by the position and orientation calculation unit and CG generation unit [Figure 4] Flowchart of processing performed by the image processing unit [Diagram 5] 11 is a flowchart showing a modified example of the process performed by the image processing unit. [Figure 6] 11 is a flowchart showing a modified example of the process performed by the image processing unit. [Figure 7] FIG. 1 illustrates a real image, a virtual image, a processed image, and a synthetic image. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] A preferred embodiment of the present invention will be described in detail with reference to the drawings. However, unless otherwise specified, the dimensions, materials, shapes, and relative positions of the components described in the embodiment are not intended to limit the scope of the present invention to those. Furthermore, the materials, shapes, and the like of the members once described in the following description are the same as those described in the first description in the following description unless otherwise specified. Well-known or publicly known technologies in the technical field can be applied to configurations and processes not particularly shown or described. Although multiple features are described in the embodiment, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined in any combination. Furthermore, in the attached drawings, the same reference numbers are given to the same or similar configurations, and duplicated descriptions are omitted.

[0015] The present invention is typically applied to an information processing device capable of communicating with an HMD (head mounted display) worn by a user. That is, the information processing device is connected to the HMD wirelessly or by wire for communication, acquires an image captured by an imaging device included in the HMD, performs image processing such as rendering and synthesis processing with a virtual image, and outputs a synthetic image to be displayed on the HMD. However, the present invention is not limited to this configuration, and may be configured such that the HMD itself has an image processing function and performs image processing and synthesis processing. Even in this case, the effect of suppressing an increase in the amount of calculation can be obtained, so that suppression of power consumption can be expected. The present invention can also be regarded as relating to an image processing system including an HMD and an information processing device. The present invention can also be regarded as an information processing method for executing each step using the above-mentioned information processing device or information processing system.

[0016] <Example 1> (Device configuration) FIG. 1 is a block diagram showing an example of a functional configuration of a system 1 of an image processing device according to an embodiment of the present invention. As shown in FIG. 1, the system 1 according to the present embodiment includes an HMD 100 and an external computing device 200 that can communicate with each other. The system 1 as a whole can be considered as an image display device that performs image processing and presents the image to a user. The HMD 100 is a video see-through head-mounted display device. The external computing device 200 is an information processing device that displays an image of a mixed reality space in which an image of a virtual space and an image of a real space are combined on the HMD 100. From the viewpoint of the HMD 100, the external computing device 200 is an external device. On the other hand, when the external computing device 200 is considered as an information processing device that performs image processing according to the present invention, it can be said that the HMD 100 is an external device equipped with an imaging device and a display device. As the external computing device 200, an information processing device or a processing circuit equipped with computing resources such as a processor and a memory can be used.

[0017] First, the HMD 100 will be described. The image input unit 11 is an imaging device (imaging means) that has a lens and an imaging element, and captures moving images in real space corresponding to the eyes of a user who wears the HMD 100 on his or her head. Images of each frame captured by the image input unit 11 (captured images) are sequentially sent to a downstream transmission unit 13. Two or more image input units 11 may be used, such as one for the left eye and one for the right eye. The sensor input unit 12 is a unit that includes a lens, an imaging element, and an image processing circuit. The external computing device 200 has a path for acquiring sensor information (sensor captured image). A position and orientation computing unit 22 of the external computing device 200, which will be described later, detects markers and feature points from the acquired sensor information, and calculates the position and orientation of the image input unit 11 (i.e., the HMD 100) by calculation.

[0018] Furthermore, the information collected by the sensor input unit 12 may be any information that can be used to calculate the position and orientation of the HMD 100, such as information indicating the tilt and direction of the user's head detected by a gyro sensor. The sensor input unit 12 may be implemented as a processing circuit, or may be implemented as a computer program executed by a processor (not shown) included in the HMD 100, and the sensor input unit 12 may perform calculations of the position and orientation.

[0019] The transmission unit 13 transmits the captured image acquired from the image input unit 11 and the sensor information acquired from the sensor input unit 12 to the external computing device 200. The reception unit 14 of the HMD 100 receives the image of the mixed reality space transmitted from the external computing device 200. Data communication between the HMD 100 and the external computing device 200 may be wired communication, wireless communication, or a combination of these. The display unit 15 as a display device is attached to the HMD 100 so as to be located in front of the eyes of a user wearing the HMD 100 on his head, and has an optical system for controlling a display element such as an LCD or EL, and presenting the image of the mixed reality space received by the reception unit 14 on the display element as an enlarged virtual image. Two or more display units 15 may be used, such as one for left eye display and one for right eye display.

[0020] Next, the external calculation device 200 will be described. The external calculation device 200 generally includes a CG generation unit 201, a composite image generation unit 202, a receiving unit 21, and a transmitting unit 26. The CG generation unit 201 generates an image of a virtual space that has been aligned. The composite image generation unit 202 processes an image of a real space, and generates an image of a mixed reality space in which an image of a virtual space is superimposed on an image of a real space. The receiving unit 21 receives information from the HMD 100, and the transmitting unit 26 transmits information to the HMD 100. The external calculation device 200 has two or more processors, and the CG generation unit 201 and the composite image generation unit 202 are operated in parallel by different processors to speed up processing. This makes it possible to shorten the processing time from when the image of the real space and the sensor information are received by the receiving unit 21 until the image of the mixed reality space is transmitted to the HMD 100. In the present embodiment, an example has been given in which the CG generation unit 201 and the composite image generation unit 202 are operated by different processors, but they may be operated within the same processor.

[0021] First, the receiving unit 21 will be described. The receiving unit 21 transfers information transmitted by the transmitting unit 13 of the HMD 100, i.e., the captured image to a downstream image processing unit 24, and sensor information to a downstream position and orientation calculation unit 22. The receiving unit 21 is an acquisition unit that acquires a captured real image.

[0022] Next, the CG generating unit 201 as a generating means will be described. The position and orientation calculation unit 22 uses sensor information to obtain the position and orientation of the image input unit 11 that captured the captured image. For example, the position and orientation calculation unit 22 uses the identification information of the marker included in the sensor information to obtain the arrangement position in real space of the marker that is managed in association with the identification information. Then, the position and orientation calculation unit 22 calculates the position and orientation information of the image input unit 11 based on the obtained arrangement position and the image coordinates of the marker included in the sensor information. Since the technology for calculating the position and orientation of the device that captured the captured image based on the index in the captured image is a well-known technology, detailed description thereof will be omitted.

[0023] The virtual image generating unit 23 generates an image of the virtual space as seen from the viewpoint of the image input unit 11 that has taken the calculated position and orientation. The virtual image generating unit 23 also calculates the display ratio when the image of the virtual space is superimposed on the image of the real space based on the transparency (alpha value) set for each pixel of the generated image of the virtual space, and notifies the image processing unit 24 of the display ratio. Note that if there is a part in the image of the virtual space that is to be transparent, it does not need to be included in the display ratio. For example, If the virtual image occupies 50% of the image and half of the area of ​​the virtual image is a transparent area in the composite image, the display ratio is 25%.

[0024] Next, the composite image generating unit 202 as a composition means will be described. The image processing unit 24 as an image processing means processes the captured image of the real space obtained from the receiving unit 21 based on the display ratio of the image of the virtual space obtained from the virtual image generating unit 23. When the display ratio of the image of the virtual space is greater than the threshold, the image processing is performed after performing a resolution conversion process to lower the resolution of the image of the real space. On the other hand, when the display ratio is equal to or less than the threshold, normal image processing is performed on the image of the real space. Note that when the display ratio of the image of the virtual space is 100%, image processing may not be performed. Also, when the display ratio of the image of the virtual space becomes a value greater than the threshold, the image area of ​​the real space excluding the image of the virtual space may be processed based on the display coordinates of the image of the virtual space. Also, the processing performed by the image processing unit 24 is not limited to conversion of the resolution, and any processing may be performed on the real image, and may include, for example, rendering. Note that in the following description, expressions such as "greater than / smaller than the threshold" are not intended to limit the present invention thereto. For example, the effect of the present invention can be obtained by replacing an expression such as "if it is greater than the threshold, a certain process is performed, and if it is less than the threshold, a different process is performed" with "if it is greater than the threshold, a certain process is performed, and if it is less than the threshold, a different process is performed" or "if it is equal to the threshold, a certain process is performed, and if it is equal to the threshold, either process can be performed."

[0025] The synthesis unit 25 generates a synthetic image by superimposing the image of the virtual space generated by the virtual image generation unit 23 on the image of the real space generated by the image processing unit 24, as an image of the mixed reality space.

[0026] The transmission unit 26 is an output unit that outputs the image of the mixed reality space generated by the synthesis unit 25 to the HMD 100. The transmitted image of the mixed reality space is displayed on the display unit 15 of the HMD 100.

[0027] (Processing flow) Next, the processing flow of the HMD 100 and the external computing device 200 will be described with reference to Fig. 2 to Fig. 6. Fig. 7 shows an example of a real space image, a virtual space image, a processed real space image output from an image processing unit, and an image displayed on a display unit in this embodiment.

[0028] Fig. 2 is an example of a flowchart of processing performed by the HMD 100 and the external computing device 200 of this embodiment. Fig. 2 is a flowchart of processing for displaying one frame of an image of the mixed reality space to the user, and by performing the processing repeatedly, it is possible to display multiple frames of images of the mixed reality space to the user.

[0029] After the system startup is complete, the HMD 100 drives the image input unit 11 to acquire a captured image of the real space, and transmits the captured image to the downstream transmission unit 13 (step S201). Next, the HMD 100 drives the sensor input unit 12 to acquire sensor information for calculating position and orientation information, and transmits the sensor information to the downstream transmission unit 13 (step S202). The transmission unit 13 transmits the captured image of the real space acquired from the image input unit 11 and the sensor information acquired from the sensor input unit 12 to the external computing device 200 (step S203).

[0030] Next, the external computing device 200 drives the receiving unit 21 to receive the captured image and sensor information transmitted from the HMD 100 (step S204). Next, using the received sensor information, the position and orientation computing unit 22 calculates the position and orientation information of the HMD 100. Furthermore, the virtual image generating unit 23 generates an image of the virtual space when the HMD 100 is in the calculated position and orientation. Furthermore, the virtual image generating unit 23 calculates the display ratio of the image in the virtual space (step (Step S205) A detailed flowchart of step S205 will be described with reference to FIG.

[0031] Next, the image processing unit 24 processes the image in real space according to the calculated display ratio of the image in virtual space (step S206). Note that a detailed flowchart of step S206 will be described with reference to FIG.

[0032] Next, the synthesis unit 25 generates an image of a mixed reality space by superimposing the image of the virtual space acquired from the virtual image generation unit 23 on the image of the real space processed by the image processing unit 24 (step S207). The transmission unit 26 transmits the image of the mixed reality space generated by the synthesis unit 25 to the HMD 100 (step S208).

[0033] Next, the HMD 100 drives the receiving unit 14 to receive the image of the mixed reality space transmitted from the external computing device 200 (step S209). Next, the display unit 15 displays the received image of the mixed reality space to the user (step S210).

[0034] FIG. 3 is an example of a flowchart for generating an image in a virtual space performed by the CG generating unit 201 (the position and orientation calculating unit 22 and the virtual image generating unit 23) for explaining step S205 in detail.

[0035] Using the received sensor information, the position and orientation calculation unit 22 calculates the position and orientation information of the image input unit 11 (and thus the HMD 100) that captured the image of real space (step S301). Next, the virtual image generation unit 23 calculates the display coordinates of the image in the virtual space based on the calculated position and orientation information (step S302). Next, the virtual image generation unit 23 generates an image of the virtual space based on the calculated display coordinates of the image in the virtual space (step S303). Next, the virtual image generation unit 23 calculates the proportion of the generated image of the virtual space that is displayed in the image of the real space (step S304). Note that the order of steps S303 and S304 may be reversed.

[0036] 4 to 6 are examples of flowcharts of the processing performed by the image processing unit 24 to explain step S206 in detail. Each of the figures shows different processing contents performed when the display ratio of the image in the virtual space is greater than a predetermined threshold. FIG. 4 corresponds to the first embodiment shown in FIG. 2. FIGS. 5 and 6 each correspond to a different modified example. It is also preferable that the predetermined threshold used here can be set in advance by the user.

[0037] [Figure 4 (this embodiment)] The image processing unit 24 judges whether the display ratio of the image in the virtual space generated by the virtual image generating unit 23 is greater than a threshold value previously determined by the user (step S401). If it is greater than the threshold value (S401: Yes), the image in the real space acquired from the receiving unit 21 is subjected to a resolution conversion process to reduce the resolution of the entire image, and then image processing is performed (step S403). On the other hand, if the display ratio of the image in the virtual space is equal to or less than the threshold value (S401: No), normal image processing is performed on the image in the real space (step S402). Note that various existing algorithms can be adopted for the resolution conversion. The image processing unit 24 may perform other image processing in addition to reducing the resolution of the real image.

[0038] [Figure 5 (Variation 1)] The image processing unit 24 determines whether the display ratio of the virtual space image generated by the virtual image generating unit 23 is greater than a threshold value previously determined by the user (step S401). If it is greater than the threshold value (S401: Yes), the image processing unit 24 calculates the display area of ​​the virtual space image in the real space image obtained from the receiving unit 21 from the display coordinates of the virtual space image obtained from the virtual image generating unit 23. (Step S404). Next, image processing is performed only on the area of ​​the image in real space excluding the calculated display area of ​​the image in virtual space (Step S405). On the other hand, when the display ratio of the image in virtual space is equal to or less than the threshold (S401: No), normal image processing is performed on the image in real space (Step S402).

[0039] [Figure 6 (Variation 2)] The image processing unit 24 judges whether the display ratio of the image of the virtual space generated by the virtual image generating unit 23 is 100% (step S406). If the display ratio of the virtual image in the composite image is 100% (S406: Yes), image processing is not performed on the image of the real space acquired from the receiving unit 21 (step S407). On the other hand, if the display ratio of the image of the virtual space is not 100% (S406: No), normal image processing is performed on the image of the real space (step S402). In this case, 100% may be considered as the second threshold, and in that case, the image processing unit 24 may first make a judgment using the second threshold, and then make a judgment using the threshold of the first embodiment if the ratio is smaller than the threshold. Note that in this modified example, the second threshold, which is a criterion for judging the display ratio, may be a value lower than 100% (for example, 98%) within a range that the user does not feel unnatural.

[0040] (Example image) FIG. 7 shows an example of an image of real space acquired by image input unit 11 in this embodiment, an image of virtual space generated by virtual image generation unit 23, an image of real space after capture generated for each of the flowcharts shown in FIGS. 4 to 6, and an image of mixed reality space displayed on display unit 15.

[0041] Reference numeral 501 denotes an example of an image of real space acquired by the image input unit 11. Reference numeral 502 denotes an example of an image of virtual space generated by the virtual image generating unit 23. In the AR image, a virtual image 502 is displayed superimposed on the real image 501. The display mode of the generated AR image is determined based on the position of the virtual image 502 in the real image 501 and the size of the virtual image 502.

[0042] Reference numeral 503 denotes an image in mixed reality space (503a) and a processed image in real space (503b) when performing normal image processing. This corresponds to a case where the display ratio of the image in the virtual space is equal to or less than the threshold value (S401: No) in Fig. 4 or Fig. 5, or a case where the display ratio of the image in the virtual space is not 100% (S406: No) in Fig. 4c.

[0043] Reference numeral 504 denotes a mixed reality space image (504a) and a captured real space image (504b) in which the display ratio of the virtual space image is greater than the threshold (S401: Yes) in FIG. 4. Image 504b is characterized in that the resolution of the entire real space image is lower than that of image 501. This is due to the process of reducing the resolution performed in S403 in FIG. 4. As a result, the resolution of the real space portion of mixed reality space image 504a is also low. However, since the user focuses on the CG (image of the vehicle) which occupies a large area ratio on the screen, the user is unlikely to feel uncomfortable.

[0044] Reference numeral 505 denotes an image (505a) in mixed reality space and an image (505b) after capturing the real space, in which the display ratio of the virtual space image is greater than the threshold (S401: Yes) in FIG. 5. Image 505b is characterized in that, compared to image 501, only a portion of the real space image (505b1, 505b2) has been subjected to image processing. That is, image processing is performed only on the area that appears in image 505a in mixed reality space, and image processing is not performed on the area in image 505a that is hidden behind the virtual image, so that the amount of calculation required for image processing can be reduced. Note that the effect of reducing the amount of calculation can be obtained by not performing image processing on at least a portion of the area in image 505a that is hidden behind the virtual image.

[0045] Reference numeral 506 denotes an image (506a) in mixed reality space when the display ratio of the image in virtual space is 100% (S406: Yes) in Fig. 6, and an image (506b) after capturing the real space. A feature of 506b is that no processing of the image in real space is performed.

[0046] According to this embodiment, when the user is not gazing at the image in real space, the amount of data required for image processing can be reduced, thereby making it possible to suppress power consumption of the entire system.

[0047] Although the preferred embodiments and modifications of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention. Furthermore, the embodiments and modifications of the present invention can be applied in any combination as long as they do not cause contradictions with each other.

[0048] <Other embodiments> The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0049] [Configuration 1] An acquisition means for acquiring a real image obtained by capturing an image of a real space; a generating means for generating a virtual image which is an image of a virtual space aligned with the real space; an image processing means capable of performing image processing on the real image; a synthesis means for generating a synthetic image by synthesizing the real image and the virtual image; having The image processing means changes the content of image processing for the real image in accordance with a result of comparing a ratio of the virtual image in the composite image with a threshold value. 23. An information processing apparatus comprising: [Configuration 2] The image processing means performs image processing to reduce the resolution of the real image when the ratio of the virtual image to the composite image is greater than the threshold value. 2. The information processing device according to configuration 1. [Configuration 3] When a ratio of the virtual image to the composite image is greater than the threshold value, the image processing means does not perform image processing on a region of the real image that is hidden by the virtual image in the composite image. 3. The information processing device according to configuration 1 or 2. [Configuration 4] When a ratio of the virtual image to the composite image is greater than a second threshold value that is greater than the threshold value, the image processing means does not perform image processing on the real image. 4. The information processing device according to any one of configurations 1 to 3. [Configuration 5] When the ratio of the virtual image to the composite image is 100%, the image processing means does not perform image processing on the real image. 5. The information processing device according to configuration 4. [Configuration 6] The image processing means treats a region of the virtual image that is transparent in the composite image as the real image when calculating the ratio. 6. The information processing device according to any one of configurations 1 to 5. [Configuration 7] The information processing device further includes an output unit for outputting the composite image to an external device. 7. The information processing device according to any one of configurations 1 to 6. [Configuration 8] The acquiring means acquires the real image captured by an imaging device external to the information processing device. 8. The information processing device according to configuration 7. [Configuration 9] The information processing device is configured to be able to communicate with a head mounted display worn by a user, The head mounted display includes the external imaging device and a display device for displaying the composite image output from the output means. 9. The information processing device according to configuration 8. [Configuration 10] an acquisition step of acquiring a real image obtained by capturing an image of a real space; A generating step of generating a virtual image which is an image of a virtual space; an image processing step of performing image processing on the real image; a synthesis step of generating a synthetic image by synthesizing the real image and the virtual image; having In the image processing step, the content of image processing for the real image is changed according to a result of comparing a ratio of the virtual image in the composite image with a threshold value. 23. An information processing method comprising: [Configuration 11] A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 9. [Explanation of symbols]

[0050] 11: image input unit, 15: display unit, 23: CG generation unit, 24: image processing unit, 25: synthesis unit

Claims

1. An acquisition means for acquiring a real image obtained by capturing an image of a real space; A generating means for generating a virtual image which is an image of a virtual space; an image processing means capable of performing image processing on the real image; a synthesis means for generating a synthetic image by synthesizing the real image and the virtual image; having The image processing means changes the content of image processing for the real image in accordance with a result of comparing a ratio of the virtual image in the composite image with a threshold value.

23. An information processing apparatus comprising:

2. The image processing means performs image processing to reduce the resolution of the real image when the ratio of the virtual image to the composite image is greater than the threshold value.

2. The information processing apparatus according to claim 1,

3. When a ratio of the virtual image to the composite image is greater than the threshold value, the image processing means does not perform image processing on a region of the real image that is hidden by the virtual image in the composite image.

2. The information processing apparatus according to claim 1,

4. When a ratio of the virtual image to the composite image is greater than a second threshold value that is greater than the threshold value, the image processing means does not perform image processing on the real image.

2. The information processing apparatus according to claim 1,

5. The image processing means does not perform image processing on the real image when the ratio of the virtual image to the composite image is 100%.

5. The information processing apparatus according to claim 4.

6. The image processing means treats a region of the virtual image that is transparent in the composite image as the real image when calculating the ratio.

2. The information processing apparatus according to claim 1,

7. The information processing device further includes an output unit for outputting the composite image to an external device.

2. The information processing device according to claim 1,

8. The acquiring means acquires the real image captured by an imaging device external to the information processing device.

8. The information processing device according to claim 7,

9. The information processing device is configured to be able to communicate with a head mounted display worn by a user, The head mounted display includes the external imaging device and a display device for displaying the composite image output from the output means.

9. The information processing device according to claim 8,

10. an acquisition step of acquiring a real image obtained by capturing an image of a real space; A generating step of generating a virtual image which is an image of a virtual space; an image processing step of performing image processing on the real image; a synthesis step of generating a synthetic image by synthesizing the real image and the virtual image; having In the image processing step, the content of image processing for the real image is changed according to a result of comparing a ratio of the virtual image in the composite image with a threshold value.

23. An information processing method comprising:

11. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 9.

Images