Information processing device, control method for information processing device, and program

Abstract

This invention provides a technology that enables the display of UI images with high operability in a head-mounted display device. [Solution] The information processing device of the present invention is characterized by comprising: acquisition means for acquiring design information of a head-mounted display device; area determination means for determining an area on the display surface of the head-mounted display device that is difficult to operate based on the design information; and position determination means for determining the position of a UI image on the display surface based on the area on which operation is difficult.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method thereof, and a program.

Background Art

[0002] In a head-mounted display (HMD), a UI (user interface) image may be displayed. The display position of the UI image can be arbitrarily determined by the developer of the application that displays the UI image.

[0003] Patent Document 1 discloses a technique for changing the display position of a UI image when the HMD is set to a stereoscopic display mode and when it is set to a non-stereoscopic display mode. Further, Patent Document 2 discloses a technique for detecting a hand gesture and displaying a UI image near the position where the hand gesture is detected.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] In the technique disclosed in Patent Document 1, in the stereoscopic display mode, since the UI image is displayed outside the area that the user visually recognizes, the user cannot visually recognize the UI image while wearing the HMD, and there is a risk of accidental operation of the UI image. In the technique disclosed in Patent Document 2, the UI image is displayed within the area that the user visually recognizes. However, although the UI image is displayed near the position where the hand gesture is detected, there is a risk of low operability.

[0006] The present invention aims to provide a technology that enables the display of UI images with high operability in a head-mounted display device. [Means for solving the problem]

[0007] The information processing apparatus of the present invention is characterized by comprising: acquisition means for acquiring design information of a head-mounted display device; area determination means for determining an area on the display surface of the head-mounted display device that is difficult to operate based on the design information; and position determination means for determining the position of a UI image on the display surface based on the area on which operation is difficult. [Effects of the Invention]

[0008] According to the present invention, UI images can be displayed with high operability in a head-mounted display device. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows the hardware configuration of the image processing device. [Figure 2] This diagram shows the functional configuration of an image processing device. [Figure 3] This is a flowchart showing the region determination process. [Figure 4] This figure shows the area where operation is difficult. [Figure 5] This figure shows a method for combining multiple difficult-to-manipulate regions. [Figure 6] This diagram shows a method for combining the difficult-to-operate regions of two HMDs. [Figure 7] This is a flowchart illustrating the notification process. [Figure 8] This figure shows the positional relationship between the UI image area and the area that is difficult to operate. [Figure 9] This is a diagram showing the application screen. [Figure 10] This figure shows the functional configuration of the image processing system according to Embodiment 2. [Figure 11] This figure shows the application screen according to Embodiment 2. [Modes for carrying out the invention]

[0010] <Embodiment 1> Embodiment 1 of the present invention will be described below with reference to the drawings. Figure 1 is a block diagram showing the hardware configuration of the image processing device 100 according to Embodiment 1. The image processing device 100 is an information processing device such as a personal computer (PC). The image processing device 100 can generate images to be displayed on an HMD and change the arrangement of UI images superimposed on content images. The image processing device 100 is not limited to a PC, but may be, for example, a tablet computer, a mobile phone, a smartphone, a game console, etc. Furthermore, the configuration of the image processing device 100 shown in Figure 1 is just one example, and the configuration of the image processing device 100 is not limited to the configuration shown in Figure 1.

[0011] The CPU 101 is connected to each device of the image processing unit 100. The CPU 101 controls the image processing unit 100 comprehensively. The CPU 101 reads programs (e.g., application programs) stored in the ROM 102 and RAM 103 and executes the processing.

[0012] ROM102 is a non-volatile memory that can be electrically stored and erased. ROM102 stores programs that the CPU101 can execute, settings related to program operation, UI image data, an operating system (OS), device drivers, and boot programs.

[0013] RAM103 is used as temporary working memory to hold various data (such as programs and settings) used by the CPU101 when executing a program. RAM103 is also used as buffer memory to temporarily hold image data and other data read from the storage medium 104.

[0014] The storage medium 104 is an HDD or the like for storing various data such as applications and images according to Embodiment 1, and is composed of a semiconductor memory, a magnetic disk, or the like. The storage medium 104 may be detachable from the image processing apparatus 100 or may be built in the image processing apparatus 100. In the image processing apparatus 100, at least the CPU 101 needs to be able to access the storage medium 104.

[0015] The input I / F 105 and the output I / F 106 are interfaces for connecting the image processing apparatus 100 and an external apparatus. A signal in a format processable by the image processing apparatus 100 is input to the input I / F 105 from the external apparatus. The output I / F 106 outputs a signal in a format processable by the external apparatus to the external apparatus. Thereby, the image processing apparatus 100 can transmit and receive information to and from the external apparatus via the input I / F 105 and the output I / F 106.

[0016] The HMD 107 is a display device such as a video see-through type HMD. The HMD 107 is an example of an external apparatus connected to the image processing apparatus 100 via the input I / F 105 and the output I / F 106. The HMD 107 outputs information acquired and generated by the HMD 107 to the image processing apparatus 100 via the input I / F 105. The information output by the HMD 107 may be, for example, information on an image captured by an imaging unit provided in the HMD 107, or the position of the HMD 107 acquired by a gyro sensor provided in the HMD 107 information regarding the posture or the like. The HMD 107 acquires information generated by the image processing apparatus 100 from the image processing apparatus 100 via the output I / F 106. The information to be acquired is, for example, information on an image to be displayed on a display surface (display) provided in the HMD 107.

[0017] Note that the HMD 107 is not limited to a video see-through type HMD. The HMD 107 may be, for example, other display devices such as an optical see-through type HMD (for example, AR glasses) and a hand-held display (HHD: Hand Held Display).

[0018] Figure 2 is a block diagram showing the functional configuration of the image processing device 100. Below, we will explain how the image processing device 100 determines the difficult-to-operate areas (areas with low operability of UI images (visibility of UI images)) on the display surface of the HMD 107 based on the design information of the HMD 107. In Embodiment 1, the CPU 101 implements the functions of the control unit 201, parameter determination unit 203, design information acquisition unit 204, HMD selection unit 206, area determination unit 207, notification unit 208, and UI position determination unit 210, but is not limited to this. For example, each function may be implemented by a circuit other than the CPU 101.

[0019] The control unit 201 controls the entire image processing device 100. The control unit 201 performs various processes based on, for example, user operations 202. User operations 202 are operations performed by the user of the image processing device 100. The method of user operations 202 is not particularly limited; for example, user operations 202 may be operations input to devices such as a mouse and keyboard connected to the input I / F 105, or operations input to a touch panel. The control unit 201 controls the image processing device 100 using triggers such as user operations 202, elapsed time, and information from external devices connected to the input I / F 105.

[0020] The parameter determination unit 203 determines the values ​​input by the user operation 202 as parameters for each process of the image processing device 100. The parameters determined by the parameter determination unit 203 are, for example, thresholds used in the notification process described later. Default values ​​are predetermined for each threshold in Embodiment 1, but the user of the image processing device 100 can change each threshold arbitrarily. Furthermore, the parameters are not limited to the thresholds described in Embodiment 1; for example, the user of the image processing device 100 may set various parameters corresponding to each process. The control unit 201 stores the parameters determined by the parameter determination unit 203 in the storage unit 205.

[0021] The design information acquisition unit 204 acquires design information for the HMD 107. In Embodiment 1, the design information for the HMD 107 includes, for example, information regarding the viewing angle characteristics of the HMD 107 (the display surface of the HMD 107) and the optical resolution (MTF characteristics) of the optical system of the HMD 107. Furthermore, the design information for the HMD 107 includes information regarding the nose relief structure, information regarding the hand detectable area, information regarding the rendering resolution distribution, and information regarding the user's gaze trackable area. The design information acquisition unit 204 may acquire design information for other HMDs in addition to the design information for the HMD 107.

[0022] The design information acquisition unit 204 can acquire design information from the HMD 107 by acquiring the design information from the HMD 107 connected to the image processing device 100, or by reading the design information from the storage unit 205. The image processing device 100 allows the user to select the method of acquiring the design information by operation 202.

[0023] The following describes how the design information acquisition unit 204 acquires design information from the HMD 107. Figure 2 shows the image processing device 100 (design information acquisition unit 204) and the HMD 107 in contact. The process continues. When the design information acquisition unit 204 acquires design information from the HMD 107, for example, if the HMD 107 has a memory unit in which design information is pre-stored, and the image processing device 100 is connected to the HMD 107, the design information is acquired from the memory unit of the HMD 107. The design information acquisition unit 204 may also acquire the design information of the HMD 107 from the memory unit of the HMD 107 based on user operation 202 (for example, in response to a user operation instructing the acquisition of design information). The acquired design information is stored in working memory (RAM 103) or the like.

[0024] When the design information acquisition unit 204 acquires the design information of the HMD 107 from the storage unit 205, it may, for example, download the design information of the HMD 107 from a website or the like in advance and store it in the storage unit 205. However, the method of storing the design information of the HMD 107 in the storage unit 205 is not limited to this; for example, when the image processing device 100 is connected to the HMD 107, the design information of the HMD 107 acquired from the HMD 107 may be stored in the storage unit 205. The design information acquisition unit 204 may acquire the design information of the HMD 107 from the storage unit 205 based on user operation 202. The design information acquisition unit 204 may acquire the design information of the HMD 107 from the storage unit 205 when the image processing device 100 is connected to the HMD 107.

[0025] The memory unit 205 is composed of at least one of the ROM 102 or the storage medium 104.

[0026] The HMD selection unit 206 selects an HMD from which the design information acquisition unit 204 will acquire design information, based on the user operation 202. Here, let's assume that HMD 107 is selected. However, the HMDs that the HMD selection unit 206 can select are not limited to HMD 107. For example, if the memory unit 205 or the memory unit of HMD 107 stores multiple design information corresponding to multiple HMDs, the HMD selection unit 206 may select multiple HMDs. The same applies when multiple HMDs are connected to the image processing device 100 (design information acquisition unit 204).

[0027] The region determination unit 207 determines the difficult-to-operate region based on the design information of the HMD 107 selected by the HMD selection unit 206. However, if the HMD selection unit 206 has selected multiple HMDs, the region determination unit 207 determines multiple difficult-to-operate regions corresponding to each of the multiple HMDs based on the design information of each of the selected HMDs.

[0028] The region determination unit 207 determines the distribution of evaluation values ​​related to operability on the display surface of the HMD 107 based on the design information of the HMD 107 in order to determine the difficult-to-operate region. The region determination unit 207 sets the evaluation value to 1 for regions in which it determines that the operability of the UI image is high for the user of the HMD 107. The region determination unit 207 also sets the evaluation value to 0 for regions in which it determines that the user of the HMD 107 cannot operate (cannot see) the UI image. The region determination unit 207 generates evaluation value map information that expresses operability as an evaluation value from 0 to 1. The region determination unit 207 considers regions that include regions in which the evaluation value is below a threshold as difficult-to-operate regions. In Embodiment 1, the evaluation value can take on three or more values ​​from 0 to 1 in order to express a continuous change in operability on the display surface, but it may also take on two values, 0 and 1.

[0029] Here, the region determination process performed by the region determination unit 207 will be explained using Figure 3. Figure 3 is a flowchart of the region determination process.

[0030] In S301, the design information acquisition unit 204 acquires the design information of the HMD. The region determination unit 207 acquires the design information of the HMD selected by the HMD selection unit 206. If the HMD selection unit 206 has selected multiple HMDs, the design information acquisition unit 204 acquires multiple pieces of design information corresponding to each of the multiple HMDs. The design information acquisition unit 204 may acquire the multiple pieces of design information in the same way or in different ways. The design information includes Multiple pieces of information may be obtained using different methods.

[0031] In S302, the region determination unit 207 determines the difficult-to-operate region by generating evaluation value map information based on the HMD design information acquired in S301. If the HMD design information acquired in S301 includes multiple types of information, the region determination unit 207 determines multiple difficult-to-operate regions corresponding to each of the multiple types of information. If the HMD selection unit 206 has selected multiple HMDs, the difficult-to-operate region for each HMD is determined by generating multiple evaluation value maps corresponding to each of the multiple HMDs.

[0032] Figures 4(A) to 4(E) show the areas of difficulty in operation, determined based on various information included in the HMD's design information. In Figures 4(A) to 4(E), two display areas visible to each of the HMD user's eyes are superimposed. Here, the method for determining the areas of difficulty in operation based on various information included in the HMD's design information will be explained using Figures 4(A) to 4(E).

[0033] Figure 4(A) shows the difficult-to-operate region determined by the region determination unit 207 when the HMD design information includes information on the HMD's field of view characteristics and MTF characteristics (MTF distribution). Generally, according to the HMD's field of view characteristics and MTF characteristics, the visibility of the HMD's display surface (the image displayed on the display surface) as seen by the HMD user decreases as you approach the outer edge of the display surface. Therefore, when a UI image is placed near the outer edge of the HMD's display surface, the visibility of the UI image is low, and the operability of the UI image is low. The region determination unit 207 determines an evaluation value that is smaller the larger the angle of the HMD user's line of sight relative to the HMD's display surface and smaller the MTF. Region 400 shown in Figure 4(A) is a region near the center of the optical axis in the HMD's optical system, and since it is a region where the angle of sight relative to the HMD's display surface is large and the MTF is large (high), the region determination unit 207 sets the evaluation value of region 400 to 1. Region 401 is a region slightly away from the center of the optical axis in the HMD's optical system, and is a region where the viewing angle with respect to the HMD's display surface is small and the MTF is small (low). Therefore, the region determination unit 207 sets the evaluation value of region 401 to a value less than 1. Region 402 is a region at the image height edge (a region with a high image height), and is a region where the HMD user cannot see the UI image. Therefore, the region determination unit 207 sets the evaluation value of region 402 to 0. In Figure 4(A), regions with an evaluation value of 1 are represented in white, and regions with an evaluation value of 0 are represented in black. In Figure 4(A), the closer to black, the lower the evaluation value (lower operability).

[0034] Figure 4(B) shows the area of ​​difficulty in operation determined by the area determination unit 207 when the HMD design information includes information about the nose clearance structure. The nose clearance structure is a structure that allows the nose to move away from the display surface by leaving a space so that the nose does not hit the display surface when the user puts on the HMD. By adopting such a structure, the distance from the HMD display surface to the user's eyes can be reduced. On the other hand, because a space for the nose is provided in the HMD, the user will not be able to see the lower left of the display area with their right eye, and the lower right of the display area with their left eye. In Figure 4(B), the area that cannot be seen is shown as area 403. Since area 403 is an area that the HMD user cannot see, the area determination unit 207 sets the evaluation value of area 403 to 0. The evaluation value of an area larger than area 403, including area 403, may also be set to 0.

[0035] Figure 4(C) shows the area of ​​difficulty in operation determined by the area determination unit 207 when the HMD design information includes information about the hand-detectable area. The hand-detectable area may be limited to a predetermined size area (range) centered on the optical axis in the HMD's optical system in order to reduce the load of the hand detection process and false detection of the hand (and its gestures). Also, if the HMD cannot detect the hand, even if a UI image is displayed on the display surface, the user cannot operate the UI image with hand gestures. The area determination unit 207 determines the area outside the hand-detectable area ( Specifically, the evaluation value of the area on the display surface that corresponds to the area outside the detectable hand area is set to 0. The area determination unit 207 sets the evaluation value of the detectable hand area (specifically, the area on the display surface that corresponds to the detectable hand area) to 1. The evaluation value of the area larger than the detectable hand area, including the detectable hand area, may also be set to 1. The detectable hand area may be an area in which a hand can be detected with a high accuracy above a predetermined threshold, or it may be an area in which hand detection processing is performed.

[0036] Figure 4(D) shows the difficult-to-operate regions determined by the region determination unit 207 when the HMD design information includes information about the distribution of rendering resolutions that change within the display surface. There is a technique that changes the rendering resolution of an image in stages according to the image height. For example, rendering is performed at a high resolution near the center of the optical axis in the HMD's optical system, and at a low resolution in other areas. If a UI image is placed in a region where the rendering resolution changes within the display surface, unnecessary resolution changes occur in the UI image, reducing the visibility and operability of the UI image. Therefore, the region determination unit 207 sets the evaluation value of the region where the rendering resolution changes to 0 and the evaluation value of the other regions to 1.

[0037] Figure 4(E) shows the difficult-to-operate area determined by the area determination unit 207 when the HMD design information includes information about the area of ​​the HMD user's gaze that can be tracked. There is a technology that tracks the user's gaze and manipulates or selects a UI image in the direction of the gaze. In this technology, the user can make a UI image operable by fixating on it, and the user can select an option included in the UI image by looking at it and blinking. If the UI image is placed outside the area of ​​the user's gaze that can be tracked, the HMD user cannot manipulate the UI image with their gaze. Therefore, the area determination unit 207 sets the evaluation value of the area outside the user's gaze that can be tracked (specifically, the area on the display surface that corresponds to the area outside the area of ​​the gaze that can be tracked) to 0. The area determination unit 207 sets the evaluation value of the area of ​​the user's gaze that can be tracked (specifically, the area on the display surface that corresponds to the area of ​​the gaze that can be tracked) to 1. The evaluation value of the area including the area of ​​the gaze that can be tracked, but larger than the area of ​​the gaze that can be tracked, may also be set to 1. The trackable area may be an area where the user's gaze can be detected with a high accuracy above a predetermined threshold, or it may be an area where the user's gaze detection processing is performed.

[0038] Returning to the explanation of Figure 3, in S303, the region determination unit 207 combines the multiple difficult-to-operate regions determined in S302 based on multiple types of information included in the design information of one HMD into one difficult-to-operate region. If the HMD selection unit 206 has selected multiple HMDs, the process in S303 is performed for each of the multiple HMDs. For HMDs where there is only one difficult-to-operate region determined in S302 (for example, an HMD where there is only one type of information included in the design information), the process in S303 is omitted. Here, a method for combining two difficult-to-operate regions based on two different types of information included in the design information of one HMD into one difficult-to-operate region will be explained using Figures 5(A) and 5(B). Figures 5(A) and 5(B) show a difficult-to-operate region obtained by combining the two difficult-to-operate regions determined in S302.

[0039] Figure 5(A) shows a combined difficult-to-operate region, which is the result of combining the difficult-to-operate region shown in Figure 4(A) and the difficult-to-operate region shown in Figure 4(B). Here, matrix X represents the evaluation value map of the difficult-to-operate region based on the field of view characteristics and MTF characteristics, matrix Y represents the evaluation value map of the difficult-to-operate region based on information about the nose escape structure, and matrix Z represents the evaluation value map of the combined difficult-to-operate region. Below, as an example, a method for obtaining element Z1, which is an element of matrix Z, is shown.

[0040] Let X1 be an element of matrix X corresponding to element Z1, and Y1 be an element of matrix Y corresponding to element Z1. Then, use equation (1) below to calculate the provisional element Z11. Z11 = (1 - X1) + (1 - Y1) ... (1)

[0041] If the value of the provisional element Z11 calculated using equation (1) is 1 or greater, a threshold process is performed to limit the value of provisional element Z11 to 1. Next, the element Z1 of matrix Z is calculated by inverting the value of provisional element Z11 between 0 and 1 using equation (2). Z1 = (1 - Z11) ... (2)

[0042] In this way, it is possible to combine multiple hard-to-operate regions corresponding to each of the multiple pieces of information included in the design information within a single HMD. By using equations (1) and (2), a region containing at least one of the multiple hard-to-operate regions can be made into the combined hard-to-operate region. In Figure 5(A), two hard-to-operate regions are combined: one based on field-of-view characteristics and MTF characteristics, and another based on information regarding the nose relief structure, but this is not the only example. For example, three or more hard-to-operate regions corresponding to three or more types of information included in the design information may be combined.

[0043] Figure 5(B) shows a combined area of ​​difficulty in operation, which is obtained by combining the area of ​​difficulty in operation based on the information regarding the detectable hand area shown in Figure 4(C) and the area of ​​difficulty in operation based on the information regarding the HMD user's gaze tracking area shown in Figure 4(E). The area of ​​difficulty in operation shown in Figure 5(B) can be determined in the same way as described using Figure 5(A).

[0044] The user of the image processing device 100 may arbitrarily set the type of difficult-to-operate region used for synthesis (the type of information used to determine the difficult-to-operate region). If the HMD selection unit 206 has selected multiple HMDs, the type of difficult-to-operate region used for synthesis may be the same or different among the multiple HMDs.

[0045] Returning to the explanation of Figure 3, in S304, the region determination unit 207 checks whether multiple HMDs have been selected by the HMD selection unit 206. If multiple HMDs have been selected by the HMD selection unit 206, the region determination unit 207 proceeds to S305; otherwise, it terminates the process.

[0046] In S305, the region determination unit 207 combines the multiple difficult-to-operate regions corresponding to the multiple HMDs determined up to S303 into a single difficult-to-operate region. Here, the method of combining two difficult-to-operate regions corresponding to two HMDs into a single difficult-to-operate region will be explained using Figure 6.

[0047] Figure 6(A) shows the difficult-to-operate region corresponding to the first HMD. Figure 6(B) shows the difficult-to-operate region corresponding to the second HMD. Figure 6(C) shows the difficult-to-operate region obtained by combining the difficult-to-operate region shown in Figure 6(A) and the difficult-to-operate region shown in Figure 6(B). Let matrix L be the evaluation value map representing the difficult-to-operate region of the first HMD, matrix M be the evaluation value map representing the difficult-to-operate region of the second HMD, and matrix N be the evaluation value map representing the combined difficult-to-operate region. Below, as an example, a method for obtaining element N1, which is an element of matrix N, is shown.

[0048] Let L1 be an element of matrix L corresponding to element N1, and M1 be an element of matrix M corresponding to element N1. Then, use equation (3) below to calculate the provisional element N11. N11 = (1 - L1) + (1 - M1) ... (3)

[0049] If the value of the provisional element N11 calculated using equation (3) is 1 or greater, a threshold process is performed to limit the value of provisional element N11 to 1. Next, the element N1 of matrix N is calculated by inverting the value of provisional element N11 between 0 and 1 using equation (4). N1 = (1 - N11) ... (4)

[0050] In this way, it is possible to combine multiple hard-to-operate regions corresponding to multiple HMDs. By using equations (3) and (4), a region containing at least one of the multiple hard-to-operate regions can be made into the combined hard-to-operate region. In Embodiment 1, two hard-to-operate regions corresponding to two HMDs, the first HMD and the second HMD, were combined, but the number of HMDs is not limited to this. For example, three or more hard-to-operate regions corresponding to three or more HMDs may be combined.

[0051] Returning to the explanation of Figure 2, when the region determination unit 207 finishes the region determination process, it stores the information regarding the determined difficult-to-operate region in the storage unit 205.

[0052] The notification unit 208 controls the image processing device 100 to notify the user of information regarding the placement of UI images, based on the difficult-to-operate areas determined by the area determination unit 207. The display unit 209 displays the difficult-to-operate areas generated by the area determination unit 207 and displays information based on the control of the notification unit 208. Various information is notified to the user of the image processing device 100 through the display on the display unit 209. The UI position determination unit 210 determines the position and area of ​​the UI image on the display surface of the HMD 107 based on user operation 202.

[0053] The notification unit 208 notifies the user of the image processing device 100 that the UI image is placed on an area that is difficult to operate. The notification process (notification method) of the notification unit 208 will now be explained in detail using Figure 7. Figure 7 is a flowchart of the notification process.

[0054] In S701, the notification unit 208 acquires the difficult-to-operate region determined by the region determination unit 207.

[0055] In S702, the notification unit 208 acquires the UI image region (UI region) determined by the UI position determination unit 210.

[0056] In S703, the notification unit 208 determines the region evaluation value of the UI area. The method for determining the region evaluation value will be explained using Figures 8(A) and 8(B). Figures 8(A) and 8(B) show the positional relationship between the UI area and the difficult-to-operate area. For example, when a user of the image processing device 100 specifies the UI area (the position of the UI image), a screen like the one shown in Figures 8(A) and 8(B) (a screen showing the UI area and the difficult-to-operate area) is displayed on the display unit 209.

[0057] In Figure 8(A), UI region 801 is positioned so as to overlap with the difficult-to-operate region. UI region 801 is assigned an evaluation value greater than 0 and less than or equal to 1. In cases like that shown in Figure 8(A), the average of the evaluation values ​​within UI region 801 is calculated as the region evaluation value. In Figure 8(B), UI region 802 is positioned so as to overlap with the difficult-to-operate region. UI region 802 is assigned an evaluation value of either 0 or 1. As shown in Figure 8(B), if UI region 801 contains a region with an evaluation value of 0, the region evaluation value is set to 0. By performing this processing, a smaller region evaluation value can be obtained the lower the operability of the UI image.

[0058] Returning to the explanation of Figure 7, in S704, the notification unit 208 determines whether the region evaluation value is below the threshold. If the region evaluation value is below the threshold, the notification unit 208 proceeds to S705; otherwise, it terminates the process. The threshold is, for example, a value corresponding to user operation 202, and is a value determined by the parameter determination unit 203. The user can change the threshold through user operation 202.

[0059] In S705, the notification unit 208 notifies the user of the image processing device 100. Notification unit 2 If the area evaluation value is below the threshold and is not 0, 08 may notify the user by displaying text such as "Warning: UI visibility and operability may be reduced" on the display unit 209. If the area evaluation value is 0, 08 may notify the user by displaying text such as "Warning: UI may not be visible or operable" on the display unit 209.

[0060] However, the notification method of the notification unit 208 is not limited to display; for example, notification may be provided by sound (such as a beep) by connecting an audio output device to the image processing device 100. Furthermore, the method for preventing a decrease in the operability of the UI image is not limited to notification. For example, when the user of the image processing device 100 moves the UI image with a mouse or touch panel and the area evaluation value falls below a threshold or becomes 0, the UI image may be automatically moved to avoid areas that are difficult to operate (so that the area evaluation value is above the threshold).

[0061] In this way, the user of the image processing device 100 can be prompted to rearrange the UI images to avoid areas that are difficult to operate. As a result, the user of the image processing device 100 can arrange the UI images to avoid areas that are difficult to operate, and can select a position where the UI images are highly visible and the UI is easy to operate.

[0062] Figures 9(A) and 9(B) show the screen of an application executed by the image processing device 100. The application screen 900 shown in Figures 9(A) and 9(B) is displayed on the display unit 209 of the image processing device 100. The application screen 900 shown in Figure 9(A) displays an image representing area 901, which is a difficult-to-operate area, and a notification 902. By viewing the application screen 900, the user of the image processing device 100 can move the position of the UI image 903 while confirming area 901 and notification 902. This allows the user of the image processing device 100 to position the UI image in a position with high visibility and operability. As shown in Figure 9(B), the application screen 900 may display only notification 902 without displaying area 901.

[0063] According to Embodiment 1, the user of the image processing device 100, which creates content to be displayed on the HMD, can arrange the UI images so that high operability is achieved on the HMD by visually checking the difficult-to-operate areas and notifications displayed on the display unit 209. Furthermore, according to Embodiment 1, the area determination unit 207 generates map information that expresses operability as an evaluation value from 0 to 1, and considers areas where the evaluation value is below a threshold as difficult-to-operate areas, thereby enabling the determination of difficult-to-operate areas with simple calculations.

[0064] <Embodiment 2> Embodiment 2 of the present invention will now be described with reference to the drawings. In Embodiment 1, the image processing device 100 was equipped with a UI position determination unit 210, but the UI position determination unit 210 (the application that determines the position of the UI image) may be a separate device from the image processing device 100. Embodiment 2 describes a case where the application that displays the difficult-to-operate area on the display unit 209 and the application that arranges the UI image are different, and these two applications are installed on two different devices. However, in the description of Embodiment 2, the differences from Embodiment 1 will be described, and detailed explanations of similar points will be omitted.

[0065] Figure 10 is a block diagram showing the functional configuration of the image processing system 1000 according to Embodiment 2. The region processing device 1001 has a configuration that excludes the notification unit 208 and the UI position determination unit 210 from the image processing device 100.

[0066] The UI image processing device 1002 consists of a control unit 1003, a UI position determination unit 1004, and a display unit. The system includes 1005. The control unit 1003 controls the entire UI image processing device 1002. The control unit 1003 performs various processes based on, for example, user operations 202. The UI position determination unit 1004 determines the position of the UI image on the HMD display surface based on the user operations 202. The area processing device 1001 performs display control to display the hard-to-operate area on the display unit 209, so that the user of the image processing system 1000 can refer to the hard-to-operate area and position the UI image in a position where the UI image is highly visible and operable. The display unit 1005 displays the UI image at the determined position.

[0067] Figure 11(A) shows the application screen displayed by the UI image processing device 1002. Figure 11(B) shows the application screen displayed by the region processing device 1001. The application screen in Figure 11(A) displayed by the region processing device 1001 shows the difficult-to-operate region 1101. The user of the image processing system 1000 can adjust the position of the UI 1100 while checking the difficult-to-operate region 1101.

[0068] According to Embodiment 2, by displaying the difficult-to-operate area on the display unit 209, the user of the image processing system 1000, which creates content to be displayed on the HMD, can position the UI image in a location that is easy for the HMD user to see and operate. Furthermore, according to Embodiment 2, the application for determining the position of the UI image and the application for displaying the difficult-to-operate area can be provided separately. As a result, even if the application for creating content to be displayed on the HMD does not have a function to display the difficult-to-operate area, it is possible to position the UI image in a highly operable location without modifying the application.

[0069] The various controls described above may or may not be performed by a single piece of hardware (e.g., a processor or circuit). Multiple pieces of hardware (e.g., multiple processors, multiple circuits, or a combination of one or more processors and one or more circuits) may share the processing to control the entire device.

[0070] Furthermore, the above-mentioned processors are processors in a broad sense, including general-purpose processors and specialized processors. General-purpose processors include, for example, CPUs (Central Processing Units), MPUs (Micro Processing Units), and DSPs (Digital Signal Processors). Specialized processors include, for example, GPUs (Graphics Processing Units), ASICs (Application Specific Integrated Circuits), and PLDs (Programmable Logic Devices). Programmable logic devices include, for example, FPGAs (Field Programmable Gate Arrays) and CPLDs (Complex Programmable Logic Devices).

[0071] Furthermore, the embodiments described above (including modified examples) are merely examples, and configurations obtained by appropriately modifying or changing the above-described configurations within the scope of the gist of the present invention are also included in the present invention. Configurations obtained by appropriately combining the above-described configurations are also included in the present invention.

[0072] <Other Embodiments> The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit that implements one or more functions.

[0073] This embodiment includes the following configurations, methods, and programs. (Composition 1) A means for acquiring design information for a head-mounted display device, A region determination means for determining an area on the display surface of the head-mounted display device that is difficult to operate, based on the design information, Position determination means for determining the position of the UI image on the display surface based on the aforementioned difficult-to-operate area, has An information processing device characterized by the following: (Configuration 2) The acquisition means acquires the design information when the information processing device is connected to the head-mounted display device. The information processing device according to configuration 1, characterized by the above. (Composition 3) The acquisition means acquires the design information from the head-mounted display device. The information processing apparatus according to configuration 2, characterized in that... (Composition 4) The system further includes a storage means in which the aforementioned design information is pre-stored, The acquisition means acquires the design information from the storage means. An information processing device according to configuration 1 or 2, characterized by the above. (Composition 5) The region determination means determines the difficult-to-operate region based on design information of a plurality of head-mounted display devices. An information processing device according to any one of configurations 1 to 4, characterized by the above. (Composition 6) The area determination means determines the distribution of evaluation values ​​related to operability on the display surface based on the design information, The aforementioned difficult-to-operate region includes a region where the evaluation value is below a threshold. An information processing device according to any one of configurations 1 to 5. (Composition 7) The design information includes information regarding the viewing angle characteristics of the head-mounted display device, The region determination means determines that the evaluation value is smaller the greater the angle of the user's line of sight to the display surface of the head-mounted display device. The information processing device according to configuration 6, characterized by the features described therein. (Composition 8) The design information includes information regarding the MTF characteristics of the head-mounted display device, The region determination means determines an evaluation value that is smaller the smaller the MTF. The information processing device according to configuration 6, characterized by the features described therein. (Composition 9) The design information includes information regarding the nose escape structure of the head-mounted display device, The region determination means determines the difficult-to-operate region, which includes a region with low operability due to the nose escape structure. An information processing device according to any one of configurations 1 to 8. (Composition 10) The design information includes information regarding the detectable area of ​​the user's hand in the head-mounted display device, The region determination means determines the difficult-to-operate region, which includes a region corresponding to the area outside the detectable region. An information processing device according to any one of configurations 1 to 9. (Composition 11) The design information includes information regarding the rendering resolution that changes within the display surface, The region determination means determines the difficult-to-operate region, which includes the region where the rendering resolution changes. An information processing device according to any one of configurations 1 to 10, characterized by the above. (Composition 12) The design information includes information regarding the area of ​​the user's gaze that can be tracked by the head-mounted display device, The region determination means determines the difficult-to-operate region, which includes a region corresponding to the area outside the traceable region. An information processing device according to any one of configurations 1 to 11, characterized by the above. (Composition 13) The region determination means determines the difficult-to-operate region based on two or more of the following information included in the design information: information regarding the field of view characteristics of the head-mounted display device, information regarding the MTF characteristics of the head-mounted display device, information regarding the nose relief structure of the head-mounted display device, information regarding the detectable area of ​​the user's hand of the head-mounted display device, information regarding the rendering resolution that changes within the display surface, and information regarding the traceable area of ​​the user's gaze. An information processing device according to any one of configurations 1 to 12, characterized by the above. (Composition 14) The position determination means determines the position of the UI image in accordance with the user operation of the information processing device. The information processing device further includes control means for controlling the device to provide a predetermined notification to the user when the UI image overlaps with the area where operation is difficult. An information processing device according to any one of configurations 1 to 13, characterized by the above. (Composition 15) The control means controls the display to provide the predetermined notification. The information processing device according to configuration 14, characterized by the features described above. (Composition 16) The position determination means determines the position of the UI image in accordance with the user operation of the information processing device. The information processing device further includes display control means for controlling the display of an image representing the difficult-to-operate area to the user. An information processing device according to any one of configurations 1 to 15, characterized by the above. (method) An acquisition step to obtain design information for a head-mounted display device, A region determination step in which a region is difficult to operate on the display surface of the head-mounted display device is determined based on the design information, A position determination step in which the position of the UI image on the display surface is determined based on the aforementioned difficult-to-operate area. has A control method for an information processing device characterized by the following features. (program) A program for causing a computer to function as one of the means of an information processing device described in any one of configurations 1 to 16. [Explanation of Symbols]

[0074] 100: Image processing unit 101: CPU

Claims

1. A means for acquiring design information for a head-mounted display device, A region determination means for determining an area on the display surface of the head-mounted display device that is difficult to operate, based on the design information, Position determination means for determining the position of the UI image on the display surface based on the difficult-to-operate area, has An information processing device characterized by the following:

2. The acquisition means acquires the design information when the information processing device is connected to the head-mounted display device. The information processing apparatus according to feature 1.

3. The acquisition means acquires the design information from the head-mounted display device. The information processing apparatus according to feature 2.

4. The system further includes a storage means in which the aforementioned design information is pre-stored, The acquisition means acquires the design information from the storage means. The information processing apparatus according to feature 1.

5. The region determination means determines the difficult-to-operate region based on design information of a plurality of head-mounted display devices. The information processing apparatus according to feature 1.

6. The area determination means determines the distribution of evaluation values ​​related to operability on the display surface based on the design information, The aforementioned difficult-to-operate region includes a region where the evaluation value is below a threshold. The information processing apparatus according to feature 1.

7. The design information includes information regarding the viewing angle characteristics of the head-mounted display device, The region determination means determines that the evaluation value is smaller the greater the angle of the user's line of sight to the display surface of the head-mounted display device. The information processing apparatus according to feature 6.

8. The design information includes information regarding the MTF characteristics of the head-mounted display device, The region determination means determines a smaller evaluation value the smaller the MTF. The information processing apparatus according to feature 6.

9. The design information includes information regarding the nose escape structure of the head-mounted display device, The region determination means determines the difficult-to-operate region, which includes a region with low operability due to the nose escape structure. The information processing apparatus according to feature 1.

10. The design information includes information regarding the detectable area of ​​the user's hand in the head-mounted display device, The region determination means determines the difficult-to-operate region, which includes a region corresponding to the area outside the detectable region. The information processing apparatus according to feature 1.

11. The design information includes information regarding the rendering resolution that changes within the display surface, The region determination means determines the difficult-to-operate region, which includes the region where the rendering resolution changes. The information processing apparatus according to feature 1.

12. The design information includes information regarding the area of ​​the user's gaze that can be tracked by the head-mounted display device, The region determination means determines the difficult-to-operate region, which includes a region corresponding to the area outside the traceable region. The information processing apparatus according to feature 1.

13. The region determination means determines the difficult-to-operate region based on two or more of the following information included in the design information: information regarding the field of view characteristics of the head-mounted display device, information regarding the MTF characteristics of the head-mounted display device, information regarding the nose relief structure of the head-mounted display device, information regarding the detectable area of ​​the user's hand of the head-mounted display device, information regarding the rendering resolution that changes within the display surface, and information regarding the traceable area of ​​the user's gaze. The information processing apparatus according to feature 1.

14. The position determination means determines the position of the UI image in accordance with the user operation of the information processing device. The information processing device further includes control means for controlling the device to provide a predetermined notification to the user when the UI image overlaps with the difficult-to-operate area. The information processing apparatus according to feature 1.

15. The control means controls the display to provide the predetermined notification. The information processing apparatus according to feature 14.

16. The position determination means determines the position of the UI image in accordance with the user operation of the information processing device. The information processing device further includes display control means for controlling the display of an image representing the difficult-to-operate area to the user. The information processing apparatus according to feature 1.

17. An acquisition step to obtain design information for a head-mounted display device, A region determination step in which a region is difficult to operate on the display surface of the head-mounted display device is determined based on the design information, A position determination step in which the position of the UI image on the display surface is determined based on the aforementioned difficult-to-operate area. has A control method for an information processing device characterized by the following features.

18. A program for causing a computer to function as one of the means of an information processing apparatus described in any one of claims 1 to 16.

Images