Information processing system, information processing device, and image display device

Abstract

To provide an information processing system, an information processing device, and an image display device configured to superimpose screen information of a smartphone in a display screen of an HMD with high degree of freedom and convenience.SOLUTION: An information processing system includes an HMD (200) equipped with a camera (121) and a smartphone (100) configured to operate and display an application that is application software, the information processing system having an information communication cooperation function between the HMD (200) and the smartphone (100). The HMD (200) images a screen of the smartphone (100) by the camera (121), recognizes a display position and a display size of the read screen of the smartphone (100) as a display area, and displays an image of the screen of the smartphone (100) in the recognized display area.SELECTED DRAWING: Figure 1B

Description

[Technical field]

[0001] The present invention relates to an information processing system, an information processing device, and an image display device that is mainly worn on the head and displays images, and that displays the display screen of the information processing device within the screen of the image display device. [Background technology]

[0002] Information processing devices that have functions such as displaying apps and communicating with other devices are called smartphones (hereinafter referred to as "smartphones"), especially the portable ones. They are roughly palm-sized and can display the operation screens of various apps and various information on the display screen, making them easy to use and widely used. Foldable and rollable smartphones have also appeared, and smartphone screens have become larger, making it possible to display multiple apps at the same time.

[0003] Also, image display devices that display virtual objects have come to be widely used. One example is a head mounted display (HMD, hereinafter referred to as HMD), which is worn on the head and has a display unit, a camera, etc. An HMD is an image display device that displays virtual objects while recognizing real objects in the real space in three dimensions, using Augmented Reality (AR) technology that reflects and augments virtual objects in a virtual space made up of other images or images created with CG (Computer Graphics), etc.

[0004] Furthermore, in recent years, HMDs that display the screen of a smartphone in cooperation with a smartphone have appeared. For example, Patent Document 1 describes an information processing device that includes a photographing unit, a position and orientation calculation unit that calculates the position and orientation of the image display device in a first reference coordinate system of the information processing device based on an image of an external image display device photographed by the photographing unit, a position and orientation conversion unit that converts the position and orientation of the image display device in the first reference coordinate system to the position and orientation of the information processing device in a second reference coordinate system of the image display device, and an embedding position calculation unit that calculates the position at which screen information related to the information processing device is embedded in the screen of the image display device based on the position and orientation of the information processing device in the second reference coordinate system. That is, based on an image of the HMD photographed by a camera mounted on the smartphone, the relative position and orientation of the HMD and the smartphone are grasped, and the screen information sent from the smartphone is superimposed on the playback screen of the HMD in an appropriate position and in an appropriate orientation on the playback screen of the content being viewed. As a result, only the smartphone exists in the content playback screen in the same position and orientation as in the real world, so that the user can operate the screen of the smartphone while wearing the HMD. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2016 / 021252 Summary of the Invention [Problem to be solved by the invention]

[0006] Although Patent Document 1 describes overlaying smartphone screen information on the HMD display screen, the entire smartphone screen must always be positioned within the HMD display screen, which limits freedom of use and makes it less convenient.

[0007] In addition, when displaying a smartphone screen as a virtual object on the HMD, it is necessary to select which application content to display at which position on the HMD, but the conventional selection operation was not visually appealing, complicated, and difficult to do.In addition, when inserting and displaying a smartphone screen as an image on the HMD screen, if the HMD screen image and the smartphone screen image do not match in contrast, for example, when the surroundings are bright, such as outdoors, the brightness of the smartphone inserted image is low compared to the bright HMD screen image (transparent image), making it difficult to see, or when the surroundings are dark, such as at night, the contrast between the dark HMD screen image and the smartphone inserted image is too strong, causing eye fatigue.

[0008] In consideration of the above problems, the present invention provides an information processing system, an information processing device, and an image display device that can easily and conveniently display a screen of an application or the like selected on an information processing device such as a smartphone at a suitable display position and size on a display screen of an image display device such as an HMD. In addition, the present invention provides a high degree of freedom for the suitable display arrangement, enabling convenient operability for the user. Furthermore, the present invention provides an information processing system, an information processing device, and an image display device that can harmonize the contrast between the display screen image of the information processing device inserted into the display screen of the image display device and the display screen image of the image display device. [Means for solving the problem]

[0009] Among the inventions disclosed in this application, a brief summary of representative inventions is as follows.

[0010] (1) an image display device equipped with a camera; an information processing device that operates and displays an application, which is application software; An information processing system having a communication link function for information between an image display device and an information processing device, The image display device is an information processing system characterized by photographing and reading the screen of an information processing device with a camera, recognizing the display position and display size of the read-in screen of the information processing device as a display area, and displaying an image of the screen of the information processing device in the recognized display area.

[0011] (2) An information processing system according to (1), The information processing device selects an application screen or a partial area of ​​the application screen, The image display device is an information processing system characterized by photographing and reading a selected app screen or partial area using a camera, recognizing the display position and display size of the read app screen or partial area as a display area, and displaying an app screen image, which is an image of the selected app screen or partial area, in the recognized display area.

[0012] (3) An information processing system according to (2), The information processing device selects an application screen or a partial area of ​​the application screen, and generates and displays a specific pattern having the selected application screen or partial area as a range thereof; The image display device is an information processing system characterized in that the generated and displayed specific pattern is photographed and read in by a camera, the display position and display size of the read-in specific pattern are recognized as a display area, and an app screen image, which is an image of a selected app screen or a partial area, is displayed in the recognized display area.

[0013] (4) An information processing system according to (3), The information processing device is an information processing system characterized in that the specific pattern that is generated and displayed is made semi-transparent.

[0014] (5) An information processing system according to (3), The image display device is an information processing system characterized in that it generates an area frame indicating the range of a specific pattern generated and displayed by an information processing device, and displays the area frame on a display screen of the image display device.

[0015] (6) An information processing system according to (2), The information processing device is an information processing system characterized in that it launches and displays display application switching software, which can switch between multiple applications, on a display screen of the information processing device, and each application icon shown on the launched and displayed display application switching software is selected, thereby selecting the application to be displayed on the image display device.

[0016] (7) An information processing system according to (2), The information processing device selects an application screen or a partial area of ​​the application screen, the image display device captures an image of the selected application screen or partial area by a camera, recognizes a display position and a display size of the captured application screen or partial area as a display area, and displays an application screen image, which is an image of the selected application screen or partial area, in the recognized display area; Furthermore, the information processing device selects another application screen or another partial area of ​​the other application screen displayed on the image display device, The image display device is an information processing system characterized in that it photographs a selected other app screen or a partial area of ​​the other app screen with a camera, recognizes the display position and display size of the photographed other app screen or partial area of ​​the other app screen as a different display area, and displays a different app screen image, which is an image of the other app screen or partial area of ​​the other app screen selected by the information processing device, in the recognized other display area.

[0017] (8) An information processing system according to (2), The image display device or the information processing device identifies a degree of harmony of contrast between an application screen image to be inserted into a display screen of the image display device and an image on the display screen of the image display device; The information processing device is an information processing system characterized in that, when contrast disharmony is identified based on the identified contrast harmony state, the information processing device adjusts image quality of the application screen so that the contrast is harmonized.

[0018] (9) An information processing system according to (8), The image display device has an illuminance sensor that detects the brightness of the surroundings of a field of view, The information processing device is an information processing system that adjusts the brightness of an application screen in accordance with an output from an illuminance sensor.

[0019] (10) An information processing system according to (2), This information processing system is characterized in that when an app screen image selected by an information processing device is displayed in a display area of ​​an image display device, the information processing device selects whether to display constantly or for each trigger notification, and if constant display is selected, the image display device always displays the app screen image, and if display for each trigger notification is selected, the image display device starts or stops displaying the app screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device.

[0020] (11) An information processing device having a function of running and displaying an application that is application software, and a function of communicating and linking information with an image display device equipped with a camera, The information processing device includes: A screen of the information processing device to be displayed on the image display device is selected, The display position and the display size of the screen of the information processing device photographed and read by the camera are recognized by the image display device as a display area; supplying an image on a screen of the information processing device to be displayed on the image display device to the image display device; The information processing apparatus is characterized in that it controls an image display device so as to display the image of the screen of the supplied information processing apparatus in the recognized display area.

[0021] (12) The information processing device according to (11), The information processing device includes: An app screen or a part of the app screen is selected, The display device is caused to recognize a display position and a display size of the application screen or a partial area photographed and read by the camera as a display area; Supplying an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device; The information processing device controls an image display device so as to display the supplied selected application screen or an application screen image which is an image of a partial area in the recognized display area.

[0022] (13) An information processing device according to (12), The information processing device includes: An application screen or a partial area displayed on the image display device is selected; Generate and display a specific pattern covering the selected application screen or a partial area; A specific pattern photographed and read by the camera is recognized as a display area by the image display device; Supplying an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device; The information processing device controls an image display device so as to display an application screen image, which is an image of a supplied application screen or a partial area, in the recognized display area.

[0023] (14) An information processing device according to (13), The information processing apparatus is characterized in that the specific pattern that is generated and displayed is made semi-transparent.

[0024] (15) An information processing device according to (13), The information processing device is characterized in that it causes an image display device to generate a region frame indicating a range of a specific pattern.

[0025] (16) An information processing device according to (12), The information processing device includes: A display application switching software capable of switching between a plurality of applications is started and displayed on a display screen of the information processing device; The information processing device is characterized in that an icon of each application displayed on the activated and displayed display application switching software is selected, thereby selecting an application to be displayed on the image display device.

[0026] (17) An information processing device according to (12), The information processing device includes: An application screen or a partial area of ​​the application screen displayed on the image display device is selected, The display device is caused to recognize a display position and a display size of the application screen or a partial area photographed and read by the camera as a display area; Supplying an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device; controlling the image display device to display the supplied application screen image in the recognized display area; Furthermore, the information processing device Another application screen or another partial area of ​​another application screen displayed on the image display device is selected; The display device is caused to recognize a display position and a display size of another application screen or another partial area of ​​the other application screen photographed and read by the camera as another display area; supplying an application screen image, which is an image of another application screen or a partial area of ​​another application screen, to the image display device; The information processing device is characterized by controlling the image display device so as to display the supplied different application screen image in the recognized different display area.

[0027] (18) An information processing device according to (12), The information processing device is characterized in that, when a contrast mismatch between an application screen image to be inserted into the screen of the image display device and an image on the display screen of the image display device is identified by the information processing device or the image display device, the information processing device adjusts the image quality of the application screen so that the contrast is harmonious.

[0028] (19) An information processing device according to (18), The information processing device is characterized in that it adjusts the brightness of an application screen in accordance with an output from an illuminance sensor of an image display device that detects the brightness of the surrounding field of view.

[0029] (20) An information processing device according to (12), When an app screen image selected by the information processing device is displayed in the display area of ​​the image display device, the information processing device selects whether to display constantly or for each trigger notification, and if constant display is selected, controls the image display device to always display the app screen image, and if display for each trigger notification is selected, generates trigger notification information for starting or stopping the display, and controls the image display device to start or stop displaying the app screen image in accordance with the generated trigger notification information.

[0030] (21) An image display device having a communication link function between the image display device and an information processing device that operates and displays an application software, and a camera, The image display device includes: The screen of the information processing device selected by the information processing device is photographed by a camera and read. The display position and the display size of the screen of the read information processing device are recognized as a display area; The image display device is characterized in that an image on the screen of the information processing device selected by the information processing device is displayed in the recognized display area.

[0031] (22) The image display device according to (21), The image display device includes: An application screen or a partial area of ​​the application screen, which is a screen of the application selected on the information processing device, is photographed and read by a camera; The display position and size of the loaded application screen or a partial area are recognized as the display area, The image display device is characterized in that an application screen image, which is an image of an application screen or a partial area selected on an information processing device, is displayed in the recognized display area.

[0032] (23) The image display device according to (22), The image display device includes: A specific pattern is captured and read by a camera of the information processing device on the application screen or a partial area of ​​the application screen; The display position and size of the specific pattern that was loaded is recognized as the display area. The image display device is characterized in that an application screen image, which is an image of an application screen or a partial area selected on an information processing device, is displayed in the recognized display area.

[0033] (24) The image display device according to (23), The image display device is characterized in that a semi-transparent specific pattern generated and displayed by an information processing device is photographed and read by a camera.

[0034] (25) The image display device according to (23), The image display device is characterized in that it generates an area frame indicating the range of a specific pattern generated and displayed by an information processing device, and displays the area frame on a display screen of the image display device.

[0035] (26) The image display device according to (22), The image display device is characterized by displaying an application screen image, which is an image of the application screen or a partial area of ​​an application icon selected from the icons of applications displayed on display application switching software launched and displayed on an information processing device.

[0036] (27) The image display device according to (22), The image display device includes: The selected application screen or a partial area is photographed and read by a camera of the information processing device; The display position and size of the loaded application screen or a partial area are recognized as the display area, Displaying an application screen or an application screen image, which is an image of a partial area, in the recognized display area; Furthermore, the image display device The screen of the other application selected on the information processing device or a partial area of ​​the other application screen is photographed and read by a camera; Recognizing the display position and display size of the loaded different application screen or a different partial area of ​​the different application screen as a different display area; The image display device is characterized in that it displays, in the recognized separate display area, an application screen image which is an image of a separate application screen or a partial area of ​​the separate application screen selected on the information processing device.

[0037] (28) The image display device according to (22), The image display device is characterized in that, when an inconsistency in contrast between an application screen image inserted into the screen of the image display device and an image on the display screen of the image display device is identified by an information processing device or an image display device, the image display device displays an application screen image whose image quality has been adjusted by the information processing device so that the contrast is harmonized.

[0038] (29) The image display device according to (28), The image display device has an illuminance sensor that detects the brightness of the surrounding field of view, and displays an application screen image whose luminance has been adjusted by an information processing device in accordance with the output from the illuminance sensor.

[0039] (30) The image display device according to (22), When an app screen image selected on an information processing device is displayed in the display area of ​​the image display device, if constant display is selected on the information processing device, the image display device always displays the app screen image, and if display for each trigger notification is selected on the information processing device, the image display device starts or stops displaying the app screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device. Effect of the Invention

[0040] By using the technology of the present invention, it is possible to simply and conveniently display the screen of an app selected on an information processing device such as a smartphone at a suitable display position and size on the display screen of an image display device such as an HMD, thereby providing a high degree of freedom in selecting the suitable display arrangement, enabling convenient operability for the user, and further providing an information processing system, information processing device, and image display device in which the contrast between the display screen image of the information processing device inserted into the display screen of the image display device and the display screen image of the image display device can be harmonised.

[0041] Furthermore, problems, configurations and effects other than those described above will become apparent from the following description of the embodiments. [Brief description of the drawings]

[0042] [Figure 1A] 1A to 1C are diagrams for explaining examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention; [Figure 1B] 1A to 1C are diagrams for explaining examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention; [Figure 1C] 1A to 1C are diagrams for explaining examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention; [Figure 1D] 1A to 1C are diagrams for explaining examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention; [Figure 2A]1A to 1C are diagrams for explaining other examples of the external appearance of the information processing system, the information processing device, and the image display device according to the embodiment of the present invention; [Figure 2B] 1A to 1C are diagrams for explaining other examples of the external appearance of the information processing system, the information processing device, and the image display device according to the embodiment of the present invention; [Figure 2C] 1A to 1C are diagrams for explaining other examples of the external appearance of the information processing system, the information processing device, and the image display device according to the embodiment of the present invention; [Figure 3A] 11A to 11C are diagrams illustrating an example of a case where a partial area of ​​an application screen is selected in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 3B] 11A and 11B are diagrams illustrating an example of a case where an application displayed on the information processing device is selected by icon designation in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 4] 1 is an example of a flowchart illustrating basic operations of the information processing system, the information processing device, and the image display device according to the present embodiment. [Diagram 5] 10 is an example of a flowchart illustrating a display operation of multiple applications in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 6A] 11 is an example of a diagram for explaining the external schematic operation of an information processing device when displaying the left and right folding screens of a foldable smartphone in an information processing system, an information processing device, and an image display device according to the present embodiment. FIG. [Figure 6B] 11 is an example of a diagram for explaining the external schematic operation of an information processing device when displaying the left and right folding screens of a foldable smartphone in an information processing system, an information processing device, and an image display device according to the present embodiment. FIG. [Figure 7A] 11A to 11C are diagrams illustrating an example of contrast harmonization processing between a screen image of an information processing device and a screen image of an image display device in an information processing system, an information processing device, and an image display device according to the present embodiment. [Figure 7B]11A to 11C are diagrams illustrating an example of contrast harmonization processing between a screen image of an information processing device and a screen image of an image display device in an information processing system, an information processing device, and an image display device according to the present embodiment. [Figure 8] 11 is an example of a flowchart illustrating an image adjustment process when the contrast of a screen image of the information processing device and that of the image display device are matched in the information processing system, the information processing device, and the transmissive image display device according to the present embodiment. [Figure 9] 11 is an example of a flowchart illustrating an image adjustment process when the contrast of a screen image of the information processing device and that of the image display device are matched in the information processing system, the information processing device, and the shield-type image display device according to the present embodiment. [Figure 10] 11 is an example of a flowchart illustrating a process for performing constant display or display for each trigger notification in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 11A] 1 is a block diagram showing an example of the configuration of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 11B] 1 is a block diagram showing an example of the configuration of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 12A] 11A to 11C are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned by a position selection application in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 12B] 11A to 11C are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned by a position selection application in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 12C] 11A to 11C are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned by a position selection application in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 12D]11A to 11C are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned by a position selection application in the information processing system, the information processing device, and the image display device according to the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] An example of an embodiment of the present invention will be described below with reference to the drawings. In this embodiment, the following technologies are provided to ensure a comfortable life and convenience for users with highly advanced technology. The realization of this linkage system for video information devices will contribute to "9. Build resilient infrastructure, promote inclusive and sustainable industrialization and promote innovation" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0044] The information processing system has an image display device equipped with a camera and an information processing device that operates and displays an app (abbreviation for application software), and has a function of communicating and linking information between the image display device and the information processing device. A head mounted display (HMD, hereinafter referred to as HMD) is used as a specific example of the image display device, and a portable smartphone (abbreviation for smartphone) is used as a specific example of the information processing device. HMDs include a type that displays virtual objects on a display unit while visually confirming real objects in front of the user through a lens (so-called optical see-through type), and a type that displays images of real objects in real space captured by a camera, and content such as movies, music, and games on a display unit provided in front of the head (so-called video see-through type, occlusion type), and the user can see the displayed real objects and virtual objects up close.

[0045] 1A to 1D are examples of diagrams for explaining the appearance of an example of an information processing system, an information processing device, and an image display device according to the present embodiment. FIG. 1A shows the smartphone 100 on the display surface side on which an application stored in the smartphone 100 is running and displayed, FIG. 1B shows the smartphone 100 held by the user 10 and the HMD 200 with information communication linkage as indicated by a two-dot chain line 128, and FIG. 1C and FIG. 1D show the display screen 130 of the HMD 200. Hereinafter, a case will be described in which an application screen (hereinafter, referred to as an application screen. That is, the application screen refers to, for example, the screen of an application displayed on an information processing device such as the smartphone 100) on the display screen 101 of the smartphone 100 is inserted as a virtual object and displayed on the display screen of the HMD 200, and the application screen of the smartphone 100 can be viewed on the HMD 200 together with the surrounding scenery or image content.

[0046] 1A, the smartphone 100 displays an app screen 102 of a map app, which is an example of an app, and an app screen 103 of a weather app on a display screen 101. When selecting, for example, the app screen 102 of a map app as the app screen to be inserted and displayed on the HMD 200, the user 10 presses and holds or touches (hereinafter referred to as touch) the app screen 102 of the map app with his / her finger, thereby selecting the app screen 102 of the map app as the app screen to be inserted and displayed on the HMD 200. In this way, the information processing device selects a screen of the information processing device to be displayed on the image display device by a user input.

[0047] Furthermore, a specific pattern 104 indicated by a diagonal dotted line may be generated and displayed on the application screen 102 of the selected map application. The displayed specific pattern 104 indicates a display area of ​​the application screen to be inserted and displayed on the HMD 200 when the user 10 wearing the HMD 200 views the specific pattern 104 on the display screen of the smartphone 100 at a suitable position. The method of selecting the application screen to be inserted and displayed on the HMD 200 may be a touch such as a deep press or a strong press, or an operation via the operation input interface of the smartphone 100. The specific pattern may be made semi-transparent. In this case, the original application screen can be viewed while the specific pattern is displayed, improving convenience.

[0048] As shown in FIG. 1B, when the screen of the smartphone 100 enters the display screen of the HMD 200 worn by the user 10, the camera 121 of the HMD 200 captures and captures the specific pattern 104 of the smartphone 100. In the HMD 200, the specific pattern 104 moving in the up, down, left, right, front and back directions as shown in 122 to 127 is captured and read by the camera 121, and the position of the read specific pattern 104 is recognized as the display position of the application screen displayed on the display screen of the HMD 200. Here, 126 and 127 are cases where the specific pattern 104 moves in the perspective direction, and when the specific pattern 104 is captured and read by the camera 121, the display size can be enlarged or reduced. Then, the size of the display area of ​​the read specific pattern 104 is recognized as the display size of the application screen displayed on the display screen of the HMD 200. In this way, the image display device captures and reads the screen of the selected information processing device by the camera 121 when the user 10 holds it over the camera 121, and recognizes the display position and display size of the screen of the read information processing device as the display area.

[0049] Fig. 1C shows the display seen in the HMD when the screen of the smartphone 100 is captured by the HMD 200. The upper part of Fig. 1C shows the screen of the smartphone 100 captured by the camera mounted on the HMD 200 entering the display range of the HMD 200. The smartphone 100 displays a specific pattern 104 on top of the application screen 102 of the selected map application.

[0050] In the HMD200, the screen displayed on the smartphone 100 is captured by the camera 121, and an area frame 131 indicating the area of ​​the read specific pattern 104 is generated as shown in the middle of FIG. 1C. Note that the smartphone 100 may generate a thick line or the like indicating the area in advance in conjunction with the specific pattern 104. The generated area frame 131 is displayed on the display screen 130 of the HMD200 in accordance with the movement of the smartphone 100 on which the specific pattern 104 is displayed in the up, down, left, right, and near directions. The user 10 moves the smartphone 100 in the up, down, left, right, and near directions while watching the area frame 131 indicating the area of ​​the specific pattern, and determines a display position and display size suitable for the HMD display of the app screen. Then, as shown in the lower part of FIG. 1C, an image 132 of the app screen of the smartphone 100 is displayed at a predetermined position in the screen of the HMD200 with a predetermined size.

[0051] In this way, by moving the display area frame 131 while viewing the screen of the smartphone 100 through the field of view from the HMD 200, it becomes easy to confirm while making fine adjustments when deciding "to display at this position and at this size". In addition, when measuring the distance up, down, left, right, and near, the distance between the HMD 200 and the smartphone 100 may be measured by a distance sensor mounted on the HMD 200, and the results may be used together. In this way, the image display device captures and reads the screen of the selected information processing device by the camera 121 when the user 10 holds it over a suitable position.

[0052] Here, it is advisable to store the correspondence between the image captured by the camera mounted on the HMD200 and the area frame displayed on the screen of the HMD200 in advance by adjusting the product at the time of shipment. In addition, if the screen of the smartphone 100 is not facing directly in front of the HMD200, the screen of the smartphone 100 will appear diagonal. In such a case, the shape of the smartphone 100 is grasped to recognize the inclination of the smartphone, and the screen displayed on the HMD200 is corrected so that it is displayed in the same shape as when the screen is viewed from the front. By using such a method, it is possible to display the smartphone screen in the appropriate position and correct the distortion of the screen.

[0053] Next, FIG. 1D illustrates a case where the display position and display size suitable for display are different from those in FIG. 1C. The user 10 can arrange the smartphone 100 at any display position and any display size within the screen of the HMD 200 by adjusting the position of the smartphone 100 up, down, left, right, and near. Since the area frame 131 is displayed on the display screen of the HMD 200, the user can freely and easily determine the display position and display size suitable for inserting and displaying the app screen, for example, a place that does not interfere with the image within the display screen, while easily viewing the image within the display screen. Note that the display position within the screen of the HMD 200 may be determined by a specific pattern without the area frame.

[0054] Here, as the operations shown in the lower part of FIG. 1C and the lower part of FIG. 1D, the HMD 200 transmits information on the display area determined by the determined display position and display size to the smartphone 100. The smartphone 100 receives the display area information, generates image data of the application screen to be displayed within the display area, and transmits the data to the HMD 200. The HMD 200 displays the received image of the application screen (simply referred to as an image or application screen image. That is, the application screen image refers to, for example, the screen of the application displayed on an image display device such as the HMD 200), for example, the image 132 of the application screen 102 of the map application, in the determined display area. In this manner, the image display device displays an image of the screen of the selected information processing device in the recognized display area. Note that the application screen initially selected by the smartphone 100, for example, the image 132 of the application screen 102 of the map application, may be displayed as is, or the resolution of the image of the application screen 102 of the map application may be converted from the number of pixels and resolution of the HMD 200. In addition, if there is a setting for a display screen for output to another device, the image data may be transmitted.

[0055] In addition, after the display position and display size in the HMD 200 are determined, the user 10 may select whether the display position of the image of the app screen of the smartphone 100 is displayed at the same position within the frame of the display screen 130 of the HMD 200 or is displayed as if it is attached to a position fixed to the background position (the upper right of the tower in FIG. 1C). In this case, when the user 10 moves his / her head, in the former case, the app screen image of the smartphone 100 is displayed at the same position on the screen of the HMD 200, and in the latter case, the image moves within the screen of the HMD but is displayed at a position where the positional relationship with the background does not change. Furthermore, when the user 10 moves his / her head or the surrounding scenery changes and it becomes necessary to move the display position of the app screen image, the position of the app screen image displayed on the HMD can be moved by an operation on the screen of the HMD 200, for example, a gesture operation to grab and move the app screen image displayed on the HMD 200, or an operation to select and move the app screen image by gaze.

[0056] As described above, when displaying the application screen of the smartphone 100 on the display screen of the HMD 200, the smartphone 100 can be moved up, down, left, right, or closer to or farther away while held over the HMD 200, thereby allowing the display position and display size that define the display area of ​​the application screen to be determined with a high degree of freedom. In addition, after setting the display position and display size of the application screen, the application screen of the smartphone 100 can be viewed on the display screen of the HMD 200 even if the smartphone 100 is out of the field of view, such as when the smartphone 100 is in a pocket or bag, or when the smartphone 100 is placed in a position that is out of reach. Alternatively, the application of the smartphone 100 that has been set to be displayed on the screen of the HMD 200 may be erased from the application screen, and the remaining application of the smartphone 100 may be expanded and displayed on the screen. Furthermore, not only can the app screen of smartphone 100 be constantly displayed on the display screen of HMD 200, but for example, in the case of a telephone app, when an incoming call is received, it is also possible to display the app screen of smartphone 100 at a determined display position and size on the display screen of HMD 200.

[0057] In this embodiment, an app on smartphone 100 has been used as an example, but this is not limited to smartphone 100, and can be anything that can be connected to and linked with other devices, such as an app on a smart watch or the screen of a car navigation system.

[0058] Next, a case where a part of the application screen of the smartphone 100 is selected and displayed on the HMD 200 will be described with reference to FIGS. 2A to 2C. In addition, in FIGS. 2A to 2C, the parts shown in FIGS. 1A to 1D and denoted with the same reference numerals have the same operations as those already described in FIGS. 1A to 1D, so detailed description thereof will be omitted. FIGS. 2A, 2B, and 2C correspond to FIGS. 1A, 1B, and 1C, respectively. FIG. 2A shows the smartphone 100 on the display surface side on which an application stored in the smartphone 100 is running and displayed. FIG. 2B shows a state in which the user 10 holds the smartphone 100 in his hand and wears the HMD 200 on his head, which is connected to the smartphone 100 for information communication. FIG. 2C shows an example of the screen of the smartphone 100 seen through the HMD 200 and the display screen 130 of the HMD 200.

[0059] In FIG. 2A, when the screen to be inserted and displayed on the HMD 200 is a part of the app screen, for example, an area 201 showing tomorrow's weather in the app screen 103 of a weather app, the user 10 first touches the app screen 103 of the weather app with his / her finger to select the weather app, and then selects the area 201 showing tomorrow's weather, which is a part of the app screen 103 of the weather app, to perform partial area designation. Alternatively, the area 201 showing tomorrow's weather, which is a part of the app screen 103 of the weather app, may be surrounded with a fingertip to perform partial area designation. By these area designations, a partial app display screen to be inserted and displayed on the HMD 200 is selected. For example, the area 201 showing tomorrow's weather in the app screen 103 of the weather app is selected as shown in the upper part of FIG. 2A. In this way, the smartphone 100 may select a partial area of ​​the app screen. Furthermore, as shown in the lower part of FIG. 2A, a specific pattern 202 shown by a diagonal dotted line is generated and displayed on the screen of the selected partial app, for example, on the area 201 showing tomorrow's weather.

[0060] When the smartphone 100 enters the display screen of the HMD 200, the camera 121 of the HMD 200 captures the display of the specific pattern 202 as shown in FIG. 2B. In the HMD 200, the specific pattern 202 moving in the up / down / left / right / front / back directions 122-127 is captured by the camera 121 and read, and the position and size of the read specific pattern 202 are recognized as the display position of the application screen displayed on the display screen of the HMD 200. In particular, in the HMD 200, the specific pattern 202 moving in the front / back directions, i.e., the perspective directions 126 and 127, is captured by the camera 121 and read, and the size of the read specific pattern 202 is recognized as the display size of the application screen displayed on the display screen of the HMD 200. Furthermore, in the HMD 200, an area frame 203 indicating the range of the read specific pattern 202 is generated.

[0061] A specific pattern 202 is displayed on the display screen 130 of the HMD 200 as shown in the upper part of FIG. 2C. At this time, an area frame 203 of the same size as the area of ​​the specific pattern captured by the camera 121 is displayed in accordance with the movement of the smartphone 100 in the up, down, left, right, perspective directions as shown in FIG. 2B. The user 10 moves the smartphone 100 in the up, down, left, right, perspective directions while looking at the area frame 203 indicating the range of the specific pattern, and determines a display position and display size suitable for the HMD display of a partial screen of the app. Once the display position and display size are determined, information on the screen display of a part of the displayed app is obtained from the smartphone 100 and displayed on the screen of the HMD 200 as shown in FIG. 2C.

[0062] The information regarding the display image is the same as that described in FIG. 1, and the HMD 200 transmits information regarding the determined display position and display size to the smartphone 100, and the smartphone 100 generates image data of a partial application screen to be displayed on the HMD 200 according to the acquired information regarding the display area, and transmits the image data to the HMD 200. The HMD 200 displays a partial application screen image from the received image data, for example, an image 204 of an area 201 showing tomorrow's weather, which is a partial screen of a weather application, in a specified display area. Thus, as in the case of FIG. 1, a partial screen of the application can be displayed in a specified display area of ​​the display screen of the HMD 200, and the same effect as in the case of FIG. 1 can be obtained. In this way, an image of a partial area of ​​the application screen is also called an application screen image. In other words, an application screen image that is a partial area refers to, for example, an image of a selected partial area of ​​an application screen displayed on an image display device such as the HMD 200.

[0063] 3A shows an example of a method for specifying a partial area by encircling a part of an application screen with a fingertip. As shown in FIG. 3A, the vicinity of a tower, which is a part of the application screen 102 of a map application, is surrounded with a fingertip, and an enclosed area 301 is selected. Furthermore, a specific pattern 302 shown with a diagonal dotted line is displayed on the enclosed area 301. Alternatively, the enclosed area may be displayed with a frame such as a thick line. The operations thereafter are the same as those shown in FIG. 1 and FIG. 2.

[0064] FIG. 3B is an example of a diagram for explaining a case where an application to be displayed on the HMD 200 is selected using a display application switching software capable of switching between multiple applications. As shown in FIG. 3B, the display application switching software 305 capable of switching between multiple applications is started on the smartphone 100 and displayed on the display screen. If an application screen on the smartphone 100 is directly pointed to, it may result in an operation related to the operation of the application (such as designating a specific location in a map application), and the operation is likely to be confused. Therefore, the display application switching software 305 is started, and an icon that is a reduced display of multiple applications displayed thereon, for example, one of the icon 306 of the map application or the icon 307 of the weather application, is touched with a fingertip to select an application to be displayed on the HMD 200. In this case, the operation of selecting an application can be visually and easily performed. Note that the operation of selecting an application icon may be performed by other methods than touching with a fingertip, such as inputting the application name by voice.

[0065] Fig. 4 is an example of a flowchart for explaining the basic operation of the information processing system, information processing device, and image display device according to this embodiment. In Fig. 4, on the right side of the smartphone 100, an application to be displayed on the display screen of the HMD 200 is selected, and if necessary, a partial screen (partial area) in the application screen is specified (S401). Furthermore, specific patterns 104, 202 are displayed on the screen of the selected application screen or a partial area of ​​the specified application screen (S402). On the other hand, on the HMD 200 side, a mode is started to wait for reading of the specific patterns 104, 202 displayed on the smartphone 100 (S403). In this state, when the application screen of the smartphone 100 comes into view within the screen of the HMD 200 (S404), the HMD 200 photographs and recognizes the application screen using the camera 121 (S405), reads the specific patterns 104 and 202 on the application screen, and displays area frames 131 and 203 indicating the ranges of the specific patterns 104 and 202 on the display screen of the HMD 200 (S406). Furthermore, the user 10 adjusts the display position and size by moving the smartphone 100 while looking at the displayed area frames 131 and 203. When a suitable display position and display size for displaying the application screen or a screen of a partial area of ​​the application screen within the HMD screen is determined, information of the HMD 200 regarding the display area determined by the display position and display size is transmitted to the smartphone 100 (S407). The smartphone 100 acquires the display area information and stops displaying the specific patterns 104 and 202 (S408). Next, the user pinches in and out on the app screen of smartphone 100 as necessary to enlarge or reduce the app screen displayed on HMD 200 (S409). Then, image data of the app screen generated to match the display size is sent to HMD 200 (S410). On the HMD 200 side, image data of the app screen is received and acquired (S411), and the app screen image of smartphone 100 is displayed in a specified display area (S412). (Here, if the text on the app screen image of smartphone 100 displayed on HMD 200 is too small to see or if the necessary information is not included, the operation of S409 is repeated (not shown).)

[0066] Next, the user 10 selects whether to display the application screen image on the HMD screen at all times on the smartphone 100 side or to display it upon a predetermined trigger (S413). When all-time display (N in S413) is selected, image data for display is continuously sent from the smartphone 100 to the HMD 200. When a temporary stop of the HMD image display is set (Y in S413), the smartphone 100 transmits display stop information of the application screen image to the HMD 200 (S414). When the HMD 200 receives image display stop information of the application screen (S415), it stops displaying the application screen image on the HMD screen (S416). Here, if the user 10 wants to stop displaying the application screen image after all-time display is selected, the smartphone 100 is instructed to stop displaying the application screen image by an operation from the HMD 200 (not shown).

[0067] The above sequence makes it possible to easily display the app screen image of smartphone 100 at a suitable display position and size within the display screen of HMD 200. Note that the portion surrounded by dotted line 400 indicates the processing sequence within smartphone 100, and the portion surrounded by dotted line 402 indicates the processing sequence within HMD 200. Also, the portion surrounded by dotted line 403 indicates the processing sequence up to the setting of the display of the app screen image within the HMD screen on the smartphone 100 side, and the portion surrounded by dotted line 404 indicates the processing sequence up to the setting of the display of the app screen image within the HMD screen on the HMD 200 side.

[0068] FIG. 5 shows an example of a case where a plurality of application screen images are displayed on the HMD screen in the information processing system, information processing device, and image display device according to the present embodiment. Here, a case where two applications, a first and a second, are displayed on the HMD screen will be described. In FIG. 5, the parts shown in FIG. 4 and denoted with the same reference numerals have the same operations as those already described in FIG. 4, and therefore detailed description thereof will be omitted. In FIG. 5, the processing sequence for displaying the first application screen image of the smartphone 100 on the HMD screen is the same as that described in FIG. 4, and is performed in the processing sequence of the part surrounded by dotted lines 403 and 404, and the first application screen image of the smartphone 100 is displayed in a suitable display area on the HMD screen. Next, when it is set on the smartphone 100 side to additionally display another second application screen image on the HMD screen (Y in S501), the processing sequence of the part surrounded by dotted lines 405 and 406, which is the same processing sequence as the processing sequence of the part surrounded by dotted lines 403 and 404, is executed. Then, in addition to the first application screen image of the smartphone 100, the second application screen image of the smartphone 100 is displayed in a display area suitable for display within the HMD screen. Here, the smartphone 100 selects whether to display both the first and second application screen images within the HMD screen at all times or at a predetermined opportunity (S502). When all-time display (N in S502) is selected, image data for display is continuously sent from the smartphone 100 to the HMD 200. When the stop of the HMD image display is set (Y in S502), the smartphone 100 transmits display stop information for both application screen images to the HMD 200 (S503). When the HMD 200 receives display stop information for both application screen images (S504), it stops displaying both the first and second application screen images of the smartphone 100 on the HMD screen (S505). Therefore, the above sequence allows both the first and second app screen images of the smartphone 100 to be displayed easily and conveniently at a suitable display position and size on the display screen of the HMD. In the case of two or more multiple apps, the HMD display process of the second app screen image can be repeated continuously. In addition, although an example in which the two apps are selected to be displayed either constantly or by occasion is shown here, a different setting may be selected for each app.In that case, for example, the display method for the first app may be selected before S501.

[0069] 6A and 6B illustrate a case where application screens of a plurality of partial regions of the smartphone 100 are displayed on the HMD screen, in the case of a foldable smartphone 600 whose display screen can be folded. Note that in Figs. 6A and 6B, the parts shown in Figs. 1A to 1D and 2A to 2C and denoted with the same reference numerals have the same operations as those described in Figs. 1A to 1D and 2A to 2C, and therefore detailed description thereof will be omitted.

[0070] When the left screen 601 of the left half of the foldable smartphone 600 enters the display screen of the HMD 200, the camera 121 of the HMD 200 captures an image of a specific pattern 602 displayed on the left screen 601, as shown in Fig. 6A. In the HMD 200, the camera 121 captures and reads the image of the specific pattern 602 moving in the up / down, left / right, front / back directions 122 to 127, and the position of the read specific pattern 602 is recognized as the display position to be displayed on the display screen of the HMD 200.

[0071] Next, in FIG. 6B, the size of the read specific pattern 602 is recognized as the display size of the left screen 601 displayed on the display screen of the HMD 200 (upper left of FIG. 6B). Furthermore, the HMD 200 generates an area frame 603 indicating the range of the read specific pattern 602 (middle left of FIG. 6B). The user 10 moves the foldable smartphone 600 in the up, down, left, right, near and far directions while looking at the area frame 603 displayed on the HMD 200, and determines a suitable display position and display size at which the left screen 601 of the foldable smartphone 600 is displayed on the HMD. The HMD 200 transmits information on the display area determined by the determined display position and display size to the foldable smartphone 600, and the foldable smartphone 600 generates image data to be displayed in the acquired display area and transmits it to the HMD 200. The HMD 200 displays the image 604 of the left screen 601 in the determined display area based on the received image data.

[0072] Further, an example is shown in which the right screen 605 of the right half of the foldable smartphone 600 is displayed at a different position. Here, similar to the case of the left screen 601, the HMD 200 displays an area frame 607 indicating the range of the specific pattern 606 that has been read. While looking at the area frame 607, the user 10 moves the foldable smartphone 600 in the up, down, left, right, near and far directions to determine a suitable display position and display size at which the right screen 605 of the foldable smartphone 600 is displayed on the HMD 200. The HMD 200 displays an image 608 of the right screen 605 of the foldable smartphone 600 in a display area determined by the determined display position and display size. This operation makes it possible to display the left and right halves of the screen at different positions on the screen of the HMD 200 when the foldable smartphone 600 is folded.

[0073] Note that Figures 6A and 6B show an example of two app screens on a foldable smartphone 600, but it goes without saying that the same effect and advantages can be obtained by displaying multiple app screens on multiple smartphone 100 screens.

[0074] Next, contrast harmonization processing between the application screen image displayed on the HMD screen and the screen of the HMD 200 will be described with reference to Figures 7A and 7B. If the contrast between the application screen image displayed on the HMD screen and the image on the display screen of the HMD 200 becomes unbalanced by reading in a specific pattern, problems will occur such as the inserted application screen image (also called the application insertion screen image) becoming difficult to see, or the application insertion screen image being overemphasized, making it difficult to draw attention to the image on the HMD screen. Here, a see-through HMD will first be described as an example.

[0075] FIG. 7A is an example diagram illustrating contrast harmony processing for cases where the surroundings are bright, such as outdoors, and the app screen image is difficult to see, while FIG. 7B is an example diagram illustrating contrast harmony processing for cases where the surroundings are dark, such as at night, and the contrast with the app screen image is too strong.

[0076] As shown in FIG. 7A, in a bright environment such as outdoors, the image 802 of the app screen may be relatively dark compared to the scenery 801 around the HMD 200 that appears bright through the lens. In such a case, the app screen image becomes difficult to distinguish due to the brightness of the surroundings. In response to this, the HMD 200 is provided with an illuminance sensor for detecting the brightness or darkness of the surroundings, and is thereby capable of detecting the brightness around the display position where the app screen image is displayed. Alternatively, if the HMD 200 is equipped with a camera 121 that captures the surroundings, it is possible to detect the brightness distribution from the image information captured by the camera 121. The average brightness around the HMD or the brightness around the display position of the image 802 of the app screen is detected from the brightness information, and the detected surrounding brightness information is transmitted to the smartphone 100. The smartphone 100 increases the brightness as an image quality adjustment of the app insertion screen image according to the received surrounding brightness information, and displays the image 803 of the app screen with increased brightness on the HMD. This makes the contrast between the app screen image and the surroundings of the HMD clear, and eliminates or reduces the problem of the app screen image appearing dark and unclear.

[0077] Also, as shown in FIG. 7B, in a dark environment such as at night, the image 805 of the app screen may appear relatively bright compared to the surrounding scenery that appears dark through the lens, and the app screen image may be overemphasized. In this case as well, the illuminance sensor of the HMD 200 can detect the brightness (darkness) of the surroundings and adjust the display of the app screen image at an appropriate brightness. In addition, if the HMD 200 is equipped with a camera 121, it is possible to detect the brightness distribution from the image information captured by the camera 121. From the brightness information, the average brightness around the HMD or the brightness around the position where the image 805 of the app screen is displayed is detected, and the detected surrounding brightness information is transmitted to the smartphone 100. In the smartphone 100, the brightness is lowered as an image quality adjustment of the app insertion screen image according to the received surrounding brightness information, and the image 806 of the app screen is displayed on the HMD 200. This harmonizes the contrast between the app insertion screen image and the HMD display screen image, eliminating or reducing the overemphasis of the app insertion screen image, and reducing eye fatigue caused by strong contrast. Furthermore, as an image quality adjustment for the application insertion screen image, the color tone may be adjusted to reduce the emphasis of the application insertion screen image.

[0078] As described in FIG. 7A and FIG. 7B, when the smartphone 100 and the HMD 200 are not linked as described above, that is, when the smartphone 100 is alone, the smartphone 100 adjusts the brightness of the application screen image according to the surrounding environment. However, when the smartphone 100 and the HMD 200 are linked as described above, the brightness of the application screen image can be adjusted according to the surrounding environment of the HMD 200, thereby ensuring clarity and reducing eye fatigue. In addition to the method described in FIG. 7A and FIG. 7B, the brightness level of the application screen image to be displayed from the HMD 200 may be transmitted to the smartphone 100 and requested. In addition, when the application screen image is displayed on the HMD 200 side, the brightness and color tone of the application screen image may be adjusted and displayed.

[0079] FIG. 8 is an example of a flowchart for explaining the contrast harmony process between the application display screen and the scenery around the HMD 200 described in FIG. 7A and FIG. 7B. In FIG. 8, the parts with the same reference numerals as those in FIG. 4 and FIG. 5 have the same operations as those already described in FIG. 4 and FIG. 5, so detailed explanations of them will be omitted. In FIG. 8, the part surrounded by the dotted line 403 is a processing sequence from the selection of an application to the transmission of image data of the application screen of the smartphone 100 on the side of the smartphone 100, and the part surrounded by the dotted line 404 is a processing sequence from the reading of a specific pattern to the display of an application screen image on the side of the HMD 200, which is the same as the explanation in FIG. 4. The HMD 200 displays the application screen image as shown in the part surrounded by the dotted line 404, and detects brightness by the illuminance sensor, or captures the surroundings with the camera 121 and acquires brightness from the surrounding image of the HMD 200 (S801). Next, the brightness of the application screen image and the surroundings of the HMD 200 are compared (S802). If either one is too bright or too dark (N in S803), the illuminance sensor of the HMD 200 detects the brightness of the surroundings, or the camera 121 of the HMD 200 captures the surroundings, analyzes the brightness information, and transmits information indicating the brightness of the surroundings of the HMD 200 to the smartphone 100 (S804). When the smartphone 100 receives the information on the brightness of the surroundings (S805), if the surroundings are brighter than a predetermined threshold value for the application screen image based on the received information on the brightness of the surroundings (S806), the smartphone 100 increases the brightness as an image quality adjustment of the application screen to be displayed (S807). On the other hand, if the surroundings are darker than a predetermined threshold value for the application screen image (S806), the smartphone 100 decreases the brightness as an image quality adjustment of the application screen to be displayed, or adjusts the color tone to reduce emphasis (S808). Furthermore, the image data of the application screen after the image quality adjustment is transmitted to the HMD 200 (S809). The HMD 200 acquires image data of the application screen after image quality adjustment (S810), and displays the application screen image after image quality adjustment on the HMD 200 (S811). The application screen image is compared with the brightness around the HMD 200, and if the application screen image is clearly visible or is not extremely bright (Y in S803), the application screen image is displayed as is on the HMD display screen (S812).

[0080] The above sequence allows the image on the app screen to be displayed at an appropriate brightness relative to the brightness around the HMD 200, allowing the user 10 to view the image on the app screen within the HMD 200 without feeling fatigued or experiencing contrast-related difficulties.

[0081] Here, in FIG. 8, the case where the HMD 200 is a see-through HMD is described. Next, the case where a video see-through type HMD is described, in which a real object or a virtual object in the real space is photographed by a camera 121 provided in front of the head and the photographed object is displayed on a display in front of the eyes in a shielded HMD. First, a specific pattern of the smartphone 100 in the field of view in front is recognized, and an application screen image is inserted and displayed at a suitable display position and display size in the display screen of the HMD 200. Then, when switching the display from the surrounding screen display by the camera 121 of the HMD 200 to video content supplied within the HMD 200 or via an external device or network, the application screen image may be synthesized with a part of the video content and displayed on the display in front of the eyes. In this case, depending on the atmosphere of the video content or the purpose of the inserted application screen image, if the contrast between the image of the video content information and the application screen image becomes unbalanced, problems such as the application screen image being focused on becoming difficult to see or the application screen image being overemphasized, destroying the atmosphere of the image of the video content, may occur. In response to this, the content of the video content, for example, the brightness and type of the video content (whether it is something that you do not want to be disturbed by, such as a movie or music video, or not, etc.) are detected. The HMD 200 also analyzes the importance of the display content of the application screen image based on the type of application, etc., and adjusts the image quality (brightness, and if necessary, color tone) of the application screen image to be inserted. For example, when you want to always pay attention to the application screen image, the image quality (brightness, and if necessary, color tone) of the application screen image is adjusted so that the application screen image stands out, and the application screen image is inserted and displayed on the HMD display screen. Alternatively, when you want to concentrate on the video content, the image quality (brightness, and if necessary, color tone) of the application screen image is adjusted so that the application screen image does not stand out, and the application screen image is inserted and displayed on the HMD display screen.

[0082] In this way, by adjusting the image quality of the application screen image to be inserted according to the image content of the video content information, it is possible to achieve contrast harmony between the video content information image and the application screen image in a form corresponding to the attention importance of the video content information image or the application screen image. As a specific example, if the application screen image is information that should attract attention, such as emergency information, it can be displayed prominently, and if it is merely additional information, such as the time or weather forecast, it can be displayed in an inconspicuous manner.

[0083] In addition, a request may be made to the smartphone 100 to adjust the image quality of the application screen image displayed on the HMD 200, or the image quality of the application screen image may be adjusted and displayed when the application screen image is displayed on the HMD 200 side.

[0084] FIG. 9 is an example of a flowchart for explaining the contrast harmony process between the application screen image and the HMD display screen image described above. In FIG. 9, the same reference numerals as those in FIG. 4 and FIG. 5 have the same operations as those already explained in FIG. 4 and FIG. 5, and therefore detailed explanations thereof will be omitted. In FIG. 9, the part surrounded by the dotted line 403 is a processing sequence from the selection of an application on the smartphone 100 side to the transmission of image data of the application screen of the smartphone 100, and the part surrounded by the dotted line 404 is a processing sequence from the reading of a specific pattern on the HMD 200 side to the display of the application screen image, which is the same as the explanation in FIG. 4. In the video see-through mode of the shielded HMD, the surroundings are photographed and displayed by the camera 121 mounted on the HMD 200 (S901). In addition, the application screen image is displayed by combining it with the camera-photographed image as shown in the part surrounded by the dotted line 404. In this state, by checking whether the contrast is harmonized (S902), it is checked whether the application screen image is inserted into the HMD display screen without difficulty in viewing or discomfort. If the contrast of both the application screen image and the HMD display screen image is not in harmony (N in S903), the brightness distribution around the application screen image of the content to be displayed on the HMD is analyzed to obtain brightness information (S904). Next, the brightness distribution of the application screen image is analyzed to obtain brightness information (S910). Then, according to the brightness information around the application screen image on the HMD 200 side, if the surroundings are bright with respect to the application screen image, the brightness is increased as the image quality adjustment of the application screen image to be inserted. Also, if the surroundings are dark with respect to the application screen image, the brightness is decreased as the image quality adjustment of the application screen image to be inserted, or the color tone is adjusted so as to reduce emphasis. Then, the application screen image after the image quality adjustment is inserted and displayed on the HMD display screen (S911). If the contrast of both images is in harmony (Y in S903), the application screen image is inserted and displayed as it is on the HMD display screen (S912). Through the above sequence, it is possible to realize a harmony of contrast between the application screen image and the HMD display screen image, and it is possible to reduce the discomfort and fatigue caused by the strong contrast for the user 10.In addition, after loading an app screen in the HMD's video see-through mode, if the user switches the HMD mode and displays content such as movies or music videos on the HMD, the processing from S902 onwards adjusts the contrast so that it harmonizes with the content.

[0085] Next, a process of whether to display the application screen image of the smartphone 100 inserted and displayed in the HMD screen on the HMD display screen at all times or at every trigger notification will be described with reference to FIG. 10. FIG. 10 is an example of a flowchart showing a process in which the application screen image is displayed at all times or at every trigger notification in the information processing system, information processing device, and image display device according to this embodiment. In FIG. 10, the same reference numerals as those in FIG. 4 and FIG. 5 have the same operations as those already described in FIG. 4 and FIG. 5, and therefore detailed description thereof will be omitted. In FIG. 10, the user 10 selects whether to display the application screen image at all times or at every trigger notification on the smartphone 100 (S1001). Display form selection information representing the selected display form is transmitted to the HMD 200 (S1002). The HMD 200 receives the display mode selection information (S1003), and when the application screen image to be inserted is to be displayed for each trigger notification (S1004 below), the application screen image on the HMD screen is hidden (S1005), and is in a state of waiting for a notification to start display for each trigger notification to arrive from the smartphone 100. Also, on the smartphone 100 side, when the application screen image is to be displayed for each trigger notification (S1006), when a notification to start display for each trigger notification is issued (S1007), display start notification information indicating the start of display for each trigger notification is transmitted to the smartphone 100 (S1008). The HMD 200 receives the display start notification information (S1009) and starts displaying the application screen image (S1010). After that, when a notification to stop display for each trigger notification is issued from the smartphone 100 (S1011), display stop notification information indicating the stop of display for each trigger notification is transmitted to the smartphone 100 (S1012). The HMD 200 receives the display stop notification information (S1013) and stops displaying the application screen image (S1014). On the other hand, the HMD 200 receives the display mode selection information, and if the application screen image to be inserted and displayed is always displayed (left of S1004), the application screen image is always displayed within the HMD screen (S1015). Note that the display of the application screen on the smartphone 100 may be made non-displayable as necessary.

[0086] Also, when erasing the application screen image after displaying it for a while, an operation may be performed to return the application in the application screen image to the application screen again, and the application screen may be displayed separately on the application screen. For example, as in the case of performing the operation of capturing the application screen described in FIG. 1 and FIG. 2, the application screen displayed on the HMD 200 may be returned to the screen display of the smartphone 100 itself by performing an operation of overlaying the actual smartphone 100 as a background so as to overlap with the application screen image displayed on the HMD 200. If such an operation is specified, the user 10 can easily understand the operation because the application screen can be displayed on the HMD 200 or returned to the smartphone 100 with the same operation.

[0087] With the above sequence, when displaying for each trigger notification, the application screen image can be displayed on the HMD display screen only when necessary, and can be prevented from being a hindrance when not necessary, according to the trigger of the smartphone 100. For example, in the case of a phone application, the application screen image can be displayed on the HMD screen when an incoming call is received, and the HMD display of the application screen image can be stopped when the call ends. In addition, by setting and understanding whether the smartphone 100 is to display constantly or for each trigger notification, in the case of displaying for each trigger notification, image data of the application insertion screen can be sent only when necessary, which is an advantage in that battery consumption can be saved. Note that, when multiple applications of the smartphone 100 are inserted and displayed on the HMD 200, each display form selection information may be sent to the HMD 200. Also, the selection of whether the application screen image is to be displayed constantly or for each trigger notification may be performed on the HMD side.

[0088] FIG. 11A shows an example of a block diagram of a portable smartphone 100 as a specific example of an information processing device. FIG. 11B shows an example of a functional block diagram of an HMD 200 as a specific example of an image display device. In FIG. 11A and FIG. 11B, the parts shown in FIG. 1A to FIG. 1D and denoted with the same reference numerals have the same operations as those already described in FIG. 1A to FIG. 1D, so that detailed explanations thereof will be omitted. In FIG. 11A, the smartphone 100 is configured by appropriately using a camera 1101, a depth sensor 1102, an acceleration sensor 1104, a gyro sensor 1105, a geomagnetic sensor 1106, an operation input interface 1107, a display device 1108, a processor 1120, a memory 1130 consisting of a program 1131 and information data 1132, a vibration generating device 1141, an audio input device 1142, an audio output device 1143, and a communication device 1144, and each component is connected to each other via a bus 1150.

[0089] The camera 1101 converts light incident through a lens into an electrical signal using an image sensor, and obtains an image captured by the camera.

[0090] The depth sensor 1102 is a sensor that can measure the distance and angle to an object and capture the shape of the object as a three-dimensional object. There is no limit to the measurement method, and it may be a TOF (Time Of Flight) sensor that irradiates an object with a laser light such as infrared light, measures the scattered light that bounces back, and analyzes and detects the distance to a distant object and the state of the object, a LiDAR (Light Detection and Ranging) that measures the distance by measuring the reflection time of a pulse light irradiated on a subject for each pixel, or a millimeter wave radar that emits millimeter wave radio waves, captures the reflected waves, and detects the distance to the object that is reflecting the waves and the state of the object.

[0091] The acceleration sensor 1104 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement, vibration, impact, and the like. The acceleration sensor 1104 provided in the smartphone 100 can detect the tilt and direction of the smartphone 100. In addition, the gyro sensor 1105 is a sensor that detects the angular velocity in the rotation direction, and can capture the vertical, horizontal, and diagonal posture states. Therefore, the acceleration sensor 1104 and the gyro sensor 1105 can be used to detect the posture, such as the tilt and direction, of the smartphone 100. The geomagnetic sensor 1106 is a sensor that detects the magnetic force of the earth and detects the direction in which the smartphone 100 is facing. It is also possible to detect the movement of the smartphone 100 by using a three-axis type that detects geomagnetism in the up-down direction in addition to the forward-backward and left-right directions, and capturing geomagnetic changes in response to the movement of the smartphone 100. With these, it is possible to detect and determine the posture state of the user 10 carrying the smartphone 100.

[0092] The processor 1120 is composed of a CPU, etc., and executes a program 1131, such as an operating system (OS) or an operation control application, stored in the memory 1130, to control each component of the smartphone 100 and realize functions such as the OS, middleware, and applications, and other functions. In the smartphone 100, a specific pattern generation and display processing unit 1121, a smartphone image generation processing unit 1122, and an image quality adjustment processing unit 1123 control the operation of each function. The processor 1120 of the smartphone 100 may also control the operation of the HMD 200 via wireless communication. For example, the processor 1120 may control the HMD 200 so that an application screen image supplied to the HMD 200 is displayed on the HMD 200. Specifically, the processor 1120 may cause the image display device to recognize the display position and display size of the screen of the information processing device, which is photographed and read by the camera 121 by the user 10 holding it over the camera 121, as a display area. The processor 1120 may also control the image display device so that the image of the screen of the information processing device supplied is displayed in the recognized display area. For example, the processor 1120 may control the image display device to display the supplied application screen image in the recognized display area.

[0093] The memory 1130 is composed of a non-volatile storage device or the like, and stores various programs 1131 and information data 1132 handled by the processor 1120 or the like. The information data 1132 stores display area information, image data of the application screen of the smartphone 100, image display stop information of the application screen of the smartphone 100, ambient brightness information, display mode selection information, and the like.

[0094] The display device 1108 of the smartphone 100 is configured with a liquid crystal display or the like and displays the display contents via a highly transparent touch panel, and displays the screen of an application in operation, icons of running applications and various status indications, various operation buttons, notifications to the user, etc. Notifications to the user include a notification that information on the display area has been acquired, a notification that image quality adjustment has been completed, etc.

[0095] The operation input interface 1107 of the smartphone 100 is used by the user 10 to input input information to the smartphone 100. In the case of the smartphone 100, a touch panel configured on the display surface of the display device 1108 detects touch operations using a finger or a touch pen as operation input, and the user 10 can set and input information that he or she wants to input. In addition, the input operation information may be captured using, for example, a keyboard or key buttons, or the user 10 may utter a voice indicating the input operation and collect the voice using the voice input device 1142 to capture the input operation information. For example, the user 10 selects a screen of the information processing device to be displayed on the image display device by user input using the operation input interface 1107. In addition, the user 10 selects an app screen or a partial area of ​​the app screen to be displayed on the image display device by user input using the operation input interface 1107.

[0096] The voice input device 1142 collects external sounds and the user's own voice with a microphone and converts them into voice data. Instruction information based on the user's voice can be taken into the smartphone 100, and an operation corresponding to the instruction information can be easily executed.

[0097] The audio output device 1143 is a device that outputs audio from a speaker or headphones based on audio data, and can notify the user of notifications by audio.

[0098] The vibration generating device 1141 generates vibrations under the control of the processor 1120, and converts a notification to the user 10 sent by the smartphone 100 into vibration. The vibration generating device 1141 transmits vibrations to the hand holding the smartphone 100, and can notify the user 10 of the notification. Examples of notifications to the user include notifications indicating the generation or reading of a specific pattern, the display of an app screen image, and the like, and notification contents, and these notification information can improve usability.

[0099] The communication device 1144 is a communication interface for performing wireless communication between at least the smartphone 100 and the HMD 200 by short-distance wireless communication, wireless LAN, or base station communication, and for connecting to a network. The communication device 1144 includes a communication processing circuit and an antenna corresponding to various communication interfaces, and transmits and receives image data captured by a camera and control signals. The short-distance wireless communication is performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). The base station communication may be performed using a long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications). For example, the communication device 1144 supplies an image of a screen of the information processing device to be displayed on the image display device to the image display device. The communication device 1144 also supplies an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device.

[0100] The specific pattern generation and display processing unit 1121 of the smartphone 100 generates and displays a pattern to be read by the HMD 200 in the entire screen of the smartphone 100 or a partial area of ​​an application screen selected by the user 10. For example, the specific pattern generation and display processing unit 1121 generates and displays a specific pattern that covers the selected application screen or a partial area of ​​the application screen. In addition, the specific pattern generation and display processing unit 1121 may make the generated and displayed specific pattern semi-transparent.

[0101] The smartphone image generation processing unit 1122 of the smartphone 100 generates image data of an app screen, such as an app, to be displayed within the display area of ​​the HMD screen based on display area information from the HMD 200.

[0102] When the smartphone display screen is difficult to see or the contrast between the app screen image and the HMD screen image is not harmonious, the image quality adjustment processing unit 1123 adjusts the image quality (brightness, color tone) of the app screen image so that the two images are harmonious. For example, when contrast inharmonicity is identified based on the identified contrast harmony, the image quality adjustment processing unit 1123 adjusts the image quality of the app screen image so that the contrast is harmonious. Specifically, the image quality adjustment processing unit 1123 adjusts the brightness of the app screen image according to the output from the illuminance sensor.

[0103] Next, in FIG. 11B, the HMD 200 is configured appropriately using a camera 121, a depth sensor 1202, an illuminance sensor 1203, an acceleration sensor 1204, a gyro sensor 1205, a geomagnetic sensor 1206, an operation input interface 1207, a display processing device 1208, a processor 1220, a memory 1230 consisting of a program 1231 and information data 1232, a vibration generating device 1241, an audio input device 1242, an audio output device 1243, and a communication device 1244, and each component is connected to each other via a bus 1250.

[0104] The camera 121 captures the field of view in front of the HMD 200, and converts light incident from a lens into an electrical signal using an imaging element to obtain a camera-captured image. The camera 121 mounted on the HMD 200 captures the field of view seen by the user 10 wearing the HMD 200, and can capture specific patterns 104, 202, and 302 on the display screen 101 of the smartphone 100 held within the field of view. For example, the user 10 can capture an image with the camera 121 by holding the screen of an information processing device over the camera 121.

[0105] The depth sensor 1202 is a sensor that can measure the distance and angle to an object similar to the smartphone 100 and capture the shape of the object, such as an object, in three dimensions. When the smartphone 100 is in front of the user 10, the depth sensor 1202 provided in the HMD 200 can measure the distance to the smartphone 100 when the smartphone 100 is captured and found by the camera 121.

[0106] The acceleration sensor 1204 is a sensor that detects acceleration, which is a change in speed per unit time, and can capture movement, vibration, impact, and the like. The acceleration sensor 1204 provided in the HMD 200 can detect the inclination of the HMD 200 worn on the head of the user 10 and the direction of the front of the face. The gyro sensor 1205 is a sensor that detects the angular velocity in the rotation direction and can capture the vertical, horizontal, and oblique posture states. The acceleration sensor 1204 and the gyro sensor 1205 can be used to detect the posture, such as the inclination and direction, of the HMD 200. The geomagnetic sensor 1206 is a sensor that detects the magnetic force of the earth and detects the direction in which the HMD 200 is facing. It is also possible to detect the movement of the HMD 200 by using a three-axis type that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions and capturing geomagnetic changes in response to the movement of the HMD 200. With these, it is possible to detect and determine the posture state of the user 10 wearing the HMD 200.

[0107] The processor 1220 is composed of a CPU or the like, and executes programs 1231, such as an operating system (OS) and an operation control application, stored in the memory 1230, to control each component of the HMD 200 and realize functions of the OS, middleware, applications, and other functions. The memory 1230 is composed of a non-volatile storage device or the like, and stores various programs 1231 and information data 1232 handled by the processor 1220 or the like. The information data 1232 stores image data of an application screen captured by a camera, display area information, image data of an application screen whose image quality has been adjusted by a smartphone, image display stop information of an application of a smartphone, ambient brightness information, display form selection information, and the like.

[0108] In the case of an optical see-through type HMD 200, the display processing device 1208 of the HMD 200 projects characters, symbols, video content, virtual objects, etc. using a small projector. Also, as shown in this embodiment, it projects a camera-captured image transmitted from the smartphone 100 and notification information to the user 10. The projected virtual objects, etc. are provided with a transparent half mirror that displays an image in front of the eyes. As a result, the user 10 can see both the imaged virtual object and the imaged virtual object as if they are floating, superimposed on the real object in the field of view in front of the eyes. In the case of a video see-through type, it is composed of a display such as a liquid crystal panel that displays a composite image of the image in front of the eyes captured by the camera 121 and the virtual object, etc. As a result, the user 10 can see the real object in the field of view image in front of the eyes superimposed on the virtual object, etc.

[0109] The operation input interface 1207 of the HMD 200 often uses gesture operation or voice input. Although there are few places to attach operation input interfaces such as buttons and touch panels to the HMD, they may be provided in a position or form that allows the user 10 to easily perform input operations, or may be separated from the main body of the HMD 200 and connected to a mouse or keyboard by wire or wirelessly. In addition, a virtual input operation screen may be displayed on the display screen of the display processing device 1208, and input operation information may be captured depending on the position on the input operation screen to which the line of sight is directed, or a pointer may be displayed on the input operation screen and the pointer may be operated by the operation input interface 1207 to capture the input operation information. In addition, the user 10 may utter a voice indicating an input operation, and the voice may be collected by the voice input device 1242 to capture the input operation information.

[0110] The audio input device 1242 uses a microphone to collect external sounds and the user's own voice and converts them into audio data. Instruction information based on the voice of the user 10 can be taken into the HMD 200, and operations corresponding to the instruction information can be easily executed. The audio output device 1243 outputs sound from a speaker or headphones based on the sound data, and can inform the user of notification information by sound.

[0111] The vibration generating device 1241 generates vibrations under the control of the processor 1220, and converts notification instruction information for the user 10, which is transmitted by the HMD 200, into vibrations. The vibration generating device 1241 transmits vibrations to the head of the user wearing the HMD 200, and can notify the user 10 of the notification instruction information. Examples of notification information for the user 10 include notifications indicating an incoming call or a warning of danger information, the reading of a specific pattern in this embodiment, the display of an application screen image, and the like, and these notifications can improve usability.

[0112] The communication device 1244 is a communication interface that performs wireless communication between the smartphone 100 and the HMD 200 and connects to a network by short-distance wireless communication, wireless LAN, or base station communication. It includes a communication processing circuit and an antenna corresponding to a predetermined communication interface, and transmits and receives image data captured by a camera and control signals. Note that short-distance wireless communication is performed using wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). In addition, long-distance wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) may be used for base station communication.

[0113] The specific pattern read area frame generating unit 1221 of the HMD 200 reads the specific pattern displayed on the screen of the smartphone 100, and generates an area frame of the specific pattern on the screen of the HMD 200. For example, the specific pattern read area frame generating unit 1221 photographs and reads the screen of the selected information processing device by the user 10 holding the camera 121 over the camera 121. Then, the specific pattern read area frame generating unit 1221 recognizes the display position and display size of the read screen of the information processing device as the display area. In addition, the specific pattern read area frame generating unit 1221 may photograph and read the selected app screen or a partial area of ​​the app screen by the user 10 holding it over a suitable position, and recognize the display position and display size of the read app screen or the partial area as the display area. Furthermore, the specific pattern read area frame generating unit 1221 may generate an area frame indicating the range of the specific pattern generated and displayed by the information processing device, and display the area frame on the display screen of the image display device.

[0114] The application image synthesis display processing unit 1222 of the HMD 200 acquires image data of the application screen from the smartphone 100, and performs processing to insert and synthesize the acquired image data of the application screen into the HMD screen image. For example, the application image synthesis display processing unit 1222 displays an image of the screen of the selected information processing device in the recognized display area. The application image synthesis display processing unit 1222 may also display an application screen or an application screen image of a partial area selected on the information processing device in the recognized display area.

[0115] The contrast harmony processing unit 1224 of the HMD 200 performs a process of identifying whether or not the contrast between the application screen image and the HMD screen image is harmonized.

[0116] When the contrast between the application screen image and the HMD screen image is not harmonious, the image quality adjustment processing unit 1223 adjusts the image quality (brightness, color tone) of the application screen image so that the two images harmonize. In the case of a see-through HMD, the brightness and, if necessary, the color tone of the application screen image are adjusted according to the brightness of the surroundings of the HMD 200 detected by the illuminance sensor 1203. In the case of an HMD screen that is video content, the brightness and color tone of the application screen image are adjusted according to the image content (brightness, type) of the video content.

[0117] The luminance distribution analysis unit 1225 analyzes the average luminance and the partial luminance strength of the image displayed on the non-see-through HMD by creating a histogram showing the luminance strength for each pixel or block, etc. Based on the analysis results, the contrast is adjusted.

[0118] With the above configuration, the smartphone 100 displays a specific pattern on the selected application screen to be displayed on the HMD screen by the specific pattern generation and display processing unit 1121, and the HMD reads the specific pattern by the specific pattern read area frame generation unit 1221 and displays an area frame indicating the range of the specific pattern on the HMD screen. The display position and display size are determined by adjusting the up, down, left, right, front, and back movements of the smartphone 100 using the area frame displayed on the HMD 200. After the display area is determined, the smartphone 100 generates an application screen image to be displayed in the display area of ​​the HMD screen by the smartphone image generation processing unit 1122. In the HMD 200, the application image synthesis and display processing unit 1222 synthesizes the application screen image from the smartphone 100 on the HMD screen and displays it. In addition, when the contrast harmony processing unit 1224 identifies that the contrast between the application screen image to be inserted and the HMD screen image is not harmonious, the image quality adjustment processing unit 1123 adjusts the image quality (brightness, color tone) of the application screen image so that the contrast between the application screen image and the HMD screen image is harmonious.

[0119] In this way, by holding the screen of an information processing device such as the smartphone 100 over an image display device such as the HMD 200 and selecting its placement location, it can be freely and easily displayed at a suitable display position and display size on the display screen of the image display device such as the HMD 200. Furthermore, by harmonizing the contrast between the display screen image of the information processing device inserted into the display screen of the image display device and the display screen image of the image display device, it is possible to obtain an effect that the screen of the information processing device such as the smartphone 100 can be viewed in a harmonious form with the display screen image of the image display device. Note that the information processing device controls the image display device, and the image display device can realize the operation of the above-mentioned information processing system by mutual communication with the information processing device.

[0120] Further, other examples of the information processing system, information processing device, and image display device according to the present embodiment will be described with reference to Figs. 12A to 12D. Figs. 12A to 12D are diagrams illustrating an example in which the placement location on the HMD screen where the application screen is displayed is divided into a plurality of locations and defined in advance, and the placement location of the selected application screen is determined using a position selection application, instead of adjusting and determining a suitable display area on the HMD screen while photographing and recognizing the application screen to be displayed on the HMD screen as described in Figs. 1A to 1D, 2A to 2C, etc. Note that in Figs. 12A to 12D, the parts shown in Figs. 1A to 1D, 2A to 2C and denoted with the same reference numerals have the same operations as those already described in Figs. 1A to 1D, 2A to 2C, and detailed description thereof will be omitted. First, the placement location on the HMD screen is divided into a plurality of areas and defined. As an example, as shown in FIG. 12A, the placement location on the HMD screen is divided into 12 areas, and placement locations from Area 1 to Area 12 are defined. On the display screen of the smartphone, as shown in FIG. 12B, a position selection application 1302 showing the defined placement locations as icons is displayed together with the selected application screens 102, 103, and 1301. For example, the user 10 selects the application screen 1301 as the application screen to be displayed on the HMD screen, and then, as shown in FIG. 12C, specifies that the placement location of the selected application screen 1301 is to be displayed in Area 2 on the HMD screen by selecting an icon of the position selection application 1302. By such an operation, as shown in FIG. 12D, a screen image 1303 of the image application is displayed in Area 2 on the HMD screen. In this way, the application screen image can be easily displayed in a placement location defined in advance on the screen of the HMD 200.

[0121] In addition, in the above explanation, the smartphone 100 is described as a specific example of an information processing device, and the HMD 200 is described as a specific example of an image display device, but the information processing device is not limited to the smartphone 100 but includes all devices with an operation display function such as an app, and the image display device is not limited to the HMD but includes all devices with a function to capture the surrounding outside world with a camera and display the images.

[0122] The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the configurations described. It is also possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.

[0123] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. In addition, the above-mentioned configurations, functions, etc. may be realized in software by a processor interpreting and executing a program that realizes each function. Information such as a program, table, file, etc. that realizes each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, DVD, etc., or may be stored in a device on a communication network.

[0124] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it can be considered that almost all components are connected to each other. [Industrial Applicability]

[0125] An information processing system, an information processing device, and an image display device are provided, which are capable of displaying a display screen of an information processing device within the screen of an image display device. [Explanation of symbols]

[0126] 10: User, 100: Smartphone, 101, 130: Display screen, 102, 103: Application screen, 104, 202, 302: Specific pattern, 121, 1101: Camera, 131, 203, 603, 607: Area frame, 200: HMD, 305: Display application switching software, 1102, 1202: Depth sensor, 1203: Illuminance sensor, 1104, 1204: Acceleration sensor 1105, 1205: gyro sensor, 1106, 1206: geomagnetic sensor, 1107, 1207: operation input interface, 1108: display device, 1120, 1220: processor, 1121: specific pattern generation display processing unit, 1122: smartphone image generation processing unit, 1123: image quality adjustment processing unit, 1130, 1230: memory, 1131, 1231: program, 1132, 1232: information data, 1141, 1241: vibration generation device, 1142, 1242: voice input device, 1143, 1243: voice output device, 1144, 1244: communication device, 1150, 1250: bus, 1221: specific pattern read area frame generation unit, 1222: application image synthesis display processing unit, 1223: image quality adjustment processing unit, 122 4: contrast harmony processing unit, 1225: luminance distribution analysis unit, 128: two-dot chain line, 132, 204: image, 201, 301: area, 306, 307: icon, 400, 402, 403, 404, 405, 406: dotted line, 601: left screen, 604, 608: image, 605: right screen, 801: landscape, 802, 805, 806: image, 1302: position selection application

Claims

1. An image display device equipped with a camera; an information processing device that operates and displays an application, which is application software; An information processing system having a communication linkage function for information between the image display device and the information processing device, the information processing device selects an application screen that is a screen of the application or a partial area of ​​the application screen by a user input, and generates and displays a specific pattern having a range of the selected application screen or the partial area; The image display device captures and reads the generated and displayed specific pattern using the camera, recognizes the display position and display size of the read-in specific pattern as a display area, and displays an app screen image, which is an image of the selected app screen or partial area, in the recognized display area.

2. 2. The information processing system according to claim 1, The information processing system is characterized in that the information processing device makes the specific pattern that is generated and displayed semi-transparent.

3. 2. The information processing system according to claim 1, The information processing system is characterized in that the image display device generates an area frame indicating the range of the specific pattern generated and displayed by the information processing device, and displays the area frame on a display screen of the image display device.

4. 2. The information processing system according to claim 1, The information processing system is characterized in that the information processing device launches and displays display application switching software, which can switch between multiple applications, on a display screen of the information processing device, and an icon of each application shown on the launched and displayed display application switching software is selected, thereby selecting the application to be displayed on the image display device.

5. 2. The information processing system according to claim 1, the information processing device selects the application screen or a partial area of ​​the application screen, and generates and displays the specific pattern having a range of the selected application screen or the partial area; the image display device captures an image of the specific pattern that has been generated and displayed by the camera, recognizes a display position and a display size of the captured specific pattern as the display area, and displays an application screen image that is an image of the selected application screen or the partial area in the recognized display area; Furthermore, the information processing device selects another application screen or another partial area of ​​the another application screen displayed on the image display device, and generates and displays another specific pattern having a range of the selected another application screen or the selected another partial area; The information processing system is characterized in that the image display device photographs the generated and displayed other specific pattern using the camera, recognizes the display position and display size of the photographed other specific pattern as a other display area, and displays another app screen image, which is an image of the other app screen or the other partial area selected by the information processing device, in the recognized other display area.

6. 2. The information processing system according to claim 1, the image display device or the information processing device identifies a degree of contrast harmony between the application screen image to be inserted into the display screen of the image display device and an image on the display screen of the image display device; The information processing system is characterized in that, when the contrast disharmony is identified based on the identified contrast harmony, the information processing device adjusts the image quality of the app screen so that the contrast is harmonized.

7. 7. The information processing system according to claim 6, The image display device has an illuminance sensor that detects the brightness of an area around a field of view, The information processing system is characterized in that the information processing device adjusts the brightness of the application screen in accordance with an output from the illuminance sensor.

8. 2. The information processing system according to claim 1, An information processing system characterized in that, when the app screen image selected by the information processing device is displayed in the display area of ​​the image display device, the information processing device selects whether to display constantly or for each trigger notification, and if the constant display is selected, the image display device always displays the app screen image, and if the display for each trigger notification is selected, the image display device starts or stops displaying the app screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device.

9. An information processing device having a function of running and displaying an application that is application software, and a function of communicating and linking information with an image display device equipped with a camera, The information processing device includes: An application screen, which is a screen of the application, or a partial area of ​​the application screen is selected by a user input; generating and displaying a specific pattern having the selected application screen or the partial area as a range; causing the image display device to recognize a display position and a display size of the specific pattern photographed and read by the camera as a display area; supplying an application screen image, which is an image of the application screen or the partial region to be displayed on the image display device, to the image display device; The information processing apparatus controls the image display device so as to display the supplied application screen or the application screen image, which is an image of the partial region, in the recognized display area.

10. The information processing device according to claim 9, The information processing apparatus is characterized in that the specific pattern that is generated and displayed is made semi-transparent.

11. The information processing device according to claim 9, The information processing device is characterized in that it causes the image display device to generate a region frame indicating a range of the specific pattern.

12. The information processing device according to claim 9, The information processing device includes: activating and displaying a display application switching software capable of switching between the plurality of applications on a display screen of the information processing device; An information processing device, characterized in that an icon of each application shown on a displayed application switching software is selected, thereby selecting the application to be displayed on the image display device.

13. The information processing device according to claim 9, The information processing device includes: The application screen or a partial area of ​​the application screen displayed on the image display device is selected, and the specific pattern having a range of the selected application screen or the partial area is generated and displayed; a display position and a display size of the specific pattern photographed and read by the camera are recognized by the image display device as the display area; supplying an application screen image, which is an image of the application screen or the partial region to be displayed on the image display device, to the image display device; controlling the image display device to display the supplied application screen image in the recognized display area; Furthermore, the information processing device selecting another application screen or another partial area of ​​the another application screen displayed on the image display device, and generating and displaying another specific pattern having a range of the selected another application screen or the selected another partial area; a display position and a display size of the other specific pattern photographed and read by the camera are recognized by the image display device as another display area; supplying to the image display device another application screen image which is an image of the other application screen or the other partial region to be displayed on the image display device; an information processing device that controls the image display device so as to display the supplied different application screen image in the recognized different display area;

14. The information processing device according to claim 9, The information processing device is characterized in that when a contrast inconsistency between the app screen image inserted into the screen of the image display device and the image on the display screen of the image display device is identified by the information processing device or the image display device, the information processing device adjusts the image quality of the app screen so that the contrast is harmonized.

15. The information processing device according to claim 14, The information processing device adjusts the luminance of the application screen in response to an output from an illuminance sensor of the image display device that detects the brightness of an area surrounding the field of view.

16. The information processing device according to claim 9, When the application screen image selected by the information processing device is displayed in the display area of ​​the image display device, the information processing device selects whether to display constantly or for each trigger notification, and if the constant display is selected, controls the image display device to always display the application screen image, and if the display for each trigger notification is selected, generates trigger notification information for starting or stopping display, and controls the image display device so that the display of the application screen image is started or stopped in accordance with the generated trigger notification information.

17. An image display device having a function of communicating with an information processing device that operates and displays an application software, and a camera, The image display device includes: an application screen, which is a screen of the application selected by a user input in the information processing device, or a specific pattern generated and displayed in a partial area of ​​the application screen is photographed and read by the camera; The display position and the display size of the specific pattern that has been read are recognized as a display area; The image display device displays, in the recognized display area, an application screen image which is an image of the application screen or the partial region selected by the information processing device.

18. The image display device according to claim 17, The image display device is characterized in that the specific translucent pattern generated and displayed by the information processing device is photographed and read by the camera.

19. The image display device according to claim 17, The image display device is characterized in that it generates an area frame indicating the range of the specific pattern generated and displayed by the information processing device, and displays the area frame on a display screen of the image display device.

20. The image display device according to claim 17, The image display device is characterized in that it displays an app screen image, which is an image of the app screen or a partial area of ​​an app of an icon selected from the icons of each app displayed on display app switching software launched and displayed on the information processing device.

21. The image display device according to claim 17, The image display device includes: The specific pattern generated and displayed on the application screen or the partial region selected by the information processing device is photographed and read by the camera; Recognizing the display position and display size of the specific pattern that has been read as the display area; Displaying the application screen image in the recognized display area; Furthermore, the image display device a second specific pattern generated and displayed on a second partial area of ​​the second application screen selected by the information processing device is photographed by the camera and read in; Recognizing the display position and display size of the read different specific pattern as a different display area; An image display device, characterized in that a different application screen image, which is an image of the different application screen or the different partial region selected by the information processing device, is displayed in the recognized different display area.

22. The image display device according to claim 17, The image display device is characterized in that when a contrast inconsistency between the app screen image inserted into the screen of the image display device and the image on the display screen of the image display device is identified by the information processing device or the image display device, the image display device displays the app screen image whose image quality has been adjusted by the information processing device so that the contrast is harmonized.

23. The image display device according to claim 22, The image display device has an illuminance sensor that detects the brightness of the surrounding field of view, and displays the app screen image whose brightness has been adjusted by the information processing device in accordance with the output from the illuminance sensor.

24. The image display device according to claim 17, When the app screen image selected by the information processing device is displayed in the display area of ​​the image display device, if constant display is selected by the information processing device, the image display device always displays the app screen image, and if display for each trigger notification is selected by the information processing device, the image display device starts or stops the display of the app screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device.

Images