Information processing system, information processing device and image display device

Abstract

To provide an information processing system, an information processing device, and an image display device configured to superimpose screen information of a smartphone in a display screen of an HMD with high degree of freedom and convenience.SOLUTION: An information processing system includes an HMD (200) equipped with a camera (121) and a smartphone (100) configured to operate and display an application that is application software, the information processing system having an information communication cooperation function between the HMD (200) and the smartphone (100). The HMD (200) images a screen of the smartphone (100) by the camera (121), recognizes a display position and a display size of the read screen of the smartphone (100) as a display area, and displays an image of the screen of the smartphone (100) in the recognized display area.SELECTED DRAWING: Figure 1B

Description

[Technical Field]

[0001] The present invention relates to an information processing system, an information processing device, and an image display device that includes an information processing device that displays an app (an abbreviation for application software) and an image display device that is primarily worn on the head and displays images, and that displays the display screen of the information processing device within the screen of the image display device. [Background technology]

[0002] Information processing devices that have functions such as displaying installed apps and communicating with other devices are called smartphones (hereinafter referred to as "smartphones"), especially portable ones. They are roughly palm-sized and can display various app operation screens and various information on the display screen, making them easy to use and widely used. With the emergence of foldable and rollable smartphones, smartphone screens have become larger, making it possible to display multiple apps simultaneously.

[0003] Furthermore, image display devices that display virtual objects have become widely used. One example is a head-mounted display (HMD, hereinafter referred to as HMD), which is worn on the head and has a display unit, a camera, etc. An HMD is an image display device that displays virtual objects while recognizing real objects in the real space in three dimensions using Augmented Reality (AR) technology, which reflects and augments virtual objects in a virtual space made up of other images or images created using CG (Computer Graphics), etc.

[0004] Furthermore, in recent years, HMDs that work in conjunction with smartphones to display the smartphone screen have emerged. For example, Patent Document 1 describes an information processing device comprising: a camera; a position and orientation calculation unit that calculates the position and orientation of the image display device in a first reference coordinate system of the information processing device based on an image of an external image display device captured by the camera; a position and orientation conversion unit that converts the position and orientation of the image display device in the first reference coordinate system to the position and orientation of the information processing device in a second reference coordinate system of the image display device; and an embedding position calculation unit that calculates the position at which to embed screen information related to the information processing device within the screen of the image display device based on the position and orientation of the information processing device in the second reference coordinate system. That is, based on an image of the HMD captured by a camera mounted on the smartphone, the relative positions and orientations of the HMD and the smartphone are determined, and the screen information sent from the smartphone is superimposed and displayed on the HMD playback screen at an appropriate position and orientation on the playback screen of the content being viewed. This allows the smartphone to be the only device within the content playback screen in the same position and orientation as in the real world, allowing the user to operate the smartphone screen while wearing the HMD. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2016 / 021252 Summary of the Invention [Problem to be solved by the invention]

[0006] Although Patent Document 1 describes overlaying smartphone screen information on the HMD display screen, the entire smartphone screen must always be positioned within the HMD display screen, which reduces flexibility in use and reduces convenience.

[0007] Furthermore, when displaying a smartphone screen as a virtual object on an HMD, it is necessary to select which app content to display at which position on the HMD, but conventional selection operations have presented the challenge of being unvisual, cumbersome, and difficult to do. Furthermore, when inserting and displaying the smartphone screen as an image directly onto the HMD screen, if the HMD screen image and the smartphone screen image do not match in contrast, for example, in bright surroundings such as outdoors, the brightness of the smartphone inserted image is low compared to the bright HMD screen image (transparent image), making it difficult to see, or in dark surroundings such as at night, the contrast between the dark HMD screen image and the smartphone inserted image is too strong, causing eye fatigue.

[0008] In consideration of the above-mentioned problems, the present invention provides an information processing system, an information processing device, and an image display device that can easily and conveniently display a screen of an application or the like selected on an information processing device such as a smartphone at a suitable display position and size on the display screen of an image display device such as an HMD. Furthermore, the present invention provides a high degree of freedom in selecting a suitable display layout, enabling convenient operability for the user. Furthermore, the present invention provides an information processing system, an information processing device, and an image display device that can harmonize the contrast between the display screen image of the information processing device inserted into the display screen of the image display device and the display screen image of the image display device. [Means for solving the problem]

[0009] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0010] (1) an image display device equipped with a camera; an information processing device that runs and displays an application, which is application software; An information processing system having a function of communicating information between an image display device and an information processing device, The image display device is an information processing system characterized by photographing and reading the screen of an information processing device using a camera, recognizing the display position and display size of the read screen of the information processing device as a display area, and displaying an image of the screen of the information processing device in the recognized display area.

[0011] (2) The information processing system according to (1), The information processing device selects an application screen or a partial area of ​​the application screen, The image display device is an information processing system characterized by photographing and reading a selected application screen or partial area using a camera, recognizing the display position and display size of the read application screen or partial area as a display area, and displaying an application screen image, which is an image of the selected application screen or partial area, in the recognized display area.

[0012] (3) The information processing system according to (2), the information processing device selects an application screen or a partial area of ​​the application screen, and generates and displays a specific pattern having the selected application screen or partial area as its range; The image display device is an information processing system characterized by photographing and reading the generated and displayed specific pattern with a camera, recognizing the display position and display size of the read specific pattern as a display area, and displaying an application screen image, which is an image of a selected application screen or a partial area, in the recognized display area.

[0013] (4) The information processing system according to (3), The information processing device is an information processing system characterized in that the specific pattern that is generated and displayed is made semi-transparent.

[0014] (5) The information processing system according to (3), The image display device is an information processing system characterized in that it generates an area frame indicating the range of a specific pattern generated and displayed by an information processing device, and displays the area frame on the display screen of the image display device.

[0015] (6) The information processing system according to (2), The information processing device is an information processing system characterized in that it starts and displays display application switching software, which can switch between multiple applications, on the display screen of the information processing device, and allows the icon of each application shown on the started and displayed display application switching software to be selected, thereby selecting the application to be displayed on the image display device.

[0016] (7) The information processing system according to (2), The information processing device selects an application screen or a partial area of ​​the application screen, the image display device captures an image of the selected application screen or partial area with a camera, recognizes a display position and a display size of the captured application screen or partial area as a display area, and displays an application screen image, which is an image of the selected application screen or partial area, in the recognized display area; Furthermore, the information processing device selects another application screen or another partial area of ​​another application screen displayed on the image display device, The image display device is an information processing system characterized in that it photographs a selected other app screen or a partial area of ​​the other app screen with a camera, recognizes the display position and display size of the photographed other app screen or partial area of ​​the other app screen as a separate display area, and displays a separate app screen image, which is an image of the other app screen or partial area of ​​the other app screen selected by the information processing device, in the recognized separate display area.

[0017] (8) The information processing system according to (2), The image display device or the information processing device identifies a degree of harmony of contrast between the application screen image to be inserted into the display screen of the image display device and the image on the display screen of the image display device, The information processing device is an information processing system characterized in that, when a contrast disharmony is identified based on the identified contrast harmony state, the information processing device adjusts the image quality of the application screen so that the contrast is harmonized.

[0018] (9) The information processing system according to (8), The image display device has an illuminance sensor that detects the brightness of the surrounding field of view, The information processing device is an information processing system that adjusts the brightness of an application screen in accordance with the output from an illuminance sensor.

[0019] (10) The information processing system according to (2), When an application screen image selected by an information processing device is displayed in the display area of ​​an image display device, the information processing device is allowed to select whether to display it at all times or for each trigger notification, and if "always display" is selected, the image display device always displays the application screen image, and if "display for each trigger notification" is selected, the image display device starts or stops displaying the application screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device, in this information processing system.

[0020] (11) An information processing device having a function of running and displaying an application that is application software, and a function of communicating information with an image display device equipped with a camera, The information processing device A screen of the information processing device to be displayed on the image display device is selected, The display position and the display size of the screen of the information processing device photographed and read by the camera are recognized by the image display device as a display area; An image on the screen of the information processing device to be displayed on the image display device is supplied to the image display device; The information processing apparatus is characterized by controlling the image display device so as to display the image on the screen of the supplied information processing apparatus in the recognized display area.

[0021] (12) The information processing device according to (11), The information processing device The app screen or a part of the app screen is selected, The image display device recognizes the display position and display size of the application screen or a partial area photographed and read by the camera as a display area; supplying an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device; The information processing device is characterized by controlling the image display device so as to display the supplied selected application screen or an application screen image, which is an image of a partial area, in the recognized display area.

[0022] (13) The information processing device according to (12), The information processing device An application screen or a partial area displayed on the image display device is selected, Generate and display a specific pattern covering the selected application screen or a partial area; A specific pattern photographed and read by the camera is recognized as a display area by the image display device; supplying an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device; The information processing device is characterized by controlling the image display device so as to display the supplied application screen or an application screen image, which is an image of a partial area, in the recognized display area.

[0023] (14) The information processing device according to (13), The information processing device is characterized in that the specific pattern that is generated and displayed is made semi-transparent.

[0024] (15) The information processing device according to (13), The information processing device is characterized by causing an image display device to generate an area frame indicating the range of a specific pattern.

[0025] (16) The information processing device according to (12), The information processing device A display application switching software that can switch between multiple applications is started and displayed on the display screen of the information processing device, The information processing device is characterized in that an icon of each application displayed on the displayed display application switching software is selected, thereby selecting an application to be displayed on the image display device.

[0026] (17) The information processing device according to (12), The information processing device An application screen or a partial area of ​​the application screen displayed on the image display device is selected, The image display device recognizes the display position and display size of the application screen or a partial area photographed and read by the camera as a display area; supplying an application screen image, which is an image of an application screen or a partial area to be displayed on the image display device, to the image display device; controlling the image display device to display the supplied application screen image in the recognized display area; Furthermore, the information processing device Another application screen or another partial area of ​​another application screen displayed on the image display device is selected, causing the image display device to recognize a display position and a display size of another application screen or another partial area of ​​the application screen photographed and read by the camera as another display area; supplying to the image display device an application screen image which is an image of another application screen or a partial area of ​​another application screen to be displayed on the image display device; The information processing device is characterized by controlling the image display device so as to display the supplied screen image of another application in the recognized display area.

[0027] (18) The information processing device according to (12), The information processing device is characterized in that when the information processing device or the image display device identifies a contrast mismatch between an application screen image inserted into the screen of the image display device and an image on the display screen of the image display device, the information processing device adjusts the image quality of the application screen so that the contrast is harmonious.

[0028] (19) The information processing device according to (18), The information processing device is characterized in that it adjusts the brightness of an application screen in accordance with the output from an illuminance sensor of an image display device that detects the brightness of the surrounding field of view.

[0029] (20) The information processing device according to (12), When an application screen image selected by an information processing device is displayed in the display area of ​​an image display device, the information processing device selects whether to display it constantly or for each trigger notification, and if constant display is selected, the information processing device controls the image display device to always display the application screen image, and if display for each trigger notification is selected, the information processing device generates trigger notification information for starting or stopping the display, and controls the image display device to start or stop displaying the application screen image according to the generated trigger notification information.

[0030] (21) An image display device having a function of communicating with an information processing device that runs and displays an application software, and a camera, The image display device is The screen of the information processing device selected by the information processing device is photographed and read by a camera; The display position and display size of the screen of the read information processing device are recognized as a display area, The image display device is characterized in that an image on the screen of the information processing device selected by the information processing device is displayed in the recognized display area.

[0031] (22) The image display device according to (21), The image display device is An application screen or a partial area of ​​the application screen, which is a screen of the application selected on the information processing device, is photographed and read by a camera; The display position and size of the loaded application screen or a partial area are recognized as the display area, The image display device is characterized in that an application screen image, which is an image of an application screen or a partial area selected on the information processing device, is displayed in the recognized display area.

[0032] (23) The image display device according to (22), The image display device is A specific pattern generated and displayed on an application screen or a partial area of ​​an information processing device is photographed and read by a camera; The display position and size of the specific pattern that was read is recognized as the display area. The image display device is characterized in that an application screen image, which is an image of an application screen or a partial area selected on the information processing device, is displayed in the recognized display area.

[0033] (24) The image display device according to (23), The image display device is characterized in that a semi-transparent specific pattern generated and displayed by an information processing device is photographed and read by a camera.

[0034] (25) The image display device according to (23), The image display device is characterized in that it generates an area frame indicating the range of a specific pattern generated and displayed by an information processing device, and displays the area frame on a display screen of the image display device.

[0035] (26) The image display device according to (22), The image display device is characterized by displaying an application screen image, which is an image of an application screen or a partial area of ​​an application icon selected from the icons of applications displayed on display application switching software launched and displayed on an information processing device.

[0036] (27) The image display device according to (22), The image display device is An image of the selected application screen or a partial area is captured by a camera on the information processing device and read. The display position and size of the loaded application screen or a partial area are recognized as the display area, An application screen image, which is an image of an application screen or a partial area, is displayed in the recognized display area; Furthermore, the image display device An image of another application screen or another partial area of ​​another application screen selected on the information processing device is captured by a camera and read; The display position and display size of the loaded different application screen or a different partial area of ​​the different application screen are recognized as a different display area, The image display device is characterized in that it displays, in the recognized separate display area, a separate application screen image that is an image of a partial area of ​​the separate application screen or a separate application screen selected on the information processing device.

[0037] (28) The image display device according to (22), The image display device is characterized in that when an information processing device or an image display device identifies a contrast mismatch between an application screen image inserted into the screen of the image display device and an image on the display screen of the image display device, the image display device displays an application screen image whose image quality has been adjusted by the information processing device so that the contrast is harmonized.

[0038] (29) The image display device according to (28), The image display device has an illuminance sensor that detects the brightness of the surrounding field of view, and is characterized by displaying an application screen image whose brightness has been adjusted by an information processing device according to the output from the illuminance sensor.

[0039] (30) The image display device according to (22), This image display device is characterized in that when an app screen image selected by an information processing device is displayed in the display area of ​​the image display device, if constant display is selected by the information processing device, the image display device always displays the app screen image, and if display for each trigger notification is selected by the information processing device, the image display device starts or stops displaying the app screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device. [Effects of the Invention]

[0040] By using the technology of the present invention, it is possible to easily and conveniently display screens of apps selected on an information processing device such as a smartphone at a suitable display position and size on the display screen of an image display device such as an HMD, thereby providing a high degree of freedom in the suitable display arrangement and enabling convenient operability for the user.Furthermore, it is possible to realize an information processing system, information processing device, and image display device that can harmonize the contrast between the display screen image of the information processing device inserted into the display screen of the image display device and the display screen image of the image display device.

[0041] Furthermore, problems, configurations, and effects other than those described above will become apparent from the following description of the embodiments. [Brief explanation of the drawings]

[0042] [Figure 1A] 1A to 1C are diagrams illustrating examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 1B] 1A to 1C are diagrams illustrating examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 1C] 1A to 1C are diagrams illustrating examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 1D] 1A to 1C are diagrams illustrating examples of the external appearance of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 2A]10A to 10C are diagrams illustrating other examples of the external appearance of the information processing system, the information processing device, and the image display device according to the present embodiment; [Figure 2B] 10A to 10C are diagrams illustrating other examples of the external appearance of the information processing system, the information processing device, and the image display device according to the present embodiment; [Figure 2C] 10A to 10C are diagrams illustrating other examples of the external appearance of the information processing system, the information processing device, and the image display device according to the present embodiment; [Figure 3A] 10A and 10B are diagrams illustrating an example of a case where a partial area of ​​an application screen is selected in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 3B] 10A and 10B are diagrams illustrating an example of a case where an application displayed on the information processing device is selected by designating an icon in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 4] 1 is an example of a flowchart illustrating basic operations of an information processing system, an information processing device, and an image display device according to the present embodiment. [Figure 5] 10 is an example of a flowchart illustrating a display operation of multiple applications in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 6A] 10A and 10B are diagrams illustrating examples of the appearance of an information processing system, an information processing device, and an image display device according to the present embodiment, which illustrate the operation of the information processing device when displaying the left and right folding screens of a foldable smartphone. [Figure 6B] 10A and 10B are diagrams illustrating examples of the appearance of an information processing system, an information processing device, and an image display device according to the present embodiment, which illustrate the operation of the information processing device when displaying the left and right folding screens of a foldable smartphone. [Figure 7A] 10A and 10B are diagrams illustrating an example of contrast harmonization processing between a screen image of an information processing device and a screen image of an image display device in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 7B]10A and 10B are diagrams illustrating an example of contrast harmonization processing between a screen image of an information processing device and a screen image of an image display device in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 8] 10 is an example of a flowchart illustrating an image adjustment process when the contrast of a screen image of the information processing device and that of the image display device are harmonized in the information processing system, the information processing device, and the transmissive image display device according to the present embodiment. [Figure 9] 10 is an example of a flowchart illustrating an image adjustment process when the contrast of a screen image of the information processing device and that of the image display device are harmonized in the information processing system, the information processing device, and the shield-type image display device according to the present embodiment. [Figure 10] 10 is an example of a flowchart illustrating a process for performing constant display or display for each trigger notification in the information processing system, the information processing device, and the image display device according to the present embodiment. [Figure 11A] 1 is a block diagram illustrating an example of the configuration of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 11B] 1 is a block diagram illustrating an example of the configuration of an information processing system, an information processing device, and an image display device according to an embodiment of the present invention. [Figure 12A] 10A and 10B are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned using a position selection application in the information processing system, information processing device, and image display device according to the present embodiment. [Figure 12B] 10A and 10B are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned using a position selection application in the information processing system, information processing device, and image display device according to the present embodiment. [Figure 12C] 10A and 10B are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned using a position selection application in the information processing system, information processing device, and image display device according to the present embodiment. [Figure 12D]10A and 10B are diagrams illustrating examples of operations when the display position of the image display device is defined in advance and positioned using a position selection application in the information processing system, information processing device, and image display device according to the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0043] An example of an embodiment of the present invention will be described below with reference to the drawings. In this example, the following technologies are provided to ensure a comfortable and convenient life for users through highly advanced technology. The realization of this linked system for video information devices will contribute to the achievement of "9. Build resilient infrastructure, promote inclusive and sustainable industrialization, inclusive and sustainable technological development" of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0044] This information processing system has an image display device equipped with a camera and an information processing device that runs and displays apps (an abbreviation for application software), etc., and has a function for communicating and linking information between the image display device and the information processing device. This description will be given using a head-mounted display (HMD, hereinafter referred to as HMD) as a specific example of the image display device and a portable smartphone (an abbreviation for smartphone) as a specific example of the information processing device. HMDs include a type that displays virtual objects on a display unit while viewing real objects in front of the user through lenses (so-called optical see-through types), and a type that displays images of real objects in real space captured by a camera, as well as content such as movies, music, and games, on a display unit provided in front of the head (so-called video see-through types or occlusion types), allowing the user to view the displayed real objects and virtual objects up close.

[0045] 1A to 1D are diagrams illustrating examples of the appearance of an information processing system, an information processing device, and an image display device according to the present embodiment. FIG. 1A shows the smartphone 100 on the display surface side, on which an app stored in the smartphone 100 is running and displayed. FIG. 1B shows the smartphone 100 being held and the HMD 200, which is connected to the smartphone 100 for information communication as indicated by a two-dot chain line 128, being worn by a user 10 on the head. FIGS. 1C and 1D show the display screen 130 of the HMD 200. Below, a case will be described in which an app screen (hereinafter referred to as an app screen; that is, the app screen refers to the screen of an app displayed on an information processing device such as the smartphone 100) on the display screen 101 of the smartphone 100 is inserted as a virtual object and displayed on the display screen of the HMD 200, so that the app screen of the smartphone 100 can be viewed on the HMD 200 together with the surrounding scenery or image content.

[0046] 1A, the smartphone 100 displays an app screen 102 of a map app, which is an example of an app, and an app screen 103 of a weather app on a display screen 101. When selecting, for example, the app screen 102 of a map app as the app screen to be inserted and displayed on the HMD 200, the user 10 presses and holds or touches (hereinafter referred to as touch) the app screen 102 of the map app with his / her finger, thereby selecting the app screen 102 of the map app as the app screen to be inserted and displayed on the HMD 200. In this way, the information processing device selects a screen of the information processing device to be displayed on the image display device by a user input.

[0047] Furthermore, a specific pattern 104 indicated by a diagonal dotted line may be generated and displayed on the app screen 102 of the selected map app. The displayed specific pattern 104 indicates the display area of ​​the app screen to be inserted and displayed on the HMD 200 when the user 10 wearing the HMD 200 views the specific pattern 104 on the display screen of the smartphone 100 from a suitable position. Note that the method of selecting the app screen to be inserted and displayed on the HMD 200 may be a touch, such as a deep press or a strong press, or an operation via the operation input interface of the smartphone 100. The specific pattern may also be made semi-transparent, in which case the original app screen can be viewed while the specific pattern is displayed, improving convenience.

[0048] As shown in FIG. 1B , when the screen of the smartphone 100 enters the display screen of the HMD 200 worn by the user 10, the camera 121 of the HMD 200 captures and captures a specific pattern 104 of the smartphone 100. The HMD 200 captures and reads the specific pattern 104, which moves in the up, down, left, right, front, and back directions, as indicated by 122 to 127, using the camera 121, and recognizes the position of the read specific pattern 104 as the display position of an application screen to be displayed on the display screen of the HMD 200. Here, 126 and 127 represent cases where the specific pattern 104 moves in the near and far directions. When the specific pattern 104 is captured and read by the camera 121, the display size can be enlarged or reduced. The size of the display area of ​​the read specific pattern 104 is recognized as the display size of the application screen to be displayed on the display screen of the HMD 200. In this way, the image display device captures and reads the screen of a selected information processing device by the camera 121 when the user 10 holds it over the camera 121, and recognizes the display position and display size of the read information processing device screen as the display area.

[0049] 1C shows the display visible within the HMD when the screen of the smartphone 100 is captured by the HMD 200. The top part of FIG. 1C shows the screen of the smartphone 100 captured by the camera mounted on the HMD 200 as it enters the display range of the HMD 200. The smartphone 100 displays a specific pattern 104 on top of the app screen 102 of the selected map app.

[0050] The HMD 200 captures the screen displayed on the smartphone 100 with the camera 121 and generates an area frame 131 indicating the area of ​​the loaded specific pattern 104, as shown in the middle of FIG. 1C. Note that the smartphone 100 may also generate a thick line or other marking indicating the area in advance, in conjunction with the specific pattern 104. The generated area frame 131 is displayed on the display screen 130 of the HMD 200 as the smartphone 100, on which the specific pattern 104 is displayed, moves up, down, left, right, and near and far. The user 10 moves the smartphone 100 up, down, left, right, and near and far while looking at the area frame 131 indicating the area of ​​the specific pattern, and determines the display position and display size suitable for the HMD display of the app screen. Then, as shown in the bottom of FIG. 1C, an image 132 of the app screen of the smartphone 100 is displayed at a predetermined position and with a predetermined size on the screen of the HMD 200.

[0051] In this way, by moving the display area frame 131 while viewing the screen of the smartphone 100 through the field of view of the HMD 200, it becomes easy to make fine adjustments and confirm when deciding "to display at this position and size." Furthermore, when measuring the distances up, down, left, right, and near, a distance sensor mounted on the HMD 200 may measure the distance between the HMD 200 and the smartphone 100, and the results may also be used. In this way, the image display device captures and reads the screen of the selected information processing device by the camera 121 when the user 10 holds it over a suitable position.

[0052] Here, it is advisable to store the correspondence between the image captured by the camera mounted on the HMD 200 and the area frame displayed on the HMD 200 screen in advance, for example, through adjustments at the time of product shipment. Furthermore, if the smartphone 100 screen is not facing directly in front of the HMD 200, the smartphone 100 screen will appear tilted. In such a case, the shape of the smartphone 100 is grasped, the smartphone's tilt is recognized, and the screen displayed on the HMD 200 is corrected so that it appears in the same shape as when viewed from the front. Using this method, it is possible to display the smartphone screen in the appropriate position and correct the screen distortion.

[0053] Next, FIG. 1D illustrates an example in which the display position and size suitable for display differ from those in FIG. 1C. The user 10 can position the smartphone 100 at any display position and size within the HMD 200 screen by adjusting the position of the smartphone 100 vertically, horizontally, and in perspective. The area frame 131 displayed on the display screen of the HMD 200 allows the user to easily determine the display position and size suitable for inserting and displaying the app screen, for example, a location that does not interfere with the image within the display screen, while still easily viewing the image within the display screen. Alternatively, the display position within the HMD 200 screen may be determined using a specific pattern without the area frame.

[0054] Here, as shown in the lower part of FIG. 1C and the lower part of FIG. 1D, the HMD 200 transmits information about the display area defined by the determined display position and display size to the smartphone 100. The smartphone 100 receives the display area information, generates image data for the application screen to be displayed within the display area, and transmits the generated image data to the HMD 200. The HMD 200 displays the received application screen image (simply referred to as an image or application screen image. That is, an application screen image refers to, for example, the screen of an application displayed on an image display device such as the HMD 200), for example, image 132 of the application screen 102 of a map application, in the specified display area. In this manner, the image display device displays an image of the screen of the selected information processing device in the recognized display area. Note that the application screen initially selected by the smartphone 100, for example, image 132 of the application screen 102 of the map application, may be displayed as is, or the resolution of the image of the application screen 102 of the map application may be converted based on the pixel count and resolution of the HMD 200. Furthermore, if the smartphone 100 has settings for a display screen to be output to another device, the image data of the settings may be transmitted.

[0055] Furthermore, after the display position and display size within the HMD 200 are determined, the user 10 may be able to select whether the display position of the app screen image of the smartphone 100 is to be displayed in the same position within the frame of the display screen 130 of the HMD 200 or to be displayed as if it is attached to a position fixed relative to the background position (in FIG. 1C, the upper right of the tower). In this case, when the user 10 moves his / her head, in the former case, the app screen image of the smartphone 100 is displayed in the same position on the screen of the HMD 200, and in the latter case, the app screen image moves within the HMD screen but is displayed in a position that does not change its position relative to the background. Furthermore, if the user 10 moves his / her head or the surrounding scenery changes and wants to move the display position of the app screen image, the position of the app screen image displayed on the HMD can be moved by operating the screen of the HMD 200, for example, by a gesture operation, such as grabbing and moving the app screen image displayed on the HMD 200, or by selecting and moving the app screen image with their line of sight.

[0056] As described above, when displaying an app screen of the smartphone 100 on the display screen of the HMD 200, the display position and display size that define the display area of ​​the app screen can be determined with a high degree of freedom by holding the smartphone 100 over the HMD 200 and moving the smartphone 100 up, down, left, or right, or by moving it closer or farther away. Furthermore, after setting the display position and display size of the app screen, the app screen of the smartphone 100 can be viewed on the display screen of the HMD 200 even if the smartphone 100 is out of the field of view, such as when the smartphone 100 is in a pocket or bag or is placed in a location that is out of reach. Alternatively, the apps of the smartphone 100 that are set to be displayed on the screen of the HMD 200 can be deleted from the app screen, and the remaining apps of the smartphone 100 can be expanded and displayed on the screen. Furthermore, not only can the app screen of smartphone 100 be displayed on the display screen of HMD 200 at all times, but for example, in the case of a telephone app, when an incoming call is received, it is also possible to display the app screen of smartphone 100 at a determined display position and size on the display screen of HMD 200.

[0057] In this embodiment, an app on smartphone 100 is used as an example, but this is not limited to smartphone 100, and can be anything that can be connected to and linked with other devices, such as an app on a smart watch or the screen of a car navigation system.

[0058] Next, a case where a portion of an application screen of the smartphone 100 is selected and displayed on the HMD 200 will be described using FIGS. 2A to 2C. In FIGS. 2A to 2C, the parts shown in FIGS. 1A to 1D and denoted with the same reference numerals operate in the same manner as those already described in FIGS. 1A to 1D, and therefore detailed descriptions thereof will be omitted. FIGS. 2A, 2B, and 2C correspond to FIGS. 1A, 1B, and 1C, respectively. FIG. 2A shows the smartphone 100 on the display surface side, on which an application stored on the smartphone 100 is running and displayed. FIG. 2B shows a state in which the user 10 holds the smartphone 100 in his / her hand and wears the HMD 200, which is connected to the smartphone 100 for information communication, on his / her head. FIG. 2C shows an example of the smartphone 100 screen as seen through the HMD 200 and the display screen 130 of the HMD 200.

[0059] In FIG. 2A , if the screen to be inserted and displayed on the HMD 200 is a portion of an app screen, for example, an area 201 showing tomorrow's weather in the app screen 103 of a weather app, the user 10 first touches the app screen 103 of the weather app with his / her finger to select the weather app, and then selects the area 201 showing tomorrow's weather, which is a portion of the app screen 103 of the weather app, to specify a partial area. Alternatively, the user 10 may specify a partial area by surrounding the area 201 showing tomorrow's weather, which is a portion of the app screen 103 of the weather app, with his / her fingertip. By specifying an area in this manner, a partial app display screen to be inserted and displayed on the HMD 200 is selected. For example, as shown in the upper part of FIG. 2A , the area 201 showing tomorrow's weather in the app screen 103 of the weather app is selected. In this manner, the smartphone 100 may select a partial area of ​​the app screen. Furthermore, as shown in the lower part of FIG. 2A , a specific pattern 202 indicated by a diagonal dotted line is generated and displayed on the selected partial app screen, for example, on the area 201 showing tomorrow's weather.

[0060] When the smartphone 100 enters the display screen of the HMD 200, the camera 121 of the HMD 200 captures the display of the specific pattern 202, as shown in FIG. 2B . The HMD 200 captures and reads the specific pattern 202 moving in the up / down, left / right, and front / rear directions 122 to 127 using the camera 121, and recognizes the position and size of the read specific pattern 202 as the display position of an application screen to be displayed on the display screen of the HMD 200. In particular, the HMD 200 captures and reads the specific pattern 202 moving in the front / rear directions, i.e., the perspective directions 126 and 127, using the camera 121, and recognizes the size of the read specific pattern 202 as the display size of the application screen to be displayed on the display screen of the HMD 200. Furthermore, the HMD 200 generates an area frame 203 indicating the range of the read specific pattern 202.

[0061] A specific pattern 202 is displayed on the display screen 130 of the HMD 200, as shown in the upper part of FIG. 2C. At this time, an area frame 203 of the same size as the area of ​​the specific pattern captured by the camera 121 is displayed in accordance with the movement of the smartphone 100 in the up, down, left, right, near and near directions, as shown in FIG. 2B. The user 10 moves the smartphone 100 in the up, down, left, right, near and near directions while looking at the area frame 203 indicating the range of the specific pattern, and determines a display position and display size suitable for the HMD display of a partial screen of the app. Once the display position and display size are determined, screen display information for a portion of the displayed app is obtained from the smartphone 100 and displayed on the screen of the HMD 200, as shown in FIG. 2C.

[0062] The information about the display image is the same as that described in FIG. 1 . The HMD 200 transmits information about the determined display position and display size to the smartphone 100. The smartphone 100 generates image data for a partial app screen to be displayed on the HMD 200 according to the acquired display area information and transmits the generated image data to the HMD 200. The HMD 200 displays a partial app screen image from the received image data, for example, an image 204 of an area 201 showing tomorrow's weather, which is a partial screen of a weather app, in a specified display area. Thus, as in the case of FIG. 1 , a partial screen of the app can be displayed in a specified display area on the display screen of the HMD 200, achieving the same effect as in the case of FIG. 1 . In this way, an image of a partial area of ​​the app screen is also referred to as an app screen image. In other words, an app screen image that is a partial area refers to, for example, an image of a selected partial area of ​​an app screen displayed on an image display device such as the HMD 200.

[0063] FIG. 3A shows an example of a method for specifying a partial area by encircling a portion of the application screen with a fingertip. As shown in FIG. 3A, the vicinity of a tower, which is a portion of the application screen 102 of the map application, is surrounded with a fingertip, and an enclosed area 301 is selected. Furthermore, a specific pattern 302 indicated by a diagonal dotted line is displayed on the enclosed area 301. Alternatively, the enclosed area may be displayed as a frame such as a thick line. Subsequent operations are the same as those shown in FIGS. 1 and 2.

[0064] FIG. 3B is an example diagram illustrating the case where an app to be displayed on the HMD 200 is selected using display application switching software capable of switching between multiple apps. As shown in FIG. 3B, display application switching software 305 capable of switching between multiple apps is launched on the smartphone 100 and displayed on the display screen. Directly touching an app screen on the smartphone 100 may result in an operation related to the operation of that app (such as specifying a specific location in a map app), which can easily lead to confusion. Therefore, the display application switching software 305 is launched, and the app to be displayed on the HMD 200 is selected by touching with a fingertip one of the icons representing the reduced display of multiple apps displayed thereon, such as the map app icon 306 or the weather app icon 307. In this case, the app selection operation can be performed visually and easily. Note that the operation of selecting an app icon may be performed by other methods than touching with a fingertip, such as inputting the app name by voice.

[0065] 4 is an example of a flowchart illustrating the basic operations of the information processing system, information processing device, and image display device according to this embodiment. In FIG. 4, on the right side of the smartphone 100, an app to be displayed on the display screen of the HMD 200 is selected, and if necessary, a partial screen (partial area) within the app screen is designated (S401). Furthermore, specific patterns 104 and 202 are displayed on the selected app screen or on a partial area of ​​the designated app screen (S402). Meanwhile, on the HMD 200 side, a mode is activated to wait for the specific patterns 104 and 202 displayed on the smartphone 100 to be read (S403). In this state, when the app screen of the smartphone 100 comes into view on the screen of the HMD 200 (S404), the HMD 200 photographs and recognizes the app screen using the camera 121 (S405), reads the specific patterns 104 and 202 on the app screen, and displays area frames 131 and 203 indicating the ranges of the specific patterns 104 and 202 on the display screen of the HMD 200 (S406). Furthermore, the user 10 adjusts the display position and size by moving the smartphone 100 while looking at the displayed area frames 131 and 203. Once a suitable display position and display size for displaying the app screen or a partial area of ​​the app screen on the HMD screen are determined, information about the HMD 200 regarding the display area determined by the display position and display size is sent to the smartphone 100 (S407). The smartphone 100 acquires the display area information and stops displaying the specific patterns 104 and 202 (S408). Next, the user pinches in and out on the app screen of the smartphone 100 as needed to enlarge or reduce the app screen displayed on the HMD 200 (S409). Then, image data of the app screen generated to fit the display size is sent to the HMD 200 (S410). The HMD 200 receives and acquires the image data of the app screen (S411), and displays the app screen image of the smartphone 100 in the specified display area (S412). (Here, if the text on the app screen image of the smartphone 100 displayed on the HMD 200 is too small to see or if the necessary information is not included, the operation of S409 is repeated (not shown).)

[0066] Next, the user 10 selects on the smartphone 100 whether to display the application screen image on the HMD screen at all times or when a predetermined trigger occurs (S413). When all-time display (N in S413) is selected, image data for display is continuously sent from the smartphone 100 to the HMD 200. When a temporary stop of the HMD image display is set (Y in S413), the smartphone 100 transmits information to stop displaying the application screen image to the HMD 200 (S414). When the HMD 200 receives the information to stop displaying the application screen image (S415), it stops displaying the application screen image on the HMD screen (S416). Here, if the user 10 wants to stop displaying the application screen image after selecting all-time display, the user 10 instructs the smartphone 100 to stop displaying the application screen image by operating the HMD 200 (not shown).

[0067] The above sequence makes it possible to easily display the app screen image of smartphone 100 at a suitable display position and size on the display screen of HMD 200. Note that the area surrounded by dotted line 400 indicates the processing sequence within smartphone 100, and the area surrounded by dotted line 402 indicates the processing sequence within HMD 200. Also, the area surrounded by dotted line 403 indicates the processing sequence on the smartphone 100 side up to the setting of the app screen image for display on the HMD screen, and the area surrounded by dotted line 404 indicates the processing sequence on the HMD 200 side up to the setting of the app screen image for display on the HMD screen.

[0068] FIG. 5 illustrates an example of displaying multiple application screen images on an HMD screen in the information processing system, information processing device, and image display device according to this embodiment. Here, a case where two applications, a first and a second, are displayed on the HMD screen will be described. Note that in FIG. 5, the components shown in FIG. 4 and denoted with the same reference numerals have the same operations as those already described in FIG. 4, and therefore detailed description thereof will be omitted. In FIG. 5, the processing sequence for displaying a first application screen image of the smartphone 100 on the HMD screen is the same as that described in FIG. 4. This processing sequence is performed by the processing sequence enclosed by dotted lines 403 and 404, and the first application screen image of the smartphone 100 is displayed in a suitable display area on the HMD screen. Next, when a setting is made on the smartphone 100 side to additionally display another second application screen image on the HMD screen (Y in S501), the processing sequence enclosed by dotted lines 405 and 406, which is the same processing sequence as the processing sequence enclosed by dotted lines 403 and 404, is executed. Then, in addition to the first application screen image of the smartphone 100, the second application screen image of the smartphone 100 is displayed in a display area suitable for display on the HMD screen. Here, the smartphone 100 selects whether to display both the first and second application screen images on the HMD screen at all times or when a predetermined trigger occurs (S502). If all-time display (N in S502) is selected, image data for display is continuously sent from the smartphone 100 to the HMD 200. If it is set to stop the HMD image display (Y in S502), the smartphone 100 transmits display stop information for both application screen images to the HMD 200 (S503). If the HMD 200 receives the display stop information for both application screen images (S504), it stops displaying both the first and second application screen images of the smartphone 100 on the HMD screen (S505). Therefore, the above sequence allows both the first and second app screen images of smartphone 100 to be displayed easily and conveniently on the HMD display screen at a suitable display position and size. In the case of two or more apps, the HMD display process for the second app screen image can be repeated. While an example is shown in which both apps are selected to be displayed constantly or only when triggered, different settings may be selected for each app.In this case, for example, the display method for the first app may be selected before S501.

[0069] 6A and 6B illustrate a case where app screens of multiple partial areas of smartphone 100 are displayed on the HMD screen, in the case of a foldable smartphone 600 with a foldable display screen. Note that in FIGS. 6A and 6B, the parts shown in FIGS. 1A to 1D and 2A to 2C and denoted with the same reference numerals have the same operations as those described in FIGS. 1A to 1D and 2A to 2C, and therefore detailed description thereof will be omitted.

[0070] When the left screen 601 of the left half of the foldable smartphone 600 enters the display screen of the HMD 200, the camera 121 of the HMD 200 captures an image of a specific pattern 602 displayed on the left screen 601, as shown in Fig. 6A. The HMD 200 captures and reads the image of the specific pattern 602 moving in the up / down, left / right, front / back directions 122 to 127 with the camera 121, and recognizes the position of the read specific pattern 602 as the display position to be displayed on the display screen of the HMD 200.

[0071] Next, in FIG. 6B, the size of the loaded specific pattern 602 is recognized as the display size of the left screen 601 displayed on the display screen of the HMD 200 (top left of FIG. 6B). Furthermore, the HMD 200 generates an area frame 603 indicating the range of the loaded specific pattern 602 (middle left of FIG. 6B). While viewing the area frame 603 displayed on the HMD 200, the user 10 moves the foldable smartphone 600 in the up, down, left, right, near, and near directions to determine an appropriate display position and display size for the left screen 601 of the foldable smartphone 600 to be displayed on the HMD. The HMD 200 transmits information about the display area determined by the determined display position and display size to the foldable smartphone 600, and the foldable smartphone 600 generates image data to be displayed in the acquired display area and transmits it to the HMD 200. The HMD 200 displays an image 604 of the left screen 601 in the specified display area based on the received image data.

[0072] Next, an example will be shown in which the right screen 605 of the right half of the foldable smartphone 600 is displayed in a different position. Here, as with the left screen 601, the HMD 200 displays an area frame 607 indicating the range of the loaded specific pattern 606. While viewing the area frame 607, the user 10 moves the foldable smartphone 600 in the up, down, left, right, near, and near directions to determine an optimal display position and display size for the right screen 605 of the foldable smartphone 600 to be displayed on the HMD 200. The HMD 200 displays an image 608 of the right screen 605 of the foldable smartphone 600 in a display area determined by the determined display position and display size. This operation makes it possible to display the left and right halves of the screen of the foldable smartphone 600 in different positions on the screen of the HMD 200 when the foldable smartphone 600 is folded.

[0073] Note that Figures 6A and 6B illustrate the case of two app screens on a foldable smartphone 600, but it goes without saying that the same effect and benefits can be obtained when multiple app screens are displayed on the screens of multiple smartphones 100.

[0074] Next, the contrast harmonization process between the application screen image displayed on the HMD screen and the screen of the HMD 200 will be described using Figures 7A and 7B. If the contrast between the application screen image displayed on the HMD screen and the image on the display screen of the HMD 200 becomes unbalanced by reading a specific pattern, problems such as the inserted application screen image (also called the application insertion screen image) becoming difficult to see or the application insertion screen image being overemphasized, making it difficult to focus on the image on the HMD screen will occur. Here, we will first explain this using a see-through HMD as an example.

[0075] FIG. 7A is an example diagram illustrating contrast harmonization processing for when the surroundings are bright, such as outdoors, making it difficult to see the app screen image, while FIG. 7B is an example diagram illustrating contrast harmonization processing for when the surroundings are dark, such as at night, making the contrast with the app screen image too strong.

[0076] As shown in FIG. 7A , in bright ambient conditions, such as outdoors, the surrounding scenery 801 of the HMD 200 appears bright through the lens, whereas the app screen image 802 may appear relatively dark. In such cases, the app screen image becomes difficult to distinguish due to the ambient brightness. To address this issue, the HMD 200 is equipped with an illuminance sensor for detecting the brightness or darkness of the surroundings, which can detect the brightness around the display position where the app screen image is displayed. Alternatively, if the HMD 200 is equipped with a camera 121 that captures the surroundings, the brightness distribution can be detected from image information captured by the camera 121. The average brightness around the HMD or the brightness around the display position of the app screen image 802 is detected from the brightness information, and the detected ambient brightness information is transmitted to the smartphone 100. The smartphone 100 adjusts the image quality of the app insertion screen image by increasing the brightness according to the received ambient brightness information, and displays the brightened app screen image 803 on the HMD. This sharpens the contrast between the app screen image and the surroundings of the HMD, eliminating or reducing the problem of the app screen image appearing dark and unclear.

[0077] Furthermore, as shown in FIG. 7B , in dark surroundings, such as at night, the app screen image 805 may appear relatively bright compared to the surrounding scenery that appears dark through the lens, resulting in the app screen image being overemphasized. Similarly, in this case, the illuminance sensor of the HMD 200 can detect the ambient brightness (darkness) and adjust the display of the app screen image to an appropriate brightness. Furthermore, if the HMD 200 is equipped with a camera 121, the brightness distribution can be detected from image information captured by the camera 121. The average brightness around the HMD or the brightness around the position where the app screen image 805 is displayed is detected from the brightness information, and the detected ambient brightness information is transmitted to the smartphone 100. The smartphone 100 adjusts the image quality of the app insertion screen image by lowering the brightness according to the received ambient brightness information, and displays the app screen image 806 on the HMD 200. This harmonizes the contrast between the app insertion screen image and the HMD display screen image, eliminating or reducing overemphasis of the app insertion screen image and reducing eye fatigue caused by strong contrast. Furthermore, as an image quality adjustment for the application insertion screen image, the color tone may be adjusted to reduce the emphasis of the application insertion screen image.

[0078] As described in FIGS. 7A and 7B, when the smartphone 100 and the HMD 200 are not linked as described above, i.e., when the smartphone 100 is alone, the smartphone 100 adjusts the brightness of the app screen image to match the surrounding environment. However, when the smartphone 100 and the HMD 200 are linked as described above, the brightness of the app screen image can be adjusted to match the surrounding environment of the HMD 200, thereby ensuring clarity and reducing eye fatigue. Note that, in addition to the method described in FIGS. 7A and 7B, the brightness level of the app screen image to be displayed from the HMD 200 may be transmitted to the smartphone 100 and requested. Furthermore, when the app screen image is displayed on the HMD 200 side, the brightness and color tone of the app screen image may be adjusted and displayed.

[0079] FIG. 8 is an example of a flowchart illustrating the contrast harmonization process between the application display screen and the scenery around the HMD 200, as described in FIGS. 7A and 7B. In FIG. 8, parts with the same reference numerals as those in FIGS. 4 and 5 have the same operations as those already described in FIGS. 4 and 5, and therefore detailed description thereof will be omitted. In FIG. 8, the part surrounded by dotted line 403 represents the processing sequence on the smartphone 100 side from selecting an application to transmitting image data of the application screen of the smartphone 100, and the part surrounded by dotted line 404 represents the processing sequence on the HMD 200 side from reading a specific pattern to displaying an application screen image, which is the same as the description in FIG. 4. The HMD 200 displays the application screen image as shown in the part surrounded by dotted line 404, and detects brightness using an illuminance sensor or captures the surroundings with the camera 121 and acquires luminance from the image of the surroundings of the HMD 200 (S801). Next, the application screen image is compared with the brightness around the HMD 200 (S802). If either one is too bright or too dark (N in S803), the HMD 200 detects the ambient brightness using the illuminance sensor, or captures an image of the surroundings using the camera 121 of the HMD 200, analyzes the brightness information, and transmits information indicating the ambient brightness of the HMD 200 to the smartphone 100 (S804). When the smartphone 100 receives the ambient brightness information (S805), if the received ambient brightness information indicates that the surroundings are brighter than a predetermined threshold relative to the app screen image (S806), it increases the brightness as an image quality adjustment for the app screen to be displayed (S807). On the other hand, if the surroundings are darker than a predetermined threshold relative to the app screen image (S806), it decreases the brightness as an image quality adjustment for the app screen to be displayed, or adjusts the color tone to reduce emphasis (S808). Furthermore, it transmits image data of the app screen after image quality adjustment to the HMD 200 (S809). The HMD 200 acquires image data of the application screen after image quality adjustment (S810), and displays the application screen image after image quality adjustment on the HMD 200 (S811). The application screen image is compared with the brightness around the HMD 200, and if the application screen image is clearly visible or is not extremely bright (Y in S803), the application screen image is displayed as is on the HMD display screen (S812).

[0080] The above sequence allows the image on the app screen to be displayed at an appropriate brightness relative to the brightness around the HMD 200, and the user 10 can view the image on the app screen within the HMD 200 without feeling any contrast or fatigue.

[0081] While FIG. 8 illustrates a case where the HMD 200 is a see-through HMD, the following describes a case where a see-through HMD captures real-world objects and virtual objects using a camera 121 mounted on the front of the head and displays the captured images on a display in front of the user's eyes. First, a specific pattern of the smartphone 100 within the user's field of view is recognized, and an app screen image is inserted and displayed at an appropriate position and size on the HMD 200's display screen. When switching from the surrounding screen display captured by the HMD 200's camera 121 to video content provided within the HMD 200, an external device, or a network, the app screen image may be composited with a portion of the video content and displayed on the display in front of the user's eyes. In this case, depending on the atmosphere of the video content and the purpose of the inserted app screen image, an imbalance in the contrast between the image of the video content information and the app screen image can result in problems such as making the app screen image difficult to see or overemphasizing the app screen image, disrupting the atmosphere of the video content image. In response to this, the HMD 200 detects the content of the video content, for example, its brightness and type (whether it is something that you do not want to be distracted by, such as a movie or music video, or not). The HMD 200 also analyzes the importance of the displayed content of the app screen image based on the type of app, etc., and adjusts the image quality (brightness and, if necessary, color tone) of the app screen image to be inserted. For example, when you want to always focus on the app screen image, the image quality (brightness and, if necessary, color tone) of the app screen image is adjusted so that the app screen image stands out, and the app screen image is inserted and displayed on the HMD display screen. Alternatively, when you want to concentrate on the video content, the image quality (brightness and, if necessary, color tone) of the app screen image is adjusted so that the app screen image does not stand out, and the app screen image is inserted and displayed on the HMD display screen.

[0082] By adjusting the image quality of the application screen image to be inserted in accordance with the image content of the video content information in this way, it is possible to achieve contrast harmony between the video content information image and the application screen image in a manner that corresponds to the importance of attention to the video content information image or the application screen image. As a specific example, if the application screen image contains information that should attract attention, such as emergency information, it can be displayed prominently, and if it is simply additional information, such as the time or weather forecast, it can be displayed in an inconspicuous manner.

[0083] In addition, a request may be made to the smartphone 100 to adjust the image quality of the app screen image displayed on the HMD 200, or the image quality of the app screen image may be adjusted and displayed when the app screen image is displayed on the HMD 200 side.

[0084] FIG. 9 is an example of a flowchart illustrating the contrast harmonization process between the application screen image and the HMD display screen image described above. Note that in FIG. 9, the same reference numerals as those in FIGS. 4 and 5 represent the same operations as those already described in FIGS. 4 and 5, and therefore detailed descriptions thereof will be omitted. In FIG. 9, the portion surrounded by dotted line 403 represents the processing sequence on the smartphone 100 side from selecting an application to transmitting image data for the application screen of the smartphone 100, and the portion surrounded by dotted line 404 represents the processing sequence on the HMD 200 side from reading a specific pattern to displaying the application screen image, which are the same as those described in FIG. 4. In the video see-through mode of the non-see-through HMD, the camera 121 mounted on the HMD 200 captures and displays the surroundings (S901). Additionally, as shown in the portion surrounded by dotted line 404, the application screen image is composited with the camera-captured image and displayed. In this state, by checking whether the contrast is harmonized (S902), it is confirmed whether the application screen image is inserted into the HMD display screen without causing discomfort or discomfort. If the contrast between the application screen image and the HMD display screen image is not harmonious (N in S903), the luminance distribution around the application screen image of the content to be displayed on the HMD is analyzed to obtain luminance information (S904). Next, the luminance distribution of the application screen image is analyzed to obtain luminance information (S910). Then, based on the luminance information around the application screen image on the HMD 200 side, if the surroundings are bright relative to the application screen image, the luminance is increased as an image quality adjustment for the application screen image to be inserted. If the surroundings are dark relative to the application screen image, the luminance is decreased as an image quality adjustment for the application screen image to be inserted, or the color tone is adjusted to reduce emphasis. The application screen image after image quality adjustment is then inserted and displayed on the HMD display screen (S911). If the contrast between the two images is harmonious (Y in S903), the application screen image is inserted and displayed as is on the HMD display screen (S912). This sequence achieves harmonious contrast between the application screen image and the HMD display screen image, thereby reducing discomfort and fatigue for the user 10 due to high contrast.In addition, after loading the app screen in the HMD's video see-through mode, if the user switches the HMD mode and displays content such as a movie or music video on the HMD, the processing from S902 onwards adjusts the contrast so that it harmonizes with the content.

[0085] Next, a process for determining whether to display an app screen image of the smartphone 100 inserted into the HMD screen on the HMD display screen at all times or for each trigger notification will be described with reference to FIG. 10 . FIG. 10 is an example of a flowchart showing a process for performing a constant display or a display at each trigger notification in the information processing system, information processing device, and image display device according to this embodiment. Note that in FIG. 10 , the same reference numerals as those in FIGS. 4 and 5 represent the same operations as those already described in FIGS. 4 and 5 , and therefore detailed description thereof will be omitted. In FIG. 10 , the user 10 selects whether to display the app screen image at all times or for each trigger notification on the smartphone 100 (S1001). Display mode selection information representing the selected display mode is transmitted to the HMD 200 (S1002). The HMD 200 receives the display mode selection information (S1003). If the application screen image to be inserted is to be displayed for each trigger notification (S1004), the HMD 200 hides the application screen image on the HMD screen (S1005) and waits for a notification from the smartphone 100 to start displaying for each trigger notification. Furthermore, if the smartphone 100 is to display an application screen image for each trigger notification (S1006), when the notification for starting displaying for each trigger notification is issued (S1007), the smartphone 100 transmits display start notification information indicating the start of displaying for each trigger notification to the smartphone 100 (S1008). The HMD 200 receives the display start notification information (S1009) and starts displaying the application screen image (S1010). After that, when the smartphone 100 issues a notification to stop displaying for each trigger notification (S1011), the smartphone 100 transmits display stop notification information indicating the stop of displaying for each trigger notification to the smartphone 100 (S1012). The HMD 200 receives the display stop notification information (S1013) and stops displaying the application screen image (S1014). On the other hand, the HMD 200 receives the display mode selection information, and if the application screen image to be inserted and displayed is always displayed (left of S1004), the application screen image is always displayed on the HMD screen (S1015). Note that the display of the application screen on the smartphone 100 may be hidden as necessary.

[0086] Furthermore, when an app screen image is displayed for a while and then the app screen image is erased, an operation may be performed to return the app in the app screen image to the app screen again, so that the app is displayed separately on the app screen. For example, as when capturing an app screen as described with reference to Figures 1 and 2, an operation may be performed to overlay the actual smartphone 100 as a background so that the app screen image displayed on the HMD 200 is superimposed on the smartphone 100 screen, so that the app screen displayed on the HMD 200 is returned to the screen display on the smartphone 100 itself. By defining such an operation, the user 10 can easily understand the operation because the user 10 can display the app screen on the HMD 200 and return it to the smartphone 100 with the same operation.

[0087] The above sequence allows the smartphone 100 to trigger the display of an app screen image on the HMD display screen only when necessary, and prevents it from being a nuisance when unnecessary. For example, in the case of a phone app, the app screen image can be displayed on the HMD screen when an incoming call is received, and the HMD display of the app screen image can be stopped when the call ends. Furthermore, by setting and understanding whether the smartphone 100 should display the app screen image at all times or for each trigger notification, the smartphone 100 can transmit image data of the app insertion screen only when necessary when displaying the app screen image at each trigger notification, which has the advantage of saving battery consumption. When multiple apps from the smartphone 100 are inserted and displayed on the HMD 200, display format selection information for each app may be sent to the HMD 200. The HMD may also select whether to display the app screen image at all times or for each trigger notification.

[0088] FIG. 11A shows an example of a block diagram of a portable smartphone 100 as a specific example of an information processing device. FIG. 11B shows an example of a functional block diagram of an HMD 200 as a specific example of an image display device. In FIGS. 11A and 11B, components shown in FIGS. 1A to 1D and labeled with the same reference numerals have the same operations as those already described in FIGS. 1A to 1D, and therefore detailed descriptions thereof will be omitted. In FIG. 11A, the smartphone 100 is configured appropriately using a camera 1101, a depth sensor 1102, an acceleration sensor 1104, a gyro sensor 1105, a geomagnetic sensor 1106, an operation input interface 1107, a display device 1108, a processor 1120, a memory 1130 including a program 1131 and information data 1132, a vibration generating device 1141, an audio input device 1142, an audio output device 1143, and a communication device 1144, and each component is connected to each other via a bus 1150.

[0089] The camera 1101 converts light incident from the lens into an electrical signal using an image sensor to obtain an image captured by the camera.

[0090] The depth sensor 1102 is a sensor that can measure the distance and angle to an object and capture the shape of the object in three dimensions. There are no limitations on the measurement method, and it can be a TOF (Time Of Flight) sensor that irradiates an object with infrared or other laser light, measures the reflected scattered light, and analyzes and detects the distance to a distant object and the state of the object, a LiDAR (Light Detection and Ranging) that measures the distance by irradiating a subject with pulsed light and measuring the reflection time for each pixel, or a millimeter wave radar that emits millimeter wave radio waves, captures the reflected waves, and detects the distance to the object and the state of the object.

[0091] The acceleration sensor 1104 detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, etc. The acceleration sensor 1104 installed in the smartphone 100 can detect the tilt and orientation of the smartphone 100. The gyro sensor 1105 detects angular velocity in the rotational direction and can detect vertical, horizontal, and diagonal orientations. Therefore, the acceleration sensor 1104 and gyro sensor 1105 can be used to detect the orientation of the smartphone 100, such as the tilt and orientation. The geomagnetic sensor 1106 is a sensor that detects the Earth's magnetic force and detects the direction in which the smartphone 100 is facing. A three-axis type sensor that detects geomagnetic fields in the up and down directions in addition to the front-back and left-right directions can be used to detect the movement of the smartphone 100 by capturing changes in geomagnetic field in response to the movement of the smartphone 100. This makes it possible to detect and determine the orientation of the user 10 carrying the smartphone 100.

[0092] The processor 1120, which is configured with a CPU or the like, executes programs 1131, such as an operating system (OS) and operation control applications stored in the memory 1130, to control each component of the smartphone 100 and realize functions of the OS, middleware, applications, and other functions. In the smartphone 100, a specific pattern generation and display processing unit 1121, a smartphone image generation processing unit 1122, and an image quality adjustment processing unit 1123 control the operation of each function. The processor 1120 of the smartphone 100 may also control the operation of the HMD 200 via wireless communication. For example, the processor 1120 may control the HMD 200 to display an application screen image provided to the HMD 200 on the HMD 200. Specifically, the processor 1120 may cause the image display device to recognize the display position and display size of the screen of the information processing device, which is captured and read by the camera 121 when the user 10 holds it over the camera 121, as a display area. The processor 1120 may also control the image display device to display the provided screen image of the information processing device in the recognized display area. For example, the processor 1120 may control the image display device to display the supplied application screen image in the recognized display area.

[0093] The memory 1130 is configured with a nonvolatile storage device or the like, and stores various programs 1131 and information data 1132 handled by the processor 1120 or the like. The information data 1132 stores display area information, image data of the application screen of the smartphone 100, image display stop information of the application screen of the smartphone 100, ambient brightness information, display mode selection information, and the like.

[0094] The display device 1108 of the smartphone 100 is configured with a liquid crystal display or the like and displays content via a highly transparent touch panel, and displays the screen of an application in operation, icons for running applications and various status indications, various operation buttons, notifications to the user, etc. Notifications to the user may include an indication that information on the display area has been acquired, an indication that image quality adjustment has been completed, etc.

[0095] The operation input interface 1107 of the smartphone 100 is used by the user 10 to input information to the smartphone 100. In the case of the smartphone 100, a touch panel configured on the display surface of the display device 1108 detects touch operations using a finger, a touch pen, or the like as operation input, allowing the user 10 to set and input the information they want to input. The input operation information may also be captured using, for example, a keyboard or key buttons, or the user 10 may vocalize an input operation and collect the sound using the audio input device 1142 to capture the input operation information. For example, the user 10 selects a screen of the information processing device to be displayed on the image display device by user input using the operation input interface 1107. The user 10 may also select an app screen or a partial area of ​​the app screen to be displayed on the image display device by user input using the operation input interface 1107.

[0096] The voice input device 1142 collects external sounds and the user's own voice using a microphone and converts them into voice data. Instruction information based on the user's voice can be captured in the smartphone 100, and operations corresponding to the instruction information can be easily performed.

[0097] The audio output device 1143 is a device that outputs audio from a speaker or headphones based on audio data, and can notify the user of notifications by audio.

[0098] The vibration generating device 1141 generates vibrations under the control of the processor 1120, and converts notifications sent by the smartphone 100 to the user 10 into vibrations. The vibration generating device 1141 transmits vibrations to the hand holding the smartphone 100, thereby notifying the user 10 of the notification. Examples of notifications to the user include notifications indicating the generation or reading of a specific pattern, the display of an app screen image, and the like, as well as notification contents, and these notification information can improve usability.

[0099] The communication device 1144 is a communication interface for wirelessly communicating at least between the smartphone 100 and the HMD 200 via short-range wireless communication, wireless LAN, or base station communication, and for connecting to a network. It includes a communication processing circuit and an antenna corresponding to each communication interface, and transmits and receives image data captured by a camera, control signals, and the like. Note that short-range wireless communication is performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). Base station communication may be performed using a long-range wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications). For example, the communication device 1144 supplies an image of the screen of the information processing device to the image display device. The communication device 1144 also supplies an application screen image, which is an image of a partial area or an application screen image to be displayed on the image display device, to the image display device.

[0100] The specific pattern generation and display processing unit 1121 of the smartphone 100 generates and displays a pattern to be read by the HMD 200 on the entire screen of the smartphone 100 or a partial area of ​​an application screen selected by the user 10. For example, the specific pattern generation and display processing unit 1121 generates and displays a specific pattern that covers the selected application screen or a partial area of ​​the application screen. The specific pattern generation and display processing unit 1121 may also make the generated and displayed specific pattern semi-transparent.

[0101] The smartphone image generation processing unit 1122 of the smartphone 100 generates image data of an application screen such as an application to be displayed within the display area of ​​the HMD screen based on the display area information from the HMD 200.

[0102] When the smartphone display screen is difficult to see or the contrast between the app screen image and the HMD screen image is not harmonious, the image quality adjustment processor 1123 adjusts the image quality (brightness, color tone) of the app screen image so that the two images are harmonious. For example, when the identified contrast harmony identifies a contrast mismatch, the image quality adjustment processor 1123 adjusts the image quality of the app screen image so that the contrast is harmonious. Specifically, the image quality adjustment processor 1123 adjusts the brightness of the app screen image according to the output from the illuminance sensor.

[0103] Next, in FIG. 11B, the HMD 200 is configured appropriately using a camera 121, a depth sensor 1202, an illuminance sensor 1203, an acceleration sensor 1204, a gyro sensor 1205, a geomagnetic sensor 1206, an operation input interface 1207, a display processing device 1208, a processor 1220, a memory 1230 consisting of a program 1231 and information data 1232, a vibration generating device 1241, an audio input device 1242, an audio output device 1243, and a communication device 1244, and each component is interconnected via a bus 1250.

[0104] The camera 121 captures the field of view in front of the HMD 200, and acquires a camera-captured image by converting light incident from a lens into an electrical signal using an imaging element. The camera 121 mounted on the HMD 200 captures the field of view seen by the user 10 wearing the HMD 200, and can capture specific patterns 104, 202, 302 on the display screen 101 of a smartphone 100 held within the field of view. For example, the user 10 can capture an image with the camera 121 by holding the screen of an information processing device over the camera 121.

[0105] The depth sensor 1202 is a sensor that can measure the distance and angle to an object similar to the smartphone 100 and capture the shape of the object, such as an object, in three dimensions. When the smartphone 100 is in front of the user 10, the depth sensor 1202 provided in the HMD 200 can measure the distance to the smartphone 100 when the camera 121 captures and finds the smartphone 100.

[0106] The acceleration sensor 1204 detects acceleration, which is a change in speed per unit time, and can detect movement, vibration, shock, and the like. The acceleration sensor 1204 provided in the HMD 200 can detect the tilt of the HMD 200 worn on the head of the user 10 and the direction of the face. The gyro sensor 1205 detects the angular velocity in the rotational direction and can detect the vertical, horizontal, and diagonal posture. The acceleration sensor 1204 and gyro sensor 1205 can be used to detect the posture, such as the tilt and direction, of the HMD 200. The geomagnetic sensor 1206 detects the Earth's magnetic force and detects the orientation of the HMD 200. A three-axis sensor that detects geomagnetism in the up-down direction in addition to the front-back and left-right directions can be used to detect the movement of the HMD 200 by detecting geomagnetic changes in response to the movement of the HMD 200. These sensors make it possible to detect and determine the posture of the user 10 wearing the HMD 200.

[0107] The processor 1220 is comprised of a CPU or the like, and executes programs 1231, such as an operating system (OS) and operation control applications, stored in the memory 1230, to control each component of the HMD 200 and realize functions of the OS, middleware, applications, and other functions. The memory 1230 is comprised of a non-volatile storage device or the like, and stores various programs 1231 and information data 1232 handled by the processor 1220 or the like. The information data 1232 stores image data of an app screen for camera photography, display area information, image data of an app screen whose image quality has been adjusted by the smartphone, image display stop information for smartphone apps, ambient brightness information, display mode selection information, and the like.

[0108] In the case of an optical see-through type HMD 200, the display processing device 1208 of the HMD 200 projects characters, symbols, video content, virtual objects, etc. using a small projector. Also, as shown in this embodiment, it projects camera-captured images transmitted from the smartphone 100 and notification information for the user 10. The projected virtual objects, etc., are provided with a transparent half mirror that forms an image in front of the user's eyes. This allows the user 10 to view the formed virtual objects superimposed on real objects in the field of view in front of the user's eyes, as if they were floating. In the case of a video see-through type HMD 200, it is configured with a display such as an LCD panel that combines and displays the image in front of the user's eyes captured by the camera 121 with virtual objects, etc. This allows the user 10 to view the real objects in the field of view image in front of the user's eyes and the virtual objects, etc., superimposed on each other.

[0109] The operation input interface 1207 of the HMD 200 often uses gesture operation or voice input. Although there are few locations on the HMD for installing operation input interfaces such as buttons or touch panels, they may be provided in a position or form that makes it easy for the user 10 to perform input operations. Alternatively, they may be separated from the main body of the HMD 200 and connected to a mouse or keyboard via a wired or wireless connection. Alternatively, a virtual input operation screen may be displayed on the display screen of the display processing device 1208, and input operation information may be captured based on the position on the input operation screen to which the gaze is directed. Alternatively, a pointer may be displayed on the input operation screen and the input operation information may be captured by operating the pointer using the operation input interface 1207. Alternatively, the user 10 may utter a voice indicating an input operation, and the voice may be collected by the voice input device 1242, and the input operation information may be captured.

[0110] The audio input device 1242 uses a microphone to collect external sounds and the user's own speech and converts it into audio data. Instruction information based on the speech of the user 10 can be captured in the HMD 200, and actions corresponding to the instruction information can be easily executed. The audio output device 1243 outputs audio from a speaker or headphones based on the audio data, so that notification information can be provided to the user by audio.

[0111] The vibration generating device 1241 generates vibrations under the control of the processor 1220, and converts notification instruction information for the user 10 transmitted by the HMD 200 into vibrations. The vibration generating device 1241 transmits vibrations to the head of the user wearing the HMD 200, thereby notifying the user 10 of the notification instruction information. Examples of notification information for the user 10 include notifications indicating an incoming call or a warning about danger, the reading of a specific pattern in this embodiment, the display of an application screen image, etc., and these notifications can improve usability.

[0112] The communication device 1244 is a communication interface that performs wireless communication between the smartphone 100 and the HMD 200 and connects to a network via short-range wireless communication, wireless LAN, or base station communication. It includes a communication processing circuit, an antenna, and the like corresponding to a predetermined communication interface, and transmits and receives image data captured by a camera, control signals, and the like. Note that short-range wireless communication is performed using a wireless LAN such as Bluetooth (registered trademark), IrDA (Infrared Data Association, registered trademark), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or Wi-Fi (registered trademark). Note that long-range wireless communication such as W-CDMA (Wideband Code Division Multiple Access, registered trademark) or GSM (Global System for Mobile Communications) may be used for base station communication.

[0113] The specific pattern read area frame generation unit 1221 of the HMD 200 reads a specific pattern displayed on the screen of the smartphone 100 and generates an area frame of the specific pattern on the screen of the HMD 200. For example, the specific pattern read area frame generation unit 1221 captures and reads the screen of the selected information processing device by having the user 10 hold the camera 121 over it. The specific pattern read area frame generation unit 1221 then recognizes the display position and display size of the read screen of the information processing device as the display area. Alternatively, the specific pattern read area frame generation unit 1221 may capture and read a selected app screen or a partial area of ​​the app screen by having the user 10 hold the camera 121 over a suitable position, and recognize the display position and display size of the read app screen or partial area as the display area. Furthermore, the specific pattern read area frame generation unit 1221 may generate an area frame indicating the range of the specific pattern generated and displayed by the information processing device and display the area frame on the display screen of the image display device.

[0114] The application image synthesis display processing unit 1222 of the HMD 200 acquires image data of the application screen from the smartphone 100 and performs processing to insert and synthesize the acquired image data of the application screen into the HMD screen image. For example, the application image synthesis display processing unit 1222 displays an image of the screen of the selected information processing device in the recognized display area. Furthermore, the application image synthesis display processing unit 1222 may display an application screen or an application screen image of a partial area selected on the information processing device in the recognized display area.

[0115] The contrast harmony processing unit 1224 of the HMD 200 performs a process to identify whether the contrast between the application screen image and the HMD screen image is harmonized.

[0116] If the contrast between the application screen image and the HMD screen image does not harmonize, the image quality adjustment processing unit 1223 adjusts the image quality (brightness, color tone) of the application screen image so that the two images harmonize. In the case of a see-through HMD, the brightness and, if necessary, color tone of the application screen image are adjusted according to the brightness of the surroundings of the HMD 200 detected by the illuminance sensor 1203. If the HMD screen is video content, the brightness and color tone of the application screen image are adjusted according to the image content (brightness, type) of the video content.

[0117] The luminance distribution analysis unit 1225 analyzes the average luminance and local luminance intensity within the screen by creating a histogram showing the luminance intensity for each pixel or block of the image displayed on the non-viewable HMD, and adjusts the contrast based on the analysis results.

[0118] With the above configuration, the smartphone 100 displays a specific pattern on a selected app screen displayed on the HMD screen by the specific pattern generation and display processing unit 1121. The HMD reads the specific pattern using the specific pattern read area frame generation unit 1221 and displays an area frame indicating the range of the specific pattern on the HMD screen. The display position and display size are determined by adjusting the smartphone 100's up / down, left / right, front / back movement using the area frame displayed on the HMD 200. After the display area is determined, the smartphone 100 generates an app screen image to be displayed in the display area on the HMD screen by the smartphone image generation processing unit 1122. The HMD 200 composites the app screen image from the smartphone 100 onto the HMD screen and displays it by the app image synthesis and display processing unit 1222. Furthermore, if the contrast harmony processing unit 1224 determines that the contrast between the app screen image to be inserted and the HMD screen image is not harmonized, the image quality adjustment processing unit 1123 adjusts the image quality (brightness, color tone) of the app screen image so that the contrast between the app screen image and the HMD screen image is harmonized.

[0119] In this way, by holding the screen of an information processing device such as the smartphone 100 over an image display device such as the HMD 200 and selecting its placement location, it can be freely and easily displayed at a suitable display position and size on the display screen of the image display device such as the HMD 200. Furthermore, by harmonizing the contrast between the display screen image of the information processing device inserted into the display screen of the image display device and the display screen image of the image display device, it is possible to obtain the effect of allowing the screen of the information processing device such as the smartphone 100 to be viewed in a manner that is harmonious with the display screen image of the image display device. Note that the information processing device controls the image display device, and the image display device communicates with the information processing device to realize the operation of the above-mentioned information processing system.

[0120] Other examples of the information processing system, information processing device, and image display device according to the present embodiment will be described with reference to FIGS. 12A to 12D. FIGS. 12A to 12D are diagrams illustrating an example in which, instead of adjusting and determining a suitable display area on the HMD screen while photographing and recognizing an app screen to be displayed on the HMD screen on the HMD 200 side as described in FIGS. 1A to 1D, 2A to 2C, and the like, the placement locations on the HMD screen for displaying app screens are predefined into multiple divisions, and the placement location of a selected app screen is determined using a position selection app. Note that in FIGS. 12A to 12D, the parts shown in FIGS. 1A to 1D and 2A to 2C and denoted with the same reference numerals operate in the same manner as those already described in FIGS. 1A to 1D and 2A to 2C, and therefore detailed description thereof will be omitted. First, the placement locations on the HMD screen are divided into multiple areas and defined. For example, as shown in FIG. 12A, the placement locations on the HMD screen are divided into 12 areas, and placement locations Area 1 to Area 12 are defined. On the display screen of the smartphone, as shown in FIG. 12B, a location selection app 1302 is displayed, which indicates the defined placement locations as icons, along with selected application screens 102, 103, and 1301. For example, the user 10 selects application screen 1301 as the application screen to be displayed on the HMD screen. Next, as shown in FIG. 12C, the user 10 specifies that the placement location of the selected application screen 1301 should be displayed in Area 2 on the HMD screen by selecting an icon in the location selection app 1302. By this operation, a screen image 1303 of the image application is displayed in Area 2 on the HMD screen, as shown in FIG. 12D. In this way, application screen images can be easily displayed in predefined placement locations on the screen of the HMD 200.

[0121] Furthermore, in the above explanation, the smartphone 100 is described as a specific example of an information processing device, and the HMD 200 is described as a specific example of an image display device, but the information processing device is not limited to the smartphone 100 but includes all devices with an operation display function such as an app, and the image display device is not limited to the HMD but includes all devices with a function to capture images of the surrounding outside world and display the images.

[0122] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0123] Furthermore, the above-described configurations, functions, processing units, processing means, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software, with a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function may be stored in a memory, a recording device such as a hard disk or SSD (Solid State Drive), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communication network.

[0124] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Industrial Applicability]

[0125] An information processing system, an information processing device, and an image display device are provided that display a display screen of an information processing device within the screen of an image display device. [Explanation of symbols]

[0126] 10: User, 100: Smartphone, 101, 130: Display screen, 102, 103: Application screen, 104, 202, 302: Specific pattern, 121, 1101: Camera, 131, 203, 603, 607: Area frame, 200: HMD, 305: Display application switching software, 1102, 1202: Depth sensor, 1203: Illuminance sensor, 1104, 1204: Acceleration sensor 1105, 1205: gyro sensor, 1106, 1206: geomagnetic sensor, 1107, 1207: operation input interface, 1108: display device, 1120, 1220: processor, 1121: specific pattern generation and display processing unit, 1122: smartphone image generation processing unit, 1123: image quality adjustment processing unit, 1130, 1230: memory, 1131, 1231: program, 1132, 1232: information data, 1141, 1241: vibration generating device, 1142, 1242: audio input device, 1143, 1243: audio output device, 1144, 1244: communication device, 1150, 1250: bus, 1221: specific pattern reading area frame generation unit, 1222: application image synthesis and display processing unit, 1223: image quality adjustment processing unit, 122 4: Contrast harmony processing unit, 1225: Brightness distribution analysis unit, 128: Two-dot dash line, 132, 204: Image, 201, 301: Area, 306, 307: Icon, 400, 402, 403, 404, 405, 406: Dotted line, 601: Left screen, 604, 608: Image, 605: Right screen, 801: Landscape, 802, 805, 806: Image, 1302: Location selection app

Claims

1. an image display device equipped with a camera; an information processing device that runs and displays an application, which is application software; and an information processing system having a communication linkage function for information between the image display device and the information processing device, the information processing device selects an application screen that is a screen of the application or a partial area of ​​the application screen by a user input, and generates and displays a specific pattern having a range of the selected application screen or the partial area; The image display device photographs and reads the generated and displayed specific pattern using the camera, recognizes the display position and display size of the read specific pattern as a display area, and displays an app screen image, which is an image of the selected app screen or partial area, in the recognized display area.

2. 2. The information processing system according to claim 1, The information processing system is characterized in that the information processing device makes the specific pattern that is generated and displayed semi-transparent.

3. 2. The information processing system according to claim 1, The information processing system is characterized in that the image display device generates an area frame indicating the range of the specific pattern generated and displayed by the information processing device, and displays the area frame on the display screen of the image display device.

4. 2. The information processing system according to claim 1, The information processing system is characterized in that the information processing device launches and displays display application switching software on the display screen of the information processing device, which can switch between multiple applications, and the icon of each application shown on the launched and displayed display application switching software is selected, thereby selecting the application to be displayed on the image display device.

5. 2. The information processing system according to claim 1, the information processing device selects the application screen or a partial area of ​​the application screen, and generates and displays the specific pattern having a range covering the selected application screen or the partial area; the image display device photographs the generated and displayed specific pattern with the camera, recognizes a display position and a display size of the photographed specific pattern as the display area, and displays an app screen image that is an image of the selected app screen or the partial area in the recognized display area; Furthermore, the information processing device selects another application screen or another partial area of ​​the other application screen displayed on the image display device, and generates and displays another specific pattern having the selected other application screen or another partial area as a range; The image display device photographs the generated and displayed other specific pattern using the camera, recognizes the display position and display size of the photographed other specific pattern as a other display area, and displays another app screen image, which is an image of the other app screen or the other partial area selected by the information processing device, in the recognized other display area.

6. 2. The information processing system according to claim 1, the image display device or the information processing device identifies a degree of harmony of contrast between the application screen image to be inserted into the display screen of the image display device and an image on the display screen of the image display device; The information processing system is characterized in that, when a contrast incongruity is identified based on the identified contrast harmony, the information processing device adjusts the image quality of the app screen so that the contrast is harmonious.

7. 7. The information processing system according to claim 6, the image display device has an illuminance sensor that detects the brightness of the surrounding field of view; The information processing system is characterized in that the information processing device adjusts the brightness of the application screen in accordance with the output from the illuminance sensor.

8. 2. The information processing system according to claim 1, When the application screen image selected by the information processing device is displayed in the display area of ​​the image display device, the information processing device is allowed to select whether to display it constantly or for each trigger notification, and if the constant display is selected, the image display device always displays the application screen image, and if the display for each trigger notification is selected, the image display device starts or stops displaying the application screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device.

9. An information processing device having a function of running and displaying an application that is application software, and a function of communicating information with an image display device equipped with a camera, The information processing device includes: An application screen or a partial area of ​​the application screen is selected by a user input, generating and displaying a specific pattern having the selected application screen or the partial area as a range; causing the image display device to recognize a display position and a display size of the specific pattern photographed and read by the camera as a display area; supplying an application screen image, which is an image of the application screen or the partial region to be displayed on the image display device, to the image display device; An information processing apparatus, characterized in that the image display device is controlled so as to display the supplied application screen or the application screen image, which is an image of the partial region, in the recognized display area.

10. 10. The information processing device according to claim 9, The information processing apparatus is characterized in that the specific pattern that is generated and displayed is made semi-transparent.

11. 10. The information processing device according to claim 9, The information processing device is characterized in that it causes the image display device to generate an area frame indicating the range of the specific pattern.

12. 10. The information processing device according to claim 9, The information processing device includes: Launching and displaying display application switching software that can switch between the plurality of applications on a display screen of the information processing device; An information processing device characterized in that an icon of each application displayed on displayed application switching software is selected, thereby selecting the application to be displayed on the image display device.

13. 10. The information processing device according to claim 9, The information processing device includes: the application screen or a partial area of ​​the application screen displayed on the image display device is selected, and the specific pattern having the selected application screen or the partial area as a range is generated and displayed; causing the image display device to recognize the display position and display size of the specific pattern photographed and read by the camera as the display area; supplying an application screen image, which is an image of the application screen or the partial region to be displayed on the image display device, to the image display device; controlling the image display device to display the supplied application screen image in the recognized display area; Furthermore, the information processing device another application screen or another partial area of ​​the another application screen displayed on the image display device is selected, and another specific pattern having the selected another application screen or another partial area as a range is generated and displayed; causing the image display device to recognize the display position and display size of the other specific pattern photographed and read by the camera as another display area; supplying to the image display device another application screen image that is an image of the other application screen or the other partial region to be displayed on the image display device; an information processing device that controls the image display device so as to display the supplied screen image of the other application in the recognized display area;

14. 10. The information processing device according to claim 9, The information processing device is characterized in that when a contrast mismatch between the app screen image inserted into the screen of the image display device and the image on the display screen of the image display device is identified by the information processing device or the image display device, the information processing device adjusts the image quality of the app screen so that the contrast is harmonious.

15. 15. The information processing device according to claim 14, The information processing device adjusts the brightness of the application screen in accordance with an output from an illuminance sensor of the image display device that detects the brightness of an area around the field of view.

16. 10. The information processing device according to claim 9, When the application screen image selected by the information processing device is displayed in the display area of ​​the image display device, the information processing device selects whether to display it constantly or for each trigger notification, and if the constant display is selected, the information processing device controls the image display device to always display the application screen image, and if the display for each trigger notification is selected, the information processing device generates trigger notification information for starting or stopping display, and controls the image display device to start or stop displaying the application screen image according to the generated trigger notification information.

17. An image display device having a function of communicating with an information processing device that runs and displays an application software, and a camera, The image display device includes: an application screen, which is a screen of the application selected by a user input in the information processing device, or a specific pattern generated and displayed in a partial area of ​​the application screen, is photographed and read by the camera; The display position and display size of the read specific pattern are recognized as a display area; An image display device, characterized in that an application screen image, which is an image of the application screen or the partial region selected by the information processing device, is displayed in the recognized display area.

18. 18. The image display device according to claim 17, The image display device is characterized in that the specific translucent pattern generated and displayed by the information processing device is photographed and read by the camera.

19. 18. The image display device according to claim 17, The image display device is characterized in that it generates an area frame indicating the range of the specific pattern generated and displayed by the information processing device, and displays the area frame on a display screen of the image display device.

20. 18. The image display device according to claim 17, The image display device is characterized in that it displays an application screen image, which is an image of the application screen or a partial area of ​​the application of the icon selected from the icons of each application displayed on display application switching software launched and displayed on the information processing device.

21. 18. The image display device according to claim 17, The image display device includes: photographing and reading the specific pattern generated and displayed on the application screen or the partial area selected by the information processing device with the camera; The display position and display size of the read specific pattern are recognized as the display area; Displaying the application screen image in the recognized display area; Furthermore, the image display device photographing and reading, by the camera, another specific pattern generated and displayed on another application screen or another partial area of ​​another application screen selected by the information processing device; The display position and the display size of the read other specific pattern are recognized as another display area; An image display device characterized in that another application screen image, which is an image of the other application screen or the other partial area selected by the information processing device, is displayed in the recognized other display area.

22. 18. The image display device according to claim 17, The image display device is characterized in that when the information processing device or the image display device identifies a contrast mismatch between the app screen image inserted into the screen of the image display device and the image on the display screen of the image display device, it displays the app screen image whose image quality has been adjusted by the information processing device so that the contrast is harmonized.

23. 23. The image display device according to claim 22, The image display device has an illuminance sensor that detects the brightness of the surrounding field of view, and displays the app screen image whose brightness has been adjusted by the information processing device in accordance with the output from the illuminance sensor.

24. 18. The image display device according to claim 17, When the application screen image selected by the information processing device is displayed in the display area of ​​the image display device, if constant display is selected by the information processing device, the image display device always displays the application screen image, and if display per trigger notification is selected by the information processing device, the image display device starts or stops the display of the application screen image in accordance with trigger notification information for starting or stopping display generated by the information processing device.

Images