Information processing system and information processing method

Abstract

The present disclosure provides an information processing system comprising a processor that recognizes movement of a part of the body of a user in the field of view of a head-mounted display worn by the user and performs image processing of a video corresponding to part of the field of view in accordance with the movement of the part of the body of the user in the field of view.

Description

Information processing system and information processing method

[0001] The present invention relates to an information processing system and an information processing method.

[0002] For example, Patent Document 1 describes a technology for displaying an image that is transparent to the outside scenery and can be viewed together with the outside scenery.

[0003] Japanese Patent Application Laid-Open No. 2016-224086

[0004] The technology described in the above-mentioned Patent Document 1 transmits the external scene, and therefore cannot perform image processing on objects included in the field of view. On the other hand, a "video see-through type" is a configuration style of head-mounted display for an augmented reality system using AR (Augmented Reality). This is a method in which a video camera captures an image of the reality in front of the user's eyes and displays the image on a head-mounted display, so that reality is observed as a video image instead of being viewed directly with the naked eye.

[0005] However, since the original idea behind AR was to superimpose computer information (AR images) on the real field of vision, video see-through images were usually presented at the same viewing angle as the real field of vision.

[0006] Therefore, the present disclosure proposes an information processing system and an information processing method that can present an image that differs from the actual field of view in response to a user's operation.

[0007] According to the present disclosure, an information processing system is provided that includes a processor that recognizes the movement of a part of a user's body in the field of view of a head-mounted display worn by the user, and performs image processing of an image corresponding to the part of the field of view in accordance with the movement of the part of the user's body in the field of view.

[0008] 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment; FIG. 2 is a block diagram illustrating an example of the functional configuration of an information processing device according to an embodiment; FIG. 3 is a diagram illustrating an overview and comparison of an optical see-through type and a video see-through type; FIG. 4 is a diagram illustrating an example of an operation in which a field of view is enlarged by a fingertip gesture in a video see-through field of view; FIG. 5 is a diagram illustrating an example of an operation in which a field of view change area ES is operated; FIG. 6 is a diagram illustrating an example of an operation in which the range of the field of view change area ES is changed; FIG. 7 is a diagram illustrating an example of an operation in which a field of view change area ES is made to appear by a fingertip gesture; FIG. 8 is a diagram illustrating an example of a configuration of an information processing system when a controller is used; FIG. 9 is a diagram illustrating an example of a designation operation using a cursor on a controller; FIG. 10 is a diagram illustrating an example of operation with one hand; FIG. 11 is a diagram illustrating an example of a user's field of view before and after enlargement; FIG. 12 is a diagram illustrating an example in which only the central portion is enlarged without losing continuity; FIG. 13 is a diagram illustrating an example of a transformation of the time axis; FIG. 14 is a hardware configuration diagram illustrating an example of a computer that realizes the functions of an information processing device;

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] The explanation will be given in the following order: 1. Embodiment 1-1. Video see-through augmented reality system 1-2. Operation of field of view change area 1-3. Changing the range of field of view change area 1-4. Variations 1-4-1. Displaying area with gestures 1-4-2. Use of controller 1-4-3. Operation with one hand 1-4-4. Continuous transformation from normal field of view 1-4-5. Preventing blurring when enlarging 1-4-6. Example of image processing 1-4-7. Application examples other than image enlargement 1-4-8. Displaying multiple field of view change areas 1-4-9. Displaying multiple field of view change areas in multiple stages (nested structure) 1-4-10. Displaying field of view change areas in full screen 1-4-11. Images different from the real thing 1-4-12. Operating field of view change area with voice 1-4-13. Operating field of view change area with the back of the hand, arm, or elbow 1-4-14. Operating the field of view change area with your feet 1-4-15. Handle for the field of view change area 1-4-16. Displaying the field of view change area in an optical see-through format 2. Hardware configuration example 3. Summary

[0011] 1. Embodiment First, a configuration example of an information processing system according to an embodiment will be described using Fig. 1. Fig. 1 is a diagram showing a configuration example of an information processing system according to an embodiment. As shown in Fig. 1, the information processing system according to the embodiment includes, for example, an information processing device 100.

[0012] The information processing device 100 is a wearable display worn on a user's head to realize, for example, AR (Augmented Reality). More specifically, the information processing device 100 may be, for example, a head-mounted display (HMD) or AR glasses. Furthermore, the information processing device 100 may also be a wearable display worn on a user's head to realize not only AR but also VR (Virtual Reality) and MR (Mixed Reality).

[0013] The information processing device 100 has, for example, a sealed or transmissive display unit (display). The information processing device 100 may be, for example, either an optical see-through type or a video see-through type. In this embodiment, the video see-through type will be described as an example. The information processing device 100 also performs so-called AR display. For example, the information processing device 100 displays a virtual object represented by CG or the like on the display by superimposing it on an image of the real world (real space). The information processing device 100 may also perform so-called VR display. For example, the information processing device 100 displays a virtual world (virtual space) represented by CG (Computer Graphics) or the like on the display. The information processing device 100 may also perform so-called MR display. For example, the information processing device 100 displays a fusion of the virtual world and the real world.

[0014] In the information processing system according to the embodiment, for example, the information processing device 100 includes a sensor that detects a user's finger or hand, and the user's finger or hand movements (gestures) detected by the sensor can cause the user to perform various operations in a three-dimensional space such as AR. For example, the user can perform a selection operation, a determination operation, a scroll operation, or the like on a virtual object displayed on the display of the information processing device 100 by the user's finger or hand movements detected by the sensor.

[0015] Furthermore, in the information processing system according to the embodiment, for example, the information processing device 100 may further include a sensor that detects the user's line of sight, and in addition to the movement of the user's fingers or hands, the user's line of sight detected by the sensor may also enable the user to perform various operations in a three-dimensional space such as AR.

[0016] An example of the functional configuration of the information processing apparatus 100 according to the embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the information processing apparatus 100 according to the embodiment.

[0017] The information processing device 100 includes an operation unit 110, a sensor unit 120, a control unit 130, a storage unit 140, a communication unit 150, and a display unit 160. The information processing device 100 may be, for example, an information processing device, but the configuration for realizing general functions of an information processing device is not shown in the figure because it is common.

[0018] The operation unit 110 is an input device that accepts various operations from the user. The operation unit 110 acquires user operations on a UI (User Interface), such as virtual buttons and icons, displayed on the screen of the display unit 160. The operation unit 110 also includes, for example, a camera for acquiring user gesture operations, a microphone for accepting voice input from the user, and operation buttons (physical buttons) for the user to directly operate and input.

[0019] The sensor unit 120 includes various sensors mounted on or connected to the information processing device 100. The sensor unit 120 may include, for example, an image sensor such as a camera, a sound sensor such as a microphone, a LiDAR (Light Detection and Ranging / Laser Imaging Detection and Ranging), a ToF (Time of Flight) sensor, an IMU (Inertial Measurement Unit), and the like. The sensor unit 120 may also include an acceleration sensor, a gyro sensor, a motion sensor, a temperature sensor, a light sensor, a barometric pressure sensor, a magnetic sensor, and the like. The sensor unit 120 may be configured to include some of the sensors, or may include other sensors in addition to or instead of the sensors. The sensor unit 120 detects the environment surrounding the information processing device 100, the user's actions or circumstances, and the like, and outputs sensor values ​​indicating the detection results to the control unit 130. The sensor values ​​indicating the detection results may be output signals from the sensors. The sensor unit 120 may also be an external device.

[0020] The control unit 130 is realized by a processor such as a CPU (Central Processing Unit), MPU (Micro Processing Unit), GPU (Graphics Processing Unit), or NPU (Neural network Processing Unit) executing a program stored inside the information processing device 100 (e.g., an information processing program according to the present disclosure) using RAM (Random Access Memory) or the like as a working area. The control unit 130 may also be a semiconductor integrated circuit (IC chip) having a function of controlling other elements, an electronic board or component on which such a circuit is mounted, or may be realized by an integrated circuit such as an ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array), or MCU (Micro Controller Unit).

[0021] The control unit 130 controls the overall operation of the information processing device 100. For example, the control unit 130 displays an image captured by a camera on the display unit 160 and performs image processing such as enlarging or reducing the image, changing the frame rate, or changing the image quality of the image displayed on the display unit 160. At this time, the control unit 130 recognizes a part of the user's body captured by the camera or a gesture operation by the user, determines and identifies the type of the user's body part or the user's gesture operation, and performs image processing on the image displayed on the display unit 160 based on the result. The control unit 130 also superimposes a UI that the user can operate on the image displayed on the display unit 160. The control unit 130 also acquires information indicating user operations acquired by the operation unit 110 and performs image processing on the image displayed on the display unit 160 in accordance with the user operations. The control unit 130 also acquires sensor values ​​indicating detection results from the sensor unit 120 and recognizes the environment around the information processing device 100, the user's actions or situations, etc.

[0022] The control unit 130 also transmits various types of information or data to an external server or the like (which may be a cloud server) via the communication unit 150 and the network NW, and receives various types of information or data from the external server or the like. The control unit 130 may also be controlled by an external server or the like via the communication unit 150 and the network NW. For example, the control unit 130 may receive display control from the control device 400 regarding the display mode of various types of information or data displayed on the display unit 160. The control unit 130 may also communicate with / link to smart devices such as smartphones and tablet terminals via the communication unit 150.

[0023] Furthermore, the control unit 130 receives signals transmitted from GPS (Global Positioning System) satellites via a GPS receiver, and acquires location information indicating the current location of the information processing device 100, such as latitude and longitude, based on the received signals. That is, the control unit 130 measures the location of the information processing device 100. Note that GPS is merely one example of a GNSS (Global Navigation Satellite System). Furthermore, the control unit 130 can measure the location using various methods other than GPS.

[0024] The storage unit 140 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk drive (HDD), a solid state drive (SSD), or an optical disk. The storage unit 140 temporarily or permanently stores various information or data handled by the control unit 130. For example, the storage unit 140 temporarily stores information indicating operations acquired by the operation unit 110, sensor values ​​indicating detection results from the sensor unit 120, information received by the communication unit 150 via the network NW, and the like. Furthermore, for example, the storage unit 140 may store a program executed by a processor that realizes the control unit 130.

[0025] The communication unit 150 is realized by, for example, a communication circuit, an antenna, a slave (adapter), a NIC (Network Interface Card), a network interface controller, or the like according to various communication methods. The communication unit 150 is connected to a network NW via a wired or wireless connection and transmits and receives various information or data via the network NW. The network NW is realized by a wireless communication standard or method such as Bluetooth (registered trademark), Wi-Fi (registered trademark), a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, UWB (Ultra Wide Band), LPWA (Low Power Wide Area), or a mobile network. Examples of mobile networks include LTE (Long Term Evolution), 4G (4th Generation), and 5G (5th Generation).

[0026] The display unit 160 is realized by various display devices such as a liquid crystal display (LCD) or an organic electro-luminescent display (OLED). The display unit 160 displays an image of the real world (real space) captured by a camera. The display unit 160 also displays a virtual object by superimposing it on the image of the real world (real space).

[0027] <1-1. Video See-Through Augmented Reality System> The information processing system according to the embodiment realizes a video see-through augmented reality system that can expand the real world.

[0028] Traditionally, for detailed work in the real world, it is common to use magnifying glasses, and sometimes even higher magnification devices such as stereo microscopes. This suggests that the resolution of vision is not sufficient compared to the precision of manual manipulation.

[0029] On the other hand, one configuration style of head-mounted display (AR-compatible HMD) for an augmented reality system using AR is the "video see-through type." This is a method in which a video camera captures an image of reality in front of the user's eyes and displays the image on a head-mounted display. Instead of viewing reality directly with the naked eye, the user observes reality as a video image. As shown in Figure 3, compared to the "optical see-through" method, which superimposes an AR image on the user's field of view using a half mirror or the like, this method has the advantage of making it easier to adjust the occlusion and positional relationship between the live-action and AR image. Figure 3 is a diagram illustrating an overview and comparison of the optical see-through type and the video see-through type.

[0030] The video see-through method has long been proposed as a configuration style for AR-compatible HMDs, but in recent head-mounted display products, the delay between capturing images with a video camera and displaying them on the display is very small, and the images are high-resolution, making it possible to reproduce the view seen with the naked eye.

[0031] Here, recent head-mounted display products and the like are based on the original idea of ​​AR, which is to "superimpose computer information (AR images) on the real field of vision," so video see-through images have usually been presented at the same viewing angle as the real field of vision.

[0032] However, with video see-through images, there is no need to be limited to reproducing the actual viewing angle. By reproducing images larger than reality as mentioned at the beginning, it is possible to support detailed work such as working with a magnifying glass.

[0033] Switching between a normal field of view and an enlarged field of view can be achieved in various ways. For example, as shown in Figure 4, a field of view change area ES (or its frame) can be superimposed on the image of real space by switching modes using a button or voice, by using a fingertip gesture or pinching with an AR input device, or by holding the viewpoint in one place. Then, by enlarging the field of view change area ES with a fingertip or the like, the object within the field of view change area ES is enlarged, thereby expanding the field of view. Figure 4 is a diagram showing an example of an operation in which the field of view is enlarged using a fingertip gesture in a video see-through field of view.

[0034] The field of view change area ES is displayed in the shape of a frame that allows a range to be specified. The field of view change area ES may have a rectangular, circular, or other shape. The shape of the field of view change area ES is not limited to these shapes and may be any shape. The shape of the field of view change area ES may be changeable by a pre-setting, or by a mode switching operation by a user's input operation or voice after display, or by a user's fingertip gesture or input using an AR input device. The method of change is arbitrary. For example, the information processing device 100 may change the shape of the field of view change area ES according to a pre-setting, or by a mode switching operation by a user's input operation or voice after display, or by a user's fingertip gesture or input using an AR input device.

[0035] Furthermore, the field of view change area ES can be not only enlarged but also reduced. For example, the field of view change area ES can be enlarged with a fingertip or the like, and then reduced to its original size (normal field of view). Furthermore, by reducing the field of view change area ES from its original size (initial state: default) with a fingertip or the like, it is possible to achieve a field of view smaller than the normal field of view. Alternatively, when the field of view change area ES is reduced from its original size, the inside of the field of view change area ES may be maintained at its original size (normal field of view), while the outside of the field of view change area ES may be enlarged. In addition to enlargement and reduction, it is also possible to change the frame rate, image quality, etc. What is changed in the field of view change area ES may be determined in advance, or the setting may be changed by the user operating a button, etc. In this embodiment, enlargement will be described.

[0036] <1-2. Operation of the field of view change area> Operation of the field of view change area ES will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of operation of the field of view change area ES.

[0037] First, as shown in (a) of Figure 5, when the information processing device 100 receives a mode switch via user input operation or voice, or when it detects the user's fingertip gesture, pinching using an AR input device, or holding the viewpoint in one place, it superimposes and displays a field of view change area ES centered on the user's viewpoint on an image of real space (step S101).

[0038] For example, the information processing device 100 makes a field of view change area ES appear on the screen when the user performs an input operation to make a field of view change area ES appear virtually in the field of view (on the screen) or physically in the real world, such as pressing a button, when voice recognition is performed on a keyword spoken by the user to make a field of view change area ES appear, when image recognition is performed on a gesture of the user's fingertip in the field of view that is intended to make a field of view change area ES appear (for example, making a frame or circle with the finger), or when it detects that the viewpoint is remaining in one place on the image.

[0039] Next, as shown in (b) of Figure 5, the information processing device 100 changes the shape of the field of view change area ES in response to user input operations, mode switching by voice, user fingertip gestures, or input by an AR input device (step S102).

[0040] The information processing device 100 may determine the shape of the field of view change area ES according to settings. For example, the information processing device 100 may be able to change the initial shape of the field of view change area ES through pre-settings. Furthermore, the information processing device 100 may change the shape of the field of view change area ES according to the object (subject) included in the field of view change area ES. For example, the information processing device 100 may associate the shape of the field of view change area ES with each type of object included in the field of view change area ES and change the field of view change area ES to one corresponding to the object. Alternatively, the information processing device 100 may determine the shape of the field of view change area ES according to the shape of the object included in the field of view change area ES and automatically change it.

[0041] Next, as shown in FIG. 5C, the information processing device 100 moves the field of view change area ES displayed on the image of real space in response to a movement of the user's viewpoint or a user operation (step S103).

[0042] For example, when the user's gaze is directed from the center of the screen to an edge, the information processing device 100 moves the field of view change area ES from the center to the edge of the screen. Furthermore, when the user taps and drags the area of ​​the field of view change area ES with their finger, the information processing device 100 moves the field of view change area ES in accordance with the movement. Alternatively, when the user pinches and holds the outer frame of the field of view change area ES with the fingertips of both hands and slides in that state, the information processing device 100 moves the field of view change area ES in accordance with the movement.

[0043] 5(d), the information processing device 100 changes the range of the field-of-view change area ES displayed on the image of the real space in response to a user operation (step S104). The details of changing the range of the field-of-view change area ES will be described later.

[0044] Next, when the information processing device 100 receives a mode change by user input operation or voice, or when it detects the user's fingertip gesture, pinching using an AR input device, or holding the viewpoint in one place, it closes the field of view change area ES displayed on the image of real space (step S105).

[0045] For example, when the user presses a button to close the field of view change area ES in the field of view or in real space, when the information processing device 100 recognizes a voice of a keyword to close the field of view change area ES spoken by the user, or when the information processing device 100 recognizes an image of a gesture made by the user's fingertip in the field of view as a gesture intended to erase / end display of the field of view change area ES (for example, making an X with the finger), or when the viewpoint is shifted from the field of view change area ES and a certain amount of time has passed, the information processing device 100 closes / erases / hides the field of view change area ES on the screen.

[0046] When closing the field-of-view changing area ES, the information processing device 100 may store and save the state (magnification, shape, etc.) of the field-of-view changing area ES at that time. When the information processing device 100 makes a previously used field-of-view changing area ES appear (redisplay), the information processing device 100 may make the field-of-view changing area ES appear in the same state as before (for example, enlarged).

[0047] <1-3. Changing the Range of the Field of View Change Region> An operation for changing the range of the field of view change region ES will be described with reference to Fig. 6. Fig. 6 is a diagram showing an example of an operation for changing the range of the field of view change region ES. Note that changing the range of the field of view change region ES corresponds to the processing of step S104 in Fig. 5.

[0048] As shown in FIG. 6A, the information processing device 100 recognizes an action of the user pinching the edge (outer frame) of the field of view change area ES with the fingers of both hands in the field of view (step S201).

[0049] The edge of the field of view change area ES may be any of the corners (for example, any of the four vertices of a rectangle). Also, the action is not limited to pinching the edge (outer frame) of the field of view change area ES with the fingers of both hands, but may be hooking the fingers of both hands around the edge (outer frame) of the field of view change area ES or grasping the edge (outer frame).

[0050] In this case, the information processing device 100 may perform image recognition and identify the user's finger. That is, the information processing device 100 may determine whether the finger is the user's finger or not. The information processing device 100 may also identify individual user fingers. For example, when the hands and fingers of multiple people are reflected in the field of view, the information processing device 100 identifies the finger of the user using / wearing the information processing device 100 from among them and accepts only the finger movements of that user. This also applies to cases where a field of view change area ES appears on the screen through a user's fingertip gesture or when an operation is performed through the movement of the user's fingertip. Furthermore, if the user is wearing gloves, the information processing device 100 may identify the user's finger based on the characteristics of the gloves or the characteristics of the user's hand or finger movements. Furthermore, the information processing device 100 may identify the user's finger based on the position and orientation of the hand and finger in the field of view (within the screen), i.e., the position and orientation of the hand and finger relative to the information processing device 100.

[0051] Next, when the information processing device 100 recognizes the user pinching the edge of the field of view change area ES with the fingers of both hands in the field of view, it determines the center (and position) of the field of view change area ES and the target range (the range of the image to be enlarged / reduced) (step S202).

[0052] For example, when the user pinches the edge of the field of view change area ES with the fingers of both hands in the field of view, the information processing device 100 fixes the center and target range of the field of view change area ES in the field of view (within the screen).

[0053] Next, as shown in FIG. 6B, the information processing device 100 recognizes the user's action of moving the fingers of both hands so as to widen the field of view change area ES in the field of view (step S203).

[0054] Next, the information processing device 100 changes the range and magnification rate (magnification of the image within the area) of the field of view change area ES in accordance with the movement of the user's finger (step S204).

[0055] The range that is changed at this time is the range (size) of the field of view change area ES itself, and the target range does not change. In other words, the range of the image displayed in the field of view change area ES relative to the original image remains the same, and the magnification ratio of the image within the area changes in accordance with the change in the range of the field of view change area ES itself.

[0056] Next, as shown in (c) of Figure 6, when the information processing device 100 recognizes the user removing both fingers from the edge of the field of view change area ES in the field of view, it determines the range and magnification ratio of the field of view change area ES (step S205).

[0057] That is, the information processing device 100 recognizes the user's action of removing both fingers from the edge of the field of view change area ES in the field of view. Then, when the user has performed the action of removing both fingers from the edge of the field of view change area ES in the field of view, the information processing device 100 determines the range and magnification rate of the field of view change area ES in the field of view (within the screen). This makes it possible to enlarge or reduce the image included in the field of view change area ES.

[0058] <1-4. Variations> <1-4-1. Displaying an area by a gesture> In practice, the present invention is not limited to the above example. For example, as another embodiment (variation), the information processing device 100 may superimpose (appear) the field-of-view change area ES in the field of view (on the screen) when the user performs a specific gesture (such as a pinch) in the field of view while the field-of-view change area ES is not displayed in the field of view (on the screen).

[0059] An operation for making the field of view change area ES appear by a fingertip gesture will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of an operation for making the field of view change area ES appear by a fingertip gesture.

[0060] As shown in Figure 7A, when the user moves the viewpoint to a part of the field of view (screen) where he or she wishes to perform image processing such as zooming in, zooming out, changing the frame rate, or changing the image quality, and pinches the desired range with the fingers of both hands, a UI (User Interface) showing the center and area is displayed along the range. This UI is the field of view change area ES.

[0061] For example, the information processing device 100 detects and recognizes a gesture of pinching a user's gaze point and an arbitrary area in the field of view (on the screen) with the fingers of both hands, and in response to the gesture of pinching the user's gaze point and an arbitrary area with the fingers of both hands, superimposes (appears) a field of view change area ES in the field of view (on the screen) in a range corresponding to the arbitrary area (the range pinched with the fingers of both hands) centered on the user's gaze point. At this time, the information processing device 100 may be configured to recognize and identify the user's fingers through image recognition. Alternatively, the information processing device 100 may be configured to identify each individual user's finger.

[0062] In this case, the depth position (horizontal distance) at which the field of view change area ES is superimposed (appears) in the field of view (on the screen) is determined according to the position of the user's finger on the information processing device 100. For example, the position at which the field of view change area ES is superimposed (appears) changes depending on whether the user makes a fingertip gesture near the information processing device 100 or near the object (subject). When the user makes a fingertip gesture near the information processing device 100, the field of view change area ES is superimposed (appears) in front of the user's eyes (closer). When the user makes a fingertip gesture near an object at the back of the screen, the field of view change area ES is superimposed (appears) near the object (at the back of the screen). The depth position (horizontal distance) of the field of view change area ES can also be changed according to the user's operation (such as moving the finger back and forth).

[0063] Next, as shown in FIG. 7B, the user pinches the edge (outer frame) of the field of view change area ES with the fingers of both hands to set the pulling area. That is, the information processing device 100 changes the range of the field of view change area ES and the image processing content in accordance with the movement of the user's fingers. The image processing content includes, for example, the magnification ratio, frame rate, or image quality. This processing is similar to the processing in steps S203 and S204 of FIG. 6.

[0064] Next, as shown in (c) of Fig. 7, the user releases the knob to confirm the area. That is, when the user removes the fingers of both hands from the edge of the field-of-view change area ES in the field of view, the information processing device 100 confirms the range of the field-of-view change area ES and the image processing content. This process is similar to the process of step S205 of Fig. 6.

[0065] Note that the action of pinching and pulling the edge of the field of view change area ES with the fingers of both hands is merely one example. In reality, the action may involve forming a frame shape with the fingers of both hands and then widening the frame (widening the distance between the hands). In this case, when the user forms a frame shape with the fingers of both hands, the information processing device 100 superimposes (appears) the field of view change area ES along the shape of the frame, and changes the range and image processing content of the field of view change area ES in accordance with the action of widening the frame. Alternatively, the action may involve drawing a frame shape with the fingers of one hand and indicating whether to enlarge or reduce the frame. In this case, when the user draws a frame shape with the fingers of one hand, the information processing device 100 superimposes (appears) the field of view change area ES along the shape of the frame, and changes the range and image processing content of the field of view change area ES in accordance with the action of enlarging or reducing the frame. In other words, the fingertip gesture may be any gesture.

[0066] <1-4-2. Use of Controller> An example configuration of an information processing system using an operation controller 200 (operation device) will be described with reference to Fig. 8. Fig. 8 is a diagram showing an example configuration of an information processing system using the controller 200. As shown in Fig. 8, the information processing system according to the embodiment may use a controller 200 capable of some kind of pointing, not limited to the movement of a user's finger or hand detected by a sensor, and may allow the user to perform various operations in three-dimensional space based on the pointing using the controller 200.

[0067] The controller 200 is a controller device that, when operated by a user, moves a virtual object in a three-dimensional space such as VR or AR, operates a virtual operation panel, and performs various operations. The controller 200 is, for example, a VR controller that can input 6 DoF (Degree of Freedom) information. More specifically, the controller 200 includes sensors such as an inertial sensor, an acceleration sensor, a gravity sensor, and a biometric sensor, and detects position and orientation information of the controller 200. The controller 200 then transmits the detected position and orientation information of the controller 200 to the information processing device 100.

[0068] The controller 200 may include, for example, one or more types of input devices selected from the group consisting of an input device, a ring-type input device, a pointing device, and a 6DoF (six degrees of freedom) input device.

[0069] The controller 200 is, for example, an input device held in the user's hand. The controller 200 may include, for example, an operation unit such as a button that can be operated by the user. For example, by pressing a button on the controller 200, the user can perform a selection operation, a confirmation operation, a scroll operation, or the like on a virtual object displayed on the display of the information processing device 100. The controller 200 may also include, for example, a touch sensor and a motion sensor.

[0070] A ring-shaped device, which is an example of the controller 200, is, for example, a ring-shaped input device worn on a user's finger. The ring-shaped device may include, for example, an operating member such as a button that can be operated by the user. For example, by operating the ring-shaped device, the user can change the position and orientation of a virtual object (e.g., a three-dimensional model) in a three-dimensional space such as VR or AR with six degrees of freedom (DoF). Note that the input device is not limited to a ring-shaped input device worn on a user's finger, and may also be a watch-shaped input device worn on the user's wrist.

[0071] A pointing device, which is an example of the controller 200, is an input device capable of pointing to any position in a three-dimensional space such as VR or AR. For example, the information processing device 100 recognizes the 6DoF position and orientation of the pointing device using a tracking method such as a bright spot tracking method, a magnetic tracking method, or an ultrasonic tracking method.

[0072] The 6DoF input device, which is an example of the controller 200, is, for example, an input device that can be operated in 6DoF.

[0073] In the example shown in FIG. 8 , the user holds (or wears) controller 200 in each of their left and right hands to operate them. For example, the user holds and operates controller 200R in their right hand and controller 200L in their left hand. In this case, controllers 200 may be paired, one for the user's right hand and one for the user's left hand. If they are paired, the shapes and functions of the controllers for the right and left hands may be the same or different. Furthermore, predetermined functions may be assigned to the controllers by the user.

[0074] In the example shown in Fig. 8, the user is holding a controller 200 in each hand, but in reality, the user may hold and operate one controller 200 in either the right or left hand. The user may also hold and operate one controller 200 with both hands. For example, one controller 200 may be provided with controls for the right hand and controls for the left hand. Alternatively, the user may be able to switch between right-handed and left-handed use by pressing a button or toggling a switch provided on the controller 200.

[0075] A designation operation using a cursor on the controller 200 will be described with reference to Fig. 9. Fig. 9 is a diagram showing an example of a designation operation using a cursor on the controller 200.

[0076] 9 , when the user holds and operates controller 200 in both hands, information processing device 100 displays (makes appear) two superimposed cursors corresponding to the fingers of both hands in the field of view (on the screen). At this time, it does not matter whether the user holds and operates controller 200 in each hand, or whether the user holds and operates one controller 200 with both hands.

[0077] Alternatively, the information processing device 100 may superimpose (appear) a cursor in the field of view (on the screen) when it receives a mode switch via user input operation or voice, or when it detects the controller 200 itself or a signal from the controller 200.

[0078] The user operates the controller 200 to align two cursors with different positions (e.g., the left and right edges) on the edge (outer frame) of the field of view change area ES, perform a designation operation (click, etc.), and then set the area by dragging each cursor in that state. In other words, the information processing device 100 changes the range of the field of view change area ES and the image processing content in accordance with the movement of the cursors. The image processing content includes, for example, the magnification ratio, frame rate, or image quality. This process is similar to the processes in steps S203 and S204 of FIG. 6.

[0079] When the user holds and operates the controller 200 with one hand, the information processing device 100 may display (appear) only one cursor superimposed in the field of view (on the screen). The user operates the controller 200 to align the cursor with the edge (or a corner) of the field of view change area ES, perform a designation operation (click, etc.), and then drag the cursor in that state to set the area.

[0080] <1-4-3. Operation with One Hand> Furthermore, if one of the user's hands is occupied due to work or the like and the other hand is free (unoccupied), designation may be performed using the viewpoint and a gesture with one hand. An example of operation with one hand will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of operation with one hand.

[0081] 10, when the user has only one hand free, the information processing device 100 recognizes the viewpoint and gestures of the one hand, and when it recognizes that the user is pinching the edge of the field of view change area ES with the fingers of one hand in the field of view, it determines the center (and position) of the field of view change area ES and the target range (the range of the image to be enlarged / reduced). At this time, the information processing device 100 fixes the position of the field of view change area ES with the user's viewpoint as the center.

[0082] Next, the information processing device 100 recognizes the user's action of moving the fingers of one hand to expand the field of view change area ES within the field of view, and changes the range and magnification rate (magnification of the image within the area) of the field of view change area ES in accordance with the movement of the user's fingers.

[0083] Next, when the user removes one finger from the edge of the field-of-view change area ES in the field of view, the information processing device 100 determines the range of the field-of-view change area ES and the image processing content, such as the magnification ratio, frame rate, or image quality.

[0084] The user's field of view before and after enlargement will be described with reference to Fig. 11. Fig. 11 is a diagram showing an example of the user's field of view before and after enlargement. Fig. 11 (A) shows an example of the user's field of view before enlargement. Fig. 11 (B) shows an example of the user's field of view after enlargement.

[0085] 11A, the user holds tweezers in one hand in the field of view (on the screen) and pinches the edge of the field of view changeable area ES with the fingers of the other hand. The information processing device 100 recognizes the user's action of moving the fingers of one hand so as to expand the field of view changeable area ES in the field of view, and changes the range and magnification rate (magnification of the image within the area) of the field of view changeable area ES in accordance with the movement of the user's fingers.

[0086] As a result, as shown in FIG. 11B, the range of the field of view change area ES is expanded compared to the pre-enlargement state shown in FIG. 11A, and the image within the area is displayed in an enlarged form.

[0087] The field of view can be enlarged for the entire video see-through area or for a part of it. Enlarging only a part of the field of view is similar to working with a magnifying glass.

[0088] <1-4-4. Continuous deformation from normal field of view> Furthermore, it is also possible to continuously deform the enlarged portion and the normal field of view. Here, as an example of continuous deformation from the normal field of view, a field of view like that seen through a fisheye lens (fisheye display) will be described. However, this is just one example, and in reality, it is not limited to a fisheye display.

[0089] An example of enlarging only the central portion without losing continuity will be described with reference to FIG. 12 . FIG. 12 is a diagram showing an example of enlarging only the central portion without losing continuity. For example, as shown in FIG. 12 , a user pinches and pulls any portion (virtual area) in the normal field of view with the fingers of both hands, thereby continuously deforming that portion relative to the normal field of view. The example shown in FIG. 12 is an example of enlarging only the central portion without losing continuity using a technique called "Focus + Context." In this way, a visualization technique can be applied that enlarges a focused portion of the real field of view without losing continuity with the entire field of view.

[0090] This embodiment makes it possible to expand the field of view in front of the user's eyes in accordance with the user's purpose by using a video see-through HMD and image processing, thereby supporting precision work in the real world.

[0091] Conventional AR mainly involves superimposing information without changing the image (field of view) of the real world (real space), but in this embodiment, the image (field of view) of the real world itself is transformed. Also, in this embodiment, there is no division of an area such as the field of view change area ES, and the enlarged portion is displayed continuously in the normal field of view.

[0092] <1-4-5. Preventing blurring during enlargement> When an image is enlarged, the movement of the image in response to head rotation is also enlarged, which can be uncomfortable for the user. By applying processing similar to image stabilization in video equipment, blurring of the enlarged field of view can be reduced.

[0093] <1-4-6. Examples of Image Processing> Furthermore, the information processing device 100 may be capable of various image processing operations, not limited to enlargement, reduction, frame rate change, image quality change, and the like.

[0094] The information processing device 100 may change the color temperature or wavelength of the image in the field of view changing region ES. For example, in medical applications, the information processing device 100 may change the wavelength so that blood vessels displayed in the field of view changing region ES can be clearly seen.

[0095] Furthermore, the information processing device 100 may use an image (image outside the current field of view) from another camera attached to the HMD to show something outside the field of view (FOV). For example, the information processing device 100 displays a field of view (image outside the current field of view) different from the current field of view on the entire screen (the entire video see-through area). Alternatively, the information processing device 100 simultaneously displays a field of view (image outside the current field of view) different from the current field of view in a superimposed display area such as the field of view change area ES.

[0096] The field of view inside and outside the FOV may be switched by a user's input operation or mode switching by voice, or by a user's finger gesture or input using an input device for AR. The switching method is arbitrary.

[0097] <1-4-7. Application Examples Other Than Image Enlargement> The information processing device 100 may enable the transformation of the time axis. The transformation of the time axis will be described with reference to FIG. 13 . FIG. 13 is a diagram illustrating an example of the transformation of the time axis. For example, the information processing device 100 displays video of a specified range (field of view change area ES) in slow motion in response to a user's input operation, voice instruction, or a user's fingertip gesture. For example, as shown in FIG. 13 , the information processing device 100 changes normal video to video with a reduced playback rate. Alternatively, the information processing device 100 changes the image change speed (playback speed) like a flip book, rather than the normal video frame rate. This makes it possible to grasp objects or phenomena that would normally be difficult to see due to rapid changes.

[0098] <1-4-8. Displaying Multiple Field-of-View Change Areas> The information processing device 100 may display multiple field-of-view change areas ES. That is, the information processing device 100 may display multiple field-of-view change areas ES. For example, after displaying and confirming a field-of-view change area ES, the information processing device 100 may superimpose (appear) another (new) field-of-view change area ES. That is, the information processing device 100 may individually enlarge multiple portions in the field of view.

[0099] <1-4-9. Multi-stage display (nested structure) of field-of-view change area> The information processing device 100 may display the field-of-view change area ES in a multi-stage manner. For example, after displaying and confirming the field-of-view change area ES, the information processing device 100 may superimpose (appear) another (new) field-of-view change area ES within the field-of-view change area ES when it recognizes that the user is pinching with the fingers of both hands within the field-of-view change area ES. In other words, the information processing device 100 may nest the field-of-view change area ES and further enlarge a specific portion of the enlarged area. Note that the display is not limited to a multi-stage display, and a fisheye display may be used to continuously deform each enlarged portion.

[0100] In this way, the information processing device 100 according to this embodiment can present an image that differs from the actual field of view in response to a user's operation.

[0101] <1-4-10. Full-Screen Display of Field-of-View Change Area> The information processing device 100 may expand the field-of-view change area ES to cover the entire screen. For example, when the user pinches the edges of the field-of-view change area ES with the fingers of both hands in the field of view and then pulls and expands the edges of the field-of-view change area ES to both ends of the field of view (screen) or beyond both ends, the information processing device 100 makes the entire screen the field-of-view change area ES. In other words, the field-of-view change area ES is displayed full-screen.

[0102] <1-4-11. Images that are different from the actual objects> When the information processing device 100 replaces / converts images of the fingers of both hands of the user with other images such as CG, polygons, or processed images, the information processing device 100 may determine the movement of the replaced / converted images as the movement of the fingers of both hands of the user. In other words, the information processing device 100 recognizes not only the fingers of both hands of the user but also CG, polygons, or processed images corresponding to the fingers of both hands of the user as the fingers of both hands of the user, and controls the field of view change area ES. Alternatively, the information processing device 100 may determine the movement of an object corresponding to the fingers of both hands of the user, such as the tip of a pen or a pointer, as the movement of the fingers of both hands of the user.

[0103] <1-4-12. Operating the Field-of-View Change Area by Voice> The information processing device 100 may be configured to allow the user to operate the field-of-view change area ES by voice, for example, when the user's hands are occupied, such as when holding an object in both hands. For example, the information processing device 100 may enlarge or reduce the field-of-view change area ES in response to a voice instruction from the user indicating the size of the field-of-view change area ES.

[0104] <1-4-13. Operation of the field of view change area with the back of the hand, arm, or elbow> The information processing device 100 may change the range of the field of view change area ES and the image processing content in response to an action of aligning / placing the back of the hand, arm, or elbow of the user with the edge of the field of view change area ES (or pressing the edge with the back of the hand, arm, or elbow) and spreading it, not limited to the fingers of both hands of the user. The image processing content is, for example, the magnification ratio, frame rate, image quality, etc. In this case, the information processing device 100 may also determine the back of the hand, arm, or elbow of the user as the fingers of both hands of the user.

[0105] <1-4-14. Operating the Field-of-View Change Area with Feet> The information processing device 100 may change the range of the field-of-view change area ES and the image processing content in response to the action of pinching (or pressing) and pulling the edges of the field-of-view change area ES with the toes of both the user's feet, not just the fingers of both hands of the user. The image processing content may be, for example, the magnification ratio, frame rate, or image quality. In this case, the information processing device 100 may also determine the toes of both the user's feet as the fingers of both the user's hands. Alternatively, the information processing device 100 may change the range of the field-of-view change area ES and the image processing content in response to the action of the user stepping on (or pressing) the edges of the field-of-view change area ES with both feet (even while wearing socks or shoes) and spreading both feet apart.

[0106] <1-4-15. Handles of the Field of View Change Area> The information processing device 100 may display handles (such as small white circles) on the edges or corners of the field of view change area ES, allowing the user to swipe the handles to set the range. In this case, the information processing device 100 may superimpose (appear) the handles when the user taps (on the edges or corners of) the field of view change area ES with their finger. Furthermore, when superimposing (appearing) the field of view change area ES, the information processing device 100 may superimpose (appear) the field of view change area ES including the handles.

[0107] <1-4-16. Display of Field-of-View Change Area in Optical See-Through Type> The information processing device 100 may superimpose (appear) a field-of-view change area ES on the field of view, not only in a video see-through type head-mounted display, but also in an optical see-through type head-mounted display, and change the range of the field-of-view change area ES using fingertip gestures. That is, the normal field of view is not limited to an image of the real world (real space), but may also be a scene of the real world (real space), i.e., the actual field of view. In this case, the information processing device 100 captures an image of the real world (real space) with a camera, just as with a video see-through type head-mounted display. The information processing device 100 then superimposes the field-of-view change area ES on the field of view and displays an image within the field of view change area ES in the field of view change area ES. When performing a continuous transformation from the normal field of view, the information processing device 100 seamlessly superimposes an image of the real world (real space) on the field of view.

[0108] 2. Hardware Configuration Example The information processing device 100 according to the embodiment of the present disclosure described above is realized by, for example, a computer 1000 configured as shown in FIG. 14 . The information processing device 100 will be described as an example. FIG. 14 is a hardware configuration diagram showing an example of the computer 1000 that realizes the functions of the information processing device 100. The computer 1000 includes a processing circuitry 1100, a RAM 1200, a ROM 1300, a secondary storage device 1400, a communication interface 1500, an input / output interface 1600, a display unit 1700, a camera unit 1800, a microphone 1900, and a speaker 2000. The components of the computer 1000 are connected by a bus 1050.

[0109] The processing circuit 1100 operates and controls each unit based on programs stored in the ROM 1300 or the secondary storage device 1400. For example, the processing circuit 1100 loads the programs stored in the ROM 1300 or the secondary storage device 1400 into the RAM 1200 and executes processing corresponding to the various programs.

[0110] The ROM 1300 stores boot programs such as a Basic Input Output System (BIOS) that is executed by the processing circuit 1100 when the computer 1000 is started up, and programs that depend on the hardware of the computer 1000 .

[0111] The secondary storage device 1400 is a computer-readable recording medium that non-temporarily records programs executed by the processing circuit 1100 and data used by such programs. Specifically, the secondary storage device 1400 is a recording medium that records programs for each process of the information processing device 100 according to an embodiment of the present disclosure, which are examples of program data 1450.

[0112] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550. The communication interface 1500 corresponds to the communication unit 150 provided in the information processing device 100. For example, the processing circuit 1100 receives data from other devices and transmits data generated by the processing circuit 1100 to other devices via the communication interface 1500.

[0113] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the processing circuit 1100 receives data from an input device such as a microphone 1900 or a touch panel via the input / output interface 1600. The processing circuit 1100 also transmits data to an output device such as a display unit 1700 or a speaker 2000 via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading programs and the like recorded on a predetermined recording medium. Examples of the media include optical recording media such as a DVD (Digital Versatile Disc) or a PD (Phase Change Rewritable Disc), magneto-optical recording media such as an MO (Magneto-Optical Disk), tape media, magnetic recording media, and semiconductor memories.

[0114] The display unit 1700 is an interface for displaying information processed by the computer 1000. The display unit 1700 is, for example, a liquid crystal display or an organic electroluminescence display (EL display). The display unit 1700 may also be a touch panel display device or a video projection device.

[0115] The camera unit 1800 is an interface through which the computer 1000 captures images. The microphone 1900 is an interface through which the computer 1000 captures audio. The speaker 2000 is an interface through which the computer 1000 outputs audio processed by the computer 1000. The components of the computer 1000 are connected by a bus 1050. The interfaces do not necessarily need to be provided inside the computer 1000, but may be provided outside the computer 1000 via a network or the like. Furthermore, the components constituting the computer 1000 may be controlled by a circuit different from the processing circuit 1100. For example, the display unit 1700 may be controlled not by the processing circuit 1100 but by a circuit dedicated to display processing provided in the display unit 1700.

[0116] For example, when the computer 1000 functions as the information processing device 100 according to an embodiment of the present disclosure, the processing circuit 1100 of the computer 1000 functions as the control unit 130 by executing a program loaded onto the RAM 1200. The secondary storage device 1400 stores the information processing program according to the present disclosure and various data stored in the storage device 140. The processing circuit 1100 reads and executes program data 1450 from the secondary storage device 1400. Alternatively, the processing circuit 1100 may obtain these programs from another device via an external network 1550. That is, the secondary storage device 1400 does not need to be located inside the computer 1000, but may also be located outside the computer 1000. The processing circuit 1100 is an example of an integrated circuit, and a CPU, an MPU, a GPU, an APU, an NPU, an ASIC, an FPGA, and an MCU can all be considered integrated circuits.

[0117] The above shows an example of the hardware configuration of the computer 1000. Each of the above components may be configured using general-purpose components, or may be configured using hardware specialized for the function of each component. This configuration may be modified as appropriate depending on the technical level at the time of implementation.

[0118] 3. Summary Although preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, the technical scope of the present disclosure is not limited to such examples. It is clear that a person skilled in the art of the present disclosure can conceive of various modified or altered examples within the scope of the technical idea described in the claims, and it is understood that these also naturally fall within the technical scope of the present disclosure.

[0119] Among the processes described in the above embodiments of the present disclosure, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using known methods. In addition, the process procedures, specific names, and information including various data and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0120] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0121] Furthermore, the above-described embodiments of the present disclosure can be combined as appropriate within the scope of the processing content without causing inconsistencies. Furthermore, the order of the steps shown in the sequence diagrams or flowcharts of the present embodiments can be changed as appropriate. For example, the steps may be processed in chronological order, repeatedly, or partially in parallel.

[0122] Furthermore, the effects described herein are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects that will be apparent to those skilled in the art from the description of this specification, in addition to or in place of the above-described effects.

[0123] Note that the following configurations also fall within the technical scope of the present disclosure. (1) An information processing system including a processor that recognizes a movement of a part of a user's body in a field of view of a head-mounted display worn by the user, and performs image processing of a video corresponding to the part of the field of view in accordance with the movement of the user's body part in the field of view. (2) The information processing system described in (1), in which the processor enlarges the video corresponding to the part of the field of view in accordance with the movement of the user's body part in the field of view. (3) The information processing system described in (1), in which the processor identifies a part of the field of view to be subjected to image processing in accordance with the movement of the user's body part in the field of view. (4) The information processing system described in (1), in which the processor superimposes a UI for performing image processing of the video corresponding to the part of the field of view in accordance with the movement of the user's body part in the field of view, and performs image processing of the video corresponding to the part of the field of view in accordance with the user's operation on the UI in the field of view. (5) The information processing system described in (4), in which the processor superimposes a UI for performing image processing of the video corresponding to the part of the field of view in accordance with the user's voice, the user's input operation, or the user's gesture operation. (6) The information processing system according to (4) or (5), wherein the processor superimposes an area displaying an image corresponding to a part of the field of view as the UI, and performs image processing of the image displayed in the area in accordance with a gesture operation of the user on the area. (7) The information processing system according to (6), wherein the processor recognizes the user's hand in the field of view, and enlarges the range of the area and the image displayed in the area in accordance with a gesture operation of the user's hand on the area. (8) The information processing system according to (6) or (7), wherein the processor recognizes the user's finger in the field of view, and enlarges the range of the area and the image displayed in the area in accordance with a gesture operation of the recognized user's finger on the area.(9) The information processing system according to (8), wherein the processor recognizes an action of the user pinching both ends of the region with the fingers of both hands, and enlarges the range of the region and the image displayed in the region in response to the user's operation of pulling the both ends of the region with the fingers of both hands. (10) The information processing system according to (8) or (9), wherein the processor further recognizes the user's viewpoint in the field of view, sets the position of the viewpoint to be the center of the region, recognizes the user's action of pinching the edge of the region with the fingers of one hand, and enlarges the range of the region and the image displayed in the region in response to the user's operation of pulling the edge of the region with the fingers of one hand. (11) The information processing system according to any one of (8) to (10), wherein the processor identifies the user's finger from among multiple fingers of multiple people included in the field of view, and enlarges the range of the region and the image displayed in the region in response to a gesture operation on the region with the identified user's finger. (12) The information processing system according to any one of (4) to (11), wherein the processor superimposes an area displaying an image corresponding to a portion of the field of view as the UI, and performs image processing of the image displayed in the area in response to an operation of the user using a controller on the area. (13) The information processing system according to (12), wherein the processor displays a cursor in the field of view when recognizing the controller or an operation using the controller, and enlarges the range of the area and the image displayed in the area in response to cursor movement corresponding to the operation of the user using the controller on the area. (14) The information processing system according to any one of (1) to (13), wherein the processor continuously deforms the image corresponding to a portion of the field of view without separating it from the normal portion of the field of view in response to a movement of a portion of the user's body in the field of view. (15) The information processing system according to any one of (1) to (14), wherein the processor changes the wavelength of the image in response to a movement of a portion of the user's body in the field of view.(16) The information processing system according to any one of (1) to (15), wherein the processor acquires an image of a field of view different from the current field of view from another camera, and displays the image of the field of view different from the current field of view in accordance with a movement of a part of the user's body in the field of view. (17) The information processing system according to any one of (1) to (16), wherein the processor changes a normal image to an image with a reduced playback rate in accordance with a movement of a part of the user's body in the field of view. (18) The information processing system according to any one of (6) to (11), wherein the processor displays a plurality of the regions in the field of view, and performs image processing of the image displayed in each region in accordance with a gesture operation by the user on each of the plurality of displayed regions. (19) The information processing system according to any one of (6) to (11), wherein the processor superimposes another region on the region, displaying a different image corresponding to a part of the image displayed in the region, and performs image processing of the image displayed in the other region in accordance with a gesture operation by the user on the other region. (20) An information processing method in which a processor recognizes a movement of a part of a user's body in a field of view of a head-mounted display worn by the user, and performs image processing of an image corresponding to the part of the field of view in accordance with the movement of the part of the user's body in the field of view.

[0124] REFERENCE SIGNS LIST 100 Information processing device 110 Operation unit 120 Sensor unit 130 Control unit 140 Storage unit 150 Communication unit 160 Display unit 200 Controller

Claims

1. An information processing system comprising a processor that recognizes the movement of a part of a user's body in the field of view of a head-mounted display worn by the user, and performs image processing of an image corresponding to the part of the field of view in accordance with the movement of the part of the user's body in the field of view.

2. The information processing system according to claim 1, wherein the processor enlarges an image corresponding to a portion of the field of view in response to a movement of a portion of the user's body in the field of view.

3. The information processing system according to claim 1, wherein the processor identifies a portion of the visual field to be subjected to image processing in accordance with a movement of a part of the user's body in the visual field.

4. The information processing system of claim 1, wherein the processor superimposes a UI onto the field of view for performing image processing of an image corresponding to a portion of the field of view in accordance with the movement of a portion of the user's body in the field of view, and performs image processing of an image corresponding to a portion of the field of view in accordance with the user's operation on the UI in the field of view.

5. The information processing system according to claim 4, wherein the processor superimposes on the field of view a UI for performing image processing of an image corresponding to a portion of the field of view in response to one of the user's voice, the user's input operation, and the user's gesture operation.

6. The information processing system according to claim 4, wherein the processor superimposes an area displaying an image corresponding to a part of the field of view as the UI, and performs image processing of the image displayed in the area in response to a gesture operation by the user on the area.

7. The information processing system according to claim 6, wherein the processor recognizes the user's hand in the field of view, expands the range of the area in response to a gesture operation of the user's hand on the area, and enlarges the image displayed in the area.

8. The information processing system according to claim 6, wherein the processor recognizes the user's finger in the field of view, and expands the range of the area and the image displayed in the area in response to a gesture operation on the area with the recognized user's finger.

9. The information processing system according to claim 8, wherein the processor recognizes the user's action of pinching both ends of the area with the fingers of both hands, and expands the range of the area and the image displayed in the area in response to the user's action of pulling both ends of the area with the fingers of both hands.

10. The information processing system of claim 8, wherein the processor further recognizes the user's viewpoint in the field of view, sets the position of the viewpoint as the center of the area, recognizes the user's action of pinching the edge of the area with the fingers of one hand, and expands the range of the area and enlarges the image displayed in the area in response to the user's action of pulling the edge of the area with the fingers of one hand.

11. The information processing system of claim 8, wherein the processor identifies the user's finger from among the fingers of multiple people within the field of view, expands the range of the area in accordance with a gesture operation on the area with the identified user's finger, and enlarges the image displayed in the area.

12. The information processing system of claim 4, wherein the processor superimposes an area displaying an image corresponding to a portion of the field of view as the UI, and performs image processing of the image displayed in the area in response to the user's operation on the area using a controller.

13. The information processing system according to claim 12, wherein the processor, when recognizing the controller or an operation using the controller, displays a cursor in the field of view, expands the range of the area in accordance with the movement of the cursor corresponding to the user's operation using the controller in the area, and enlarges the image displayed in the area.

14. The information processing system according to claim 1, wherein the processor continuously transforms the image corresponding to a portion of the field of view in accordance with the movement of a portion of the user's body in the field of view, without separating it from the normal portion of the field of view.

15. The information processing system of claim 1, wherein the processor changes the wavelength in the image in response to movement of a part of the user's body in the field of view.

16. The information processing system of claim 1, wherein the processor acquires an image of a field of view different from the current field of view from another camera, and displays the image of the field of view different from the current field of view in accordance with the movement of a part of the user's body in the field of view.

17. The information processing system according to claim 1, wherein the processor changes the normal image to an image with a reduced playback rate in response to a movement of a part of the user's body in the field of view.

18. The information processing system according to claim 6, wherein the processor displays a plurality of the areas in the field of view, and performs image processing of the image displayed in each area in accordance with the user's gesture operation on each of the plurality of displayed areas.

19. An information processing system as described in claim 6, wherein the processor superimposes another area within the area, displaying another image corresponding to a part of the image displayed in the area, and performs image processing of the image displayed in the other area in accordance with a gesture operation by the user on the other area.

20. An information processing method in which a processor recognizes the movement of a part of a user's body in the field of view of a head-mounted display worn by the user, and performs image processing of an image corresponding to the part of the field of view in accordance with the movement of the part of the user's body in the field of view.

Images