Information processing device, information processing method, and program

Abstract

To enable excellent display control to be performed even in the time of walking and running of an operator in a wearable type information display device.SOLUTION: A wearable type information display device 1 having a display function of contents comprises: a display unit 10 which displays the contents; and a control unit 13 which executes processing of acquiring a photographed image of an operator of the information display device 1 from a camera 11 attached to the display unit 10, processing of recognizing a face image being an image of the face of the operator from the photographed image, and processing of controlling a display range of the contents on the basis of a state of a principal portion in the face.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Wearing a wearable information processing device, such as a wristwatch, allows users to access a variety of information anytime, anywhere. Such wearable information processing devices have various limitations, such as a limited screen size for displaying information and difficulty in appropriately controlling the display due to the operator's easily changing movements and posture. To address this issue, a technology has been proposed that uses acceleration and angular velocity sensors to detect the operator's movements, such as checking the time on a wristwatch, in order to determine the operator's movements.

[0003] Patent Document 1 proposes a technology for adapting to situations where content is displayed on a limited screen size, in which an acceleration sensor is used to detect movement of a display unit, and the display range of the content is moved in accordance with that movement, thereby displaying the entire content. Patent Document 2 proposes a technology for determining the accuracy of detection of the movement distance of the display unit by calculating an attitude parameter of the display unit and improving the accuracy of the amount of movement of the display unit based on the attitude parameter. Patent Document 3 proposes a screen display control method for an electronic device equipped with a display screen and a camera on the display screen side, in order to easily control the display range of an object displayed on the display screen with operations similar to human sensations. The screen display control method determines the orientation of the user's face based on changes in feature points of the user's face in an image captured by the camera, and controls the scrolling direction and amount of the object displayed on the display screen according to the determined face orientation and the amount of change in the feature points. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2015-176246 [Patent Document 2] Japanese Patent Application Laid-Open No. 2014-182612 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-044729 Summary of the Invention [Problem to be solved by the invention]

[0005] As such, various technologies have been proposed that attempt to display content by effectively utilizing display units with limited display area, such as wearable information processing devices.However, with regard to controlling the display range, there have been problems such as a lack of convenience, as the display range may move in an unintended manner as the information processing device is moved, or the operator may be forced to repeat specific actions when moving the display range significantly.

[0006] The present invention has been made in consideration of such circumstances, and aims to provide a technology in a wearable information processing device that does not limit the display range of content even if the display size of the content is larger than the display unit, and enables good display control in line with the operator's intentions. [Means for solving the problem]

[0007] One aspect of the present invention is a wearable information processing device having a content display function, comprising: a display unit that displays the content; and a control unit that executes a process of acquiring a captured image of an operator of the information processing device from a camera attached to the display unit, a process of recognizing a face image that is an image of the operator's face from the captured image, and a process of controlling a display range of the content based on a state of a main part of the face. and when a face image is recognized in the captured image and the state of the main part of the face remains as a predetermined state for a certain period of time or more, the control unit stops changing the display form of the content on the display unit. . [Effects of the Invention]

[0008] According to the present invention, in a wearable information processing device, even if the display size of the content is larger than the display unit, the display range of the content is not limited, and good display control in line with the operator's intentions is possible. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an overview of an embodiment of an information processing device of the present invention. [Figure 2] 1 is a block diagram showing a configuration of an information display device according to an embodiment of the present invention. [Figure 3A] FIG. 1 is an explanatory diagram showing an overview of a content display operation in an information display device according to an embodiment of the present invention. [Figure 3B] FIG. 1 is an explanatory diagram showing an overview of a content display operation in an information display device according to an embodiment of the present invention. [Figure 4] FIG. 2 is an explanatory diagram showing an example of a face area and its landmarks detected in one embodiment of the present invention. [Figure 5A] FIG. 1 is an explanatory diagram regarding assumed facial poses in one embodiment of the present invention. [Figure 5B] FIG. 1 is an explanatory diagram regarding assumed facial poses in one embodiment of the present invention. [Figure 6A] FIG. 2 is a diagram showing a coordinate system of a clock in one embodiment of the present invention. [Figure 6B] FIG. 2 is a diagram illustrating a camera coordinate system according to an embodiment of the present invention. [Figure 6C] FIG. 2 is a diagram illustrating a coordinate system of content in one embodiment of the present invention. [Figure 7A] FIG. 2 is a diagram showing an example of content to be displayed in one embodiment of the present invention. [Figure 7B] 10A to 10C are diagrams illustrating an example of a display transition on a watch according to an embodiment of the present invention. [Figure 7C] 10A to 10C are diagrams illustrating an example of a display transition on a watch according to an embodiment of the present invention. [Figure 7D] FIG. 10 is a diagram showing an example of the correspondence between the display on the watch and the face in one embodiment of the present invention. [Figure 8] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 9A]10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 9B] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 10A] FIG. 3 is a diagram showing an example of reference face data according to an embodiment of the present invention. [Figure 10B] 10A and 10B are diagrams illustrating examples of measurement face positions in an embodiment of the present invention. [Figure 10C] FIG. 2 is a diagram illustrating an example of a coordinate system of content according to an embodiment of the present invention. [Figure 11A] 10A and 10B are diagrams illustrating an example of a correspondence relationship between a display transition and a facial movement in one embodiment of the present invention. [Figure 11B] 10A to 10C are diagrams illustrating an example of transition of an operator's actions in one embodiment of the present invention. [Figure 11C] 10A and 10B are diagrams illustrating an example of transition of a content display area in an embodiment of the present invention. [Figure 12] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 13A] 10A and 10B are diagrams illustrating an example of transition between approaching and moving away from a face in one embodiment of the present invention. [Figure 13B] 10A to 10C are diagrams illustrating an example of transition of an operator's actions in one embodiment of the present invention. [Figure 13C] 10A and 10B are diagrams illustrating an example of transition of a content display area in an embodiment of the present invention. [Figure 14A] FIG. 2 is a diagram showing a coordinate system of a clock in one embodiment of the present invention. [Figure 14B] FIG. 1 illustrates an initial image in one embodiment of the present invention. [Figure 14C] FIG. 10 is a diagram showing an image after rotation in one embodiment of the present invention. [Figure 15] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 16] FIG. 10 is a diagram showing an example of a face image of an operator in one embodiment of the present invention. [Figure 17A]10A and 10B are diagrams illustrating an example of transition of a display area accompanying scrolling in one embodiment of the present invention. [Figure 17B] 10A to 10C are diagrams showing an example of a transition of the display area at each timing and an entire content image according to an embodiment of the present invention. [Figure 18A] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 18B] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. [Figure 19] 10 is a flowchart showing a flow of information display control in one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0010] <Information display device 1> Hereinafter, an embodiment of the present invention will be described with reference to the accompanying drawings. Fig. 1 is a diagram showing an overview of an information display device 1 as an information processing device in one embodiment of the present invention. The information display device 1 illustrated in Fig. 1 is a wristwatch-type information processing device that appropriately displays content stored by receiving it over a network, and is worn on the wrist 100 of an operator.

[0011] This information display device 1 is equipped with a display unit 10 and a camera 11 attached to the display unit 10. The camera 11 is set and positioned in various conditions such as the angle of view so that the operator faces the display unit 10 and captures the entire face when operating the display unit 10.

[0012] Specifically, the information display device 1 in this embodiment can be considered to be a wristwatch. As indicated by arrow 2, when an operator makes a gesture such that the display unit 10 of the information display device 1 faces in a substantially vertical direction while raising their arm with their elbow bent, the information display device 1 detects this gesture and displays a time image representing time information on the display unit 10. Here, the vertical direction is the direction perpendicular to the plane of the paper in FIG. 1 and penetrating the plane from the back to the front.

[0013] The information display device 1 of this embodiment detects the angular velocity generated in the device, compares the time series data of the detected angular velocity with time series data of angular velocity that represents the gesture pattern of the action of looking at the time on a wristwatch, and detects the gesture based on the similarity. These basic operations and functions are similar to those of the technology disclosed in Patent Document 2 (JP 2014-182612 A), and detailed explanations thereof will be omitted (the same applies hereinafter).

[0014] 2 is a block diagram showing an example of a hardware configuration including the information display device 1 according to this embodiment. As shown in FIG. 2, the information display device 1 includes a display unit 10, a camera 11, an operation unit 12, a control unit 13, a face recognition unit 14, and an attitude sensor 15.

[0015] Of these, the control unit 13 is implemented by a microcomputer having an arithmetic processing unit including a CPU (Central Processing Unit) and a main memory including a ROM (Read Only Memory) and a RAM (Random Access Memory).

[0016] The CPU implements necessary functions by loading the programs stored in the ROM into the RAM and executing them, and controls each part of the information display device 1.

[0017] Furthermore, the camera 11 repeatedly captures the face of the operator and sequentially sends the images of the operator obtained by the capture to the face recognition unit 14. The face detection unit 140 in the face recognition unit 14 detects the face of the operator, which is the subject of the image, from the images sent from the camera 11.

[0018] The face recognition unit 14 is made up of a face landmark extraction unit 141, a face position detection unit 142, a face posture detection unit 143, and a face state detection unit 144 in addition to the face detection unit 140 described above.

[0019] Of these, the face landmark extraction unit 141 identifies the positions of the main features of the operator's face, such as the eyes, nose, eyebrows, and mouth, detected by the face detection described above.

[0020] Furthermore, the face position detection unit 142 detects the position of the face based on the landmarks identified by the face landmark extraction unit 141 .

[0021] Furthermore, the face pose detection unit 143 identifies the three-dimensional direction and pose of the face.

[0022] Furthermore, the face state detection unit 144 identifies the face state of the operator, such as whether the eyes and mouth are open or closed, and whether or not the operator is smiling.

[0023] The display unit 10 is configured using, for example, a liquid crystal display, and displays images generated by the control unit 13. The operation unit 12 is used to input user operations, and corresponds to physical keys mounted on the side of the display unit 10 or a touch panel mounted on the top surface of the display unit 10. Operation input signals from the user (operator) are sent via this operation unit 12 to the control unit 13 for processing. The operation unit 12 may also be configured to accept voice input. In this case, the operator can give instructions to the control unit 13 by voice.

[0024] <Display operation of information display device 1> 3A and 3B are diagrams illustrating an overview of the content display operation performed by the information display device 1. Of these, Fig. 3A is a diagram illustrating content C, which is an example of content to be displayed. Content C here may include image elements such as text, illustrations, and photographs.

[0025] Furthermore, content C is a rectangular content placed in the virtual space, and the vertical and horizontal sizes of content C are assumed to be larger than the vertical and horizontal display sizes of display unit 10.

[0026] When the information display device 1 displays content C on its display unit 10, the display range of the content C is moved in accordance with the operator's movement of the camera 11 including the display unit 10. The display range is a part of the content C displayed on the display unit 10.

[0027] When the main body of information display device 1 is at position A shown in FIG. 3B, the image displayed on display unit 10 is determined to be within the display range indicated by the rectangular range of frame 3A in FIG. 3A.

[0028] When the operator moves information display device 1 from position A to B-H along dotted line 3B and finally reaches display position H, display unit 10 determines display range 3C (FIG. 3A) centered on display position H. That is, the main body of information display device 1 moves the display range of content C in accordance with the movement of camera 11 (a method for detecting camera movement will be described later).

[0029] In this way, by determining the cut-out position of the content C according to the forward and backward and left and right movements of the clock, which is the information display device 1, it is possible to make the user feel as if they are looking at the large content C through a peephole.

[0030] The attitude sensor 15 provided in the information display device 1 is composed of a three-axis magnetic sensor 151 that measures the direction of the geomagnetic field, a three-axis acceleration sensor 152 that measures the acceleration applied to the information display device 1, and a three-axis angular velocity sensor 153 that measures the rotational movement of the information display device 1. Each sensor is capable of making measurements in the directions of three axes, x, y, and z.

[0031] <Face Recognition Unit 14> Next, the functions related to the face recognition unit 14 in the information display device 1 of this embodiment will be described. Many face recognition technologies have already been commercialized. They are also provided as software libraries that are applied to various applications and services. Therefore, detailed descriptions of how they are implemented will be omitted. Here, the explanation will focus on the functions used in the present invention, and will explain the main configuration and operation of face recognition that can be applied in a general purpose manner.

[0032] 2 determines whether a human face is included in the image input from camera 11. If the result of this determination indicates that a face is present in the image, face detection unit 140 extracts the area in which the face exists and its center position.

[0033] Furthermore, the face landmark extraction unit 141 detects landmarks of the face based on the face area detected by the face detection unit 140. Here, landmarks are characteristic points on the face, such as the right eye, left eye, tip of the nose, eyebrows, center of the mouth, edges of the lips, bridge of the nose, chin, etc. Figure 4 shows an example of the area of ​​a face F detected from a photograph P and its landmarks L.

[0034] The face position detection unit 142 determines the center position of the face F (for example, the center position between the right and left eyes) based on the position of the landmark L point detected by the face landmark extraction unit 141. After the face area is detected once by the face detection unit 140, the accuracy is recursively improved again using the landmark L. The determined center position and area of ​​the face F are stored in a register.

[0035] Furthermore, the face pose detection unit 143 detects the pose of the face F based on the landmarks L and the contour and texture of the face F (for specific functional details, see, for example, https: / / developers.google.com / vision / face-detection-concepts).

[0036] The face posture is a numerical representation of the posture of the face tilted relative to the front, based on the coordinate system shown in Figure 5A, as shown in the variations in Figure 5B. When representing this posture, Euler angles are usually used and expressed using the values ​​of Euler X, Euler Y, and Euler Z.

[0037] Although it is not directly used for the display position of content C, it is a concept that can be used to detect facial gestures such as "nodding" or "shaking head" by using changes in the tilt of the face over time. This detected facial posture is stored in a register and can be read out by the control unit 13.

[0038] The facial state detection unit 144 also classifies the state of each part of the face F. For example, the eyes are classified into two states, either closed or open. The mouth is classified into two states, either closed or open. It is also possible to combine the classification results for each part of the face and classify the facial state as either "smiling or not smiling."

[0039] The results of the classification described above are stored in a register as a facial state, which is a combination of values ​​for each state, such as right eye: closed / open, left eye: closed / open, right eyebrow: raised / lowered, left eyebrow: raised / lowered, mouth: open / closed, and facial expression: smiling / not smiling.

[0040] Here, the various detection results regarding the face detected by the face recognition unit 14, such as the face position and area, face posture, and face state, are read out by the control unit 13. The control unit 13 controls the display of the content C in accordance with this information. The detailed display operation in this case and the control method for realizing this display operation will be described in order.

[0041] 6A to 6C are diagrams that define the direction of movement of a clock serving as information display device 1 (hereinafter referred to as "clock 1" for simplicity), the coordinate system of camera 11, and the coordinate system of content C, for the sake of convenience.

[0042] As shown in Figure 6A, in watch 1, the display center of display unit 10 is taken as the origin, the 3 o'clock direction is the X axis, the 6 o'clock direction is the Y axis, and the Z axis is from the surface of display unit 10 toward the zenith. This Z axis is perpendicular to the X axis and Y axis. The Y axis points downward in order to align the direction of the coordinate system of content C.

[0043] 6B, the origin of the image of face F captured by camera 11 is the center of the image, the horizontal direction is the u-axis, and the downward vertical direction is the v-axis. The v-axis of camera 11 points upward in the drawing so that, as will be described later, when watch 1 is moved in the positive direction along the y-axis, face F captured by camera 11 also moves in the positive direction of the v-axis.

[0044] 6C, the coordinate system for content C has the x-axis running horizontally to the right and the y-axis running vertically downward. The y-axis is downward because content C, such as text, is generally arranged from top to bottom, and the eyes move from top to bottom, so the coordinate system is also downward.

[0045] The direction of the coordinate system of watch 1 coincides with the direction of the coordinate system of content C. The black dot between the eyebrows of face F in Figure 6B represents the center of face F (the center of the right and left eyes) captured by camera 11. The coordinates of this center of face F are defined as (uf, vf).

[0046] 7A to 7D are diagrams illustrating the display control of content C and the operation of camera images. FIG. 7A shows content C to be displayed. FIG. 7B is a diagram that allows one to visualize this content C and a portion 7B of that content C that is displayed on the display unit 10 of the watch 1. In this case, the dashed line 7B1 represents the path that the watch 1 follows when the operator moves their arm while keeping their face F fixed.

[0047] The points A, B, C, D, E, F, G, and H on the path of dashed line 7B1 represent positions along the path of movement of watch 1 in response to arm movement. Note that background image 7B2, on which content C is supposed to be displayed on the screen, virtually represents the content image, and actually displays only the portion of the display unit 10 of watch 1.

[0048] The watch 1 initially displays an area of ​​content C centered on A, and then displays the virtual background content in response to the movement of the operator's arm.

[0049] 7C shows what is displayed on the clock 1 at A, B, C, D, E, F, G, and H. The thick black arrow 7C1 in the figure indicates the direction from the previous position to the current reading position, and the black dashed line 7C2 indicates the direction of movement from here on. In other words, the tip of the arrow on this dashed line 7C2 indicates the direction to the next place of interest that the operator wants to read.

[0050] Figure 7D shows face F captured by camera 11 attached to display unit 10 of watch 1 at each of the positions A to H described above. Thick black line 7D1 indicates the direction in which face F has moved from the previous position. Black dot 7D2 in the figure indicates the center of face F (the center between the right and left eyes).

[0051] In this way, the position of face F captured by camera 11 changes in accordance with the movement of clock 1. When clock 1 moves to the right, the position of face F moves to the right, and when clock 1 moves down, face F moves up.

[0052] As a result, the vertical axis direction (the Y axis of watch 1 and the V axis of camera 11) is reversed between the coordinate system of camera 11 and the coordinate system for the operation of watch 1. Therefore, control unit 13 of watch 1 detects the center position of face F and performs control to change the location where content C is displayed according to that position, thereby realizing the transition of the display area of ​​content C on display unit 10 as described above.

[0053] Next, a control method for realizing the above-mentioned operation will be described with reference to Figures 8, 9A, and 9B. Figure 8 illustrates the operation flow of the face detection control unit 145 in the face recognition unit 14.

[0054] 8, the flow starts with a face recognition start command issued by the control unit 13 of the main body. First, the face detection control unit 145 sets a register indicating the recognition operation status to "recognizing" (step S201). The value of this register is used to notify the control unit 13 that the recognition process has ended.

[0055] The face detection control unit 145 reads the latest image from the camera 11 (step S202). The image is taken into the face recognition unit 14. The face detection control unit 145 operates the face detection unit 140 and waits for face detection to be performed (step S203).

[0056] The face detection control unit 145 acquires the face detection result in the face detection unit 140 (step S204), and if a face cannot be detected (step S204: not detected), branches the process to step S212 and sets an error in the detection result register. The face detection control unit 145 also sets the recognition operation status to "completed" (step S211) and ends the flow.

[0057] On the other hand, if a face is detected in step S204 (step S204: detected), the face detection control unit 145 causes the face landmark extraction unit 141 to extract landmarks L from the face detection area data and the image (step S205).

[0058] The information about the landmark L obtained here is configured to be sent to the face position detection unit 142, face pose detection unit 143, and face state detection unit 144. First, the face position detection unit 142 determines the face center position and face area (step S206). Next, the face pose detection unit 143 determines the face pose (step S207). Furthermore, the face state detection unit 144 detects the state of each face (step S208).

[0059] Information such as the face region, face center position, face posture, and face state is held in registers of the respective detection units and is configured to be read out by an external control device.

[0060] The face detection control unit 145 waits for these pieces of information to be updated (step S209). When the update is completed (step S209: Yes), the face detection control unit 145 sets the detection result to "success" (step S210). Then, the face detection control unit 145 sets the recognition operation status to "completed" (step S211) and ends the recognition operation flow.

[0061] <Content display by control unit 13> 9A and 9B show an example of a flow when content display is executed by the control unit 13. Due to space limitations, the illustration is divided into parts, but FIGS. 9A and 9B show a series of operational flows. The control unit 13 starts this flow when it transitions to content display mode. In this case, first, the control unit 13 turns the display lock mode to OFF (step S1). This "display lock mode" indicates whether or not to control display changes, and "ON" means that display changes are not executed. The initial value starts from a state in which display changes are executed.

[0062] Next, the control unit 13 reads out the content C to be displayed onto the memory (step S2). It is assumed that this content C includes image data and image size information.

[0063] Next, the control unit 13 initializes the display range information (display zoom factor scale, display offset) according to the content image size as follows (step S3). Display scale: The ratio of content coordinates to display coordinates. x_scale = horizontal display size of display unit 10 / horizontal size of content y_scale = vertical display size of display unit 10 / vertical size of content Display scale scale=x_scale, y_scale, whichever is larger if scale <1 / 1.5 then scale =1.0 Display offset (xc, yc): The coordinates of the content to be displayed at the center of the display. Display offset xc = content width / 2 off1=content vertical size / 2 off2 = vertical display size of display 10 / display scale / 2 Display offset yc = the smaller of off1 and off2

[0064] It is desirable to set the scale so that the entire image is displayed, but if the ratio exceeds 1 / 1.5, the text becomes small and difficult to read. Therefore, in this case, we decided to set it to 1.0, which is optimal for display. The display offset was determined to display the center of the top of the content image. As an example, the initial display range for content C in Figure 10C is represented by A.

[0065] Next, the function of F1 processed in step S4 will be described.

[0066] F1: The control unit 13 first creates an image of the display range based on the display range information. If the vertical and horizontal display sizes of the display unit 10 are P and Q, the coordinates (x, y) of the content data corresponding to the pixel on the display unit 10 with coordinates (p, q) 0≧p>P, 0≧q>Q are as follows: [Formula 1] x=(pP / 2) / scale+xc [Formula 2] y = (qQ / 2) / scale+yc...Equation (2) Pixel data of display pixel (p, q) = pixel data of content (x, y)

[0067] Since the content coordinates (x, y) are not integers, nearby pixel data is calculated using image interpolation methods such as bilinear. If the coordinates are negative or exceed the content size, the pixel on the boundary can be selected.

[0068] In this way, the control unit 13 creates image data of P×Q pixels from the content image for the display unit 10. The control unit 13 sends the created image data to the display unit 10.

[0069] In this embodiment, operation using a conventional touch panel without using face recognition is also supported for scrolling content C. The initial scroll mode is set to the touch panel.

[0070] Steps S1 to S4 and S21 are the initialization part, after which a loop process for displaying content is started.

[0071] In this embodiment, when the operator presses and holds the touch panel of the display unit 10, the control unit 13 switches the scroll mode between a touch panel control mode and a face detection control mode. In the face detection control mode, the screen can be operated by face recognition, which ensures the operator's convenience in operating the screen even in situations where it is difficult for the operator to operate the screen by hand, such as when the operator is wearing gloves, carrying luggage, or wearing a watch on the dominant hand. Switching from the touch panel control mode to the face recognition mode can also be performed by simply pressing and holding the touch panel, eliminating the need for complicated operations and providing the advantage of easy switching even in situations where it is difficult to control the screen by hand. The control unit 13 detects the state of the scroll mode and switches between them in a scroll detection process (step S22). Note that the switching between scroll modes may also be enabled by voice input to the control unit 13.

[0072] Details of the above-mentioned scroll mode detection (step S22) will be described using the flow in Fig. 15. The control unit 13 detects whether the touch panel is in a pressed state (step S401). If this detection reveals that the touch panel is not in a pressed state (step S401: No), the control unit 13 sets the pressing start time of the touch panel to the current time (step S402).

[0073] Furthermore, the control unit 13 changes the return value depending on the current scroll mode state. In the face recognition control mode, the "continue face recognition" state is returned (step S21: continue face recognition), and in the touch panel control mode, the "touch panel" state is returned (step S21: touch panel).

[0074] When the touch panel is not pressed in this way, the control unit 13 returns the current mode state without changing the mode.

[0075] On the other hand, if the touch panel is pressed (step S401: Yes), the control unit 13 measures the pressing time by calculating the difference between the start time of pressing the touch panel and the current time (step S404).Until the measured pressing time reaches a predetermined time, the control unit 13 branches the process to step S403, and the current mode status is returned.

[0076] On the other hand, if the pressing time exceeds the long pressing time (step S405: Yes), the control unit 13 checks the current scroll mode and inverts the mode (step S406). If the current mode is the touch panel mode, the control unit 13 changes the scroll mode to face recognition (step S407) and sets the return value status to "face recognition new" (step S21: face recognition new). On the other hand, if the current mode is face recognition (step S406: face recognition), the control unit 13 sets the return value to the touch panel status (step S408). In this way, the above-mentioned function is achieved.

[0077] 9A and 9B, the current mode state is returned in the scroll mode detection process, and the control unit 13 branches depending on the value (step S22).

[0078] If the mode state is the touch panel (step S22: touch panel), the control unit 13 branches the process to step S23 and executes display control by the touch panel. This process is display control by operations such as swiping and pinching on the touch panel realized in smartphones and is well known, so a description thereof will be omitted.

[0079] On the other hand, if the mode changes from the touch panel to the face recognition processing mode, the state becomes "new face recognition" (step S22: new face recognition), and the control unit 13 branches the process to step S5. In this step S5, the control unit 13 initializes the face recognition process, setting a reference face data save request flag to use the current face position and face size as the reference (step S5).

[0080] Moreover, if the mode is "continue face recognition" (step S22: continue face recognition), the control unit 13 skips the process of step S5 and does not execute initialization.

[0081] From here on, a display control loop process is performed by face recognition control. In this case, first, the control unit 13 accesses the face recognition unit 14 to start a face recognition operation (step S6).

[0082] The face recognition unit 14 starts face recognition according to the flow of Fig. 8. The recognition operation status here is "recognizing" while the recognition operation is in progress, and becomes "completed" when the recognition is complete, so the control unit 13 continues to monitor this status in step S7 and waits for completion. After completion, the control unit 13 reads the detection result from the face recognition unit 14 (step S8).

[0083] The control unit 13 makes a judgment on the result in step S9, and if a face is detected as a result (step S9: Yes), it obtains the face center position and face area from the face recognition unit 14 (step S10), and also obtains face landmarks (step S11).

[0084] On the other hand, if the face is not recognized in step S9 (step S9: No), the control unit 13 skips the processes of steps S10 and S11.

[0085] When the control unit 13 acquires the face recognition information, it detects the display lock state and performs processing based on this information (step S12). During normal display processing, no special processing is performed here. The detailed operation will be described later.

[0086] Next, the control unit 13 determines whether the display lock mode is active (step S13). This mode is OFF in normal processing, and the process branches to No in this step. Here, the control unit 13 checks whether a face has been detected (step S14), and if a face has not been detected (step S14: No), the control unit 13 retains the current display data and skips to step S20.

[0087] On the other hand, if a face has been detected (step S14: Yes), the control unit 13 first checks the reference face save request flag (step S15), and if it is set (step S15: Yes), creates and saves reference face data (step S16).

[0088] The reference face data is data that serves as a reference when determining the relationship with the display range information from the position and size of the face recognized by the camera 11. Specifically, it is the reference position and size of the face (the distance between the eyes). FIG. 10A shows a camera image taken when creating the display reference data. In this example, the center position of the face in the data acquired from the face recognition unit 14 is (u0, v0), and the distance calculated from the coordinates of the right eye and left eye in the landmark is d0. These values, along with the current display range data, are saved in memory as reference face data. Reference face data: {Face position (u0, v0), face size (d0), display scale (scale0), display offset (x0, y0)}

[0089] Upon receiving the above-described completion of the creation and storage process of the reference face data, the control unit 13 resets the storage request signal for the reference face data (step S17).

[0090] Next, the control unit 13 calculates the display range from the current face information (step S18). This calculation function F2 will be explained below. Figure 10B shows the camera image currently being measured. The center position of the face detected in this camera image is (uf, vf), and the eye distance is de. Therefore, the control unit 13 creates display range data based on the currently saved reference face data.

[0091] When the center position of the detected face is (uf, vf) and the eye distance is de, the display scale is calculated as follows: [Formula F2-1] scale=scale0×de / d0 Camera ratio k: The movement ratio of the display pixel per pixel of the camera 11 (the optimum value is determined in advance through trial and error) is [Formula 3] xc=x0+(uf-u0)×k / scale [Formula 4] yc=y0+(vf-v0)×k / scale

[0092] The control unit 13 creates an image of the display range using the function F1 (same as in step S4) based on the scale, xc, and yc calculated in step S18, and transfers it to the display unit 10 (step S19).

[0093] Next, the control unit 13 checks whether the operator has operated the operation unit 12 to end the content display (step S20). If the result of this check is that the content display has not been ended (step S20: No), the control unit 13 returns the process to step S6, which is the first step of the display processing loop. On the other hand, if an end request has been issued (step S20: Yes), the control unit 13 ends this flow process.

[0094] In this way, the display area of ​​content C can be changed as shown in FIG. 7 based on the facial information at the time the content display starts and the facial information that changes according to the subsequent arm movements.

[0095] <Display lock detection process> Next, the details of "display lock detection and processing" (step S12) in the flow of FIG. 9B will be explained. Here, the operation that this processing is intended to achieve will first be explained using FIG. 11. For example, when an operator moves their arm to move the display area of ​​content C, they may reach a position where they cannot move their arm even though there is still part of the display area of ​​content C remaining. In such a case, the operator would like to move their arm further while keeping the display of content C locked. Therefore, the function corresponding to step S12 realizes a function that meets such needs.

[0096] 11B shows the movement of the operator's arm in a time series A to F. The operator moves his / her arm toward his / her body at timings A, B, and C, and the display position on display unit 10 at that time changes in accordance with the movement of the arm, as shown in FIG.

[0097] On the other hand, at timing C, the operator cannot move their arm any further, so they close their eyes (an action other than closing their eyes may also be used as the trigger). With their eyes closed, the operator moves watch 1 away from their body (timings D and E). In this state, control unit 13 locks the display and continues to display the same area.

[0098] Meanwhile, at timing E, when the operator has moved their arm to a position where it can be moved sufficiently, they open their eyes (timing F), and then move their arm again, bringing it closer to their body (timings F to H). At this time, control unit 13 controls the display position of the locked image so that it changes from that position. Figure 11C shows the display area of ​​the content at each time point.

[0099] A method for realizing the operation shown in Fig. 11 will be described with reference to Fig. 12 and Fig. 9B. Fig. 12 shows details of the processing in step S12 in Fig. 9B. Step S12 is a process inserted into the display processing loop, and is a display lock process for realizing the operation shown in Fig. 11, and is responsible for switching the display lock ON and OFF depending on the state of the user's face, and for controlling the processing that occurs when the display lock is switched ON and OFF.

[0100] When the process of step S12 is executed in the flow of Fig. 9B, the flow starts from the start of Fig. 12. First, the process flow when a normal slide operation is performed will be described. First, in response to detection of a face (step S301: Yes), the control unit 13 resets the face non-detection time (step S302).

[0101] Furthermore, the control unit 13 acquires information about facial expression from the face recognition unit 14 (step S303), and determines whether the eyes are closed with the facial expression (step S304). If it is determined that the user is performing a slide operation with their eyes open (step S304: No), the control unit 13 turns off the lock mode (step S305), and ends this flow.

[0102] The control unit 13 detects the trigger for transitioning to the display lock mode when the operator closes both eyes, but if the control unit 13 detects that the operator blinks during the display slide operation and the eyes are closed for only a moment, the control unit 13 will transition to the lock mode. To avoid this, the control unit 13 measures the face non-detection time so that the control unit 13 will not transition to the lock mode even if it detects that the eyes are closed for only a short time.

[0103] The control unit 13 detects that the operator has closed their eyes for a moment during a slide operation and branches to Yes in step S304, but then updates the time the eyes are closed (step S306), and then determines in step S307 whether the time the eyes are closed has exceeded a predetermined time, thereby avoiding unnecessary mode transitions. In other words, if the time is as short as a blink, the control unit 13 branches to No in step S307 and ends the processing of this flow. Therefore, an unnecessary lock state does not occur.

[0104] On the other hand, the flow when transitioning to lock mode will be explained. Assume that the operator continues to close both eyes for a certain period of time or more. At this time, the control unit 13 passes through the branches of steps S304 to S307 for a while, and eventually determines in step S307 that the predetermined period of time has passed, branches to Yes, and turns on the lock mode state (step S308). In addition, the control unit 13 also sets a save request signal for reference face data (step S309) and ends this flow. The save request signal for reference face data is used to set the face state as a new face reference when the lock mode is released.

[0105] If the operator is in a dangerous situation where it would be dangerous to operate the watch with both eyes closed, it is also possible to transition to lock mode by covering the camera 11 with the hand opposite the hand that wears the watch 1. If the camera 11 is covered by the hand in this way, the control unit 13 will no longer detect a face in the first step S301, and will branch to No. This also measures the time that no face is detected, and when a face has not been detected for a sufficient period of time, it will branch to Yes in step S311, proceeding to steps S308 and S309, which are the processing flow for the locked state, and transitioning to lock mode.

[0106] In this way, the operator can transition the watch 1 to lock mode by closing both eyes or covering the camera 11 with a hand or the like. When transitioning to lock mode, in the display control flow of FIG. 9B, the flow branches to Yes in step S13, which determines the display lock mode, skipping steps S18 and S19, which update the display range, and locking the display. Then, when the display lock is released, the flow branches to No in step S13.

[0107] If a face is detected in step S14, a request to save reference face data is set in step S15, so that the control unit 13 saves the face currently captured by the camera 11 as a reference image in step S16, and enables slide operation according to the face that changes as the operator moves his / her arm thereafter.

[0108] <Zoom in, zoom out> In this embodiment, the operator can zoom in and out of the content by moving their arm up and down. Figure 13 illustrates a zoom operation (scale change) with display lock. Figure 13B shows an example of the operation in chronological order.

[0109] First, at A, B, and C, the operator raises his / her arm to perform a zoom-out operation. To further enlarge content C, the operator closes his / her eyes at D and then lowers his / her arm. Since the lock mode is in effect from D to E, even if the operator lowers his / her arm, the control unit 13 does not control the zoom-in of the screen. When the operator opens his / her eyes at F, the control unit 13 releases the lock mode. Here, the operator also raises his / her arm to continue the zoom-out operation.

[0110] Fig. 13A shows the face photographed by the camera 11 at each time point during these operations, and Fig. 13C shows the display area at each time point.

[0111] A method for realizing the operation of Fig. 13 will be described with reference to Fig. 12 and Fig. 9B. As already mentioned, in step S18 (F2 processing) in the display processing loop of Fig. 9B, the SCALE rate changes depending on the size of the face (the distance between the right eye and the left eye).

[0112] When the distance between the right eye and the left eye is small, as in screen A in FIG. 13, SCALE is small according to (Equation F2-1). On the other hand, when the distance between the right eye and the left eye is large, as in screen C, SCALE becomes large. The control unit 13 changes the area cut out from the content according to this SCALE. As a result, the display control shown in FIG. 13C is performed. Similarly to the screen lock operation described above, the open / closed state of both eyes is detected according to the flow in FIG. 12, and the display lock is controlled to be turned on or off, thereby realizing the operation in FIG. 13.

[0113] <Correction according to the position of the watch> If the operator keeps his / her face fixed, the relative positions of the face and arm are detected based on the position of the face in the image being captured by the camera 11, and the forward / backward and left / right movements of the arm can be stably detected even when the operator is walking or running.

[0114] However, if the user rotates around the axis of the arm while walking and rotates watch 1 counterclockwise around the Y axis of Fig. 14A as shown in Fig. 14A, the position of the face captured by camera 11 changes. When watch 1 rotates in the direction of the arrow as shown in Fig. 14A, the image captured by camera 11 changes from Fig. 14B to Fig. 14C.

[0115] It is difficult to distinguish this image from an image of the face alone when the arm is moved in the opposite direction of the Y axis. This movement can also be used in place of arm movement, so this in itself is unlikely to be a problem, but it may be possible to use an angular velocity sensor in the orientation sensor 15 to detect the rotational movement of the arm and suppress the movement.

[0116] <Other scroll controls> For content that requires a large display size, a display lock mode is provided, which allows the user to move their arm back while stopping the scrolling of the content. However, repeatedly moving their arm back can be cumbersome for the user.

[0117] To solve this problem, a mode is provided in which, when the operator moves his / her arm and the position of the arm becomes farther from the body than a certain distance, the screen is controlled to automatically scroll from that point on.

[0118] The speed of this automatic scrolling is controlled so that it increases as the arm moves away from the body. When the operator approaches the desired position to view, the arm returns to its original position. At this time, the control unit 13 detects that the arm position has come closer than a certain distance from the body and stops the automatic scrolling. After that, the scrolling mode returns to the original position of the arm, allowing the operator to make fine adjustments by moving the arm.

[0119] This operation will be explained using Figures 16 and 17. Figure 16 shows the face of the operator captured by camera 11. Automatic scrolling control is determined by the distance between the center of the camera 11 and the center of the face (in the scrolling by arm movement described above, scrolling control is determined by the relative movement (position) of the face).

[0120] In Figure 16, (uc, vc) is the center coordinate of the shooting range of camera 11. A circle 161 is illustrated with a distance rb centered on this point. When the face is at position f1, the center coordinate of the face is (uf1, vf1), and the distance between this coordinate and the center coordinate is r1. On the other hand, when the face is at position f3, the center coordinate of the face is (uf3, vf3), and the distance between this coordinate and the center coordinate is r3.

[0121] Since r1 is within the radius rb, scrolling within this range is determined by the position of the face. On the other hand, since r3 is outside the radius rb, in this case, the control unit 13 determines the scrolling speed according to the amount of rb-r3 or rb / r3, and executes automatic scrolling in the direction according to the vector direction from the center coordinates.

[0122] An example of this scrolling will be explained using Figures 17A and 17B. Figure 17A shows the image of the entire content being displayed, and the area being displayed at each timing is shown as a dotted square. Figure 17B also shows the face being captured by camera 11 in response to the movement of the operator's arm.

[0123] First, the correspondence between the position of the face in the camera image when the face is at position f1 (FIG. 17B) and the display position of the map (FIG. 17A) is shown. At this time, if the operator wants to display the left side of the map content, he or she moves his or her arm to the right to look at the left side. In this case, the position of the face in camera 11 changes from f1 to f2, and the display position on the map also changes from f1 to f2 in accordance with the movement of the arm.

[0124] If the operator moves his / her arm further to the right, the position of the face moves from f2 to f3, and the map display also moves to f3. As a result, if the position of the face moves outside the circular area 161, the control unit 13 changes the scrolling operation to automatic scrolling. In this case, scrolling continues even if the face stops at position f3, and the display position of the map continues to move from f3 to f3' at a constant speed. The direction of movement is the vector direction from the center coordinate to the position of the face.

[0125] Note that the up and down directions of the camera 11 coordinate system (v direction) and the content coordinate system (y direction) are reversed, so the up and down directions of the face and the map movement are reversed.

[0126] If the operator sees that the map is not only scrolling left but also scrolling upward unintentionally, he or she will move his or her arm slightly downward to adjust the map so that it does not scroll upward. At this time, the control unit 13 changes the map display position from f3 to f4, maintains this state, continues scrolling left, and moves the map display from f4 to f4'.

[0127] When the display position reaches the part the operator wants to see, the operator moves his / her arm back to the right to stop the automatic scrolling. At this time, the position of the face in the image captured by camera 11 becomes f5, which falls within circular area 161. Control unit 13 detects this, stops the automatic scrolling, and returns to the scrolling mode controlled by arm movement. In this way, control unit 13 switches between automatic scrolling and manual scrolling in response to the operator's up / down / left / right movement of his / her arm, thereby providing the operator with good scrolling control.

[0128] A control method for realizing such scrolling operation will be described with reference to Figures 18A, 18B, and 19. Figures 18A and 18B are obtained by removing the display lock function of the scrolling control in Figures 9A and 9B. Also, the processing of step S18 in Figure 19 has been replaced with a new processing step S500. Other flow operations are the same, so a description thereof will be omitted.

[0129] The flow of step S500 described above is shown in more detail in Figure 19. In this step S500, the coordinates of the detected face are used to determine the display area of ​​the content. As shown in Figure 16, the distance r between the coordinates (uf, vf) of the detected face and the center coordinates (uc, vc) of the camera 11 is calculated using the following equation (5) (step S501). [Formula 5] r = ((u f -u c ) 2 + (v f -v c ) 2 ) 1 / 2

[0130] Here, the control unit 13 determines whether the distance r is inside or outside the circular area 161, which is the boundary for switching scroll control (step S502). If the result of this determination is that the distance r is inside the circular area 161 (step S502: Yes), the control unit 13 determines the center coordinates of the content according to the difference between the reference face data and the current face data, as follows, using the same method as in Figures 9A and 9B.

[0131] When the center position of the detected face is (uf, vf) and the eye distance is de, the display scale (scale) and the display center (xc, yc) of the content are calculated using the following equations (6) and (7). [Formula 6] scale = scale0x (de / d0) [Formula 7] x c = x0+ (u f - u0) xk / scale, y c = y0+ (v f - v0) xk / scale

[0132] On the other hand, if the distance r is outside the circular area 161 (step S501: No), the control unit 13 measures the time elapsed since the previous processing (step S505). The elapsed time at this time is defined as Tp. Based on this elapsed time and the distance r extends beyond the circular area 161, the control unit 13 calculates the display scale (scale), the display center of the content, and the origin offset (x0, y0) of the content using the following equations (8) to (11) (step S506). [Formula 8] scale = scale0x (de / d0) [Formula 9] deltaXY = ((r - r b ) xa s + b s ) * T p [Formula 10] x0= x0+ deltaXY / scale, y0= y0+ deltaXY / scale [Formula 11] x c = x0+ (u f - u0) xk / scale, y c = y0+ (v f - v0) xk / scale

[0133] Here, as and bs are predetermined proportional multipliers that can be determined through trial and error to achieve the optimal scrolling. Since scrolling is a fixed amount, the elapsed processing is multiplied to take into account the case when the processing flow cycle is not constant.

[0134] The above operation allows switching between automatic scrolling and manual scrolling depending on the up, down, left and right movement of the operator's arm, thereby achieving good scrolling control.

[0135] According to the present invention, by determining the relative position (three axes) between the display unit and the face and the facial posture (pitch, yaw, roll) in an information display device worn on the wrist, it is possible to perform highly accurate display control even when walking or running.

[0136] For example, if an operator walks with their arms fixed so that the display unit is visible, the relative position of the display unit and the operator remains fixed, and the content displayed on the display unit remains stationary. Also, if the operator moves their arms while walking, the content display is controlled according to the amount of movement relative to their face.

[0137] As described above, in this embodiment, the wearable information processing device having a content display function includes a display unit 10 that displays the content, and a control unit 13 that executes a process of acquiring a photographed image of the operator from a camera 11 attached to the display unit 10, a process of recognizing a facial image that is an image of the operator's face from the photographed image, and a process of controlling the display range of the content based on the state of the main parts of the face.

[0138] In this way, it becomes possible to perform good display control in accordance with the operator's intentions regarding the display of content on a wearable information processing device.

[0139] The control unit 13 may execute a process of determining a display range of the content to be displayed on the display unit 10 based on the captured image.

[0140] In this way, the operator can determine the display range without touching the information display device 1.

[0141] The control unit 13 may execute a process to detect the position and posture of the face from the facial image and the relative distance between the face and the camera 11, and when determining the display range of the content to be displayed on the display unit 10 based on the captured image, the display range may be determined based on the position and posture of the face and the relative distance.

[0142] In this way, the operator can determine the display range by moving the information display device 1.

[0143] The control unit 13 may further execute a process of controlling the display range based on at least one of whether the facial image is recognized or not and the moving distance of the main part on the display unit.

[0144] In this way, it becomes possible to control the display of content on a wearable information processing device in a more convenient manner in line with the operator's intentions.

[0145] When detecting the position and posture of a face, the control unit 13 may identify the position of the main part of the face in the facial image, and identify the position, orientation, and posture of the face based on the position information of the main part.

[0146] In this way, the position, direction, and posture of each operator's face can be determined with high accuracy based on the main parts of the face.

[0147] The control unit 13 may identify the state of the main part of the operator's face, and, depending on the state and whether or not the face is detected, control at least one of enlarging or reducing the content on the display unit and moving the display range of the content displayed on the display unit.

[0148] In this way, it is possible to perform display control such as enlarging or reducing the display range of the content or moving it based on whether or not the operator's face has been detected and the state of the main part of the face.

[0149] When detecting the position and posture of the face, the control unit 13 may calculate the center position of the face based on the position information of the main parts, and detect the posture of the face based on the position information of the main parts and the contour information of the face.

[0150] In this way, the control unit 13 can accurately determine the posture of the operator's face.

[0151] When the face recognition unit 14 recognizes a face image in the captured image and determines that the state of the main part of the face has been predetermined for a certain period of time or more, the change in the display format of the content on the display unit 10 may be stopped.

[0152] In this way, it is possible to prevent the display mode of the content from being changed against the operator's will.

[0153] The control unit 13 may identify the timing when the distance moved by the main part of the operator's face on the display screen exceeds a predetermined standard, and from that timing onwards, automatically scroll the display range of the content on the display unit according to the direction of the face.

[0154] In this way, the operator can scroll the display range without having to perform a specific operation continuously.

[0155] The wearable information processing device may be a wristwatch-type terminal device.

[0156] In this way, an operator wearing a wristwatch-type terminal device can freely control the display of content displayed on the display unit 10 based on the movement of the arm wearing the device, the position and posture of the face, the state of the main parts, etc.

[0157] The above-described series of processes can be executed by hardware or software. In other words, the functional configuration of FIG. 2 is merely illustrative and is not particularly limited. That is, it is sufficient for the information display device 1 to have the functionality to execute the above-described series of processes as a whole, and the functional blocks used to realize these functions are not particularly limited to the example of FIG. 2. Furthermore, a single functional block may be configured as a single piece of hardware, a single piece of software, or a combination thereof. The functional configuration of this embodiment is realized by a processor that executes arithmetic processing. Processors that can be used in this embodiment include those configured as various processing units, such as single processors, multiprocessors, and multicore processors, as well as those that combine these various processing units with processing circuits, such as ASICs (Application Specific Integrated Circuits) and FPGAs (Field-Programmable Gate Arrays).

[0158] When a series of processes is executed by software, the programs that make up the software are installed into a computer or the like from a network or a recording medium. The computer may be a computer built into dedicated hardware. Alternatively, the computer may be a computer that can execute various functions by installing various programs, such as a general-purpose personal computer.

[0159] The recording medium containing such a program may be a removable medium such as a USB memory that is distributed separately from the device main body in order to provide the program to the user, or may be a recording medium that is provided to the user in a state where it is pre-installed in the device main body. Removable media may be, for example, a magnetic disk (including a floppy disk), an optical disk, or a magneto-optical disk. Optical disks may be, for example, CD-ROMs (Compact Disk-Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Discs, etc. Magneto-optical disks may be, for example, MDs (Mini-Disks). Furthermore, the recording medium that is provided to the user in a state where it is pre-installed in the device main body may be, for example, a ROM on which the program is recorded.

[0160] In this specification, the steps of describing a program to be recorded on a recording medium include not only processes that are performed chronologically in accordance with the order, but also processes that are not necessarily performed chronologically but are performed in parallel or individually.

[0161] Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. The present invention can take various other embodiments, and it is also possible to combine the configurations of the above embodiments and modifications. Furthermore, various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications thereof are included within the scope and spirit of the invention described in this specification, etc., and are included in the invention described in the claims and their equivalents.

[0162] The inventions described in the claims of the present application as originally filed are set forth below. [Appendix 1] A wearable information processing device having a content display function, a display unit that displays the content; A process of acquiring a photographed image of the operator of the information processing device from a camera attached to the display unit; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; and a control unit that executes a process of controlling a display range of the content based on a state of a main part of the face. An information processing device comprising: [Appendix 2] 2. The information processing device according to claim 1, wherein the control unit executes a process of determining a display range of the content to be displayed on the display unit based on the captured image. [Appendix 3] the control unit executes a process of detecting a position and posture of a face from the face image and a relative distance between the face and the camera; An information processing device as described in Appendix 2, wherein, when determining a display range of the content to be displayed on the display unit based on the captured image, the display range is determined based on the position and posture of the face and the relative distance. [Appendix 4] The information processing device according to claim 1, wherein the control unit further executes a process of controlling the display range based on at least one of whether or not the facial image is recognized and the distance the main part has moved on the display unit. [Appendix 5] The control unit In the process of detecting the position and orientation of the face, the position of the main part of the face in the face image is identified, and the position, orientation, and orientation of the face are identified based on the position information of the main part. 4. The information processing device according to claim 3. [Appendix 6] The control unit a state of the main part of the face of the operator is identified, and at least one of enlarging or reducing the content on the display unit and moving a display range of the content to be displayed on the display unit is controlled according to the state and whether or not the face has been detected; 2. The information processing device according to claim 1. [Appendix 7] The control unit When detecting the position and posture of the face, a center position of the face is calculated based on the position information of the main part, and the posture of the face is detected based on the position information of the main part and the contour information of the face. 4. The information processing device according to claim 3. [Appendix 8] The control unit When a face image is recognized in the captured image and the state of the main part of the face remains in a predetermined state for a certain period of time or more, the change in the display format of the content on the display unit is stopped. 2. The information processing device according to claim 1. [Appendix 9] The control unit a timing when the moving distance of the main part of the face of the operator on the display screen reaches or exceeds a predetermined standard, and from that timing onwards, the display range of the content on the display unit is automatically scrolled in accordance with the direction of the face; 2. The information processing device according to claim 1. [Appendix 10] Execute a process of detecting the position and posture of the face from the face image and the relative distance between the face and the camera; An information processing device as described in Appendix 8, wherein, when determining the display range of the content to be displayed on the display unit based on the captured image, the display range of the content to be displayed on the display unit is determined based on the position and posture of the face and the relative distance. [Appendix 11] Execute a process of detecting the position and posture of the face from the face image and the relative distance between the face and the camera; An information processing device as described in Appendix 9, wherein, when determining the display range of the content to be displayed on the display unit based on the captured image, the display range of the content to be displayed on the display unit is determined based on the position and posture of the face and the relative distance. [Appendix 12] 12. The information processing device according to any one of appendices 1 to 11, wherein the wearable information processing device is a wristwatch-type terminal device. [Appendix 13] A wearable information processing device having a content display function, A process of acquiring a photographed image of the operator of the information processing device from a camera attached to a display unit that displays the content; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; and a process of controlling a display range of the content based on a state of a main part of the face. An information processing method that performs the above. [Appendix 14] A wearable information processing device having a content display function, A process of acquiring a photographed image of the operator of the information processing device from a camera attached to a display unit that displays the content; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; and a process of controlling a display range of the content based on a state of a main part of the face. A program that executes the following. [Explanation of symbols]

[0163] 1 Information display device 10 Display 11 Camera 12 Control section 13 Control Unit 14 Face Recognition Unit 140 Face detection unit 141 Face landmark detection unit 142 Face position detection unit 143 Face Pose Detection Unit 144 Face state detection unit 145 Face detection control unit 15 Attitude Sensor 151 Magnetic Sensor 152 Acceleration Sensor 153 Angular rate sensor

Claims

1. A wearable information processing device having a content display function, a display unit that displays the content; A process of acquiring a photographed image of the operator of the information processing device from a camera attached to the display unit; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; and a control unit that executes a process of controlling a display range of the content based on a state of a main part of the face. Equipped with The control unit An information processing device that stops changing the display format of the content on the display unit when a facial image is recognized in the captured image and the state of the main part of the face remains predetermined for a certain period of time or more.

2. A wearable information processing device having a content display function, a display unit that displays the content; A process of acquiring a photographed image of the operator of the information processing device from a camera attached to the display unit; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; and a control unit that executes a process of controlling a display range of the content based on a state of a main part of the face. Equipped with The control unit An information processing device that identifies the timing when the distance moved by the main part of the operator's face on the display screen exceeds a predetermined standard, and from that timing onwards, automatically scrolls the display range of the content on the display unit in accordance with the direction of the face.

3. The information processing device according to claim 1 , wherein the control unit executes a process of determining a display range of the content to be displayed on the display unit based on the captured image.

4. the control unit executes a process of detecting a position and posture of a face from the face image and a relative distance between the face and the camera; The information processing device according to claim 3 , wherein, when a display range of the content to be displayed on the display unit is determined based on the captured image, the display range is determined based on the position and posture of the face and the relative distance.

5. 3 . The information processing device according to claim 1 , wherein the control unit further executes a process of controlling the display range based on at least one of whether the facial image is recognized or not and a moving distance of the main part on the display unit.

6. The control unit In the process of detecting the position and orientation of the face, the position of the main part of the face in the face image is identified, and the position, orientation, and orientation of the face are identified based on the position information of the main part. The information processing device according to claim 4 .

7. The control unit a state of the main part of the face of the operator is identified, and at least one of enlarging or reducing the content on the display unit and moving a display range of the content to be displayed on the display unit is controlled according to the state and whether or not the face has been detected; 3. The information processing device according to claim 1.

8. The control unit When detecting the position and posture of the face, a center position of the face is calculated based on the position information of the main part, and the posture of the face is detected based on the position information of the main part and the contour information of the face. The information processing device according to claim 4 .

9. The information processing device according to claim 1 or 2, wherein the wearable information processing device is a wristwatch-type terminal device.

10. A wearable information processing device having a content display function, A process of acquiring a photographed image of the operator of the information processing device from a camera attached to a display unit that displays the content; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; a process of controlling a display range of the content based on a state of a main part of the face; and a process of stopping the change of the display format of the content on the display unit when a face image is recognized in the captured image and the state of the main part of the face remains in a predetermined state for a certain period of time or more. An information processing method that performs the above.

11. A wearable information processing device having a content display function, A process of acquiring a photographed image of the operator of the information processing device from a camera attached to a display unit that displays the content; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; a process of controlling a display range of the content based on a state of a main part of the face; and a process of identifying a timing at which a moving distance of the main part of the face of the operator on the display screen reaches or exceeds a predetermined standard, and automatically scrolling a display range of the content on the display unit according to a direction of the face from that timing onward. An information processing method that performs the above.

12. A wearable information processing device having a content display function, A process of acquiring a photographed image of the operator of the information processing device from a camera attached to a display unit that displays the content; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; a process of controlling a display range of the content based on a state of a main part of the face; and a process of stopping the change of the display format of the content on the display unit when a face image is recognized in the captured image and the state of the main part of the face remains in a predetermined state for a certain period of time or more. A program that executes the following.

13. A wearable information processing device having a content display function, A process of acquiring a photographed image of the operator of the information processing device from a camera attached to a display unit that displays the content; a process of recognizing a face image, which is an image of the face of the operator, from the captured image; a process of controlling a display range of the content based on a state of a main part of the face; and a process of identifying a timing at which a moving distance of the main part of the face of the operator on the display screen reaches or exceeds a predetermined standard, and automatically scrolling a display range of the content on the display unit according to a direction of the face from that timing onward. A program that executes the following.

Images