Image processing apparatus, wearable device, image processing method, control method, and program

Abstract

To provide support that can improve the user's work efficiency. [Solution] The image processing device according to the present disclosure is an image processing device that performs display control for a display unit 120 of a wearable device such as a head-mounted display 100, and performs display control to obtain specific text from among the text associated with a moving image and to display the specific text and the moving image on different display surfaces.

Description

Technical Field

[0006] , , ,

[0001] The present disclosure relates to a display control technique for wearable devices.

Background Art

[0002] There is a technique for visually assisting the work of a wearer of a wearable device (hereinafter referred to as a "user") by displaying a moving image related to the work on a display unit of the wearable device such as a head-mounted display. Patent Document 1 discloses a technique for detecting an object included as an image in a captured image by analyzing the captured image obtained by capturing with an imaging device provided in a head-mounted display, and displaying a moving image related to the work related to the detected object on a display unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] Conventionally, it has sufficiently satisfied the improvement in work efficiency desired at that time. However, in recent years, a mechanism for further improving work efficiency has been demanded.

Means for Solving the Problems

[0005] An image processing apparatus according to the present disclosure is an image processing apparatus that performs display control for a display unit included in a wearable device, and includes an acquisition unit that acquires specific text from text associated with a moving image, and a display control unit that displays the specific text and the moving image on different display surfaces.

Effects of the Invention

[0006] The technology disclosed herein can provide support that can improve the user's work efficiency. [Brief explanation of the drawing]

[0007] [Figure 1] This is a block diagram showing an example of the hardware configuration of a head-mounted display according to Embodiment 1. [Figure 2] This is a block diagram showing an example of the logical configuration of the image processing unit according to Embodiment 1. [Figure 3] This figure shows an example of how a typical video streaming site looks. [Figure 4] This figure illustrates an example of a method for selecting a mode in the text acquisition unit according to Embodiment 1. [Figure 5] This figure illustrates an example of a method for extracting specific text in the text acquisition unit according to Embodiment 1. [Figure 6] This figure illustrates an example of specific text acquired by the text acquisition unit according to Embodiment 1. [Figure 7] This figure shows an example of a display screen shown on the display unit of the head-mounted display according to Embodiment 1. [Figure 8] This flowchart shows an example of the processing flow of the image processing unit according to Embodiment 1. [Figure 9] This flowchart shows an example of the processing flow of the image processing unit according to Embodiment 2. [Figure 10] This figure shows an example of the arrangement of candidate configurations according to Embodiment 3. [Figure 11] This flowchart shows an example of the processing flow of the image processing unit according to Embodiment 3. [Modes for carrying out the invention]

[0008] Embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the technology of this disclosure. While multiple features are described in the embodiments, not all of these features are necessarily essential to the technology of this disclosure, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar components are denoted by the same reference numerals, and redundant explanations are omitted. Note that the following embodiments will be described using video-transmissive or optical-transmissive head-mounted displays as examples, but the scope of application of the technology of this disclosure is not limited to head-mounted displays. The technology of this disclosure is applicable, for example, to wearable display devices (wearable devices) having a display unit, such as AR (augmented reality) glasses (also called "smart glasses").

[0009] [Embodiment 1] <Hardware configuration of head-mounted displays> A head-mounted display 100 according to Embodiment 1 will be described with reference to Figures 1 to 8. Figure 1 is a block diagram showing an example of the hardware configuration of the head-mounted display 100 according to Embodiment 1. The head-mounted display 100 has a hardware configuration consisting of a processor 111, memory 112, non-volatile memory 113, storage medium 114, communication unit 115, display unit 120, imaging unit 130, operation unit 140, sensor 150, and battery 160. Hereinafter, the image processing unit 110 will be described as being composed of the processor 111, memory 112, non-volatile memory 113, storage medium 114, and communication unit 115. Each part of the head-mounted display 100's hardware configuration is connected to each other via a bus 101 so as to be able to communicate with one another.

[0010] The processor 111 consists of a processing unit such as a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit) and controls the entire head-mounted display 100. The memory 112 consists of RAM (Random Access Memory) and operates as the work area of ​​the processor 111. The non-volatile memory 113 consists of ROM (Read Only Memory) and stores a computer program for controlling the head-mounted display 100, and various data used to execute the computer program. Hereinafter, "computer program" will be simply referred to as "program". The storage medium 114 consists of flash memory or a hard disk drive and stores the above-mentioned program, various data used to execute the program, and other image data and audio data. The communication unit 115 is a communication interface used for sending and receiving data with external devices. When the communication unit 115 communicates with an external device via wireless communication, the communication unit 115 has a communication antenna.

[0011] The display unit 120 is composed of a display device such as a liquid crystal display and displays a display image based on a signal indicating the display image output from the image processing unit 110. Specifically, the display unit 120 is positioned so as to be included in the user's field of view when the user is wearing the head-mounted display 100. The imaging unit 130 is composed of an image sensor including a CCD (Charge Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor), and an optical system such as a lens. The imaging unit 130 uses an optical system to image light rays from the outside onto the light-receiving surface of the image sensor, converts the image obtained by imaging into an electrical signal by photoelectric conversion, and outputs the electrical signal. Specifically, the imaging unit 130 is positioned and oriented so as to be able to image in a direction corresponding to the direction of the user's field of view when the user is wearing the head-mounted display 100.

[0012] The operation unit 140 is composed of a push button switch, a touch sensor, etc., receives an operation from the user (hereinafter referred to as "user operation"), and outputs an electrical signal corresponding to the user operation. The sensor 150 detects the position and orientation of the head-mounted display 100. For example, the sensor 150 is composed of a gyro sensor, an acceleration sensor, or a GNSS (Global Navigation Satellite System) receiver such as a GPS (Global Positioning System) receiver. The battery 160 is composed of a storage battery such as a lithium-ion battery, and supplies power to each part that the head-mounted display 100 has as a hardware configuration via the bus 101.

[0013] The processor 111 controls the head-mounted display 100 by expanding and executing a program read from the non-volatile memory 113 or the storage medium 114, or received by communication through the communication unit 115 in the memory 112. Further, the processor 111 also operates as a control unit that controls each of the display unit 120, the imaging unit 130, the operation unit 140, the sensor 150, and the battery 160.

[0014] <Logical Configuration of Image Processing Unit> FIG. 2 is a block diagram showing an example of the logical configuration of the image processing unit 110 in the head-mounted display 100 according to Embodiment 1. The image processing unit 110 has, as a logical configuration, a data acquisition unit 201, a text acquisition unit 202, an arrangement unit 203, and a display control unit 204. Each part that the image processing unit 110 has as a logical configuration is realized by the processor 111 expanding and executing a program read from the non-volatile memory 113 or the storage medium 114, or received by communication through the communication unit 115 in the memory 112.

[0015] The data acquisition unit 201 acquires video data (hereinafter referred to as "video data"). Specifically, the data acquisition unit 201 acquires video data selected by user operation by reading it from the storage medium 114, or by receiving it through communication by the communication unit 115. For example, the user inputs keywords using the operation unit 140. Note that the method of inputting keywords is not limited to using the operation unit 140; for example, the user's voice, collected through a sound collector such as a microphone (not shown in Figure 1), may be analyzed and converted into text, i.e., input by voice input. Alternatively, input may be made by operating a terminal such as a smartphone (not shown in Figure 1). The data acquisition unit 201 searches for video images corresponding to the keyword and acquires thumbnail data representing each of the multiple video images that satisfy the search conditions. The thumbnails acquired by the data acquisition unit 201 are displayed on the display unit 120 by display control by the display control unit 204, which will be described later. The user selects a desired thumbnail from among the thumbnails displayed on the display unit 120 using, for example, the operation unit 140. The data acquisition unit 201 acquires video data corresponding to the thumbnail selected by the user.

[0016] The text acquisition unit 202 acquires data of specific text (hereinafter referred to as "specific text") from the text (hereinafter referred to as "related text") associated with the moving image data acquired by the data acquisition unit 201. Specifically, for example, the text acquisition unit 202 can acquire the specific text by extracting the specific text from the related text based on the mode (hereinafter referred to as "selected mode") selected by the user operation from among a plurality of modes. More specifically, for example, the text acquisition unit 202 acquires the specific text by extracting text indicating at least any one of the materials, tools, and working steps corresponding to the selected mode from the related text based on the selected mode. For example, when the selected mode is a mode related to "cooking", the text acquisition unit 202 acquires the specific text by extracting text indicating at least any one of the materials used for cooking and the working steps of cooking from the related text.

[0017] Referring to FIGS. 3 to 6, a general display method of moving images and related texts, a method of selecting a mode, a method of extracting text based on the selected mode, and specific examples of specific texts will be described. FIG. 3 is a diagram showing an example of the display of a general moving image viewing site. The display screen 300 of the moving image viewing site includes, for example, the following regions as shown in FIG. 3 as an example. Specifically, the display screen 300 includes a region 301 where the moving image being viewed is displayed, and a region 302 where information of the poster of the moving image being viewed and the title of the moving image are displayed. Further, the display screen 300 includes a region 303 where the related text of the moving image being viewed is displayed, and a region 305 where thumbnails of moving images related to the moving image being viewed are displayed. When the number of characters of the related text is larger than the size of the region 303, a part of the related text, such as a predetermined number of lines from the beginning of the related text, is displayed in the region 303. In this case, when the button 304 is pressed, the entire related text can be displayed by scrolling or the like.

[0018] Figure 4 is a diagram illustrating an example of a mode selection method in the text acquisition unit 202 according to Embodiment 1. The user performs a gesture with their hand or fingers within the field of view of the imaging unit 130. The imaging unit 130 captures the gesture by the user, and the text acquisition unit 202 acquires the data of the captured image obtained by the imaging unit 130. Figure 4 shows an example of a captured image 400. The text acquisition unit 202 analyzes the acquired captured image to identify the gesture included as an image in the captured image, and selects a mode corresponding to the identified gesture.

[0019] Examples of gestures include the following. For example, the user makes a gesture of touching the fingertips of their index finger and thumb with the palm of their right hand facing the imaging unit 130. When such a gesture is captured by the imaging unit 130, the text acquisition unit 202 instructs the display control unit 204 to display a list of pre-prepared modes. Figure 4 shows an example of the mode list display 401. For example, the mode to be focused on is switched by moving the fingertips of the index finger and thumb of the right hand up and down while they are in contact. The focused mode is highlighted, for example, in the mode list display so that it can be distinguished from other modes. For example, when the user separates the fingertips of the index finger and thumb of the right hand, the focused mode is determined as the selected mode. Also, for example, when the user twists their wrist so that the palm of the right hand is not captured while the fingertips of the index finger and thumb of the right hand are in contact, the list display is canceled.

[0020] Figure 5 is a diagram illustrating an example of a method for extracting specific text in the text acquisition unit 202 according to Embodiment 1. In Table 500 shown in Figure 5, as an example, modes 501 and extraction conditions 502 are associated with each other. The text acquisition unit 202 acquires specific text by extracting text from related text based on the extraction conditions associated with the selected mode in Table 500. For example, if the selected mode is cooking mode, the text acquisition unit 202 extracts text from the related text in the range from lines containing the word "ingredients" to lines that do not contain any of those words, and acquires that text as specific text. Note that it is not limited to the word "ingredients"; for example, the text acquisition unit 202 may extract text from lines containing words such as "recipe" to lines that do not contain any of those words. The characters used to identify the extraction range are not limited to specific ones. Also, the characters used to identify the extraction range may be common across multiple modes.

[0021] Figure 6 is a diagram illustrating an example of specific text acquired by the text acquisition unit 202 according to Embodiment 1. Figure 6 shows an example of related text 600 and specific text 601 extracted from related text 600. Note that the extraction conditions are not limited to those shown in Table 500. For example, when extracting text related to "ingredients" from related text, the text acquisition unit 202 may extract lines containing numerical values ​​or characters or strings indicating numerical units as specific text. Characters or strings indicating numerical units include "grams," "g," "kg," "liters," "l," "ml," "pieces," or "books." In this case, for example, the text acquisition unit 202 may exclude lines in which numerical values ​​are expressed in units indicating time, such as "hours," "minutes," or "seconds," from the extraction target. In this case, for example, if the number of characters in a single line exceeds a predetermined number, the text acquisition unit 202 may exclude that line from the extraction target, assuming that the line contains information other than information related to ingredients, such as a description of the cooking process.

[0022] The method for extracting specific text by the text acquisition unit 202 is not limited to the method based on the extraction conditions 502 associated with mode 501, as shown as an example in Table 500. For example, the text acquisition unit 202 may input relevant text into a pre-prepared generating AI (Artificial Intelligence), and acquire the text output by the generating AI as the specific text. Alternatively, the text acquisition unit 202 may convert the captions contained in the frames constituting the video into text using optical character recognition, and extract the text corresponding to the specific text from the converted string. For example, in this case, the text acquisition unit 202 acquires the frame corresponding to the timestamp specified by the video uploader or user, and converts the captions contained in the acquired frame into text by performing optical character recognition on the acquired frame.

[0023] The arrangement unit 203 arranges a first display surface on which moving images acquired by the data acquisition unit 201 are displayed, and a second display surface on which specific text acquired by the text acquisition unit 202 is displayed. The method by which the arrangement unit 203 arranges the first and second display surfaces will be described later.

[0024] The display control unit 204 displays a display image including the specific text and the moving image on the display unit 120 by controlling the display of the moving image acquired by the data acquisition unit 201 and the specific text acquired by the text acquisition unit 202 on different display surfaces. Figure 7 is a diagram showing an example of a display screen 700 displayed on the display unit 120 of the head-mounted display 100 according to Embodiment 1. The display screen 700 includes a moving image 711 that is controlled to be displayed on the first display surface 710, an image 721 of the specific text that is controlled to be displayed on the second display surface 720, and an image 701 obtained by video pass-through or optical see-through.

[0025] For example, the first display surface 710 and the second display surface 720 are fixed and arranged in predetermined positions on the display screen 700. The positions on the display screen 700 to which the first display surface 710 and the second display surface 720 are fixed may be predetermined or specified by user operation. As shown in Figure 7 as an example, the first display surface 710 is positioned, for example, in the center of the upper part of the user's field of view, and the second display surface 720 is positioned, for example, on the left or right of the upper part of the user's field of view. With such an arrangement, the user can prepare the ingredients or tools required for cooking or work while checking the image 721 of specific text, while proceeding with the work while checking the moving image 711 and the image 701 via video passthrough or optical seethrough.

[0026] Furthermore, the image processing unit 110 may accept user input to change the position of at least one of the first display surface 710 and the second display surface 720 while a moving image and specific text are displayed. For example, the user can select the first or second display surface to change the position of using the operation unit 140 or by performing a gesture input, and move the selected display surface to the desired position. When the image processing unit 110 accepts such user input, the placement unit 203 moves the selected display surface on the display screen 700 based on the user input. The display control unit 204 controls the display of the moving image or specific text on the display surface. With this configuration, the user can secure a field of view that can improve work efficiency according to the task.

[0027] The display control unit 204 may also control the display so that at least one of the moving image 711 displayed on the first display surface 710 and the specific text displayed on the second display surface 720 is displayed semi-transparently. By displaying the moving image 711 and the specific text semi-transparently, the background of the first and second display surfaces is visible, thus widening the user's field of view. Furthermore, as the work process progresses, the display control unit 204 may also control the display so that strings corresponding to materials that have already been used are deleted from the specific text, or so that it is indicated as used by superimposing a strikethrough or graying out. The progress of the work process may be determined, for example, by sequentially analyzing the frames of the moving image that have been played back so far, or by sequentially analyzing the captured images obtained by imaging by the imaging unit 130.

[0028] <Operation of the image processing unit> The operation of the image processing unit 110 will be described with reference to Figure 8. Figure 8 is a flowchart showing an example of the processing flow of the image processing unit 110 according to Embodiment 1. The processing shown in the flowchart in Figure 8 is realized by the processor 111 loading a program stored in the non-volatile memory 113 or storage medium 114, etc., into the memory 112 and executing it. The flowchart shown in Figure 8 is started, for example, when the user turns on the power of the head-mounted display. In the following description, the symbol "S" preceding the reference numerals means a processing step.

[0029] First, in S801, the data acquisition unit 201 acquires video data and related text data corresponding to the video. The method for selecting the video to be acquired has been described above, so it will not be explained. Next, in S802, the text acquisition unit 202 acquires data for a specific text from the related text acquired in S801. The method for acquiring the specific text, specifically the method for extracting text corresponding to the specific text from the related text, including the method for selecting the mode, has been described above, so it will not be explained. Next, in S803, the arrangement unit 203 arranges a first display surface on which the video acquired in S801 is played and displayed, and a second display surface on which the image of the specific text acquired in S802 is displayed. The method for determining the arrangement positions of the first and second display surfaces has been described above, so it will not be explained.

[0030] Next, in S804, the display control unit 204 controls the display of the video acquired in S801 on the first display surface and the specific text acquired in S802 on the second display surface, thereby controlling the display of the display image including the specific text and the video. The method of display control in the display control unit 204 has been described above and will not be explained here. Next, in S805, the image processing unit 110 determines whether or not termination has been instructed by user operation. If it is determined in S805 that termination has been instructed, the image processing unit 110 terminates the process shown in the flowchart in Figure 8.

[0031] If it is determined in S805 that termination has not been instructed, in S806 the display control unit 204 determines whether or not it has controlled the display of the video acquired in S801 until the last frame. If it is determined in S806 that it has not controlled the display until the last frame, the image processing unit 110 returns to the process in S804 and repeatedly executes the processes from S804 to S806 until the following determination is made. Specifically, the image processing unit 110 repeatedly executes the processes from S804 to S806 until it is determined in S805 that termination has been instructed, or until it is determined in S806 that it has controlled the display until the last frame. If it is determined in S806 that it has controlled the display until the last frame, the image processing unit 110 terminates the process shown in the flowchart in Figure 8.

[0032] The head-mounted display 100 configured as described above can provide support that can improve the user's work efficiency.

[0033] [Embodiment 2] Embodiment 1 described a configuration in which a first display surface on which a moving image is played and displayed, and a second display surface on which an image of specific text is displayed, are fixed and arranged at different positions on the display screen. Embodiment 2 describes a configuration in which the first display surface is arranged at a predetermined position in a virtual space corresponding to real space.

[0034] The hardware configuration of the head-mounted display according to Embodiment 2 is the same as the hardware configuration of the head-mounted display 100 according to Embodiment 1, which is shown as an example in Figure 1. Therefore, the head-mounted display according to Embodiment 2 will be referred to as "head-mounted display 100" in the following description. Furthermore, the parts that the image processing unit 110 according to Embodiment 2 has as a logical configuration are the same as the parts that the image processing unit 110 according to Embodiment 1 has as a logical configuration, although some functions differ. Therefore, the names of the parts that the image processing unit 110 according to Embodiment 2 has as a logical configuration will be written in accordance with the names of the parts that the image processing unit 110 according to Embodiment 1 has as a logical configuration. That is, the image processing unit 110 according to Embodiment 2 (hereinafter simply referred to as "image processing unit 110") has, as a logical configuration, a data acquisition unit 201, a text acquisition unit 202, a layout unit 203, and a display control unit 204 according to Embodiment 2.

[0035] The data acquisition unit 201 and text acquisition unit 202 according to Embodiment 2 (hereinafter simply referred to as "data acquisition unit 201" and "text acquisition unit 202") are the same as the data acquisition unit 201 and text acquisition unit 202 according to Embodiment 1. Therefore, a description of the data acquisition unit 201 and text acquisition unit 202 will be omitted.

[0036] The arrangement unit 203 according to Embodiment 2 (hereinafter simply referred to as "arrangement unit 203") arranges the first display surface at a predetermined position in a virtual space corresponding to real space. In this case, the display control unit 204 according to Embodiment 2 (hereinafter simply referred to as "display control unit 204") controls the display of a moving image on the first display surface, while the image processing unit 110 performs the following processing. Specifically, for example, the display control unit 204 generates a virtual viewpoint image corresponding to how the moving image displayed on the first display surface looks from a position in the virtual space corresponding to the user's viewpoint in real space, based on the position and orientation of the head-mounted display 100. The position and orientation of the head-mounted display 100 can be determined based on signals from the sensor 150.

[0037] Furthermore, the placement unit 203 places a second display surface on which specific text is displayed at a predetermined position in the virtual viewpoint image generated by the display control unit 204. The display control unit 204 controls the display of the specific text on the second display surface placed at the predetermined position in the virtual viewpoint image by the placement unit 203, thereby controlling the display image including the image of the specific text and the image of the moving image displayed in the virtual space to be displayed on the display unit 120.

[0038] The position in the virtual space where the first display surface is positioned by the positioning unit 203 may be predetermined or specified by user operation. For example, if the selected mode is a mode related to "cooking," the positioning unit 203 positions the first display surface in the virtual space at a position corresponding to the back of the cooking counter in the depth direction. By positioning it in such a location, the user can work efficiently and safely without their hands being obscured by the moving image. Alternatively, the positioning unit 203 may determine the position in the virtual space where the first display surface is positioned based on the captured image obtained by the imaging unit 130. Specifically, for example, the positioning unit 203 analyzes the captured image to identify the position of a predetermined object, such as a cutting board, included as an image in the captured image, and positions the first display surface in the virtual space at a position corresponding to a predetermined position, such as the back of the object.

[0039] Furthermore, the position of the second display surface in the virtual viewpoint image may be predetermined or specified by user operation. For example, the placement unit 203 places the second display surface to the left or right of the top of the virtual viewpoint image. Alternatively, the placement unit 203 may determine the position of the second display surface in the virtual viewpoint image based on the captured image obtained by the imaging unit 130. For example, the placement unit 203 analyzes the captured image to identify the direction in which the refrigerator door opens, etc., included as an image in the captured image, and determines the position of the second display surface in the virtual viewpoint image based on this identification result. Alternatively, the placement unit 203 may place the second display surface in a position such that the display of the moving image and the display of specific text do not overlap, based on the display position of the moving image in the virtual viewpoint image generated by the display control unit 204.

[0040] Furthermore, the image processing unit 110 may accept user input to change the position of at least one of the first and second display surfaces while a moving image and specific text are displayed. For example, the user can select the first or second display surface to change the position of using the operation unit 140 or by performing a gesture input, and move the selected display surface to the desired position. When the image processing unit 110 accepts such user input, the placement unit 203 moves the selected display surface in the virtual space or on the virtual viewpoint image based on the user input. The display control unit 204 controls the display of the moving image or specific text on the display surface. With this configuration, the user can secure a field of view that can improve work efficiency according to the task.

[0041] Figure 9 is a flowchart showing an example of the processing flow of the image processing unit 110 according to Embodiment 2. In the explanation of the flowchart in Figure 9, steps that perform the same processing as those in the flowchart in Figure 8 are denoted by the same reference numerals and their explanation is omitted. The processing shown in the flowchart in Figure 9 is realized by the processor 111 loading a program stored in the non-volatile memory 113 or storage medium 114, etc., into the memory 112 and executing it. Note that the flowchart in Figure 9 is started, for example, when the user turns on the power to the head-mounted display.

[0042] First, the image processing unit 110 executes the processes in S801 and S802. After S802, in S901, the placement unit 203 places the first display surface on which the moving image is played back into a virtual space corresponding to the real space. The method for determining the placement position of the first display surface in the virtual space has been described above and will be omitted from this explanation. Next, in S902, the display control unit 204 controls the display of the moving image acquired in S801 onto the first display surface. Next, in S903, the display control unit 204 generates a virtual viewpoint image corresponding to how the moving image displayed on the first display surface looks from a position in the virtual space corresponding to the user's viewpoint in the real space, based on the position and orientation of the head-mounted display 100. Next, in S904, the placement unit 203 places a second display surface on which specific text is displayed onto the virtual viewpoint image generated in S903. The method for determining the placement position of the second display surface on the virtual viewpoint image has been described above and will be omitted from this explanation.

[0043] Next, in S905, the display control unit 204 controls the display of the specific text acquired in S802 on the second display surface. As a result, a display image including the moving image displayed on the first display surface located in the virtual space and the specific text displayed on the second display surface located on the virtual viewpoint image is displayed on the display unit 120. After S905, the image processing unit 110 executes the processes of S805 and S806 as appropriate. If it is determined in S806 that display control has not been performed until the last frame, the image processing unit 110 returns to the process of S902 and repeatedly executes the processes from S902 to S806 until the following determination is made. Specifically, the image processing unit 110 repeatedly executes the processes from S902 to S806 until it is determined that termination has been instructed in S805, or until it is determined in S806 that display control has been performed until the last frame. In this repeated process, the process of S904 may be omitted.

[0044] Generally, videos related to cooking and other tasks should be displayed as large as possible without interfering with the task. Therefore, when a video is displayed, for example, when a user takes ingredients out of a refrigerator, the video may obstruct the user's view, potentially reducing work efficiency. On the other hand, in this case, to easily identify the ingredients needed for cooking and other tasks, it is desirable that the text display of ingredients be continuously displayed in a position visible to the user.

[0045] With the head-mounted display 100 configured as described above, the first display surface on which moving images are displayed is fixed in a virtual space position corresponding to the position around a predetermined object such as a cutting board. Therefore, for example, when a user is performing a task such as retrieving materials from a storage area located away from the object, such as a refrigerator, specific text will be displayed on the display screen, while moving images will not be displayed on the display screen. As a result, the user can confirm the content of specific text while maintaining a good field of view. Thus, the head-mounted display 100 according to Embodiment 2 can provide support that can further improve the user's work efficiency compared to the head-mounted display 100 according to Embodiment 1.

[0046] [Embodiment 3] Embodiment 2 describes an embodiment in which the first display surface is positioned at a predetermined location in a virtual space corresponding to real space. Embodiment 3 describes an embodiment in which the first display surface is positioned at a location selected from a plurality of pre-set positioning candidates in a virtual space corresponding to real space.

[0047] The hardware configuration of the head-mounted display according to Embodiment 3 is the same as the hardware configuration of the head-mounted display 100 according to Embodiment 1, which is shown as an example in Figure 1. Therefore, the head-mounted display according to Embodiment 3 will be referred to as "head-mounted display 100" in the following description. Furthermore, the parts that the image processing unit 110 according to Embodiment 3 has as a logical configuration are the same as the parts that the image processing unit 110 according to Embodiment 1 has as a logical configuration, although some functions differ. Therefore, the names of the parts that the image processing unit 110 according to Embodiment 3 has as a logical configuration will be written in accordance with the names of the parts that the image processing unit 110 according to Embodiment 1 has as a logical configuration. That is, the image processing unit 110 according to Embodiment 3 (hereinafter simply referred to as "image processing unit 110") has, as a logical configuration, a data acquisition unit 201, a text acquisition unit 202, a layout unit 203, and a display control unit 204 according to Embodiment 3.

[0048] The data acquisition unit 201 and text acquisition unit 202 according to Embodiment 3 (hereinafter simply referred to as "data acquisition unit 201" and "text acquisition unit 202") are the same as the data acquisition unit 201 and text acquisition unit 202 according to Embodiment 1. Therefore, a description of the data acquisition unit 201 and text acquisition unit 202 will be omitted. The display control unit 204 according to Embodiment 3 (hereinafter simply referred to as "display control unit 204") is the same as the display control unit 204 according to Embodiment 2. Therefore, a description of the display control unit 204 will be omitted.

[0049] The placement unit 203 according to Embodiment 3 (hereinafter simply referred to as "placement unit 203") places the first display surface at a predetermined position in a virtual space corresponding to real space, similar to the placement unit 203 according to Embodiment 2. Here, the placement unit 203 according to Embodiment 2 determined the position in the virtual space where the first display surface is placed based on an image obtained by imaging by the imaging unit 130, for example. In contrast, the placement unit 203 differs from the placement unit 203 according to Embodiment 2 in that it places the first display surface at a position of a selected placement candidate from a plurality of placement candidates pre-set in the virtual space. The processing of the placement unit 203 other than this point is the same as the processing of the placement unit 203 according to Embodiment 2, so a description is omitted.

[0050] Figure 10 is a diagram showing an example of the arrangement of arrangement candidates 1001 to 1003 according to Embodiment 3. As an example, Figure 10 shows the layout of a kitchen 1000 that exists in real space, and the kitchen 1000 includes a stove 1011, a cutting board 1012 placed on a countertop, and a sink 1013. Figure 10 also shows a plurality of arrangement candidates 1001 to 1003 that are arranged in a virtual space corresponding to the real space. Specifically, for example, arrangement candidate 1001 is arranged around the virtual space location corresponding to the location of the stove 1011. Similarly, arrangement candidate 1002 is arranged around the virtual space corresponding to the location of the cutting board 1012 in real space, and arrangement candidate 1003 is arranged around the virtual space corresponding to the location of the sink 1013 in real space.

[0051] If the kitchen 1000 is large, the user will move around while cooking according to each step of the cooking process. Therefore, if the first display surface, which shows video related to cooking, is fixed in a predetermined position in the virtual space, it may become difficult for the user to cook while viewing the video when they move. To address this, multiple placement candidates 1001 to 1003 are set in advance, and from among the placement candidates 1001 to 1003, the placement candidate for the first display surface is determined, for example, according to the user's position. The user's position can be determined, for example, based on the position of the head-mounted display 100, which is determined based on the signal from the sensor 150.

[0052] Specifically, for example, the placement unit 203 places the first display surface at the placement candidate 1001 to 1003 that is closest to the user's position, i.e., the position of the head-mounted display 100. Here, the distance between the position of the placement candidate and the position of the head-mounted display 100 is the distance between the position of the placement candidate in virtual space and the position in virtual space corresponding to the position of the head-mounted display 100 in real space. In other words, the placement unit 203 moves the position of the first display surface by changing the placement candidate to which the first display surface is placed in accordance with the movement of the user (head-mounted display 100). The initial placement positions of the multiple placement candidates 1001 to 1003 can be determined, for example, by object recognition through analysis of captured images obtained by imaging by the imaging unit 130, at a predetermined distance from a predetermined object. These initial placement positions may also be determined by user operation.

[0053] Furthermore, although it has been explained above that the placement candidates for the first display surface are determined according to the user's position, the method for determining the placement candidates for the first display surface is not limited to this. For example, the placement unit 203 may determine the placement candidates for the first display surface according to the user's position and orientation, i.e., the position and orientation of the head-mounted display 100. Specifically, for example, the placement unit 203 determines the placement candidates for the first display surface as those included within the display area of ​​the head-mounted display 100, based on the position and orientation of the head-mounted display 100. Here, the position and orientation of the head-mounted display 100 can be determined, for example, based on signals from the sensor 150. Alternatively, for example, the placement unit 203 may determine the placement candidates for the first display surface by analyzing the frames of the video being played back to identify the content of the work at the time of that frame, and estimating the user's position when performing that work. Alternatively, for example, the placement unit 203 may place the first display surface on a placement candidate selected by the user through user operation.

[0054] Furthermore, the image processing unit 110 may accept user input to change the position of at least one of several placement candidates while a moving image and specific text are displayed. For example, the user can select a placement candidate to change the position of using the operation unit 140 or by making a gesture input, and move the selected placement candidate to the desired position. When the image processing unit 110 receives such user input, the placement unit 203 moves the selected placement candidate in the virtual space based on the user input.

[0055] Figure 11 is a flowchart showing an example of the processing flow of the image processing unit 110 according to Embodiment 3. In the explanation of the flowchart shown in Figure 11, steps that perform the same processing as those shown in the flowcharts of Figure 8 or 9 are denoted by the same reference numerals and their explanation is omitted. The processing shown in the flowchart of Figure 11 is realized by the processor 111 loading a program stored in the non-volatile memory 113 or storage medium 114, etc., into the memory 112 and executing it. Note that the flowchart shown in Figure 11 is started, for example, when the user turns on the power of the head-mounted display.

[0056] First, the image processing unit 110 executes the processes in S801 and S802. After S802, in S1101, the placement unit 203 places multiple placement candidates in a virtual space corresponding to the real space, for example, based on the captured image obtained by the imaging unit 130. Next, in S1102, the placement unit 203 obtains the distance between each placement candidate placed in S1101 and the head-mounted display 100. Next, in S1103, the placement unit 203 determines a placement candidate for the first display surface based on the distance between each placement candidate obtained in S1102 and the head-mounted display 100, for example, and places (moves) the first display surface to that placement candidate. After S1103, the image processing unit 110 sequentially executes the processes in S902 to S905. After S905, the image processing unit 110 executes the processes in S805 and S806 as appropriate.

[0057] If it is determined in S806 that display control has not been performed until the last frame, the image processing unit 110 returns to the process in S1102 and repeatedly executes the processes from S1102 to S806 until the following determination is made. Specifically, the image processing unit 110 repeatedly executes the processes from S1102 to S806 until it is determined in S805 that termination has been instructed, or until it is determined in S806 that display control has been performed until the last frame. In this repeated process, the process in S904 may be omitted.

[0058] With the head-mounted display 100 configured as described above, a placement candidate for the first display surface on which the moving image is displayed is selected and determined from among a plurality of placement candidates pre-placed in the virtual space, based on the user's position, etc. Therefore, even if the user moves in accordance with their work, for example, the moving image can be displayed in a position that is easy for the user to see. Accordingly, the head-mounted display 100 according to Embodiment 3 can provide support that can further improve the user's work efficiency compared to the head-mounted display 100 according to Embodiment 2.

[0059] In the above description, the placement unit 203 was described as selectively selecting a placement candidate from among multiple placement candidates for the placement of the first display surface on which the moving image is displayed, but it is not limited to this. For example, the placement unit 203 may select two or more placement candidates from among multiple placement candidates and place the first display surface for each of the two or more selected placement candidates. In this case, for example, the placement unit 203 may select the placement candidate closest to the user's position and the placement candidate second closest to the user's position, and place the first display surface for each of the two selected placement candidates.

[0060] [Other embodiments] The technology of this disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. Furthermore, the technology of this disclosure can also be implemented by a processing circuit such as an ASIC (Application-Specific Integrated Circuit) that implements one or more functions.

[0061] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0062] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0063] <Configuration 1> An image processing apparatus that performs display control for a display unit of a wearable device, A means for obtaining specific text from among the text associated with a video image, A display control means that displays the aforementioned specific text and the aforementioned moving image on different display surfaces, An image processing apparatus characterized by having

[0064] <Configuration 2> The acquisition means acquires the specific text based on the mode selected by the user from among a plurality of modes. An image processing apparatus according to configuration 1, characterized by the above.

[0065] <Structure 3> The acquisition means acquires, based on the selected mode, text indicating at least one of the materials, tools, and work processes corresponding to the selected mode as the specific text. An image processing apparatus according to configuration 2, characterized by the above.

[0066] <Structure 4> The acquisition means, when the selected mode is a cooking mode, acquires text indicating at least one of the ingredients used in cooking and the cooking process as the specific text. An image processing apparatus according to configuration 3, characterized by the above.

[0067] <Composition 5> Arrangement means for arranging the display surface on which the moving image is displayed in a virtual space corresponding to real space, Furthermore, to have, An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0068] <Composition 6> The arrangement means arranges the display surface on which the moving image is displayed at a specified location among a plurality of arrangement candidates provided in the virtual space. The image processing apparatus described in configuration 5, characterized by the above.

[0069] <Composition 7> The arrangement means identifies an arrangement candidate for arranging the display surface on which the moving image is displayed, based on the distance between the virtual space position corresponding to the user's position and each of the plurality of arrangement candidates. An image processing apparatus according to configuration 6, characterized by the above.

[0070] <Structure 8> The display surface on which the aforementioned specific text is displayed is positioned on a virtual viewpoint image obtained by rendering the virtual space. An image processing apparatus according to any one of configurations 5 to 7, characterized by the above.

[0071] <Composition 9> The position of the display surface where the aforementioned specific text is displayed does not change depending on the user's position and orientation. An image processing apparatus according to any one of configurations 1 to 8 characterized by the above.

[0072] <Composition 10> An image processing device as described in any one of configurations 1 to 9, A display unit in which the specific text and the moving image are displayed by the display control of the image processing device, A wearable device characterized by having the following features.

[0073] <Method 1> An image processing method for controlling the display of a display unit in a wearable device, The acquisition process involves obtaining specific text from the text associated with a video image, A display control step of displaying the aforementioned specific text and the aforementioned moving image on different display surfaces, An image processing method characterized by including

[0074] <Method 2> The acquisition process involves obtaining specific text from the text associated with a video image, A display control step of displaying the aforementioned specific text and the aforementioned moving image on different display surfaces, A method for controlling a wearable device, characterized by including [a specific feature].

[0075] <Program> A program for causing a computer to function as an image processing device described in any one of configurations 1 through 9. [Explanation of symbols]

[0076] 100 Head-Mounted Displays 110 Image Processing Unit 120 Display section 202 Text Acquisition Unit 204 Display Control Unit

Claims

1. An image processing apparatus that performs display control for a display unit of a wearable device, A means for obtaining specific text from among the text associated with a video image, A display control means that displays the aforementioned specific text and the aforementioned moving image on different display surfaces, An image processing apparatus characterized by having

2. The acquisition means acquires the specific text based on the mode selected by the user from among a plurality of modes. The image processing apparatus according to claim 1, characterized in that

3. The acquisition means acquires, based on the selected mode, text indicating at least one of the materials, tools, and work processes corresponding to the selected mode as the specific text. The image processing apparatus according to claim 2, characterized by the above.

4. The acquisition means, when the selected mode is a cooking mode, acquires text indicating at least one of the ingredients used in cooking and the cooking process as the specific text. The image processing apparatus according to claim 3, characterized by the following:

5. Arrangement means for arranging the display surface on which the moving image is displayed in a virtual space corresponding to real space, Furthermore, to have, The image processing apparatus according to claim 1, characterized in that

6. The arrangement means arranges the display surface on which the moving image is displayed at a specified location among a plurality of arrangement candidates provided in the virtual space. The image processing apparatus according to claim 5, characterized by the following:

7. The arrangement means identifies an arrangement candidate for arranging the display surface on which the moving image is displayed, based on the distance between the virtual space position corresponding to the user's position and each of the plurality of arrangement candidates. The image processing apparatus according to claim 6, characterized by the above.

8. The display surface on which the aforementioned specific text is displayed is positioned on a virtual viewpoint image obtained by rendering the virtual space. The image processing apparatus according to claim 5, characterized by the following:

9. The position of the display surface where the aforementioned specific text is displayed does not change depending on the user's position and orientation. The image processing apparatus according to claim 1, characterized in that

10. An image processing apparatus according to any one of claims 1 to 9, A display unit in which the specific text and the moving image are displayed by the display control of the image processing device, A wearable device characterized by having the following features.

11. An image processing method for controlling the display of a display unit in a wearable device, The acquisition process involves obtaining specific text from the text associated with a video image, A display control step of displaying the aforementioned specific text and the aforementioned moving image on different display surfaces, An image processing method characterized by including

12. The acquisition process involves obtaining specific text from the text associated with a video image, A display control step of displaying the aforementioned specific text and the aforementioned moving image on different display surfaces, A method for controlling a wearable device, characterized by including [a specific feature].

13. A program for causing a computer to function as an image processing device according to any one of claims 1 to 9.

Images