Display system and display device

Abstract

The present invention provides a display system and a display device, which are suitable for workers working in real space and can display information related to the work. The display system (10) is configured to include a display (160) and a detector for detecting the orientation of the display. The display system (10) includes: a display source image generating unit, which generates a display source image including a first image and a second image associated with the first image; and a field of view image display unit, which displays the first image in a first display area of ​​the display and displays the second image in a second display area of ​​the display. When the display mode associated with the first image is the first display mode, the field of view image display unit uses a predetermined fixed area of ​​the display as the first display area, and when the display mode is the second display mode, the first display area is determined according to the orientation of the display.

Description

Technical Field

[0001] The present invention relates to a display system and a display device for displaying information for assisting work. Background Art

[0002] In the manufacture of products in factories (workshops), inspection and repair of factory equipment, etc., sometimes the work is performed while observing work documents such as process documents and pictures. However, depending on the work environment, it is sometimes difficult to configure a device such as a display that displays the work documents near the work object. As a display device that can be used in such a situation, a perspective head-mounted display device (Head-mounted Display, Head-Mounted Display, hereinafter also referred to as HMD) or smart glasses that are worn on the operator's head and display images of the virtual space superimposed on the real space have attracted attention. If a perspective HMD or the like is used, the operator does not need to hold the display device or watch a display device in the distance, which can improve work efficiency.

[0003] Regarding display control in the HMD, the display screen is changed / configured according to the state of the HMD, the operator, or the display content, thereby making it easy to use. For example, in the HMD described in Patent Document 1, an image of a virtual camera that captures a virtual space is generated according to the angle of the line of sight measured by the HMD, and is displayed together with an image that does not change according to the angle.

[0004] In the invention described in Patent Document 1, for example, a first image that indicates the state of a virtual space according to the orientation of a non-perspective HMD and a second image that does not change according to the orientation of the HMD in a predetermined area of ​​the display screen are displayed. On the other hand, in an HMD that assists on-site operations such as factories, the appropriate display methods of the first image and the second image differ depending on the equipment used, the scene used, and the display content. Therefore, the technology described in Patent Document 1 cannot be said to be suitable as a display device for on-site operations.

[0005] Patent Document 1: Japanese Patent Application Publication No. 2019-101330 Summary of the invention

[0006] The present invention has been made in view of such a background, and an object of the present invention is to provide a display system and a display device which are suitable for a worker who performs work in a real space and can display information related to the work.

[0007] In order to solve the above-mentioned problems, the display system involved in the present invention is constructed to include a display and a detector for detecting the orientation of the display, and comprises: a display source image generating unit, which generates a display source image including a first image and a second image associated with the first image; and a field of view image display unit, which displays the first image in a first display area of ​​the display and displays the second image in a second display area of ​​the display, if the display mode associated with the first image is the first display mode, the field of view image display unit sets a predetermined fixed area of ​​the display as the first display area, and if the display mode is the second display mode, the field of view image display unit determines the first display area according to the orientation of the display.

[0008] According to the present invention, a display system and a display device that can display information related to work suitable for a worker who performs work in a real space can be provided. Other problems, structures, and effects than those described above will become clear from the following description of the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a diagram showing the configuration of a display system according to the first embodiment.

[0010] Figure 2 This is a functional block diagram of the HMD according to the first embodiment.

[0011] Figure 3 This is a functional block diagram of the smartphone according to the first embodiment.

[0012] Figure 4 This is a diagram showing the data structure of the job file database according to the first embodiment.

[0013] Figure 5 It is a diagram showing a structure of a display source image according to the first embodiment.

[0014] Figure 6 This is a diagram showing a structure of a field of view image displayed on a display in the first display mode of the first embodiment.

[0015] Figure 7 This is a diagram showing a structure of a field of view image displayed on a display in the second display mode of the first embodiment.

[0016] Figure 8 It is a diagram showing a structure of a field of view image displayed on a display in the third display mode of the first embodiment.

[0017] Fig. 9 This is a sequence diagram of the configuration definition process according to the first embodiment.

[0018] Fig.10This is a flowchart of the display source image generation process in the first embodiment.

[0019] Fig.11 This is a flowchart of the field of view image display processing in the first embodiment.

[0020] Fig.12 This is a functional block diagram of an HMD according to the second embodiment.

[0021] Fig.13 This is a functional block diagram of a smartphone according to a third embodiment.

[0022] Fig.14 This is a flowchart of the display source image generation process in the third embodiment.

[0023] Description of Reference Numerals

[0024] 10. Display system;

[0025] 100, 100A HMD (display device);

[0026] 111 data receiving unit;

[0027] 112 visual field image display unit;

[0028] 113 display mode control unit;

[0029] 122 Display source image;

[0030] 126 display mode;

[0031] 160 display;

[0032] 170 sensor (detector);

[0033] 170A Sensors (detectors, microphones);

[0034] 175 Operation Department;

[0035] 200, 200B Smartphone (portable device, external device);

[0036] 211 Configuration Definition Department;

[0037] 212 Display source image generation unit;

[0038] 213 display mode control unit;

[0039] 214 process file emphasis unit (display source image generation unit);

[0040] 224 display mode;

[0041] 230 Display source image;

[0042] 231 process file configuration area (area of ​​the first image);

[0043] 232 picture configuration area (area of ​​the second image);

[0044] 233, 233A visual field area;

[0045] 240 Job File Database;

[0046] 250 image database;

[0047] 420, 420A, 420B visual field images;

[0048] 421 process file area in the field of view (first display area, fixed area);

[0049] 422 image area within the field of view (second display area);

[0050] 423 The process file area in the field of view (the first display area). DETAILED DESCRIPTION

[0051] 《Overview of Display System》

[0052] The following is a description of a display system in a method (implementation method) for implementing the present invention. The display system is configured to include a smartphone and a see-through HMD (head mounted display). The smartphone stores a process file and a picture of the operation, generates an image (also referred to as a display source image) containing the process file and the picture of the operation steps being performed by the operator in each predetermined area, and sends it to the HMD.

[0053] The HMD displays the flow file and the picture included in the display source image at a predetermined position corresponding to the display mode. For example, in the case of the first display mode, the HMD cuts out the image from the area of ​​the flow file included in the display source image and displays it at a predetermined position (for example, the lower center) on the screen of the HMD. In addition, the HMD displays the picture according to the orientation of the subject. For example, suppose the area of ​​the picture is located on the right side of the display source image (refer to the following description). Figure 5 The HMD displays the picture configuration area 232 in the display source image 230. Thus, the HMD does not display the picture when the operator is facing forward. When the HMD detects that the operator is slightly facing right, the left side of the picture is displayed on the right side of the screen. When the HMD detects that the operator is facing right, the entire picture is displayed on the right side of the screen.

[0054] In this way, by displaying the process file and the picture on the HMD, the operator can always refer to the process file and perform the work. In addition, by facing the right, the operator can refer to the picture. When facing the front, the picture is not displayed, and the operator's field of vision of the work object and the work site (actual space) is not blocked. As a result, the operator can work with both hands while referring to the process file and the picture without operating the smartphone or HMD with his hands.

[0055] In the second display mode, the HMD displays the image cut out from the display source image directly on the screen of the HMD. Compared with the first display mode, since there is no fixed display flow file, the visibility of the actual space (workplace, work space) is improved.

[0056] The display system switches between the first display mode and the second display mode according to the work. By appropriately setting the display mode in advance according to the work content (flow file), the operator can perform the work without performing the operation of switching the display mode.

[0057] 《First Embodiment: Overall Structure of Display System》

[0058] Figure 1 : This is a diagram showing the structure of the display system 10 of the first embodiment. The display system 10 is configured to include HMD100 (display device) and a smart phone 200 (portable device). HMD100 is, for example, a goggle-type display device that is worn on the head in a manner that covers the field of vision. HMD100 and the smart phone 200 are connected, for example, via a USB (Universal Serial Bus) cable, but can also be connected via cables of other standards or wirelessly. In addition, the smart phone 200 can also be connected to the USB, and is converted to an image output connector such as HDMI (registered trademark) (High-Definition Multimedia Interface: High Definition Multimedia Interface) or DisplayPort (display port) via a conversion adapter and connected to the HMD100. In addition, the HMD100 also includes smart glasses that are used like glasses. Similarly, the smart phone 200 (portable device) also includes portable terminals such as tablet computers and notebook PCs.

[0059] <First embodiment: Structure of HMD>

[0060] Figure 21 is a functional block diagram of HMD 100 according to the first embodiment. HMD 100 includes a control unit 110, a storage unit 120, a display 160, a sensor 170, and a communication unit 180. The communication unit 180 has one or more communication interfaces such as USB and Wi-Fi (registered trademark), and transmits and receives data with a smartphone 200.

[0061] The display 160 is a see-through display device having high transmittance provided on the front of the HMD 100 (see Figure 1 ). In addition to Figure 1 In addition to such a binocular type, a monocular type that can be used by either the left or right eye may be used. The operator can perform work while wearing HMD 100 and referring to information displayed on display 160 .

[0062] The sensor 170 is a sensor including a MEMS (Micro Electro Mechanical Systems) gyroscope, for example, and detects the angle or angular velocity of the HMD 100 and outputs it to the control unit 110. The sensor 170 may include an illuminance sensor for measuring the surrounding illuminance and a camera for photographing the surroundings or the operator.

[0063] The storage unit 120 is composed of a ROM (Read Only Memory), a RAM (Random Access Memory), a flash memory, etc. The storage unit 120 stores a program 121, a display source image 122, a flow file coordinate 123, a related image coordinate 124, a flow file coordinate 125 within the field of view, and a display mode 126. The program 121 is a program executed by the CPU (Central Processing Unit) constituting the control unit 110, and controls the HMD 100. The program 121 includes a field of view image display process (see below). Fig.11 ) and other processing procedures.

[0064] The display source image 122 is a display source image 230 (see below) received from the smartphone 200. Figure 3 , Figure 5 ), which is an image that is the basis of the visual field image displayed on the display 160. The process file coordinates 123, the associated image coordinates 124, and the visual field process file coordinates 125 are coordinate data representing the area in the display source image 122 received from the smartphone 200, and are respectively related to the smartphone 200 (see Figure 3) are equivalent to the process file coordinates 221, the associated image coordinates 222, and the process file coordinates 223 in the field of view. In addition, the display mode 126 indicates the display method of the field of view image, and is equivalent to the display mode 224 stored in the smartphone 200 described later.

[0065] The control unit 110 includes a CPU and has a data receiving unit 111 and a field of view image display unit 112. The data receiving unit 111 stores the data received from the smartphone 200 in a display source image 122, a flow file coordinate 123, a related picture coordinate 124, a field of view flow file coordinate 125, and a display mode 126.

[0066] The visual field image display unit 112 displays the source image 122 (see the display mode 126) according to the display mode 126. Figure 5 The image is cut out from the display source image 230 and displayed on the display 160. When cutting out the image, the field of view image display unit 112 refers to the flow file coordinates 123, the related picture coordinates 124, the field of view flow file coordinates 125, and the display mode 126.

[0067] 《First Embodiment: Structure of Smartphone》

[0068] Figure 3 1 is a functional block diagram of the smartphone 200 according to the first embodiment. The smartphone 200 includes a control unit 210, a storage unit 220, a touch panel display 260, a microphone 270, and a communication unit 280. The communication unit 280 has one or more communication interfaces such as USB and Wi-Fi, and transmits and receives data with the HMD 100.

[0069] The storage unit 220 is composed of ROM, RAM, flash memory, etc. The storage unit 220 stores a display source image 230, a process file coordinate 221, a related image coordinate 222, a process file coordinate 223 within the field of view, a display mode 224, and a work file database 240 (see below). Figure 4 ), picture database 250 and program 229.

[0070] The program 229 is a program executed by the CPU constituting the control unit 210, and controls the smartphone 200. The program 229 includes a display source image generation process (see the following description). Fig.10 ) and other processing procedures.

[0071] Figure 4 2 is a data structure diagram of the work file database 240 of the first embodiment. The work file database 240 is, for example, data in a table format. One row (record) of the work file database 240 represents one work step performed by an operator who is a user of the display system 10. The record is composed of a step number 241 (in Figure 4 ), work target part 242, work content 243, related pictures 244, completion mark 245 (in Figure 4 F), completion date and time 246, and display mode 247 (in Figure 4 Recorded as display M).

[0072] The step number 241 is a number assigned to the operation step, indicating the order of the operation steps. The operation target part 242 indicates the part that becomes the target of the operation such as repair and inspection, for example, the name of the operation target such as "machine No. 1 of model A". The operation content 243 is displayed on the display 160 (refer to Figure 1 as well as Figure 2 ) is an explanation of the work step, for example, an explanation of the work content such as "turn off switch B". The associated image 244 is an image associated with the work step, and is identification information of the image displayed on the display 160. The completion mark 245 indicates whether the work step is completed ("yes") or not completed ("no"). The completion date and time 246 is the date and time when the work step is completed. The display mode 247 indicates how to display the process file and the associated images, and the first to third display modes are specified in the first embodiment. The details of the display mode are described later.

[0073] Back to Figure 3 The picture database 250 stores pictures associated with the work steps. Figure 4 ) corresponding identification information. The control unit 210 can access the image by specifying the identification information. With respect to the display source image 230, the process file coordinates 221, the associated image coordinates 222, and the process file coordinates 223 within the field of view, reference is made to the following description. Figures 5 to 8 While explaining.

[0074] The display mode 224 is a mode related to the structure of the field of view image including the flow chart file and the picture. In the first embodiment, there are three display modes, namely, the first to the third. For details of the display modes, see Figures 5 to 8 Provide explanation.

[0075] 《First Embodiment: Displaying Source Image and Field of View Image》

[0076] Figure 5 The display source image 230 is a diagram showing the structure of the display source image 230 according to the first embodiment. The display source image 230 is displayed on the display 160 (see Figure 1 as well as Figure 2 ) of the image based on the image, and Figure 2The process file configuration area 231 is a partial area of ​​the display source image 230, which is used to display the content of the process file ( Figure 4 The image configuration area 232 is a partial area for displaying the source image 230, and is an area for displaying images associated with the process file (by Figure 4 The associated picture 244 identifies an area of ​​the picture stored in the picture database 250).

[0077] The viewing area 233 is a partial area of ​​the source image 230, which is displayed on the display 160 in the first and second display modes (see Figure 1 , Figure 2 The size of the field of view area 233 is the display size of the display 160.

[0078] The visual field area 233 moves within the display source image 230 according to the orientation of the HMD 100. For example, when the operator faces left and the sensor 170 (see Figure 2 ) When the left rotation of HMD 100 is detected, the field of view area 233A located on the left side of the display source image 230 is displayed on the display 160.

[0079] Process file coordinates 221 (refer to Figure 3 ) are the configuration coordinates of the process file configuration area 231 in the display source image 230. The configuration coordinates are, for example, the coordinates of the upper left vertex and the lower right vertex of the area in the display source image 230. The configuration coordinates may also be the coordinates of the upper left vertex of the area and the size of the area. The associated picture coordinates 222 are the configuration coordinates of the picture configuration area 232 in the display source image 230.

[0080] Below, while referring to Figure 6 to Figure 8 , while describing the structures of the field of view images 420 , 420A, and 420B in the first to third display modes (“of” is omitted, and are also described as the first to third display modes).

[0081] Figure 6 1 is a diagram showing a structure of a visual field image 420 displayed on the display 160 in the first display mode of the first embodiment. The visual field image 420 is a diagram showing a structure of a visual field image 420 displayed on the display 160 in the first display mode of the first embodiment. Figure 5) is a field of view image in the first display mode. Field of view image 420 is an image obtained by cutting out a field of view picture area 422 described later from display source image 230, to which a process file area 421 in the field of view is added. In display 160, pixels are not displayed in a portion other than field of view picture area 422 and field of view process file area 421, so that an operator wearing HMD 100 can see the work place (actual space) through this portion of the field of view.

[0082] The in-view process file area 421 is a partial area of ​​the view image 420, and is an area where the process file is displayed. In the first embodiment, the in-view process file area 421 is arranged at the lower center of the view image 420, but it can also be arranged at other positions. Figure 3 ) are the configuration coordinates of the in-field flow file area 421 in the field of view image 420.

[0083] The image area 422 in the field of view is the field of view area 233 (see Figure 5 ) is an area that overlaps with the picture configuration area 232, and is an area that includes a portion of the picture. The field of view area 233 moves in the display source image 230 according to the orientation of the HMD 100, so the area in which the field of view area 233 overlaps with the picture configuration area 232, that is, the in-field picture area 422 also changes according to the orientation of the HMD 100. For example, when the operator faces the left side, the field of view area 233 moves to the field of view area 233A. As a result, the field of view area 233A does not overlap with the picture configuration area 232, so the in-field picture area 422 does not exist, and the picture is not displayed on the display 160.

[0084] In the first display mode, the process file area 421 in the field of view image 420 is fixed, and the process file is always displayed on the display 160, so the operator can check the process file even if the orientation of the HMD 100 (head) is not changed. Therefore, the work efficiency is improved when multiple relatively short processes are performed on the same work object.

[0085] Figure 7 4 is a diagram showing a structure of a visual field image 420A displayed on the display 160 in the second display mode of the first embodiment. The visual field image 420A is a diagram showing a structure of a visual field image 420A displayed on the display 160 in the second display mode of the first embodiment. Figure 5 ) is a field of view image in the second display mode. Field of view image 420A is an image obtained by cutting out the field of view image area 422 and the field of view process file area 423 from the display source image 230. In the display 160, pixels are not displayed in the portion other than the field of view image area 422 and the field of view process file area 423, so the operator wearing HMD 100 can see the work place through this portion of the field of view.

[0086] The in-view flow file area 423 is an area where the view area 233 and the flow file arrangement area 231 overlap. Also, the in-view picture area 422 is an area where the view area 233 and the picture arrangement area 232 overlap.

[0087] In the second display mode, the in-view process file area 423 in the view image 420A is not fixed, and the process file is not displayed fixedly, so the visibility of the actual space is improved. Therefore, for the case where relatively precise work is performed intensively, the work efficiency is improved.

[0088] Figure 8 This is a diagram showing the structure of a field of view image 420B displayed on the display 160 in the third display mode of the first embodiment. The field of view image 420B is an image corresponding to the display source image 230 and is a field of view image in the third display mode. The number of pixels of the display 160 is smaller than the number of pixels of the original image, and the field of view image 420B is an image obtained by reducing the display source image 230. The process file configuration area 231 and the picture configuration area 232 are also reduced to become the field of view process file area 425 and the field of view picture area 424, respectively.

[0089] In the third display mode, since the process file and the image can be checked at the same time, it is convenient to check the content before the operation is performed. In addition, when using a single-eye HMD, even if the process file and the image are always displayed on the display, the visual recognition of the actual space can be ensured, so it can also be set to the third display mode all the time.

[0090] 《First Embodiment: Configuration of Smartphone: Control Unit》

[0091] Back to Figure 3 The control unit 210 includes a CPU, and has a configuration definition unit 211, a display source image generation unit 212, and a display mode control unit 213. The configuration definition unit 211 receives instructions from an operator who is a user of the display system 10, and defines (sets) a display source image 230 (see Figure 5 ) in the process file configuration area 231, the configuration (configuration coordinates) of the image configuration area 232, and the field of view image 420 (refer to Figure 6 ) in the field of view process file area 421. The configuration definition unit 211 sends the defined configuration coordinates to the HMD 100.

[0092] The display source image generating unit 212 generates a display source image 230 including a flow document and a related image related to a work step currently being performed by the worker, and transmits the display source image 230 to the HMD 100 .

[0093] The display mode control unit 213 determines the display mode and stores it as the display mode 224 .

[0094] "First Implementation Method: Configuration Definition Processing"

[0095] Fig. 9 This is a sequence diagram of the configuration definition process of the first embodiment. Fig. 9 , while performing and displaying the source image 230 (refer to Figure 5 ) and visual field image 420 (refer to Figure 6 )The processing of the smartphone 200 and HMD 100 related to the settings is explained.

[0096] In step S111, the arrangement definition unit 211 of the smartphone 200 obtains the display source image 230 (see FIG. 1 ) instructed by the operator who is the user of the display system 10. Figure 5 ) and stored as the process file coordinates 221 (refer to Figure 3 ). The configuration coordinates are, for example, the coordinates of the upper left vertex and the lower right vertex of the region in the display source image 230 .

[0097] In step S112 , the arrangement definition unit 211 obtains the arrangement coordinates of the picture arrangement area 232 in the display source image 230 designated by the operator, and stores the coordinates as the associated picture coordinates 222 .

[0098] In step S113, the configuration definition unit 211 obtains the visual field image 420 (see Figure 6 ) and stored as the in-view process file coordinates 223.

[0099] In step S114 , the configuration definition unit 211 transmits the flow file coordinates 221 , the related image coordinates 222 , and the in-field flow file coordinates 223 to the HMD 100 .

[0100] In step S115 , the data receiving unit 111 of the HMD 100 stores the received flow file coordinates 221 , related image coordinates 222 , and in-view flow file coordinates 223 as flow file coordinates 123 , related image coordinates 124 , and in-view flow file coordinates 125 , respectively.

[0101] Next, the processing of the display system 10 when the operator performs work using the display system 10 will be described.

[0102] <First Embodiment: Display Source Image Generation Processing>

[0103] Fig.10 1 is a flowchart of the display source image generation process in the first embodiment. Fig.10 , while explaining the process in which the smartphone 200 generates the display source image 230 and transmits it to the HMD 100.

[0104] In step S131, if there is a completion instruction of the work step (step S131→Yes), the display source image generation unit 212 proceeds to step S132, and if there is no completion instruction of the work step (step S131→No), it proceeds to step S133. The completion instruction of the work step refers to the situation that the display system 10 instructs the operator to record the completion of one work step. The display source image generation unit 212 generates a display source image based on the microphone 270 (refer to Figure 3 ) and detects the operator's utterance of "operation step completed" to determine whether there is an instruction to complete the operation step. Alternatively, the display source image generation unit 212 may detect a click of the operation step completion button displayed on the touch panel display 260 to determine that there is an instruction to complete the operation step.

[0105] In step S132, the display source image generation unit 212 records the completion of the work step. Specifically, the display source image generation unit 212 sets the completion mark 245 (see Figure 4 ) is updated to "yes", and the completion date and time 246 is updated to the current time. The current operation step is the operation step with the completion flag 245 being "no" and the smallest step number 241.

[0106] In step S133 , the display source image generating unit 212 generates a blank image and stores it as the display source image 230 .

[0107] In step S134, the display source image generation unit 212 generates a flow file configuration area 231 (see Figure 5 Specifically, the display source image generation unit 212 depicts the step number 241, the work target part 242, the work content 243, the related image 244, and the completion mark 245 of the current work step in the process file configuration area 231 (see Figure 4 ).

[0108] In step S135 , the display mode control unit 213 acquires the display mode 247 of the current work step and stores it as the display mode 224 .

[0109] In step S136 , the display source image generation unit 212 draws the related picture in the picture placement area 232 of the display source image 230 . Specifically, the display source image generation unit 212 obtains the picture corresponding to the identification information in the related picture 244 of the current work step from the picture database 250 and draws it in the picture placement area 232 .

[0110] In step S137 , the display source image generation unit 212 transmits the display source image 230 and the display mode 224 to the HMD 100 , and the process returns to step S131 . In addition, the display mode control unit 213 may transmit the display mode 224 to the HMD 100 .

[0111] 《First Embodiment: Field of View Image Display Processing》

[0112] Fig.11 FIG. 1 is a flowchart of the visual field image display process in the first embodiment. Fig.11 , while HMD 100 receives the information from smartphone 200 (refer to Fig.10 The process of displaying the source image 230 and the field of view image on the display 160 in step S137) and the display mode 224 is described below.

[0113] In step S151 , the data receiving unit 111 receives the display source image 230 and the display mode 224 transmitted from the smartphone 200 , and stores them as the display source image 122 and the display mode 126 , respectively.

[0114] In step S152, if the display mode 126 is the first display mode (step S152→first display mode), the field of view image display unit 112 enters step S153; if it is the second display mode (step S152→second display mode), it enters step S156; if it is the third display mode (step S152→third display mode), it enters step S159.

[0115] In step S153, the visual field image display unit 112 places the flow file configuration area 231 (see Figure 5 , the image of the partial area of ​​the display source image 122 represented by the process file coordinates 123) is displayed in the area of ​​the display 160 represented by the process file coordinates 125 in the field of view (refer to Figure 6 In the process file area 421 within the field of view.

[0116] In step S154, the visual field image display unit 112 calculates the visual field area 233 (see FIG. 234 ) based on the orientation (angle) of the HMD 100 acquired by the sensor 170. Figure 5 ).

[0117] In step S155, the visual field image display unit 112 displays the overlapping area of ​​the visual field area 233 and the picture arrangement area 232 in the corresponding area of ​​the display 160, and returns to step S151. The corresponding area of ​​the display 160 refers to the area where the display image of the display 160 is regarded as the visual field image 420 (see Figure 6 ), the area corresponding to the in-field image area 422 within the field of view image 420.

[0118] Step S156 is the same as step S154.

[0119] In step S157, the field of view image display unit 112 displays the overlapping area of ​​the field of view area 233 and the flow file configuration area 231 in the corresponding area of ​​the display 160. In other words, the field of view image display unit 112 determines the area of ​​the field of view area 233 that overlaps with the flow file configuration area 231 as the in-field flow file area 423 according to the orientation of the HMD 100.

[0120] Step S158 is the same as step S155.

[0121] In step S159, the visual field image display unit 112 displays the display source image 122 on the display 160, and returns to step S151. At this time, the size of the display source image 122 may be changed so that the display source image 122 is displayed on the display 160 in full screen.

[0122] 《First Embodiment: Characteristics of Display System》

[0123] The display 160 of the HMD 100 displays a flow document of a work step that the worker is currently working on and a related image.

[0124] In the case of the first display mode, the process file is always displayed at a set position of the display 160 (see Figure 6 Flow file area 421 within the field of view). Therefore, the operator can always refer to the flow file while working. In addition, the related picture is displayed according to the direction of the operator's head (HMD100). It is assumed that the position of the picture is set to the right side of the display source image 230. Therefore, the picture is not displayed when the operator faces the front, and the operator can work without being hindered by the picture. The operator can refer to the picture by facing the right side. As a result, the operator can work with both hands without operating the smartphone 200 and HMD100 with hands, and can refer to the flow file and pictures.

[0125] In the case of the second display mode, since the flow chart is not displayed fixedly in the field of view, the visibility of the actual space is improved, and the work efficiency is improved when more detailed work is performed intensively.

[0126] In the case of the third display mode, the process file and the image can be confirmed on one display, which is convenient when confirming the content before the operation is performed. In addition, when using a single-eye HMD, even if the process file and the image are always displayed on the display, the visual recognition for the actual space can be ensured, so it can be always set to the third display mode.

[0127] The display mode switches according to the operation step (refer to Fig.10 Steps S135, S137, Fig.11 By setting the operation mode according to the operation content corresponding to the operation step, the operator does not need to switch the operation mode during the operation, and can concentrate on the operation, thereby improving the operation efficiency.

[0128] 《First Embodiment: Modification: Switching of Display Mode》

[0129] In addition, the display system 10 of the first embodiment changes the first to third display modes according to the display mode 247 set in the work file database 240, but other methods may be used. For example, the display mode control unit 213 may identify the type of content to be displayed, and if it is a setting screen (see Fig. 9 ) is set to the third display mode, and if it is a work screen, it is set to the first display mode. In addition, the display mode control unit 213 can also obtain the type of HMD (monocular / binocular) from the HMD 100, and set it to the third display mode in the case of a monocular type, and set it to the first display mode in the case of a binocular type ... Figure 3 ) to switch the display mode based on the operator's voice command.

[0130] Second Implementation Method

[0131] In the first embodiment, the HMD 100 switches the display mode according to the display mode 126 transmitted from the smartphone 200. The display mode may be switched by an operator operating the HMD 100.

[0132] Fig.12 1 is a functional block diagram of HMD 100A according to the second embodiment. Figure 2 ) compared to HMD100A, HMD100A has an operation unit 175, the control unit 110 has a display mode control unit 113, and the sensor 170A has a microphone.

[0133] The operation unit 175 is, for example, a button. The display mode control unit 113 recognizes the voice picked up by the microphone provided in the sensor 170A to switch the display control mode. The display mode control unit 113 recognizes, for example, a voice such as "display mode 1" and switches to the first display mode. In addition, each time the button of the operation unit 175 is pressed, the display mode control unit 113 sequentially switches the first display mode, the second display mode, and the third display mode with a toggle.

[0134] In the first embodiment, the display mode is set to display mode 247 (see Figure 4), which is fixed according to the operation steps (operation contents). In the second embodiment, the operator can switch according to the situation, and the usability of the display system 10 is improved.

[0135] 《Third Implementation Mode》

[0136] Fig.13 This is a functional block diagram of a smartphone 200B according to the third embodiment. In the third embodiment, a process file highlighting unit 214 is added to highlight the difference in process files. The process file highlighting unit 214 highlights the difference between the work content 243 (work process) in the work file database 240 and the previous work process. This can prevent reading errors in the work process, etc.

[0137] Fig.14 This is a flowchart of the display source image generation process in the third embodiment.

[0138] Steps S311 to S313 are respectively Fig.10 The described steps S131 to S133 are the same.

[0139] In step S314, the flow chart highlighting unit 214 calculates the difference between the flow chart of the current work step and the flow chart of the previous work step. The difference is calculated by using, for example, the Levenshtein distance.

[0140] In step S315, the flow file emphasis unit 214 proceeds to step S316 if the Levenshtein distance D calculated in step S314 is greater than 0 and less than a predetermined threshold value (D_a) (step S315→Yes), and proceeds to step S318 in other cases (step S315→No).

[0141] In step S316 , the process flow document highlighting unit 214 identifies characters that are different between the current process flow document and the previous process flow document.

[0142] In step S317, the display source image generation unit 212 changes the color of the characters to emphasize the characters determined in step S316, and displays the characters in the process file configuration area 231 (see Figure 5 ) describes the process file.

[0143] Steps S318 to S321 are respectively Fig.10 The recorded steps S134 to S137 are the same.

[0144] <<Features of the Third Embodiment>>

[0145] In the third embodiment, when the flow charts (work contents) of the preceding and following work steps are similar, the flow charts are displayed to make the differences clear. This can prevent the operator from making assumptions or reading errors, and enable more reliable work.

[0146] In the third embodiment, the process flow files of the preceding and following work steps are compared. When the current work step includes a plurality of work processes, the differences between the plurality of work processes may be emphasized.

[0147] 《Variant: Field of View》

[0148] In the above-mentioned embodiment, the field of view area 233 (see Figure 5 ) is calculated based on the orientation of the HMD 100 detected by the sensor 170 (refer to Fig.11 In contrast, the field of view area 233 may be determined based on other information.

[0149] As a first example, the sensor 170 may include a line of sight sensor for detecting the line of sight of the operator wearing the HMD 100, and the field of view image display unit 112 may determine the moving direction of the field of view area 233 based on the line of sight. For example, based on the direction of the line of sight such as up, upper right, or left, the field of view area 233 in the display source image 230 may be moved from the current position to the same direction as the line of sight.

[0150] As a second example, the visual field image display unit 112 may determine the moving direction of the visual field area 233 based on the voice of the operator detected by the microphone provided in the HMD 100 or the smartphone 200. For example, the voice of the operator indicating a direction such as up, upper right, or left may be detected, and the visual field area 233 in the display source image 230 may be moved in the same direction as the direction indicated by the voice.

[0151] As a third example, the visual field image display unit 112 may determine the moving direction of the visual field area 233 according to the direction of the sound detected by the microphone provided in the HMD 100. The sound is the sound of the work object (predetermined sound source), such as the engine sound or the motor sound. For example, the direction of the sound of the work object may be detected, and the visual field area 233 may be moved in the same direction as the sound.

[0152] In the first to third examples, the visual field image display unit 112 determines the moving direction of the visual field area 233 based on the line of sight, voice, or sound of the work object. However, the visual field image 420 (see Figure 6 , Figure 7) in the visual field image area 422. For example, the visual field image area 422 may be arranged at the upper, upper right, left, etc. of the visual field image 420 (an area above, upper right, left, etc., of the visual field image 420) according to the visual line of sight, voice, or sound of an object. When arranged, the visual field image display unit 112 displays the display source image 230 (see FIG. 1 ) at an area above, upper right, left, etc., of the center of the display 160. Figure 5 )'s picture configuration area 232.

[0153] In addition, in the case of the sound of the object, the in-view image area 422 may be arranged in the upper, upper right, left, etc. area of ​​the center of the view image 420 in the opposite direction to the direction of the sound of the object. By arranging in the opposite direction, the operator can ensure the view in the direction where the object is located, making it easier to work.

[0154] Furthermore, in order to ensure a field of view, picture arrangement area 232 is preferably displayed in an area above, to the upper right, or to the left of the center of display 160 so as to be in contact with an end (periphery) of display 160 .

[0155] 《Other Variations》

[0156] The present invention can adopt other various embodiments, and various changes such as omission and substitution can be made without departing from the scope of the present invention. For example, the functional parts and data of the smartphone 200 can also be acquired by the HMD 100. In this case, the display system 10 can be regarded as the HMD 100.

[0157] The components (functional parts) of the control unit 110 of the HMD 100 and the control unit 210 of the smartphone 200 may also be moved. For example, the configuration definition unit 211, the display source image generation unit 212, and the display mode control unit 213 may all be included in the HMD 100. In this case, the display source image generation unit 212 and the display mode control unit 213 in the HMD 100 access the job file database 240 and the picture database 250 stored in the smartphone 200. Such an embodiment is preferable when the storage unit 120 of the HMD 100 is small.

[0158] In the above-mentioned embodiment and modification, the smartphone 200 generates a display source image and transmits it to the HMD 100, and the HMD 100 cuts out an image of a picture and an image of a process file from the display source image and displays them on the display 160. The smartphone 200 may transmit an image of a picture and an image of a process file, and the HMD 100 may cut out an image of the picture or an image of the process file and display them on the display 160. In addition, the smartphone 200 may transmit text to the HMD 100 instead of an image of the text, and the HMD 100 may convert the text into an image.

[0159] These embodiments and modifications thereof are included in the scope and gist of the invention described in the present specification, etc. and are included in the scope of the invention described in the claims and their equivalents.

Claims

1. A display system comprising a display and a detector for detecting the orientation of the display, characterized in that: The display system comprises: A display source image generating unit that generates a display source image including a first image and a second image associated with the first image; and a visual field image display unit that displays the first image in a first display area of ​​the display and displays the second image in a second display area of ​​the display, If the display mode associated with the first image is the first display mode, the visual field image display unit sets a predetermined fixed area of ​​the display as the first display area, If the display mode is the second display mode, the visual field image display unit determines the first display area according to the orientation of the display, The first image is an image of text, The display source image generating unit generates a plurality of the display source images. The display source image generating unit obtains a difference between text of a first image included in a display source image generated last time and text of the first image included in a display source image generated this time, and generates the display source image by emphasizing the difference when the difference satisfies a predetermined condition.

2. A display system comprising a display and a detector for detecting the orientation of the display, characterized in that: The display system comprises: A display source image generating unit that generates a display source image including a first image and a second image associated with the first image; and a visual field image display unit that displays the first image in a first display area of ​​the display and displays the second image in a second display area of ​​the display, If the display mode associated with the first image is the first display mode, the visual field image display unit sets a predetermined fixed area of ​​the display as the first display area, If the display mode is the second display mode, the visual field image display unit determines the first display area according to the orientation of the display, The first image is an image of text, The text includes multiple partial texts. The display source image generating unit obtains a difference between the plurality of partial texts, and generates the display source image by emphasizing the difference when the difference satisfies a predetermined condition.

3. The display system according to claim 1 or 2, characterized in that: If the display mode is the third display mode, the visual field image display unit displays the display source image on the display.

4. The display system according to claim 3, characterized in that: When the display is a single-eye display, the visual field image display unit sets the display mode to the third display mode and displays the display source image on the display.

5. The display system according to claim 1 or 2, characterized in that: The display source image generating unit calculates the difference using the Levenshtein distance.

6. A display device of a display system, the display system comprising a display device and a portable device, characterized in that: The portable device comprises: A display source image generating unit that generates a display source image including a first image and a second image associated with the first image, and transmits the display source image to the display device; and a display mode control unit that sends a display mode associated with the first image to the display device, The first image is an image of text, The display source image generating unit generates a plurality of the display source images. The display source image generating unit calculates a difference between the text of the first image included in the display source image generated last time and the text of the first image included in the display source image generated this time, and generates the display source image by emphasizing the difference when the difference satisfies a predetermined condition. The display device comprises: monitor; a detector that detects an orientation of the display; as well as a visual field image display unit that displays the first image in a first display area of ​​the display and displays the second image in a second display area of ​​the display, If the display mode is the first display mode, the visual field image display unit sets a predetermined fixed area of ​​the display as the first display area, If the display mode is the second display mode, the visual field image display unit determines the first display area according to the orientation of the display.

7. A display device of a display system, the display system comprising a display device and a portable device, characterized in that: The portable device comprises: A display source image generating unit, which generates a display source image including a first image and a second image associated with the first image, and sends the display source image to the display device; as well as a display mode control unit that sends a display mode associated with the first image to the display device, The first image is an image of text, The text includes multiple partial texts. The display source image generating unit obtains a difference between the plurality of partial texts, and generates the display source image by emphasizing the difference when the difference satisfies a predetermined condition. The display device comprises: monitor; a detector that detects an orientation of the display; as well as a visual field image display unit that displays the first image in a first display area of ​​the display and displays the second image in a second display area of ​​the display, If the display mode is the first display mode, the visual field image display unit sets a predetermined fixed area of ​​the display as the first display area, If the display mode is the second display mode, the visual field image display unit determines the first display area according to the orientation of the display.

Images