Image display method and image display system

Abstract

The present invention provides an image display method and an image display system that can assist humans in performing tasks on objects located outside their field of vision. [Solution] Position information of a predetermined object is obtained from the image captured by an imaging device attached to at least one of the person and the item the person is wearing. If the predetermined object is not within the person's field of view, an auxiliary image B2, which is generated from the image captured based on the position information of the predetermined object, includes the predetermined object, and assists the person in performing the task, is displayed on a terminal attached to at least one of the person and the item the person is wearing.

Description

Technical Field

[0001] The present invention relates to an image display method and an image display system.

Background Art

[0002] When a display device that superimposes and displays a virtual image on the actual scene of an operator's field of view displays a virtual image of a component that conforms to the coordinate system of the actual scene to assist the operator's assembly work, identification information for identifying the operator's work content is extracted from the captured image of the imaging device, information on the assembly position and assembly orientation of the component corresponding to the identification information is obtained, and the virtual image of the component is drawn on the assembly target in the captured image in the assembled state position and orientation. A manufacturing support system is known (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] [[ID=S35]]In the above prior art, when an assembly target exists outside the operator's field of view, there is a problem that the operator's assembly work cannot be supported.

[0005] The problem to be solved by the present invention is to provide an image display method and an image display system that can support work when a human performs work on an object existing outside the field of view.

Means for Solving the Problems

[0006] The present invention solves the above problem by acquiring location information of a predetermined object from an image captured by an imaging device attached to at least one of a person or an item worn by the person, and if the predetermined object is not within the person's field of view, displaying an auxiliary image, which is generated from the image captured based on the location information of the predetermined object, includes the predetermined object, and assists in the work performed by the person, on a terminal attached to at least one of the person or the item worn by the person. [Effects of the Invention]

[0007] According to the present invention, when a person performs an action on an object that is outside their field of vision, the present invention can assist that action. [Brief explanation of the drawing]

[0008] [Figure 1] A block diagram showing one embodiment of the image display system according to the present invention. [Figure 2A] This is a figure (part 1) showing an example of an image captured by the imaging device shown in Figure 1. [Figure 2B] This is Figure (Part 2) showing an example of an image captured by the imaging device shown in Figure 1. [Figure 3] This figure shows an example of an auxiliary image displayed on the terminal in Figure 1. [Figure 4] This figure shows another example of an image captured by the imaging device shown in Figure 1. [Figure 5] This figure shows another example of an auxiliary image displayed on the terminal in Figure 1. [Figure 6] Figure 1 is a flowchart showing an example of the processing procedure in the image display system. [Figure 7A] This is a flowchart (part 1) showing another example of the processing procedure in the image display system shown in Figure 1. [Figure 7B] This is a flowchart (part 2) showing another example of the processing procedure in the image display system shown in Figure 1. [Modes for carrying out the invention]

[0009] Embodiments of the present invention will be described below with reference to the drawings.

[0010] [Image display system configuration] Figure 1 is a block diagram showing one embodiment of the image display system according to the present invention. The image display system of this embodiment displays images (hereinafter also referred to as auxiliary images) to assist a person in performing a task. The attributes of the person are not particularly limited and include vehicle occupants (hereinafter simply referred to as occupants), workers engaged in vehicle assembly work in a factory (hereinafter simply referred to as workers), and mechanics at vehicle dealerships (hereinafter simply referred to as mechanics). Furthermore, the content of the work is not particularly limited and includes the operation of in-vehicle equipment by occupants, the operation of tools by workers, and the operation of maintenance tools by mechanics.

[0011] As shown in Figure 1, the image display system 10 of this embodiment comprises an imaging device 11, a terminal 12, and a display control device 20. These devices are connected to each other via known means such as wired or wireless LAN, and exchange information with each other. Note that the imaging device 11 and terminal 12 do not need to be provided together with the display control device 20, and may be provided in a location separate from the display control device 20. For example, the display control device 20 may be provided on a remote server located away from the imaging device 11 and terminal 12.

[0012] The imaging device 11 is a camera equipped with an image sensor such as a CCD, which images objects around the vehicle and generates an image including the objects. The imaging device 11 may also be an infrared camera, an ultrasonic camera, or the like. The objects include the person performing the work (hereinafter also referred to as the worker) and objects present around the person. Specific examples of objects include switches placed around the occupant, parts being assembled by the worker, and tools used by the mechanic.

[0013] The imaging device 11 is attached to at least one of a person or an item worn by the person (hereinafter also simply referred to as "item"). The item is not particularly limited, but includes, for example, clothing and protective gear worn by workers. Examples of clothing and protective gear include work clothes, hats, safety shoes, helmets, safety glasses, and watches. The imaging device 11 is attached to at least one of the person or the item using hook-and-loop fasteners, clamps, adhesive tape, adhesive, etc. The imaging device 11 may also be installed in a location where the posture of a worker can be detected, such as on the dashboard or roof of a vehicle, or on an assembly line or workbench in an assembly plant.

[0014] Terminal 12 provides information to the worker. Terminal 12 is equipped with a display unit (e.g., a liquid crystal display) that displays auxiliary images and, like the imaging device 11, is attached to at least one of the person or the object. Terminal 12 may be a portable device such as a tablet or smartphone, or a wearable device such as a smartwatch.

[0015] Terminal 12 may be a glasses-type wearable device (so-called smart glasses) that can be worn on a person's head. Smart glasses are a type of wearable device using a head-mounted display (HMD) system, and for example, they include a display unit and a headband for attaching the smart glasses to a person's head. The display is a transmissive holographic lens that transmits light from real objects like the lens of ordinary glasses, and can also display images (e.g., stereoscopic images) in the worker's field of view using holograms. Therefore, the display unit of the smart glasses can display auxiliary images using holograms superimposed on the real scene within the human field of view. The real scene within the human field of view refers to the parts of the objects around the worker that are included in the worker's field of view, and is the scenery that a worker not wearing smart glasses would actually see.

[0016] In this embodiment, the imaging device 11 is attached to at least one of the worker and the article worn by the worker, and is, for example, attached to the worker's head, the worker's shoulder, the helmet worn by the worker, or the like. Further, the imaging device 11 is, for example, a wide-angle camera equipped with a wide-angle lens or a fisheye lens, has an angle of view wider than the field of view of the worker, and can image a range wider than the field of view of the worker. Furthermore, the terminal 12 is attached to at least one of the worker and the article worn by the worker so that the worker can visually recognize the information displayed on the display unit. The display unit of the terminal 12 attached to the worker may be arranged to always exist in the line-of-sight direction of the worker, or may move in accordance with the line-of-sight direction of the worker. Note that the imaging device 11 and the terminal 12 may be integrated. For example, when the terminal 12 is smart glasses, the imaging device 11 is provided on the visor or headband of the smart glasses.

[0017] The display control device 20 controls the devices constituting the image display system 10 to cooperate with each other, and causes an auxiliary image to be displayed on the display unit of the terminal 12. The display control device 20 acquires image information from the imaging device 11 at a predetermined time interval (for example, every 0.1 to 1 millisecond). The image information is information regarding the image captured by the imaging device 11. Further, the display control device 20 outputs the generated auxiliary image to the terminal 12.

[0018] The display control device 20 is, for example, a computer, and includes a CPU (Central Processing Unit) as a processor, a ROM (Read Only Memory) in which a program is stored, and a RAM (Random Access Memory) that functions as an accessible storage device. The CPU executes the program stored in the ROM and is an operation circuit for realizing the functions of the display control device 20. Note that an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), or the like may be used instead of the CPU or together with the CPU.

[0019] [Functions of the Display Control Device] The ROM of the display control device 20 stores a program for displaying an auxiliary image on the display unit of the terminal 12, and the CPU of the display control device 20 executes this program to display the auxiliary image. Figure 1 shows, for convenience, an extracted functional block for displaying the auxiliary image, consisting of an acquisition unit 21, a generation unit 22, and a display unit 23.

[0020] The acquisition unit 21 acquires an image from the imaging device 11 and determines whether or not a predetermined object is included in the acquired image. Hereinafter, the image acquired from the imaging device 11 will also simply be referred to as the image. A predetermined object is an object that exists around the worker to whom the imaging device 11 is attached and is related to the worker's work. Examples include switches operated by the crew, computers used by the worker, parts of the vehicle that are maintained by mechanics, and the worker's hands. The predetermined objects are set in advance according to the type of work the worker is doing.

[0021] The imaging device 11 is, for example, attached to the worker's head, shoulder, or helmet, and captures images in the direction of the worker's line of sight, generating images from the worker's first-person perspective. The imaging device 11 may continuously generate a time-series of images linked to the worker's movements at a predetermined frame rate (e.g., 1 to 120 fps) depending on the processing capacity of the imaging device 11. The acquisition unit 21 acquires image information related to these images from the imaging device 11. When the acquisition unit 21 has acquired a time-series of images from the imaging device 11, it may detect predetermined objects included in the captured images at arbitrary frame intervals (e.g., every 3 to 10 frames) depending on the processing capacity of the display control device 20. The imaging process by the imaging device 11 starts, for example, when the worker operates a switch (not shown) provided on the imaging device 11, and ends when the worker operates the switch again.

[0022] If it is determined that the captured image does not contain the specified object, the acquisition unit 21 acquires the captured image again from the imaging device 11. On the other hand, if it is determined that the captured image contains the specified object, the acquisition unit 21 acquires the position information of the specified object included in the captured image. The position information of the specified object includes information such as the relative position of the specified object with respect to the imaging device 11, the relative position of the specified object with respect to stationary objects such as building columns and floors, and the relative position of the specified object with respect to parts of the worker's body (nose, shoulder, etc.).

[0023] The acquisition unit 21 calculates, for example, the position of a predetermined object relative to the imaging device 11 and defines a rectangular area in the captured image that includes the predetermined object. The acquisition unit 21 may detect the predetermined object using a trained model (e.g., a convolutional neural network) that has been trained to detect the predetermined object, or it may use a pattern matching method to detect the predetermined object if the predetermined object has a distinctive color or shape. Furthermore, if the acquisition unit 21 detects a part of a worker's body as the predetermined object, it may use a known skeletal estimation method.

[0024] The generation unit 22 generates an auxiliary image containing a predetermined object from the image captured by the imaging device 11 based on the position information acquired by the acquisition unit 21. The generation unit 22 sets a range to be cropped from the image as an auxiliary image (hereinafter also referred to as the cropping range) based on the size and position of the predetermined object, and enlarges or reduces the cropped auxiliary image to match the size of the display unit of the terminal 12. The range to be cropped from the image is set so that the size of the predetermined object displayed on the display unit of the terminal 12 is large enough for the operator to recognize it appropriately. The range to be cropped from the image may be rectangular, circular, or elliptical. When enlarging or reducing the cropped auxiliary image, the cropped auxiliary image may be enlarged or reduced along one direction, or along multiple directions. Alternatively, the cropped auxiliary image may be enlarged along one direction and reduced along other directions.

[0025] The generation unit 22 may adjust the size of the cropped auxiliary image according to a predetermined rule. For example, if a rule is set to crop an area three times the size of the detected predetermined object from the captured image, and the detected predetermined object occupies 100 x 200 pixels in the captured image, the generation unit 22 will crop a 300 x 600 pixel area including the detected predetermined object from the captured image. The area to be cropped from the captured image as an auxiliary image is set, for example, so that the predetermined object is located in the center of the cropped auxiliary image. Furthermore, if there are multiple predetermined objects, the image cropping process may be performed for each object, and the size adjustment process (enlargement or reduction) may be performed for each cropped image.

[0026] The display unit 23 determines whether the location of a predetermined object is within the operator's field of view. The operator's field of view relative to the mounted imaging device 11 (i.e., the operator's field of view relative to the imaging device 11) is preset. If it is determined that the predetermined object is within the operator's field of view, the display unit 23 does not display the auxiliary image. On the other hand, if it is determined that the predetermined object is not within the operator's field of view, the display unit 23 displays the auxiliary image superimposed on the actual scene on the display unit of the terminal 12. The display unit 23 displays the auxiliary image so that both the actual scene and the auxiliary image are visible to the operator. That is, it does not display an auxiliary image that covers the entire field of view of the operator, but displays an auxiliary image on the display unit that is sized to occupy a predetermined percentage (for example, 10-40%) of the operator's field of view.

[0027] Figures 2A and 2B show examples of images captured by the imaging device 11. In the examples shown in Figures 2A and 2B, a wide-angle camera equipped with a fisheye lens is attached to the head of worker P, and images A1 and A2 are captured by this wide-angle camera. Images A1 and A2 are two separate images of the same image captured at the same time. Image A1 in Figure 2A corresponds to the imaging range of the lower side of the wide-angle camera (towards the feet of worker P), and image A2 in Figure 2B corresponds to the imaging range of the upper side of the wide-angle camera (towards the head of worker P).

[0028] In the examples shown in Figures 2A and 2B, processing by the acquisition unit 21 begins when the operator operates the switch on the wide-angle camera. When the switch operation is detected, the acquisition unit 21 acquires information about the time-series images from the wide-angle camera and detects a predetermined object every 5 frames. If the predetermined object is the computer X used by operator P, the acquisition unit 21 detects the computer X by pattern matching.

[0029] The generation unit 22 sets a cropping area B1 that includes computer X and generates an auxiliary image corresponding to the cropping area B1. The display unit 23 displays the auxiliary image on the display unit of terminal 12 when it is determined that computer X is not in the field of view of worker P. For example, if worker P is wearing smart glasses, the display unit 23 displays the auxiliary image D1 on the smart glasses display C1 shown in Figure 3. The generation unit 22 enlarges the auxiliary image corresponding to the cropping area B1 vertically and horizontally to the size of the auxiliary image D1 shown in Figure 3. The size of the auxiliary image D1 is, for example, preset relative to the size of the display C1. The display unit 23 terminates the display of the auxiliary image D1 when it is determined that worker P has operated the wide-angle camera switch again.

[0030] The designated object may be the worker's hand. In this case, the generation unit 22 generates an auxiliary image including the worker's hand, and the display unit 23 displays the auxiliary image including the worker's hand on the display unit of the terminal 12. For example, in the example shown in Figure 2B, the acquisition unit 21 detects the worker P's right hand Pa and left hand Pb from the captured image A2, the generation unit 22 generates an auxiliary image including the worker P's right hand Pa and left hand Pb, and the display unit 23 displays the auxiliary image including the worker P's right hand Pa and left hand Pb on the display unit of the terminal 12.

[0031] When the generation unit 22 generates an auxiliary image by cropping a portion of the captured image, if the worker's left and right hands are included in the captured image, it may set the cropping range from the captured image to the auxiliary image according to the distance between the left and right hands. For example, the generation unit 22 determines whether the worker's hands are included in the captured image, and if it determines that the worker's hands are included in the captured image, it sets the cropping range based on the position of the worker's hands and the size of the worker's hands in the captured image. For example, the generation unit 22 sets a rectangular cropping range where the worker's left hand is located at the lower left corner and the right hand is located at the lower right corner.

[0032] The generation unit 22 may set the cutting range so that the object located between the worker's right and left hands or at the fingertips is at its center. The generation unit 22 may also set the cutting range wider the greater the distance between the worker's left and right hands, and narrower the cutting range the closer the distance between the worker's left and right hands is. If the worker's right and left hands are separated by a certain distance (for example, 1.5 to 3 m) or more, the generation unit 22 may set the cutting range to include the hand that is at a higher position.

[0033] The display unit 23 may determine from the captured image whether the position of the worker's hand is higher than the position of the worker's eye. If it is determined that the position of the worker's hand is higher than the position of the worker's eye, the display unit 23 displays an auxiliary image on the display unit of the terminal 12. On the other hand, if it is determined that the position of the worker's hand is lower than or equal to the position of the worker's eye, the display unit 23 does not display an auxiliary image on the display unit of the terminal 12. The position of the worker's eye (line of sight) is set in advance, for example, as the relative position of the eye to the imaging device 11 (relative position of the line of sight).

[0034] If the generation unit 22 detects a worker's arm from the captured image, it calculates the position of the worker's arm and the angle between the worker's arm and torso. The display unit 23 displays an auxiliary image on the display of the terminal 12 if the position of the worker's arm is higher than the position of the worker's shoulder, and does not display an auxiliary image on the display of the terminal 12 if the position of the worker's arm is lower than or equal to the position of the worker's shoulder. The generation unit 22 may also determine that the position of the worker's arm is higher than the position of the worker's shoulder if the position of the arm falls within a predetermined range. The predetermined range can be set to an appropriate range within which the position of the worker's arm and the position of the shoulder can be accurately compared.

[0035] The display unit 23 may detect the worker's hand movements using a known skeletal estimation method or the like, and determine whether the worker's hand movements are predetermined movements. If it is determined that the worker's hand movements are predetermined movements, the display unit 23 stops displaying the auxiliary image. On the other hand, if it is determined that the worker's hand movements are not predetermined movements, the display unit 23 does not stop (or continues to display) the auxiliary image. The predetermined movements are not particularly limited, but examples include extending all the fingers of the hand, crossing the right and left hands, and swinging one hand from side to side.

[0036] If the display unit 23 detects a worker's arm from the captured image and determines that the worker's arm has moved relatively large from side to side (for example, 1 to 1.5 m from side to side) for a certain number of frames (for example, 10 to 20 frames), it may assume that the worker has issued an instruction to stop displaying the auxiliary image and may hide the auxiliary image. The display unit 23 may hide the auxiliary image, for example, if the arm movement is not a movement specified in a pre-set work procedure.

[0037] The display unit 23 may perform a process to cut out or fade out the auxiliary image when hiding the auxiliary image. The display unit 23 may also perform a process to move the auxiliary image toward the center of the display unit of the terminal 12 and fade out the auxiliary image when stopping the display of the auxiliary image. For example, in the example shown in Figure 3, when hiding the auxiliary image D1, the display unit 23 moves the auxiliary image D1 toward the center of the display C1 and gradually increases the transparency of the auxiliary image D1 to fade it out.

[0038] The display unit 23 may change the display position of the auxiliary image according to the position of the worker's hand if the captured image includes the worker's hand. The display unit 23 may also change the display position of the auxiliary image according to the position of the worker's arm if the captured image includes the worker's arm. For example, if the position of the hand or arm is above the reference position, the display unit 23 displays the auxiliary image above the display unit of the terminal 12 (above the center in the height direction), and if the position of the hand or arm is lowered, it lowers the display position of the auxiliary image in conjunction with the movement of the hand or arm. The reference position is, for example, the position of the worker's head, the position of the worker's shoulder, or the position of the helmet worn by the worker. Furthermore, if the position of the worker's hand or arm is below the reference position, the display unit 23 may stop displaying the auxiliary image.

[0039] Figure 4 shows another example of an image captured by the imaging device 11. In the example shown in Figure 4, a wide-angle camera equipped with a wide-angle lens is attached to the mechanic's head, and this wide-angle camera captures an image A3 including the vehicle V being serviced.

[0040] In the example shown in Figure 4, the acquisition unit 21 acquires information about a time-series image from the wide-angle camera and detects a predetermined object every 5 frames. If the predetermined object is the mechanic's hand, the acquisition unit 21 detects the mechanic's right hand Pc and left hand Pd using a known skeletal estimation method. The generation unit 22 sets a cropping range B2 that includes the mechanic's right hand Pc and left hand Pd and generates an auxiliary image corresponding to the cropping range B2. The display unit 23 displays the auxiliary image on the display unit of the terminal 12 if it is determined that the right hand Pc and left hand Pd are not in the mechanic's field of view.

[0041] For example, if the mechanic is wearing smart glasses, the display unit 23 displays the auxiliary image D2 on the smart glasses' display C2 as shown in Figure 5. The generation unit 22 enlarges the auxiliary image corresponding to the cropping area B2 vertically and horizontally to the size of the auxiliary image D2 shown in Figure 5. The size of the auxiliary image D2 is, for example, pre-set relative to the size of the display C2. The display position of the auxiliary image D2 may be set to a position that does not overlap with the actual view of the vehicle V being serviced. Furthermore, if the position of the mechanic's right hand Pc and left hand Pd is lowered, the display unit 23 lowers the display position of the auxiliary image D2 in conjunction with the movement of the right hand Pc and left hand Pd, and stops displaying the auxiliary image D2 when the position of the right hand Pc and left hand Pd is below the position of the mechanic's shoulders.

[0042] If the display unit 23 determines that at least one of the worker's right and left hands is positioned higher than the worker's eye level before a predetermined time has elapsed since the display of the auxiliary image was stopped, it may display the auxiliary image of the same size as when it was stopped on the display unit of the terminal 12. The predetermined time can be set to an appropriate amount of time within a range that does not cause discomfort to the worker when the auxiliary image is displayed again, for example, 1 to 3 seconds.

[0043] For example, in the example shown in Figure 5, if it is determined that the mechanic's right hand Pc is positioned higher than the mechanic's eye level within 3 seconds of stopping the display of auxiliary image D2, the display unit 23 redisplays auxiliary image D2 at the same position on display C2. Even if the right hand Pc is no longer included in the captured image after the display unit 23 redisplays auxiliary image D2, the display unit 23 will continue to display auxiliary image D2 for a certain period of time (e.g., 3 to 5 seconds), and then hide it after that period of time has elapsed. Alternatively, even if the right hand Pc is no longer included in the captured image after the display unit 23 redisplays auxiliary image D2, the display unit 23 will continue to display auxiliary image D2 until it is determined that the mechanic's hand has made a predetermined movement.

[0044] [Processing in image display systems] Referring to Figures 6 and 7A-7B, the procedure for information processing by the display control device 20 will be explained below. The processing described below is executed by the processor (CPU) of the display control device 20 at predetermined time intervals (for example, every 0.1 to 1 millisecond).

[0045] Figure 6 is a flowchart showing an example of a processing procedure in the image display system 10.

[0046] First, in step S1, the display control device 20 acquires an image from the imaging device 11 attached to the worker. In the following step S2, it uses a method such as pattern matching to detect a predetermined object included in the image for each arbitrary frame. In step S3, the display control device 20 determines whether or not the image contains the predetermined object based on the detection result of the predetermined object. If it is determined that the image does not contain the predetermined object, the process proceeds to step S1. On the other hand, if it is determined that the image contains the predetermined object, the process proceeds to step S4.

[0047] In step S4, the display control device 20 sets a range to be cropped from the captured image as an auxiliary image based on the position information of a predetermined object, and in the following step S5, it enlarges or reduces the cropped auxiliary image to match the size of the display unit of the terminal 12. In the following step S6, the display control device 20 displays the auxiliary image superimposed on the actual scene on the display unit of the terminal 12 worn by the worker.

[0048] In step S7, the display control device 20 determines whether or not an instruction to stop displaying the auxiliary image has been received from the operator. If it is determined that an instruction to stop displaying the auxiliary image has been received from the operator, the process ends. On the other hand, if it is determined that no instruction to stop displaying the auxiliary image has been received from the operator, the process proceeds to step S1.

[0049] Next, Figures 7A to 7B are flowcharts showing other examples of processing procedures in the image display system 10.

[0050] First, in step S11 of Figure 7A, the display control device 20 acquires an image from the imaging device 11 attached to the worker. In the following step S12, it uses a method such as pattern matching to detect the worker's arms included in the image for each arbitrary frame. In the following step S13, it detects the worker's hands included in the image for each arbitrary frame. In step S14, the display control device 20 determines, based on the detection results, whether or not the worker's hands are included in the image. If it is determined that the worker's hands are included in the image, the process proceeds to step S15.

[0051] In step S15, the display control device 20 determines whether the worker's arm is above shoulder level and whether both of the worker's hands are above the worker's line of sight. If it is determined that the worker's arm is not above shoulder level and both of the worker's hands are not above the worker's line of sight, the process proceeds to step S11. On the other hand, if it is determined that the worker's arm is above shoulder level and both of the worker's hands are above the worker's line of sight, the process proceeds to step S16.

[0052] In step S16, the display control device 20 sets a cropping area in the captured image that includes both of the worker's hands, and in the following step S17, it sets the display position of the auxiliary image according to the position of the worker's hands based on the position information of the worker's hands. In step S18, the display control device 20 enlarges or reduces the cropped auxiliary image to match the size of the smart glasses' display, and in the following step S19, it displays the auxiliary image superimposed on the real scene on the smart glasses' display.

[0053] In step S20, the display control device 20 determines whether the worker's hand movement is a predetermined movement using a method such as pattern matching. If it is determined that the worker's hand movement is a predetermined movement, the process proceeds to step S22. Conversely, if it is determined that the worker's hand movement is not a predetermined movement, the process proceeds to step S21.

[0054] In step S21, the display control device 20 determines whether or not the operator has given an instruction to stop displaying the auxiliary image. If it is determined that the operator has given an instruction to stop displaying the auxiliary image, the process proceeds to step S22, where the display control device 20 stops displaying the auxiliary image and terminates the process. On the other hand, if it is determined that the operator has not given an instruction to stop displaying the auxiliary image, the process proceeds to step S11.

[0055] On the other hand, if it is determined in step S14 that the captured image does not include both the worker's right and left hands, the process proceeds to step S31 in Figure 7B. In step S31, the display control device 20 uses methods such as pattern matching to determine whether or not one of the worker's hands is included in the captured image acquired from the imaging device 11. If it is determined that neither the worker's right nor left hand is included in the captured image, the process proceeds to step S11 in Figure 7A. Conversely, if it is determined that either the worker's right or left hand is included in the captured image, the process proceeds to step S32 in Figure 7B.

[0056] In step S32, the display control device 20 determines whether or not an auxiliary image was displayed on the display unit of the terminal 12 immediately before (i.e., whether or not the period during which the auxiliary image was not displayed is longer than a predetermined period). The predetermined period can be set to an appropriate time within a range that does not cause discomfort to the operator when the auxiliary image is displayed again, for example, 1 to 3 seconds. If it is determined that an auxiliary image was displayed on the display unit of the terminal 12 immediately before (i.e., the period during which the auxiliary image was not displayed is shorter than the predetermined period), the process proceeds to step S33, and the display control device 20 redisplays an auxiliary image with the same scale and field of view as the auxiliary image that was displayed immediately before.

[0057] In step S34, the display control device 20 determines whether a certain amount of time has elapsed since the start of displaying the auxiliary image. If it is determined that a certain amount of time has not elapsed since the start of displaying the auxiliary image, the process proceeds to step S36, and the display control device 20 continues to display the auxiliary image. After that, the process proceeds to step S34. On the other hand, if it is determined that a certain amount of time has elapsed since the start of displaying the auxiliary image, the process proceeds to step S35, and the display control device 20 stops displaying the auxiliary image. After that, the process proceeds to step S11 in Figure 7A.

[0058] On the other hand, if it is determined in step S32 that the period during which the auxiliary image is not displayed is longer than a predetermined period, the process proceeds to step S37. In step S37, the display control device 20 sets a cropping range in the captured image that includes one of the worker's hands, and in the subsequent step S38, it sets the display position of the auxiliary image according to the position of one of the worker's hands based on the position information of one of the worker's hands. After that, the process proceeds to step S18 in Figure 7A.

[0059] [Embodiments of the present invention] According to this embodiment, an image display method and image display system 10 are provided that acquire positional information of a predetermined object from an image captured by an imaging device 11 attached to at least one of a person and an item worn by the person, and if the object is not within the person's field of view, an auxiliary image generated from the image captured based on the positional information, including the object, and assisting the person's work is displayed on a terminal 12 attached to at least one of the person and the item. This allows a person to perform work on an object that is outside their field of view, and also allows the worker to check the condition around the predetermined object without moving their head or body significantly.

[0060] In the image display method and image display system 10 of this embodiment, the terminal 12 is a glasses-type wearable terminal that can be worn on the head of a person, and the display control device 20 displays the auxiliary image superimposed on the actual scene within the person's field of view on the display unit of the wearable terminal. This allows the worker to check the condition of the area around a predetermined object while ensuring their field of view.

[0061] In the image display method and image display system 10 of this embodiment, the object is the human hand, and the auxiliary image including the hand is displayed on the display unit of the terminal 12. This allows the auxiliary image to be displayed even when the worker's work object cannot be detected, or when there are multiple work objects for the worker.

[0062] In the image display method and image display system 10 of this embodiment, when generating the auxiliary image by cropping a portion of the captured image, if the left and right hands of the person are included in the captured image, the range to be cropped from the captured image as the auxiliary image is set according to the distance between the left and right hands. This makes it possible to generate an auxiliary image in which the object of the work is centered.

[0063] In the image display method and image display system 10 of this embodiment, it is determined from the captured image whether the position of the human hand is higher than the position of the human eye. If it is determined that the position of the hand is higher than the position of the eye, the auxiliary image is displayed on the display unit of the terminal 12. This allows the auxiliary image to be displayed when the worker needs to see it.

[0064] In the image display method and image display system 10 of this embodiment, it is determined whether the movement of the human hand is a predetermined movement, and if it is determined that the hand movement is the predetermined movement, the display of the auxiliary image is stopped. This allows the auxiliary image to be hidden when the worker does not need to see it.

[0065] In the image display method and image display system 10 of this embodiment, when the display of the auxiliary image is to be stopped, the auxiliary image is moved toward the center of the display unit of the terminal 12, and the process of fading out the auxiliary image is executed. This prevents the auxiliary image from suddenly disappearing.

[0066] In the image display method and image display system 10 of this embodiment, if the captured image includes a human hand, the display position of the auxiliary image is changed according to the position of the hand. This prevents the auxiliary image from being displayed in a position that does not match the movement of the worker's hand, thus preventing the worker from feeling uneasy (or uncomfortable).

[0067] In the image display method and image display system 10 of this embodiment, if it is determined that at least one of the person's right hand and left hand is positioned higher than the person's eyes before a predetermined time has elapsed since the display of the auxiliary image was stopped, the auxiliary image is displayed on the display unit of the terminal 12 at the same size as when the display was stopped. This allows the auxiliary image to be quickly redisplayed when the worker needs to see it. [Explanation of Symbols]

[0068] 10...Image display system, 11...Imaging device, 12...Terminal, 20...Display control device, 21...Acquisition unit, 22...Generation unit, 23...Display unit, A1, A2, A3...Imagine captured image, B1, B2...Cropping area, C1, C2...Display, D1, D2...Auxiliary image, P...Worker, Pa, Pc...Right hand, Pb, Pd...Left hand, V...Vehicle, X...Computer

Claims

1. In an image display method executed by a display control device, The aforementioned display control device is Position information of a predetermined object is obtained from the image captured by an imaging device attached to at least one of a human being and an item worn by the human being. If the object is not within the human's field of vision, an image display method that displays an auxiliary image, generated from the captured image based on the location information, which includes the object and assists in the work performed by the human, on a terminal attached to at least one of the human and the article.

2. The aforementioned terminal is a glasses-type wearable terminal that can be worn on the head of a person, The image display method according to claim 1, wherein the display control device displays the auxiliary image superimposed on the actual scene within the human field of view on the display unit of the wearable terminal.

3. The aforementioned object is the human hand, The image display method according to claim 1 or 2, wherein the display control device displays the auxiliary image including the hand on the display unit of the terminal.

4. The image display method according to claim 1 or 2, wherein when the display control device generates the auxiliary image by cropping a part of the captured image, if the left hand and right hand of the person are included in the captured image, the range to be cropped from the captured image as the auxiliary image is set according to the distance between the left hand and the right hand.

5. The aforementioned display control device is From the captured image, it is determined whether the position of the human hand is higher than the position of the human eye. The image display method according to claim 1 or 2, wherein if it is determined that the position of the hand is higher than the position of the eye, the auxiliary image is displayed on the display unit of the terminal.

6. The aforementioned display control device is Determine whether the aforementioned human hand movement is a predetermined movement, The image display method according to claim 1 or 2, wherein the display of the auxiliary image is stopped if it is determined that the hand movement is the predetermined movement.

7. The image display method according to claim 6, wherein when the display control device stops displaying the auxiliary image, it moves the auxiliary image toward the center of the display unit of the terminal and performs a process to fade out the auxiliary image.

8. The image display method according to claim 1 or 2, wherein the display control device changes the display position of the auxiliary image according to the position of the hand if the captured image includes the hand.

9. The image display method according to claim 6, wherein if the display control device determines that at least one of the person's right hand and left hand is positioned higher than the person's eyes before a predetermined time has elapsed since the display of the auxiliary image was stopped, the auxiliary image of the same size as when the display was stopped is displayed on the display unit of the terminal.

10. An acquisition unit that acquires location information of a predetermined object from an image captured by an imaging device attached to at least one of a human being and an item worn by the human being, If the object is not within the human's field of vision, the image display system includes a display unit attached to at least one of the human and the article, which displays an auxiliary image generated from the captured image based on the position information, including the object, and assisting the work performed by the human.

Images