Microscope system, method for operating microscope system, and program

Abstract

This microscope system comprises: a microscope optical system that includes an eyepiece lens and forms an intermediate image of an observation object on an object side of the eyepiece lens; a sensing device (500) that is different from a microscope camera; a control unit that outputs an auxiliary image including work assistance information which is determined on the basis of a work condition with respect to the observation object, said work condition being specified from work information acquired by the sensing device (500), and reference information corresponding to the work condition, said reference information being acquired from a storage unit; and a superimposition device that superimposes the auxiliary image outputted by the control unit on the image plane on which the intermediate image is formed.

Description

Microscope system, microscope system operation method, and program

[0001] The disclosure of the present specification relates to a microscope system, a method of operating the microscope system, and a program.

[0002] Even today, with the increasing automation of work using robots and other tools, there are still many products that require manual assembly, such as medical equipment. The assembly of precision equipment like medical equipment involves many delicate tasks, so it is often done under a microscope, often using a stereo microscope that allows the object to be viewed in three dimensions with both eyes.

[0003] However, in order to check the instructions while performing assembly work while observing an object through a stereo microscope, the worker must temporarily take their eyes off the stereo microscope's eyepiece and shift their gaze to a display or other device displaying the instructions. After checking, they must then look through the eyepiece again to continue the assembly work, which makes it difficult to improve work efficiency.

[0004] A technique related to such technical issues is described, for example, in Patent Document 1. The microscope system described in Patent Document 1 projects information onto an intermediate image observed through an eyepiece, allowing an operator to receive information without taking their eyes off the eyepiece.

[0005] International Publication No. 2020 / 066041

[0006] The information required by a worker may change depending on the work status. Therefore, it is desirable to provide information to the worker according to the work status. In light of the above circumstances, an object of one aspect of the present invention is to provide a technology for providing more appropriate information to a worker looking through an eyepiece.

[0007] A microscope system according to one aspect of the present invention comprises a microscope optical system including an eyepiece that forms an intermediate image of an observed object on the object side of the eyepiece; a sensing device different from a microscope camera; a control unit that outputs an auxiliary image including work support information determined based on a work state for the observed object identified from work information acquired by the sensing device and reference information corresponding to the work state acquired from a memory unit; and a superimposition device that superimposes the auxiliary image output by the control unit on an image plane on which the intermediate image is formed.

[0008] An operating method according to one aspect of the present invention is a method for operating a microscope system equipped with a microscope optical system that forms an intermediate image of an observed object on the object side of an eyepiece, in which work support information is determined based on a work state for the observed object identified from work information acquired by a sensing device other than a microscope camera and reference information corresponding to the work state acquired from a memory unit, and an auxiliary image including the determined work support information is output to a superimposition device that superimposes an input image on an image plane on which the intermediate image is formed.

[0009] A program according to one aspect of the present invention causes a computer of a microscope system having a microscope optical system that forms an intermediate image of an observed object on the object side of an eyepiece to determine work support information based on a work state for the observed object identified from work information acquired by a sensing device other than a microscope camera and reference information corresponding to the work state acquired from a memory unit, and to execute a process of outputting an auxiliary image including the determined work support information to a superimposing device that superimposes an input image on an image plane on which the intermediate image is formed.

[0010] According to the above aspect, more appropriate information can be provided to the worker looking through the eyepiece.

[0011] 1 is a diagram illustrating a microscope system according to an embodiment of the present invention. FIG. 1 is a diagram illustrating an example of the configuration of an optical system included in the microscope system. FIG. 2 is a diagram illustrating an example of the hardware configuration of a computer for realizing a control device. FIG. 3 is a diagram illustrating an example of a flowchart of control processing of the microscope system. FIG. 4 is a diagram illustrating an operator using the microscope system. FIG. 5 is an example of an image acquired by a microscope camera. FIG. 6 is an example of an image acquired by a macro camera. FIG. 7 is an example of a table storing reference information. FIG. 8 is an example of a superimposed image. FIG. 9 is another example of a table storing reference information. FIG. 10 is another example of a superimposed image. FIG. 11 is another example of an image acquired by a microscope camera. FIG. 12 is another example of an image acquired by a macro camera. FIG. 13 is yet another example of a table storing reference information. FIG. 14 is yet another example of a superimposed image. FIG. 15 is yet another example of an image acquired by a macro camera. FIG. 16 is yet another example of a table storing reference information. FIG. 17 is yet another example of a superimposed image. FIG. 18 is a diagram illustrating a series of tasks performed by multiple operators in separate steps. FIG. 19 is yet another example of a table storing reference information. FIG. 19 is yet another example of a superimposed image. FIG. 19 is a diagram illustrating a temperature sensor. FIG. 19 is yet another example of a table storing reference information. FIG. 19 is yet another example of a superimposed image. 10 is yet another example of a table storing reference information. FIG. 10 is yet another example of a superimposed image. FIG. 10 is yet another example of a table storing reference information. FIG. 10 is yet another example of a superimposed image. FIG. 10 is yet another example of a table storing reference information. FIG. 10 is yet another example of a superimposed image. FIG. 10 is yet another example of a table storing reference information. FIG. 10 is yet another example of a superimposed image. FIG. 10 is yet another example of a table storing reference information. FIG. 10 is yet another example of a superimposed image. FIG. 10 is yet another example of a table storing reference information. FIG. 10 is yet another example of a superimposed image. FIG. 10 is yet another example of a table storing reference information.10 is yet another example of a superimposed image; FIG. 11 is yet another example of a table storing reference information; FIG. 12 is yet another example of a superimposed image; FIG. 13 is a diagram showing an example of a temperature transition; FIG. 14 is a diagram showing another example of a temperature transition; FIG. 15 is yet another example of a superimposed image; FIG. 16 is a diagram explaining a pressure sensor; FIG. 17 is yet another example of a table storing reference information; FIG. 18 is yet another example of a superimposed image.

[0012] FIG. 1 is a diagram illustrating a microscope system according to an embodiment of the present invention. FIG. 2 is a diagram illustrating the configuration of an optical system included in the microscope system. FIG. 3 is a diagram illustrating the hardware configuration of a computer for realizing a control device. The microscope system 1 shown in FIG. 1 uses work information acquired by a sensing device other than a microscope camera to provide a user with appropriate information at the necessary timing while the user is working under a microscope while looking through an eyepiece 106. The configuration of the microscope system 1 will be described with reference to FIGS. 1 to 3.

[0013] The microscope system 1 includes a microscope 100 , a control device 200 , a display device 300 , a plurality of input devices 400 (a mouse 401 , a keyboard 402 , a foot switch 403 , and a barcode reader 404 ), and a web camera 500 .

[0014] The microscope 100 is a stereo microscope capable of stereoscopically viewing a sample, and includes a microscope optical system 110 shown in FIG. 2. The microscope optical system 110 is an optical system for a stereo microscope that includes eyepieces 106 (eyepieces 106a and 106b). A user can observe an intermediate image (hereinafter referred to as an optical image) of the sample, which is the observation object formed by the microscope optical system 110 on the object side of the eyepieces 106, with both eyes via the eyepieces 106, thereby enabling stereoscopic observation of the sample. For this reason, the microscope 100 is suitable for applications such as precision instrument assembly work.

[0015] The microscope 100 is equipped with zoom lenses 102 (zoom lenses 102a and 102b) that can be operated with a zoom handle 130. By operating the zoom handle 130, the observation magnification can be changed while continuing to observe a sample by looking through the eyepiece 106.

[0016] The microscope 100 is equipped with a focusing handle 140. By operating the focusing handle 140, the distance between the sample and the objective lens 101 can be changed to bring the sample into focus.

[0017] The microscope 100 is equipped with an imaging device 112 that captures an image of a sample and acquires an image of the sample. An eyepiece tube 120 to which the eyepiece 106 is attached is a triocular tube, and the imaging device 112 is attached to the eyepiece tube 120. The imaging device 112 is provided with a two-dimensional image sensor. The image sensor is not particularly limited, but may be, for example, a CCD image sensor or a CMOS image sensor.

[0018] The imaging device 112 is a microscope camera that acquires an image of a sample magnified by the microscope 100, and the image acquired by the imaging device 112 is output to the control device 200. The image acquired by the imaging device 112 may also be output directly to the display device 300. Note that, hereinafter, an image acquired by the imaging device 112, which is a microscope camera, will be referred to as a microscope image.

[0019] 2, light split by a beam splitter 103a, such as a half mirror, from one of the left and right optical paths of the microscope optical system 110 enters the imaging device 112 via an imaging lens 111. Note that an ND prism 103b is provided in the other optical path to compensate for the optical path length between the left and right optical paths caused by the beam splitter 103a and to suppress the difference in light intensity.

[0020] The microscope 100 includes a projector 113 that projects an auxiliary image onto an image plane where the imaging lenses 105 (imaging lenses 105a and 105b) form an optical image. The projector 113 is a superimposing device that superimposes an input image onto the image plane where the optical image is formed, and projects and superimposes an auxiliary image output by the control device 200 onto the image plane. The type of projector 113 is not particularly limited. The projector 113 may be configured using a display device such as a liquid crystal device or a digital mirror device, for example.

[0021] The projector 113 is provided inside the eyepiece tube 120. Light from the projector 113 is guided to the left and right optical paths of the microscope optical system 110 via a projection lens 114 and a plurality of beam splitters (beam splitter 115, beam splitter 104a, beam splitter 104b).

[0022] 1, the eyepiece tube 120 is provided with an operation unit 121. By operating the operation unit 121, the user can switch the projector 113 on and off and instruct the projector to start or stop superimposing the auxiliary image on the image plane.

[0023] The control device 200 is a device that controls the microscope 100. The control device 200 is, for example, a control unit that generates the above-mentioned auxiliary image and outputs it to the microscope 100 (projector 113). The process of outputting the auxiliary image will be described in detail later.

[0024] 3, the hardware configuration of the computer 200a that realizes the control device 200 includes a processor 201, a memory 202, a storage device 203, a reading device 204, a communication interface 206, and an input / output interface 207. The processor 201, the memory 202, the storage device 203, the reading device 204, the communication interface 206, and the input / output interface 207 are connected to each other via a bus 208, for example.

[0025] The processor 201 may be, for example, a single processor, a multiprocessor, or a multi-core processor. The processor 201 reads and executes programs stored in the storage device 203 to perform various control processes, such as the process of outputting the auxiliary image shown in FIG. 4 .

[0026] The memory 202 is, for example, a semiconductor memory and may include a RAM area and a ROM area. The storage device 203 is, for example, a semiconductor memory such as a hard disk or a flash memory, or an external storage device.

[0027] The reader 204 accesses the removable storage medium 205 in accordance with instructions from the processor 201, for example. The removable storage medium 205 is realized by, for example, a semiconductor device, a medium for inputting and outputting information by magnetic action, or a medium for inputting and outputting information by optical action. An example of a semiconductor device is a USB (Universal Serial Bus) memory. An example of a medium for inputting and outputting information by magnetic action is a magnetic disk. An example of a medium for inputting and outputting information by optical action is a CD (Compact Disc)-ROM, a DVD (Digital Versatile Disc), a Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0028] The communication interface 206 communicates with other devices (e.g., the microscope 100, the web camera 500, etc.) according to instructions from the processor 201, for example. The input / output interface 207 is, for example, an interface between an input device 400 and an output device. The input device 400 is, for example, a device such as a mouse 401, a keyboard 402, or a foot switch 403 that accepts instructions from a user. The output device is, for example, a display device 300 and an audio device such as a speaker.

[0029] The program executed by the processor 201 is provided to the computer 200a in the following forms, for example: (1) pre-installed in the storage device 203; (2) provided by a removable storage medium 205; or (3) provided from a server such as a program server.

[0030] The hardware configuration of the computer 200a for realizing the control device 200 described with reference to FIG. 3 is an example, and the embodiment is not limited thereto. For example, part of the above-described configuration may be deleted, or new configuration may be added. Furthermore, in another embodiment, for example, part or all of the functions of the above-described processing device may be implemented as hardware using a field programmable gate array (FPGA), a system-on-a-chip (SoC), an application specific integrated circuit (ASIC), a programmable logic device (PLD), or the like.

[0031] The display device 300 and the input device 400 are connected to the control device 200. The display device 300 is, for example, a liquid crystal display or an organic EL display. The web camera 500 is connected to the control device 200 directly or via the Internet. The web camera 500 is an example of a sensing device different from the imaging device 112, and transmits captured images to the control device 200.

[0032] The web camera 500 is a macro camera that captures a wider range than the imaging device 112. The web camera 500 may capture, for example, the microscope 100 and the operator using the microscope 100, or may capture a wide area above the stage of the microscope 100, including the operator's hands. It is sufficient for the web camera 500 to acquire an image that allows the operator's working state on the sample to be understood.

[0033] Fig. 4 is an example of a flowchart of the control process of the microscope system 1. The microscope system 1 configured as described above performs the control process shown in Fig. 4. The control process shown in Fig. 4 is a process of outputting to the superimposing device (projector 113) an auxiliary image to be superimposed on an image plane on which an optical image observed through the eyepiece 106 is formed. The control process shown in Fig. 4 is performed by the control device 200. The control process performed by the control device 200 will be described below with reference to Fig. 4.

[0034] First, the microscope system 1 acquires work information (step S1). Here, the processor 201 of the control device 200 acquires the work information acquired by a sensing device other than the image capturing device 112.

[0035] The sensing device may be, for example, a web camera 500, and the work information may be a macro image acquired by the web camera 500. The sensing device may also be a sensor that measures a state quantity of a work piece of equipment, and the work information may be state quantity information acquired by the sensor. For example, the sensing device may be a temperature sensor that measures the temperature of a soldering iron, and the work information may be temperature information acquired by the temperature sensor. The type of the sensing device may also be, for example, a pressure sensor, an acceleration sensor, a torque sensor, etc., and the form is not particularly limited.

[0036] Next, the microscope system 1 identifies the working state (step S2). Here, the processor 201 of the control device 200 identifies the working state of the operator with respect to the observation object based on the work information acquired in step S1.

[0037] The work state may be the content of the work. For example, if a worker is assembling or inspecting a precision device, it may be the work process or work content of the precision device assembly or inspection. The work state may also be the status of the work. For example, it may be a status where the work is completed or a status where the work is in progress and not completed, or it may be a normal status of the work or an abnormal status of the work (abnormality in the work). The work state may also be the status of the worker. For example, it may be the worker's working posture or whether the worker is seated (seated) or not. The work state may also be the status of the work object. For example, it may be the type, color, material, etc. of the work object. The work state may also be the status of the work equipment. For example, it may be the temperature of a soldering iron, etc.

[0038] The method for identifying the work state from the work information is not particularly limited. For example, the processor 201 may identify the work state by analyzing a macro image acquired by the web camera 501. In this case, for example, a machine learning model on which machine learning such as deep learning has been performed in advance may be used for the image analysis. Furthermore, the processor 201 may identify a state quantity, which is an example of the work state, based on state quantity information measured by a sensor.

[0039] When the work state is identified, the microscope system 1 acquires reference information (step S3). Here, the processor 201 acquires the reference information from a storage unit in which work states and reference information are previously stored in association with each other. The reference information acquired in step S3 may include at least reference information corresponding to the work state identified in step S2.

[0040] The storage unit is, for example, but is not limited to, the storage device 203 of the control device 200. The storage unit may be provided in an external device different from the control device 200, and the control device 200 may acquire the reference information from, for example, a server (storage unit) placed on a cloud connected via the Internet.

[0041] Thereafter, the microscope system 1 determines work support information (step S4). Here, the processor 201 determines the work support information based on the work state identified in step S2 and the reference information acquired in step S3 corresponding to the work state identified in step S2. Typically, the processor 201 identifies reference information corresponding to the work state based on the work state identified in step S2, and determines the work support information based on the identified reference information.

[0042] The work support information may be any information that is useful to the worker during work by being displayed in the field of view of the microscope 100 during work, such as a message or an image that the worker wishes to display in the field of view. Specific examples of work support information include work procedure manuals, messages for calling attention, and sample images (macro images). The work support information may be the reference information itself, which corresponds to a case where, for example, a message is stored as reference information in a storage unit. The work support information may also be the work information itself, which corresponds to a case where, for example, information specifying a macro image (work information) is stored as reference information in a storage unit.

[0043] Finally, the microscope system 1 outputs the auxiliary image to the projector 113 of the microscope 100 (step S5). Here, the processor 201 outputs the auxiliary image including the work support information determined in step S4 to the projector 113.

[0044] An auxiliary image is an image in which work support information is visualized. For example, an auxiliary image is an image that includes a macro image, which is work support information, at a predetermined position in a predetermined size. Alternatively, an auxiliary image may be an image in which an annotation, a message, or the like, which is also work support information, is superimposed on a macro image, which is work support information. Alternatively, an auxiliary image may be an image that includes a message to call attention, which is work support information.

[0045] By performing the control process shown in FIG. 4 , the microscope system 1 projects an auxiliary image including work support information onto the image plane where the optical image is formed. This work support information is determined according to the work state of the operator, which is identified from work information acquired using the sensing device. Therefore, the operator can obtain information according to the work state while observing the sample without taking his / her eye away from the eyepiece 106. This allows the operator to work under the microscope while obtaining appropriate information without having to frequently move his / her line of sight back and forth between the eyepiece 106 and the display device 300. Therefore, the microscope system 1 can provide the operator with appropriate information, significantly improving the efficiency of the operator's work under the microscope.

[0046] Below, in each embodiment, a specific example of the control process performed by the control device 200 will be described. <First embodiment> Fig. 5 is a diagram showing an operator using the microscope system 1. Fig. 6 is an example of an image acquired by the microscope camera (imaging device 112). Fig. 7 is an example of an image acquired by the macro camera (Web camera 500). Fig. 8 is an example of a table storing reference information. Fig. 9 is an example of a superimposed image. With reference to Figs. 5 to 9, a specific example of the above-mentioned control process will be described using an example in which the Web camera 500 is a sensing device that acquires work information.

[0047] In this example, as shown in Figures 5 to 7, worker H is performing soldering, which is one step in the assembly work of a precision device. The optical image X1 shown in Figure 6 shows the state in which soldering is being performed on terminals on a circuit board with a soldering iron.

[0048] First, in step S1, the processor 201 acquires a macro image Y1 of the hands of the worker H, captured by the web camera 500. As shown in Fig. 7 , the macro image Y1 is an image that makes it possible to grasp that work is being performed on a circuit board, the position of the work target on the circuit board, the work equipment (soldering iron) used for the work, and the like, and is acquired as work information that makes it possible to identify the work status.

[0049] Next, in step S2, the processor 201 analyzes the macro image Y1 to identify a work process. That is, the processor 201 identifies a work process for assembling a precision device as a work state based on the image (macro image Y1) acquired by the macro camera (Web camera 500).

[0050] The processor 201 may identify the work process as the soldering process (process B) by using, for example, a learning model that has learned the relationship between a macro image taken during work and the work process in advance. Note that a work process is one process included in a predetermined procedure for assembling precision equipment.

[0051] The reference information is stored in association with a work process, which is an example of a work state, in, for example, table T1 shown in Fig. 8 stored in storage device 203. In step S3, processor 201 acquires reference information corresponding to the soldering process (process B), which is the work process identified in step S2, from storage device 203. Here, processor 201 acquires a "sample image at low magnification" as reference information corresponding to the work process.

[0052] From the content of the acquired reference information, it is determined that a low-magnification sample image will be helpful for work in process B, and therefore in step S4, the processor 201 determines the macro image Y1 shown in Fig. 7 acquired by the web camera 500 as work support information. In other words, the processor 201 determines the work support information based on the reference information corresponding to the work process identified in step S2.

[0053] Finally, in step S5, the processor 201 outputs the macro image Y1 as the auxiliary image A1. As a result, as shown in Fig. 9, a superimposed image Z1 in which the auxiliary image A1 (macro image Y1) is superimposed on the optical image X1 is formed on the image plane, and is observed by the operator H through the eyepiece 106. By performing soldering work while looking at the auxiliary image A1 (macro image Y1) which captures a wide area in the operator's hands, the operator H can, for example, check whether there are any errors in the soldering locations.

[0054] 8, the content of the reference information varies depending on the work process identified from the work information (macro image). Therefore, the microscope system 1 projects onto the image plane an auxiliary image including work support information that varies depending on the work process identified from the work information (macro image). This allows the microscope system 1 to provide the worker H with information appropriate for each work process.

[0055] For example, in process C, which requires complex work, a procedure manual may be displayed as work support information to prevent errors in the work procedure. Also, in process A, where no special information display is required for work support, the projection of additional information onto the optical image may be refrained from. This may provide an environment for operator H that allows him or her to easily concentrate on observing the sample.

[0056] Second Example Fig. 10 shows another example of a table storing reference information. Fig. 11 shows another example of a superimposed image. With reference to Figs. 10 and 11 , another specific example of the above-mentioned control process will be described using an example in which the web camera 500 is a sensing device that acquires work information.

[0057] This embodiment differs from the first embodiment in that, as shown in table T2 in FIG. 10, the reference information corresponding to the soldering process (process B) is "sample image at low magnification + annotation."

[0058] 7 acquired by the web camera 500, as the work support information, and outputs the image of the macro image Y1 with the annotation added thereto as the auxiliary image A2 to the projector 113. As a result, as shown in FIG. 11 , a superimposed image Z2 in which the auxiliary image A2 (the image of the macro image Y1 with the annotation AN added thereto) is superimposed on the optical image X1 is formed on the image plane, and is observed by the worker H through the eyepiece 106.

[0059] The annotation AN added to the macro image Y1 may be, for example, information automatically added by the control device 200 after analyzing the macro image Y1. Alternatively, the annotation AN added to the macro image Y1 may be, for example, information input by a user other than the worker H using the input device 400 while viewing the macro image Y1 displayed on the display device 300. By displaying information input by a user other than the worker H as the annotation AN within the field of view, the auxiliary image A2 can be used, for example, as a means for an experienced worker to instruct an inexperienced worker.

[0060] <Third Example> Fig. 12 is another example of an image acquired by a microscope camera. Fig. 13 is another example of an image acquired by a macro camera. Fig. 14 is yet another example of a table storing reference information. Fig. 15 is yet another example of a superimposed image. With reference to Figs. 12 to 15, a further specific example of the above-mentioned control process will be described using an example in which a web camera 500 is a sensing device that acquires work information.

[0061] 12 and 13, worker H is installing a connector, which is one step in the assembly work of a precision device. The optical image X3 shown in Fig. 12 shows the state in which the pins of a plug held with tweezers are being inserted into a socket.

[0062] First, in step S1, the processor 201 acquires a macro image Y3 of the worker H's hands captured by the web camera 500. As shown in Fig. 13 , the macro image Y3 is an image that makes it possible to grasp that the worker H is performing an operation to insert a plug into a socket of a connector part fixed by a vice, as well as the work tools (tweezers) used for the operation, and is acquired as work information that enables the work status to be identified.

[0063] Next, in step S2, the processor 201 analyzes the macro image Y3 to identify the work process. The processor 201 may identify that the work process is the connector installation process (process A), for example, by using a learning model that has previously learned the relationship between the macro image captured during the work and the work process.

[0064] The reference information is stored in association with a work process, which is an example of a work state, in, for example, table T3 shown in Fig. 14 stored in storage device 203. In step S3, processor 201 acquires reference information corresponding to the connector attachment process (process A), which is the work process identified in step S2, from storage device 203. Here, processor 201 acquires "a sample image at low magnification + guide lines + work instructions" as reference information corresponding to the work process.

[0065] Then, in step S4, the processor 201 determines that the work support information is an image in which a guide line G1 (in this example, two parallel lines indicating horizontal) and a work instruction message M1 (in this example, "Keep it parallel!") have been added to the macro image Y3 shown in FIG. 13 acquired by the web camera 500.

[0066] Finally, in step S5, the processor 201 outputs an image in which the guide lines G1 and the work instruction message M1 are added to the macro image Y3 as the auxiliary image A4. As a result, as shown in Fig. 15, a superimposed image Z4 in which the auxiliary image A4 is superimposed on the optical image X3 is formed on the image plane and is observed by the worker H through the eyepiece 106. By performing the connector attachment work while observing the wide area around his / her hand using the auxiliary image A4 including the guide lines G1 and the message M1, the worker H can prevent, for example, the connector from being connected at an angle.

[0067] <Fourth Example> Fig. 16 is yet another example of an image acquired by a macro camera. Fig. 17 is yet another example of a table storing reference information. Fig. 18 is yet another example of a superimposed image. With reference to Figs. 16 to 18 , a further specific example of the above-mentioned control process will be described using an example in which a web camera 500 is a sensing device that acquires work information.

[0068] This example differs from the above-described example in that the web camera 500 captures the worker H himself, rather than the worker H's hands, as shown in Fig. 16. Fig. 16 shows a macro image Y5 capturing the entire upper body of the worker.

[0069] In step S2, the processor 201 identifies the working posture of the worker H as a working condition based on the macro image Y5 acquired as the work information in step S1. Here, the processor 201 identifies, for example, poor posture based on the macro image Y5.

[0070] In step S3, the processor 201 acquires reference information stored in association with the worker's posture, which is an example of a working state, in table T4 shown in Fig. 17, which is stored in the storage device 203. In this example, since poor posture is identified as the working state in step S2, "warning" is acquired as the corresponding reference information.

[0071] Thereafter, the processor 201 determines work support information based on the reference information corresponding to the identified work posture. In this example, it is determined from the content of the acquired reference information that a warning about the worker's posture is necessary, so in step S4, the processor 201 determines the message "Please correct your posture" as the work support information.

[0072] Finally, in step S5, the processor 201 outputs an image including the message for warning determined in step S4 as auxiliary image A5. As a result, as shown in FIG. 18 , a superimposed image Z5 in which auxiliary image A5 is superimposed on optical image X1 is formed on the image plane and is observed by worker H through the eyepiece 106. Because the warning about posture is displayed only when posture deteriorates, worker H can be more urged to take action in accordance with the warning than if the warning were displayed regardless of posture. This makes it possible for workers to work in the correct posture, and is expected to improve work efficiency.

[0073] Fifth Example Fig. 19 is a diagram showing a series of tasks being performed by a plurality of workers sharing the work process. Fig. 20 is yet another example of a table storing reference information. Fig. 21 is yet another example of a superimposed image. With reference to Figs. 19 to 21 , a further specific example of the above-described control process will be described, taking as an example a case where a web camera 500 is a sensing device that acquires work information.

[0074] This embodiment is an example in which a plurality of steps constituting a series of work such as assembly work are shared among a plurality of workers (worker H1, worker H, worker H2), and differs from the above-described embodiment in that an auxiliary image including the progress status of preceding and following workers is projected onto the image surface as work support information. The auxiliary image provided to worker H, one of the multiple workers shown in FIG. 19, will be described below.

[0075] In step S1, the processor 201 acquires macro images captured by the web cameras (web cameras 501 and 502) corresponding to the workers in charge of the previous and next processes (in this example, workers H1 and H2) as work information.

[0076] In step S2, the processor 201 identifies the progress of the work in the process in which worker H1 is responsible from the macro image acquired by the web camera 501, and identifies the progress of the work in the process in which worker H2 is responsible from the macro image acquired by the web camera 502. Furthermore, the processor 201 identifies the delay status of the preceding and succeeding workers as the work status from the progress status of workers H1 and H2. In this example, a case will be described in which the progress of worker H1's work is slightly behind schedule. In this case, the processor 201 identifies a delay in the preceding process as the work status.

[0077] Thereafter, in step S3, the processor 201 acquires reference information stored in association with a delay situation, which is an example of a work status, in table T5 shown in Fig. 20, which is stored in the storage device 203. In this example, since a delay in the previous process is identified as the work status in step S2, the processor 201 acquires a message saying "Work in the previous process is delayed" as the corresponding reference information.

[0078] Thereafter, in step S4, the processor 201 determines the message acquired as the reference information as work support information, and in step S5, outputs an image including the message determined in step S4 as auxiliary image A6. As a result, as shown in FIG. 21 , a superimposed image Z6 in which auxiliary image A6 is superimposed on optical image X1 is formed on the image plane and is observed by worker H through the eyepiece 106. Since worker H can grasp the situations of workers in front and behind him while continuing his work, he can work in accordance with the progress of those around him, and can avoid situations such as making a mistake by working more quickly than necessary.

[0079] Sixth Example Fig. 22 is a diagram illustrating a temperature sensor. Fig. 23 is yet another example of a table storing reference information. Fig. 24 is yet another example of a superimposed image. With reference to Figs. 22 to 24 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the temperature sensor 405 is a sensing device that acquires work information.

[0080] 22 , a temperature sensor 405 that measures the temperature of a soldering iron 600 used by a worker H in a soldering process is connected to the control device 200. The temperature sensor 405 is an example of a sensor that measures a state quantity of a work device, and measures the temperature of the soldering iron 600.

[0081] When worker H brings soldering iron 600 into contact with temperature sensor 405 to check whether soldering iron 600 is at an appropriate temperature before soldering, in step S1, processor 201 acquires temperature information acquired by temperature sensor 405 as work information that can identify the work state. Temperature information is an example of state quantity information.

[0082] In step S2, the processor 201 identifies the temperature of the soldering iron 600 as the working state based on the temperature information. That is, the processor 201 identifies the state quantity of the work equipment as the working state. In step S3, the processor 201 acquires reference information stored in association with temperatures in table T6 shown in FIG. 23, which is stored in the storage device 203. For example, if 300°C is identified as the working state in step S2, the processor 201 acquires a message saying "appropriate temperature" as the corresponding reference information.

[0083] Thereafter, in step S4, the processor 201 determines work support information based on the reference information. Specifically, the processor 201 determines the message acquired as the reference information as the work support information.

[0084] Finally, in step S5, the processor 201 outputs the image including the message determined in step S4 as auxiliary image A7. As a result, as shown in Fig. 24, a superimposed image Z7 in which auxiliary image A7 is superimposed on optical image X1 is formed on the image plane and is observed by operator H through the eyepiece 106. Since operator H can determine whether the condition of the work equipment is suitable for the work while continuing the work, it is possible to avoid having to rework the work and improve work efficiency.

[0085] Seventh Example Fig. 25 shows yet another example of a table storing reference information. Fig. 26 shows yet another example of a superimposed image. With reference to Figs. 25 and 26 , a further specific example of the above-described control process will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0086] This embodiment differs from the above-described example in that, as shown in table T7 in FIG. 25, in addition to information specifying the content of the work support information, information specifying the display settings of the work support information within the auxiliary image is stored in the storage unit as reference information.

[0087] In step S3, the processor 201 acquires the message "Be careful of the temperature" and the position at which to display the message, "center of the field of view," as reference information corresponding to the work state identified in step S2. Furthermore, in step S4, the processor 201 determines the work support information and the display settings for the work support information in the auxiliary image based on the reference information. Specifically, the processor 201 determines the message "Be careful of the temperature" acquired as reference information as the work support information, and determines the position at which to display the message acquired as reference information as the display settings for the work support information in the auxiliary image.

[0088] Finally, in step S5, the processor 201 outputs an image including the work support information (message) determined in step S4 in accordance with the display settings (display position) determined in step S4 as auxiliary image A8. As a result, as shown in Fig. 26, a superimposed image Z8 in which auxiliary image A8 including the message is superimposed in the center of the field of view on optical image X1 is formed on the image plane and is observed by worker H through the eyepiece 106. In this embodiment, the work support information is displayed in a position according to the work status, so that necessary information can be reliably provided to the worker while displaying the work support information in a position that does not interfere with the worker's work.

[0089] Eighth Example Fig. 27 shows yet another example of a table storing reference information. Fig. 28 shows yet another example of a superimposed image. With reference to Figs. 27 and 28 , a further specific example of the above-described control process will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0090] This embodiment differs from the seventh embodiment in that, as shown in table T8 in FIG. 27, the image size is stored instead of the display position as information specifying the display settings of the work support information included in the reference information.

[0091] In this case, in step S5, the processor 201 outputs the sample image determined in step S4 as auxiliary image A9 at the size determined in step S4. As a result, as shown in Fig. 28, a superimposed image Z9 in which auxiliary image A9 is superimposed on optical image X1 is formed on the image plane and is observed by operator H through the eyepiece 106. In this example, the work support information is displayed at a size according to the work status, so that necessary information can be reliably provided to the operator while displaying the work support information at a size that does not interfere with the worker's work.

[0092] 29 shows another example of a table storing reference information. A further specific example of the control process described above will be described with reference to the case where the web camera 500 is a sensing device that acquires work information.

[0093] In this embodiment, as shown in Table T9 in Figure 29, the color of the part being worked on is identified as the work status from the macro image acquired by the web camera 500, and the brightness of the image is stored as information that identifies the display settings of the work support information included in the reference information, which is different from the above-mentioned embodiment.

[0094] In this case, in step S5, the processor 201 outputs a sample image with brightness determined according to the color of the identified work part as an auxiliary image. This prevents the auxiliary image (sample image) from being obscured by the color of the part, for example, and allows the worker to reliably understand the auxiliary image.

[0095] <Tenth Example> Fig. 30 shows yet another example of a table storing reference information. Fig. 31 shows yet another example of a superimposed image. With reference to Figs. 30 and 31 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0096] In this embodiment, as shown in table T10 in FIG. 30, the type of part to be worked on is identified as the work status from a macro image acquired by the web camera 500, and the color of a mark indicating the range of the region of interest (ROI) (hereinafter referred to as the ROI mark) is stored as information specifying the display settings of the work support information included in the reference information, which is different from the above-described embodiment.

[0097] In this case, in step S5, the processor 201 outputs an auxiliary image A10 that displays an ROI mark on the work target part in a color determined according to the type of the work target part. This prevents a situation in which the auxiliary image A10 is not noticeable because the part and the auxiliary image A10 (ROI) are displayed in similar colors, as shown in Fig. 31, and allows the worker to reliably understand the auxiliary image A10.

[0098] 32 shows another example of a table storing reference information. With reference to FIG. 32, a further specific example of the control process described above will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0099] This embodiment differs from the tenth embodiment in that, as shown in table T11 in FIG. 32, the transparency of the ROI mark is stored as information specifying the display settings of the work support information included in the reference information.

[0100] In this case, in step S5, the processor 201 outputs an auxiliary image that displays an ROI mark on the work target part with a transparency determined according to the type of the work target part. This allows the ROI mark to be displayed with transparency adjusted according to the reflectance of the part, allowing the worker to reliably grasp the ROI without interfering with observation of the part.

[0101] <Twelfth Example> Fig. 33 is a diagram showing how the position of a webcam is changed. Fig. 34 is yet another example of a table storing reference information. Fig. 35 is yet another example of a superimposed image. With reference to Figs. 33 to 35 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the webcam 500 is a sensing device that acquires work information.

[0102] 33 , the position of the web camera 500 may be adjustable. For example, the web camera 500 may be moved to a position where it can reliably capture an image of the sample S depending on the work process. The movement of the web camera 500 may be controlled by the processor 201 or may be performed manually by an operator.

[0103] When a macro image (sample image) is displayed as an auxiliary image within the field of view, it is desirable that the auxiliary image is always displayed as an image captured from a fixed orientation regardless of the work process. Therefore, the macro image acquired by the web camera 500 may be inverted so that the auxiliary image appears to be an image captured from a fixed orientation regardless of the work process. As shown in Table T12 in Fig. 34 , whether or not inversion processing of the macro image is required is stored in the storage unit as information specifying the display settings of the work support information included in the reference information.

[0104] In this case, in step S5, the processor 201 outputs, as an auxiliary image, a macro image (sample image) that has been subjected to inversion processing determined according to the work process. As a result, as shown in Fig. 35, a superimposed image Z12 in which the auxiliary image A12 is superimposed on the optical image X12 of the sample S is formed on the image plane and is observed by the worker H through the eyepiece 106. As a result, even if the macro image (sample image) is acquired by the web camera 500 from different directions depending on the work process, an image acquired from a fixed direction regardless of the work process can be displayed as the auxiliary image A12.

[0105] <Thirteenth Example> Fig. 36 is yet another example of a table storing reference information. Fig. 37 is yet another example of a superimposed image. With reference to Figs. 36 and 37 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0106] In this embodiment, the processor 201 displays the current task position and the next task position on the macro image, and displays a guide to the next task when the task changes. As shown in table T13 in Fig. 36, information necessary for displaying a guide to visually support the transition to the next task, such as the current task position and the next task position, is stored in the storage unit as reference information corresponding to each task step.

[0107] In this case, in step S5, the processor 201 outputs an image including a display guiding the next task according to the task process as the auxiliary image A13. As a result, as shown in Fig. 37, a superimposed image Z13 in which the auxiliary image A13 is superimposed on the optical image X1 is formed on the image plane and is observed by the worker H through the eyepiece 106.

[0108] The auxiliary image A13 includes a sample image IM in which a mark R1 indicating the current work position and a mark R2 indicating the next work position are superimposed, and a guide display G2. In this example, the marks R1 and R2 are color-coded so that the worker can distinguish between the current work position and the next work position at a glance, and the guide display G2 is also displayed in a different color. This allows the worker to smoothly proceed to the next work without taking their eyes off the eyepiece 106. The auxiliary image A13 may also be displayed when the control device 200 detects that the current work position has been completed.

[0109] <Fourteenth Example> Fig. 38 shows yet another example of a table storing reference information. Fig. 39 shows yet another example of a superimposed image. With reference to Figs. 38 and 39 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0110] In this embodiment, the processor 201 may identify the work situation as the work state based on the macro image. Specifically, the processor 201 may identify the work situation as a normal work situation, or may identify the work situation as an abnormal work situation (abnormal work situation) as the work state. As shown in table T14 in FIG. 38 , the storage unit stores reference information corresponding to each work situation. In particular, a message for calling attention is stored as reference information corresponding to each abnormality.

[0111] In step S5, the processor 201 outputs the abnormality identified from the macro image, i.e., an image including a message corresponding to the detected abnormality, as auxiliary image A14. As a result, as shown in Fig. 39, a superimposed image Z14 in which auxiliary image A14 is superimposed on optical image X1 is formed on the image plane and is observed by operator H through the eyepiece 106. By displaying a message at the time an abnormality is detected, operator H can recognize the abnormal situation early and improve it.

[0112] <Fifteenth Example> Fig. 40 is a diagram showing the state of a worker when an abnormality occurs. Fig. 41 is yet another example of a table storing reference information. Fig. 42 is yet another example of a superimposed image. With reference to Figs. 40 to 42 , a further specific example of the above-mentioned control process will be described, taking as an example a case where a web camera 500 is a sensing device that acquires work information.

[0113] In this embodiment, the processor 201 identifies the work state based on the microscopic image acquired by the imaging device 112 and the macro image acquired by the web camera 500, and determines the work support information based on the reference information corresponding to the identified work state.

[0114] Specifically, as shown in FIG. 40 , the processor 201 monitors the microscopic image output from the imaging device 112, which is a microscopic camera. The processor 201 then detects an abnormality in the work based on the microscopic image. When the processor 201 detects an abnormality in the work from the microscopic image, it identifies the work status based on the macro image acquired by the web camera 500 (second camera). The abnormality in the work is, for example, a delay in the work. Furthermore, the work status may be, for example, whether the worker is aware of the abnormality in the work and is taking action, or whether the worker is performing the work without noticing the abnormality in the work.

[0115] Figure 40 shows an operator adjusting the microscope 100 to deal with an abnormality that occurs during task A, and an operator continuing the work without taking any particular action to deal with an abnormality that occurs during task B.

[0116] In this example, when an abnormality occurs in task A, the processor 201 identifies the task status as "in progress." Then, the processor 201 obtains the message "Please resume from task [X]" corresponding to "in progress" from table T15 in FIG. 41 as reference information corresponding to the task status, and determines this as task support information. Finally, if the abnormality occurs during task A, the processor 201 outputs an image including the message "Please resume from task A" as auxiliary image A15. As a result, as shown in FIG. 42, a superimposed image Z15 in which auxiliary image A15 is superimposed on optical image X1 is formed on the image plane, and is observed by the worker H through the eyepiece 106.

[0117] On the other hand, when an abnormality occurs in task B, the processor 201 identifies the task status as "before action is taken." Then, the processor 201 obtains the message "The task is behind schedule" corresponding to "before action is taken" from table T15 in FIG. 41 as reference information corresponding to the task status, and determines this as task support information. Finally, the processor 201 outputs an image including the message "The task is behind schedule" as an auxiliary image.

[0118] In this way, in this embodiment, even if the same abnormality (for example, a work delay) occurs, different messages can be displayed depending on the subsequent response status of the worker, thereby encouraging the worker to take appropriate action when an abnormality occurs.

[0119] Even if an abnormality in work is detected, it may be determined that there is no abnormality by comparing the microscope image and macro image (second camera image) from the same time. For example, even if there is a delay in work, if the worker adjusts the microscope, there is no need to issue a warning. In this way, even if an abnormality in work is detected, if it is an abnormality that does not require a message to be displayed as an auxiliary image, it may be determined that there is no abnormality.

[0120] <Sixteenth Example> Fig. 43 is a diagram showing the working state when a worker leaves his / her desk. Fig. 44 is yet another example of a table storing reference information. Fig. 45 is yet another example of a superimposed image. With reference to Figs. 43 to 45, a further specific example of the above-mentioned control process will be described, taking as an example a case where a web camera 500 is a sensing device that acquires work information.

[0121] This embodiment is similar to the fifteenth embodiment in that the processor 201 identifies the work state based on the microscopic image acquired by the imaging device 112 and the macro image acquired by the web camera 500, and determines the work support information based on the reference information corresponding to the identified work state.

[0122] The 15th embodiment differs from the 15th embodiment in that, whereas in the 15th embodiment, when a specific situation (abnormality) is detected from a microscopic image, the control device 200 identifies the work status based on a macroscopic image, in the present embodiment, when a specific situation (leaving the desk) is detected from a macroscopic image, the control device 200 identifies the work status based on a microscopic image.

[0123] 43, the processor 201 monitors macro images output from the Web camera 500, which is a macro camera. The processor 201 then detects whether the worker has left his / her seat based on the macro images. When the processor 201 detects that the worker has left his / her seat from the macro images, it identifies the work status based on the microscopic image acquired by the image capture device 112. The work status is, for example, a state in which a specific work has been completed or a state in which the work is in progress and not completed.

[0124] FIG. 43 shows that the first absence occurred when task A was completed, and the second absence occurred in the middle of task B.

[0125] In this example, when the user leaves the desk for the first time, the processor 201 identifies the work status as "work completed." The processor 201 then acquires "display OFF" from table T16 in Fig. 44 as reference information corresponding to the work status "work completed." This allows the processor 201 to recognize that projection of an auxiliary image is not necessary, and stops generating the auxiliary image.

[0126] On the other hand, when the worker leaves his / her desk for the second time, the processor 201 identifies the work status as "in progress." Then, the processor 201 obtains "Please resume from work [X]" from table T16 in FIG. 44 as reference information corresponding to the work status "in progress." If the worker is in the middle of work B, the processor 201 outputs an image including the message "Please resume from work B" as auxiliary image A16. As a result, as shown in FIG. 45, a superimposed image Z16 in which auxiliary image A16 is superimposed on optical image X1 is formed on the image plane, and is observed by worker H through the eyepiece 106.

[0127] In this way, in this embodiment, even if the worker leaves his / her seat, it is possible to switch between displaying and hiding the message depending on whether or not the worker is in the middle of work. This makes it possible to prompt the worker to take appropriate action when an abnormality occurs. Note that instead of switching between displaying and hiding the message depending on whether or not the worker is in the middle of work, a different message may be displayed.

[0128] <Seventeenth Example> Fig. 46 shows yet another example of a table storing reference information. Fig. 47 shows yet another example of a superimposed image. With reference to Figs. 46 and 47 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the web camera 500 is a sensing device that acquires work information.

[0129] This embodiment differs from the above-described embodiments in that the control device 200 determines work support information based on statistical information obtained from records of work performed in the past.

[0130] The processor 201 identifies a work process from the macro image and acquires reference information corresponding to the identified work process from table T17 shown in Fig. 46. If the work process identified from the macro image is process B, the processor 201 acquires "statistical information" as the reference information.

[0131] When the processor 201 acquires the "statistical information," it determines work support information based on statistical information that has been generated in advance based on past work records. In this example, the average work time for each work process is acquired as work support information. Finally, the processor 201 generates an auxiliary image A17 that includes the work support information and outputs it to the projector 113. As a result, as shown in FIG. 47 , a superimposed image Z17 in which the auxiliary image A17 is superimposed on the optical image X1 is formed on the image plane, and is observed by the worker H through the eyepiece 106.

[0132] Auxiliary image A17 is an image including a graph comparing the average work time for each work step with the work time required for each step up to the current step by worker H. By checking auxiliary image A17, worker H can understand whether his or her own work is faster or slower than the average.

[0133] In this embodiment, the average work time for each work process is exemplified as the statistical information, but the statistical information is not limited to the average work time. The statistical information may also be information that counts the number of times a situation requiring a warning has occurred for each process. Specifically, the statistical information may be, for example, the number of work mistakes, the number of times posture has deteriorated, the number of times the temperature has exceeded the limit, etc. The control device 200 may identify a situation requiring a warning that is particularly likely to occur for each process based on the statistical information, and output an auxiliary image including a message corresponding to the identified situation. This makes it possible to issue a warning appropriate to the work process.

[0134] <Eighteenth Example> Fig. 48 is a diagram showing an example of temperature transition. Fig. 49 is a diagram showing another example of temperature transition. Fig. 50 is yet another example of a superimposed image. With reference to Figs. 48 to 50 , a further specific example of the above-mentioned control process will be described, taking as an example a case where the temperature sensor 405 shown in Fig. 22 is a sensing device that acquires work information.

[0135] In this embodiment, the control device 200 predicts whether the temperature of the soldering iron 600 will exceed the allowable range in the future based on the temperature history (time-series information) of the soldering iron 600 acquired by the temperature sensor 405. That is, the control device 200 predicts future work information (future temperature information) based on current work information (current temperature information) and past work information (past temperature information). The past work information may be stored in the memory unit. The control device 200 extracts the past work information from the memory unit. The past work information refers to information about work performed at a time before the current work information, and it does not necessarily have to be the same work or a different work from the work for which the current work information is acquired.

[0136] 48 and 49 both show the temperature change of the soldering iron 600 measured by the temperature sensor 405, and in both cases the current temperature of the soldering iron 600 is temperature Ta. Note that the solid line shows the change in the actually measured temperature, and the dashed line shows the predicted change in the temperature.

[0137] By taking into account not only the current temperature but also the past temperature, the control device 200 predicts that the temperature of the soldering iron 600 will exceed the maximum allowable temperature Tb in the future in the case of the transition shown in FIG. 48 , and outputs an auxiliary image A18 including a warning message to the projector 113. On the other hand, in the case of the transition shown in FIG. 49 , the control device 200 predicts that the temperature of the soldering iron 600 will not exceed the maximum allowable temperature Tb in the future, and does not output an auxiliary image to the projector 113. As a result, only in cases where the maximum allowable temperature is likely to be exceeded, as in the transition shown in FIG. 48 , a superimposed image Z18 in which the auxiliary image A18 is superimposed on the optical image X1 is formed on the image plane as shown in FIG. 50 , and is observed by the worker H through the eyepiece 106. In this case, the control device 200 acquires reference information stored in association with temperatures in a table stored in the storage unit, and acquires work support information corresponding to the future temperature information.

[0138] The specific prediction algorithm is not particularly limited, but does not include a determination method that uses only the current temperature as a determination material and does not use past temperatures as a determination material, such as whether the measured temperature exceeds a predetermined threshold. The prediction algorithm may be one that predicts the future using past temperatures and the current temperature, i.e., time-series information. In the above example, the current and past temperature information is information indicating the temperature itself, and the future temperature information is information indicating whether the maximum allowable temperature Tb will be exceeded. However, the future temperature information may also be information indicating the temperature itself.

[0139] Unlike the present embodiment, if a warning is issued based solely on the current temperature without future predictions—that is, if a warning is issued simply because of a deviation from the tolerance—the warning is issued after the tolerance is exceeded, resulting in a large overshoot after the deviation and a long time until the temperature returns to an acceptable state. Furthermore, if a temperature lower than the tolerance is set as the threshold to prevent overshoot, false alarms will increase. In contrast, this embodiment predicts future values ​​(future work conditions) and issues a warning, thereby reducing false alarms and preventing work conditions from deviating from the tolerance range. This allows for a longer period of time for normal operation.

[0140] Although the work information described above is temperature information acquired from a temperature sensor, the work information is not limited to temperature information. If the work information is numerical data recorded as time-series information, future values ​​can be predicted from the time-series information, allowing for the presentation of information based on future values, as well as the presentation of cautions and warnings. For example, a macro camera is used as the sensing device for a process in which the same task is repeatedly performed. By repeatedly recording and accumulating the work time of a single task as time-series information, the most recent work time information can be used to predict the expected completion time of the entire task and the number of remaining tasks that can be completed within a specified time, and this information can be presented to the worker. Even when the work information is not temperature information, the control device 200 performs the same processing as when the work information is temperature information. The control device 200 predicts (generates) future work information based on current work information and past work information. The past work information may be stored in a memory unit in association with the type of sensing device used. The control device 200 extracts past work information from the memory unit that corresponds to the work information acquired by the sensing device in use. The work support information is determined based on the reference information corresponding to the predicted future work information. In this case, the control device 200 acquires the reference information stored in association with temperatures in a table stored in the storage unit, and acquires the work support information corresponding to the future temperature information.

[0141] Furthermore, future value predictions (future work information) can be performed not only over a short-term time axis but also over a long-term time axis. For example, consider the case where evaluation values ​​are obtained as numerical data during a performance inspection of an assembled part. If the evaluation value falls within the tolerance range, it is deemed to have passed. The tolerance range is composed of an upper and lower limit. By accumulating the evaluation values ​​of assembled parts as time-series information, it is possible to warn of the possibility that the evaluation value may exceed the tolerance range in the future. While evaluation values ​​are typically expected to vary randomly and their median value to be near the center of the tolerance range, over the long term, the distribution of evaluation values ​​may shift in a consistent direction. This could be due to factors such as changes in the dimensions of the parts being used compared to the past due to fluctuations in the 5Ms (the five manufacturing elements: "people," "machines," "materials," "methods," and "measurements"), or a deterioration in the performance of the performance evaluation equipment itself. These factors are unexpected situations that can lead to process defects. By analyzing time-series information over a long-term time axis to predict future values ​​and provide warnings, it is possible to take measures before defects occur frequently.

[0142] <19th Example> Fig. 51 is a diagram illustrating a pressure sensor. Fig. 52 is yet another example of a table storing reference information. Fig. 53 is yet another example of a superimposed image. With reference to Figs. 51 to 53, a further specific example of the above-mentioned control process will be described, taking as an example a case where the pressure sensor 700 is a sensing device that acquires work information.

[0143] 51 , in this embodiment, a pressure sensor 700 that measures the pressure that the soldering iron 600 applies to the sample S (e.g., a substrate) is connected to the control device 200. The pressure sensor 700 is, for example, a thin sensor that can be placed on a stage, and the sample S is placed on the pressure sensor 700. The pressure sensor 700 is an example of a sensor that measures the state quantity of a work device, and measures the pressure that the soldering iron 600 applies to the sample S.

[0144] When worker H brings soldering iron 600 into contact with sample S, in step S1, while soldering is being performed, processor 201 acquires pressure information acquired by pressure sensor 700 as work information that can identify the work state.

[0145] In step S2, the processor 201 identifies the pressure applied to the sample S as the working state based on the pressure information. In step S3, the processor 201 acquires reference information stored in association with the working state (pressure) in table T19 shown in FIG. 52, which is stored in the storage device 203. For example, if 2 cN is identified as the working state in step S2, the processor 201 acquires a message saying "appropriate pressure" as the corresponding reference information.

[0146] Thereafter, in step S4, the processor 201 determines work support information based on the reference information. Specifically, the processor 201 determines the message acquired as the reference information as the work support information.

[0147] Finally, in step S5, the processor 201 outputs an image including the message determined in step S4 as auxiliary image A19. As a result, as shown in Fig. 53, a superimposed image Z19 in which auxiliary image A19 is superimposed on optical image X1 is formed on the image plane and is observed by operator H through the eyepiece 106. Since operator H can grasp the state of pressure being applied to sample S while continuing his work, he can, for example, avoid applying excessive pressure to sample S or soldering iron 600.

[0148] The above-described embodiments are illustrative examples provided to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modifications of the above-described embodiments and alternatives to the above-described embodiments may be included. In other words, the components of each embodiment may be modified without departing from the spirit and scope of the embodiment. Furthermore, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. Furthermore, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in each embodiment. Furthermore, the order of the processing steps shown in each embodiment may be reversed as long as there is no contradiction.

[0149] In this specification, the expression "based on A" does not mean "based only on A," but also means "based at least on A," and further means "based at least partially on A." That is, "based on A" may be based on B in addition to A, or may be based on a part of A.

[0150] 1: Microscope system 100: Microscope 106, 106a, 106b: Eyepiece 110: Microscope optical system 112: Imaging device 113: Projector 200: Control device 200a: Computer 201: Processor 202: Memory 203: Storage device 405: Temperature sensor 500, 501, 502: Web camera 600: Pressure sensor

Claims

1. A microscope system comprising: a microscope optical system including an eyepiece lens that forms an intermediate image of an observed object on the object side of the eyepiece lens; a sensing device different from a microscope camera; a control unit that outputs an auxiliary image including work support information that is determined based on a work state for the observed object identified from work information acquired by the sensing device and reference information corresponding to the work state acquired from a memory unit; and a superimposition device that superimposes the auxiliary image output by the control unit on an image plane on which the intermediate image is formed.

2. A microscope system as described in claim 1, wherein the sensing device is a macro camera different from the microscope camera, and the control unit identifies, as the work state, a work process or work content of assembly or inspection of a precision instrument based on an image acquired by the macro camera, and determines the work support information based on reference information corresponding to the identified work process or work content.

3. A microscope system as described in claim 1, wherein the sensing device is a macro camera different from the microscope camera, and the control unit identifies, as the work state, a work process or work content of assembly or inspection of a precision instrument based on an image acquired by the macro camera, and determines, based on reference information corresponding to the identified work process or work content, the work support information and display settings for the work support information in the auxiliary image.

4. A microscope system as described in claim 1, wherein the sensing device is a macro camera different from the microscope camera, and the control unit identifies the working posture of the worker as the work state based on an image acquired by the macro camera, and determines the work support information based on reference information corresponding to the identified working posture.

5. A microscope system as described in claim 1, wherein the sensing device is a temperature sensor attached to a work equipment, and the control unit identifies the temperature of the work equipment as the work status based on temperature information acquired by the temperature sensor, and determines the work support information based on reference information corresponding to the identified temperature.

6. A microscope system as described in claim 5, wherein the control unit acquires past temperature information, which is past work information, from the memory unit, generates future temperature information based on the temperature information and the past temperature information, and determines the work support information based on reference information corresponding to the generated future temperature information.

7. A microscope system as described in claim 1, characterized in that the control unit acquires from the memory unit past work information corresponding to the work information acquired by the sensing device in use, generates future work information based on the work information and the past work information, and determines the work support information based on reference information corresponding to the generated future work information.

8. A microscope system as described in claim 1, wherein the sensing device is a macro camera different from the microscope camera, and the control unit identifies an abnormality in work as the work status based on an image acquired by the macro camera, and determines the work support information based on reference information corresponding to the identified abnormality in work.

9. A microscope system as described in claim 1, further comprising a microscope camera that acquires a microscopic image of the observed object, the sensing device being a macro camera different from the microscope camera, and the control unit identifying the work state based on the microscope image acquired by the microscope camera and an image acquired by the macro camera, and determining the work support information based on reference information corresponding to the identified work state.

10. A microscope system as described in claim 9, characterized in that the control unit detects an abnormality in the work based on the microscope image acquired by the microscope camera, and, upon detecting the abnormality in the work, identifies the work state based on the image acquired by the macro camera.

11. A microscope system as described in claim 9, characterized in that the control unit detects when an operator leaves the desk based on an image acquired by the macro camera, and, upon detecting that the operator has left the desk, identifies the work status based on the microscope image acquired by the microscope camera.

12. An operating method for a microscope system equipped with a microscope optical system that forms an intermediate image of an observed object on the object side of an eyepiece, comprising the steps of: determining work support information based on a work state for the observed object identified from work information acquired by a sensing device other than a microscope camera and reference information corresponding to the work state acquired from a memory unit; and outputting an auxiliary image including the determined work support information to a superimposition device that superimposes an input image on an image plane on which the intermediate image is formed.

13. A program for causing a computer of a microscope system having a microscope optical system that forms an intermediate image of an observed object on the object side of an eyepiece to execute a process of determining work support information based on a work state for the observed object identified from work information acquired by a sensing device other than a microscope camera and reference information corresponding to the work state acquired from a memory unit, and outputting an auxiliary image including the determined work support information to a superimposition device that superimposes an input image on an image plane on which the intermediate image is formed.

Images