Microscope system, superposition unit, and operating method

Abstract

To improve efficiency of work done through a microscope.SOLUTION: A microscope system provided herein comprises a microscope optical system 110 equipped with an eyepiece lens 106 and configured to form an optical image of a sample on an object side of the eyepiece lens 106, a control unit configured to generate auxiliary image data based on information on a target slide selected from among ordered multiple slides of a slide set, and a projector 113 acting as a superimposing unit configured to superimpose an auxiliary image including the target slide on an image plane where the optical image is formed according to the auxiliary image data. The control unit responds to an instruction to switch the target slide by selecting a slide determined according to a first order in which the multiple slides are arranged as a new target slide.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The disclosure of this specification relates to a microscope system, an overlay unit, and an operation method.

Background Art

[0002] Even today when the automation of work by robots and the like is progressing, there are still many products that require manual assembly. For example, medical devices are one such example. Since the assembly of precision devices such as medical devices involves many detailed operations, it is often performed under a microscope, and a stereomicroscope that allows the object to be viewed stereoscopically with both eyes is often used.

[0003] However, in order to check the procedure manual while performing the assembly work while observing the object with a stereomicroscope, the user has to temporarily take their eyes off the eyepiece of the stereomicroscope and shift their line of sight to a display or the like on which the procedure manual is displayed. And after confirmation, since the user has to look into the eyepiece again to continue the assembly work, it is difficult to improve the work efficiency.

[0004] Techniques related to such problems are described in, for example, Patent Document 1. In the system described in Patent Document 1, by projecting an image at the intermediate image position of the microscope, necessary information can be obtained while looking into the eyepiece.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] By the way, when assuming an assembly work consisting of a plurality of steps, it is desirable to appropriately switch and display appropriate information at the necessary timing instead of constantly displaying specific information.

[0007] Based on the circumstances described above, one aspect of the present invention is to provide a technology that improves the efficiency of work performed under a microscope. [Means for solving the problem]

[0008] A microscope system according to one aspect of the present invention includes an eyepiece, and a microscope optical system that forms an optical image of a sample on the object side of the eyepiece, User work performed under a microscope The system includes a memory for storing a slide set consisting of a plurality of slides pre-ordered according to a work procedure, a control device for generating auxiliary image data based on information about a target slide selected from the plurality of ordered slides included in the slide set, and a superimposing device for superimposing the auxiliary image, including the target slide, onto the image plane on which the optical image is formed, based on the auxiliary image data. In response to an instruction to switch the target slide, the control device selects a slide determined according to a first order based on the ordering of the plurality of slides stored in the memory as the new target slide.

[0009] An overlay unit according to one aspect of the present invention is an overlay unit that is attached to a microscope equipped with a microscope optical system that forms an optical image of a sample on the object side of the eyepiece, The user's work performed under the aforementioned microscope The system includes a memory for storing a slide set consisting of a plurality of slides pre-ordered according to a work procedure, a control device for generating auxiliary image data based on information about a target slide selected from the plurality of ordered slides included in the slide set, and a superimposing device for superimposing the auxiliary image, including the target slide, onto the image plane on which the optical image is formed, based on the auxiliary image data. In response to an instruction to switch the target slide, the control device selects a slide determined according to a first order based on the ordering of the plurality of slides stored in the memory as the new target slide.

[0010] An operating method according to one aspect of the present invention is an operating method for a control device that controls a microscope comprising a microscope optical system and a superimposing device that form an optical image of a sample on the object side of the eyepiece, wherein the control device has a slide set stored in the memory of the control device. The user's work performed under the aforementioned microscope The system includes reading a slide set consisting of a plurality of slides pre-ordered according to a work procedure, selecting a slide determined according to a first order based on the ordering of the plurality of slides stored in the memory as the new target slide in response to an instruction to switch a target slide selected from the plurality of ordered slides included in the slide set, generating auxiliary image data based on information about the new target slide, and superimposing the auxiliary image including the new target slide onto the image plane where the optical image is formed, based on the auxiliary image data, using the superimposing device. [Effects of the Invention]

[0011] According to the above embodiment, the efficiency of work performed under a microscope can be improved. [Brief explanation of the drawing]

[0012] [Figure 1] This figure illustrates a microscope system according to one embodiment of the present invention. [Figure 2] This diagram illustrates the configuration of the optical system included in a microscope system. [Figure 3] This is a diagram illustrating the configuration of a slide set. [Figure 4] This is an example of a flowchart for the image projection process performed by a microscope system. [Figure 5] This is a diagram illustrating the composition of the image formed on the image plane. [Figure 6] This is a diagram showing an example of a home screen displayed on a monitor. [Figure 7] This figure shows an example of a screen for configuring the microscope setup. [Figure 8] This diagram shows an example of a screen for adjusting AR display. [Figure 9] This is a diagram for explaining the adjustment method of AR display. [Figure 10] This is a diagram showing an example of a screen for adjusting a zoom sensor. [Figure 11] This is a diagram showing an example of a home screen observed from an eyepiece. [Figure 12] This is an example of a flowchart of an assembly work support process. [Figure 13] This is a diagram showing an example of a superimposed image observed from an eyepiece. [Figure 14] This is a diagram for explaining a slide switching operation. [Figure 15] This is a diagram showing another example of a superimposed image observed from an eyepiece. [Figure 16] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 17] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 18] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 19] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 20] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 21] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 22] [[ID=第40]]This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 23] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 24] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [Figure 25] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. [[ID=第51]] [Figure 26] This is a diagram showing yet another example of a superimposed image observed from an eyepiece. It should be noted that there seems to be an error in the original text where "第40" and "第51" are used instead of "ID=40" and "ID=51". I have translated them as they are in the original but this might need to be corrected in the source text. [Figure 27] This figure shows yet another example of a superimposed image observed through the eyepiece. [Figure 28] This figure shows yet another example of a superimposed image observed through the eyepiece. [Figure 29] This figure shows an example of the image captured as seen through the eyepiece. [Figure 30] This figure shows an example of a captured image observed on a monitor. [Figure 31] This is a diagram showing an example of a recording image. [Figure 32] This figure shows another example of a recording image. [Figure 33] This figure shows yet another example of a recording image. [Figure 34] This is a diagram illustrating the image recording method. [Figure 35] This diagram illustrates a configuration in which a trainer gives instructions from a remote location. [Figure 36] This is an example of a flowchart for the slide set creation process. [Figure 37] This diagram shows an example of a screen for setting parameters for a slide file. [Figure 38] This is a diagram showing an example of a slide settings screen. [Figure 39] This diagram illustrates the hardware configuration of a computer used to implement a control device. [Figure 40] This figure shows yet another example of a superimposed image observed through the eyepiece. [Figure 41] This figure shows yet another example of a superimposed image observed through the eyepiece. [Modes for carrying out the invention]

[0013] Figure 1 is an example of a microscope system according to one embodiment of the present invention. Figure 2 is an example of the configuration of the optical system included in the microscope system. Figure 3 is an example of the configuration of the slide set. The microscope system 1 shown in Figure 1 uses a pre-prepared slide set to provide the user with appropriate information at the necessary time while the user is working under the microscope while looking through the eyepiece 106. The configuration of the microscope system 1 will be explained with reference to Figures 1 to 3.

[0014] Microscope system 1 includes a microscope 100, a control device 200, a monitor 300, multiple input devices 400 (mouse 401, keyboard 402, foot switch 403, barcode reader 404), and a webcam 500.

[0015] Microscope 100 is a stereomicroscope capable of viewing a sample in three dimensions and is equipped with a microscope optical system 110 as shown in Figure 2. The microscope optical system 110 is an optical system for a stereomicroscope. The user can observe the optical image formed by the microscope optical system 110 on the object side of the eyepiece 106 (eyepiece 106a, eyepiece 106b) with both eyes via the eyepiece 106, thereby observing the sample in three dimensions. For this reason, microscope 100 is suitable for applications such as the assembly of precision instruments.

[0016] The microscope 100 is equipped with zoom lenses 102 (zoom lenses 102a and 102b) that can be operated with a zoom handle 130. By operating the zoom handle 130, the observation magnification can be changed while continuing to observe the sample by looking through the eyepiece 106.

[0017] The microscope 100 is equipped with a focusing handle 140. By operating the focusing handle 140, the distance between the sample and the objective lens 101 can be changed to focus on the sample.

[0018] The microscope 100 is equipped with an imaging device 112 that images a sample and acquires a digital image of the sample. The eyepiece tube 120, to which the eyepiece lens 106 is attached, is a tri-eyepiece tube, and the imaging device 112 is attached to the eyepiece tube 120. The imaging device 112 is equipped with a two-dimensional image sensor. The image sensor is not particularly limited, but examples include a CCD image sensor and a CMOS image sensor. The digital image acquired by the imaging device 112 is output to the control device 200. Alternatively, the digital image may be output directly to the monitor 300.

[0019] As shown in Figure 2, light is incident on the imaging device 112 via the imaging lens 111, after being split by a beam splitter 103a, such as a half-mirror, from one of the left or right optical paths of the microscope optical system 110. In order to compensate for the difference in optical path length between the left and right optical paths caused by the beam splitter 103a and to suppress the difference in light intensity, an ND prism 103b is provided in the other optical path.

[0020] The microscope 100 is equipped with a projector 113 that projects an auxiliary image onto the image plane formed by the imaging lenses 105 (imaging lenses 105a and 105b) that create the optical image. The projector 113 is a device that projects and superimposes the auxiliary image onto the image plane according to commands from the control device 200. More specifically, the projector 113 superimposes the auxiliary image onto the image plane based on the auxiliary image data described later. The type of projector 113 is not particularly limited. The projector 113 may be configured using, for example, a liquid crystal device or a digital mirror device.

[0021] The projector 113 is located inside the eyepiece tube 120. Light from the projector 113 is guided to the left and right optical paths of the microscope optical system 110 via the projection lens 114 and multiple beam splitters (beam splitter 115, beam splitter 104a, beam splitter 104b).

[0022] Furthermore, the eyepiece barrel 120 is provided with an operating unit 121, as shown in Figure 1. By operating the operating unit 121, the user can switch the projector 113 ON / OFF and instruct it to start or stop the superposition of the auxiliary image onto the image plane.

[0023] The control device 200 is a device that controls the microscope 100. The control device 200 generates the auxiliary image data described above and outputs it to the microscope 100 (projector 113). The auxiliary image data is generated using the slide set 10, as exemplified in Figure 3, which is stored in the control device 200 beforehand.

[0024] Slide set 10 consists of multiple slides (slide 11, slide 12, slide 13, slide 14, slide 15, slide 16). Slide set 10 may contain information to assist in the assembly of precision equipment, or more specifically, it may be a procedure manual for assembly work. The order of the multiple slides is also predetermined. In other words, slide set 10 consists of multiple slides that are pre-ordered.

[0025] Each slide contains one or more pieces of content to be projected onto the image plane as an auxiliary image. Each of the slides contains information about each step in the assembly of precision equipment, and more specifically, information including the work procedures and precautions for each step.

[0026] The control device 200 generates an auxiliary image based on information about a target slide selected from among multiple slides included in the slide set 10. The auxiliary image includes the target slide. By appropriately switching the target slide, the control device 200 can switch the auxiliary image projected onto the image plane, thereby providing the user with appropriate information.

[0027] The monitor 300 and input device 400 are connected to the control device 200. The monitor 300 is, for example, a liquid crystal display or an organic EL display. The webcam 500 transmits the captured images to the control device 200 via a network, such as the internet. The webcam 500 captures, for example, a user using the microscope system 1.

[0028] The microscope system 1, configured as described above, performs the image projection process shown in Figure 4. Figure 4 is an example of a flowchart of the image projection process performed by the microscope system. Figure 5 is a diagram illustrating the composition of the image formed on the image plane. The image projection process performed by the microscope system 1 will be described below with reference to Figures 4 and 5.

[0029] First, the microscope system 1 projects an optical image of the sample onto the image plane (step S1). Here, the imaging lens 105 focuses the light from the sample, which has been captured by the objective lens 101, onto the image plane, forming an optical image of the sample. As a result, for example, optical image A1 in Figure 5 is projected onto the image plane.

[0030] Next, the microscope system 1 selects a target slide (step S2). Here, the control device 200 selects a target slide from the slide set 10. If it is immediately after the start of the image projection process shown in Figure 4, the control device 200 may select, for example, the first slide 11 from among the ordered slides included in the slide set 10 as the target slide. Also, when it detects an instruction to switch the target slide, the control device 200 may select the next slide (for example, slide 12) as the new target slide if the instruction is to switch to the next slide, or it may select the previous slide (for example, slide 16) as the new target slide if the instruction is to switch to the previous slide. In other words, in response to an instruction to switch the target slide, the control device 200 selects a slide as the new target slide according to the order in which the ordered slides are arranged.

[0031] The instruction to switch the target slide may, for example, be input by the user to the control device 200 using an input device. Alternatively, the instruction to switch the target slide may be generated by the control device 200 itself.

[0032] The control device 200 may generate a switching instruction based on, for example, the elapsed time measured by a timer. Alternatively, the control device 200 may generate a switching instruction based on the analysis results of the image captured by the imaging device 112. In this case, the control device 200 may select the slide determined according to the analysis results of the image captured by the imaging device 112 as the new target slide. Furthermore, the control device 200 may generate a switching instruction based on, for example, the user's gestures recognized via the webcam 500.

[0033] When a target slide is selected, the microscope system 1 generates auxiliary image data (step S3). Here, the control device 200 generates auxiliary image data based on information about the target slide selected in step S2. For example, if slide 11 shown in Figure 3 is selected as the target slide, the control device 200 generates auxiliary image data corresponding to the auxiliary image B1 in Figure 5, which includes the target slide (slide 11), based on information such as the type of content contained in the target slide (rectangular gauge), the size of the content (width, height), the position of the content (coordinates), and other options (presence or absence of dimension display).

[0034] Finally, the microscope system 1 projects an auxiliary image onto the image plane (step S4). Here, the control device 200 outputs the auxiliary image data generated in step S3 to the microscope 100, and the projector 113 of the microscope 100 projects the auxiliary image B1 onto the image plane based on the auxiliary image data output from the control device 200. As a result, as shown in Figure 5, the auxiliary image B1 is superimposed on the image plane on which the optical image A1 is formed, and a superimposed image C1 is formed in which the optical image A1 and the auxiliary image B1 are superimposed.

[0035] In this way, in the microscope system 1, a slide selected from a pre-prepared slide set is projected as an auxiliary image onto the image plane where the optical image is formed. Furthermore, the slides projected onto the image plane are switched according to a predetermined order in response to a switching instruction. Therefore, even when performing tasks where different information is required for each process, such as assembly work, the user can obtain information appropriate to the work process while observing the optical image of the sample without taking their eye off the eyepiece 106 by switching slides. This allows the user to perform a series of tasks, such as assembly work, without frequently shifting their gaze between the eyepiece 106 and the monitor 300. Consequently, the microscope system 1 can significantly improve the efficiency of the user's work performed under the microscope.

[0036] The following describes in more detail the functions of the software application (hereinafter referred to as the work support application) provided by the microscope system 1 to assist in assembly work performed under the microscope. Figure 6 shows an example of the home screen displayed on the monitor. In the microscope system 1, the work support application is launched when the control device 200 executes a predetermined program, and window W1 shown in Figure 6 is displayed on the monitor 300.

[0037] Immediately after the work support application is launched, the Home tab is selected in window W1, and the Home screen shown in Figure 6 is displayed. From the Home screen, the user can select a work mode to support assembly work, a training mode to receive guidance from a trainer, and a procedure manual mode to create procedure manuals to be used in assembly work.

[0038] Furthermore, various settings can be adjusted by selecting the Setup tab. The following explains the various settings necessary for the proper operation of the work support application, referring to Figures 7 through 10.

[0039] Figure 7 shows an example of a screen for setting the microscope configuration. Figure 8 shows an example of a screen for adjusting the AR display. Figure 9 is a diagram illustrating how to adjust the AR display. Figure 10 shows an example of a screen for adjusting the zoom sensor.

[0040] Selecting the Microscope Configuration tab within the Setup tab displays the screen shown in Figure 7. By following the steps displayed on the screen shown in Figure 7, the user can select the zoom lens, objective lens, intermediate tube, camera adapter, and camera from the pull-down lists to match the configuration of the microscope 100, thereby allowing the control device 200 to correctly recognize the configuration of the microscope 100. This enables the control device 200 to recognize, for example, the reference magnification (magnification excluding zoom magnification) of the microscope optical system 110.

[0041] Selecting the AR display adjustment tab within the setup tab displays the screen shown in Figure 8. By adjusting the AR display according to the procedure shown on the screen in Figure 8, the user can match the position on the digital image acquired by the imaging device 112 with the position of the AR display. This allows the control device 200 to correctly display the AR, that is, each content of the slide included in the auxiliary image, at the position recognized via the digital image.

[0042] The specific procedure for adjusting the AR display is as follows: First, the user (1) places the sample to be adjusted on the stage, and (2) looks through the eyepiece 106. During AR display adjustment, an auxiliary image B2, which includes the digital image acquired by the imaging device 112, is projected onto the image plane by the projector 113. As a result, the user can see the superimposed image C2 shown in Figure 9 through the eyepiece 106. The superimposed image C2 is an image in which the optical image A2 of the sample to be adjusted and the auxiliary image B2 are superimposed. The auxiliary image B2 includes the digital image of the sample to be adjusted (content B2a) and the adjustment menu (content B2b).

[0043] Subsequently, the user adjusts the AR display so that it overlaps with the sample used for adjustment. Specifically, by operating the adjustment menu to adjust the projection position, angle, and size of the digital image, the user ensures that the digital image of the sample and the optical image of the sample perfectly match on the image plane. Once the adjustment is successfully completed, the user presses the registration button. This records setting information for appropriately projecting the auxiliary image onto the image plane, based on the adjusted display position (X, Y), angle, and size information, in the control device 200. Specifically, this setting information is a conversion formula that converts the pixel positions of the imaging device 112 to the pixel positions of the projector 113. This conversion formula is calculated from the adjusted display position, angle, and size information described above. Figures 8 and 9 illustrate an example including the three elements mentioned above, but the conversion formula only needs to include at least one of the elements of size (scaling), rotation, translation, and distortion coefficient.

[0044] Selecting the Zoom Sensor Adjustment tab within the Setup tab displays the screen shown in Figure 10. By adjusting the zoom sensor according to the procedure displayed on the screen shown in Figure 10, the user can allow the control device 200 to correctly recognize the settings of the zoom lens 102. As a result, the control device 200 can correctly recognize the zoom magnification changed by operating the zoom handle 130 via the zoom sensor, and thus correctly recognize the current magnification of the microscope optical system 110 from the reference magnification and zoom magnification of the microscope optical system 110.

[0045] The specific procedure for adjusting the zoom sensor is as follows: (1) The user operates the zoom handle 130 to set the zoom dial to each click position (0.8x, 2.0x, 3.2x, 4.0x, 5.6x), and then presses the acquire button. This causes the control device 200 to acquire sensor information generated from the zoom sensor at each magnification. Next, the user (2) instructs the device to perform the adjustment. Here, the control device 200 updates the information showing the relationship between the sensor information output from the zoom sensor and the zoom magnification, based on the sensor information acquired in (1) and the corresponding zoom magnification information. Finally, the user (3) checks the adjustment result. Here, the user rotates the zoom dial to any position and checks whether the zoom magnification indicated by the dial matches the magnification sensor value displayed on the screen.

[0046] Next, we will describe the work modes of the work support application. Figure 11 shows an example of the home screen as observed through the eyepiece. Figure 12 is an example of a flowchart of the assembly work support process.

[0047] The work mode is started when the user presses the start button for assembly work on the home screen shown in Figure 6, which is displayed on the monitor 300. Alternatively, when the user performs a predetermined operation using the input device 400 (for example, pressing an assigned key), the projector 113 projects the auxiliary image B3 shown in Figure 11 onto the image plane, and the menu screen is displayed within the field of view. The work mode may also be started when the user selects "Open" on the menu screen (auxiliary image B3) observed through the eyepiece 106. Once the work mode is started, the control device 200 starts the assembly work support process shown in Figure 12.

[0048] First, the control device 200 reads the slide file (step S11). More specifically, the control device 200 reads the slide file selected by the user using the input device 400. The slide file is a file that records information about the slide set and is created and edited in the procedure mode described later.

[0049] While the selection of slide files is typically done by the user clicking on the slide file icon in a folder, the selection method is not limited to this. For example, a barcode reader 404 may obtain identification information of the slide set (e.g., the path to the slide file) from a barcode, and the control device 200 may read the slide file and obtain the slide set based on the identification information obtained by the barcode reader 404. In other words, the barcode reader 404 is an example of an acquisition device that obtains identification information of a slide set, and the control device 200 may obtain the slide set based on the identification information obtained by the acquisition device.

[0050] The acquisition device for obtaining identification information of the slide set is not limited to a one-dimensional code reader such as the barcode reader 404, but may also be a two-dimensional code reader such as a QR code reader (QR code is a registered trademark), or an RF tag reader. Furthermore, identification information may be obtained by detecting, for example, a two-dimensional code from the digital image acquired by the imaging device 112. In other words, the acquisition device may include at least one of a one-dimensional code reader, a two-dimensional code reader, an RF tag reader, and an imaging device.

[0051] When the slide file is loaded, the control device 200 selects the first slide as the target slide (step S12). More specifically, the control device 200 selects the first slide as the target slide from among a sequence of slides included in the slide set corresponding to the slide file selected in step S1.

[0052] Next, the control device 200 generates auxiliary image data (step S13). More specifically, the control device 200 generates auxiliary image data based on information about the target slide (first slide) selected in step S12.

[0053] Furthermore, if the size of the auxiliary image projected onto the image plane is to be changed according to the observation magnification, the control device 200 may generate auxiliary image data based on information about the target slide and the magnification information of the microscope optical system 110. Since the zoom sensor adjustment described above has been performed in advance, the control device 200 can easily obtain accurate information about the magnification information of the microscope optical system 110.

[0054] Once auxiliary image data is generated, the control device 200 controls the projector 113, which is a superimposing device, so that the auxiliary image is superimposed on the image plane (step S14). If the first slide contains a bulleted list of the overall flow of the assembly work, for example, as shown in Figure 13, a superimposed image C4 is formed on the image plane, in which an auxiliary image B4 consisting of text content is superimposed on the optical image A1. This allows the user to quickly confirm the content of the work to be performed in the initial stages of the assembly work by looking at the auxiliary image B4 displayed in the field of view. Figure 13 shows an example of a superimposed image observed through the eyepiece.

[0055] Subsequently, the control device 200 determines whether or not it has detected a switching instruction (step S15). A switching instruction can be input to the control device 200 by, for example, a user using the input device 400. Figure 14 is a diagram illustrating the operation of switching slides. For example, a switching instruction may be input to the control device 200 by pressing a switch (switch 131, switch 132) equipped with a zoom handle 130 as shown in Figure 14. Pressing switch 131 may input an instruction to switch to the next slide to the control device 200, and pressing switch 132 may input an instruction to switch to the previous slide to the control device 200.

[0056] A switching instruction may be input to the control device 200 by rotating the wheel of the mouse 401. Rotating the wheel towards the user may input an instruction to switch to the next slide to the control device 200, and rotating the wheel away from the user may input an instruction to switch to the previous slide to the control device 200. Alternatively, a switching instruction may be input to the control device 200 by pressing a shortcut key on the keyboard 402. Furthermore, a switching instruction may be input to the control device 200 by pressing the foot switch 403. Stepping on the right-hand area of ​​the foot switch 403 may input an instruction to switch to the next slide to the control device 200, and stepping on the left-hand area of ​​the foot switch 403 may input an instruction to switch to the previous slide to the control device 200. Furthermore, the switching of slides may also be instructed by voice, and a microphone (not shown) may function as an input device for inputting switching instructions. In other words, the input device for inputting switching instructions may include at least one of the mouse, keyboard, switches on the handle, foot switch, and microphone. The input device for issuing the switching instruction should preferably allow the user to issue the switching instruction with a single, simple operation that can be performed while looking through the eyepiece 106.

[0057] When a switch instruction is detected (step S15 YES), the control device 200 selects a new target slide (step S16). Here, when the control device 200 detects an instruction to switch to the next slide, it selects the slide following the current target slide as the target slide from among the multiple slide sets included in the slide set acquired in step S11, according to the order of those multiple slides. Similarly, when the control device 200 detects an instruction to switch to the previous slide, it selects the slide immediately preceding the current target slide as the target slide from among the multiple slide sets, according to the order of those multiple slides.

[0058] Subsequently, the control device 200 executes steps S13 and S14 again. If the new target slide contains text containing operational precautions for the process to be performed, for example, a superimposed image C5 is formed on the image plane, where an auxiliary image B5 consisting of text content is superimposed on the optical image A1, as shown in Figure 15. This allows the user to check operational precautions immediately before starting work, and is expected to reduce human error during the process. Figure 15 shows an example of a superimposed image observed through the eyepiece.

[0059] The control device 200 repeatedly executes the processes from steps S13 to S16 in response to the input of a switching instruction. As a result, for example, multiple slides included in the slide set are projected onto the image plane in sequence, and the information to be provided is delivered to the user at the time required. Furthermore, even if the user restarts the work, the slides are rewound and the appropriate information is delivered to the user.

[0060] Figures 16 to 28 show other examples of superimposed images observed through the eyepiece. The following describes an example of the assembly process performed by the user while switching between slides, referring to Figures 16 to 28.

[0061] Figures 13 and 15 illustrate the case where a slide containing text content is selected as the target slide, but the slide may also contain geometric content such as rectangles, circles, and lines. Figure 16 shows an example in which an auxiliary image B6 containing rectangular content B6a and circular content B6b is projected onto the image plane. The auxiliary image B6 may be used, for example, for sample alignment. While observing the superimposed image C6, which is the superimposed image of the optical image A1 and the auxiliary image B6, the user may move the stage so that the content contained in the auxiliary image B6 overlaps with a predetermined structure in the sample.

[0062] Once alignment using auxiliary image B6 is complete, the user may instruct the slide to switch and check whether the size of a specific structure in the sample matches the specifications. For example, Figure 17 shows an example where an overlaid image C7 is formed on the image plane, with auxiliary image B7 containing rectangular gauge content and optical image A1 superimposed. The gauge content may include vertical and horizontal dimension markings, as shown in Figure 17. The gauge content is pre-made to the specified size. The user may check whether the size of a specific structure matches the specifications by comparing the gauge content contained in auxiliary image B7 with the specific structure of the sample appearing in the optical image.

[0063] The user may change the observation magnification by operating the zoom handle 130 during the inspection. By observing at a higher magnification, the user may carefully check whether specific structures of the sample conform to the specifications. Figure 18 illustrates a superimposed image C8 in which the optical image A8 of the sample, magnified at a higher magnification than optical image A1, and the auxiliary image B8 are superimposed. The auxiliary image B8 includes gauge content (content B8a) that has been magnified to match optical image A2.

[0064] Furthermore, as shown in Figure 18, auxiliary image B8 may include setting information representing the current magnification of the microscope 100 as text content (content B8b), in addition to gauge content (content B8a). While Figure 18 shows an example where auxiliary image B8 includes both the overall magnification and zoom magnification of the microscope 100, it may include only one of these. Additionally, the magnifications of each optical system affecting the overall magnification (e.g., objective lens magnification, zoom magnification, intermediate tube magnification, eyepiece magnification, etc.) may be projected onto the image plane. This allows the user to recognize the set magnification while observing. In particular, displaying magnifications adjusted while looking through the eyepiece, such as zoom magnification, as an auxiliary image eliminates the need to take one's eye away from the eyepiece to check the magnification, significantly improving work efficiency. The setting information may be displayed at predetermined timings, such as after changing the magnification. Furthermore, the setting information is not limited to magnification information; it may also include information about any current setting of the microscope 100. For example, the type of objective lens and filter settings may be included as setting information in the auxiliary image.

[0065] Projection of auxiliary images containing content sized according to the zoom magnification can be achieved by the control device 200 generating auxiliary image data based on the magnification information of the microscope optical system 110 and information about the target slide. More specifically, the control device 200 should generate auxiliary image data such that a predetermined piece of content from one or more pieces of content contained in the target slide is included in the auxiliary image at a size corresponding to the magnification information. The predetermined content is content of the first category, such as the gauges mentioned above, or reticles. Since gauges and reticles are used for dimensional measurement, it is desirable to project them with their size changed according to the observation magnification.

[0066] On the other hand, the control device 200 only needs to generate auxiliary image data such that, among one or more contents included in the target slide, the contents other than the predetermined content are included in the auxiliary image at a predetermined size. The contents other than the predetermined content are the second category of content, such as text content. It is desirable that the text be displayed at a constant ratio to the field of view, that is, at a constant size, regardless of the observation magnification.

[0067] The control device 200 should, for example, classify the content contained in the slides into either Category 1 or Category 2 content based on information about the type of content, and generate auxiliary image data of different sizes according to the classification result. Examples of content types include pens, shapes, text, still images, moving images, gauges, reticles, image analysis results, timers, dashboards, external device information, and measurement content.

[0068] Figure 19 shows an example in which an overlaid image C9 is formed on the image plane, with an auxiliary image B9 containing cross-shaped reticle content and the optical image A1 superimposed. The reticle content, like the gauge content, is classified as Category 1 content and is projected enlarged or reduced according to the zoom level. Users may switch slides to display the auxiliary image B9 and inspect the sample by recognizing the size of a specific structure using the scale of the reticle contained in the auxiliary image B9.

[0069] Figure 20 shows an example in which an auxiliary image C10 is formed on the image plane by superimposing an auxiliary image B10 containing text content (content B10a), line measurement content (content B10b), and text content (content B10c) onto the optical image A1. The user may change the state in which the auxiliary image B10 is projected by switching slides, measure the size of a predetermined structure, and determine whether the inspection is successful or not. As shown in Figure 20, the user may also instruct the measurement by specifying the range to be measured with an input device such as a mouse 401, and the control device 200 may project the auxiliary image B10 containing the measurement results onto the image plane using the projector 113. Such dynamic display is possible by the control device 200 generating auxiliary image data based on the magnification information of the microscope optical system 110 and information about the target slide, as well as position information entered by the user. Note that the auxiliary image may include not only measurement content but also surface measurement content. Furthermore, setting information (magnification) displayed as text content is particularly effective when the magnification is specified as a requirement during measurement. By including magnification in the auxiliary image, users can confirm that the magnification requirements specified in the text content are met without taking their eyes off the eyepiece, and input measurement instructions.

[0070] Furthermore, slides may include still images and moving images as content. Figure 21 shows an example where an overlaid image C11 is formed on the image plane by superimposing an auxiliary image B11 containing a still image and an optical image A11. Users may perform the work while checking the state of the sample after the correct work has been done using the still image, i.e., the sample. Figure 22 shows an example where an overlaid image C12 is formed on the image plane by superimposing an auxiliary image B12 containing a moving image and an optical image A11. Users may perform the work while checking the correct work procedure using the moving image.

[0071] Furthermore, the image displayed on the image plane as an auxiliary image may be an image captured in real time by the webcam 500. This allows the user to perform tasks while checking both the micro-image (optical image of the sample) and the macro-image (auxiliary image) captured by the webcam 500, which shows the area around the microscope 100 and the user's hands. Figure 23 shows an example in which an overlaid image C121 is formed on the image plane, where auxiliary image B121, which includes an image of tools placed around the microscope 100 captured by the webcam 500, and the optical image A11 are superimposed. Figure 24 shows an example in which an overlaid image C122 is formed on the image plane, where auxiliary image B122, which includes an image of the work area and tools placed around it captured simultaneously by the webcam 500, is superimposed on the optical image A11. As shown in Figures 23 and 24, displaying surrounding tools as an auxiliary image allows for smoother tool changes and other operations.

[0072] Furthermore, the slide may include timer and other sensor information as content. Figure 25 shows an example where an overlaid image C13 is formed on the image plane by superimposing an auxiliary image B13 containing a still image (content B13a) and elapsed time since the start of work (content B13b) onto an optical image A11. Users may perform the work while checking whether too much time is being taken by checking the elapsed time with content B13b. Figure 26 shows an example where an overlaid image C14 is formed on the image plane by superimposing an auxiliary image B14 containing a still image (content B14a) and the temperature of the work area measured by a thermometer (not shown) (content B14b) onto an optical image A11. Users may perform the work while checking that the temperature of the work area is within an appropriate range using content B14b.

[0073] The final slide may include text content showing the work results. Figure 27 shows an example in which an overlaid image C15 is formed on the image plane by superimposing an auxiliary image B15 containing text content and an optical image A11. Users may check the progress and results of the assembly work using the auxiliary image B15. The control device 200 may also generate auxiliary image data corresponding to the auxiliary image B15 by communicating with other devices to collect information.

[0074] The control device 200 controls the projector 113 to display multiple slides included in the slide set selected by the user in a predetermined order according to the switching instruction, but it may also control the projector 113 to display a specific slide via an interrupt. For example, if the manufacturing line in charge of the user is shut down, the microscope system 1 may notify the user of such information via an auxiliary image. Figure 28 shows an example in which an overlaid image C16 is formed on the image plane by superimposing an auxiliary image B16 containing dashboard content indicating the status of the manufacturing line and an optical image A11. The user may check the status of the manufacturing line using the auxiliary image B16.

[0075] Figure 29 shows an example of the captured image as observed through the eyepiece. Figure 30 shows an example of the captured image as observed on the monitor. Figures 31 to 33 are illustrative images for recording. Figure 34 is a diagram illustrating the image recording method. The following explanation will describe how to create work records including images, referring to Figures 29 to 34.

[0076] During assembly, the user may instruct the system to photograph or record the sample, and may record still images or videos as evidence of the work in the control device 200. The user may input instructions for photography or video recording while looking through the eyepiece 106. For example, when the camera button is pressed on the home screen shown in Figure 11, the auxiliary image B17 shown in Figure 29 is projected onto the image plane, and the shooting screen is displayed. The user may input shooting instructions while viewing the live image in the auxiliary image B17 shown in Figure 29. The live image can be acquired, for example, by the imaging device 112. By inputting shooting instructions while checking the live image, the user can adjust the focus, brightness, etc., before shooting.

[0077] Alternatively, the shooting instructions may be entered via window W2, as shown in Figure 30, displayed on monitor 300. Figure 30 shows the live image LV acquired by the imaging device 112 and the auxiliary images (content B10a, content B10b) projected onto the image plane displayed within the field of view mark FV, which indicates the field of view of the microscope 100. The user may also enter the shooting instructions via window W2.

[0078] When the control device 200 detects a shooting or recording instruction (step S17YES), it controls the imaging device 112 to perform shooting or recording (step S18). Here, the control device 200 associates the digital image of the sample captured by the imaging device 112 (captured image) with the auxiliary image that was projected onto the image plane at the time of shooting, and records it in the recording device. The recording device may be located within the control device, or it may be located on a server on a network.

[0079] The control device 200 may, for example, create a recording image F1 by combining the captured image D1 and the auxiliary image E1, as shown in Figure 31, and record it in the recording device. Alternatively, the control device 200 may record the captured image D1 and the auxiliary image E1 as separate files and additionally record information relating them. If only a portion of the auxiliary image E1 is within the field of view of the imaging device 112, the auxiliary image E1 may be recorded after cropping to include only the portion within the field of view.

[0080] Figure 31 illustrates the configuration of the recording image as an example where the field of view of the imaging device 112 is narrower than the field of view of the microscope optical system 110. If the field of view of the imaging device 112 is wider than the field of view of the microscope optical system 110, the control device 200 may create a recording image F2 by combining the captured image D2 and auxiliary image E2 with a field of view mark FV indicating the field of view of the microscope optical system 110, as shown in Figure 32, and record it in the recording device. The control device 200 may record the captured image D2, auxiliary image E2, and field of view mark FV as separate files and additionally record information relating them.

[0081] Furthermore, as shown in Figure 33, the control device 200 may create image data of a recording image F3 by combining the captured image D2, auxiliary image E2, and field mark FV with an annotation image H3, and record it on the recording device. The annotation image H3 is projected using annotation image data created in response to drawing instructions input by the user using the input device 400. In this respect, the annotation image H3 differs from the slide-based auxiliary image E2. However, the annotation H3 is similar to the auxiliary image E2 in that it is superimposed on the image plane by the projector 113 based on the image data generated by the control device 200.

[0082] The user may input drawing instructions while looking through the eyepiece 106. For example, the user may draw characters or shapes on the image plane by pressing the pen button on the home screen shown in Figure 11. Alternatively, the user may add characters or shapes on window W2 shown in Figure 30. In this case, when text is written as content on the live image LV with the pen, the content is projected onto the optical image at a position corresponding to the position on the live image LV where the content is placed. The content may be characters written with the pen, as well as shapes such as lines, arrows, rectangles, and circles, and may also be text, images (still images, videos), etc. The user can use this drawing function to leave notes and comments about the work results. The control device 200 may record the captured image D2, auxiliary image E2, and annotation image H3 as separate files and additionally record information that links them together.

[0083] Furthermore, as shown in Figure 34, the control device 200 may record image G4, captured by the webcam 500, in the recording device in association with the recorded image F4. By recording image G4 in association with the recorded image F4, the user's condition during work can also be recorded. Image G4 and recorded image F4 may be recorded as a single image.

[0084] In the above example, a recorded image was taken at the end of the assembly work and used as evidence of the work performed by the user. However, work records may be taken during the assembly work, or multiple times at any point during the work. Also, although still images were used as examples of recorded images, recorded images may also be moving images, and the control device 200 may record the entire process from the start to the end of the work as a moving image. For example, image G4 and recorded image F4 shown in Figure 34 may be recorded as a video and also combined and recorded as a single image. This makes it possible to play back image G4 and recorded image F4 simultaneously on the same time axis, making it easy to recognize the relationship between the work performed under the microscope and the user's condition. Therefore, work analysis also becomes easier. Furthermore, when recording moving images, chapter information linked to the page of the slide being played back may be written to the moving image as an auxiliary image. This improves searchability when reviewing the moving image later.

[0085] Furthermore, the work record is not limited to images; it may also contain other information. The control device 200 may record information that is not projected onto the image plane, such as the settings information of the microscope system 1, as metadata along with the image, as part of the work record. The control device 200 may also record audio data or operation logs of the microscope system 1 as part of the work record. In addition, the control device 200 may record information such as time information obtained from a time server or the username used to log in to the work support application as part of the work record, or it may record information that identifies the sample (part name or serial number) as part of the work record. The work record may be recorded automatically by the control device 200, not only when the user explicitly instructs it to be recorded.

[0086] If a termination command is entered after the shooting command (step S19YES), the control device 200 terminates the assembly work support process shown in Figure 12.

[0087] The microscope system 1 performs the assembly work support process shown in Figure 12, allowing the user to switch slides as needed according to the work process. Therefore, the user can obtain necessary information at unnecessary times without taking their eye off the eyepiece 106. Consequently, the microscope system 1 significantly improves the efficiency of the user's work.

[0088] Next, the training mode of the work support application will be described. The training mode is started when the user presses the training start button on the home screen shown in Figure 6, which is displayed on the monitor 300. The training mode differs from the work mode in that it uses slide files created for training, but otherwise it is the same as the work mode. Therefore, even in training mode, the control device 200 executes the assembly work support process shown in Figure 12.

[0089] In training mode, the microscope system 1 is used by the trainee who works by looking through the eyepiece 106 and the trainer who gives instructions to the trainee while viewing the image displayed on the monitor 300. The trainer can give instructions to the trainee in real time by projecting annotated images onto the projector 113 using the drawing function that was used to create evidence in work mode.

[0090] Alternatively, instead of the trainer using the input device 400 to operate the control device 200 and give instructions to the trainee, instructions may be given to the trainee from a remote location. Figure 35 illustrates a configuration in which the trainer gives instructions from a remote location. As shown in Figure 35, the trainer may access the microscope system 1 from remote terminals (remote terminal 601, remote terminal 602) connected to the microscope system 1 via a network such as the Internet. For example, the trainee may access the control device 200 from the remote terminal using a remote desktop function or the like and give instructions to the trainee by directly operating the control device 200. Alternatively, instructions may be given to the trainee from the remote terminal by installing a work support application on the remote terminal.

[0091] Figure 36 is an example of a flowchart for the slide set creation process. Figure 37 shows an example of a screen for setting slide file parameters. Figure 38 shows an example of a slide settings screen. The procedure mode of the work support application will be explained below with reference to Figures 36 to 38.

[0092] The procedure manual mode is started when the user presses the "Create New" or "Edit" button for creating a procedure manual on the home screen shown in Figure 6, which is displayed on the monitor 300. When the "Create New" button is pressed and the procedure manual mode is started, the control device 200 starts the slide set creation process shown in Figure 36.

[0093] First, the control device 200 creates a new slide file (step S21). Once the slide file is created, the control device 200 sets the parameters of the slide file according to the user's input (step S22). The user can specify the number of slides, for example, by adding or deleting slides in window W3 shown in Figure 37. They can also specify the order of these slides by rearranging them.

[0094] After setting the parameters for the slide file, the control device 200 creates the slides one by one according to the user's input. Specifically, for each slide, the control device 200 repeats the process of setting the background (step S23) and arranging the content and setting the content parameters (step S24).

[0095] In step S23, the control device 200 sets a background for the slide selected by the user from area 20 of window W3 shown in Figure 38. Specifically, when the user presses button 31 in area 30, the control device 200 sets no background for that slide. When the user presses button 32, the control device 200 sets a live image acquired by the imaging device 112 as the background for that slide. When the user presses button 33, the control device 200 sets a pre-acquired video as the background for that slide. When the user presses button 34, the control device 200 sets a pre-acquired still image as the background for that slide. Figure 38 shows that, as a result of the user pressing button 34, a pre-acquired still image is displayed in area 40, which displays the slide selected in area 20.

[0096] Setting a live image as the background is particularly suitable when the sample to be used in the work mode is available when creating a slide set in procedure mode. By creating the slides while displaying the sample to be used in the work mode as a live image, the content can be positioned and sized appropriately to match the sample in step S24.

[0097] Furthermore, setting still images or live images as the background is particularly suitable when creating a slide set in procedure mode and it is not possible to prepare the sample to be used in work mode. Even if a sample cannot be prepared, by creating slides with a still image or video of the sample taken in advance as the background, the content can be placed in the appropriate position and size according to the sample in step S24.

[0098] The background set in step S23 is used to facilitate the user's placement and configuration of content in step S24. In other words, it is used only for creating and editing slides, and the background set here will not be displayed in the auxiliary image projected in work mode.

[0099] In step S24, the control device 200 places content on the slide whose background was set in step S23 and sets the parameters of that content. Specifically, the control device 200 places content on the slide according to the input from the user and incorporates that content as a component of the slide.

[0100] The type of content to be placed on the slide can be selected in area 50. Once you select a content type from area 50, you can further select subdivided content (also called objects) belonging to that content from area 60.

[0101] For example, if you select graphic content in area 50, you can select content such as lines, arrows, rectangles, circles, text, images (still images), and videos (moving images) from area 60. Similarly, if you select gauge content in area 50, you can select content such as straight line gauges, rectangular gauges, and circular gauges from area 60. Furthermore, if you select reticle content in area 50, you can select content such as cross reticles and grid reticles from area 60. Figure 38 shows how, after a user selects gauge content in area 50 and then selects a rectangular gauge in area 60, the rectangular gauge is placed as content in area 40, which displays the slide selected in area 20.

[0102] Content parameters can be set in area 70. Content parameters include height, width, rotation (orientation), vertical position, and horizontal position. For gauges and reticles, whether or not to display the length can also be set as a parameter. Alternatively, content parameters may be set by moving or scaling the content in area 40. Note that the parameters for each content item are not limited to those exemplified here. Each content item may include other parameters such as color, line width, and line type.

[0103] Once the processing in steps S23 and S24 is completed for all slides, the control device 200 saves the slide file (step S25) and terminates the slide set creation process shown in Figure 36. In this way, the microscope system 1 creates a new slide set in response to operations performed on the creation screen displayed on the monitor 300, which is the display device.

[0104] Although Figure 36 illustrates the process of creating a new slide set, the process of editing a slide set is basically the same as shown in Figure 36. When editing a slide file, in step S21, region 20 reads an existing slide file instead of creating a new one, and then performs the subsequent processing. In other words, in the microscope system 1, the control device 200 edits an existing slide set in response to operations on the editing screen displayed on the display device, monitor 300.

[0105] As described above, with the microscope system 1, by using the procedure manual mode, users can freely create and edit slide sets to be used as procedure manuals. By using slide sets created in procedure manual mode, in which the user has arranged the necessary information in the required order, the microscope system 1 can provide the user with the appropriate information at the necessary time while working under the microscope in work mode.

[0106] Figure 39 is a diagram illustrating the hardware configuration of a computer 200a for realizing the control device 200 according to the above embodiment. The hardware configuration shown in Figure 39 includes, for example, a processor 201, memory 202, storage device 203, reader 204, communication interface 206, and input / output interface 207. The processor 201, memory 202, storage device 203, reader 204, communication interface 206, and input / output interface 207 are connected to each other, for example, via a bus 208.

[0107] The processor 201 may be a single processor, a multi-processor, or a multi-core processor. The processor 201 reads and executes the program stored in the memory device 203, thereby performing the control processes exemplified in Figures 12 and 36.

[0108] Memory 202 is, for example, a semiconductor memory and may include a RAM area and a ROM area. Storage device 203 is, for example, a hard disk, a semiconductor memory such as flash memory, or an external storage device.

[0109] The reader 204 accesses the removable storage medium 205, for example, according to instructions from the processor 201. The removable storage medium 205 can be implemented by, for example, a semiconductor device, a medium through which information is input / output by magnetic action, or a medium through which information is input / output by optical action. A semiconductor device is, for example, a USB (Universal Serial Bus) memory. A medium through which information is input / output by magnetic action is, for example, a magnetic disk. A medium through which information is input / output by optical action is, for example, a CD (Compact Disc)-ROM, DVD (Digital Versatile Disk), Blu-ray Disc, etc. (Blu-ray is a registered trademark).

[0110] The communication interface 206 communicates with other devices (e.g., a microscope 100, a webcam 500, etc.) according to instructions from the processor 201. The input / output interface 207 is, for example, an interface between an input device 400 and an output device. The input device 400 is, for example, a device that accepts instructions from the user, such as a mouse 401, a keyboard 402, or a foot switch 403. The output device is, for example, a monitor 300 and an audio device such as a speaker.

[0111] The program executed by processor 201 is provided to the computer in the following form, for example: (1) It is pre-installed on storage device 203. (2) Provided by a removable storage medium 205. (3) Provided from a server such as a program server.

[0112] The computer hardware configuration for realizing the control device described with reference to Figure 39 is illustrative, and the embodiment is not limited thereto. For example, some of the above configurations may be deleted, or new configurations may be added. In another embodiment, for example, some or all of the functions of the above-described processing unit may be implemented as hardware such as an FPGA (Field Programmable Gate Array), SoC (System-on-a-Chip), ASIC (Application Specific Integrated Circuit), and PLD (Programmable Logic Device).

[0113] The embodiments described above are specific examples provided to facilitate understanding of the invention, and the present invention is not limited to these embodiments. Modified forms of the embodiments described above and alternative forms that replace the embodiments described above may be included. In other words, each embodiment can be modified in terms of its components without departing from its spirit and scope. Furthermore, new embodiments can be implemented by appropriately combining multiple components disclosed in one or more embodiments. In addition, some components may be deleted from the components shown in each embodiment, or some components may be added to the components shown in the embodiments. Moreover, the processing procedures shown in each embodiment may be performed in a different order, as long as they do not contradict each other. That is, the microscope system, superposition unit, and operating method of the present invention can be modified in various ways without departing from the scope of the claims.

[0114] In the embodiment described above, an example was shown in which a control device 200 that controls the operation of the entire microscope system 1 generates auxiliary image data. However, the auxiliary image data may also be generated by a control device provided inside the eyepiece tube 120. That is, the eyepiece tube 120 attached to the microscope 100 may function as a superposition unit comprising a control device that generates auxiliary image data based on information about a target slide selected from a plurality of ordered slides included in a slide set, and a superposition device that superimposes the auxiliary image including the target slide onto the image plane where an optical image is formed, based on the auxiliary image data.

[0115] Furthermore, although not specifically mentioned in the embodiments described above, the image plane (AR screen) of the microscope optical system 110 onto which the auxiliary image is projected may be a screen that mirrors the contents of the slide displayed on the monitor 300, or a part of the video displayed on the monitor 300 may be projected onto the image plane as an auxiliary image. In addition, the mouse cursor displayed on the monitor 300 may be projected onto the auxiliary image as a pseudo-mouse cursor. The pseudo-mouse cursor does not need to be projected at all times; for example, if there is no cursor movement for a certain period of time, the pseudo-mouse cursor may be removed from the auxiliary image. This prevents the display of an unnecessary pseudo-mouse cursor from distracting the user. Also, the position of the pseudo-mouse cursor does not necessarily have to coincide with the position of the mouse cursor displayed on the monitor 300. For example, if the mouse cursor on the monitor 300 moves outside the field of view of the microscope optical system 110, the pseudo-mouse cursor may be displayed at the edge of the field of view. This prevents the user from losing sight of the pseudo-mouse cursor when it moves outside the field of view.

[0116] Furthermore, although not specifically mentioned in the embodiments described above, auxiliary images may be generated using a trained model created by machine learning such as deep learning. The slide set may include slides containing content (hereinafter referred to as AI content) obtained by inputting digital images acquired by the imaging device 112 into the trained model. Specifically, for example, by using a trained model that has learned defects in the sample to be inspected, defects in the sample may be detected from the digital image, and an auxiliary image B18 including a bounding box indicating the defect location may be output, as shown in Figure 40. In addition, the AI ​​content may be intended not only for inspection purposes but also for work support. For example, by using a trained model that has learned areas that require soldering, areas to be soldered may be detected from the digital image, and an auxiliary image B19 including a bounding box indicating the detected area may be output, as shown in Figure 41. The shape of the bounding box is not particularly limited and may be a rectangle as shown in Figure 40 or a circle as shown in Figure 41. The slide set may include multiple slides containing AI content corresponding to the work process, and the user may be provided with necessary information using various trained models by switching between multiple slides as appropriate.

[0117] In this specification, the expression "based on A" does not mean "based solely on A," but rather "based on at least A," and furthermore, "based at least partially on A." That is, "based on A" may also mean based on B in addition to A, or based on a part of A. [Explanation of symbols]

[0118] 1. Microscope System 10 slide set Slides 11-16 100 Microscopes 101 Objective lens 102, 102a, 102b zoom lenses 106, 106a, 106b eyepieces 110 Microscope Optical Systems 112 Imaging device 113 Projector 120 Eyepiece tube 130 Zoom Handle 200 Control device 201 Processor 202 memory 203 Storage device 204 Reader 205 Storage medium 206 Communication Interfaces 207 Input / Output Interfaces Bus 208 300 monitors 401 Mouse 402 keyboard 403 Footswitch 404 Barcode Reader 500 Webcams 601, 602 Remote terminals

Claims

1. A microscope optical system including an eyepiece, which forms an optical image of the sample on the object side of the eyepiece, A memory that stores a slide set consisting of multiple slides pre-ordered based on the user's work procedure performed under a microscope, A control device that generates auxiliary image data based on information about a target slide selected from the ordered plurality of slides included in the slide set, The system includes a superposition device that superimposes the auxiliary image, including the target slide, onto the image plane on which the optical image is formed, based on the auxiliary image data. In response to an instruction to switch the target slide, the control device selects a slide as the new target slide that is determined according to a first order based on the ordering of the plurality of slides stored in the memory. A microscope system characterized by the following features.

2. In the microscope system according to claim 1, The control device generates the auxiliary image data based on the magnification information of the microscope optical system and the information regarding the target slide. A microscope system characterized by the following features.

3. In the microscope system according to claim 2, The aforementioned target slide contains one or more pieces of content, The control device is The auxiliary image data is generated such that the content of the first category among the one or more contents is included in the auxiliary image at a size corresponding to the magnification information. The auxiliary image data is generated such that the content of the second category among the one or more contents is included in the auxiliary image at a predetermined size. A microscope system characterized by the following features.

4. In the microscope system according to claim 3, The contents of the first classification include at least one of a gauge and a reticle. A microscope system characterized by the following features.

5. In the microscope system according to claim 3 or claim 4, The one or more contents include at least one of the following: a pen, a shape, text, a still image, a moving image, a gauge, a reticle, image analysis results, a timer, a dashboard, external device information, and measurement content. A microscope system characterized by the following features.

6. In the microscope system according to any one of claims 1 to 5, further, The system includes an input device for inputting the instruction to switch the target slide. A microscope system characterized by the following features.

7. In the microscope system according to claim 6, The input device includes at least one of a mouse, keyboard, switch on a handle, foot switch, and microphone. A microscope system characterized by the following features.

8. A microscope system according to any one of claims 1 to 7, further, The system includes an acquisition device that acquires identification information to identify the slide set, The control device acquires the slide set based on the identification information acquired by the acquisition device. A microscope system characterized by the following features.

9. In the microscope system according to claim 8, The acquisition device includes at least one of a one-dimensional code reader, a two-dimensional code reader, an RF tag reader, and an imaging device. A microscope system characterized by the following features.

10. In the microscope system according to any one of claims 1 to 9, further, An imaging device for imaging the aforementioned sample, A recording device is provided, The control device causes the recording device to record the image of the sample captured by the imaging device in association with the auxiliary image. A microscope system characterized by the following features.

11. In the microscope system according to any one of claims 1 to 9, The control device generates annotation image data in response to a drawing instruction. The superposition device superimposes the annotation image corresponding to the annotation image data onto the image plane. A microscope system characterized by the following features.

12. In the microscope system according to claim 11, further, An imaging device for imaging the aforementioned sample, A recording device is provided, The control device causes the recording device to record the image of the sample captured by the imaging device in association with the auxiliary image and the annotation image. A microscope system characterized by the following features.

13. In the microscope system according to any one of claims 1 to 9, further, An imaging device for imaging the aforementioned sample, The control device selects a slide determined according to the analysis results of the image of the sample captured by the imaging device as the new target slide. A microscope system characterized by the following features.

14. In the microscope system according to any one of claims 1 to 13, further, The device includes an eyepiece tube for attaching the aforementioned eyepiece, The eyepiece barrel is equipped with an operating section for instructing the start or stop of superimposing the auxiliary image onto the image plane. A microscope system characterized by the following features.

15. A microscope system according to any one of claims 1 to 14, further, Equipped with a display device, The control device edits the slide set in response to operations performed on the editing screen displayed on the display device. A microscope system characterized by the following features.

16. A microscope system according to any one of claims 1 to 14, further, Equipped with a display device, The control device creates a new slide set in response to operations performed on the creation screen displayed on the display device. A microscope system characterized by the following features.

17. In the microscope system according to any one of claims 1 to 16, The aforementioned microscope optical system is an optical system for a stereomicroscope. A microscope system characterized by the following features.

18. In the microscope system according to any one of claims 1 to 17, The aforementioned slide set contains information to assist in the assembly of precision equipment. Each of the aforementioned slides contains information related to each step in the assembly process of the precision instrument. A microscope system characterized by the following features.

19. In the microscope system according to any one of claims 1 to 18, The aforementioned multiple slides include slides containing AI content obtained by inputting the images of the sample into a trained model. A microscope system characterized by the following features.

20. A superposition unit that is attached to a microscope equipped with a microscope optical system that forms an optical image of a sample on the object side of the eyepiece, A memory that stores a slide set consisting of multiple slides pre-ordered based on the work procedure of the user's work performed under the microscope, A control device that generates auxiliary image data based on information about a target slide selected from the ordered plurality of slides included in the slide set, The system includes a superposition device that superimposes the auxiliary image, including the target slide, onto the image plane on which the optical image is formed, based on the auxiliary image data. In response to an instruction to switch the target slide, the control device selects a slide as the new target slide that is determined according to a first order based on the ordering of the plurality of slides stored in the memory. A superimposed unit characterized by the following features.

21. A method for controlling a control device that controls a microscope equipped with a microscope optical system and superimposition device that form an optical image of a sample on the object side of the eyepiece, The control device, The control device reads out a slide set stored in its memory, which consists of a set of slides that have been pre-ordered based on the work procedure of the user's work performed under the microscope. In response to an instruction to switch a target slide selected from the ordered plurality of slides included in the slide set, the slide determined according to the first order based on the ordering of the plurality of slides stored in the memory is selected as the new target slide. Based on the aforementioned new information regarding the target slide, auxiliary image data is generated. The superimposing device superimposes the auxiliary image, including the new target slide, onto the image plane on which the optical image is formed, based on the auxiliary image data. A method that characterizes something.

Images