Display method and device for augmented reality image corresponding to microscope view

Abstract

A method for displaying an augmented reality (AR) image, comprising controlling an AR display to display the AR image as at least partially overlaid on a view of a microscope, the view being visible to a user of the microscope, the AR image including a cursor; receiving user input from a user interface; setting an anchor point based on the user input; updating the AR image to include the anchor point; controlling the AR display to display the updated AR image; detecting movement of a slide of the microscope; adjusting the AR image by moving at least one of the cursor and the anchor point within the AR image based on the detected movement; and controlling the AR display to display the adjusted AR image.

Description

Technical Field

[0001] The present disclosure relates to augmented reality. In particular, the present disclosure relates to displaying an augmented reality image corresponding to a microscope view. Background Art

[0002] Currently, most microscopists rely purely on their experience to give rough measurements, such as the size of a lesion or the number of cells. This approach is subjective and therefore inaccurate.

[0003] Some solutions suggest first taking digital photos from a microscope and then analyzing them using computer software. For example, CN109029247A and EP2473928B1 propose using software to measure photos taken by a special optical microscope.

[0004] However, in order to perform measurements on a computer, the user needs to switch between the microscope and the computer for operation, which is inefficient and time-consuming. Therefore, such a system is rarely adopted by users such as pathologists.

[0005] Also, because the measurements are taken on screenshots of the microscope view, the area that can be measured is limited by the microscope's field of view. If the tissue is too large to be captured in one screenshot, the system cannot measure it. Summary of the invention

[0006] According to one embodiment, a method for displaying an augmented reality (AR) image may include controlling an AR display to display the AR image as at least partially overlaid on a view obtained through a microscope, and the view is visible to a user of the microscope, wherein the AR image includes a cursor; receiving user input from a user interface; setting an anchor point based on the user input; updating the AR image to include the anchor point; controlling the AR display to display the updated AR image; detecting movement of a slide of the microscope; moving at least one of the cursor and the anchor point within the AR image based on the detected movement to adjust the AR image; and controlling the AR display to display the adjusted AR image.

[0007] According to one embodiment, a device for displaying an augmented reality (AR) image may include an AR display for displaying the AR image as at least partially overlaid on a view of a microscope, while the view is visible to a user of the microscope; at least one motion sensor for detecting movement of a slide of the microscope; a user interface for receiving user input; at least one memory for storing program code; and at least one processor for reading the program code and operating according to instructions of the program code, the program code including: a first display code for causing the at least one processor to control the AR display to display the AR image including a cursor; a receiving code for causing the at least one processor to receive from the user interface receiving the user input; setting code for causing the at least one processor to set an anchor point based on the user input; updating code for causing the at least one processor to update the AR image to include the anchor point; a second display code for causing the at least one processor to control the AR display to display the updated AR image; a detection code for causing the at least one processor to detect the movement of the slide from the motion sensor; an adjustment code for causing the at least one processor to move at least one of a cursor and an anchor point within the AR image based on the detected movement to adjust the AR image; and a third display code for causing the at least one processor to control the AR display to display the adjusted AR image.

[0008] According to one embodiment, a non-transitory computer-readable medium may store instructions, comprising: one or more instructions that, when executed by one or more processors of a device to display an augmented reality (AR) image, cause the one or more processors to: control an AR display to display the AR image as at least partially overlaid on a view of a microscope, the view being visible to a user of the microscope, wherein the AR image includes a cursor; receive user input from a user interface; set an anchor point based on the user input; update the AR image to include the anchor point; control the AR display to display the updated AR image; detect movement of a slide of the microscope; move at least one of a cursor and an anchor point within the AR image based on the detected movement to adjust the AR image; and control the AR display to display the adjusted AR image. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 is a schematic diagram of an exemplary implementation of an augmented reality microscope system of one embodiment.

[0010] Figure 2A-2B is a schematic diagram of the movement of a slide of an augmented reality microscope system according to an embodiment.

[0011] Figure 3is a flow chart of an exemplary method for displaying an augmented reality image.

[0012] Figure 4 is a schematic diagram of an exemplary environment in which the systems and / or methods described herein are applicable.

[0013] Figure 5 yes Figure 4 A diagram of exemplary components of one or more devices. DETAILED DESCRIPTION

[0014] Augmented reality (AR) may refer to a technology that superimposes computer-generated images on a user's view of the real world, thereby providing a composite view. Embodiments of the present disclosure relate to methods, devices, and systems that may allow a user to directly measure an object under a microscope using an AR image overlaid on a microscope view. According to various embodiments, a user may directly draw virtual objects such as points, lines, circles, rectangles, or areas of arbitrary shapes under a microscope. Various embodiments may allow a user to generate measurement data based on the user's drawing, such as the length of a line, the size of an area, an intensity histogram in an area, the type of area, or the number of objects (e.g., cells) in an area. Virtual objects may also be saved and used to emphasize some areas on a screenshot of a view observed through a microscope.

[0015] The various embodiments of the present disclosure allow users to take measurements and display the results directly under the microscope. The user can freely move the tissue during the measurement, thereby reducing the limitation of the field of view. The user can also use the system to plot and highlight areas of interest under the microscope.

[0016] Figure 1 1 is a schematic diagram of an embodiment of an AR microscope system 100. According to various embodiments, the AR microscope system 100 may include an image sensor, such as a camera 102, which can capture a microscope image of a view observed through a microscope 104. The computer 108 can calculate and generate graphics, such as an AR image 110, based on the microscope image captured by the camera 102. The AR device 112 can overlay the computer-generated AR image 110 on the view of the microscope 104. The user can use the AR image 110 as an AR overlay view 116 with the microscope to observe the slide 114. In this way, a large amount of information can be presented through the microscope 104. In addition, the user can interact with the computer 108 through voice control, which can be performed through a microphone 118 or a physical input device (including but not limited to a button 120 or a keyboard, a mouse, or a joystick).

[0017] When taking measurements, a virtual cursor 122 will be displayed as a point at the center of the AR device 112. The user can use this virtual cursor 122 to place an anchor point 124. The virtual cursor 122 can be similar to, for example, a cursor displayed on a computer monitor. However, in embodiments, as the slide 114 is moved, the virtual cursor 122 can stay in the middle of the view observed through the microscope 104. This can be consistent with the operation of the microscope 104. In various embodiments, the slide 114 can be placed on top of a stage 126, and the operator moves the stage 126 back and forth to observe different parts of the slide 114. When the slide 114 is moved, all previous plots (e.g., anchor points 124) can move with the slide.

[0018] In various embodiments, the AR microscope system 100 may include a microscope 104 that can zoom and view a slide 114. When the objective is turned on, the microscope 104 may send a signal to the computer 108 to inform the computer 108 that the objective of the microscope 104 is currently in use, or the computer 108 sends a request signal for this information. The AR microscope system 100 may include an image sensor, such as a digital camera 102, which may be mounted on a lens tube of the microscope 104 to capture an image 106 of the view of the microscope 104. The AR microscope system 100 may include an AR device 112, such as an AR lens, which may be mounted on an eyepiece portion of the microscope 104 for overlaying a virtual screen such as an AR image 110 on the slide view. The AR microscope system 100 may include a computer 108 for providing image calculations and data storage.

[0019] According to various embodiments, the AR microscope system 100 may include a microphone 118, which may be a voice input device for a user to control the AR microscope system 100. The AR microscope system 100 may also include other inputs, which may be physical input devices for a user to control the AR microscope system 100, such as a button 120, or a keyboard, a mouse, or a joystick.

[0020] According to various embodiments, the AR microscope system 100 may include software and a graphical user interface. For example, the motion tracking algorithm of the AR microscope system 100 tracks the movement of the slide 114 under the microscope 104. The AR microscope system 100 may include a virtual layer, such as the layer shown in the AR image 110, which can move with the slide 114 and record virtual objects, such as anchor points 124 drawn by the user. The AR microscope system 100 may include a virtual cursor 122 placed in the center of the view. The AR microscope system 100 may include a user interface that enables the user to draw under the microscope 104, such as by drawing lines 128. The AR microscope system 100 may include a user interface that can display measurement results 130 under the microscope 104 in an interactive manner. The AR microscope system 100 may include a storage system that can save images and videos taken by the camera 102. The storage system can save virtual drawings made by the user, such as shown in the AR image 110. The AR microscope system 100 may include computer software that allows the user to load the saved images and videos together with the virtual drawings. The AR microscope system 100 may include a printing service that can print captured photos with or without overlaid virtual plots.

[0021] like Figure 2A-2B As shown, the user can experience the operation of the AR microscope system 100 as if there is a virtual screen, such as the AR image 110, which moves with the slide 114. All plots, such as the anchor points 124 and the lines 128, can be plotted on the virtual screen. For example, Figure 2A 1 shows an example of the slide 114, the AR image 110, and the AR overlay view 116 before the slide 114 is moved in the indicated direction of movement, Figure 2B An example of the slide 114 , the AR image 110 , and the AR overlay view 116 is shown after the slide 114 has been moved in the indicated direction of movement.

[0022] According to one embodiment, the computer 108 can track the movement of the slide 114. For example, the camera 102 can record the microscope view in a series of frames. These frames are transmitted to the computer 108. The computer 108 can compare adjacent frames and detect and calculate the movement of the slide 114 from the previous frame. To current frame displacement.

[0023] One possible way to calculate such motion or displacement is the following steps:

[0024] 1) Calculate the frame Key points on ,

[0025] 2) Find out Corresponding points in the current frame: ,

[0026] 3) Calculation and Distance between:

[0027] ,

[0028] 4) Use The virtual screen is shifted and the graphical depiction included in the AR image 110 of the AR display 112 is updated.

[0029] In various embodiments, the motion tracking algorithm may also be implemented in other ways. For example, the motion tracking algorithm may include an optical flow algorithm or an end-to-end deep learning twin network or a multi-model ensemble for estimating the motion between two adjacent frames to improve accuracy. In addition, the motion tracking algorithm may consider motion across multiple previous frames to improve temporal stability.

[0030] According to various embodiments, a user may draw a virtual plot, such as a line 128 on the AR image 110. For example, the line 128 may include a ruler, an arrow, or any other desired visual representation. In one embodiment, the user may start using a voice command (e.g., saying "draw a line") or a keyboard command (e.g., pressing the key "l") or by pressing the button 120 in a predetermined pattern (e.g., single click). After starting, the AR microscope system 100 may record the current position of the virtual cursor 122 by placing the anchor point 124 on the virtual screen 110 accordingly. When the stage 114 is moved, the line 128 may be plotted between the virtual cursor 122 and the anchor point 124. If the user selects the ruler function, measurement data, such as the length of the line, may be displayed next to the virtual cursor 122. If the user selects the arrow function, an arrow pointing from the virtual cursor 122 to the anchor 124 may be displayed. The user may complete and save the plotting by a voice command (e.g., saying "save") or a keyboard command (e.g., pressing the key "s") or by pressing the button 120 in a predetermined pattern (e.g., double clicking). The user can complete and delete the drawing by voice command (eg, saying "stop") or keyboard command (eg, pressing key "x") or by pressing button 120 in a predetermined pattern (eg, long press).

[0031] To make the measurements, the AR microscope system 100 can take into account the physical size of the pixels of the camera 102 and the physical size of the objective lens. This can be set manually, or can be automatically corrected by a standard grid ruler slide together with an automatic correction program. The grid ruler can be a slide with a checkerboard-like grid pattern. Each square on the detection plate can have a fixed size, such as 10 microns. The automatic correction program can detect those grids and count the number of pixels within the grid, thereby automatically estimating the physical size of the pixels.

[0032] According to various embodiments, a user may draw a virtual polygon on the AR image 110. In one embodiment, the user may start using a voice command (e.g., saying "draw a polygon") or a keyboard command (e.g., pressing the key "p") or pressing a button in a predetermined pattern (e.g., double clicking). After starting, the AR microscope system 100 may record the current position of the virtual cursor 122 by placing an anchor point 124 on the virtual screen 110 accordingly. When the stage 114 is moved, a line 128 may be drawn between the virtual cursor 122 and the anchor point 124. The user may place a new anchor point at the location of the current virtual cursor by a voice command (e.g., saying "next") or a keyboard command (e.g., pressing the key "k") or pressing the button 120 in a predetermined pattern (e.g., single clicking). When the user does this, a fixed line may be drawn between the new anchor point and the previous anchor point, and another line may be drawn starting from the new anchor point to the virtual cursor 122. The user can complete and save the drawing by voice command (e.g., saying "save") or keyboard command (e.g., pressing key "s") or by pressing button 120 in a predetermined pattern (e.g., double-clicking). When the user does this, a line can be drawn between the first cursor and the last cursor, thereby generating a closed polygon. After the user is finished, the area size of the polygon can be displayed next to the polygon on the virtual screen 110. The user can complete and delete the drawing by voice command (e.g., saying "stop") or keyboard command (e.g., pressing key "x") or by pressing button 120 in a predetermined pattern (e.g., long press).

[0033] According to various embodiments, the user can draw a sphere or a rectangular figure on the AR image 110. In one embodiment, the user can start using a voice command (e.g., saying "draw a sphere") or a keyboard command (e.g., pressing the key "o") or pressing the button 120 in a predetermined pattern (e.g., long press). After starting, the AR microscope system 100 can record the current position of the virtual cursor 122 by placing the anchor point 124 on the virtual screen 110 accordingly. When the stage 114 is moved, a sphere can be drawn by taking the anchor point 124 as the center and the line 128 between the virtual cursor 122 and the anchor point 124 as the radius. If the user selects the rectangle function, a rectangle can be drawn by taking the line 128 between the virtual cursor 122 and the anchor point 124 as the diagonal line. The user can complete and save the drawing by a voice command (e.g., saying "save") or a keyboard command (e.g., pressing the key "s") or pressing the button 120 in a predetermined pattern (e.g., double-clicking). The user can complete and delete the drawing by voice command (eg, saying "stop") or keyboard command (eg, pressing key "x") or by pressing button 120 in a predetermined pattern (eg, long press).

[0034] According to various embodiments, a user may take measurements. Measurement data, such as measurement data 130, may be displayed during plotting or after plotting is complete. A user may turn measurements on or off, or select displayed measurement data through voice or physical controls. The types of measurement data available include, but are not limited to:

[0035] 1) Length: line length, circle radius, rectangle diagonal

[0036] 2) Size: The size of the object

[0037] 3) Color distribution: color mean / standard deviation, color histogram, frequency distribution

[0038] 4) Cell counting: the number of cells in the area

[0039] 5) Tissue count: The percentage of different types of tissues in the region

[0040] Accordingly, the AR microscope system 100 can allow a user to interactively plot under a microscope to directly highlight and / or measure areas under the microscope that cannot be done with a traditional microscope. The AR microscope system 100 can allow a user to measure objects under the microscope, even when the object is too large to be fully photographed under the microscope, but current microscope software cannot implement these functions. In addition, the AR microscope system 100 can allow a user to save and / or print an AR image together with the captured image, such as a plot performed on the image, which helps to record data for use in future inspections.

[0041] Figure 3 is a flow chart of an exemplary method 300 for displaying an augmented reality (AR) image. In some implementations, Figure 3 One or more of the flow blocks may be executed by computer 108 or platform 420, as described below. In some implementations, Figure 3 One or more flow blocks may be performed by another device or group of devices independent of or including the computer 108 or platform 420, such as the microscope 104, camera 102, AR device 112, or user device 410 described below.

[0042] like Figure 3 As shown, method 300 may include controlling an AR display to display an AR image at least partially overlaid on a view of the microscope while the view is visible to a user of the microscope, wherein the AR image includes a cursor (block 310 ).

[0043] like Figure 3 As further shown, method 300 may include receiving user input from a user interface (block 320 ).

[0044] like Figure 3 As further shown, method 300 may include setting an anchor point based on user input (block 330 ).

[0045] like Figure 3 As further shown, the method 300 may include updating the AR image to include the anchor point (block 340 ).

[0046] like Figure 3 As further shown, the method 300 may include controlling the AR display to display the updated AR image (block 350 ).

[0047] like Figure 3 As further shown, method 300 may include detecting movement of a microscope slide (block 360 ).

[0048] like Figure 3 As further shown, if there is no motion (block 360 —NO), the method 300 may return to block 310 .

[0049] like Figure 3 As further shown, if motion is detected (block 360 —Yes), method 300 may include moving at least one of a cursor and an anchor point within the AR image based on the detected motion, thereby adjusting the AR image (block 370 ).

[0050] like Figure 3 As further shown, the method 300 may include controlling the AR display to display the adjusted AR image (block 380 ).

[0051] According to one embodiment, the adjusted AR image may include a line displayed between the anchor point and the cursor.

[0052] According to one embodiment, the method 300 may further include defining a region within the AR image based on the line.

[0053] According to one embodiment, the anchor point may be one of a plurality of anchor points, the line may be one of a plurality of lines determined based on the plurality of anchor points, and the area may be defined by the plurality of lines.

[0054] According to one embodiment, the area may include a rectangle, and the line may be a diagonal line of the rectangle.

[0055] According to one embodiment, the area may include a circle, the cursor may be located at the center of the circle, the anchor point may be located on the circumference of the circle, and the line may be the radius of the circle.

[0056] According to one embodiment, the adjusted AR image may include measurement data determined based on the line, wherein the measurement data may include at least one of: the length of the line, the size of the area, the color distribution of the colors included in the area, the color histogram of the colors included in the area, the frequency distribution of the colors included in the area, the number of cells included in the area, and the percentage of the area that may include a predetermined type of tissue.

[0057] According to one embodiment, the user input may include at least one of a button press, a keyboard input, a mouse input, and a voice input.

[0058] According to one embodiment, method 300 may further include capturing multiple image frames of the microscope view, wherein the distance of movement and the direction of movement may be determined based on a difference between a first point in a first image frame among the multiple image frames and a second point in a second image frame among the multiple image frames, the second point corresponding to the first point, and the adjusted AR image may be adjusted based on the determined distance and the determined direction.

[0059] According to one embodiment, the method 300 may further include storing an image including the microscope view at least partially overlaid with the adjusted AR image.

[0060] although Figure 3 An exemplary step block of method 300 is shown. In some implementations, Figure 3 Compared with the step blocks depicted in , method 300 may include more blocks, fewer blocks, different blocks, or blocks in a different order. Additionally or alternatively, two or more blocks of method 300 may be executed in parallel.

[0061] Figure 4 4 is a schematic diagram of an exemplary environment 400 in which the systems and / or methods described herein may be used. Figure 4 As shown, environment 400 may include user device 410, platform 420, and network 430. For example, user device 410 may correspond to various components, such as microscope 104, camera 102, and AR device 112. In addition, platform 420 may correspond to computer 108. The devices of environment 400 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0062] User device 410 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with platform 420. For example, user device 410 may include a computing device (e.g., a desktop computer, a laptop computer, a tablet computer, a handheld computer, a smart speaker, a server, etc.), a mobile phone (e.g., a smart phone, a wireless phone, etc.), a wearable device (e.g., smart glasses or a smart watch), or a similar device. In some implementations, user device 410 may receive information from platform 420 and / or send information to platform 420.

[0063] The platform 420 includes one or more devices capable of generating an AR image that is displayed as an overlay on a microscope view, as described elsewhere herein. In some implementations, the platform 420 may include a cloud server or a group of cloud servers. In some implementations, the platform 420 may be designed to be modular so that certain software components can be swapped in or out according to specific needs. Therefore, the platform 420 can be easily and / or quickly reconfigured for different purposes.

[0064] In some implementations, as shown, platform 420 can be hosted in a cloud computing environment 422. It is worth noting that although the implementations described herein describe platform 420 as being hosted in a cloud computing environment 422, in some implementations, platform 420 is not cloud-based (i.e., can be implemented outside of a cloud computing environment) or can be partially cloud-based.

[0065] Cloud computing environment 422 includes an environment hosting platform 420. Cloud computing environment 422 can provide computing, software, data access, storage, and other services that do not require end users (e.g., user devices 410) to be aware of the physical location and configuration of the systems and / or devices hosting platform 420. As shown, cloud computing environment 422 can include a set of computing resources 424 (collectively referred to as "various computing resources 424", and a single resource is referred to as "computing resource 424").

[0066] The computing resources 424 include one or more personal computers, workstation computers, server devices, or other types of computing and / or communication devices. In some implementations, the computing resources 424 can host the platform 420. Cloud resources can include computing instances executed in the computing resources 424, storage devices provided in the computing resources 424, data transmission devices provided by the computing resources 424, etc. In some implementations, the computing resources 424 can communicate with other computing resources 424 via wired connections, wireless connections, or a combination of wired and wireless connections.

[0067] like Figure 4As further shown, the computing resources 424 include a set of cloud resources, such as one or more applications ("APP") 424-1, one or more virtual machines ("VM") 424-2, virtualized storage ("VS") 424-3, one or more hypervisors ("HYP") 424-4, and the like.

[0068] Applications 424-1 include one or more software applications that can be provided to or accessed by user device 410. Applications 424-1 can eliminate the need for user device 410 to install and execute software applications. For example, applications 424-1 can include software associated with platform 420 and / or any other software that can be provided via cloud computing environment 422. In some implementations, one application 424-1 can send / receive information to / from one or more other applications 424-1 via virtual machine 424-2.

[0069] Virtual machine 424-2 includes a software implementation of a machine (e.g., a computer) that executes programs like a physical machine. Virtual machine 424-2 can be a system virtual machine or a process virtual machine, depending on the use and correspondence of virtual machine 424-2 to any real machine. System virtual machines can provide a complete system platform that supports the execution of a complete operating system ("OS"). Process virtual machines can execute a single program and can support a single process. In some implementations, virtual machine 424-2 can execute on behalf of a user (e.g., user device 410) and manage the infrastructure of cloud computing environment 422, such as data management, synchronization, or long-term data transfer.

[0070] Virtualized storage 424-3 includes one or more storage systems and / or one or more devices using virtualization technology within the storage system or device of computing resource 424. In some implementations, in the context of a storage system, the types of virtualization may include block virtualization and file virtualization. Block virtualization may refer to abstracting (or separating) logical storage from physical storage so that the storage system may be accessed without regard to physical storage or heterogeneous structures. Separation may allow administrators of storage systems to flexibly manage storage for end users. File virtualization may eliminate the dependency between file-level data access and the physical storage location of the file. This may optimize the performance of storage usage, server consolidation, and / or non-destructive file migration.

[0071] Hypervisor 424-4 may provide hardware virtualization technology that allows multiple operating systems (e.g., "guest operating systems") to execute concurrently on a host such as computing resource 424. Hypervisor 424-4 may provide a virtual operating platform for the guest operating systems and may manage the execution of the guest operating systems. Multiple instances of various operating systems may share virtualized hardware resources.

[0072] The network 430 includes one or more wired and / or wireless networks. For example, the network 430 may include a cellular network (e.g., a fifth generation (5G) network, a long term evolution (LTE) network, a third generation (3G) network, a code division multiple access (CDMA) network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, etc., and / or a combination of these or other types of networks.

[0073] Figure 4 The number and arrangement of devices and networks shown are provided as examples. Figure 4 There may be more devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged differently than those shown. Figure 4 Two or more of the devices shown may be implemented in a single device, or Figure 4 The single device shown may be implemented by multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 200 may perform one or more functions described as being performed by another set of devices of environment 200.

[0074] Figure 5 4 is a schematic diagram of exemplary components of device 300. Device 300 may correspond to user device 410 and / or platform 420. Figure 5 As shown, device 300 may include a bus 510 , a processor 520 , a memory 330 , a storage component 540 , an input component 550 , an output component 560 , and a communication interface 570 .

[0075] The bus 510 includes components that allow communication between components of the device 300. The processor 520 is implemented in hardware, firmware, or a combination of hardware and software. The processor 520 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or other types of processing components. In some implementations, the processor 520 includes one or more processors that can be programmed to perform a function. The memory 330 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic storage, and / or optical storage) that stores information and / or instructions for use by the processor 520.

[0076] Storage component 540 stores information and / or software related to the operation and use of device 300. For example, storage component 540 may include a hard disk (e.g., a magnetic disk, an optical disk, a magneto-optical disk, and / or a solid-state disk), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium and a corresponding drive.

[0077] Input components 550 include components that allow device 300 to receive information, such as through user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 550 may include sensors for sensing information (e.g., a global positioning system (GPS) component, an accelerometer, a gyroscope, and / or an actuator). Output components 560 include components that provide output information of device 300 (e.g., a display, a speaker, and / or one or more light emitting diodes (LEDs)).

[0078] The communication interface 570 includes transceiver-like components (e.g., a transceiver and / or a separate receiver and transmitter) that enable the device 300 to communicate with other devices, such as through a wired connection, a wireless connection, or a combination of wired and wireless connections. The communication interface 570 can allow the device 300 to receive information from another device and / or provide information to another device. For example, the communication interface 570 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0079] Device 300 can perform one or more methods described herein. Device 300 can perform these methods in response to processor 520 executing software instructions, which can be stored in a non-transitory computer-readable storage medium, such as memory 330 and / or storage component 540. Computer-readable media is defined herein as a non-transitory storage device. A storage device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0080] The software instructions may be read into the memory 330 and / or storage component 540 from another computer-readable medium or from another device via the communication interface 570. When executed, the software instructions stored in the memory 330 and / or storage component 540 may cause the processor 520 to perform one or more methods described herein. Additionally, or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more methods described herein. Thus, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0081] Figure 5The number and arrangement of devices and networks shown are provided as examples. Figure 5 Compared to the components shown, the device 300 may include additional components, fewer components, different components, or components in a different deployment. Additionally or alternatively, a set of components (e.g., one or more components) of the device 300 may perform one or more functions described as being performed by another set of components of the device 300.

[0082] The foregoing disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the specific forms disclosed. Modifications and variations of the implementation may be obtained based on the foregoing disclosure or from the practice of these implementations.

[0083] As used herein, the term component should be broadly interpreted as hardware, firmware, or a combination of hardware and software.

[0084] It is apparent that the systems and / or methods described herein may be implemented by various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods is not limited by these implementations. Therefore, the operation and behavior of these systems and / or methods are described herein without reference to specific software code—it should be understood that software and hardware can be designed based on the description herein to implement these systems and / or methods.

[0085] Even though specific combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of possible implementations. In fact, many of these features may be combined in ways not specifically described in the claims and / or disclosed in the specification. Although each dependent claim listed below may only be directly dependent on one claim, the disclosure of possible implementations includes the combination of each dependent claim with every other claim in the claim set.

[0086] Unless explicitly stated, any element, action or instruction used in this article should not be interpreted as critical or necessary. In addition, as used herein, the articles "a" and "an" are intended to include one or more items and can be used interchangeably with "one or more". In addition, as used herein, the term "set" is intended to include one or more items (e.g., related items, unrelated items, combinations of related items and unrelated items, etc.), and can be used interchangeably with "one or more". In the case where only one item is desired, the term "one" or similar language is used. In addition, as used herein, the terms "has", "have", "having", etc. are intended to be open terms. Further, unless otherwise explicitly stated, the phrase "based on" means "based at least in part on".

Claims

1. A method for displaying an augmented reality (AR) image, characterized in that: include: controlling the AR display to display an AR image at least partially overlaid on a view of the microscope, the view being visible to a user of the microscope, the AR image including a cursor; receiving user input from a user interface; setting an anchor point based on the user input; Updating the AR image to include the anchor point; Controlling the AR display to display the updated AR image; detecting movement of a slide of the microscope; adjusting the AR image by moving at least one of the cursor and the anchor point within the AR image based on the detected motion; and Controlling the AR display to display the adjusted AR image; wherein the adjusted AR image includes a line displayed between the anchor point and the cursor; The method further comprises: taking a plurality of image frames of the view of the microscope, Wherein, the distance of the movement and the direction of the movement are determined based on the difference between a first point in a first image frame among the multiple image frames and a second point in a second image frame among the multiple image frames, the second point corresponds to the first point, and the adjusted AR image is adjusted based on the determined distance and the determined direction.

2. The method according to claim 1, characterized in that Further comprising defining a region within the AR image based on the line.

3. The method according to claim 2, characterized in that The anchor point is one of a plurality of anchor points, The line is one of a plurality of lines determined based on the plurality of anchor points, and wherein the area is defined by the plurality of lines.

4. The method according to claim 2, characterized in that: The region comprises a rectangle, and The lines are the diagonals of the rectangle.

5. The method according to claim 2, characterized in that: The area includes a circle, The cursor is located at the center of the circle. The anchor point is located on the circumference of the circle, and The line is the radius of the circle.

6. The method according to claim 2, characterized in that The adjusted AR image includes measurement data determined based on the line, The measurement data includes at least one of the following: the length of the line, the size of the area, the color distribution of each color included in the area, the color histogram of the colors included in the area, the frequency distribution of the colors included in the area, the number of cells included in the area, and the area percentage of a predetermined type of tissue.

7. The method according to claim 1, characterized in that The user input includes at least one of button pressing, keyboard input, mouse input, and voice input.

8. The method according to claim 1, characterized in that: Further included is storing an image including a view of the microscope at least partially overlaid with the adjusted AR image.

9. A device for displaying augmented reality (AR) images, characterized in that: include: an AR display for displaying the AR image at least partially overlaid on a view of the microscope, the view being visible to a user of the microscope; at least one motion sensor for detecting movement of a slide of the microscope; A user interface for receiving user input; at least one memory for storing program code; and At least one processor is configured to read the program code and perform the following according to instructions of the program code: Controlling the AR display to display the AR image, where the AR image includes a cursor; receiving the user input from the user interface; setting an anchor point based on the user input; Updating the AR image to include the anchor point; Controlling the AR display to display the updated AR image; detecting the motion of the slide from the motion sensor; Based on the detected motion, adjusting the AR image by moving at least one of the cursor and the anchor point within the AR image; as well as Controlling the AR display to display the adjusted AR image; wherein the adjusted AR image includes a line displayed between the anchor point and the cursor; wherein the at least one motion sensor comprises at least one image sensor for capturing a plurality of image frames of a view of the microscope, Wherein, the distance of the movement and the direction of the movement are determined based on the difference between a first point in a first image frame among the multiple image frames and a second point in a second image frame among the multiple image frames, the second point corresponds to the first point, and the adjusted AR image is adjusted based on the determined distance and the determined direction.

10. The device according to claim 9, characterized in that The program code further causes the at least one processor to define an area within the AR image based on the line.

11. The device according to claim 10, characterized in that The anchor point is one of a plurality of anchor points, The line is one of a plurality of lines determined based on the plurality of anchor points, and The area is defined by the plurality of lines.

12. The device according to claim 10, characterized in that The region comprises a rectangle, and The lines are the diagonals of the rectangle.

13. The device according to claim 10, characterized in that The area includes a circle, The cursor is located at the center of the circle. The anchor point is located on the circumference of the circle, and The line is the radius of the circle.

14. The device according to claim 10, characterized in that The adjusted AR image includes measurement data determined based on the line, The measurement data includes at least one of the following: the length of the line, the size of the area, the color distribution of the colors included in the area, the color histogram of the colors included in the area, the frequency distribution of the colors included in the area, the number of cells included in the area, and the area percentage of a predetermined type of tissue.

15. The device according to claim 9, characterized in that The program code further causes the at least one processor to store an image including a view of the microscope at least partially overlaid with the adjusted AR image.

16. A non-transitory computer-readable medium storing instructions, characterized in that: The method comprises one or more instructions, which, when executed by one or more processors of a device to display an augmented reality (AR) image, cause the one or more processors to perform the method according to any one of claims 1 to 8.

Images