Apparatus and method for slide imaging

Abstract

To solve the problem in which the cost of out-of-focus errors in scanning of a slide can be high, particularly in the time that is lost to make a decision based on that slide.SOLUTION: An exemplary apparatus for real time image generation includes at least one optical system; a slide port configured to hold a slide; an actuator mechanism mechanically connected to a mobile element; a user interface; and at least one processor configured to receive at least one region of interest, capture, using the at least one optical system, a first image of the slide at a first position within the at least one region of interest, identify a focus pattern as a function of the first image and the first position, extrapolate a focal distance for a second position as a function of the focus pattern, and capture, using the at least one optical system, a second image of the slide at a second position and at the focal distance.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates generally to the field of medical imaging. In particular, the present invention relates to an apparatus and method for slide imaging. [Background technology]

[0002] In slide scanning, errors due to out-of-focus can be costly, especially if decisions are made based on the slide. A single-pass approach to slide digitization necessarily requires determining the Z plane of best focus during a single scan, and therefore forces decisions in the presence of tissue, dust, pen marks, etc. on the slide. For example, if dust is encountered, the scan will select the wrong Z reference plane and will not focus on the tissue beyond the dust. Similarly, pen marks will cause tissue to be completely out of focus, because the Z plane of the pen mark is above the cover slip. Summary of the Invention [Means for solving the problem]

[0003] In one aspect, an apparatus for imaging a slide may include at least one optical system including an optical sensor, a slide port configured to hold a slide, at least one processor, and a memory communicatively connected to the at least one processor, the memory storing instructions to configure the at least one processor to receive at least one region of interest, capture a first image of the slide at a first location within the at least one region of interest using the at least one optical system, identify a focal pattern as a function of the first image and the first location, extrapolate a focal length of a second location as a function of the focal pattern, and capture a second image of the slide at a second location and the focal length using the at least one optical system.

[0004] In another aspect, a method of imaging a slide may include receiving at least one region of interest using at least one processor; capturing a first image of the slide at a first location within the at least one region of interest using at least one processor and at least one optical system; identifying a focal pattern as a function of the first image and the first location using at least one processor; extrapolating a focal length of a second location as a function of the focal pattern using at least one processor; and capturing a second image of the slide at the second location and the focal length using at least one processor and the at least one optical system.

[0005] These and other aspects and features of non-limiting embodiments of the present invention will become apparent to those of ordinary skill in the art upon review of the following description of specific non-limiting embodiments of the present invention in conjunction with the accompanying drawings. [Brief description of the drawings]

[0006] For the purpose of illustrating the invention, the drawings show aspects of one or more embodiments of the invention. It is to be understood, however, that the invention is not limited to the precise arrangements and instrumentalities shown in the drawings. [Figure 1] FIG. 1 illustrates an exemplary apparatus for slide imaging. [Diagram 2] FIG. 1 illustrates an exemplary machine learning model. [Diagram 3] FIG. 1 illustrates an exemplary neural network. [Figure 4] FIG. 2 illustrates an exemplary neural network node. [Diagram 5] FIG. 1 illustrates an exemplary method of slide imaging. [Figure 6A] FIG. 1 illustrates a slide including various features, identifying a region of interest, and scanning lines within the region of interest. [Figure 6B]FIG. 1 illustrates a slide including various features, identifying a region of interest, and scanning lines within the region of interest. [Figure 6C] FIG. 1 illustrates a slide including various features, identifying a region of interest, and scanning lines within the region of interest. [Figure 7] FIG. 1 illustrates an exemplary method of slide imaging. [Figure 8] FIG. 1 is a block diagram of a computing system that may be used to implement any one or more of the methodologies and any one or more portions thereof disclosed herein. The drawings are not necessarily to scale and may be illustrated with phantom lines, schematic representations, and fragmentary views. In certain instances, details that are not necessary for an understanding of the embodiments or that make other details difficult to perceive may be omitted. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0007] At a high level, aspects of the disclosure relate to devices and methods for slide imaging. The devices described herein can generate images of slides and / or samples on slides. In one embodiment, the device can capture a first image and identify one or more regions of interest. The regions of interest can include features such as writing, dust, samples, etc. The samples can include tissue. In one embodiment, the device can identify a focus pattern of the region of interest. For example, the device can identify the focus pattern as a plane based on a number of points at which best focus is determined. In one embodiment, the device can determine which regions include a sample. In some embodiments, which regions include a sample can be determined after one or more other steps described herein. In some embodiments, delaying the determination of which regions include a sample can make the slide imaging process more efficient. For example, efficiency can be improved because it is difficult to run sophisticated models on a scanning device during scanning. In another example, efficiency can be improved because the risk of false positives is minimized when classifying a region as dust or annotation and skipping scanning the region. In other instances, it may be useful to scan for annotations, and therefore performing the steps in this order may be optimal.Exemplary embodiments illustrating aspects of the present disclosure are described below in the context of several illustrative examples.

[0008] Referring now to FIG. 1, an exemplary embodiment of an apparatus 100 for slide imaging is illustrated. The apparatus 100 may include a computing device. The apparatus 100 may include a processor 104. The processor 104 may include, but is not limited to, any processor 104 described in this disclosure. The processor 104 may be included in a computing device. The apparatus 100 may include at least one processor 104 and a memory 108 communicatively coupled to the at least one processor 104, the memory 108 storing instructions 112 that configure the at least one processor 104 to perform one or more processes described herein. The computing device may include any computing device described in this disclosure, including but not limited to a microcontroller, a microprocessor, a digital signal processor (DSP), and / or a system on a chip (SoC) described in this disclosure. The computing device may include, be included in, and / or communicate with a mobile device, such as a mobile phone or a smartphone. A computing device may include a single computing device operating independently, or may include two or more computing devices operating in cooperation, parallel, serially, etc., and the two or more computing devices may be included together in a single computing device or may be included in two or more computing devices. A computing device may interface or communicate with one or more additional devices, as described in more detail below, via a network interface device. A network interface device may be utilized to connect a computing device to one or more of various networks and one or more devices. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof.Examples of networks include, but are not limited to, wide area networks (e.g., the Internet, corporate networks), local area networks (e.g., networks associated with an office, building, campus, or other relatively small geographic space), telephone networks, data networks associated with a telephone / voice provider (e.g., a mobile communications provider's data and / or voice network), a direct connection between two computing devices, and any combination thereof. Networks may employ wired and / or wireless communication modes. In general, any network topology may be used. Information (e.g., data, software, etc.) may be communicated to and / or from computers and / or computing devices. Computing devices may include, but are not limited to, a computing device or cluster of computing devices at a first location and a second computing device or cluster of computing devices at a second location. Computing devices may include one or more computing devices specialized for data storage, security, traffic distribution for load balancing, etc. A computing device may distribute one or more computing tasks, as described below, across multiple computing devices that may operate in parallel, serially, redundantly, or any other manner used to distribute tasks or memory among computing devices. As a non-limiting example, a computing device may be implemented using a "shared nothing" architecture.

[0009] With continued reference to FIG. 1 , a computing device may be designed and / or configured to repeatedly perform any method, method step, or sequence of method steps in any embodiment described in this disclosure, in any order, and to any degree. For example, a computing device may be configured to repeatedly perform a single step or sequence until a desired or ordered result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively using the output of a previous iteration as input for a subsequent iteration, aggregation of the inputs and / or outputs of an iteration to generate an aggregate result, reduction or decrement of one or more variables, such as global variables, and / or division of a large processing task into a set of smaller processing tasks that are addressed iteratively. A computing device may perform any step or sequence of steps described in this disclosure in parallel, such as performing a step two or more times simultaneously and / or nearly simultaneously, using two or more parallel threads, processor cores, etc., and the division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize upon reviewing this disclosure in its entirety the various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise processed using iterative, recursive, and / or parallel processing.

[0010] Still referring to FIG. 1, "communicatively connected" as used in this disclosure means connected by a connection, attachment, or coupling that allows for the receipt and / or transmission of information between two or more entities. For example, but not limited to, the connection may be wired or wireless, direct or indirect, and between two or more components, circuits, devices, systems, etc., allowing for the receipt and / or transmission of data and / or signals. The data and / or signals therebetween may include, but are not limited to, among others, electric, electromagnetic, magnetic, visual, audio, radio and microwave data and / or signals, combinations thereof, and the like. The communication connection may be achieved, for example, but not limited to, by wired or wireless electronic, digital or analog communication, directly or through one or more intervening devices or components. Furthermore, the communication connection may include electrically coupling or connecting at least one output of one device, component or circuit to at least one input of another device, component or circuit. For example, but not limited to, it may be via a bus or other facility for mutual communication between computing device elements. A communicative connection may also include an indirect connection via, for example, but not limited to, a wireless connection, a wireless communication, a low power wide area network, optical communication, magnetic coupling, capacitive coupling, optical coupling, etc. In some cases, the term "communicatively coupled" may be used in this disclosure instead of communicatively connected.

[0011] Still referring to FIG. 1 , in some embodiments, the device 100 may be used to generate images of the slide 116 and / or a sample on the slide 116. As used herein, a "slide" is a container or surface that holds a sample of interest. In some embodiments, the slide 116 may include a glass slide. In some embodiments, the slide 116 may include a formalin-fixed paraffin-embedded slide. In some embodiments, the sample on the slide 116 may be stained. In some embodiments, the slide 116 may be substantially transparent. In some embodiments, the slide 116 may include a thin, flat, substantially transparent glass slide. In some embodiments, a transparent cover may be applied to the slide 116 such that the sample is between the slide 116 and the cover. The sample may include, in non-limiting examples, blood smears, cervical smears, bodily fluids, and non-biological samples. In some embodiments, the sample on the slide 116 may include tissue. In some embodiments, the sample on the slide 116 may be frozen.

[0012] Still referring to FIG. 1 , in some embodiments, the slide 116 and / or the samples on the slide 116 may be illuminated. In some embodiments, the apparatus 100 may include a light source. As used herein, a "light source" is any device configured to emit electromagnetic radiation. In some embodiments, the light source may emit light having substantially one wavelength. In some embodiments, the light source may emit light having a range of wavelengths. The light source may emit, without limitation, ultraviolet, visible, and / or infrared light. In non-limiting examples, the light source may include light emitting diodes (LEDs), organic LEDs (OLEDs), and / or other light emitters. Such a light source may be configured to illuminate the slide 116 and / or the samples on the slide 116. In non-limiting examples, the light source may illuminate the slide 116 and / or the samples on the slide 116 from below.

[0013] Still referring to FIG. 1, in some embodiments, the device 100 may include at least one optical system 120. An "optical system" as used in this disclosure is an arrangement of one or more components that act on or employ electromagnetic radiation. In non-limiting examples, the optical system may include light, such as electromagnetic radiation, visible light, infrared light, ultraviolet light, etc. The optical system may include one or more optical elements, including but not limited to lenses, mirrors, windows, filters, etc. The optical system may form an optical image corresponding to an optical object. For example, the optical system may form an optical image at or on an optical sensor, which may capture (e.g., digitize) the optical image. In some cases, the optical system may have at least one magnification. For example, the optical system may include an objective lens (e.g., a microscope objective lens) and one or more reimaging optical elements that together form an optical magnification. In some cases, optical magnification may be referred to as zoom. As used herein, an "optical sensor" is a device that measures light and converts the measured light into one or more signals, which may include, but are not limited to, one or more electrical signals. In some embodiments, the optical sensor 120 may include at least one photodetector. As used herein, a "photodetector" is a device that is sensitive to light and can thereby detect light. In some embodiments, the photodetector may include a photodiode, a photoresistor, a photosensor, a photovoltaic chip, and the like. In some embodiments, the optical sensor 120 may include multiple photodetectors. The optical sensor 120 may include, but is not limited to, a camera. The optical sensor 120 may be in electronic communication with at least one processor 104 of the device 100. As used in this disclosure, "electronic communication" is a shared data connection between two or more devices. In some embodiments, the device 100 may include two or more optical sensors 120.

[0014] Still referring to FIG. 1, "image data" as used herein is information that represents at least one physical scene, space, and / or object. Image data may include, for example, information that represents a sample, a slide 116, or an area of ​​a sample or slide. In some cases, image data may be generated by a camera. "Image data" may be used interchangeably with "image" throughout this disclosure, with image being used as a noun. An image may be optical, such as, but not limited to, one in which at least one optical system is used to generate an image of an object. An image may be digital, such as, but not limited to, one that is represented as a bitmap. Alternatively, an image may include any medium capable of representing a physical scene, space, and / or object. Alternatively, when "image" is used as a verb in this disclosure, it refers to the generation and / or formation of an image.

[0015] Still referring to FIG. 1 , in some embodiments, the device 100 may include a slide port 140. In some embodiments, the slide port 140 may be configured to hold the slide 116. In some embodiments, the slide port 140 may include one or more alignment features. As used herein, an "alignment feature" is a physical characteristic that serves to secure the slide in place and / or align the slide with other components of the device. In some embodiments, the alignment feature may include a component that secures the slide 116, such as a clamp, latch, clip, recess, or other fastener. In some embodiments, the slide port 140 may facilitate removal or insertion of the slide 116. In some embodiments, the slide port 140 may include a transparent surface through which light passes. In some embodiments, the slide 116 may rest on such a transparent surface and / or be illuminated by light traveling through such a transparent surface. In some embodiments, the slide port 140 may be mechanically connected to the actuator mechanism 124, as described below.

[0016] Still referring to FIG. 1 , in some embodiments, the apparatus 100 may include an actuator mechanism 124. As used herein, an "actuator mechanism" is a mechanical component configured to change the relative position of the slide and the optical system. In some embodiments, the actuator mechanism 124 may be mechanically connected to the slide 116, such as the slide 116 in the slide port 140. In some embodiments, the actuator mechanism 124 may be mechanically connected to the slide port 140. For example, the actuator mechanism 124 may move the slide port 140 to move the slide 116. In some embodiments, the actuator mechanism 124 may be mechanically connected to at least one optical system 120. In some embodiments, the actuator mechanism 124 may be mechanically connected to a movable element. As used herein, a "movable element" refers to any movable or portable object, component, and device in the apparatus 100, such as, but not limited to, a slide, a slide port, an optical system. In some embodiments, the movable element may move such that the optical system 120 is properly positioned with respect to the slide 116 so that the optical system 120 can capture an image of the slide 116 according to a set of parameters. In some embodiments, the actuator mechanism 124 may be mechanically connected to an item selected from the list consisting of the slide port 140, the slide 116, and at least one optical system 120. In some embodiments, the actuator mechanism 124 may be configured to change the relative position of the slide 116 and the optical system 120 by moving the slide port 140, the slide 116, and / or the optical system 120.

[0017] Still referring to FIG. 1 , the actuator mechanism 124 may include machine components responsible for moving and / or controlling a mechanism or system. The actuator mechanism 124 may require a control signal and / or an energy source or power in some embodiments. In some cases, the control signal may be relatively low energy. Exemplary forms of the control signal include electrical potential or current, air pressure or flow, or hydraulic fluid pressure or flow, mechanical force / torque or speed, or even human power. In some cases, the actuator may have an energy source or power source other than the control signal. This may include a primary energy source, which may include, for example, electrical power, hydraulics, pneumatic pressure, mechanical power, etc. In some embodiments, upon receiving a control signal, the actuator mechanism 124 responds by converting source power into mechanical motion. In some cases, the actuator mechanism 124 may be understood as a form of automation or automatic control.

[0018] Still referring to FIG. 1 , in some embodiments, the actuator mechanism 124 may include a hydraulic actuator. A hydraulic actuator may be comprised of a cylinder or fluid motor that uses hydraulic power to facilitate mechanical motion. The output of the hydraulic actuator mechanism 124 may include mechanical motion, such as, but not limited to, linear, rotary, or oscillatory motion. In some embodiments, the hydraulic actuator may employ a hydraulic fluid. Because liquids are potentially incompressible, hydraulic actuators may exert large forces. Additionally, because force is equal to pressure multiplied by area, hydraulic actuators may function as force transducers with changes in area (e.g., cross-sectional area of ​​the cylinder and / or piston). An exemplary hydraulic cylinder may be comprised of a hollow cylindrical tube within which a piston can slide. In some cases, the hydraulic cylinder may be considered to be single-acting. “Single-acting” may be used when fluid pressure is applied substantially only to one side of the piston. Thus, the single-acting piston can only move in one direction. In some cases, a spring may be used to provide a return stroke for the single-acting piston. In some cases, the hydraulic cylinder may be double-acting. "Double acting" may be used where pressure is applied substantially to both sides of the piston. The force difference between the two sides of the piston causes the piston to move.

[0019] Still referring to FIG. 1 , in some embodiments, the actuator mechanism 124 may include a pneumatic actuator mechanism 124. In some cases, pneumatic actuators can generate large forces from relatively small changes in gas pressure. In some cases, pneumatic actuators can respond more quickly than other types of actuators, such as hydraulic actuators. Pneumatic actuators can use compressible fluids (e.g., air). In some cases, pneumatic actuators can run on compressed air. Operation of hydraulic and / or pneumatic actuators includes control of one or more valves, circuits, fluid pumps, and / or fluid manifolds.

[0020] Still referring to FIG. 1 , in some cases, the actuator mechanism 124 may include an electric actuator. The electric actuator mechanism 124 may include either an electromechanical actuator, a linear motor, or the like. In some cases, the actuator mechanism 124 may include an electromechanical actuator. An electromechanical actuator can convert the rotational force of an electric rotary motor into linear motion and generate linear motion through a mechanism. Exemplary mechanisms include rotary to translational converters, such as, but not limited to, belts, screws, cranks, cams, linkages, scotch yokes, and the like. In some cases, control of the electromechanical actuator may include control of an electric motor, e.g., a control signal may control one or more electric motor parameters to control the electromechanical actuator. Exemplary non-limiting electric motor parameters include rotational position, input torque, speed, current, and potential. The electric actuator mechanism 124 may include a linear motor. A linear motor may differ from an electromechanical actuator because power from a linear motor is directly output as translational motion, rather than output as rotational motion and converted to translational motion. In some cases, a linear motor may incur less friction loss than other devices. Linear motors may be designated into at least three different categories, such as flat linear motors, U-channel linear motors, and tubular linear motors. Linear motors may be directly controlled by control signals that control one or more linear motor parameters. Exemplary linear motor parameters include, but are not limited to, position, force, velocity, potential, and current.

[0021] 1 , in some embodiments, the actuator mechanism 124 may include a mechanical actuator mechanism 124. In some cases, the mechanical actuator mechanism 124 may function to execute motion by converting one type of motion, such as rotational motion, to another type of motion, such as linear motion. An exemplary mechanical actuator includes a rack and pinion. In some cases, a mechanical power source, such as a power take-off, may function as a power source for the mechanical actuator. The mechanical actuator may employ any number of mechanisms, including, for example, but not limited to, gears, rails, pulleys, cables, linkages, and the like.

[0022] Still referring to FIG. 1 , in some embodiments, the actuator mechanism 124 can be in electronic communication with the actuator control. As used herein, an “actuator control” is a system configured to operate the actuator mechanism such that the slide and the optical system are in a desired relative position. In some embodiments, the actuator control can operate the actuator mechanism 124 based on input received from the user interface 136. In some embodiments, the actuator control can be configured to operate the actuator mechanism 124 such that the optical system 120 is in a position to capture an image of the entire sample. In some embodiments, the actuator control can be configured to operate the actuator mechanism 124 such that the optical system 120 is in a position to capture an image of a region of interest, a particular horizontal row, a particular point, a particular focal depth, etc. The electronic communication between the actuator mechanism 124 and the actuator control can include the transmission of signals. For example, the actuator control can generate a physical movement of the actuator mechanism in response to an input signal. In some embodiments, the input signal can be received by the actuator control from the processor 104 or the input interface 128.

[0023] Still referring to FIG. 1, a "signal" as used in this disclosure is any understandable representation of data, for example, from one device to another. A signal may include an optical signal, a hydraulic signal, a pneumatic signal, a mechanical signal, an electrical signal, a digital signal, an analog signal, etc. In some cases, a signal may be used to communicate with a computing device, for example, via one or more ports. In some cases, a signal may be transmitted and / or received by a computing device, for example, via an input / output port. An analog signal may be digitized, for example, by an analog-to-digital converter. In some cases, an analog signal may be processed, for example, with analog signal processing steps described in this disclosure, before being digitized. In some cases, a digital signal may be used to communicate between two or more devices, including, but not limited to, computing devices. In some cases, a digital signal may be communicated by one or more communication protocols, including, but not limited to, Internet Protocol (IP), Controller Area Network (CAN) protocols, serial communication protocols (e.g., Universal Asynchronous Receiver Transmitter [UART]), parallel communication protocols (e.g., IEEE 128 [printer port]), etc.

[0024] Still referring to FIG. 1, in some embodiments, the device 100 can perform one or more signal processing steps on a signal. For example, the device 100 can analyze, modify, and / or synthesize a signal representing data to improve the signal, for example by improving transmission, storage efficiency, or signal-to-noise ratio. Exemplary methods of signal processing can include analog, continuous-time, discrete, digital, nonlinear, statistical, etc. Analog signal processing can be performed on non-digitized or analog signals. Exemplary analog processing can include passive filters, active filters, summing mixers, integrators, delay lines, companders, multipliers, voltage-controlled filters, voltage-controlled oscillators, phase-locked loops, etc. Continuous-time signal processing can be used in some cases to process signals that vary continuously in a domain, for example, the time domain. Exemplary non-limiting continuous-time processes can include time-domain processing, frequency-domain processing (Fourier transform), complex frequency-domain processing. Discrete-time signal processing can be used when a signal is sampled at non-continuous or discrete time intervals (i.e., quantized in time). Analog discrete-time signal processing can process signals using the following exemplary circuits: sample and hold circuits, analog time division multiplexers, analog delay lines, analog feedback shift registers. Digital signal processing can be used to process digitized discrete-time sampling signals. In general, digital signal processing can be performed by a computing device or other specialized digital circuitry, such as, but not limited to, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or a specialized digital signal processor (DSP). Digital signal processing can be used to perform any combination of typical arithmetic operations, such as fixed-point, floating-point, real-valued, complex-valued, multiplication, addition, etc. Digital signal processing can also operate circular buffers and look-up tables.Further non-limiting examples of algorithms that may be implemented according to digital signal processing techniques include Fast Fourier Transform (FFT), Finite Impulse Response (FIR) filters, Infinite Impulse Response (IIR) filters, and adaptive filters such as Wiener filters and Kalman filters. Statistical signal processing may be used to exploit statistical properties to process signals as random functions (i.e., stochastic processes). For example, in some embodiments, a signal may be modeled with a probability distribution indicative of noise, which may be used to reduce noise in the processed signal.

[0025] 1, in some embodiments, the device 100 may include a user interface 136. The user interface 136 may include the output interface 132 and the input interface 128.

[0026] Still referring to FIG. 1 , in some embodiments, output interface 132 may include one or more elements through which device 100 may communicate information to a user. In a non-limiting example, output interface 132 may include a display. The display may include a high-resolution display. The display may output images, videos, and the like to the user. In another non-limiting example, output interface 132 may include a speaker. The speaker may output audio to the user. In another non-limiting example, output interface 132 may include a haptic device. The speaker may output haptic feedback to the user.

[0027] Still referring to FIG. 1 , in some embodiments, the optical system 120 may include a camera. In some cases, the camera may include one or more optical systems. Exemplary non-limiting optical systems include spherical lenses, aspheric lenses, reflectors, polarizers, filters, windows, aperture stops, and the like. In some embodiments, one or more optical systems associated with the camera may be adjusted to change the zoom, depth of field, and / or focal length of the camera, in non-limiting examples. In some embodiments, one or more of such settings may be configured to detect features of a sample on the slide 116. In some embodiments, one or more of such settings may be configured based on a parameter set, as described below. In some embodiments, the camera may capture images with a shallow depth of field. In a non-limiting example, the camera may capture images that are focused at a first depth of the sample and not focused at a second depth of the sample. In some embodiments, an autofocus mechanism may be used to determine the focal length. In some embodiments, the focal length may be set by a parameter set. In some embodiments, the camera may be configured to capture multiple images at different focal lengths. In a non-limiting example, the camera can capture multiple images at different focal lengths such that an image is captured at each focal depth of the sample in at least one image. In some embodiments, the at least one camera can include an image sensor. Exemplary non-limiting image sensors include digital image sensors, such as, but not limited to, charge-coupled device (CCD) sensors and complementary metal-oxide semiconductor (CMOS) sensors. In some embodiments, the camera can be sensitive in the non-visible range of electromagnetic radiation, such as, but not limited to, infrared.

[0028] Still referring to FIG. 1 , in some embodiments, the input interface 128 may include controls for operating the device 100. Such controls may be operated by a user. The input interface 128 may include, in non-limiting examples, a camera, a microphone, a keyboard, a touch screen, a mouse, a joystick, a foot pedal, a button, a dial, and the like. The input interface 128 may accept, in non-limiting examples, mechanical input, voice input, visual input, text input, and the like. In some embodiments, voice input into the input interface 128 may be interpreted using an automatic speech recognition function, allowing a user to control the device 100 via voice. In some embodiments, the input interface 128 may approximate the controls of a microscope.

[0029] Still referring to FIG. 1 , in some embodiments, the speech input may be processed using automatic speech recognition. In some embodiments, the automatic speech recognition may require training (i.e., enrollment). In some cases, training the automatic speech recognition model may require an individual speaker to read text or isolated vocabulary. In some cases, the speech training data may include speech components having audible linguistic content, which content is known a priori by the computing device. Thus, the computing device may train the automatic speech recognition model according to the training data including the audible linguistic content that correlates to the known content. In this manner, the computing device may analyze a person's particular speech and train the automatic speech recognition model on that person's voice, resulting in improved accuracy. Alternatively or additionally, in some cases, the computing device may include a speaker-independent automatic speech recognition model. As used in this disclosure, a "speaker-independent" automatic speech recognition process does not require training for an individual speaker. Conversely, as used in this disclosure, an automatic speech recognition process that employs training specific to an individual speaker is "speaker-dependent."

[0030] Still referring to FIG. 1, in some embodiments, the automatic speech recognition process may perform speech recognition or speaker identification. "Speech recognition" as used in this disclosure refers to identifying a speaker from audio content, not what the speaker says. In some cases, the computing device may first recognize the speaker of the linguistic audio content, and then automatically recognize the speaker's voice, for example, by a speaker-dependent automatic speech recognition model or process. In some embodiments, the automatic speech recognition process may be used to authenticate or verify the identity of the speaker. In some cases, the speaker may or may not include the subject. For example, the subject may speak during the voice input, but other people may speak as well.

[0031] Still referring to Figure 1, in some embodiments, the automatic speech recognition process may include one or all of acoustic modeling, language modeling, and statistically-based speech recognition algorithms. In some cases, the automatic speech recognition process may employ Hidden Markov Models (HMMs). As will be described in more detail below, language modeling, such as that employed in natural language processing applications such as document classification and statistical machine translation, may also be employed in the automatic speech recognition process.

[0032] Still referring to FIG. 1, an exemplary algorithm employed for automatic speech recognition may include or be based on a hidden Markov model. A hidden Markov model (HMM) may include a statistical model that outputs a sequence of symbols or quantities. HMMs may be used for speech recognition because speech signals may be considered piecewise stationary or short-term stationary signals. For example, on short time scales (e.g., 10 ms), speech may be approximated as a stationary process. Speech (i.e., audible speech content) may be understood as a Markov model for many probabilistic purposes.

[0033] Still referring to FIG. 1, in some embodiments, the HMM can be trained automatically and can be relatively simple and computationally feasible to use. In an exemplary automatic speech recognition process, the hidden Markov model can output a sequence of n-dimensional real-valued vectors (where n is a small integer such as 10) at a rate of approximately one vector every 10 milliseconds. The vectors may be composed of cepstral coefficients. Cepstral coefficients should use the spectral domain. Cepstral coefficients are found by Fourier transforming a short time window of speech to generate a spectrum, decorrelating the spectrum using a cosine transform, and taking the first (i.e., most significant) coefficients. In some cases, the HMM may have a statistical distribution at each state that is a mixture of diagonal covariance Gaussians, yielding a likelihood for each observation vector. In some cases, each word or phoneme may have a different output distribution. An HMM for a sequence of words or phonemes can be created by concatenating HMMs for separate words or phonemes.

[0034] Still referring to FIG. 1, in some embodiments, the automatic speech recognition process may use various combinations of several techniques to improve results. In some cases, the large vocabulary automatic speech recognition process may include context dependency of phonemes. For example, in some cases, phonemes with different left and right contexts may have different recognition as states of the HMM. In some cases, the automatic speech recognition process may use cepstral normalization to normalize for different speakers and recording conditions. In some cases, the automatic speech recognition process may use vocal tract length normalization (VTLN) for gender normalization and maximum likelihood linear regression (MLLR) for more general speaker adaptation. In some cases, the automatic speech recognition process may determine so-called delta and delta-delta coefficients to capture speech dynamics and may use heteroscedastic linear discriminant analysis (HLDA). In some cases, the automatic speech recognition process may use projections based on splicing and linear discriminant analysis (LDA), which may include heteroscedastic linear discriminant analysis or global semi-joint covariance transform (also called maximum likelihood linear transform [MLLT]). In some cases, an automatic speech recognition process may forego a purely statistical approach to HMM parameter estimation and instead use discriminative training techniques that optimize some classification-related measure of the training data, examples of which include maximum mutual information (MMI), minimum classification error (MCE), and minimum phoneme error (MPE).

[0035] Still referring to FIG. 1, in some embodiments, an automatic speech recognition process may be said to decode speech (i.e., audible language content). Speech decoding occurs when an automatic speech recognition system is presented with a new utterance and must calculate the most likely sentence. In some cases, speech decoding may include a Viterbi algorithm. The Viterbi algorithm may include a dynamic programming algorithm to obtain a maximum a posteriori probability estimate of the most likely sequence of hidden states (i.e., the Viterbi path) that results in an observed sequence of events. The Viterbi algorithm may be employed in the context of Markov sources and hidden Markov models. The Viterbi algorithm may be used to find the best path, for example, using a statically created combinatorial hidden Markov model (e.g., a finite state transducer [FST] approach) and using a dynamically created combinatorial hidden Markov model that has both acoustic and language model information.

[0036] Still referring to FIG. 1, in some embodiments, decoding of speech (i.e., audible language content) may include considering good candidates as well as best candidates when a new utterance is presented. In some cases, a refined scoring function (i.e., rescoring) may be used to evaluate each of the set of good candidates and select the best candidate according to this refined score. In some cases, the set of candidates may be maintained either as a list (i.e., N-best list approach) or as a subset of the model (i.e., lattice). In some cases, rescoring may be performed by optimizing the Bayes risk (or an approximation thereof). In some cases, rescoring may include optimizing the sentence (including keywords) that minimizes the expected value of a given loss function with respect to all possible transcriptions. For example, rescoring allows the selection of sentences that minimize the average distance to other possible sentences weighted by their estimated probabilities. In some cases, the loss function employed may include the Levenshtein distance, although, for example, different distance calculations may be performed for a particular task. In some cases, the set of candidates may be pruned to maintain tractability.

[0037] Still referring to FIG. 1, in some embodiments, the automatic speech recognition process may employ a dynamic time warping (DTW) based approach. Dynamic time warping may include algorithms that measure similarity between two sequences that differ in time or speed. For example, a person may walk slowly in one video and fast in another video, and may accelerate and decelerate during the same observation, but similarity in walking patterns may be detected. DTW has been applied to video, audio, and graphics, and in fact any data that can be converted to a linear representation may be analyzed with DTW. In some cases, DTW may be used by the automatic speech recognition process to accommodate different speech (i.e., audible language content) rates. In some cases, DTW may enable a computing device to find an optimal match between two given sequences (e.g., time series) subject to certain constraints. That is, in some cases, sequences may be "warped" nonlinearly to match each other. In some cases, DTW-based sequence alignment methods may be used in the context of hidden Markov models.

[0038] Still referring to FIG. 1, in some embodiments, the automatic speech recognition process may include a neural network. The neural network may include any neural network, such as those disclosed with reference to FIGS. 2-4. In some cases, the neural network may be used for automatic speech recognition, including phoneme classification, phoneme classification by multi-objective evolutionary algorithms, isolated word recognition, audio-visual speech recognition, audio-visual speaker recognition, and speaker adaptation. In some cases, the neural network employed in automatic speech recognition may have fewer explicit assumptions about feature statistical properties than HMMs, and thus may have some qualities that make it an attractive recognition model for speech recognition. When used to estimate the probability of speech feature segments, the neural network may provide discriminative training in a natural and efficient manner. In some cases, the neural network may be used to effectively classify audible language content over short time intervals, such as individual phonemes and isolated words. In some embodiments, the neural network may be employed by the automatic speech recognition process for preprocessing, feature transformation, and / or dimensionality reduction, for example, prior to HMM-based recognition. In some embodiments, long short-term memory (LSTM) and related recurrent neural networks (RNN) as well as time-delay neural networks (TDNN) may be used for automatic speech recognition, e.g., over longer time intervals for continuous speech recognition.

[0039] 1, in some embodiments, the apparatus 100 captures a first image of the slide 116 at a first position. In some embodiments, the first image may be captured using at least one optical system 120.

[0040] 1 , in some embodiments, capturing a first image of the slide 116 at a first position may include using the actuator mechanism 124 and / or actuator control to move the optical system 120 and / or the slide 116 to a desired position. In some embodiments, the first image includes an image of the entire sample and / or the entire slide 116. In some embodiments, the first image includes an image of an area of ​​the sample. In some embodiments, the first image includes a wider angle image than the second image (described below). In some embodiments, the first image may include a lower resolution image than the second image.

[0041] Still referring to FIG. 1, in some embodiments, the apparatus 100 can identify at least one region of interest in the first image. In some embodiments, machine vision may be used to identify the at least one region of interest. As used herein, a "region of interest" is a particular area within a slide or a digital image of a slide where features are detected. Features may include, in non-limiting examples, samples, dust, writing on the slide, cracks in the slide, air bubbles, etc.

[0042] Still referring to FIG. 1, in some embodiments, the device 100 may include a machine learning module 144. Machine learning is described with reference to FIG. 2. In some embodiments, the device 100 may use an ROI identification machine learning model 148 to identify at least one region of interest. In some embodiments, the ROI identification machine learning model 148 may be trained using supervised learning. In some embodiments, the ROI identification machine learning model 148 may include a classifier. The ROI identification machine learning model 148 may be trained on a data set including example images of slides associated with example regions of the images in which features are present. Such a training data set may be collected, for example, by collecting slide imaging device data regarding which regions of the images of the slides an expert focuses on or zooms in on. Once trained, the ROI identification machine learning model 148 may accept images of slides as input and output data regarding the location of the regions of interest present. In some embodiments, a neural network, such as a convolutional neural network, may be used to identify at least one region of interest. For example, a convolutional neural network may be used to detect edges in images of slides, and at least one region of interest may be identified based on the presence of the edges. In some embodiments, at least one region of interest may be identified as a function of brightness and / or color difference in comparison to the brightness and / or color of the background. In some embodiments, the device 100 may identify the region of interest using a classifier. In some embodiments, a segment of the image may be input to the classifier, which may categorize the segment of the image based on whether a region of interest is present. In some embodiments, the classifier may output a score indicating the degree to which a region of interest is detected and / or a confidence that a region of interest is present. In some embodiments, features may be detected using a neural network or other machine learning model trained to detect features and / or objects.For example, edges, corners, blobs, or ridges may be detected, and whether a location is determined to be within the region of interest may depend on the detection of such features. In some embodiments, machine learning models, such as support vector machine techniques, may be used to determine features based on the detection of features such as edges, corners, blobs, or ridges.

[0043] Still referring to FIG. 1 , in some embodiments, the device 100 can identify at least one region of interest as a function of user input. For example, the user can modify a setting of the degree of sensitivity for detecting the region of interest. In this example, if the user input indicates low sensitivity, the device 100 can only detect large regions of interest. This can include, for example, ignoring potential regions of interest below a certain size. In another example, this can include applying a machine learning model, such as a classifier, to the image (or a segment of the image) and identifying the image (or a segment of the image) as a region of interest only if the machine learning model outputs a score higher than a threshold. The score indicates the degree to which the region of interest is detected and / or the confidence that the region of interest is present. In some embodiments, the image can be divided into smaller segments and the segments can be analyzed to determine whether at least one region of interest is present.

[0044] Still referring to FIG. 1 , in some embodiments, the device 100 may receive at least one region of interest. In some embodiments, the device 100 may receive at least one region of interest without first capturing a first image. In a non-limiting example, a user may input the region of interest. In some embodiments, the device 100 may capture a first image and apply the received region of interest to the first image. This may be done, for example, when the region of interest is received before the first image is captured. In some embodiments, the device 100 may capture the first image as a function of the region of interest. In a non-limiting example, the device 100 may receive a region of interest from a user through user input and capture the first image at a first location within the region of interest.

[0045] 1, apparatus 100 may identify a focus pattern. In some embodiments, apparatus 100 may identify a focus pattern in at least one region of interest, such as each region of interest detected as described herein. In some embodiments, identifying a focus pattern may include identifying a row, identifying a point within the row, determining a best focus at a point, and / or identifying a plane.

[0046] Still referring to FIG. 1, as used herein, a "row" of a digital image of a slide is a segment of the digital image of a slide between two parallel lines. A row may include, for example, a row of pixels having a width of one pixel. In another example, a row may have a width of multiple pixels. A row may or may not cross a grid of pixels diagonally. As used herein, a "point" on a digital image of a slide refers to a specific location within the digital image of a slide. For example, in a digital image composed of a grid of pixels, a point may have a specific (x,y) location. As used herein, "best focus" is the focal distance at which the object being focused on is in focus.

[0047] Still referring to FIG. 1, in some embodiments, the apparatus 100 may identify a row in the region of interest. The row may be identified based on a first row sample presence score. The first row sample presence score may be identified using machine vision. The first row sample presence score may be identified based on the output of the row identification machine learning model 152. The first row sample presence score may be identified by determining one or more row sample presence scores of adjacent rows. For example, the first row sample presence score may be determined as a function of a second row sample presence score and a third row sample presence score, where the second and third row sample presence scores are based on rows adjacent to the row of the first row sample presence score. As used herein, a "sample presence score" is a value that represents or estimates the likelihood that a sample is present at a location. As used herein, a "row sample presence score" is a sample presence score with the location as the row. The sample presence score does not have to represent the likelihood of the sample being present as a percentage between 0 and 100%. For example, a row sample presence score of 2000 for a first row and a row sample presence score of 3000 for a second row indicates that the second row is more likely to contain a sample than the first row. The first row sample presence score may be identified based on, in non-limiting examples, the sum of the row sample presence scores of adjacent rows, and / or a weighted sum of the row sample presence scores of adjacent rows, and / or the minimum of the row sample presence scores of adjacent rows. In some embodiments, a first set of row sample presence scores is identified using a machine learning model such as row identification machine learning model 152, and a second set of row sample presence scores is identified based on the row sample presence scores of rows adjacent to the first set of row sample presence scores. In some embodiments, such a second set of row sample presence scores may be used to identify the best row. The row sample presence scores may be identified by determining the sample presence scores of rows that are not directly adjacent to the row in question.For example, line sample presence scores may be determined for the line in question, adjacent lines, and lines one line away from the line in question, and each of these line sample presence scores may be factored into identifying the line (e.g., using a weighted sum of the sample presence scores). The sample presence scores may be determined using machine vision. In some embodiments, the line sample presence scores may be determined based on a section of the line that does not run completely across the image and / or slide. For example, the line sample presence score may be determined relative to the width of the region of interest. In another example, the line sample presence score may be determined for a section of the line having a pixel width.

[0048] Still referring to FIG. 1 , in some embodiments, identifying the focal pattern may include identifying a row that includes a particular (X,Y) location, such as a location where an optimal focal length is calculated. In some embodiments, identifying the focal pattern may include capturing a plurality of images at such locations, each of the plurality of images having a different focal length. Such images may represent a Z-stack, as described further below. Identifying the focal pattern may further include determining an optimally focused image from among the plurality of images. The focal length of such image may be determined to be the optimal focal length for that (X,Y) location. In this manner, an optimal focal length may be determined for a plurality of points in the row. The focal pattern may be identified using the focal lengths of the plurality of optimally focused images at a plurality of points in such row.

[0049] Still referring to FIG. 1 , in some embodiments, the row may further include a second location. The apparatus 100 may capture a plurality of second images at such second locations, each of the plurality of second images having a different focal length. The apparatus 100 may determine an optimally focused second image among the plurality of best-focused second images. The apparatus 100 may identify a focal pattern using the focal lengths of the optimally focused first image and the optimally focused second image. For example, the focal pattern may be determined to include a line connecting the two points. The apparatus 100 may extrapolate a third focal length for the third location as a function of the focal pattern. In some embodiments, extrapolating the third focal length may include using the first or second focal length as the third focal length. In some embodiments, the extrapolation may include linear extrapolation, polynomial extrapolation, conic extrapolation, geometric extrapolation, and the like. In some embodiments, such a third location may be located outside the row that includes the first location and / or the second location. In some embodiments, such a third location may be located in a different region of interest than the first location and / or the second location.

[0050] Still referring to FIG. 1 , in some embodiments, the sample presence score may be determined using a line identification machine learning model 152. In some embodiments, the line identification machine learning model 152 may be trained using supervised learning. The line identification machine learning model may be trained on a dataset that includes examples of lines from images of slides that are associated with whether or not a sample is present. Such a dataset may be collected, for example, by capturing images of slides and manually identifying which lines contain samples and extracting lines from the larger image. Once trained, the line identification machine learning model 152 may accept as input lines from images of slides, such as lines from a region of interest, and output a determination of whether or not a sample is present and / or a sample presence score, such as a line sample presence score.

[0051] Still referring to FIG. 1 , in some embodiments, the device 100 can identify a point within a row, such as the row identified above. In some embodiments, the point may be identified using machine vision. In some embodiments, the point may be identified based on the point within a particular row having the maximum point sample presence score. As used herein, a "point sample presence score" is a sample presence score given the location of the point. The point sample presence score may be determined, in a non-limiting example, based on the color of the point and / or the surrounding pixels, or whether the point is inside or outside a potential specimen boundary (which may be determined, for example, by identifying edges in the image and areas of the image surrounded by the edges). In another non-limiting example, the point sample presence score may be determined based on the distance between the point and an edge of the slide.

[0052] Still referring to FIG. 1 , in some embodiments, the points may be identified using a point identification machine learning model 156. In some embodiments, the point identification machine learning model 156 may be trained using supervised learning. The point identification machine learning model may be trained on a dataset that includes examples of points from images of slides that are associated with whether or not a sample is present. Such a dataset may be collected, for example, by capturing images of slides, manually identifying which points contain samples, and extracting points from the larger image. Once trained, the point identification machine learning model 156 may accept as input points from images of slides, such as points from a region of interest, and output a determination of whether or not a sample is present and / or a point sample presence score. In some embodiments, the device 100 may identify points from a row based on which point has the highest point sample presence score.

[0053] Still referring to FIG. 1 , in some embodiments, the device 100 can determine the best focus at a point, such as the point identified as described above. In some embodiments, the best focus can be determined using an autofocus mechanism. In some embodiments, the best focus can be determined using a rangefinder. In some embodiments, the actuator mechanism 124 can move the optical sensor 120 and / or the sliding port 140 to cause the autofocus mechanism to focus at a desired location. In some embodiments, the autofocus mechanism can be capable of focusing on multiple points in a frame and can select which point to focus on based on the points identified as described above. In some embodiments, one or more camera parameters other than focus can be adjusted to improve focus and / or improve the image. For example, the aperture can be adjusted to change how well a point is focused. For example, the aperture can be adjusted to increase the depth of field to focus on a point. The best focus can be expressed as a focal length or depth of focus, in non-limiting examples.

[0054] Still referring to FIG. 1, in some embodiments, the device 100 may identify a focus pattern based on the best focus and / or point. The best focus and point may be expressed as positions in three-dimensional space. For example, the X and Y coordinates (horizontal axis) may be determined based on the location of the point on the slide and / or the location of the point on the image. The Z coordinate may be determined based on the best focus. For example, the focal length of the best focus may be used as the Z coordinate. As used herein, a "focus pattern" is a pattern that approximates the best focus level at a number of points, including at least one point where best focus has not been measured. One or more (X,Y,Z) coordinates may be used to determine the focus pattern. One or more default parameters may be used to determine the focus pattern (such as defaulting to horizontal if there is insufficient data to determine otherwise). For example, a single (X,Y,Z) coordinate may be determined and the focus pattern may be determined as a plane extending horizontally in the X and Y directions with the Z level constant. In some embodiments, the focus pattern varies vertically (Z direction). For example, two (X,Y,Z) coordinates may be determined, and a focal pattern may be determined that includes a line connecting the two (X,Y,Z) locations and extends horizontally (forming a plane that includes the line) when moved horizontally perpendicular to the line. In another example, three (X,Y,Z) coordinates may be determined, and a focal pattern may be determined as a plane that includes all three locations. In another example, several (X,Y,Z) coordinates may be determined, and a regression algorithm may be used to determine a plane that best fits the coordinates. For example, least squares regression may be used. In some embodiments, the focal pattern is not a plane. The focal pattern may include a surface, such as a surface in a three-dimensional space. The focal pattern may include one or more curves, bumps, edges, etc. For example, the focal pattern may include a first plane in a first region, a second plane in a second region, and an edge where the planes intersect. In another example, the focus pattern may include a curved and / or bumpy surface where the Z level of each location on the focus pattern surface is determined based on the (X,Y,Z) coordinates of neighbors and the distance to their Z levels.In another example, the focus pattern may include multiple shapes with (X,Y,Z) coordinates as vertices and bounded by lines between the (X,Y,Z) coordinates. In some embodiments, the focus pattern may have only one Z value for each (X,Y) coordinate. In some embodiments, which points are evaluated for their best focus may be determined as a function of, in non-limiting examples, the desired density of points in the region of interest, the likelihood that samples are present (such as output from the ML model described above), and / or user input. For example, a user may manually select points and / or input a desired point density. In some embodiments, the focus pattern may be updated as additional points are scanned. For example, the focus pattern may be the shape of a plane based on ten (X,Y,Z) coordinates, an eleventh (X,Y,Z) coordinate may be scanned, and the focus pattern may be recalculated and / or updated to account for the new coordinate. In another example, the focus pattern may start as a plane based on a single (X,Y,Z) coordinate and be updated as additional (X,Y,Z) coordinates are identified. In some embodiments, the focus pattern may be updated for every additional (X,Y,Z) coordinate. In some embodiments, the focus pattern may be updated at a rate less than the rate of (X,Y,Z) coordinate identification. In non-limiting examples, the focus pattern may be updated every 2, 3, 4, 5, or more (X,Y,Z) coordinates.

[0055] Still referring to FIG. 1, in some embodiments, the data used to identify the focus pattern may be filtered. In some embodiments, one or more outliers may be removed. For example, if nearly all (X,Y,Z) points suggest a focus pattern in the shape of a plane, and a single (X,Y,Z) point has a Z value that is significantly different from that estimated from the plane, that (X,Y,Z) point may be removed. In another example, (X,Y,Z) points identified as focusing on features other than the sample may be removed. For example, (X,Y,Z) points identified as focusing on an annotation may be removed.

[0056] Still referring to FIG. 1 , in some embodiments, the process described above may be used to determine a focus pattern for each region of interest. In a region of interest that includes a sample, this results in determining a focus pattern based on one or more points that include the sample. This may help to efficiently identify the focus pattern so that follow-up images are captured at the correct focus distance. In some regions of interest, such as regions of interest that do not include a sample, but instead include features such as annotations, focusing on non-sample features may occur. This may be desirable, for example, because capturing a focused image of a feature such as an annotation may help an expert read the annotation and / or aid the optical character recognition process in transcribing characters. In some regions of interest, both sample and non-sample features may be present. In this case, focusing on the sample is desirable, and this can be achieved by the processes described herein. In some embodiments, the processes described herein may provide a more efficient method of identifying focus patterns than alternatives. For example, the processes described herein may require focusing on fewer points to determine a focus pattern.

[0057] Still referring to FIG. 1 , in some embodiments, a focal pattern, such as a plane, may be identified as a function of a first image and a first location. An (X,Y) location of a point may be determined from the first location. A best focus value may be determined from the first image. Together, these may be used to identify (X,Y,Z) coordinates that may be used to identify the focal pattern, as described herein.

[0058] Still referring to FIG. 1 , in some embodiments, a focus pattern, such as the Z level of best focus, may be used to scan the remaining portions of the row. For example, the best focus may be used to scan the remaining row that includes the identified point. In some embodiments, the focus pattern may be used to scan additional rows. For example, a focus pattern determined as a function of the Z level of the point may be used to scan rows adjacent to the row that includes the point. In some embodiments, the focus pattern may be used to scan other rows in the same region of interest. In some embodiments, the focus pattern may be used to scan rows in other regions of interest, such as neighboring regions of interest.

[0059] Still referring to FIG. 1 , in some embodiments, the identification of the region of interest, the identification of the points, the identification of the points, and / or the determination of the focus pattern may be performed locally. For example, the device 100 may include a machine learning model already trained and may apply the model to the image. In some embodiments, the identification of the region of interest, the identification of the points, the identification of the points, and / or the determination of the focus pattern may be performed externally. For example, the device 100 may transmit image data to another computing device and may receive outputs as described herein. In some embodiments, the region of interest may be identified, the lines may be identified, the points may be identified, and / or the focus pattern may be determined in real time.

[0060] 1, in some embodiments, the device 100 can determine a scanning pattern. The scanning pattern can be based on, for example, the morphology of the sample. The scanning pattern can include, in non-limiting examples, a zigzag, a snake line, and a spiral.

[0061] Still referring to FIG. 1 , in some embodiments, a snake pattern may be used to scan the slide. The snake pattern may proceed in any horizontal direction. In a non-limiting example, the snake pattern may proceed across the length or width of the slide. In some embodiments, the direction in which the snake pattern proceeds may be selected to minimize the number of turns required. For example, the snake pattern may proceed along the shortest dimension of the slide. In another example, the shape of the sample may be identified, for example using machine vision, and the snake pattern may proceed in a direction according to the shortest dimension of the sample. In some embodiments, a snake pattern may be selected when fast scanning is desired. In some embodiments, the snake pattern may minimize the camera and / or slide movement required to scan the slide. In some embodiments, a zigzag pattern may be used to scan the slide. Similar to what was described with respect to snake pattern scanning, the zigzag pattern scan may be performed in any horizontal direction, and the direction may be selected to minimize the number of turns and / or rows used to scan a feature, such as a slide and / or a sample. In some embodiments, a spiral pattern may be used to scan a slide, feature, region of interest, etc. In some embodiments, a snake and / or zigzag pattern may be best suited for Z-direction movement from one row to the next, hi some embodiments, a spiral pattern may be best suited for dynamic grid region of interest expansion.

[0062] Still referring to FIG. 1 , in some embodiments, the apparatus 100 can extrapolate the focal length of the second position as a function of the focal pattern. In some embodiments, the focal pattern can be determined as a function of one or more (X,Y,Z) points local to the region of interest and / or a sub-region of the region of interest. In some embodiments, a focal pattern, such as a plane, can be used to approximate the optimal focus level using extrapolation (rather than interpolation). For example, the focal pattern can be used to approximate the optimal focus level at (X,Y) points outside the range of (X,Y) points already scanned, such as outside the range of X values, outside the range of Y values, or outside the range of shapes that encompass the already scanned (X,Y) points. In another example, the focal pattern can be used to approximate the optimal focus level, where the optimal focus level includes Z values ​​outside the range of Z values ​​used to determine the focal pattern. In another example, the (X,Y,Z) points can be extrapolated to a focal pattern that spans the rows. In some embodiments, one or more additional (X,Y,Z) points can be used to update the focal pattern. In some embodiments, a focus pattern identified for one row may be extrapolated to another row, such as an adjacent row. In another example, a local focus pattern, such as a plane, may be extrapolated to identify an optimal focus level outside the local region. In another example, a focus pattern in a first region of interest may be extrapolated to generate a focus pattern in a second region of interest and / or to identify an optimal focus level at an (X,Y) point in the second region of interest.

[0063] Still referring to FIG. 1 , in some embodiments, the apparatus 100 can capture a second image of the slide at a second position with a focal distance based on the focal pattern. For example, if the focal pattern is a plane and a focused image of a particular (X,Y) point is desired, the apparatus 100 can capture the image with a focal distance based on the Z coordinate of the plane at those (X,Y) coordinates. In some embodiments, the first position (such as the position where best focus is measured) and the second position can be set to image positions within the same region of interest. In some embodiments, an actuator mechanism can be mechanically connected to the movable element. Also, the actuator mechanism can move the movable element to the second position. In some embodiments, acquiring the second image can include capturing multiple images taken with a focal distance based on the focal pattern and constructing the second image from the multiple images. In some embodiments, the second image can include images taken as part of a Z-stack, which is described below.

[0064] Still referring to FIG. 1 , in some embodiments, the device 100 may determine which regions of interest contain a sample. In some embodiments, this may be applied to a second image, such as a second image taken with a focal distance based on the focal pattern. In some embodiments, a sample identification machine learning model 160 may be used to determine which regions of interest contain a sample. In some embodiments, the sample identification machine learning model 160 may include a classifier. In some embodiments, the sample identification machine learning model 160 may be trained using supervised learning. The sample identification machine learning model 160 may be trained on a dataset that includes example images of slides and / or segments of images of slides that are associated with whether or not a sample is present. Such a dataset may be collected, for example, by capturing images of slides and manually identifying those that contain a sample. In some embodiments, multiple machine learning models may be trained to identify different types of samples. Once trained, the sample identification machine learning model 160 may accept images of regions of interest as input and output a determination of whether or not a sample is present. In some embodiments, the sample identification machine learning model 160 may be improved, such as through the use of reinforcement learning. The feedback used to determine the cost function of the reinforcement learning model may include, for example, user input or annotations on the slide. For example, if annotations transcribed using optical character recognition indicate a particular type of sample and the sample identification machine learning model 160 indicates that the region of interest does not contain the sample, a cost function may be determined that indicates the output is false. In another example, a user may input a label to be associated with the region of interest. If the label indicates a particular type of sample and the output of the sample identification machine learning model 160 indicates that the sample is present in the region of interest, a cost function may be determined that indicates the output is correct.

[0065] Still referring to FIG. 1 , in some embodiments, whether a sample is present may be determined locally. For example, the device 100 may include a pre-trained sample identification machine learning model 160 and apply the model to an image or a segment of an image. In some embodiments, whether a sample is present may be determined externally. For example, the device 100 may transmit image data to another computing device and receive a determination of whether a sample is present. In some embodiments, whether a sample is present may be determined in real-time.

[0066] 1, in some embodiments, a machine vision system and / or an optical character recognition system may be used to determine one or more characteristics of the sample and / or slide 116. In a non-limiting example, an optical character recognition system may be used to identify writing on the slide 116, which may be used to annotate an image of the slide 116.

[0067] Still referring to FIG. 1, in some embodiments, the apparatus 100 can capture multiple images at different sample focal depths. As used herein, "sample focal depth" is the depth within a sample at which an optical system is focused. As used herein, "focal length" is the object-side focal length. In some embodiments, the first and second images may have different focal lengths and / or sample focal depths.

[0068] Still referring to FIG. 1 , in some embodiments, the device 100 may include a machine vision system. In some embodiments, the machine vision system may include at least one camera. The machine vision system may use images, such as images from at least one camera, to make decisions about a scene, a space, and / or an object. For example, in some cases, the machine vision system may be used for world modeling and alignment of objects in a space. In some cases, alignment may include image processing, such as, but not limited to, object recognition, feature detection, edge / corner detection, etc. Non-limiting examples of feature detection include Scale Invariant Feature Transform (SIFT), Canny edge detection, Shi Tomasi corner detection, etc. In some cases, alignment may include one or more transformations that orient the camera frame (or image or video stream) with respect to a three-dimensional coordinate system. Exemplary transformations include, but are not limited to, homography transformations and affine transformations. In one embodiment, the alignment of the first frame with respect to the coordinate system may be verified and / or corrected using object identification and / or computer vision, as described above. For example, but not by way of limitation, an initial alignment into two dimensions, e.g., represented as an alignment into x and y coordinates, may be performed using a two-dimensional projection of the three-dimensional point onto the first frame. A third dimension of alignment, representing depth and / or z-axis, may be detected by comparison of the two frames. For example, if the first frame includes a pair of frames captured using a pair of cameras (e.g., also referred to as a stereo camera in this disclosure), image recognition and / or edge detection software may be used to detect a pair of stereo views of the image of the object. The two stereo views are compared to derive z-axis values ​​for points on the object, allowing for the derivation of additional z-axis points within and / or around the object, e.g., using interpolation. This may be repeated for multiple objects in the field of view, including, but not limited to, environmental features of interest identified by an object classifier and / or indicated by an operator.In one embodiment, the x and y axes may be selected to span a plane common to the two cameras used to capture the stereoscopic image and / or the xy plane of the first frame, so that the x and y translation components and φ may be pre-entered into the translation and rotation matrices for the affine transformation of the object's coordinates, as described above. As described above, the estimation of the initial x and y coordinates and / or transformation matrix may alternatively or additionally be performed between the first and second frames. As described above, for each point of the object and / or edges and / or points on the object's edges, the x and y coordinates of the first stereoscopic frame may be entered, along with an initial estimate of the z coordinate, based on assumptions about the object, such as, for example, the assumption that the ground is approximately parallel to the xy plane, as selected above. The Z coordinates and / or x, y, z coordinates aligned using the image capture and / or object identification process as described above may then be compared to coordinates predicted using an initial guess of the transformation matrix, and an error function may be used to calculate by comparing the two sets of points and the new x, y, and / or z coordinates, and may be iteratively estimated and compared until the error function falls below a threshold level. In some cases, the machine vision system may use a classifier, such as any of the classifiers described throughout this disclosure.

[0069] Still referring to FIG. 1 , in some embodiments, the image data may be processed using optical character recognition. In some embodiments, optical character recognition or optical character reader (OCR) involves automatic conversion of an image of written (e.g., typed, handwritten, or printed) text into machine-encoded text. In some cases, recognizing at least one keyword from the image data may include one or more processes, including but not limited to optical character recognition (OCR), optical word recognition, intelligent character recognition, intelligent word recognition, and the like. In some cases, OCR may recognize written text one glyph or character at a time. In some cases, optical word recognition can recognize written text one word at a time, for example, for languages ​​that use spaces as word separators. In some cases, intelligent character recognition (ICR) can recognize written text one glyph or character at a time, for example, by employing machine learning processes. In some cases, intelligent word recognition (IWR) can recognize written text one word at a time, for example, by employing machine learning processes.

[0070] Still referring to FIG. 1, in some cases, OCR may be an "offline" process that analyzes a static document or image frame. In some cases, handwriting motion analysis may be used as input for handwriting recognition. For example, rather than simply using glyphs or word shapes, the technique can capture the order in which segments are drawn, the direction, pen placement and lift patterns, and other motions. This additional information can result in more accurate handwriting recognition. In some cases, the technique is also referred to as "online," dynamic, real-time, or intelligent character recognition.

[0071] Still referring to FIG. 1, in some cases, OCR processing may employ pre-processing of the image data. Pre-processing processes may include, but are not limited to, deskewing, despeckling, binarization, line removal, layout analysis or "zoning", line and word detection, script recognition, character separation or "segmentation", and normalization. In some cases, deskewing processing may include applying a transformation (e.g., homography or affine transformation) to the image data to align the text. In some cases, despeckling processing may include removing positive and negative spots and / or smoothing edges. In some cases, binarization processing may include converting the image from color or grayscale to black and white (i.e., a binary image). Binarization may be performed as a simple way to separate text (or any other desired image components) from the background of the image data. In some cases, binarization may be required, for example, if the OCR algorithm being employed only supports binary images. In some cases, line removal processing may include removing glyphs and non-character images (e.g., boxes and lines). In some cases, a layout analysis or "zoning" process may identify columns, paragraphs, captions, etc. as separate blocks. In some cases, a line and word detection process may establish word and character shape metrics and separate words as needed. In some cases, a script recognition process may identify scripts, for example in a multilingual document, allowing an appropriate OCR algorithm to be selected. In some cases, a character separation or "segmentation" process may separate signal characters, for example in a character-based OCR algorithm. In some cases, a normalization process may normalize the aspect ratio and / or scale of the image data.

[0072] Still referring to FIG. 1, in some embodiments, the OCR process may include an OCR algorithm. Exemplary OCR algorithms include a matrix matching process and / or a feature extraction process. Matrix matching may involve a pixel-by-pixel comparison of the image to the stored glyphs. In some cases, matrix matching is also known as "pattern matching," "pattern recognition," and / or "image correlation." Matrix matching may depend on the input glyph being properly separated from the rest of the image data. Matrix matching may also depend on the stored glyphs being in a similar font and scale as the input glyphs. Matrix matching may work best with typed text.

[0073] Still referring to FIG. 1, in some embodiments, the OCR process may include a feature extraction process. In some cases, feature extraction may decompose a glyph into at least one feature. Exemplary non-limiting features may include corners, edges, lines, closed loops, line directions, line intersections, and the like. In some cases, feature extraction may reduce the dimensionality of the representation, making the recognition process more computationally efficient. In some cases, the extracted features are compared to an abstract, vector-like representation of the character and reduced to one or more glyph prototypes. Common techniques of feature detection in computer vision may be applied to this type of OCR. In some embodiments, a machine learning process such as a nearest neighbor classifier (e.g., a k-nearest neighbor algorithm) may be used to compare image features to stored glyph features and select the closest match. The OCR may employ any machine learning process described in this disclosure, such as the machine learning processes described with reference to FIGS. 2-4. Exemplary non-limiting OCR software includes Cuneiform and Tesseract. Cuneiform is a multilingual, open-source optical character recognition system originally developed by Cognitive Technologies of Moscow, Russia. Tesseract is a free OCR software originally developed by Hewlett-Packard of Palo Alto, California, USA.

[0074] Still referring to FIG. 1, in some cases, OCR may employ a two-pass approach to character recognition. The first pass may attempt to recognize characters. Each good character is passed through an adaptive classifier as training data. The adaptive classifier gets a chance to more accurately recognize characters by further analyzing the image data. A second pass is performed on the image data, since the adaptive classifier may have learned something useful in the first pass that is a little too late to recognize characters. The second pass may include adaptive recognition, where characters that were recognized with high confidence in the first pass may be used to better recognize the remaining characters in the second pass. In some cases, the two-pass approach may be advantageous for specialized fonts or low-quality image data. Another exemplary OCR software tool is OCRopus. Development of OCRopus is led by the German Research Center for Artificial Intelligence (DFKI) in Kaiserslautern, Germany. In some cases, the OCR software may employ neural networks.

[0075] Still referring to FIG. 1, in some cases, OCR can include post-processing. For example, OCR accuracy can be improved in some cases when the output is constrained by a lexicon. The lexicon may include a list or set of words that are allowed to occur in the document. In some cases, the lexicon may include, for example, all words in the English language, or a more specialized lexicon for a particular domain. In some cases, the output stream may be a plain text stream or a file of characters. In some cases, the OCR process can preserve the original layout of the image data. In some cases, near-neighbor analysis can utilize co-occurrence frequency and correct errors by noting that certain words are frequently found together. For example, "Washington, DC" is usually much more common in English than "Washington DOC." In some cases, the OCR process can utilize a priori knowledge of the grammar of the language being recognized. For example, grammar rules may be used to determine whether a word is a verb or a noun. Distance conceptualizations can be employed for recognition and classification. For example, the Levenshtein distance algorithm can be used in post-processing of the OCR to further optimize the results.

[0076] 1, in some embodiments, the device 100 can remove artifacts from an image. As used herein, an "artifact" is a visual imprecision, an element of an image that distracts from an element of interest, an element of an image that obscures an element of interest, or any other undesirable element of an image.

[0077] Still referring to FIG. 1 , device 100 may include an image processing module. An "image processing module" as used in this disclosure is a component designed to process digital images. In one embodiment, the image processing module may include software algorithms that can analyze, manipulate, or otherwise improve an image, such as, but not limited to, image processing techniques as described below. In another embodiment, the image processing module may include hardware components, such as, but not limited to, one or more graphic processing units (GPUs) that can speed up the processing of large amounts of images. In some cases, the image processing module may be implemented using one or more image processing libraries, such as, but not limited to, OpenCV, PIL / Pillow, ImageMagick, etc.

[0078] 1, the image processing module may be configured to receive images from the optical sensor 120. One or more images may be transmitted from the optical sensor 120 to the image processing module via any suitable electronic communication protocol, including, but not limited to, packet-based protocols such as Transmission Control Protocol / Internet Protocol (TCP-IP), File Transfer Protocol (FTP), etc. Receiving the images may include retrieving the images from a data store that contains the images, as described below. For example, but not limited to, the images may be retrieved using a query that specifies a timestamp that the images are required to match.

[0079] Still referring to FIG. 1 , the image processing module may be configured to process the image. In one embodiment, the image processing module may be configured to compress and / or encode the image to reduce file size and storage requirements while maintaining essential visual information necessary for further processing steps as described below. In one embodiment, compressing and / or encoding the image may facilitate high speed transmission of the image. In some cases, the image processing module may be configured to perform lossless compression on the image, which can maintain the original image quality of the image. In non-limiting examples, the image processing module may perform lossless compression on the image, which can maintain the original image quality of the image, including but not limited to Huffman coding, Lempel-Ziv-Welch (LZW), Run-Length Compression (LLC), and lossless compression. One or more lossless compression algorithms, such as Random Leakage Encoding (RLE), may be utilized to identify and remove image redundancy without losing information. In such an embodiment, compressing and / or encoding each image of the image may include converting the file format of each image to PNG, GIF, lossless JPEG2000, or the like. In one embodiment, an image compressed via lossless compression may be fully reconstructed to the image's original form (e.g., the original image's resolution, dimensions, color representation, format, etc.). In another case, the image processing module may be configured to perform lossy compression on the image, which may sacrifice some image quality to achieve a higher compression ratio. In a non-limiting example, the image processing module may utilize one or more lossy compression algorithms, such as, but not limited to, JPEG's Discrete Cosine Transform (DCT) or JPEG2000's Wavelet Transform, to discard less important information in the image, resulting in a smaller file size but with only a small loss in image quality. In such an embodiment, compressing and / or encoding the image may include converting the file format of each image to JPEG, WebP, lossy JPEG2000, or the like.

[0080] Still referring to FIG. 1, in one embodiment, processing the images can include determining a degree of depiction quality of the region of interest of the image. In one embodiment, the image processing module can determine the blurriness of the image. In a non-limiting example, the image processing module can perform blur detection by taking a Fourier transform, or an approximation such as a Fast Fourier Transform (FFT), of the image and analyzing the distribution of low and high frequencies in the resulting frequency domain depiction of the image. For example, but not by way of limitation, the number of high frequency values ​​below a threshold level may indicate blurriness. In another non-limiting example, blur detection can be performed by convolving the images, channels of the image, etc. with a Laplacian kernel, which can generate a numerical score reflecting the number of abrupt changes in intensity shown in each image, such that a high score indicates sharpness and a low score indicates blurriness. In some cases, blur detection can be performed using a gradient-based operator that measures the operator based on the gradient or first derivative of the image, based on the hypothesis that abrupt changes indicate sharp edges in the image and therefore a low degree of blurriness. In some cases, blur detection may be performed using a wavelet-based operator that utilizes the ability of discrete wavelet transform coefficients to describe the frequency and spatial content of an image. In some cases, blur detection may be performed using a statistics-based operator that utilizes some image statistics as texture descriptors to calculate the focus level. In other cases, blur detection may be performed using discrete cosine transform (DCT) coefficients to calculate the focus level of an image from its frequency content. Additionally or alternatively, the image processing module may be configured to rank the images according to a degree of quality of depiction of the region of interest and select the highest ranked image from the plurality of images.

[0081] Still referring to FIG. 1 , processing the image may include enhancing the image or at least one region of interest by a number of image processing techniques to improve the quality (or degree of quality of depiction) of the image for better processing and analysis, as further described in this disclosure. In one embodiment, the image processing module may be configured to perform a noise reduction operation on the image, which may remove or minimize noise (caused by various causes such as sensor limitations, poor lighting conditions, image compression, etc.), resulting in a cleaner, more visually consistent image. In some cases, the noise reduction operation may be performed using one or more image filters, for example, but not limited to, the noise reduction operation may include Gaussian filtering, median filtering, bilateral filtering, etc. The noise reduction process may be performed by the image processing module by averaging or filtering pixel values ​​in the neighborhood of each pixel of the image to reduce random fluctuations.

[0082] Still referring to FIG. 1, in another embodiment, the image processing module may be configured to perform a contrast enhancement operation on the image. In some cases, the image may exhibit low contrast, e.g., features may be difficult to distinguish from the background. A contrast enhancement operation may improve the contrast of the image by expanding the intensity range of the image and / or redistributing the intensity values ​​(i.e., the degree of lightness or darkness of pixels in the image). In a non-limiting example, the intensity values ​​represent the gray level or color of each pixel and may be scaled from an intensity range of 0 to 255 for an 8-bit image and from 0 to 16,777,215 for a 24-bit color image. In some cases, the contrast enhancement operation may include, but is not limited to, histogram equalization, adaptive histogram equalization (CLAHE), contrast stretching, and the like. The image processing module may be configured to adjust the light and dark levels in the image to make features more distinguishable (i.e., to increase the degree of depiction quality). Additionally or alternatively, the image processing module may be configured to perform a brightness normalization operation to correct for variations in lighting conditions (i.e., non-uniform brightness levels). In some cases, the image may contain consistent brightness levels across an entire region after the brightness normalization operation performed by the image processing module. In a non-limiting example, the image processing module may perform a global normalization or a local mean normalization, where a mean intensity value for the entire image or a region of the image may be calculated and used to adjust the brightness level.

[0083] Still referring to FIG. 1, in other embodiments, the image processing module may be configured to perform color space conversion operations to enhance the degree of rendering quality. In a non-limiting example, in the case of color images (i.e., RGB images), the image processing module may be configured to convert the RGB image to grayscale or HSV color space. Such conversion may enhance the difference in intensity values ​​between the region or feature of interest and the background. The image processing module may be further configured to perform image sharpening operations, such as, but not limited to, unsharp masking, Laplacian sharpening, high-pass filtering, and the like. The image processing module may use image sharpening operations to enhance edges and fine details associated with the region or feature of interest in the image by enhancing high frequency components in the image.

[0084] Still referring to FIG. 1, processing the image may include isolating a region or feature of interest from the remainder of the image as a function of a number of image processing techniques. The image may include the highest ranked image selected by the image processing module as described above. In one embodiment, the number of image processing techniques may include one or more morphological operations, which are techniques developed based on set theory, lattice theory, topology, and random functions used to process geometric structures using a structuring element. For purposes of this disclosure, a "structuring element" is a small matrix or kernel that defines the shape and size of a morphological operation. In some cases, a structuring element may be placed at the center of each pixel of the image and used to determine an output pixel value for that location. In a non-limiting example, isolating a region or feature of interest from the image may include applying a dilation operation, which is a basic morphological operation configured to expand or grow the boundaries of objects (e.g., cells, dust particles, etc.) in the image. In another non-limiting example, isolating a region or feature of interest from the image may include applying an erosion operation, which is a basic morphological operation configured to shrink or shrink the boundaries of objects in the image. In another non-limiting example, isolating the region of interest or feature from the image may include applying an opening operation, which is a basic morphological operation configured to remove small objects or thin structures from the image while preserving larger structures. In a further non-limiting example, isolating the region of interest or feature from the image may include applying a closing operation, which is a basic morphological operation configured to fill small gaps or holes in objects in the image while preserving the overall shape and size of the objects. These morphological operations can be performed by an image processing module to enhance edges of objects, remove noise, or fill gaps in the region of interest or feature prior to further processing.

[0085] Still referring to FIG. 1, in one embodiment, isolating a region or feature of interest from an image may include utilizing edge detection techniques that may detect one or more shapes defined by edges. As used in this disclosure, "edge detection techniques" include mathematical methods that identify points in a digital image where the image brightness changes abruptly and / or there is a discontinuity. In one embodiment, such points may be organized into straight and / or curved line segments called "edges." Edge detection techniques may be performed by the image processing module using any suitable edge detection algorithm, including but not limited to Canny edge detection, Sobel operator edge detection, Prewitt operator edge detection, Laplacian operator edge detection, and / or differential edge detection. Edge detection techniques may include edge detection based on phase congruency, which finds all locations in an image where all sinusoids in the frequency domain, generated, for example, using Fourier decomposition, may have a matching phase that may indicate the location of an edge. Edge detection techniques may be used to detect the shape of a feature of interest, such as a cell, indicative of a cell membrane or cell wall. In one embodiment, edge detection techniques may be used to find closed figures formed by edges.

[0086] Still referring to FIG. 1 , in a non-limiting example, isolating the feature of interest from the image can include determining the feature of interest by edge detection techniques. The feature of interest can include a particular region in the digital image that contains information relevant for further processing as described below. In a non-limiting example, image data located outside the feature of interest can include irrelevant or redundant information. Portions of the image that contain irrelevant or redundant information can be ignored by the image processing module, thereby allowing resources to be focused on the feature of interest. In some cases, the feature of interest can vary in size, shape, and / or location within the image. In a non-limiting example, the feature of interest can be depicted as a circle surrounding a cell nucleus. In some cases, the feature of interest can specify one or more coordinates, distances, etc., such as the center and radius of the circle surrounding the cell nucleus in the image. The image processing module can then be configured to isolate the feature of interest from the image based on the feature of interest. In a non-limiting example, the image processing module can crop the image according to a bounding box that surrounds the feature of interest.

[0087] Still referring to FIG. 1, the image processing module may be configured to perform connected component analysis (CCA) on the image for feature of interest isolation. As used in this disclosure, "connected component analysis (CCA)", also known as connected component labeling, is an image processing technique used to identify and label connected regions in a binary image (i.e., an image in which each pixel has only two possible values: 0 or 1, black or white, or foreground or background). A "connected region" as described herein is a group of adjacent pixels that share the same value and are connected based on a predefined neighborhood system, such as, but not limited to, a 4-connected neighborhood or an 8-connected neighborhood. In some cases, the image processing module may convert the image to a binary image by thresholding, which may include setting a threshold that separates pixels of the image corresponding to the feature of interest (foreground) from pixels corresponding to the background. Pixels with intensity values ​​above the threshold may be set to 1 (white) and pixels below the threshold may be set to 0 (black). In one embodiment, CCA may be employed to detect and extract features of interest by identifying multiple connected regions that exhibit certain characteristics or features of the feature of interest. The image processing module can then filter the connected regions by analyzing characteristics of the connected regions, such as, but not limited to, area, aspect ratio, height, width, perimeter, etc. In a non-limiting example, the image processing module may retain connected components that closely resemble the dimensions and aspect ratio of the feature of interest as the feature of interest, while discarding other components. The image processing module may be further configured to extract the feature of interest from the image for further processing, as described below. Still referring to FIG. 1, in one embodiment, isolating the feature of interest from the image may include dividing a region depicting the feature of interest into a plurality of sub-regions.

[0088] Dividing a region into sub-regions can include dividing the region as a function of the features of interest and / or the CCA via an image segmentation process. An "image segmentation process" as used in this disclosure is a process of dividing a digital image into one or more segments, each segment representing a different portion of the image. The image segmentation process can change the representation of the image. The image segmentation process can be performed by an image processing module. In a non-limiting example, the image processing module can perform region-based segmentation, which includes growing regions from one or more seed points or pixels on the image based on similarity criteria. The similarity criteria may include, but are not limited to, color, intensity, texture, and the like. In a non-limiting example, the region-based segmentation can include region growing, region merging, watershed algorithms, and the like.

[0089] 1, in some embodiments, the device 100 can remove artifacts identified by the machine vision system or optical character recognition system described above. Non-limiting examples of artifacts that may be removed include dust particles, air bubbles, cracks in the slide 116, writing on the slide 116, shadows, visual noise such as grainy images, etc. In some embodiments, the artifacts may be partially removed and / or become less visible.

[0090] Still referring to FIG. 1 , in some embodiments, artifacts may be removed using an artifact removal machine learning model. In some embodiments, the artifact removal machine learning model may be trained on a dataset that includes images associated with artifact-free images. In some embodiments, the artifact removal machine learning model may accept as input an image that includes artifacts and output an image that is artifact-free. For example, the artifact removal machine learning model may accept as input an image that includes air bubbles in a slide and output an image that does not include air bubbles. In some embodiments, the artifact removal machine learning model may include a generative machine learning model, such as a diffusion model. A diffusion model may learn the structure of a dataset by modeling how data points diffuse through a latent space. In some embodiments, artifact removal may be performed locally. For example, the device 100 may include an already trained artifact removal machine learning model and apply the model to the image. In some embodiments, artifact removal may be performed externally. For example, the device 100 may send image data to another computing device and receive an image with artifacts removed. In some embodiments, artifacts may be removed in real time. In some embodiments, artifacts may be removed based on identification by a user. For example, a user can use a mouse cursor to drag a box around the artifacts, and the apparatus 100 can remove the artifacts within the box.

[0091] Still referring to FIG. 1 , in some embodiments, device 100 may display an image to a user. In some embodiments, the first image may be displayed to the user in real time. In some embodiments, the image may be displayed to the user using output interface 132. For example, the first image may be displayed on a display such as a screen. In some embodiments, the first image may be displayed to the user in the context of a graphical user interface (GUI). For example, the GUI may include controls to navigate the image, such as zoom controls and controls to change the location being displayed. The GUI may include a touch screen.

[0092] Still referring to FIG. 1, in some embodiments, the device 100 can receive a parameter set from a user. As used herein, a "parameter set" is a set of values ​​that identify how to capture an image. A parameter set may be implemented as a data structure, as described below. In some embodiments, the device 100 can receive a parameter set from a user using the input interface 128. The parameter set can include X and Y coordinates that indicate where the user wants to look. The parameter set can include a desired magnification level. As used herein, a "magnification level" is an item of data that indicates how much to zoom in / out to capture an image. The magnification level can take into account optical zoom and / or digital zoom. As a non-limiting example, the magnification level can be "8x." The parameter set may include a desired depth of focus and / or focal length of the sample. In a non-limiting example, the user can manipulate the input interface 128 such that the parameter set includes X and Y coordinates and a magnification level that corresponds to a more magnified view of a particular area of ​​the sample. In some embodiments, the parameter set corresponds to a more magnified view of a particular area of ​​the sample included in the first image. This may be done, for example, to obtain a more detailed view of a small object. As used herein, unless otherwise specified, "X coordinate" and "Y coordinate" refer to coordinates along the vertical axis, and the plane defined by these axes is parallel to the plane of the slide 116 surface. In some cases, setting the magnification may include changing one or more optical elements in the optical system. For example, setting the magnification may include replacing a first objective lens with a second objective lens having a different magnification. Additionally, replacing one or more optical components that "beam down" from the objective lens can change the overall magnification of the optical system and set the magnification. In some cases, setting the magnification may include changing the digital magnification. Digital magnification includes outputting the image at a different resolution, i.e., after rescaling the image, using an output interface.In some embodiments, device 100 can capture an image with a particular field of view, which can be determined, for example, based on the level of magnification or how wide the camera captures the image.

[0093] Still referring to FIG. 1 , in some embodiments, the apparatus 100 can move one or more of the slide port 140, the slide 116, and the at least one optical system 120 to a second position. In some embodiments, the location of the second position can be based on a parameter set. In some embodiments, the location of the second position can be based on an identification of an area of ​​interest, a line, or a point, as described above. The second position can be determined, for example, by changing the position of the optical system 120 relative to the slide 116 based on the parameter set. For example, the parameter set can indicate that the second position is achieved by changing the X coordinate by 5 mm in a particular direction. In this example, the second position can be determined by changing the original position of the optical system by 5 mm in that direction. In some embodiments, such movement can be performed using the actuator mechanism 124. In some embodiments, the actuator mechanism 124 can move the slide port 140 such that the slide 116 is in a position relative to the at least one optical system 120 where the optical sensor 120 can capture an image as dictated by the parameter set. For example, the slide 116 can rest on the slide port 140, and movement of the slide port 140 can move the slide 116. In some embodiments, the actuator mechanism 124 can move the slide 116 such that the slide 116 is in a position relative to the at least one optical system 120 such that the optical sensor 120 can capture an image as dictated by the set of parameters. For example, the slide 116 can be connected to the actuator mechanism 124 such that the actuator mechanism 124 can move the slide 116 relative to the at least one optical system 120. In some embodiments, the actuator mechanism 124 can move the at least one optical system 120 such that the slide 116 is in a position relative to the slide 116 such that the optical sensor 120 can capture an image as dictated by the set of parameters.For example, the slide 116 may be stationary and the actuator mechanism 124 may move the at least one optical system 120 to a position relative to the slide 116. In some embodiments, the actuator mechanism 124 may move more than one of the slide port 140, the slide 116, and the at least one optical system 120 so that they are in the correct relative positions. In some embodiments, the actuator mechanism 124 may move the slide port 140, the slide 116, and / or the at least one optical system 120 in real time. For example, user input of a set of parameters may result in substantially instantaneous movement of an item by the actuator mechanism 124.

[0094] Still referring to FIG. 1, in some embodiments, the device 100 may capture a second image of the slide 116 at a second position. In some embodiments, the device 100 may capture the second image using at least one optical system 120. In some embodiments, the second image may include an image of an area of ​​the sample. In some embodiments, the second image may include an image of an area captured in the first image. For example, the second image may include a larger magnified image of an area in the first image, with a higher resolution per unit area. In another example, the second image may be captured using a focal distance based on a focal pattern. This allows the second image to be displayed, allowing the user to detect smaller details in the imaged area.

[0095] 1, in some embodiments, the second image includes an X and Y coordinate shift relative to the first image. For example, the second image may partially overlap the first image.

[0096] Still referring to FIG. 1 , in some embodiments, the device 100 may capture the second image in real time. For example, the user may manipulate the input interface 128 to create a parameter set, the actuator mechanism 124 may begin moving the slide 116 relative to the optical system 120 substantially immediately after the input interface 128 is manipulated, and the optical system 120 may capture the second image substantially immediately after the actuator mechanism 124 completes its movement. In some embodiments, artifacts may also be removed in real time. In some embodiments, the images may also be annotated in real time. In some embodiments, the focus pattern may also be determined in real time, and images taken after the first image may be taken with a focal length according to the focus pattern.

[0097] Still referring to FIG. 1, in some embodiments, the device 100 can display a second image to the user. In some embodiments, the second image can be displayed to the user using the output device 132. The second image can be displayed as described above with respect to the output device and displaying the first image. In some embodiments, displaying the second image to the user can include replacing an area of ​​the first image with the second image to generate a hybrid image and displaying the hybrid image to the user. As used herein, a "hybrid image" is an image constructed by combining a first image and a second image. In some embodiments, generating a hybrid image in this manner can preserve the second image. For example, if the second image has a higher resolution per unit area and covers a smaller area than the first image, the second image can replace a lower resolution per unit area segment of the first area corresponding to the area covered by the second image. In some embodiments, at the boundary between the first image and the second image in the hybrid image, image adjustments can be made to offset visual differences between the first image and the second image. In a non-limiting example, colors may be adjusted so that the background color of an image is constant along the borders of the image. As another non-limiting example, the brightness of an image may be adjusted so that there is no noticeable difference between the brightness of the images. In some embodiments, artifacts may be removed from the second image and / or the hybrid image, as described above. In some embodiments, the second image may be displayed to the user in real time. For example, adjustments (such as annotations and / or artifact removal) may begin substantially immediately after the second image is captured, and an adjusted version of the second image may be displayed to the user substantially immediately after the adjustments are made. In some embodiments, an unadjusted version of the second image may be displayed to the user while the adjustments are being made. In some embodiments, if there are multiple images covering a particular area, a lower resolution image of the area may be displayed when the user zooms out using the user interface 136.

[0098] 1, in some embodiments, apparatus 100 may transmit a data structure including the first image, the second image, the hybrid image, and / or the multiple images to an external device. Such external devices may include, in non-limiting examples, a phone, a tablet, or a computer. In some embodiments, such transmission may configure the external device to display the images.

[0099] Still referring to FIG. 1 , in some embodiments, the device 100 can annotate an image. In some embodiments, the device 100 can annotate a first image. In some embodiments, the device 100 can annotate a second image. In some embodiments, the device 100 can annotate a hybrid image. For example, during creation of the hybrid image, the device 100 can recognize a cell depicted in the hybrid image as a particular type of cell and annotate the hybrid image with an indication of the type of cell. In a non-limiting example, the device 100 can associate text with a particular location in the image, where the text describes a feature present at that location in the image. In some embodiments, the device 100 can annotate an image selected from a list consisting of the first image, the second image, and the hybrid image.

[0100] Still referring to FIG. 1 , in some embodiments, annotation may be performed as a function of a user input of annotation instructions to the input interface 128. An “annotation instruction,” as used herein, is data generated based on a user input indicating whether to create an annotation or describing an annotation to be created. For example, a user may select an option that controls whether the device 100 annotates an image. In another example, a user may manually annotate an image. In some embodiments, annotation may be performed automatically. In some embodiments, annotation may be performed using an annotation machine learning model. In some embodiments, the annotation machine learning model may include an optical character recognition model, as described above. In some embodiments, the annotation machine learning model may be trained using a dataset that includes image data associated with text depicted in the image. In some embodiments, the annotation machine learning model may accept input image data and may output annotated image data and / or annotations to apply to the image data. In some embodiments, the annotation machine learning model may be used to convert text written on the slide 116 into annotations on the image. In some embodiments, the annotation machine learning model may include a machine vision model, as described above. In some embodiments, the annotation machine learning model, including a machine vision model, may be trained on a dataset that includes image data associated with annotations that indicate features of the image data. In some embodiments, the annotation machine learning model may accept input image data and may output annotated image data and / or annotations to apply to the image data. Non-limiting examples of features that the annotation machine learning model may be trained to recognize include cell types, cell features, and objects within the slide 116, such as air bubbles. In some embodiments, images may be annotated in real-time. For example, annotation may begin immediately after the image is captured and / or immediately after a command to annotate the image is received, and the annotated image may be displayed to the user immediately after annotation is completed.

[0101] Still referring to FIG. 1 , in some embodiments, device 100 can determine a visual element data structure. In some embodiments, device 100 can display visual elements to a user as a function of the visual element data structure. As used herein, a "visual element data structure" is a data structure that describes a visual element. As non-limiting examples, the visual elements can include a first image, a second image, a hybrid image, and an element of a GUI.

[0102] Still referring to FIG. 1 , in some embodiments, the visual element data structure may include visual elements. As used herein, a “visual element” is data that is visually displayed to a user. In some embodiments, the visual element data structure may include rules for displaying visual elements. In some embodiments, the visual element data structure may be determined as a function of the first image, the second image, and / or the hybrid image. In some embodiments, the visual element data structure may be determined as a function of items from a list consisting of the first image, the second image, the hybrid image, a GUI element, and an annotation. In a non-limiting example, the visual element data structure may be generated such that visual elements describing features of the first image, such as annotations, are displayed to a user.

[0103] 1, in some embodiments, the visual elements can include one or more elements of text, images, shapes, charts, particle effects, interactable features, etc. As a non-limiting example, the visual elements can include a touch screen button that sets a magnification level.

[0104] 1, the visual element data structure may include rules governing whether or when a visual element is displayed. In a non-limiting example, the visual element data structure may include a rule that causes a visual element including annotations describing the first image, the second image, and / or the hybrid image to be displayed when a user selects a particular area of ​​the first image, the second image, and / or the hybrid image using the GUI.

[0105] 1, the visual element data structure can include multiple visual elements or rules for presenting multiple visual elements at one time. In one embodiment, about 1, 2, 3, 4, 5, 10, 20, or 50 visual elements are displayed simultaneously. For example, multiple annotations can be displayed simultaneously.

[0106] Still referring to FIG. 1 , in some embodiments, the device 100 can transmit the visual elements to a display, such as the output interface 132. The display can convey the visual elements to a user. The display can include, for example, a smartphone screen, a computer screen, a tablet screen, etc. The display can be configured to provide a visual interface. The visual interface can include one or more virtual interactive elements, such as, but not limited to, buttons, menus, etc. The display can include one or more physical interactive elements, such as buttons, a computer mouse, or a touch screen, that allow a user to input data into the display. The interactive elements can be configured to enable interaction between a user and a computing device. In some embodiments, the visual element data structure is determined as a function of data input by a user into the display.

[0107] Still referring to FIG. 1 , the variables and / or data described herein may be represented as data structures. In some embodiments, the data structures may include one or more functions and / or variables, such as a class in object-oriented programming. In some embodiments, the data structures may include data in the form of a Boolean, an integer, a floating point, a string, a date, etc. In a non-limiting example, the annotation data structure may include a string value representing the text of the annotation. In some embodiments, the data of the data structure may be organized as a linked list, a tree, an array, a matrix, a tensor, etc. In a non-limiting example, the annotation data structure may be organized in an array. In some embodiments, the data structure may include or be associated with one or more elements of metadata. The data structure may include one or more self-referential data elements that the processor 104 may use in interpreting the data structure. In a non-limiting example, the data structure may include a "tag" indicating that the content between the tags is a date. <date> "and"< / date> " tag.

[0108] Still referring to FIG. 1, the data structure may be stored in, for example, memory 108 or a database. The database may be implemented as, but not limited to, a relational database, a key-value database such as a NOSQL database, or any other format or structure for use as a database that one of ordinary skill in the art would recognize as appropriate upon reviewing the entirety of this disclosure. The database may alternatively or additionally be implemented using a distributed data storage protocol and / or data structure, such as a distributed hash table. The database may include multiple data entries and / or records, as described above. Data entries in the database may be flagged or linked to one or more additional information elements, which may be reflected in data entry cells and / or linked tables, such as tables related by one or more indexes in a relational database. One of ordinary skill in the art, upon reviewing the entirety of this disclosure, will recognize various ways in which data entries in a database may store, search, organize, and / or reflect the data and / or records, and categories and / or populations of data, used herein, consistent with this disclosure.

[0109] 1, in some embodiments, the data structure may be read and / or manipulated by the processor 104. In a non-limiting example, the image data structure may be read and displayed to a user. In another non-limiting example, the image data structure may be modified to remove artifacts, as described above.

[0110] Still referring to FIG. 1, in some embodiments, the data structure may be calibrated. In some embodiments, the data structure may be trained using a machine learning algorithm. In a non-limiting example, the data structure may include an array of data representing biases of connections of a neural network. In this example, the neural network may be trained on a set of training data, and a backpropagation algorithm may be used to correct the data of the array. Machine learning models and neural networks are further described herein.

[0111] One or more features of the device 100 may correspond to features disclosed in one or more of the following: (A) U.S. Patent Application No. 18 / 217,378, filed July 25, 2023, entitled "APPARATUS AND A METHOD FOR DETECTING ASSOCIATIONS AMONG DATASETS OF DIFFERENT TYPES," which is incorporated herein by reference in its entirety; (B) U.S. Patent Application No. 18 / 226,017, filed July 25, 2023, entitled "APPARATUS AND A METHOD FOR GENERATING A CONFIDENCE SCORE ASSOCIATED WITH A SCANNED LABEL," which is incorporated herein by reference in its entirety; or (C) U.S. Patent Application No. 18 / 226,058, filed July 25, 2023, entitled "IMAGING DEVICE AND A METHOD FOR IMAGE GENERATION OF A SPECIMEN" (incorporated herein by reference in its entirety) (D) U.S. Patent Application No. 18 / 226,100, filed July 25, 2023, entitled "APPARATUS AND METHODS FOR REAL-TIME IMAGE GENERATION" (incorporated herein by reference in its entirety).

[0112] 2, an exemplary embodiment of a machine learning module 200 is illustrated that may perform one or more machine learning processes as described in this disclosure. The machine learning module may use the machine learning processes to perform the determining, classifying, and / or analyzing steps, methods, processes, etc. as described in this disclosure. As used in this disclosure, a "machine learning process" is a process that automatically uses training data 204 to generate algorithms instantiated in hardware or software logic, data structures, and / or functions executed by a computing device / module to generate an output 208 when data is provided as input 212, in contrast to a non-machine learning software program in which the commands to be executed are predetermined by a user and written in a programming language.

[0113] Still referring to FIG. 2, as used herein, "training data" is data that includes correlations that a machine learning process can use to model relationships between two or more categories of data elements. For example, but not by way of limitation, the training data 204 may include a plurality of data entries, also known as "training examples," each of which represents a set of data elements recorded, received, and / or generated together, where the data elements may be correlated by the presence of commonality in a given data entry, by closeness in a given data entry, etc. The plurality of data entries of the training data 204 may indicate one or more trends in correlations between categories of data elements. For example, but not by way of limitation, a high value of a first data element belonging to a first category of data elements tends to correlate with a high value of a second data element belonging to a second category of data elements, indicating the possibility of a proportional or other mathematical relationship linking values ​​belonging to the two categories. The plurality of categories of data elements may be associated in the training data 204 according to various correlations. Correlations may indicate causal and / or predictive links between categories of data elements and may be modeled as relationships, such as mathematical relationships, by machine learning processes, as described in further detail below. Training data 204 may be formatted and / or organized by categories of data elements, for example, by associating the data elements with one or more descriptors that correspond to the categories of the data elements. As a non-limiting example, training data 204 may include data entered into a standardized form by a person or process such that entry of a given data element in a given field of the form is mapped to one or more descriptors of the category. Elements of training data 204 may be linked to the descriptors of the category by tags, tokens, or other data elements.For example, and without limitation, the training data 204 may be provided in a format that links the location of the data to a category, such as a fixed-length format, a comma-separated values ​​(CSV) format, and / or a self-describing format, such as Extensible Markup Language (XML) or JavaScript Object Notation (JSON), that allows a process or device to detect the category of the data.

[0114] Alternatively or additionally, with continued reference to FIG. 2, the training data 204 may include one or more elements that are uncategorized. That is, the training data 204 may be unformatted or may not include descriptors for some elements of the data. Machine learning algorithms and / or other processes may sort the training data 204 according to one or more categorizations, for example, using natural language processing algorithms, tokenization, detecting correlation values ​​in the raw data, etc. Categories may be generated using correlation and / or other processing algorithms. As a non-limiting example, in a corpus of text, phrases that make up "n" compounds, such as nouns modified by other nouns, may be identified according to the statistically significant prevalence of n-grams that contain such words in a particular order. Such n-grams may be categorized as elements of language such as "words" that are tracked similarly to single words, generating new categories as a result of the statistical analysis. Similarly, in a data entry containing text data, a person's name may be identified by reference to a list, dictionary, or other glossary, allowing for ad-hoc categorization by a machine learning algorithm and / or automatic association of the data in the data entry to a descriptor or a given format. The ability to automatically categorize data entries allows the same training data 204 to be applied to two or more different machine learning algorithms, as described in more detail below. The training data 204 used by the machine learning module 200 may correlate any input data, as described in this disclosure, with any output data, as described in this disclosure. As a non-limiting example, the input may include an image of a region of interest, and the output may include a determination of whether a sample is present.

[0115] With further reference to FIG. 2, the training data may be filtered, sorted, and / or selected using one or more supervised and / or unsupervised machine learning processes and / or models, as described in further detail below, including, but not limited to, the training data classifier 216. The training data classifier 216 may include a "classifier," which as used in this disclosure is a machine learning model, as defined below, and is a data structure that represents and / or uses a mathematical model, neural net, or program generated by a machine learning algorithm known as a "classification algorithm," as described in further detail below, for example, which sorts inputs into categories or bins of data and outputs the categories or bins of data and / or their associated labels. The classifier may be configured to output at least one datum that labels or otherwise identifies a set of data, such as those that have been clustered, found to be close under a distance metric as described below, etc. The distance metric may include any norm, such as, but not limited to, the Pythagorean norm. The machine learning module 200 may generate a classifier using a classification algorithm, which is defined as a process in which the computing device and / or any modules and / or components operating on the computing device derive a classifier from the training data 204. The classification may be performed using a linear classifier, such as, but not limited to, a logistic regression and / or a naive Bayes classifier, a nearest neighbor classifier, such as a k-nearest neighbor classifier, a support vector machine, a least squares support vector machine, Fisher's linear discriminant, a quadratic classifier, a decision tree, a boosted tree, a random forest classifier, learning vector quantization, and / or a neural network based classifier. As a non-limiting example, the training data classifier 216 may classify elements of the training data as being present or absent of the sample.

[0116] With further reference to FIG. 2, the training examples used as training data may be selected from a population of potential examples according to a cohort relevant to the analytical problem, classification task, etc. to be solved. Alternatively or additionally, the training data may be selected to span a set of situations or inputs that the machine learning model and / or process is likely to encounter upon deployment. For example, but not limited to, for each category of input data to a machine learning process or model that may be present in a range of values ​​in a population of phenomena such as images, user data, processing data, physical data, etc., the computing device, processor, and / or machine learning model may select training examples that represent each possible value on such range, and / or a representative sample of values ​​on such range. Selection of representative samples may include selecting training examples in a proportion that matches a statistically determined and / or predicted distribution of such values ​​according to relative frequency, e.g., such that values ​​that are more frequently encountered in the population of analyzed data are represented by more training examples than values ​​that are less frequently encountered. Alternatively or additionally, the set of training examples may be compared to a collection of representative values ​​in a database and / or presented to a user, allowing the process to detect, automatically or via user input, one or more values ​​not included in the set of training examples. A computing device, processor, and / or module may automatically generate missing training examples by receiving and / or acquiring missing input and / or output values ​​and associating the missing input and / or output values ​​with corresponding output and / or input values ​​that coexist in the data record with the acquired values, such as provided by a user and / or other device.

[0117] Still referring to FIG. 2, the computer, processor, and / or module may be configured to sanitize the training data. As used in this disclosure, "sanitizing" the training data is a process in which training examples that prevent the convergence of a machine learning model and / or process to a useful result are removed. For example, but not limited to, the training examples may include input and / or output values ​​that are outliers from typically encountered values ​​so that the machine learning algorithm using the training examples adapts to unlikely quantities as inputs and / or outputs. For example, values ​​that are more than a threshold number of standard deviations away from the average, mean, or expected value may be removed. Alternatively or additionally, one or more training examples may be identified as having low quality data, where "low quality" is defined as having a signal-to-noise ratio below a threshold.

[0118] As a non-limiting example, and with further reference to FIG. 2, images used to train an image classifier or other machine learning model and / or process that takes images as input or produces images as output may be rejected if the image quality is below a threshold. For example, without limitation, a computing device, processor, and / or module may perform blur detection and eliminate one or more blurs. Blur detection may be performed by, as a non-limiting example, taking a Fourier transform, or an approximation such as a Fast Fourier Transform (FFT), of the image and analyzing the distribution of low and high frequencies in the resulting frequency domain representation of the image. The number of high frequency values ​​below a threshold level may indicate blur. In a further non-limiting example, blur detection may be performed by convolving the image, a channel of the image, or the like, with a Laplacian kernel, which may generate a numerical score reflecting the number of abrupt changes in intensity shown in the image, with a high score indicating sharpness and a low score indicating blur. Blur detection can be performed using gradient-based operators that measure the operator based on the gradient or first derivative of the image, with the hypothesis that abrupt changes indicate sharp edges in the image and therefore less blur. Blur detection may be performed using wavelet-based operators that exploit the ability of discrete wavelet transform coefficients to describe the frequency and spatial content of an image. Blur detection may be performed using statistics-based operators that exploit some image statistics as texture descriptors to calculate the focus level. Blur detection may be performed using discrete cosine transform (DCT) coefficients to calculate the focus level of an image from its frequency content.

[0119] Still referring to FIG. 2, the computing device, processor, and / or module may be configured to be preconditioned on one or more training examples. For example, but not by way of limitation, if a machine learning model and / or process has one or more inputs and / or outputs that require, transmit, or receive a certain number of bits, samples, or other units of data, the elements of one or more training examples used as inputs and / or outputs or compared to the inputs and / or outputs may be modified to have such units of data. For example, the computing device, processor, and / or module may convert a smaller number of units, such as an image with a low number of pixels, to a desired number of units, such as by upsampling or interpolation. As a non-limiting example, an image with a low number of pixels may have 100 pixels, but the desired number of pixels may be 128 pixels. The processor may interpolate the image with a low number of pixels to convert the 100 pixels to 128 pixels. It should also be noted that one of ordinary skill in the art would know, upon reading this disclosure, various methods of interpolating a smaller number of data units, such as samples, pixels, bits, etc., to a desired number of such units. In some examples, the set of interpolation rules may be trained with highly detailed inputs and / or outputs, a corresponding set of inputs and / or outputs downsampled to a smaller number of units, and a neural network or other machine learning model trained to predict the interpolated pixel values ​​using the training data. As a non-limiting example, sample inputs and / or outputs, such as a sample image having sample augmented data units (e.g., pixels added between original pixels), may be input to a neural network or machine learning model, which may output a pseudo-replica sample image in which pixels between the original pixels are assigned dummy values ​​based on the set of interpolation rules.As a non-limiting example, in the context of an image classifier, a machine learning model may have a set of interpolation rules trained with a set of high definition images and images downsampled to a smaller number of pixels, and a neural network or other machine learning model trained using those examples to predict interpolated pixel values ​​in the context of face images. As a result, an input with sample expanded data units (with dummy values ​​added between the original data units) can be run through the trained neural network and / or model to pad values ​​to replace the dummy values. Alternatively or additionally, the processor, computing device, and / or module may utilize a sample expander method, a low pass filter, or both. A "low pass filter" as used in this disclosure is a filter that passes signals with frequencies below a selected cutoff frequency and attenuates signals with frequencies above the cutoff frequency. The exact frequency response of the filter depends on the filter design. The computing device, processor, and / or module may use averaging, such as luma averaging or chroma averaging in the image, to pad data units between the original data units.

[0120] In some embodiments, with continued reference to FIG. 2, a computing device, processor, and / or module may downsample elements of the training examples to a desired smaller number of data elements. As a non-limiting example, an image with a high pixel count may have 256 pixels, but a desired number of pixels may be 128 pixels. The processor may downsample the image with a high pixel count to convert the 256 pixels to 128 pixels. In some embodiments, the processor may be configured to perform downsampling on the data. Downsampling, also known as decimation, may include removing every Nth entry, all but every Nth entry, etc. in a sequence of samples, a process known as "compression," which may be performed, for example, by an N-sample compressor implemented using hardware or software. Anti-aliasing and / or anti-imaging filters, and / or low-pass filters may be used to clean up compression side effects.

[0121] Still referring to FIG. 2, the machine learning module 200 may be configured to execute a lazy learning process 220 and / or protocol. This may alternatively be referred to as a "lazy loading" or "call-when-needed" process and / or protocol, in which machine learning is performed by combining the input with a training set upon receipt of the input to be converted into an output to derive an algorithm that is used to generate the output on demand. For example, an initial set of simulations may be run to cover initial heuristics and / or "first guesses" at the outputs and / or relationships. As a non-limiting example, the initial heuristics may include ranking associations between the inputs and elements of the training data 204. The heuristics may include selecting a few of the highest ranked associations and / or elements of the training data 204. Lazy learning may implement any suitable lazy learning algorithm, including but not limited to a K-nearest neighbor algorithm, a lazy Naive Bayes algorithm, etc., and one of skill in the art, upon review of this disclosure in its entirety, will recognize a variety of lazy learning algorithms that may be applied to generate outputs as described in this disclosure, including but not limited to lazy learning applications of machine learning algorithms, as described in further detail below.

[0122] Alternatively or additionally, and still referring to FIG. 2, a machine learning process as described in this disclosure may be used to generate the machine learning model 224. A "machine learning model" as used in this disclosure is a data structure that represents and / or instantiates a mathematical and / or algorithmic representation of a relationship between inputs and outputs, generated using any machine learning process, including but not limited to any process as described above, and stored in memory, where the inputs, once created, are submitted to the machine learning model 224, which generates an output based on the derived relationship. For example, but not by way of limitation, a linear regression model generated using a linear regression algorithm may use coefficients derived during the machine learning process to calculate a linear combination of the input data and calculate the output data. As a further non-limiting example, the machine learning model 224 may be generated by creating an artificial neural network, such as a convolutional neural network, that includes an input layer of nodes, one or more hidden layers, and an output layer of nodes. The connections between nodes may be created through a process of "training" the network, in which elements from a set of training data 204 are applied to the input nodes, and an appropriate training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithm) is then used to adjust the connections and weights between nodes in adjacent layers of the neural network to produce desired values ​​at the output nodes. This process is sometimes referred to as deep learning.

[0123] Still referring to FIG. 2, the machine learning algorithm may include at least one supervised machine learning process 228. At least one supervised machine learning process 228, as defined herein, includes an algorithm that receives a training set that relates a number of inputs to a number of outputs and attempts to generate one or more data structures that represent and / or instantiate one or more mathematical relationships that relate the inputs to the outputs, each of which is optimal according to some criteria specified to the algorithm using some scoring function. For example, the supervised learning algorithm may include an image of a region of interest as described above as an input, a determination of whether a sample is present or not as an output, and a scoring function that represents the desired form of the relationship to be found between the input and the output. The scoring function may, for example, attempt to maximize the probability that a given input and / or combination of elements of the inputs is associated with a given output and minimize the probability that a given input is not associated with a given output. The scoring function may be expressed as a risk function that represents the "expected loss" of the algorithm relating inputs to outputs, where the loss is calculated as an error function that represents the degree to which a prediction produced by a relationship is inaccurate when compared to a given input-output pair provided in the training data 204. Those skilled in the art will recognize, upon reviewing this disclosure in its entirety, various possible variations of at least one supervised machine learning process 228 that may be used to determine a relationship between an input and an output. A supervised machine learning process may include a classification algorithm as defined above.

[0124] With further reference to FIG. 2, training of a supervised machine learning process may include iteratively updating coefficients, biases, weights based on, but not limited to, an error function, an expected loss, and / or a risk function. For example, outputs generated by a supervised machine learning model using example inputs of training examples may be compared to example outputs from the training examples, and an error function may be generated based on the comparison, which may include any error function suitable for use with any machine learning algorithm described in this disclosure, including the squared difference between one or more sets of compared values, and the like. Such error functions may in turn be used to update one or more weights, biases, coefficients, or other parameters of the machine learning model through any suitable process, including but not limited to a gradient descent process, a least squares process, and / or other processes described in this disclosure. This may be done iteratively and / or recursively to gradually adjust the weights, biases, coefficients, or other parameters. The updates may be performed using one or more backpropagation algorithms in the neural network. Iterative and / or recursive updates to weights, biases, coefficients, or other parameters as described above may be performed until the currently available training data is exhausted and / or a convergence test is passed, where a "convergence test" is a test on a condition selected as indicating that the model and / or its weights, biases, coefficients, or other parameters have reached a certain degree of accuracy. The convergence test may, for example, compare the difference between two or more successive error or error function values, where a difference below a threshold may be considered to indicate convergence. Alternatively or additionally, one or more error and / or error function values ​​evaluated in a training iteration may be compared to a threshold.

[0125] Still referring to FIG. 2, a computing device, processor, and / or module may be configured to repeatedly execute the methods, method steps, sequences of method steps, and / or algorithms described with reference to this figure in any order and to any degree. For example, a computing device, processor, and / or module may be configured to repeatedly execute a single step, sequence, and / or algorithm until a desired or commanded result is achieved. The repetition of a step or sequence of steps may be performed iteratively and / or recursively using the output of a previous iteration as input for a subsequent iteration, aggregation of the inputs and / or outputs of an iteration to generate an aggregate result, reduction or decrement of one or more variables, such as global variables, and / or division of a large processing task into a set of smaller processing tasks that are addressed iteratively. A computing device, processor, and / or module may execute any step, sequence of steps, or algorithm in parallel, such as performing a step two or more times simultaneously and / or substantially simultaneously using two or more parallel threads, processor cores, etc., and the division of tasks among parallel threads and / or processes may be performed according to any protocol suitable for dividing tasks among iterations. Those skilled in the art will recognize upon reviewing this disclosure in its entirety the various ways in which steps, sequences of steps, processing tasks, and / or data may be subdivided, shared, or otherwise processed using iterative, recursive, and / or parallel processing.

[0126] 2, the machine learning process can include at least one unsupervised machine learning process 232. An unsupervised machine learning process, as used herein, is a process that derives inferences in a dataset without regard to labels, such that the unsupervised machine learning process is free to discover any structure, relationships, and / or correlations provided in the data. The unsupervised machine learning process 232 may not require a response variable, and the unsupervised machine learning process 232 can be used to find interesting patterns and / or inferences between variables, to determine the degree of correlation between two or more variables, etc.

[0127] Still referring to FIG. 2, the machine learning module 200 may be designed and configured to create the machine learning model 224 using techniques for the development of linear regression models. The linear regression model may include ordinary least squares regression, which aims to minimize the squared difference between the predicted outcome and the actual outcome according to an appropriate norm (e.g., a vector space distance norm) that measures such difference. The coefficients of the resulting linear equation may be modified to improve the minimization. The linear regression model may include a ridge regression method, where the function to be minimized may include, in addition to the least squares function, a term in which the square of each coefficient is multiplied by a scalar amount to penalize large coefficients. The linear regression model may include a least absolute shrinkage selection operator (lasso) model, where ridge regression is combined with multiplying the least squares term by a factor of 1 divided by twice the number of samples. The linear regression model may include a multitasking lasso model, where the norm applied to the least squares term of the lasso model is the Frobenius norm, which corresponds to the square root of the sum of the squares of all the terms. The linear regression model may include an elastic net model, a multitask elastic net model, a least-angle regression model, a LARS lasso model, an orthogonal matching pursuit model, a Bayesian regression model, a logistic regression model, a stochastic gradient descent model, a perceptron model, a passive-aggressive algorithm, a robustness regression model, a Huber regression model, or other suitable models that may occur to one of ordinary skill in the art upon reviewing this disclosure in its entirety. The linear regression model may be generalized in embodiments to a polynomial regression model, whereby a polynomial (e.g., quadratic, cubic, or higher order) that provides the best predicted output / actual output fit is found. As will be apparent to one of ordinary skill in the art upon reviewing this disclosure in its entirety, methods similar to those described above may be applied to minimize the error function.

[0128] Still referring to FIG. 2 , the machine learning algorithm may include, but is not limited to, linear discriminant analysis. The machine learning algorithm may include quadratic discriminant analysis. The machine learning algorithm may include kernel ridge regression. The machine learning algorithm may include support vector machines, including but not limited to regression processes based on support vector classification. The machine learning algorithm may include stochastic gradient descent algorithms, including classification and regression algorithms based on stochastic gradient descent. The machine learning algorithm may include nearest neighbor algorithms. The machine learning algorithm may include various forms of latent space regularization, such as variational regularization. The machine learning algorithm may include Gaussian processes, such as Gaussian process regression. The machine learning algorithm may include cross decomposition algorithms, including partial least squares and / or canonical correlation analysis. The machine learning algorithm may include naive Bayes methods. The machine learning algorithm may include decision tree-based algorithms, such as decision tree classification and regression algorithms. The machine learning algorithm may include ensemble methods, such as bagged meta-estimators, forests of random trees, AdaBoost, gradient tree boosting, and / or voting classifier methods. The machine learning algorithms may include neural net algorithms, including convolutional neural net processing.

[0129] Still referring to FIG. 2, the machine learning model and / or process may be deployed or instantiated by incorporation into a program, device, system and / or module. For example, but not limited to, the machine learning model, neural network, and / or some or all of its parameters may be stored and / or deployed in any memory or circuit. Parameters such as coefficients, weights, and / or biases may be stored as circuit-based constants such as an array of wires and / or binary inputs and / or outputs set to logic "1" and "0" voltage levels in a logic circuit to represent numbers according to any suitable encoding system, including binary complements, or may be stored in any volatile and / or non-volatile memory. Similarly, mathematical operations and data inputs and / or outputs to or from the model, neural network layers, etc. may be instantiated in hardware circuits and / or in the form of firmware, machine code such as binary opcode instructions, assembly language, or instructions in any higher-level programming language. To instantiate a machine learning process and / or model, any technique for hardware and / or software instantiation of memory, instructions, data structures, and / or algorithms may be used, which may include, but is not limited to, the manufacture and / or configuration of non-reconfigurable hardware elements, circuits, and / or modules, such as, but not limited to, ASICs; the manufacture and / or configuration of reconfigurable hardware elements, circuits, and / or modules, such as, but not limited to, FPGAs; the manufacture and / or configuration of non-reconfigurable memory elements, circuits, and / or modules, such as, but not limited to, non-reconfigurable ROMs; the manufacture and / or configuration of reconfigurable and / or reconfigurable memory elements, circuits, and / or modules, such as, but not limited to, reconfigurable ROMs or other memory technologies described in this disclosure; and / or the manufacture and / or configuration of any computing device and / or components thereof, as described in this disclosure, in any combination thereof.Such deployed and / or instantiated machine learning models and / or algorithms may receive inputs from, and generate outputs for, any other processes, modules, and / or components described in this disclosure.

[0130] 2, any process of training, retraining, deployment, and / or instantiation of a machine learning model and / or algorithm may be performed and / or repeated after initial deployment and / or instantiation to modify, refine, and / or improve the machine learning model and / or algorithm. Such retraining, deployment, and / or instantiation may be performed as a periodic or periodic process, for example, retraining, deployment, and / or instantiation at periodic elapsed time periods, after a measure of volume such as the number of bytes or other measures of data processed, the number of uses or executions of the processes described in this disclosure, and / or according to a software, firmware, or other update schedule. Alternatively or additionally, retraining, deployment, and / or instantiation may be event-based and may be triggered, without limitation, by user input indicating suboptimal or otherwise problematic performance, and / or by an automated field testing and / or audit process that may compare the output of the machine learning model and / or algorithm, and / or error and / or error function thereof, to any threshold value, convergence determination, etc., and / or compare the output of the processes described herein to similar threshold values, convergence determinations, etc. Event-based retraining, deployment, and / or instantiation may alternatively or additionally be triggered by the receipt and / or generation of one or more new training examples, where the number of new training examples may be compared to a pre-configured threshold, where exceeding the pre-configured threshold may trigger retraining, deployment, and / or instantiation.

[0131] Still referring to FIG. 2, the retraining and / or additional training may be performed using any of the processes for training described above, using any version of a current or previously deployed machine learning model and / or algorithm as a starting point. Training data for retraining may be collected, pre-processed, sorted, classified, sanitized, or otherwise processed according to any process described in this disclosure. Training data may include, without limitation, training examples including inputs and associated outputs used, received, and / or generated from any version of any system, module, machine learning model or algorithm, apparatus, and / or method described in this disclosure, such examples may be modified and / or labeled according to user feedback or other processing to indicate desired results, and / or may have actual or measured results from a process being modeled and / or predicted by the system, module, machine learning model or algorithm, apparatus, and / or method as a "desired" result to be compared to the output for the training process as described above.

[0132] The redeployment may be performed using any reconfiguration and / or rewriting of reconfigurable and / or rewriteable circuitry and / or memory elements; alternatively, the redeployment may be performed by the fabrication of new hardware and / or software components, circuits, instructions, etc., which may be added to and / or replace existing hardware and / or software components, circuits, instructions, etc.

[0133] With further reference to FIG. 2, one or more of the processes or algorithms described above may be performed by at least one dedicated hardware unit 236. For purposes of this figure, a "dedicated hardware unit" is a hardware component, circuitry, etc., other than the main control circuitry and / or processor that performs the method steps described in this disclosure, that is specifically designated or selected to perform one or more particular tasks and / or processes described with reference to this figure, such as, but not limited to, preconditioning and / or sanitizing training data and / or training machine learning algorithms and / or models. The dedicated hardware unit 236 may include hardware units that can efficiently perform repetitive or intensive calculations, such as, but not limited to, matrix-based calculations that update or adjust parameters, weights, coefficients, and / or biases of machine learning models and / or neural networks using pipelined processing, parallel processing, etc., and such hardware units may be optimized for such processing, for example, by including dedicated circuitry for matrix operations and / or signal processing operations including multiple arithmetic and / or logic circuit units, such as multipliers and / or adders, capable of operating simultaneously and / or in parallel, etc. Such dedicated hardware units 236 may include, but are not limited to, graphic processing units (GPUs), dedicated signal processing modules, FPGAs, or other reconfigurable hardware configured to instantiate parallel processing units for one or more specific tasks. A computing device, processor, apparatus, or module may be configured to instruct one or more dedicated hardware units 236 to perform one or more operations described herein, such as evaluation of model and / or algorithmic outputs, one-time or iterative updates of parameters, coefficients, weights, and / or biases, and / or any other operations, such as vector and / or matrix operations, described in this disclosure.

[0134] Referring now to FIG. 3, an exemplary embodiment of a neural network 300 is illustrated. A neural network 300, also known as an artificial neural network, is a network of "nodes" or data structures having one or more inputs, one or more outputs, and a function that determines the output based on the inputs. Such nodes may be organized into a network, such as, but not limited to, a convolutional neural network, including an input layer of nodes 304, one or more hidden layers 308, and an output layer of nodes 312. The connections between the nodes may be created through a process of "training" the network, in which elements from a set of training data are applied to the input nodes, and a suitable training algorithm (such as Levenberg-Marquardt, conjugate gradient, simulated annealing, or other algorithms) is then used to adjust the connections and weights between nodes in adjacent layers of the neural network to generate desired values ​​at the output nodes. This process is sometimes referred to as deep learning. Connections may only run from input nodes to output nodes in a "feedforward" network, or the output of one layer may be fed back to the input of the same or a different layer in a "recurrent network." As a further non-limiting example, a neural network can include a convolutional neural network that includes an input layer of nodes, one or more hidden layers, and an output layer of nodes. A "convolutional neural network," as used in this disclosure, is a neural network in which at least one hidden layer is a convolutional layer that convolves the input to that layer with a subset of the input known as a "kernel," along with one or more additional layers, such as a pooling layer, a fully connected layer, etc.

[0135] 4, an exemplary embodiment of a neural network node 400 is illustrated. The node has multiple inputs x that can receive values ​​from, but are not limited to, inputs to the neural network that contains the node and / or from other nodes. iA node may implement one or more activation functions to generate its output given one or more inputs. Activation functions include, but are not limited to, a binary step function that compares an input to a threshold and outputs a logic 1 or logic 0 output or the equivalent, a linear activation function where the output is directly proportional to the input, and / or a non-linear activation function where the output is not proportional to the input. A non-linear activation function includes, but is not limited to, a binary step function that compares an input x to a threshold and outputs a logic 1 or logic 0 output or the equivalent, a linear activation function where the output is directly proportional to the input, and / or a non-linear activation function where the output is not proportional to the input.

number

number

number

number

number

number

[0136] Still referring to Figure 4, a "convolutional neural network" as used in this disclosure is a neural network in which at least one hidden layer is a convolutional layer that convolves the input to that layer with a subset of the input known as a "kernel," along with one or more additional layers, such as a pooling layer, a fully connected layer, etc. CNNs can include, but are not limited to, extensions of deep neural networks (DNNs), which are defined as neural networks with two or more hidden layers.

[0137] Still referring to FIG. 4, in some embodiments, a convolutional neural network can learn from images. In a non-limiting example, the convolutional neural network may perform tasks such as classifying images, detecting objects depicted in images, segmenting images, and / or processing images. In some embodiments, the convolutional neural network may operate such that each node in the input layer is only connected to regions of nodes in the hidden layer. In some embodiments, the regions may collectively create a map of features from the input layer to the hidden layer. In some embodiments, the convolutional neural network may include layers in which the weights and biases of all nodes are identical. In some embodiments, this allows the convolutional neural network to detect features, such as edges, at different locations in the image.

[0138] Referring now to FIG. 5, an exemplary embodiment of a method 500 of slide imaging is illustrated. This may help reduce errors due to out-of-focus. In step 505, an entire slide image may be captured at a low resolution (e.g., 1x). In step 510, image segmentation may be performed to detect regions where material is present. This may delimit all regions on the slide, including dust regions, pen marks, printed text (e.g., annotations), with bounding boxes. In step 515, for each bounding box encompassing the region of interest, a sample presence probability score may be calculated. This may be performed by various algorithms such as K-means, NLMD (Non-Local Means Denoising), features such as hue color space of each row, segmentation models such as U-Net, etc. This may be used to find the best row 520. The best row may include rows sandwiched between the rows above and below, and the sum of the weighted scores of the row is the highest score for the presence of the sample. Once the best row is determined, the best (x,y) location may be determined. The best (x,y) location may include the (x,y) point that maximizes the probability of the sample being present at that point, based on specimen boundaries, color, etc. The best focus of the bounding box may be determined by collecting a Z-stack at the best (x,y) point 525. A "Z-stack" as used herein is a number of images at a particular (x,y) location, with varying focal lengths. In some embodiments, the Z-stack may have a height less than the height of the slide. In some embodiments, the Z-stack may include 2, 3, 4, 5, 6, 7, 8, 9, 10, or more images captured at various focal lengths. In some embodiments, the images captured as part of the Z-stack may be spaced less than a millimeter apart. For example, capturing fewer images may increase efficiency, due to increased speed. Capturing images closer together may allow the best focal length to be determined with greater precision. In some embodiments, these factors may make it desirable to capture a relatively small number of images over a relatively small height.This may increase the importance of selecting a focal length for capturing the Z-stack such that the object to be focused is located within the height of the Z-stack. The focal pattern may be used to estimate the focal length at the (x,y) location where the Z-stack is captured. Using such an estimate may improve the certainty that the object to be focused is within the height of the Z-stack. In some embodiments, the distance of the Z-stack may be identified and / or captured using the planarity of the camera field of view and / or at least two points along its row. Such planarity may include a focal pattern as described herein. In steps 530 and 535, the Z-level may be used to scan the row and identify a focal length pattern, such as a plane. A focal length pattern, such as a plane, may be identified using a set of points along the row as a function of the best focus of the points in that row. Such a plane may be used to estimate the focal length at other locations, such as the location of an adjacent row. Such a plane may be recalculated based on additional data as new data is obtained. For example, the best focus may be identified at an additional point and the plane updated based on this additional data. Such a plane may also be used to estimate the focal length in another region of interest 540. This procedure may be repeated for all regions of interest on slide 545 .

[0139] 6A-6C, the progression of a slide through various steps described herein is illustrated. Slide 604 may include annotations 608A, 608B, dust 612A, 612B, and samples 616A, 616B. The steps described herein address the challenge of selecting the correct focus for areas where samples are present despite the presence of dust and annotations. By dividing slide 604 into separate regions of interest 620A-620F, each segment can be scanned at a different focus that is optimal for that region. This makes the focus decision independent of the spatial distribution of samples, dust, and annotations. In some embodiments, image classification to detect dust and annotations to avoid scanning is best done downstream, not during scanning. In some embodiments, this solves the challenge of running advanced models live on the scanning device during scanning, and the risk of false positives when classifying areas as dust or annotations and skipping scanning the associated areas. Additionally, annotations that may be useful for downstream tasks may need to be scanned for use in downstream multimodal learning. In some embodiments, segmentation followed by a best focus determination can be a useful approach to find all regions of interest at the optimal focus obtained with the best row estimate.

[0140] 7, an exemplary embodiment of a method 700 of slide imaging is illustrated. One or more steps of method 700 may be implemented as, but not limited to, described herein with reference to other figures. One or more steps of method 700 may be implemented using, but not limited to, at least one processor.

[0141] Still referring to FIG. 7, in some embodiments, the method 700 may include receiving at least one region of interest 705 .

[0142] Still referring to FIG. 7, in some embodiments, method 700 may include capturing a first image of the slide at a first position within at least one region of interest 710 using at least one optical system.

[0143] 7, in some embodiments, method 700 may include identifying a focus pattern as a function of the first image and the first location 715. In some embodiments, identifying the focus pattern includes identifying a row that includes the first location, capturing a plurality of first images at the first location, each of the plurality of first images having a different focal length, determining an optimally focused first image among the optimally focused plurality of first images, and identifying the focus pattern using the focal lengths of the plurality of optimally focused images at a set of points along the row. In some embodiments, the row further includes a second location, and the method further includes capturing a plurality of second images at the second location using at least one processor and at least one optical system, each of the plurality of second images having a different focal length, determining an optimally focused second image of the plurality of best-focused second images using at least one processor, identifying a focus pattern using the focal lengths of the optimally focused first image and the optimally focused second image using at least one processor, and extrapolating a third focal length of the third location as a function of the focus pattern using at least one processor. In some embodiments, the third location is located outside the row. In some embodiments, the third location is located within a different region of interest than the first location. In some embodiments, identifying the row includes identifying the row based on a first row sample presence score from a first set of row sample presence scores. In some embodiments, identifying the row based on the first row sample presence score includes determining a row from a second set of sample presence scores whose adjacent rows have the highest sample presence scores, the second set of sample presence scores being determined using machine vision. In some embodiments, identifying the plane includes identifying a plurality of points and a plurality of best focuses at the plurality of points, and generating the plane as a function of a subset of the plurality of points and a corresponding subset of the best focuses at the points.In some embodiments, identifying the focus pattern further includes updating the focus pattern, and updating the focus pattern includes identifying the additional point and an optimal focus at the additional point, and updating the focus pattern as a function of the additional point and the optimal focus at the additional point.

[0144] Still referring to FIG. 7, in some embodiments, the method 700 may include extrapolating the focal length of the second position as a function of the focal pattern 720.

[0145] 7, in some embodiments, method 700 may include capturing a second image of the slide using the at least one optical system at a second position and focal length 725. In some embodiments, capturing the second image includes capturing a plurality of images taken at focal lengths based on the focal pattern and constructing the second image from the plurality of images.

[0146] Still referring to FIG. 7, in some embodiments, the method 700 may further include moving the movable element to the second position using an actuator mechanism.

[0147] Still referring to FIG. 7, in some embodiments, the method 700 may further include using machine vision to identify the point in the row having the maximum point sample presence score.

[0148] Still referring to FIG. 7, in some embodiments, method 700 may further include capturing a low magnification image of the slide using at least one processor and optical system, the low magnification image having a lower magnification than the magnification of the first image, identifying at least one region of interest in the low magnification image using at least one processor and machine vision, and determining, using the at least one processor, whether the sample is included in either the first image or the second image.

[0149] It should be noted that any one or more of the aspects and embodiments described herein may be suitably implemented using one or more machines (e.g., one or more computing devices utilized as user computing devices for electronic documents, one or more server devices, such as document servers) programmed in accordance with the teachings herein, as would be apparent to one of ordinary skill in the computer arts. Appropriate software coding may be readily produced by skilled programmers based on the teachings of the present disclosure, as would be apparent to one of ordinary skill in the software arts. The above-described aspects and implementations employing software and / or software modules may also include appropriate hardware to assist in the implementation of the machine-executable instructions of the software and / or software modules.

[0150] Such software may be a computer program product employing a machine-readable storage medium. A machine-readable storage medium may be any medium capable of storing and / or encoding a sequence of instructions for execution by a machine (e.g., a computing device) and causing the machine to perform any one of the methodologies and / or embodiments described herein. Examples of machine-readable storage media include, but are not limited to, magnetic disks, optical disks (e.g., CDs, CD-Rs, DVDs, DVD-Rs, etc.), magneto-optical disks, read-only memory "ROM" devices, random access memory "RAM" devices, magnetic cards, optical cards, solid-state memory devices, EPROMs, EEPROMs, and any combination thereof. A machine-readable medium as used herein is intended to include a single medium as well as a collection of physically separate media, such as, for example, a collection of compact discs, one or more hard disk drives in combination with a computer memory, and the like. A machine-readable storage medium as used herein does not include a transitory form of signal transmission.

[0151] Such software may also include information (e.g., data) carried as a data signal on a data carrier, such as a carrier wave. For example, the machine-executable information may be included as a data carrier signal embodied in a data carrier, the signal encoding a sequence of instructions, or portions thereof, for execution by a machine (e.g., a computing device), and any associated information (e.g., data structures and data) that causes the machine to perform any one of the methodologies and / or embodiments described herein.

[0152] Examples of computing devices include, but are not limited to, e-book reading devices, computer workstations, terminal computers, server computers, handheld devices (e.g., tablet computers, smartphones, etc.), web appliances, network routers, network switches, network bridges, any machine capable of executing a sequence of instructions that specify actions to be performed by that machine, and any combination thereof. In one example, a computing device may include and / or be included in a kiosk.

[0153] 8 shows a diagrammatic representation of one embodiment of a computing device in the exemplary form of a computer system 800 upon which a set of instructions may be executed that causes a control system to perform any one or more of the aspects and / or methodologies of the present disclosure. It is also contemplated that multiple computing devices may be utilized to execute a set of instructions specifically configured to cause one or more of the devices to perform any one or more of the aspects and / or methodologies of the present disclosure. The computer system 800 includes a processor 804 and a memory 808 that communicate with each other and with other components via a bus 812. The bus 812 may include any of several types of bus structures, including, but not limited to, a memory bus, a memory controller, a peripheral bus, a local bus, and any combination thereof, using any of a variety of bus architectures.

[0154] The processor 804 may include any suitable processor, such as, but not limited to, a processor incorporating logic circuitry for performing arithmetic and logical operations, such as an arithmetic logic unit (ALU), which may be controlled by a state machine and directed by operational input from memory and / or sensors. The processor 804 may be configured according to, as non-limiting examples, the Von Neumann architecture and / or the Harvard architecture. The processor 804 may include, incorporate, and / or be incorporated in, but is not limited to, a microcontroller, a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), a complex programmable logic device (CPLD), a graphical processing unit (GPU), a general purpose GPU, a tensor processing unit (TPU), an analog or mixed signal processor, a trusted platform module (TPM), a floating point unit (FPU), and / or a system on a chip (SoC).

[0155] Memory 808 may include a variety of components (e.g., machine-readable media), including but not limited to random access memory components, read-only components, and any combination thereof. In one example, a basic input / output system 816 (BIOS), including basic routines that help to transfer information between elements within computer system 800, such as during start-up, may be stored in memory 808. Memory 808 may also include (e.g., stored on one or more machine-readable media) instructions (e.g., software) 820 that embody any one or more of the aspects and / or methodologies of the present disclosure. In another example, memory 808 may further include any number of program modules, including but not limited to an operating system, one or more application programs, other program modules, program data, and any combination thereof.

[0156] Computer system 800 may also include a storage device 824. Examples of storage devices (e.g., storage device 824) include, but are not limited to, hard disk drives, magnetic disk drives, optical disk drives in combination with optical media, solid-state memory devices, and any combination thereof. Storage device 824 may be connected to bus 812 by an appropriate interface (not shown). Examples of interfaces include, but are not limited to, SCSI, Advanced Technology Attachment (ATA), Serial ATA, Universal Serial Bus (USB), IEEE 1394 (FIREWIRE®), and any combination thereof. In one example, storage device 824 (or one or more components thereof) may be removably interfaced with computer system 800 (e.g., via an external port connector (not shown)). In particular, storage device 824 and associated machine-readable media 828 may provide non-volatile and / or volatile storage of machine-readable instructions, data structures, program modules, and / or other data for computer system 800. In one example, the software 820 may reside completely or partially within the machine-readable medium 828. In another example, the software 820 may reside completely or partially within the processor 804.

[0157] Computer system 800 may also include input devices 832. In one example, a user of computer system 800 may input commands and / or other information to computer system 800 via input devices 832. Examples of input devices 832 include, but are not limited to, alphanumeric input devices (e.g., keyboards), pointing devices, joysticks, gamepads, audio input devices (e.g., microphones, voice response systems, etc.), cursor control devices (e.g., mice), touchpads, optical scanners, video capture devices (e.g., still cameras, video cameras), touch screens, and any combination thereof. Input devices 832 may be interfaced to bus 812 via any of a variety of interfaces (not shown), including, but not limited to, a serial interface, a parallel interface, a game port, a USB interface, a FIREWIRE interface, a direct interface to bus 812, and any combination thereof. Input devices 832 may include a touch screen interface, which may be part of or separate from display 836, as described below. Input devices 832 may be utilized as a user selection device to select one or more graphical representations in a graphical interface, as described above.

[0158] A user may also input commands and / or other information to computer system 800 via storage device 824 (e.g., removable disk drive, flash drive, etc.) and / or network interface device 840. A network interface device, such as network interface device 840, may be utilized to connect computer system 800 to one or more of a variety of networks, such as network 844, and one or more remote devices 848 connected thereto. Examples of network interface devices include, but are not limited to, a network interface card (e.g., a mobile network interface card, a LAN card), a modem, and any combination thereof. Examples of networks include, but are not limited to, a wide area network (e.g., the Internet, an enterprise network), a local area network (e.g., a network associated with an office, building, campus, or other relatively small geographic space), a telephone network, a data network associated with a telephone / voice provider (e.g., a mobile communications provider's data and / or voice network), a direct connection between two computing devices, and any combination thereof. A network, such as network 844, may employ wired and / or wireless modes of communication. In general, any network topology may be used. Information (eg, data, software 820 , etc.) may be communicated to and / or from computer system 800 via network interface device(s) 840 .

[0159] The computer system 800 may further include a video display adapter 852 that communicates displayable images to a display device, such as a display device 836. Examples of display devices include, but are not limited to, a liquid crystal display (LCD), a cathode ray tube (CRT), a plasma display, a light emitting diode (LED) display, and any combination thereof. The display adapter 852 and the display device 836 may be utilized in combination with the processor 804 to provide a graphical representation of aspects of the present disclosure. In addition to a display device, the computer system 800 may include one or more other peripheral output devices, including, but not limited to, audio speakers, a printer, and any combination thereof. Such peripheral output devices may be connected to the bus 812 via a peripheral interface 856. Examples of peripheral interfaces include, but are not limited to, a serial port, a USB connection, a FIREWIRE connection, a parallel connection, and any combination thereof.

[0160] The above is a detailed description of exemplary embodiments of the present invention. Various modifications and additions can be made without departing from the spirit and scope of the present invention. Features of each of the various embodiments described above can be combined with features of other described embodiments as appropriate to provide various combinations of features in related new embodiments. Moreover, while a number of separate embodiments have been described above, what has been described herein is merely illustrative of the application of the principles of the present invention. Furthermore, although certain methods herein may be illustrated and / or described as being performed in a particular order, the order can be varied considerably within the ordinary skill of the art to implement the methods, systems, and software according to the present disclosure. Accordingly, the present specification is intended to be interpreted as illustrative only, and not as otherwise limiting the scope of the present invention.

[0161] Exemplary embodiments have been disclosed above and illustrated in the accompanying drawings. It will be understood by those skilled in the art that various modifications, omissions, and additions may be made to what is specifically disclosed herein without departing from the spirit and scope of the invention.

Claims

1. 1. An apparatus for imaging a slide, comprising: At least one optical system including an optical sensor; a slide port configured to hold the slide; At least one processor; a memory communicatively coupled to the at least one processor, the at least one processor comprising: Receiving at least one region of interest; capturing a plurality of first images of the slide at a first location within the at least one region of interest using the at least one optical system, each of the plurality of first images having a different focal length; and identifying a focus pattern, the identifying the focus pattern comprising: identifying rows of the plurality of first images; determining an optimally focused first image from among the plurality of best-focused first images; identifying the focus pattern using focal lengths of each of a plurality of optimally focused images at a set of points along the row; Including, extrapolating a focal length of a second position as a function of the focal pattern; capturing a second image of the slide at the second position and at the focal length using the at least one optical system. and a memory storing instructions for configuring the An apparatus comprising:

2. 10. The apparatus of claim 1, further comprising an actuator mechanism mechanically connected to the at least one optical system, the actuator mechanism configured to move the at least one optical system to the second position.

3. The line further includes a second location, and the instructions cause the processor to: using the optical system to capture a plurality of second images at the second location, each of the plurality of second images having a different focal length; determining an optimally focused second image from the plurality of best-focused second images; identifying the focus pattern using the focal lengths of the first optimally focused image and the second optimally focused image; Extrapolating a third focal length at a third position as a function of the focal pattern. The apparatus of claim 1 further configured to:

4. The apparatus of claim 3 , wherein the third location is located outside the row.

5. The apparatus of claim 3 , wherein the third location is located in a different region of interest than the first location.

6. The apparatus of claim 1 , wherein identifying the row comprises identifying the row based on a first row sample presence score from a first set of row sample presence scores.

7. 7. The apparatus of claim 6, wherein identifying the row based on the first row sample presence score includes determining a row from a second set of sample presence scores whose adjacent rows have the highest sample presence scores, the second set of sample presence scores being determined using machine vision.

8. The apparatus of claim 1 , wherein the instructions further configure the processor to identify, using machine vision, a point in the row having a maximum point sample presence score.

9. identifying the focal pattern includes identifying a plane; Identifying a plurality of points and a plurality of best focuses at the plurality of points; generating a plane as a function of a subset of the plurality of points and a corresponding subset of best focus at those points; The apparatus of claim 1 , comprising:

10. Identifying the focus pattern further includes updating the focus pattern, where updating the focus pattern comprises: identifying an additional point and a best focus at said additional point; updating a focus pattern as a function of the additional point and best focus at the additional point; The apparatus of claim 9 , comprising:

11. Capturing the second image includes: capturing a plurality of images taken at focal lengths based on the focus pattern; constructing the second image from the plurality of images; and The apparatus of claim 1 , comprising:

12. The processor, capturing a low magnification image of the slide using the at least one optical system, the low magnification image having a magnification lower than a magnification of the first image; identifying the at least one region of interest in the low magnification image using machine vision; Determining whether the sample is included in either the first image or the second image The apparatus of claim 1 , further configured to:

13. 1. A method of imaging a slide, comprising: receiving, using at least one processor, at least one region of interest; capturing, using the at least one processor and at least one optical system, a plurality of first images of the slide at a first location within the at least one region of interest, each of the plurality of first images having a different focal length; and identifying a focus pattern using the at least one processor, the identifying the focus pattern comprising: identifying rows of the plurality of first images; determining an optimally focused first image from among the plurality of best-focused first images; identifying the focus pattern using focal lengths of each of a plurality of optimally focused images at a set of points along the row; and extrapolating, using the at least one processor, a focal length of a second position as a function of the focal pattern; capturing a second image of the slide at the second position and at the focal length using the at least one processor and the at least one optical system; The method includes:

14. The method of claim 13 , further comprising moving the optical system to the second position using an actuator mechanism.

15. the row further comprises a second location, and the method further comprises: capturing a plurality of second images at the second location using the at least one processor and the at least one optical system, each of the plurality of second images having a different focal length; determining, using the at least one processor, an optimally focused second image of the plurality of best-focus second images; identifying, using the at least one processor, the focal length of the first optimally focused image and the second optimally focused image, the focal pattern; extrapolating, using the at least one processor, a third focal length at a third position as a function of the focal pattern; The method of claim 13 further comprising:

16. The method of claim 15 , wherein the third location is located outside the row.

17. The method of claim 15 , wherein the third location is located in a different region of interest than the first location.

18. The method of claim 13 , wherein identifying the row comprises identifying the row based on a first row sample presence score from a first set of row sample presence scores.

19. 20. The method of claim 18, wherein identifying the row based on the first row sample presence score includes determining a row from a second set of sample presence scores whose adjacent rows have the highest sample presence scores, the second set of sample presence scores being determined using machine vision.

20. The method of claim 13 , further comprising identifying, using machine vision, the point in the row having the maximum point sample presence score.

21. identifying the focal pattern includes identifying a plane; Identifying a plurality of points and a plurality of best focuses at the plurality of points; generating a plane as a function of a subset of the plurality of points and a corresponding subset of best focus at those points; The method of claim 13, comprising:

22. Identifying the focus pattern further includes updating the focus pattern, where updating the focus pattern comprises: identifying an additional point and a best focus at said additional point; updating a focus pattern as a function of the additional point and best focus at the additional point; 22. The method of claim 21 , comprising:

23. Capturing the second image includes: capturing a plurality of images taken at focal lengths based on the focus pattern; constructing the second image from the plurality of images; and The method of claim 13, comprising:

24. capturing a low magnification image of the slide using at least one processor and the optical system, the low magnification image having a magnification lower than a magnification of the first image; identifying the at least one region of interest in the low magnification image using the at least one processor and machine vision; determining, using the at least one processor, whether the sample is included in either the first image or the second image; The method of claim 13 further comprising:

Images