Image Composition for Mixed Transducer Arrays.

Abstract

The imaging method may include receiving a first signal from one or more array elements of a first type in the mixed transducer array, receiving a second signal from one or more array elements of a second type in the mixed transducer array, where at least one of the first type and the second type is an optical sensor, generating a first image from the first signal and a second image from the second signal, and combining the first image and the second image to generate a composite image.

Description

[Technical Field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Patent Application No. 63 / 104,886, filed October 23, 2020, which is incorporated herein by reference in its entirety.

[0002]

[0002] The present disclosure relates generally to the field of imaging, and more particularly to methods and devices that enable the formation of composite images from images collected by mixed arrays including optical sensors and arrays of other transducers. The methods and devices disclosed herein include optical sensors with high sensitivity and / or high operating bandwidth for improved imaging performance. [Background technology]

[0003]

[0003] Ultrasound sensing is used in various industries, including medical imaging and medical diagnostics, due to several advantages. For example, ultrasound sensing utilizes ultrasonic signals, which have a significant penetration depth. Furthermore, ultrasound imaging is known to be advantageously a non-invasive form of imaging, since it is based on non-ionizing radiation. Summary of the Invention [Problem to be solved by the invention]

[0004]

[0004] Various known ultrasonic transducers used in ultrasound imaging have many drawbacks. For example, some ultrasonic transducers are made of piezoelectric materials such as lead zirconate titanate (PZT). However, the 6 dB bandwidth of PZT materials is generally limited to only about 70%. Some composite PZT materials have slightly increased bandwidth, but still only achieve a maximum bandwidth of about 80%. As another example, single-crystal materials are increasingly being used to improve the performance of ultrasound probes, but they have low Curie temperatures and are brittle. Another type of transducer material is silicon, which can be processed to construct capacitive micromachined ultrasonic transducer (CMUT) probes that can have increased bandwidth. However, CMUT probes are not very sensitive or reliable. Furthermore, CMUT probes have several operational limitations. For example, CMUT probes are nonlinear sensors and therefore generally not suitable for harmonic imaging. Therefore, there is a need for an ultrasound probe with a mixed transducer array (mixed array) that includes sensors with higher bandwidth and sensitivity. Additionally, a back-end and / or front-end device is needed to process the signals and / or images produced by the mixed array. [Means for solving the problem]

[0005] In general, in some variations, an apparatus (e.g., an image combination system) for imaging (e.g., ultrasound imaging of a patient) may include a combination transducer array including one or more array elements of a first type configured to receive a first signal and one or more array elements of a second type configured to receive a second signal, where at least one of the first and second types is an optical sensor. The apparatus may further include one or more processors configured to generate a first image from the first signal, generate a second image from the second signal, and combine the first and second images to generate a combination image.

[0006] In some variations, the first type of array element may include a non-optical transducer, and the second type of array element may include an optical sensor. The one or more array elements of the first type may include, for example, a piezoelectric transducer, a single crystal material transducer, a piezoelectric micromachined ultrasonic transducer (PMUT), or a capacitive micromachined ultrasonic transducer (CMUT). The optical sensor may include, for example, a whispering gallery mode (WGM) optical resonator, a microbubble optical resonator, a photonic integrated circuit (PIC) optical resonator, a microsphere resonator, a microtoroid resonator, a microring resonator, a microbottle resonator, a microcylinder resonator, and / or a microdisk optical resonator.

[0007] In some variations, the second type of array elements may include optical sensors having different characteristics (e.g., different designs and / or different operating parameters). For example, in some variations, the second type of array elements may include one or more high quality factor (high Q) optical sensors and one or more low quality (low Q) optical sensors. Additionally or alternatively, the second type of array elements may include one or more tunable optical resonators configured to operate as high Q optical resonators, and / or the second type of array elements may include one or more tunable optical resonators configured to operate as low Q optical resonators. For example, such tunable optical resonators may be selectively operable in a high Q mode or a low Q mode, depending on the imaging setting, etc.

[0008] Furthermore, in some variations, the mixed transducer array may include a combination of one or more non-optical transducers and multiple types of optical sensors. For example, the mixed transducer array may include one or more array elements of a first type that include at least one non-optical transducer, one or more array elements of a second type that include at least one type of optical sensor, and one or more array elements of a third type that include at least another type of optical sensor. The one or more processors may be further configured to generate a third image from the third signal and combine the first, second, and third images to generate a composite image. The different types of optical resonator sensors may include, for example, high-Q and low-Q optical resonators (or tunable optical resonator sensors configured to operate as high-Q or low-Q optical resonators). As another example, the different types of optical resonator sensors may include wide-bandwidth and ultra-high-sensitivity optical resonators.

[0009] In some variations, one or more array elements (e.g., transducers) of the mixed transducer array may transmit an acoustic signal at a fundamental frequency f. In response, one or more array elements of the first type, the second type, or both the first and second types may produce one or more responses upon receiving harmonic acoustic echoes (including supra- and sub-harmonics) corresponding to the transmitted acoustic signal. One or more array elements of the second type may have a bandwidth ranging from at least f / M to Nf, where M and N are integers greater than 1. In some variations, one or more array elements of the first type may transmit acoustic signals at a first fundamental frequency f1 and a second fundamental frequency f2. In response, one or more array elements of the second type may produce one or more optical responses upon receiving acoustic echoes corresponding to frequencies of one or more linear combinations nf1+mf2, where n and m are integers such that nf1+mf2 is positive. At least one of the first image and the second image may be or include a harmonic image.

[0010] In some variations, the one or more processors may be configured to filter various signals from different types of array elements in the mixed transducer array using one or more suitable filters. Such suitable filters may include, for example, harmonic bandpass filters that may enable extraction of harmonic signals, including subharmonic and ultraharmonic signals.

[0011] Combining the first and second images may be performed by a suitable combining algorithm. For example, the one or more processors may be configured to combine the first and second images at least in part by determining an average of the first and second images. For example, the one or more processors may be configured to combine the first and second images at least in part by determining an arithmetic or geometric mean of the first and second images. Additionally or alternatively, the one or more processors may be configured to combine the first and second images at least in part by determining a weighted average of the first and second images. In some variations, such weighted averaging may include determining one or more combining coefficients for the first and second images, and the first and second images may be combined based on the one or more combining coefficients.

[0012] For example, in some variations, the one or more processors may be configured to determine the one or more composite coefficients, at least in part, by transforming the first and second images into first and second transform domain images using at least one transform operator, determining one or more transform domain composite coefficients for the first and second transform domain images, and inverse transforming the one or more transform domain composite coefficients to determine the one or more composite coefficients for the first and second images. The transform domain composite coefficients may be determined, for example, at least in part, by applying one or more coefficient combination rules (e.g., predetermined, heuristic-based, or learned rules, etc.) to the first and second transform domain images. The transform operator may include any suitable type of transform supporting 1:1 forward and inverse transforms (e.g., Fourier transform, discrete wavelet transform (DWT), discrete cosine transform (DCT), or wave atom transform).

[0013]

[0013] In some variations, the one or more processors may additionally or alternatively be configured to determine the one or more composite coefficients, at least in part, by determining a first image quality factor map for the first image and a second image quality factor map for the second image, determining a first composite coefficient for the first image based on the first image quality factor map, and determining a second composite coefficient for the second image based on the second image quality factor map.

[0014]

[0014] Additionally or alternatively, in some variations, the one or more processors may be configured to determine the one or more composite coefficients, at least in part, by determining a local entropy for each pixel in the first image and a local entropy for each pixel in the second image, and determining the one or more composite coefficients based on the determined local entropies.

[0015]

[0015] Other suitable techniques for determining the composite coefficients include, at least in part, applying a linear filter (e.g., a Gaussian difference filter) to each of the first and second images to estimate and weight the image content, determining one or more composite coefficients as a function of imaging depth, and / or determining one or more composite coefficients by applying a saturation mask that reduces the weight (e.g., composite coefficient) of at least a portion of the first and / or second images that exceeds a predetermined saturation threshold.

[0016]

[0016] In other words, the one or more processors may be configured to combine images from different types of sensors in the mixed transducer array using one or more suitable combining techniques described herein, including, for example, one or more of arithmetic averaging, geometric averaging, transform domain combining, image quality factor-based (IQF) combining, local entropy weighted combining, image content weighted combining, depth-dependent weighted combining, or saturation masking. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a block diagram of an exemplary image compositing system having a mixed array. [Figure 2] FIG. 1 is a block diagram of an exemplary image compositing system having a mixed array. [Figure 3] FIG. 1 is a block diagram of an exemplary image compositing system having a mixed array. [Figure 4] FIG. 1 is a block diagram of an exemplary image compositing system having a mixed array. [Figure 5] FIG. 1 is a block diagram of an exemplary image compositing system having a mixed array. [Figure 6] 1 is a flowchart of an exemplary method for performing image compounding on images collected by a mixing array. [Figure 7] 1 is a flowchart of an exemplary method for performing image compounding on images collected by a mixing array. [Figure 8] 1 is a flowchart of an exemplary method for performing image compounding on images collected by a mixing array. [Figure 9] 1 is a flowchart of an exemplary method for performing image compounding on images collected by a mixing array. [Figure 10] 1 is a flowchart of an exemplary method for performing image compounding on images collected by a mixing array. [Figure 11A] 1A-1C illustrate exemplary signals generated by a mixing array and harmonic filtering of those signals. [Figure 11B] 1A-1C illustrate exemplary signals generated by a mixing array and harmonic filtering of those signals. [Figure 11C] 1A-1C illustrate exemplary signals generated by a mixing array and harmonic filtering of those signals. [Figure 11D] 1A-1C illustrate exemplary signals generated by a mixing array and harmonic filtering of those signals. [Figure 11E] 1A-1C illustrate exemplary signals generated by a mixing array and harmonic filtering of those signals. [Figure 12] FIG. 1 illustrates a method for performing image compounding on images collected by a mixed array. [Figure 13] FIG. 1 illustrates a method for performing image compounding on images collected by a mixed array. DETAILED DESCRIPTION OF THE INVENTION

[0018]

[0030] Non-limiting examples of various aspects and variations of the present invention are described herein and illustrated in the accompanying drawings.

[0019]

[0031] Described herein are methods and devices for compositing (e.g., synthesizing) images collected using a mixed array containing multiple types of array elements. The mixed array described herein includes one or more array elements of a first type and one or more array elements of a second type different from the first type. One or more array elements of the first type can be used to form a first image, and one or more array elements of the second type can be used to form a second image. The first type can include non-optical transducers such as piezoelectric transducers, single-crystal material transducers, piezoelectric micromachined ultrasonic transducers (PMUTs), and / or capacitive micromachined ultrasonic transducers (CMUTs). The second type can include optical sensors, which can be optical resonators (e.g., whispering gallery mode (WGM) optical resonators or photonic integrated circuit (PIC) optical resonators) or interference-based optical sensors such as optical interferometers. The optical sensors can have any suitable shape. For example, the optical sensor may be a microbubble resonator, a microsphere resonator, a microtoroid resonator, a microring resonator, a microbottle resonator, a microcylinder resonator, and / or a microdisk optical resonator, etc. Optical sensors have high sensitivity and / or wide bandwidth in receiving ultrasonic signals compared to other types of ultrasonic sensors.

[0020]

[0032] Various suitable combinations of non-optical transducers and one or more types of optical sensors can be included in the hybrid transducer array. For example, in some variations, a first type of array element can include a non-optical transducer, and a second type of array element can include an optical sensor. One or more array elements of the first type can include a non-optical transducer (a non-optical subarray) for transmitting acoustic signals and / or detecting acoustic echoes to form a first image. One or more array elements of the second type (e.g., an optical sensor in an optical subarray) can be used to detect acoustic echoes (e.g., full spectrum, baseband, subharmonic, superharmonic, and / or differential harmonic), which can be used to form a second image. The second image generated by the high-sensitivity and / or wide-bandwidth optical sensor can be used independently or can be combined with the first image to form an even more improved image. Due to the high sensitivity and wide bandwidth of the optical resonator, images produced by optical sensors may have improved spatial resolution, improved contrast resolution, improved penetration depth, improved signal-to-noise ratio (SNR), improved tissue harmonic imaging, and / or improved Doppler sensitivity. However, because optical and non-optical subarrays have inherently different characteristics, a composite image produced by combining images produced using signals produced by different types of sensors may have more features, better image quality, and provide a more complete understanding of the underlying imaging target.

[0021]

[0033] Furthermore, optical sensors do not generate ultrasound waves and, therefore, are used together in mixed arrays with other transducers that generate ultrasound waves (e.g., piezoelectric, CMUT, etc.). Mixed arrays can include sensor elements arranged in various configurations and with various noise levels, amplitude responses, phase delays, frequency ranges, etc. As a result, conventional beamforming methods and devices commonly used for probes with one type of sensor are not optimal for probes that use mixed arrays of multiple types of sensors. The optical resonators described herein have an ultra-high quality factor (10 3 , 10 5 , 10 7 , 10 9Such an optical resonator may have a low quality factor (such as a high-quality factor) and thus have ultra-high sensitivity for ultrasound detection but a smaller dynamic range. Such an ultra-high quality factor optical resonator may be particularly suitable for ultra-deep imaging but may suffer from undesirable nonlinear distortion in the near field. On the other hand, optical resonators may be designed to have a lower quality factor and therefore lower sensitivity compared to optical resonators with ultra-high quality factors. Such low-quality factor optical resonators may be particularly suitable for imaging in the near field without undesirable nonlinear distortion. Furthermore, optical resonators may support many different resonance modes. Thus, the operating mode of the optical resonator may be switched from a first operating mode to a second operating mode, for example, by switching the wavelength of a laser source coupled to the optical resonator. In some variations, the imaging composite system may operate the optical resonators in an ultra-high quality factor operating mode at a first time and in a low-quality factor operating mode at a second time. In some variations, the imaging composite system may operate a first set of optical resonators in an ultra-high quality factor operating mode and a second set of optical resonators in a low-quality factor operating mode. Additionally, subarrays of different types of optical resonators can be deployed in the same image compositing system to produce different images showing different aspects of a target. Combining images produced by different optical resonators or by operating the optical resonators in different modes of operation using a compositing algorithm such as those described herein can produce or otherwise generate images with better image quality than images produced or generated by a single type of sensor.

[0022]

[0034] Thus, in some variations, the second type of array elements may include optical resonator sensors with different characteristics (e.g., different designs and / or different operating parameters). For example, in some variations, the second type of array elements may include one or more high-quality factor (high-Q) optical resonators and one or more low-quality (low-Q) optical resonators. Additionally or alternatively, the second type of array elements may include one or more tunable optical resonators configured to operate as high-Q optical resonators and one or more tunable optical resonators configured to operate as low-Q optical resonators. For example, such tunable optical resonators may be selectively operable in a high-Q mode or a low-Q mode, depending on the imaging settings, etc. Additionally or alternatively, the second type of array elements may include one or more optical resonator sensors designed for wide bandwidth and one or more optical resonator sensors designed for ultra-high sensitivity.

[0023]

[0035] Additionally, in some variations, a hybrid transducer array may include a combination of one or more non-optical transducers with multiple types of optical sensors. Thus, image combination systems and methods such as those described herein may be used to combine different types of input images (e.g., from non-optical transducers and / or from one or more different types of optical sensors) to obtain a composite image of better quality than any individual input image.

[0024]

[0036] Image Composite System FIG. 1 is a block diagram of an exemplary imaging combination system 100 having a mixed array. The imaging combination system 100 includes a probe 125, an imaging system 160, and a display 170. The probe 125 may be operably coupled to the imaging system 160. The probe 125 may receive and / or transmit a set of signals (e.g., electrical signals, electromagnetic signals, optical signals, etc.) to and from the imaging system 160. The probe 125 includes a mixing array 110 that may receive and / or transmit a set of signals (e.g., acoustic signals, etc.) to and from a medium for use in forming an image. The imaging system 160 may include a front end 140 and a back end 150 that collectively determine physical parameters (e.g., timing, location, angle, intensity, etc.) of signals transmitted to the probe (e.g., via one or more transmit channels) and post-process signals received by the probe 125 (e.g., via one or more receive channels) to form an image. Imaging system 160 may also be coupled to display 170 to transmit a set of signals (e.g., electrical signals, electromagnetic signals, etc.) to display 170. For example, in some variations, display 170 may be configured to display images produced by imaging system 160 (e.g., in a graphical user interface (GUI)). Additionally or alternatively, imaging system 160 may receive signals from display 170. For example, display 170 may further include an interactive interface (e.g., a touch screen, a keyboard, a motion sensor, etc.) for receiving commands from a user of imaging combination system 100, such as to control operation of imaging combination system 100.

[0025]

[0037] As shown in FIG. 1 , the probe 125 may include a mixed array 110, a multiplexer 120, and an optical sensor cable 130. The mixed array 110 may include one or more non-optical array elements (e.g., PZT transducers, CMUT transducers, etc.) and one or more optical array elements (e.g., optical sensors such as WGM resonators). The non-optical transducers may be configured to transmit acoustic waves and, in some variations, may be configured to further receive and detect acoustic echoes in response to the transmitted acoustic waves. The optical sensors may be configured to receive and detect echo signals with high sensitivity and / or wide bandwidth response. In some variations, the mixed array may be similar to any of the mixed arrays described in International Patent Application No. PCT / US2021 / 033715, which is incorporated herein by reference in its entirety. In some variations, the mixed array may be configured to perform harmonic imaging as described in International Patent Application No. PCT / US2021 / 039551, which is incorporated herein by reference in its entirety. In some variations, the probe 125 may be configured to repeatedly scan across the field of view by using the mixed array 110. In some variations, signals from the mixed array may be synthesized by synthetic aperture techniques, such as those described in International Patent Application No. PCT / US2021 / 049226, the entirety of which is incorporated herein by reference. Such signals may be used to generate images using optical sensors and / or non-optical transducers, as described in more detail below.

[0026]

[0038] The mixed array 110 may include an array of transducer elements and may be configured for operation in a one-dimensional (1D), one-and-a-half-dimensional (1.25D), one-and-a-half-dimensional (1.5D), one-and-a-half-dimensional (1.75D), or two-dimensional (2D) array configuration. Generally, the dimensionality of an ultrasonic sensor array relates to the range of elevation beamwidths (or elevation beam slice thicknesses) achievable when imaging with the ultrasonic sensor array and the amount of control the system has over the elevation beam size, focus, and / or steering of the sensor array across the imaging field (e.g., across the imaging depth). A 1D array has only one row of elements in the elevation dimension and a fixed elevation aperture size. A 1.25D array has multiple rows of elements in the elevation dimension and variable elevation aperture sizes, but a fixed elevation focus via an acoustic lens. A 1.5D array has multiple rows of elements in the elevation dimension, a variable elevation aperture size, and a variable elevation focus via electronic delay control. A 1.75D array is a 1.5D array with additional elevation beam steering capabilities. A 2D array has a large number of elements in both the lateral and elevation dimensions to meet the minimum pitch requirements for large beam steering angles in both the lateral and elevation directions.

[0027]

[0039] In some variations, the image combining system may be configured to convert a 1.5D or 2D array configuration into a 1D array configuration. The mixed array 110 may include a large number of elements (e.g., 16, 32, 64, 128, 256, 1024, 4096, 8192, 16384, etc.). In some variations, the mixed array 110 may be arranged in a rectangular configuration and include N x M elements, where N is the number of rows and M is the number of columns. In some variations, for example, the mixed array 110 includes one or more array elements of a first type and one or more array elements of a second type, where the first type may be a piezoelectric transducer or other non-optical transducer configured to transmit ultrasound, and the second type may be an optical sensor such as an optical resonator. The non-optical transducers and optical sensors may be collectively arranged in a rectangular, curved, circular, or sparse array configuration.

[0028]

[0040] The non-optical transducers in the mixed array 110 may be, for example, lead zirconate titanate (PZT) transducers, polymer thick film (PTF) sensors, polyvinylidene fluoride (PVDF) sensors, capacitive micromachined ultrasonic transducers (CMUT), piezoelectric micromachined ultrasonic transducers (PMUT), single crystal materials (e.g., LiNbO3 (LN), Pb(Mg 1 / 3 Nb 2 / 3 )-PbTiO3(PMN-PT), and Pb(In 1 / 2 Nb 1 / 2 )-Pb(Mg 1 / 3 Nb 2 / 3 The transducers may include transducers based on PbTiO3 (PIN-PMN-PT)-PbTiO3 (PIN-PMN-PT) and / or any transducer suitable for acoustic sensing.

[0029]

[0041] The optical sensor may be or include, for example, an interference-based optical sensor such as an optical interferometer or an optical resonator (e.g., a whispering gallery mode (WGM) optical resonator). In variations in which the optical sensor is an optical resonator, the optical sensor may have any suitable shape or form (e.g., a microring resonator, a microsphere resonator, a microtoroid resonator, a microbubble resonator, a fiber-based resonator, an integrated photonic resonator, a microdisk resonator, etc.). In some variations, the optical sensor may be / include, for example, a Fabry-Perot (FP) resonator, a fiber-based resonator (e.g., a fiber ring resonator), a photonic crystal resonator, a waveguide resonator, or any other suitable optical resonator capable of localizing optical energy in space and time. For example, in some variations, the optical resonator may be similar to any of the optical resonators described in International Patent Applications Nos. PCT / US2020 / 064094 and PCT / US2021 / 022412, each of which is incorporated herein by reference in its entirety.

[0030]

[0042] An optical resonator may include a closed loop of a transparent medium (e.g., glass, transparent polymer, silicon nitride, titanium dioxide, or any other material that is suitably optically transparent at the operating wavelength of the optical resonator) that allows several allowed frequencies of light to propagate continuously within the closed loop and accumulate optical energy of the allowed frequencies of light within the closed loop. This is equivalent to saying that the optical resonator may allow propagation of modes (e.g., whispering gallery modes (WGMs)) corresponding to the allowed frequencies to travel along the surface of the optical resonator and circulate around the resonator. Each mode corresponds to the propagation of at least one frequency of light from the allowed frequencies of light. The allowed frequencies and quality factors of the optical resonators described herein may be based, at least in part, on the geometric parameters of the optical resonator, the refractive index of the transparent medium, and the refractive index of the environment surrounding the optical resonator.

[0031]

[0043] The optical resonators described herein may have a set of resonant frequencies including a first subset of resonant frequencies and a second subset of resonant frequencies. In some variations, the optical resonator may be operated at the first subset of resonant frequencies having a high quality factor. Alternatively or additionally, in some variations, the optical resonator may be operated at the second subset of resonant frequencies having a low quality factor. The high quality factor subset of resonant frequencies may be suitable for operation in a high-sensitivity sensing probe (or subarray), and the low quality factor subset of resonant frequencies may be suitable for high dynamic range applications.

[0032]

[0044] In some variations, the sensitivity of the optical resonator can be controlled by tuning the geometric and / or characteristic material parameters of the optical resonator to tune the quality factor of the optical resonator. In some variations, the space inside and / or surrounding the optical resonator can be filled with an ultrasonically enhancing material, such as polyvinylidene fluoride, parylene, polystyrene, etc. The ultrasonically enhancing material can increase the sensitivity of the optical resonator.

[0033]

[0045] The optical resonator can be coupled to other components to receive / transmit light. In some implementations, the optical resonator can be operably coupled to a light source (e.g., a laser, a tunable laser, an erbium-doped fiber amplifier, etc.) and / or a photodetector (e.g., a p-doped / intrinsic / n-doped (PIN) diode) via an optical medium (e.g., an optical fiber, a tapered optical fiber, a free-space medium, etc.). Acousto-optic systems based on optical resonators can directly measure ultrasound waves (e.g., ultrasound echoes) through the photoelastic effect and / or physical deformation of the resonator in response to the ultrasound. Therefore, the optical resonator can be considered an optoacoustic transducer that can convert mechanical energy (e.g., acoustic energy) into optical energy. For example, in the presence of ultrasound (or any pressure wave), modes traveling within the resonator can undergo spectral shifts or amplitude changes caused by changes in the resonator's refractive index and / or shape. The spectral changes can be easily monitored and analyzed in the spectral domain using an optical detector. Amplitude changes can also be detected by the optical detector. The optical detector ultimately converts the optical energy (i.e., optical signal) propagating through the optical resonator and optical fiber into electrical energy (i.e., electrical signal) suitable for processing using electronic circuitry. By monitoring and analyzing the optical response of the optical resonator during the mixing array, additional spatial and other information can be further derived. Exemplary mixing transducer arrays are described herein. Additionally or alternatively, the signal from the optical resonator can be processed by optical circuitry before being converted to electrical energy by the optical detector.

[0034]

[0046] The mixed array 110 may have one or more non-optical array elements (e.g., ultrasound transducers or other non-optical sensors) and one or more optical array elements (e.g., optical resonators such as WGM optical resonators) arranged in various configurations (similar to any of the mixed arrays described in the above-incorporated U.S. Patent Application No. 63 / 029,044). For example, in some configurations, the non-optical and optical array elements may be collectively arranged in a rectangular array including several rows and several columns. The rectangular array may include N×M sensor elements, where N is the number of rows and M is the number of columns, both of which are integers. In some implementations, such as in the case of a 2D array, the number of rows and / or columns may be greater than 31 rows and / or 31 columns. For example, a 2D mixed array may include 64×96=6,144 sensor elements.

[0035]

[0047] In some variations, the mixing array 110 may include multiple different types of optical sensors. For example, as described further below, the different types of optical sensors may include wide-bandwidth optical resonators and ultra-high-sensitivity optical resonators. As another example, the mixing array 110 may include one or more high-quality factor (high-Q) optical resonators and one or more low-quality (low-Q) optical resonators. Additionally or alternatively, the mixing array 110 may include one or more tunable optical resonators configured to operate in different quality factor modes. For example, the tunable optical resonators may be operated in a low-quality factor (low-Q) operating mode for a high dynamic response or in a high-quality factor (high-Q) operating mode for a high-sensitivity response. In some implementations, the tunable optical resonators may be or include a first set of tunable optical resonators and a second set of tunable optical resonators that can be operated in different operating modes. In some implementations, the tunable optical resonators may be operated in a high-Q operating mode during a first time interval and in a low-Q operating mode during a second time interval. In other words, in some variations, the mixing array 110 may include one or more tunable optical resonators configured to operate as high-Q optical resonators and / or one or more tunable optical resonators configured to operate as low-Q optical resonators. For example, such tunable optical resonators may be selectively operable in a high-Q mode or a low-Q mode, depending on the imaging settings, etc.

[0036]

[0048] In some configurations, the spatial distribution of the positions of the multiple array element types may be random. Using a sparse spatial distribution of array elements may reduce and / or prevent the generation of grating lobes in images produced by the mixed array. The spatial distribution of the first type of array elements may be the same as, similar to, or different from the spatial distribution of the second type of array elements. In some configurations, the spatial distribution of the positions of the first and second type of array elements may follow a placement pattern (e.g., may be the same, may be shifted one cell to the right between sensor elements, or may be shifted two cells down between sensor elements). In some cases, one or more array elements of the second type may be smaller than or the same as one or more array elements of the first type.

[0037]

[0049] The non-optical transducers in the mixed array 110 may be operably coupled to a multiplexer 120 that processes electrical signals transmitted and / or received between the imaging system 160 and the non-optical transducers. The optical sensors in the mixed array 110 may be operably coupled to an optical sensor cable 130 that processes optical signals transmitted and / or received between the imaging system 160 and the optical sensors.

[0038]

[0050] The multiplexer 120 functions to selectively connect individual system channels to desired array elements. The multiplexer 120 may include analog switches. The analog switches may include multiple high-voltage analog switches. Each analog switch may be connected to an individual system channel. As a result, the multiplexer 120 may selectively connect individual system channels from the set of system channels of the imaging system 160 to desired transducer elements of the mixed array 110.

[0039]

[0051] The optical sensor cable 130 may include dedicated optical paths for transmitting and / or receiving optical signals to and from the optical sensors. The optical sensor cable 130 may include one or more optical waveguides, such as, for example, a fiber optic cable. The characteristics of the optical sensor cable 130 may depend on the type of optical signal, the type of optical sensor, and / or the arrangement of the optical sensors. In some configurations, multiple optical sensors (e.g., any two or more optical sensors forming an entire subarray of optical sensors, or a portion thereof) may be optically coupled to a single optical waveguide. Thus, signals from multiple optical sensors may be coupled to a single optical waveguide and communicated by the single optical waveguide. In some configurations, the subarray of optical sensors may be optically coupled to the array of optical waveguides in a 1:1 ratio (e.g., each optical sensor may be coupled to a respective optical waveguide). Thus, optical signals from the subarray of optical sensors may be coupled to one or more optical waveguides in the optical sensor cable 130 and communicated by those optical waveguides to the imaging system 160.

[0040]

[0052] The imaging system 160 may include a front end 140 and a back end 150. Generally, the front end 140 interfaces with the probe 125 to generate acoustic beams and receive electrical and / or optical signals. For example, the front end 140 may drive non-optical transducers (e.g., transducers) in the probe to transmit ultrasound signals in a predetermined beam pattern and may receive reflected ultrasound signals from non-optical transducers and optical sensors in a mixed array in the probe. The front end may also be tasked with performing both transmit and receive beamforming. The back end 150 may include one or more processors for processing signals received from the mixed array 110 via the front end to generate images, memory operably coupled to the processors for storing the images, and / or a communications interface for presenting the images to a user (e.g., via a graphical user interface). For example, the back end 150 may receive separately reconstructed images from a receive beamformer in the front end, perform additional back-end processes, and perform image compositing operations. Various back-end processes may be involved in image formation, including digital signal processing (DSP), digital scan conversion (DSC), envelope detection, etc. To perform image compositing using optical sensors, an image compositing system may include specific implementations of back-end processes for storing, analyzing, synthesizing, and transmitting data, signals, and / or images. Such specific implementations are illustrated and described below with respect to FIGS. 2-5.

[0041]

[0053] Display 170 may display the set of images generated by imaging system 160. In some variations, display 170 may additionally or alternatively include an interactive user interface (e.g., a touch screen) and be configured to send a set of commands (e.g., pause, resume, etc.) to imaging system 160. In some variations, image compositing system 100 may further include a set of one or more auxiliary devices (not shown) used to input information to or output information from image compositing system 100. The set of auxiliary devices may include, for example, a keyboard, a mouse, a monitor, a webcam, a microphone, a touch screen, a printer, a scanner, a virtual reality (VR) head-mounted display, a joystick, a biometric reader, etc. (not shown).

[0042]

[0054] 2 shows a block diagram of an exemplary image combining system 102 having a mixing array 110. As shown, the mixing array 110 may include a non-optical sub-array 113 and an optical resonator sub-array 114. The front end 140 may include a transmitter 142, a non-optical receiver 143, an optical resonator receiver 144, a transmit beamformer 145, a non-optical receive beamformer 146, and an optical resonator receive beamformer 147. The back end 150 may include a non-optical back-end processor 151 and an optical resonator back-end processor 152. The non-optical back-end processor 151 and the optical resonator back-end processor 152 may be involved in implementing one or more techniques, including digital signal processing (DSP), digital scan conversion (DSC), envelope detection, etc.

[0043]

[0055] The transmit beamformer 145 generates various transmit waveforms based on the transmit beamformer settings 181. The waveforms may be amplified by the transmitter 142, which may include analog circuitry, digital circuitry, and / or a computer system, before being applied to the non-optical subarray 113. After receiving the waveforms and / or the amplified waveforms by the transmitter 142, the non-optical subarray 113 may generate a set of acoustic waves (e.g., ultrasound signals) toward the target. The acoustic waves insonify the target, which then reflects a portion of the acoustic waves (i.e., echo signals) back to the mixing array probe. The non-optical receiver 143 receives the echo signals detected by the non-optical transducer and processes them to produce digitized signals as an output. The signals detected by the optical resonator subarray 114 may be processed and digitized by the optical resonator receiver 144. Non-optical receive beamformer 146, optical resonator receive beamformer 147, non-optical backend processor 151, and optical resonator backend processor 152 use the signals processed by the two receivers to form non-optical image 182 and optical resonator image 183. Non-optical image 182 and optical resonator image 183 often have different characteristics. The different characteristics of non-optical image 182 and optical resonator image 183 may depend on factors including the placement of sensing elements (non-optical transducers or optical resonators) within the mixed array, physical parameters of the sensing elements, etc.

[0044]

[0056] 3 shows a block diagram of an exemplary imaging combination system 103 having a mixing array 110 including an optical resonator sensor including subarrays with different quality factors (Q factors). As shown, the mixing array 110 may include a non-optical subarray 113, a high-quality factor (high-Q) optical resonator subarray 115, and a low-quality factor (low-Q) optical resonator subarray 116. The front-end 140 may include a transmit beamformer 145, a transmitter 142, a high-Q optical resonator receiver 148 that receives signals from the high-Q optical resonator subarray, a low-Q optical resonator receiver 149 that receives signals from the low-Q optical resonator subarray, and an optical resonator receive beamformer 147. While separate optical resonator receivers (high-Q optical resonator receiver 148 and low-Q optical resonator receiver 149) are shown in FIG. 3 as receiving signals from high-Q and low-Q optical resonators, respectively, it should be understood that in some variations, receivers 148 and 149 may be replaced by one or more receivers capable of receiving a wide range of Q-factor signals. For example, a single receiver may be dynamically tuned or otherwise configured to receive low-Q signals (e.g., in one or more “low-Q” modes) and tuned or otherwise configured to receive high-Q signals (e.g., in one or more “high-Q” modes). A single receiver may be dynamically configured across a spectrum of Q-factors or may be operable between different discrete modes corresponding to respective ranges of Q-factors. Backend 150 may include one or more optical resonator backend processors 152. The optical resonator back-end processor 152 may involve implementing one or more techniques including digital signal processing (DSP), digital scan conversion (DSC), envelope detection, and the like.

[0045]

[0057] The signals collected by the high-Q optical resonator subarray 115 may generate one or more high-sensitivity images 184, in which weaker signals from less reflective features or deeper depths may be better visualized, and strong signals from highly reflective features or shallower depths may be saturated. Meanwhile, the low-Q optical resonator subarray generates one or more high-dynamic-range images 185, which may miss weaker signals from smaller and less reflective features or deeper depths. The one or more high-sensitivity images 184 and the one or more high-dynamic-range images 185 may be used in the optical resonator back-end processor 152 to generate a composite image that includes the benefits of each of the signals from the high-Q and low-Q optical resonator subarrays.

[0046]

[0058] 3, in some variations, high-Q optical resonator subarray 115 and low-Q optical resonator subarray 116 may share optical resonator receive beamformer 147 and optical resonator back-end processor 152. Alternatively, in some variations, high-Q optical resonator subarray 115 and low-Q optical resonator subarray 116 may have different respective receive beamformers and / or different respective back-end processors. For example, high-Q optical resonator subarray 115 may be operably coupled to a high-Q optical resonator receive beamformer (not shown) and a high-Q optical resonator back-end process (not shown), and low-Q optical resonator subarray 116 may be operably coupled to a low-Q optical resonator receive beamformer (not shown) and a low-Q optical resonator back-end process (not shown).

[0047]

[0059] In some variations, front end 140 may further include a non-optical receiver and a non-optical receive beamformer (e.g., non-optical receiver 143 and non-optical receive beamformer 146 shown and described with respect to FIG. 2). Consequently, back end 150 may also include a non-optical back end processor, such as non-optical back end processor 151 that produces non-optical image 182 shown and described with respect to FIG. 2. Thus, image combination system 103 may be configured to form a combined image based on high sensitivity image 184 and high dynamic range image 185, and optionally further based on non-optical image 182.

[0048]

[0060] FIG. 4 shows a block diagram of an exemplary imaging combining system 104 having a mixing array 110, which is similar to the imaging combining system 103 shown and described above with respect to FIG. 3, except that the mixing array 110 includes a tunable optical resonator subarray 117 that can operate in two or more modes with different Q-factor values. Tuning into different modes may be achieved, for example, by selectively changing the ambient temperature of the mixing array 110 and / or by changing the optical wavelength. Such a tunable optical resonator subarray 117 may be used to collect both high-sensitivity and high-dynamic-range images. For example, in some variations, at least one optical resonator in the tunable optical resonator subarray 117 may receive signals at multiple times in response to different sets of transmit sequences, where at least one optical resonator operates in a high-Q mode at some times and a low-Q mode at different times. In other words, in some variations, at least a portion of the tunable optical resonator subarray 117 may be operated in a first time interval and a second time interval that does not overlap the first time interval, where at least a portion of the tunable optical resonator subarray 117 may be operated as a high-Q optical resonator in the first time interval to generate a high-sensitivity image 184 and as a low-Q optical resonator in the second time interval to generate a high-dynamic range image 185. In some variations, at least one tunable optical resonator may operate in a high-Q mode before operating in a low-Q mode. Additionally or alternatively, at least one tunable optical resonator may operate in a low-Q mode before operating in a high-Q mode. At least two sets of transmit sequences may be implemented to insonify the target multiple times to collect signals from both the high-Q optical resonator receiver 148 and the low-Q optical resonator receiver.

[0049]

[0061] Additionally or alternatively, in some variations, at least a first portion (e.g., a first set) of the tunable optical resonator subarray 117 may be consistently designated to operate in a high-Q mode, and at least a second portion (e.g., a second set) of the tunable optical resonator subarray 117 may be consistently designated to operate in a low-Q mode. Signals from the first portion of the tunable optical resonators may be received by high-Q optical resonator receiver 148, and signals from the second portion of the tunable optical resonators may be received by low-Q optical resonator receiver 149. In some variations in which the tunable optical resonator subarray simultaneously includes some optical resonators tuned to operate in a high-Q mode and some optical resonators tuned to operate in a low-Q mode, the mixing array 110 may be functionally similar to the mixing array 110 illustrated and described above with respect to FIG. 3 . While separate optical resonator receivers (high-Q optical resonator receiver 148 and low-Q optical resonator receiver 149) are shown in FIG. 4 as receiving high-Q and low-Q signals, respectively, similar to those described above with respect to FIG. 3, it should be understood that in some variations, receivers 148 and 149 may be replaced by one or more receivers capable of receiving a wide range of Q-factor signals. For example, a single receiver may be dynamically tuned or otherwise configured to receive low-Q signals (e.g., in one or more “low-Q” modes) and tuned or otherwise configured to receive high-Q signals (e.g., in one or more “high-Q” modes). A single receiver may be dynamically configured across a spectrum of Q-factors or may be operable between different discrete modes corresponding to respective ranges of Q-factors.

[0050]

[0062] As shown in FIG. 4 , the mixing array 110 may include a non-optical subarray 113 and a tunable optical resonator subarray. The front-end 140 may include a transmit beamformer 145, a transmitter 142, a high-Q optical resonator receiver 148, a low-Q optical resonator receiver 149, and an optical resonator receive beamformer 147. The non-optical subarray 113 in the mixing array 110 may transmit a set of acoustic signals, and the tunable optical resonator subarray may receive a set of acoustic echoes in response to the acoustic signals. The tunable optical resonator subarray 117 may be operably coupled to an optical detector configured to generate a first signal and a second signal, where the first signal includes a readout from at least a portion of the tunable optical resonator subarray 117 operating in a high-Q mode, and the second signal includes a readout from at least a portion of the tunable optical resonator subarray 117 operating in a low-Q mode. The high-Q optical resonator receiver 148 and the low-Q optical resonator receiver 149 may receive the first signal and the second signal, respectively. The back end 150 may include an optical resonator back-end processor 152. The optical resonator back-end processor 152 may perform operations including digital signal processing (DSP), digital scan conversion (DSC), envelope detection, etc. on the first signal and the second signal to generate a high-sensitivity image 184 and a high-dynamic range image 185. The back end 150 may be further configured to combine the high-sensitivity image 184 and the high-dynamic range image 185 to generate a composite image that includes the advantages of the signals of each of the high-Q and low-Q modes of the tunable optical resonator sub-array 117.

[0051]

[0063] In some variations, multiple transmit sequences are transmitted using the transmit beamformer configuration 181, the transmit beamformer 145, the transmitter 142, and the non-optical subarray 113 to insonify the target multiple times. For example, the non-optical subarray 113 may transmit a first transmit sequence and a second transmit sequence. In response, the tunable optical resonator subarray 117 may collect a first signal in response to the first transmit sequence and a second signal in response to the second transmit sequence. The backend may then create a first image from the first signal and a second image from the second signal.

[0052]

[0064] 5 shows a block diagram of an exemplary imaging combination system 105 having a mixed array 110 including optical resonators in both subarrays with wide bandwidth and subarrays with high sensitivity. For example, the mixed array may include a non-optical subarray 113, a wide-bandwidth optical resonator subarray 118, and an ultra-high-sensitivity optical resonator subarray 119. The wide-bandwidth optical resonator subarray 118 may capture signals outside the baseband of the transmitted acoustic waves, such as super- and sub-harmonics from tissue and / or contrast agents (e.g., as described in International Patent Application No. PCT / US2021 / 039551, incorporated by reference above). The ultra-high-sensitivity optical resonator subarray 119 may capture signals from deeper regions both within and outside the baseband.

[0053]

[0065] The non-optical subarray 113 may be operably coupled to the transmitter 142, which is operably coupled to the transmit beamformer 145 that receives the transmit beamformer setting 181. The non-optical subarray 113 transmits acoustic signals toward a target and receives acoustic echoes in response to the acoustic signals. The non-optical subarray 113 may be further operably coupled to the non-optical receiver 143 and the non-optical receive beamformer 146 in the front end 140 to generate a first signal in response to the acoustic echoes received at the non-optical subarray 113. The non-optical back-end processor 151 may analyze the first signal to generate a first image (non-optical image 182) that visualizes the target at conventional spatial resolution and imaging depth. The wide-bandwidth optical resonator subarray 118 and the ultra-high-sensitivity optical resonator subarray 119 may be operably coupled to the optical resonator receiver 144 and the optical resonator receive beamformer 147. The optical resonator back-end processor 152 may be used to process signals from the two optical resonator sub-arrays 118 and 119 to produce one or more images (e.g., fundamental frequency image, superharmonic image, subharmonic image, etc.) and one or more high-sensitivity images. For example, a second signal originating from the wide-bandwidth optical resonator sub-array 118 may be used to generate a second image (a superharmonic image 186), and / or a third signal originating from the ultrahigh-sensitivity optical resonator sub-array 119 may be used to generate a third image (a high-sensitivity image 184). Thus, the image combining system 105 may simultaneously achieve improved spatial resolution and imaging depth.

[0054]

[0066] After the first image, the second image, and / or the third image are generated separately using the first signal, the second signal, and / or the third signal from the non-optical subarray 113, the wide-bandwidth optical resonator subarray 118, and the ultra-high-sensitivity optical resonator subarray 119, respectively, an image blending algorithm may be used to combine the first image, the second image, and / or the third image to create a composite image, as described further below.

[0055]

[0067] How to perform image compositing 6-10, described below, illustrate embodiments of exemplary methods for performing image compositing based on images received from the above-described mixing array. While the methods are described primarily with reference to optical resonator sensors, it should be understood that they may similarly be implemented using signals from other types of optical sensors (e.g., optical interferometers). The methods for performing image compositing may be performed by an image compositing computing device, for example, part of the backend 150 illustrated and described with respect to FIGS. 1-5 and / or operably coupled to an image compositing system (such as the image compositing system 100 illustrated and described with respect to FIG. 1). The image compositing computing device may include a set of electronic circuits, such as a processor, memory, and a communication interface. The processor may include, for example, a hardware-based integrated circuit (IC) or any other suitable device for operating or executing a set of instructions / code. For example, the processor may include a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a microprocessor, a field-programmable gate array (FPGA) chip, a graphics processing unit (GPU), a digital signal processing (DSP) chip, etc. The memory may store, for example, code including instructions for causing the processor to perform one or more processes or functions (e.g., signal filtering, signal amplification, phase matching, noise reduction, aperture selection, etc.). The memory may be / include, for example, a memory buffer, a random access memory (RAM), a read-only memory (ROM), a flash drive, a secure digital (SD) memory card, etc. The communication interface may be / include, for example, a universal serial bus (USB) interface, a peripheral component interconnect express (PCIe) interface, or other hardware component operably coupled to the processor and / or memory and capable of enabling communication between the imaging complex computing device and components of the imaging complex system and / or, in some variations, with external devices and / or a network of devices (e.g., the Internet).

[0056]

[0068] The image combining computing device may include an application as software stored in memory and executed by the processor. For example, the application may include code to cause the processor to select an aperture, analyze signals, generate an image, etc. Alternatively, the application may be implemented on a hardware-based device. For example, the application may include digital or analog circuitry that may cause the image combining computing device to filter, amplify, and / or delay signals.

[0057]

[0069] 6 is a flowchart of an example method 600 for performing image compounding on images collected by a mixed array. In some implementations, the method may be implemented using the compound imaging system 102 (e.g., back end 150) as shown and described with respect to FIG. 2. The method 600 may include initiating image acquisition (601) (e.g., upon receiving an instruction to begin acquisition). The method 600 may further include transmitting a non-optical signal (602), followed by receiving a non-optical signal (603), and receiving an optical resonator signal (or other optical sensor signal) (604). The method may repeat 602, 603, and / or 604 until all desired transmitting steps for transmitting acoustic signals from all non-optical array elements and all receiving steps for receiving acoustic echoes from all non-optical and optical array elements of the mixed array 110 have been performed. Once all desired transmissions and receptions have been performed for at least one desired composite image (605), method 600 may further include generating or forming a non-optical image (606) and generating or forming an optical resonator image (607) using front end 140 and back end 150 of composite imaging system 102. Back end 150 may then apply image domain filters to the non-optical image and the optical resonator image (608, 609). The image domain filters may be specifically designed according to the image characteristics of each type of image. Method 600 may include combining (610) the non-optical image and the optical resonator image (e.g., using a combining algorithm, such as those described below) and creating a composite image (611). In general, in some variations, the composite image may be formed utilizing dynamically determined weight masks, for example, along with combining coefficients indicating which features of the non-optical image and which features of the optical resonator image may be included in each composite image.

[0058]

[0070] Additionally or alternatively, in some variations, the composite image may be formed utilizing a static weight mask that may be predetermined and stored for use during a later image compositing process. For example, if the image compositing method is image content independent (such as method 700) or static, the weight mask may be pre-calculated and stored in the memory of the image compositing system. Performing the image compositing method based on a pre-calculated weight mask may be processed faster and more efficiently by the processor of the image compositing system. Figure 7 is a flowchart of an exemplary method 700 for performing image compositing on images collected by a mixing array, where the image compositing utilizes a pre-calculated weight mask along with the compositing coefficients.

[0059]

[0071] Method 700 may include steps 601-607 as shown and described with respect to FIG. 6 . However, method 700 may further include retrieving (708) a pre-calculated weight mask. Method 700 may then perform a weighted averaging of the non-optical image and the optical resonator image to generate a composite image (709). The weighted averaging may include arithmetic averaging, geometric averaging, depth-dependent weighting, region-based weighting, etc. Method 700 may further include filtering (710) the composite image and creating (711) a composite image.

[0060]

[0072] FIG. 8 is a flowchart of an exemplary method 800 for performing image compounding on images collected by a mixed array. In some implementations, method 800 may be implemented using compound imaging system 103 as shown and described with respect to FIG. 3 . Method 800 may include initiating image acquisition (801) (e.g., upon receiving an instruction to begin acquisition). Method 800 may further include transmitting (802) a non-optical signal and subsequently receiving (803) a high quality factor (high-Q) optical resonator and / or low quality factor (low-Q) optical resonator signal. Method 800 may repeat 802 and 803 until all desired transmitting steps for transmitting acoustic signals from all non-optical array elements and all receiving steps for receiving acoustic echoes from all high-Q and low-Q optical resonator array elements have been performed. Once all desired transmissions and receptions for at least one desired composite image have been performed (804), method 800 may further include generating or forming a high-Q optical resonator image (also referred to as a high-sensitivity image) (805) and a low-Q optical resonator image (also referred to as a high-dynamic-range image) (806) using front end 140 and back end 150 of composite imaging system 103. Back end 150 may then filter (807) the high-Q optical resonator image and filter (808) the low-Q optical resonator image. Method 800 may include combining (809) the high-Q and low-Q optical resonator images (e.g., using a combining algorithm) and producing (810) a composite image. As with method 700, in some variations (e.g., when method 800 is image content independent or static), the weight masks may be pre-computed and stored in memory of image combining system 103 for faster processing.

[0061]

[0073] FIG. 9 is a flowchart of an exemplary method for performing image compounding on images collected by a mixing array. In some implementations, method 900 may be implemented using compound imaging system 104 as shown and described with respect to FIG. 4 . Method 900 may include initiating image acquisition (901) (e.g., upon receiving an instruction to begin acquisition). Method 900 may further include transmitting a non-optical signal (902) followed by receiving an optical resonator signal from at least one tunable optical resonator operating in a high-Q mode (903). In some cases, the optical resonator may be operated at a high-Q setting by selecting an optical wavelength (of the light source) to match a resonant frequency with a high resonance quality factor. Method 900 may further include transmitting a non-optical signal (904) followed by receiving an optical resonator signal from at least one tunable optical resonator operating in a low-Q mode (905). While the flowchart in FIG. 9 illustrates receiving a signal from the optical resonator in the high-Q mode before receiving a signal from the optical resonator in the low-Q mode, it should be understood that, alternatively, a signal from the optical resonator in the low-Q mode may be received before receiving a signal from the optical resonator in the high-Q mode. Method 900 may repeat steps 902-905 until all desired transmit steps for transmitting acoustic signals from all non-optical array elements and all receive steps for receiving acoustic echoes at all Q optical resonator array elements at low-Q and high-Q settings have been performed. Once all desired transmits and receives for at least one desired composite image have been performed (906), method 900 may further include generating or forming a high-Q optical resonator image (907) and a low-Q optical resonator image (908) using the front end 140 and back end 150 of the composite imaging system 104. The back end 150 may then filter the high-Q optical resonator image (909) and filter the low-Q optical resonator image (910). The method 900 may include combining 911 the high-Q optical resonator image and the low-Q optical resonator image (eg, using a combining algorithm) to produce 912 a combined image.As with methods 700 and 800, in some variations (e.g., when method 900 is static), the weight masks may be pre-computed and stored in memory of image combination system 104 for faster processing.

[0062]

[0074] FIG. 10 is a flowchart of an exemplary method for performing image compounding on images collected by a compound array. In some implementations, method 1000 may be implemented using compound imaging system 105 as shown and described with respect to FIG. 5. Method 1000 may include initiating image acquisition (1001) (e.g., upon receiving an instruction to begin acquisition). Method 1000 may further include transmitting a non-optical signal (1002), followed by receiving a non-optical signal (1003), and receiving an optical resonator signal (1004) (e.g., from a wide-bandwidth optical resonator subarray and / or an ultra-high-sensitivity optical resonator subarray). Method 1000 may repeat steps 1002-1004 until all desired transmitting steps for transmitting acoustic signals from all non-optical array elements and all receiving steps for receiving acoustic echoes at all non-optical and optical resonator array elements have been performed. Once all desired transmit and receive steps for at least one desired composite image have been performed (1005), method 1000 may further include generating or forming a non-optical image (1006), generating or forming a harmonic optical resonator image (1007), and generating or forming a high-sensitivity optical resonator image (1008) using front end 140 and back end 150 of composite imaging system 105. Back end 150 may then filter the non-optical image (1009), filter the harmonic optical resonator image (1010), and filter the high-sensitivity optical resonator image (1011). Filtering the harmonic optical resonator image (i.e., the low-Q optical resonator image) may include implementing a set of bandpass filters and / or a set of one-dimensional signal filters to extract components within subharmonic and / or ultraharmonic bands. These filtered signals are then used to form harmonic images in each of the selected bands. The method 1000 may include combining (1012) the non-optical image, the harmonic optical resonator image, and the high-sensitivity optical resonator image (e.g., using a combining algorithm) to produce (1013) a combined image.

[0063]

[0075] As described above, when forming a harmonic optical resonator image, the optical resonator signal can be processed using a filter bank containing one or more filters. Figures 11A-11E show exemplary signals generated by a mixing array and harmonic filtering of those signals. As shown in Figure 11A, a first signal 1101 is received by a broadband optical resonator. By performing a transformation, such as a Fourier transform, the first signal 1101 can be converted from the time domain to the frequency domain 1111. As shown by the solid line in Figure 11B, the first signal primarily contains baseband components around 6 MHz with a bandwidth of approximately 87% (or 5.22 MHz). However, the spectrum of the first signal reveals the presence of a second harmonic component at -25 dB and a third harmonic component at -35 dB in the first signal. The first signal also contains additive 1 / f pink noise at -35 dB.

[0064]

[0076] 11C-11E illustrate the extraction of harmonic components using a suitable filter. For example, as shown in FIG. 11D, a 101-tap finite impulse response (FIR) second harmonic bandpass filter can be applied to the first signal 1101 to extract the filtered second harmonic signal 1102. Additionally, a third harmonic bandpass filter (dash-dotted line in the lower right panel) can be applied to the first signal 1101 to extract the filtered third harmonic signal 1103. In some cases, the time signal (the signal in the time domain) can be normalized, and the second and third harmonic signals can be much weaker than the baseband signal. This is because tissue-generated ultraharmonic signals are typically (e.g., by several orders of magnitude) lower than the baseband signal. Furthermore, higher frequency signals suffer greater loss in biological tissue. Without wide bandwidth sensors such as the optical resonators described herein, and methods and apparatus for compound imaging based on signals generated by optical resonators, harmonic imaging can be extremely difficult to achieve.

[0065]

[0077] Complex Algorithms Described herein are exemplary compositing algorithms for combining multiple images based on signals from non-optical array elements and / or optical resonator array elements. In some cases, n images of m dimensions (mD) are combined (via image compositing) to generate a single mD image computed as output (n and m are integers). When m is 2, mD images are sometimes referred to as "images," and when m is 3, they are sometimes referred to as "volumes." The described compositing algorithms can be applied to both images and volumes. Generally, in some variations, the compositing algorithms can produce compositing coefficients (e.g., factors) that characterize which or how much of each feature (e.g., pixel intensity) of each separate image (e.g., non-optical image, optical resonator image) can contribute to each composite image. The compositing coefficients can be written into a weighting mask that can be applied to the images to extract desired features for contribution to the composite image.

[0066]

[0078] In some variations, the combining algorithm may be or include arithmetic averaging. The concept behind arithmetic averaging for combined imaging based on signals received from a combining array is to combine n input images into one output image using direct pixel-by-pixel arithmetic averaging of pixel values.

number

[0067]

[0079] In some variations, the composite algorithm may be or include geometric averaging. Similar to the arithmetic averaging method described above, geometric averaging is also a pixel-by-pixel method implemented by:

number

[0068]

[0080] In some variations, the compositing algorithm may be or include transform domain compositing. This is a class of compositing methods that rely on transforming the input image into a transform domain that supports one-to-one forward and inverse transforms. One-to-one transforms may include, for example, a Fourier transform, a discrete wavelet transform (DWT), a discrete cosine transform (DCT), a wave atom transform, etc. After the transformation, a set of heuristic-based rules or learned rules may be applied to obtain composite coefficients in the transform domain. An inverse transform may then be performed to convert the composite coefficients to the image domain. An example of this process is shown in FIG. 12. An input image 1202 (a non-optical image and / or an optical resonator image) may undergo a transform 1204 to generate coefficients 1206. Coefficient compositing rules 1208 may be applied to these coefficients to generate composite coefficients 1210 in the transform domain. The composite coefficients may then be inverse transformed 1212 to convert the composite coefficients to the image domain for use in generating a composite image 1214.

[0069]

[0081] In some variations, the transform domain combination may use a transform that is suitable for multi-scale analysis of images, such as the DWT. In the context of the DWT, illustrative examples of coefficient combination rules include the following: For the smallest scale among multiple scales, take the smallest coefficient among the coefficients of all images (e.g., non-optical image, high-Q optical image, low-Q optical image, etc.). This rule assumes that the smallest scale contains mostly noise and should therefore be minimized. For the largest scale among multiple scales, we take the average of the coefficients across all input images. This rule assumes that the largest scale represents the general shape of the object and should be consistent across input images. For all other scales (other than the smallest and largest scales) among the scales, take the maximum value of the coefficients among all input images. This rule assumes that all other scales represent some details of the target, and different input images may be best at representing one or more aspects. By taking the maximum value, all details can be preserved.

[0070]

[0082] However, when the DWT method is applied in method 1000 as shown and described with respect to FIG. 10, greater weights may be assigned to the smaller scale coefficients of the ultraharmonic image and the larger scale coefficients of the non-optical image.

[0071]

[0083] Additionally or alternatively, a set of coefficient combination rules (e.g., rules that may be learned via a suitable machine learning algorithm, etc.) may be predefined for different ultrasound frequencies (e.g., as a look-up table, as a function of ultrasound frequency, etc.). For example, a first combination coefficient (or a first range of combination coefficients) may be associated with images generated using a high ultrasound frequency (or a range of high ultrasound frequencies), and a second combination coefficient (or a second range of combination coefficients) may be associated with images generated using a low ultrasound frequency (or a range of low ultrasound frequencies). Generally, in some variations, because higher ultrasound frequencies are more attenuated in long-distance imaging, the combination coefficients may be lower as the imaging depth increases, such that images generated using high ultrasound frequencies are given less weight in creating the combination image.

[0072]

[0084] In some variations, the composite algorithm may be or include image quality factor (IQF)-based composite, as shown in FIG. 13. Image quality factor (IQF) may be defined as a quantitative measure of image quality and may be represented or otherwise characterized, at least in part, by an image quality factor map for an image. There are various IQFs developed for different purposes and applications. For example, each and / or any combination of signal-to-noise ratio (SNR), entropy, detail resolution, contrast resolution, and penetration depth may be used as an IQF. Different IQFs enhance different aspects of the ultrasound image. In some cases, one or more IQFs 1304 may be extracted from the input image 1302. The IQFs 1304 are then converted into composite coefficients 1306. The composite image I f (x)1308 is the input image I j It can be calculated by the weighted sum of (x).

number

[0073]

[0085] In some variations, the combining algorithm may be or include local entropy weighted combining. Local entropy weighted combining combines input images by assigning a weight to each pixel of each input image based on the information content in its neighborhood. This may be done by calculating the entropy of the region surrounding each pixel of each input image. The local entropy of a pixel at coordinate x in the jth image may be calculated by:

number

number

[0074]

[0086] Instead of this particular example, H x,j Many functions can be used to transform σ to a non-negative value. The composite image can be expressed as:

number

[0075]

[0087] In some variations, the combining algorithm may be or include fast image content weighting combining. As an approximation to local entropy-based weighting, a faster linear filtering-based algorithm may also be used. Instead of calculating the local entropy of the input image, which can be computationally expensive, W j [x] is calculated by applying a Difference of Gaussian (DoG) filter to the jth image. The same formula as for local entropy weighted compositing can be used to generate the composite image.

[0076]

[0088] In some variations, the compositing algorithm may be or include depth-dependent weighting compositing. Predetermined depth-dependent weighting may be useful when the input image has well-defined characteristics that are depth-dependent. Because some input images may have better quality in shallower regions and other images may have better quality in deeper regions, depth-dependent weighting compositing may be particularly useful when the optical resonator subarray includes or is operated as an ultrasensitive optical resonator (e.g., as shown in FIGS. 3 and 4). Many depth-weighting functions may be used, including, but not limited to, linear and gamma functions.

[0077]

[0089] In some variations, the compositing algorithm may be or include saturation masking. When some input images are prone to signal saturation (e.g., images produced by high-Q optical resonators) or other types of nonlinearities due to excessive signal amplitude, a saturation masking step may be introduced into these input images before they undergo the compositing method. Signal saturation may be detected by comparing a moving average of the beamformed images to a predetermined threshold. When saturation is detected, saturated pixels of the input image under examination may be assigned a weight of 0 or near 0 so that their contribution to the composite image is reduced and the other, non-saturated, input image or images dominate.

[0078]

[0090] Although the image compositing methods and systems for mixed arrays are described in the context of ultrasound imaging, in some variations, the image compositing methods and systems may be used in applications other than ultrasound imaging. For example, in some cases, the image compositing methods and systems may be used in computed tomography, magnetic resonance imaging, metrology, signal processing, particle physics, remote sensing, aerospace applications, etc. The image compositing methods disclosed herein may also be applied to combine images generated using different imaging modalities to form a fused image. For example, ultrasound, CT, and MRI images of the same region of a patient may be fused to reveal more diagnostic information.

[0079]

[0091] Although in some variations described above, the tunable optical resonator is described as operating in a low-quality factor (low-Q) or high-quality factor (high-Q) mode of operation, in general, the tunable optical resonator can be operated in multiple modes of operation (e.g., three modes of operation, ten modes of operation, or one hundred modes of operation). For example, the tunable optical resonator can be operated in a low-Q mode of operation to generate a first image having a high linear range, a high-Q mode of operation to generate a second image having a high sensitivity, and a medium-quality factor mode of operation to generate a third image having a balance between sensitivity and linear range. The back end of the image combining system 100 can be configured to combine the first, second, and third images to generate a combined image that is better (e.g., in terms of resolution, depth, contrast, quality factor, etc.) than each of the first, second, or third images.

[0080]

[0092] The foregoing description, for purposes of explanation, used specific terms to provide a thorough understanding of the invention. However, it will be apparent to those skilled in the art that specific details are not required to practice the present invention. Thus, the foregoing descriptions of specific embodiments of the present invention are presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed, and obviously, many modifications and variations are possible in light of the above teachings. The embodiments were chosen and described in order to explain the principles of the invention and its practical application, so that they will enable others skilled in the art to utilize the present invention and various embodiments, with various modifications as may be suitable for the particular use contemplated. It is intended that the following claims and their equivalents define the scope of the invention.

Claims

1. receiving a first signal from one or more array elements of a first type in a mixed transducer array; receiving a second signal from one or more array elements of a second type in the mixed transducer array, wherein at least one of the first type and the second type is an optical sensor; generating a first image from the first signal and a second image from the second signal; Determining one or more composite coefficients for the first image and the second image, wherein determining one or more composite coefficients for the first image and the second image comprises: determining a local entropy for each pixel in the first image and in the second image; determining the one or more complex coefficients based on the determined local entropy; and combining the first image and the second image to generate a composite image based on the one or more composite coefficients; Including, 10. A method of imaging, wherein combining the first image and the second image includes determining a weighted average of the first image and the second image.

2. The method of claim 1 , wherein the first type and the second type are optical resonators having different properties.

3. 3. The method of claim 2, wherein the first type is a high-Q optical resonator and the second type is a low-Q optical resonator.

4. 3. The method of claim 2, wherein the first type is a tunable optical resonator operated as a high-Q optical resonator, and the second type is a tunable optical resonator operated as a low-Q optical resonator.

5. The method of claim 1 , wherein the first type is a non-optical transducer and the second type is an optical sensor.

6. The method of claim 5 , wherein the non-optical transducer is a piezoelectric transducer, a single crystal material transducer, a piezoelectric micromachined ultrasonic transducer (PMUT), or a capacitive micromachined ultrasonic transducer (CMUT).

7. 6. The method of claim 5, wherein the second type is a wide bandwidth optical sensor, the method further comprising receiving a third signal from one or more array elements of a third type, the third type being an ultra-high sensitivity optical sensor.

8. The method of claim 7 , comprising filtering the first signal, the second signal, and / or the third signal using one or more filters.

9. The method of claim 8 , wherein the one or more filters include a harmonic bandpass filter.

10. The method of claim 1 , wherein combining the first image and the second image comprises determining an arithmetic or geometric mean of the first image and the second image.

11. Determining one or more composite coefficients for the first image and the second image comprises: transforming the first image and the second image into a first transform domain image and a second transform domain image using at least one transform operator; determining one or more transform domain composite coefficients for the first transform domain image and the second transform domain image; inverse transforming the one or more transform domain complex coefficients to determine the one or more complex coefficients for the first image and the second image; The method of claim 1 , comprising:

12. 12. The method of claim 11 , wherein determining one or more transform domain combination coefficients for the first transform domain image and the second transform domain image comprises applying one or more coefficient combination rules to the first transform domain image and the second transform domain image.

13. The method of claim 11 , wherein the at least one transform operator comprises a Fourier transform, a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a wave atom transform.

14. Determining one or more composite coefficients for the first image and the second image comprises: determining a first image quality factor map for the first image and a second image quality factor map for the second image; determining a first composite coefficient for the first image based on the first image quality factor map and a second composite coefficient for the second image based on the second image quality factor map; The method of claim 1 , comprising:

15. The method of claim 1 , wherein determining one or more composite coefficients for the first image and the second image comprises applying a linear filter to each of the first image and the second image.

16. The method of claim 15 , wherein the linear filter comprises a Gaussian difference filter.

17. The method of claim 1 , wherein determining one or more composite coefficients for the first image and the second image comprises determining one or more composite coefficients as a function of imaging depth.

18. 2. The method of claim 1, wherein determining a weighted average of the first image and the second image comprises applying a saturation mask that reduces a weight of at least a portion of the first image and / or the second image that exceeds a predetermined saturation threshold.

19. The method of claim 1 , wherein the optical sensor is a WGM optical resonator.

20. 10. The method of claim 1, wherein the optical sensor is a microbubble optical resonator, a photonic integrated circuit (PIC) optical resonator, a microsphere resonator, a microtoroid resonator, a microring resonator, a microbottle resonator, a microcylinder resonator, or a microdisk optical resonator.

21. one or more non-optical transducers in the mixed transducer array transmit acoustic signals at a fundamental frequency f; 2. The method of claim 1 , wherein the one or more array elements of the first type, the second type, or both the first type and the second type are configured to produce one or more optical responses upon receiving harmonic or subharmonic acoustic echoes corresponding to the transmitted acoustic signal, and the one or more array elements of the second type have a bandwidth ranging from at least f / M to Nf, where M and N are integers greater than 1.

22. The one or more non-optical transducers have a first fundamental frequency f １ and the second fundamental frequency f ２ The method of claim 1 , further comprising transmitting the acoustic signal by

23. The one or more array elements of the second type are one or more linear combinations nf １ +mf ２ and n and m are configured to produce one or more optical responses upon receiving an acoustic echo corresponding to a frequency of nf １ +mf ２ 23. The method of claim 22, wherein is an integer such that is a positive number.

24. The method of claim 1 , wherein at least one of the first image and the second image is a harmonic image.

25. 25. The method of claim 24, wherein the harmonic image is a subharmonic image or an ultraharmonic image.

26. 1. An apparatus for imaging a target, comprising: one or more array elements of a first type configured to receive a first signal; one or more array elements of a second type configured to receive a second signal; and wherein at least one of the first type and the second type is an optical sensor; and generating a first image from the first signal and a second image from the second signal; determining one or more composite coefficients for the first image and the second image at least in part by determining a local entropy for each pixel in the first image and the second image, and determining one or more composite coefficients based on the determined local entropies; combining the first image and the second image to generate a composite image based on the one or more composite coefficients; one or more processors configured to perform Equipped with the one or more processors are configured to combine the first image and the second image, at least in part, by determining a weighted average of the first image and the second image.

27. 27. The apparatus of claim 26, wherein the first type and the second type are optical resonators having different properties.

28. 28. The apparatus of claim 27, wherein the first type is a high-Q optical resonator and the second type is a low-Q optical resonator.

29. 28. The apparatus of claim 27, wherein the first type is a tunable optical resonator operated as a high-Q optical resonator, and the second type is a tunable optical resonator operated as a low-Q optical resonator.

30. 27. The apparatus of claim 26, wherein the first type is a non-optical transducer and the second type is an optical sensor.

31. 31. The apparatus of claim 30, wherein the non-optical transducer is a piezoelectric transducer, a single crystal material transducer, a piezoelectric micromachined ultrasonic transducer (PMUT), or a capacitive micromachined ultrasonic transducer (CMUT).

32. 31. The apparatus of claim 30, wherein the second type is a wide bandwidth optical sensor, the mixed transducer array further comprising one or more array elements of a third type configured to receive a third signal, and the third type is an ultra-high sensitivity optical sensor.

33. 33. The apparatus of claim 32, wherein the one or more processors are configured to filter the first signal, the second signal, and / or the third signal using one or more filters.

34. 34. The apparatus of claim 33, wherein the one or more filters include a harmonic bandpass filter.

35. 27. The apparatus of claim 26, wherein the one or more processors are configured to combine the first image and the second image, at least in part, by determining an arithmetic or geometric mean of the first image and the second image.

36. The one or more processors, at least in part, transforming the first image and the second image into a first transform domain image and a second transform domain image using at least one transform operator; determining one or more transform domain composite coefficients for the first transform domain image and the second transform domain image; inverse transforming the one or more transform domain complex coefficients to determine the one or more complex coefficients for the first image and the second image; 27. The apparatus of claim 26, configured to determine one or more composite coefficients for the first image and the second image by:

37. 37. The apparatus of claim 36, wherein the one or more processors are configured to determine one or more transform domain combination coefficients for the first transform domain image and the second transform domain image, at least in part, by applying one or more coefficient combination rules to the first transform domain image and the second transform domain image.

38. 37. The apparatus of claim 36, wherein the at least one transform operator comprises a Fourier transform, a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a wave atom transform.

39. The one or more processors, at least in part, determining a first image quality factor map for the first image and a second image quality factor map for the second image; determining a first composite coefficient for the first image based on the first image quality factor map and a second composite coefficient for the second image based on the second image quality factor map; 27. The apparatus of claim 26, configured to determine one or more composite coefficients for the first image and the second image by:

40. 27. The apparatus of claim 26, wherein the one or more processors are configured to determine one or more composite coefficients for the first image and the second image, at least in part, by applying a linear filter to each of the first image and the second image.

41. 41. The apparatus of claim 40, wherein the linear filter comprises a Gaussian difference filter.

42. 27. The apparatus of claim 26, wherein the one or more processors are configured to determine one or more composite coefficients for the first image and the second image, at least in part, by determining one or more composite coefficients as a function of imaging depth.

43. 27. The apparatus of claim 26, wherein the one or more processors are configured to determine a weighted average of the first image and the second image, at least in part, by applying a saturation mask that reduces a weight of at least a portion of the first image and / or the second image that exceeds a predetermined saturation threshold.

44. 27. The apparatus of claim 26, wherein the optical sensor is a WGM optical resonator.

45. 27. The apparatus of claim 26, wherein the optical sensor is a microbubble optical resonator, a photonic integrated circuit (PIC) optical resonator, a microsphere resonator, a microtoroid resonator, a microring resonator, a microbottle resonator, a microcylinder resonator, or a microdisk optical resonator.

46. one or more non-optical transducers in the mixed transducer array transmit acoustic signals at a fundamental frequency f; 27. The apparatus of claim 26, wherein the one or more array elements of the first type, the second type, or both the first type and the second type are configured to produce one or more optical responses upon receiving harmonic or sub-harmonic acoustic echoes corresponding to the transmitted acoustic signal, and wherein the one or more array elements of the second type have a bandwidth ranging from at least f / M to Nf, where M and N are integers greater than 1.

47. The one or more non-optical transducers have a first fundamental frequency f １ and the second fundamental frequency f ２ 27. The device of claim 26, wherein the device transmits an acoustic signal.

48. The one or more array elements of the second type are one or more linear combinations nf １ +mf ２ and n and m are configured to produce one or more optical responses upon receiving an acoustic echo corresponding to a frequency of nf １ +mf ２ 48. The apparatus of claim 47, wherein is an integer such that is a positive number.

49. 27. The apparatus of claim 26, wherein at least one of the first image and the second image is a harmonic image.

50. 50. The apparatus of claim 49, wherein the harmonic image is a subharmonic image or an ultraharmonic image.

Images