Image composite for mixed transducer arrays

Abstract

Ultrasonic probes with mixed transducer arrays (mixed arrays) containing sensors with higher bandwidth and sensitivity are needed. [Solution] The imaging method may include receiving a first signal from one or more array elements of a first type in a mixed transducer array, and receiving a second signal from one or more array elements of a second type in a mixed transducer array, wherein at least one of the first and second types is an optical sensor; generating a first image from the first signal and a second image from the second signal; and combining the first image and the second image to generate a composite image.

Description

[Technical Field]

[0001]

[0001] Cross-reference of related applications This application claims priority to U.S. Patent Application No. 63 / 104,886, filed on 23 October 2020, which is incorporated herein by this reference in its entirety.

[0002]

[0002] This disclosure generally relates to the field of imaging, and more particularly to methods and devices that enable the formation of composite images from images acquired by a mixed array including an array of optical sensors and other transducers. The methods and devices disclosed herein include optical sensors having high sensitivity and / or high operating bandwidth for improved imaging performance. [Background technology]

[0003]

[0003] Ultrasonic detection is used in various industries, including medical imaging and medical diagnosis, due to several advantages. For example, ultrasonic detection utilizes ultrasonic signals with significant penetration depth. Furthermore, since ultrasonic imaging is based on non-ionizing radiation, it is advantageously known to be a non-invasive imaging method. [Overview of the project] [Problems that the invention aims to solve]

[0004]

[0004] Various known ultrasonic transducers used in ultrasonic imaging have many drawbacks. For example, some ultrasonic transducers are made of piezoelectric materials such as lead zirconate titanate (PZT). However, the 6 dB bandwidth of PZT material is generally limited to only about 70%. Some composite PZT materials have a slightly increased bandwidth, but still only achieve a maximum bandwidth of about 80%. As another example, single-crystal materials are increasingly used to improve the performance of ultrasonic probes, but they have a low Curie temperature and are brittle. Another type of transducer material is silicon, which can be processed to construct capacitive micromachine ultrasonic transducer (CMUT) probes that can have an increased bandwidth. However, CMUT probes do not have very high sensitivity or reliability. Furthermore, CMUT probes have some operational limitations. For example, CMUT probes are nonlinear sensors and are therefore generally not suitable for harmonic imaging. Therefore, there is a need for ultrasonic probes that have a mixed transducer array (mixed array) containing sensors with higher bandwidth and sensitivity. Furthermore, backend and / or frontend devices are required to process the signals and / or images generated by the mixed array. [Means for solving the problem]

[0005]

[0005] Generally, in some variations, the apparatus for imaging (e.g., ultrasound imaging of a patient) (e.g., an image composite system) may include a mixed transducer array comprising one or more array elements of a first type configured to receive a first signal and one or more array elements of a second type configured to receive a second signal, where at least one of the first and second types is an optical sensor. The apparatus may further include one or more processors configured to generate a first image from the first signal, a second image from the second signal, and to combine the first and second images to generate a composite image.

[0006]

[0006] In some variations, the first type of array element may include non-optical transducers, and the second type of array element may include optical sensors. One or more array elements of the first type may include, for example, piezoelectric transducers, single-crystal material transducers, piezoelectric micromachine ultrasonic transducers (PMUTs), or capacitive micromachine ultrasonic transducers (CMUTs). Optical sensors may include, for example, whispering gallery mode (WGM) optical resonators, microbubble optical resonators, photonic integrated circuit (PIC) optical resonators, microsphere resonators, microtoroid resonators, microring resonators, microbottle resonators, microcylinder resonators, and / or microdisk optical resonators.

[0007]

[0007] In some variations, the second type of array element may include optical sensors having different characteristics (e.g., different designs and / or different operating parameters). For example, in some variations, the second type of array element may include one or more high-quality coefficient (high-Q) optical sensors and one or more low-quality (low-Q) optical sensors. As an addition or alternative, the second type of array element may include one or more tunable optical resonators configured to operate as high-Q optical resonators, and / or the second type of array element may include one or more tunable optical resonators configured to operate as low-Q optical resonators. For example, such tunable optical resonators may be selectively operable in high-Q mode or low-Q mode depending on the imaging settings, etc.

[0008]

[0008] Furthermore, in some variations, a mixed transducer array may include a combination of one or more non-optical transducers and multiple types of optical sensors. For example, a mixed transducer array may include one or more array elements of a first type, each including at least one non-optical transducer; one or more array elements of a second type, each including at least one type of optical sensor; and one or more array elements of a third type, each including at least another type of optical sensor. One or more processors may be further configured to generate a third image from a third signal and to synthesize the first, second, and third images to generate a composite image. Different types of optical resonator sensors may include, for example, high-Q optical resonators and low-Q optical resonators (or tunable optical resonator sensors configured to operate as either high-Q or low-Q optical resonators). As another example, different types of optical resonator sensors may include broadband optical resonators and ultra-high-sensitivity optical resonators.

[0009]

[0009] In some variations, one or more array elements (e.g., transducers) of a mixed transducer array may transmit an acoustic signal at a fundamental frequency f. In response, one or more array elements of the first type, the second type, or both the first and second types may produce one or more responses upon receiving harmonic acoustic echoes (including superharmonics and subharmonics) corresponding to the transmitted acoustic signal. One or more array elements of the second type may have a bandwidth ranging from at least f / M to Nf, where M and N are integers greater than 1. In some variations, one or more array elements of the first type may transmit an acoustic signal at a first fundamental frequency f1 and a second fundamental frequency f2. In response, one or more array elements of the second type may produce one or more optical responses upon receiving acoustic echoes corresponding to the frequencies of one or more linear combinations nf1 + mf2, where n and m are integers such that nf1 + mf2 is a positive number. At least one of the first image and the second image is a harmonic image, or may include a harmonic image.

[0010]

[0010] In some variations, one or more processors may be configured to filter various signals from different types of array elements in a mixed transducer array using one or more suitable filters. Such suitable filters may include, for example, harmonic bandpass filters that can enable the extraction of harmonic signals, including low-harmonic and super-harmonic signals.

[0011]

[0011] Combining the first image and the second image can be carried out by a suitable composite algorithm. For example, one or more processors may be configured to combine the first image and the second image by determining the mean of the first image and the second image, at least in part. For example, one or more processors may be configured to combine the first image and the second image by determining the arithmetic or geometric mean of the first image and the second image, at least in part. As an addition or alternative, one or more processors may be configured to combine the first image and the second image by determining a weighted average of the first image and the second image, at least in part. In some variations, such weighted averaging may include determining one or more composite coefficients for the first image and the second image, and the first image and the second image may be combined based on one or more composite coefficients.

[0012]

[0012] For example, in some variations, one or more processors may be configured to determine one or more composite coefficients by, at least partially, using at least one transform operator to transform a first image and a second image into a first transformed region image and a second transformed region image, determining one or more transformed region composite coefficients for the first transformed region image and the second transformed region image, and inversely transforming one or more transformed region composite coefficients to determine one or more composite coefficients for the first image and the second image. The transformed region composite coefficients may be determined, for example, by, at least partially, applying one or more coefficient composite rules (e.g., predetermined, heuristic-based, or learned rules) to the first transformed region image and the second transformed region image. The transform operator may include any preferred type of transform that supports one:1 forward and inverse transforms (e.g., Fourier transform, discrete wavelet transform (DWT), discrete cosine transform (DCT), or wave-atomic transform).

[0013]

[0013] In some variations, one or more processors may be configured, in addition or alternatively, to determine one or more composite coefficients by at least partially determining a first image quality coefficient map for a first image and a second image quality coefficient map for a second image, and by determining a first composite coefficient for the first image based on the first image quality coefficient map and a second composite coefficient for the second image based on the second image quality coefficient map.

[0014]

[0014] As an addition or alternative, in some variations, one or more processors may be configured to determine one or more composite coefficients by at least partially determining the local entropy of each pixel in a first image and the local entropy of each pixel in a second image, and determining one or more composite coefficients based on the determined local entropies.

[0015]

[0015] Another suitable technique for determining the composite coefficient includes, at least in part, estimating and weighting the image content by applying a linear filter (e.g., a difference of Gaussian filter) to each of the first image and the second image, determining one or more composite coefficients as a function of the imaging depth, and / or applying a saturation mask that reduces the weight (e.g., the composite coefficient) of at least a portion of the first image and / or the second image that exceeds a predetermined saturation threshold to determine one or more composite coefficients.

[0016]

[0016] In other words, one or more processors may be configured to synthesize images from different types of sensors within a hybrid transducer array using one or more suitable composite techniques described herein, including, for example, one or more of arithmetic averaging, geometric averaging, transform domain composition, image quality factor based (IQF) composition, local entropy weighting composition, image content weighting composition, depth-dependent weighting composition, or saturation masking.

Brief Description of the Drawings

[0017] [Figure 1] FIG. [Number] is a block diagram of an exemplary image composite system having a hybrid array. [Figure 2] FIG. [Number] is a block diagram of an exemplary image composite system having a hybrid array. [Figure 3] FIG. [Number] is a block diagram of an exemplary image composite system having a hybrid array. [Figure 4] FIG. [Number] is a block diagram of an exemplary image composite system having a hybrid array. [Figure 5] FIG. [Number] is a block diagram of an exemplary image composite system having a hybrid array. [Figure 6] FIG. [Number] is a flowchart of an exemplary method for performing image composition on an image collected by a hybrid array. [Figure 7] FIG. [Number] is a flowchart of an exemplary method for performing image composition on an image collected by a hybrid array. [Figure 8] A flowchart of an exemplary method for performing image composition on an image collected by a hybrid array. [Figure 9] A flowchart of an exemplary method for performing image composition on an image collected by a hybrid array. [Figure 10] A flowchart of an exemplary method for performing image composition on an image collected by a hybrid array. [Figure 11A] A diagram showing exemplary signals generated by a hybrid array and harmonic filtering of those signals. [Figure 11B] A diagram showing exemplary signals generated by a hybrid array and harmonic filtering of those signals. [Figure 11C] A diagram showing exemplary signals generated by a hybrid array and harmonic filtering of those signals. [Figure 11D] A diagram showing exemplary signals generated by a hybrid array and harmonic filtering of those signals. [Figure 11E] A diagram showing exemplary signals generated by a hybrid array and harmonic filtering of those signals. [Figure 12] A diagram showing a method for performing image composition on an image collected by a hybrid array. [Figure 13] A diagram showing a method for performing image composition on an image collected by a hybrid array.

DETAILED DESCRIPTION OF THE INVENTION

[0018]

[0030] Non-limiting examples of various aspects and variations of the present invention are described herein and shown in the accompanying drawings.

[0019]

[0031] Methods and devices for combining (e.g., synthesizing) images acquired using a mixed array comprising multiple types of array elements are described herein. The mixed array described herein comprises one or more array elements of a first type and one or more array elements of a second type distinct from the first type. One or more array elements of the first type may be used to form a first image, and one or more array elements of the second type may be used to form a second image. The first type may include non-optical transducers such as piezoelectric transducers, single-crystal material transducers, piezoelectric micromachine ultrasonic transducers (PMUTs), and / or capacitive micromachine ultrasonic transducers (CMUTs). The second type may include optical sensors, which may be optical resonators (e.g., whispering gallery-mode (WGM) optical resonators or photonic integrated circuit (PIC) optical resonators) or interference-based optical sensors such as optical interferometers. The optical sensors may have any preferred shape. For example, optical sensors may include microbubble resonators, microsphere resonators, microtoroid resonators, microring resonators, microbottle resonators, microcylinder resonators, and / or microdisk optical resonators. Compared to other types of ultrasonic sensors, optical sensors have high sensitivity and / or wide bandwidth in receiving ultrasonic signals.

[0020]

[0032] Various suitable combinations of non-optical transducers and one or more types of optical sensors may be included in a mixed transducer array. For example, in some variations, a first type of array element may include non-optical transducers, and a second type of array element may include optical sensors. One or more array elements of the first type may include non-optical transducers (non-optical sub-arrays) for transmitting acoustic signals and / or detecting acoustic echoes to form a first image. One or more array elements of the second type (e.g., optical sensors in an optical sub-array) may be used to detect acoustic echoes (e.g., full-spectrum, baseband, low-harmonic, super-harmonic, and / or differential harmonics) which can be used to form a second image. The second image, produced by a highly sensitive and / or wide-bandwidth optical sensor, may be used independently or may be combined with the first image to form an even more improved image. Due to the high sensitivity and wide bandwidth of optical resonators, images produced by optical sensors may have improved spatial resolution, improved contrast resolution, improved penetration depth, improved signal-to-noise ratio (SNR), improved tissue harmonic imaging, and / or improved Doppler sensitivity. However, since optical and non-optical subarrays have inherently different characteristics, composite images produced by combining images generated using signals from different types of sensors may have more features, better image quality, and provide a more complete understanding of the underlying imaging target.

[0021]

[0033] Furthermore, optical sensors do not generate ultrasound and are therefore used together in a mixed array with other transducers that do generate ultrasound (e.g., piezoelectric, CMUT, etc.). Mixed arrays can be arranged in various configurations and may include sensor elements with varying noise levels, amplitude responses, phase delays, frequency ranges, etc. Consequently, conventional beamforming methods and devices commonly used for probes with one type of sensor are not optimal for probes using mixed arrays of multiple types of sensors. The optical resonators described herein have an ultra-high quality coefficient (10 3 , 10 5 , 10 7 , 10 9Such optical resonators may have high quality coefficients (e.g., high quality coefficients) and therefore have ultra-high sensitivity for ultrasonic detection, but may have a smaller dynamic range. Such ultra-high quality coefficient optical resonators may be particularly suitable for ultra-depth imaging, but may suffer from undesirable nonlinear distortion in the near field. On the other hand, optical resonators may be designed to have lower quality coefficients, and therefore lower sensitivity, compared to optical resonators with ultra-high quality coefficients. Such low quality coefficient optical resonators may be particularly suitable for near-field imaging without undesirable nonlinear distortion. Furthermore, optical resonators may support many different resonance modes. Thus, the operating mode of an optical resonator may be switched from a first operating mode to a second operating mode, for example, by switching the wavelength of a laser source coupled to the optical resonator. In some variations, the imaging composite system may operate the optical resonator in an ultra-high quality coefficient operating mode at a first time and in a low quality coefficient operating mode at a second time. In some variations, the imaging composite system may operate a first set of optical resonators in an ultra-high quality coefficient operating mode and a second set of optical resonators in a low quality coefficient operating mode. Furthermore, subarrays consisting of different types of optical resonators can be deployed in the same image composite system used to produce different images that show different aspects of a target. Synthesizing images produced by different optical resonators, or by operating the optical resonators in different operating modes using composite algorithms such as those described herein, can produce, or otherwise produce, images with better image quality than images produced or generated by a single type of sensor.

[0022]

[0034] Therefore, in some variations, the second type of array element may include optical resonator sensors having different characteristics (e.g., different designs and / or different operating parameters). For example, in some variations, the second type of array element may include one or more high-quality coefficient (high-Q) optical resonators and one or more low-quality (low-Q) optical resonators. As an addition or alternative, the second type of array element may include one or more tunable optical resonators configured to operate as high-Q optical resonators and one or more tunable optical resonators configured to operate as low-Q optical resonators. For example, such tunable optical resonators may be selectively operable in high-Q mode or low-Q mode depending on the imaging settings, etc. As an addition or alternative, the second type of array element may include one or more optical resonator sensors designed for wide bandwidth and one or more optical resonator sensors designed for ultra-high sensitivity.

[0023]

[0035] Furthermore, in some variations, a mixed transducer array may include a combination of one or more non-optical transducers and multiple types of optical sensors. Therefore, to obtain a composite image of better quality than any individual input image, different types of input images (e.g., from non-optical transducers and / or from one or more different types of optical sensors) can be synthesized using image synthesis systems and methods such as those described herein.

[0024]

[0036] Image composite system Figure 1 is a block diagram of an exemplary image composite system 100 having a mixed array. The image composite system 100 includes a probe 125, an imaging system 160, and a display 170. The probe 125 can be operably coupled to the imaging system 160. The probe 125 can receive and / or transmit a set of signals (e.g., electrical signals, electromagnetic signals, optical signals, etc.) to and from the imaging system 160. The probe 125 includes a mixed array 110 that can receive and / or transmit a set of signals (e.g., acoustic signals, etc.) to and from a medium to be used in forming an image. The imaging system 160 may include a front end 140 and a back end 150, which can collectively determine the physical parameters (e.g., timing, location, angle, intensity, etc.) of the signals transmitted to the probe (e.g., via one or more transmit channels) and post-process the signals received by the probe 125 (e.g., via one or more receive channels) to form an image. The imaging system 160 may also be coupled to the display 170 to transmit a set of signals (e.g., electrical signals, electromagnetic signals, etc.) to the display 170. For example, in some variations, the display 170 may be configured to display (e.g., on a graphical user interface (GUI)) the images produced by the imaging system 160. As an addition or alternative, the imaging system 160 may receive signals from the display 170. For example, the display 170 may further include an interactive interface (e.g., a touchscreen, keyboard, motion sensor, etc.) for receiving commands from a user of the image composite system 100, such as to control the operation of the image composite system 100.

[0025]

[0037] As shown in Figure 1, the probe 125 may include a mixed array 110, a multiplexer 120, and an optical sensor cable 130. The mixed array 110 may include one or more non-optical array elements (e.g., PZT transducers, CMUT transducers, etc.) and one or more optical array elements (e.g., optical sensors such as WGM resonators). The non-optical transducers may be configured to transmit acoustic waves and, in some variations, may be configured to further receive and detect acoustic echoes in response to the transmitted acoustic waves. The optical sensors may be configured to receive and detect echo signals with high sensitivity and / or wide bandwidth response. In some variations, the mixed array may be similar to any of the mixed arrays described in International Patent Application PCT / US2021 / 033715, which is incorporated herein by this reference in its entirety. In some variations, the mixed array may be configured to perform harmonic imaging as described in International Patent Application PCT / US2021 / 039551, which is incorporated herein by this reference in its entirety. In some variations, the probe 125 may be configured to scan iteratively across the field of view by using a mixed array 110. In some variations, the signals from the mixed array may be synthesized by a synthetic aperture technique, such as the technique described in International Patent Application PCT / US2021 / 049226, which is incorporated herein by this reference in its entirety. Such signals may be used to generate images using optical sensors and / or non-optical transducers, as will be described in more detail below.

[0026]

[0038] The mixed array 110 may include an array of transducer elements and may be configured for operation in a one-dimensional (1D), 1.25-dimensional (1.25D), 1.5-dimensional (1.5D), 1.75-dimensional (1.75D), or two-dimensional (2D) array configuration. Generally, the dimensionality of an ultrasonic sensor array relates to the range of elevation beam width (or elevation beam slice thickness) achievable when imaging with the ultrasonic sensor array, and to how much control the system has over the elevation beam size, focus, and / or steering of the sensor array across the entire imaging field (e.g., across the entire imaging depth). A 1D array has only one row of elements in the elevation dimension and a fixed elevation aperture size. A 1.25D array has multiple rows of elements in the elevation dimension and a variable elevation aperture size, but has a fixed elevation focus via an acoustic lens. A 1.5D array has multiple rows of elements in the elevation dimension, a variable elevation aperture size, and a variable elevation focus via electronic delay control. A 1.75D array is a 1.5D array with additional elevation beam steering capability. A 2D array has a large number of elements in both the lateral and elevation dimensions to meet the minimum pitch requirements for large beam steering angles in both the lateral and elevation directions.

[0027]

[0039] In some variations, the image composite system can be configured to change a 1.5D array configuration or a 2D array configuration to a 1D array configuration. The hybrid array 110 can include a number (e.g., 16, 32, 64, 128, 256, 1024, 4096, 8192, 16384, etc.) of elements. In some variations, the hybrid array 110 can be arranged in a rectangular configuration and can include N×M elements, where N is the number of rows and M is the number of columns. In some variations, for example, the hybrid array 110 can include one or more array elements of a first type and one or more array elements of a second type, where the first type can be a piezoelectric transducer or other non-optical transducer configured to transmit ultrasonic waves, and the second type can be an optical sensor such as an optical resonator. The non-optical transducer and the optical sensor can be collectively arranged in a rectangular arrangement, a curved arrangement, a circular arrangement, or a sparse array arrangement.

[0028]

[0040] The non-optical transducers within the hybrid array 110 can include, for example, lead zirconate titanate (PZT) transducers, polymer thick film (PTF) sensors, polyvinylidene fluoride (PVDF) sensors, capacitive micromachine ultrasonic transducers (CMUTs), piezoelectric micromachine ultrasonic transducers (PMUTs), single crystal materials (e.g., LiNbO3 (LN), Pb(Mg 1 / 3 Nb 2 / 3 )-PbTiO3 (PMN-PT), and Pb(In 1 / 2 Nb 1 / 2 )-Pb(Mg 1 / 3 Nb 2 / 3 )-PbTiO3 (PIN-PMN-PT))-based transducers, and / or any transducer suitable for acoustic sensing.

[0029]

[0041] The optical sensor is, or may include, an interference-based optical sensor such as an optical interferometer or optical resonator (e.g., a whispering gallery mode (WGM) optical resonator). In the variant where the optical sensor is an optical resonator, the optical sensor may have any suitable shape or form (e.g., a microring resonator, a microsphere resonator, a microtroid resonator, a microbubble resonator, a fiber-based resonator, an integrated photonic resonator, a microdisk resonator, etc.). In some variants, the optical sensor may be / may include, for example, a Fabry-Perot (FP) resonator, a fiber-based resonator (e.g., a fiber ring resonator), a photonic crystal resonator, a waveguide resonator, or any other suitable optical resonator capable of localizing optical energy in space and time. For example, in some variations, an optical resonator may be similar to either of the optical resonators described in International Patent Applications PCT / US2020 / 064094 and PCT / US2021 / 022412, each of which is incorporated herein by this reference in its entirety.

[0030]

[0042] An optical resonator may include a closed loop of a transparent medium (e.g., glass, transparent polymer, silicon nitride, titanium dioxide, or any other material that is adequately optically transparent at the operating wavelength of the optical resonator) that allows several permissible frequencies of light to propagate continuously within the closed loop and accumulate the optical energy of the permissible frequencies of light within the closed loop. The foregoing is equivalent to saying that an optical resonator may allow the propagation of modes (e.g., whispering gallery modes (WGMs)) that travel along the surface of the optical resonator and correspond to permissible frequencies to circulate around the resonator. Each mode corresponds to the propagation of at least one frequency of light from the permissible frequency of light. The permissible frequency and quality factors of light for optical resonators described herein may be based at least in part on the geometric parameters of the optical resonator, the refractive index of the transparent medium, and the refractive index of the environment surrounding the optical resonator.

[0031]

[0043] The optical resonators described herein may have a set of resonant frequencies comprising a first subset of resonant frequencies and a second subset of resonant frequencies. In some modifications, the optical resonator may operate in the first subset of resonant frequencies having high-quality coefficients. Alternatively or additionally, in some modifications, the optical resonator may operate in the second subset of resonant frequencies having low-quality coefficients. The high-quality coefficient subset of resonant frequencies may be suitable for operation in high-sensitivity sensing probes (or subarrays), while the low-quality coefficient subset of resonant frequencies may be suitable for high-dynamic-range applications.

[0032]

[0044] In some variations, the sensitivity of an optical resonator can be controlled by tuning the geometric and / or characteristic material parameters of the optical resonator for the tuning of the quality coefficient of the optical resonator. In some variations, the space inside and / or around the optical resonator can be filled with an ultrasonically strengthening material, such as polyvinylidene fluoride, parylene, or polystyrene. The ultrasonically strengthening material can increase the sensitivity of the optical resonator.

[0033]

[0045] Optical resonators can be coupled to other components for receiving / transmitting light. In some implementations, an optical resonator can be operably coupled to a light source (e.g., a laser, a tunable laser, an erbium-doped fiber amplifier, etc.) and / or a photodetector (e.g., a p-doped / intrinsic / n-doped (PIN) diode) via an optical medium (e.g., optical fiber, tapered optical fiber, free-space medium, etc.). Acousto-optic systems based on optical resonators can directly measure ultrasound (e.g., ultrasonic echoes) by the photoelastic effect and / or physical deformation of the resonator in response to the ultrasound. Thus, an optical resonator can be considered a photoacoustic transducer capable of converting mechanical energy (e.g., acoustic energy) into optical energy. For example, in the presence of ultrasound (or any pressure wave), modes moving within the resonator may undergo spectral shifts or amplitude changes caused by changes in the refractive index and / or shape of the resonator. Spectral changes can be readily monitored and analyzed in the spectral region using a photodetector. Amplitude changes can also be detected by a photodetector. The photodetector ultimately converts the optical energy (i.e., optical signal) propagating through the optical resonator and optical fiber into electrical energy (i.e., electrical signal) suitable for processing using electronic circuits. Additional spatial and other information can be derived by monitoring and analyzing the optical response of the optical resonator between the mixed arrays. An exemplary mixed transducer array is described herein. Additionally or alternatively, the signal from the optical resonator may be processed by an optical circuit before being converted into electrical energy by the photodetector.

[0034]

[0046] The mixed array 110 may have one or more non-optical array elements (e.g., ultrasonic transducers or other non-optical sensors) and one or more optical array elements (e.g., optical resonators such as WGM optical resonators) arranged in various configurations (similar to any of the mixed arrays described above in U.S. Patent Application No. 63 / 029,044). For example, in some configurations, the non-optical array elements and optical array elements may be collectively arranged in a rectangular array having some rows and some columns. The rectangular array may contain N × M sensor elements, where N is the number of rows and M is the number of columns, both of which are integers. In some implementations, such as in the case of a 2D array, the number of rows and / or columns may be greater than 31 rows and / or 31 columns. For example, a 2D mixed array may contain 64 × 96 = 6,144 sensor elements.

[0035]

[0047] In some variations, the mixed array 110 may include multiple different types of optical sensors. For example, as will be further described below, the different types of optical sensors may include broadband optical resonators and ultra-high sensitivity optical resonators. As another example, the mixed array 110 may include one or more high-quality coefficient (high-Q) optical resonators and one or more low-quality (low-Q) optical resonators. As an addition or alternative, the mixed array 110 may include one or more tunable optical resonators configured to operate in different quality coefficient modes. For example, a tunable optical resonator may operate in a low-quality coefficient (low-Q) operating mode for high dynamic response or in a high-quality coefficient (high-Q) operating mode for high sensitivity response. In some implementations, the tunable optical resonators may be a first set of tunable optical resonators and a second set of tunable optical resonators that can operate in different operating modes, or may include them. In some implementations, a tunable optical resonator may operate in a high-Q operating mode in a first time interval and in a low-Q operating mode in a second time interval. In other words, in some variations, the mixed array 110 may include one or more tunable optical resonators configured to operate as high-Q optical resonators and / or one or more tunable optical resonators configured to operate as low-Q optical resonators. For example, such tunable optical resonators may be selectively operable in high-Q mode or low-Q mode depending on the imaging settings, etc.

[0036]

[0048] In some configurations, the spatial distribution of the positions of multiple array element types can be random. By using a sparse spatial distribution of array elements, the generation of grating lobes in images produced by mixed arrays can be reduced and / or prevented. The spatial distribution of the first type of array elements can be the same as, similar to, or different from, the spatial distribution of the second type of array elements. In some configurations, the spatial distribution of the positions of the first and second type of array elements can follow an arrangement pattern (e.g., they can be the same, shifted one cell to the right between sensor elements, or shifted two cells down between sensor elements). In some cases, one or more array elements of the second type can be smaller than or the same as one or more array elements of the first type.

[0037]

[0049] Non-optical transducers within the mixed array 110 may be operably coupled to a multiplexer 120 that processes electrical signals transmitted and / or received between the imaging system 160 and the non-optical transducers. Optical sensors within the mixed array 110 may be operably coupled to an optical sensor cable 130 that processes optical signals transmitted and / or received between the imaging system 160 and the optical sensors.

[0038]

[0050] The multiplexer 120 functions to selectively connect individual system channels to desired array elements. The multiplexer 120 may include analog switches. The analog switches may include a number of high-voltage analog switches. Each analog switch may be connected to an individual system channel. As a result, the multiplexer 120 can selectively connect individual system channels from a set of system channels of the imaging system 160 to desired transducer elements of the mixed array 110.

[0039]

[0051] The optical sensor cable 130 may include a dedicated optical path for transmitting and / or receiving optical signals to and from the optical sensor. The optical sensor cable 130 may include one or more optical waveguides, such as fiber optic cables. The characteristics of the optical sensor cable 130 may depend on the type of optical signal, the type of optical sensor, and / or the arrangement of the optical sensors. In some configurations, multiple optical sensors (e.g., an entire subarray of optical sensors, or any two or more optical sensors forming a part thereof) may be optically coupled to a single optical waveguide. Thus, signals from multiple optical sensors may be coupled to a single optical waveguide and communicated by that single optical waveguide. In some configurations, a subarray of optical sensors may be optically coupled to an array of optical waveguides in a 1:1 ratio (e.g., each optical sensor may be coupled to its own optical waveguide). Thus, optical signals from the subarray of optical sensors may be coupled to one or more optical waveguides in the optical sensor cable 130 and communicated to the imaging system 160 by those optical waveguides.

[0040]

[0052] The imaging system 160 may include a front-end 140 and a back-end 150. Generally, the front-end 140 interfaces with the probe 125 to generate an acoustic beam and receive electrical and / or optical signals. For example, the front-end 140 may drive non-optical transducers (e.g., transducers) in the probe to transmit ultrasonic signals in a predetermined beam pattern and may receive reflected ultrasonic signals from non-optical transducers and optical sensors in a mixed array within the probe. The front-end may also be tasked with performing both transmit beamforming and receive beamforming. The back-end 150 may include one or more processors for processing signals received from the mixed array 110 via the front-end to generate an image, memory operably coupled to the processors for storing the image, and / or a communication interface for presenting the image to the user (e.g., via a graphical user interface). For example, the back-end 150 may receive separately reconstructed images from the receive beamformer at the front-end, perform additional back-end processing, and perform image composite operations. Various backend processes, including digital signal processing (DSP), digital scan conversion (DSC), and envelope detection, may be involved in image formation. To perform image composites using optical sensors, the image composite system may include specific implementations of backend processes for storing, analyzing, synthesizing, and transmitting data, signals, and / or images. Such specific implementations are illustrated with reference to Figures 2-5 and described below.

[0041]

[0053] The display 170 may display a set of images generated by the imaging system 160. In some variations, the display 170 may, as an addition or alternative, include an interactive user interface (e.g., a touchscreen) and be configured to send a set of commands (e.g., pause, resume, etc.) to the imaging system 160. In some variations, the image composite system 100 may further include a set of one or more auxiliary devices (not shown) used to input information into or output information from the image composite system 100. The set of auxiliary devices may include, for example, a keyboard, mouse, monitor, webcam, microphone, touchscreen, printer, scanner, virtual reality (VR) head-mounted display, joystick, biometric reader, etc. (not shown).

[0042]

[0054] Figure 2 shows a block diagram of an exemplary image composite system 102 having a mixed array 110. As shown, the mixed array 110 may include a non-optical subarray 113 and an optical resonator subarray 114. The front end 140 may include a transmitter 142, a non-optical receiver 143, an optical resonator receiver 144, a transmitting beamformer 145, a non-optical receiving beamformer 146, and an optical resonator receiving beamformer 147. The back end 150 may include a non-optical back end processor 151 and an optical resonator back end processor 152. The non-optical back end processor 151 and the optical resonator back end processor 152 may perform one or more techniques, including digital signal processing (DSP), digital scan conversion (DSC), and envelope detection.

[0043]

[0055] The transmit beamformer 145 generates various transmit waveforms based on the transmit beamformer setting 181. The waveforms may be amplified by a transmitter 142, which may include analog, digital, and / or computer systems, before being applied to the non-optical subarray 113. After receiving the waveforms and / or the amplified waveforms by the transmitter 142, the non-optical subarray 113 may generate a set of acoustic waves (e.g., ultrasonic signals) toward the target. The acoustic waves irradiate the target with sound waves, which then reflect some of the acoustic waves (i.e., echo signals) toward the mixed array probe. The non-optical receiver 143 receives the echo signals detected by the non-optical transducers and processes them to produce digitized signals as outputs. Signals detected by the optical resonator subarray 114 may be processed and digitized by the optical resonator receiver 144. The non-optical receiving beamformer 146, the optical resonator receiving beamformer 147, the non-optical backend processor 151, and the optical resonator backend processor 152 use the signals processed by the two receivers to form the non-optical image 182 and the optical resonator image 183. The non-optical image 182 and the optical resonator image 183 often have different characteristics. The different characteristics of the non-optical image 182 and the optical resonator image 183 may depend on factors including the arrangement of the sensing elements (non-optical transducers or optical resonators) in the mixed array and the physical parameters of the sensing elements.

[0044]

[0056] Figure 3 shows a block diagram of an exemplary image composite system 103 having a mixed array 110 including optical resonator sensors, which include subarrays having different quality factors (Q factors). As shown, the mixed array 110 may include a non-optical subarray 113, a high-quality factor (high-Q) optical resonator subarray 115, and a low-quality factor (low-Q) optical resonator subarray 116. The front end 140 may include a transmitting beamformer 145, a transmitter 142, a high-Q optical resonator receiver 148 that receives signals from the high-Q optical resonator subarray, a low-Q optical resonator receiver 149 that receives signals from the low-Q optical resonator subarray, and an optical resonator receiving beamformer 147. Although separate optical resonator receivers (high-Q optical resonator receiver 148 and low-Q optical resonator receiver 149) are shown in Figure 3 to receive signals from the high-Q and low-Q optical resonators, respectively, it should be understood that in some variations, receivers 148 and 149 may be replaced by one or more receivers capable of receiving a wide range of Q-factor signals. For example, a single receiver may be dynamically tuned or otherwise configured to receive low-Q signals (e.g., in one or more "low-Q" modes) and tuned or otherwise configured to receive high-Q signals (e.g., in one or more "high-Q" modes). A single receiver may be dynamically configured across the spectrum of Q-factors or may be able to operate between different discrete modes corresponding to each range of the Q-factor. The backend 150 may include one or more optical resonator backend processors 152. The optical resonator backend processor 152 may perform one or more techniques, including digital signal processing (DSP), digital scan conversion (DSC), and envelope detection.

[0045]

[0057] The signals collected by the high-Q optical resonator subarray 115 can generate one or more high-sensitivity images 184, where features with lower reflectivity or weaker signals from deeper depths can be better visualized, and features with high reflectivity or strong signals from shallow depths can be saturated. On the other hand, the low-Q optical resonator subarray generates one or more high-dynamic-range images 185 that may miss smaller and lower reflectivity features or weaker signals from deeper depths. The one or more high-sensitivity images 184 and the one or more high-dynamic-range images 185 can be used in the optical resonator backend processor 152 to generate a composite image that incorporates the advantages of the signals from the high-Q and low-Q optical resonator subarrays, respectively.

[0046]

[0058] As shown in Figure 3, in some variants, the high-Q optical resonator subarray 115 and the low-Q optical resonator subarray 116 may share an optical resonator receiving beamformer 147 and an optical resonator backend processor 152. Alternatively, in some variants, the high-Q optical resonator subarray 115 and the low-Q optical resonator subarray 116 may have different receiving beamformers and / or different backend processors. For example, the high-Q optical resonator subarray 115 may be operably coupled to a high-Q optical resonator receiving beamformer (not shown) and a high-Q optical resonator backend process (not shown), and the low-Q optical resonator subarray 116 may be operably coupled to a low-Q optical resonator receiving beamformer (not shown) and a low-Q optical resonator backend process (not shown).

[0047]

[0059] In some variants, the front-end 140 may further include a non-optical receiver and a non-optical receiving beamformer (e.g., the non-optical receiver 143 and non-optical receiving beamformer 146 illustrated and described with respect to Figure 2). Consequently, the back-end 150 may also include a non-optical back-end processor, such as the non-optical back-end processor 151 that produces the non-optical image 182 illustrated and described with respect to Figure 2. Thus, the image composite system 103 may be configured to form a composite image based on the high-sensitivity image 184 and the high-dynamic-range image 185, and optionally further based on the non-optical image 182.

[0048]

[0060] Figure 4 shows a block diagram of an exemplary image composite system 104 having a mixed array 110, which is similar to the image composite system 103 described above and shown with respect to Figure 3, except that the mixed array 110 includes a tunable optical resonator subarray 117 capable of operating in two or more modes with different Q factor values. Tuning to different modes can be achieved, for example, by selectively changing the ambient temperature of the mixed array 110 and / or changing the optical wavelength. Such a tunable optical resonator subarray 117 can be used to acquire both high-sensitivity and high-dynamic-range images. For example, in some variants, at least one optical resonator in the tunable optical resonator subarray 117 may receive signals at multiple times in response to different sets of transmission sequences, where at least one optical resonator operates in a high-Q mode at one time and in a low-Q mode at another time. In other words, in some variations, at least a portion of the tunable optical resonator subarray 117 may operate in a first time interval and a second time interval that does not overlap with the first time interval, where at least a portion of the tunable optical resonator subarray 117 may operate as a high-Q optical resonator in the first time interval to generate a high-sensitivity image 184 and as a low-Q optical resonator in the second time interval to generate a high-dynamic-range image 185. In some variations, at least one tunable optical resonator may operate in high-Q mode before operating in low-Q mode. As an addition or alternative, at least one tunable optical resonator may operate in low-Q mode before operating in high-Q mode. At least two sets of transmission sequences may be performed to irradiate the target with sound waves multiple times in order to collect signals from both the high-Q optical resonator receiver 148 and the low-Q optical resonator receiver.

[0049]

[0061] As an addition or alternative, in some variations, at least a first portion (e.g., a first set) of the tunable optical resonator subarray 117 may be consistently designated to operate in high-Q mode, and at least a second portion (e.g., a second set) of the tunable optical resonator subarray 117 may be consistently designated to operate in low-Q mode. Signals from the first portion of the tunable optical resonators may be received by a high-Q optical resonator receiver 148, and signals from the second portion of the tunable optical resonators may be received by a low-Q optical resonator receiver 149. In some variations, where the tunable optical resonator subarray simultaneously includes several optical resonators tuned to operate in high-Q mode and several optical resonators tuned to operate in low-Q mode, a mixed array 110 may be functionally similar to the mixed array 110 shown with respect to Figure 3 and described above. With respect to Figure 3, as described above, separate optical resonator receivers (high-Q optical resonator receiver 148 and low-Q optical resonator receiver 149) are shown in Figure 4 as receiving high-Q and low-Q signals, respectively. However, it should be understood that in some variations, receivers 148 and 149 may be replaced by one or more receivers capable of receiving a wide range of Q-factor signals. For example, a single receiver may be dynamically tuned or otherwise configured to receive low-Q signals (e.g., in one or more "low-Q" modes) and tuned or otherwise configured to receive high-Q signals (e.g., in one or more "high-Q" modes). A single receiver may be dynamically configured across the spectrum of Q-factors or may be able to operate between different discrete modes corresponding to each range of the Q-factor.

[0050]

[0062] As shown in Figure 4, the mixed array 110 may include a non-optical subarray 113 and a tunable optical resonator subarray. The front end 140 may include a transmit beamformer 145, a transmitter 142, a high-Q optical resonator receiver 148, a low-Q optical resonator receiver 149, and an optical resonator receive beamformer 147. The non-optical subarray 113 in the mixed array 110 may transmit a set of acoustic signals, and the tunable optical resonator subarray may receive a set of acoustic echoes in response to the acoustic signals. The tunable optical resonator subarray 117 may be operably coupled to a photodetector configured to generate a first signal and a second signal, where the first signal includes a readout from at least a portion of the tunable optical resonator subarray 117 operating in high-Q mode, and the second signal includes a readout from at least a portion of the tunable optical resonator subarray 117 operating in low-Q mode. The high-Q optical resonator receiver 148 and the low-Q optical resonator receiver 149 can receive the first signal and the second signal, respectively. The backend 150 may include an optical resonator backend processor 152. The optical resonator backend processor 152 may perform operations on the first signal and the second signal, including digital signal processing (DSP), digital scan conversion (DSC), and envelope detection, in order to generate a high-sensitivity image 184 and a high-dynamic-range image 185. The backend 150 may be further configured to synthesize the high-sensitivity image 184 and the high-dynamic-range image 185 in order to generate a composite image that includes the advantages of the high-Q mode and low-Q mode signals of the tunable optical resonator subarray 117.

[0051]

[0063] In some variations, multiple transmission sequences are transmitted using a transmit beamformer setup 181, a transmit beamformer 145, a transmitter 142, and a non-optical subarray 113 to irradiate the target with sound waves multiple times. For example, the non-optical subarray 113 may transmit a first transmission sequence and a second transmission sequence. In response, a tunable optical resonator subarray 117 may collect a first signal in response to the first transmission sequence and a second signal in response to the second transmission sequence. The backend may then produce a first image from the first signal and a second image from the second signal.

[0052]

[0064] Figure 5 shows a block diagram of an exemplary image composite system 105 having a mixed array 110 that includes optical resonators in both a broadband subarray and a highly sensitive subarray. For example, the mixed array may include a non-optical subarray 113, a broadband optical resonator subarray 118, and a highly sensitive optical resonator subarray 119. The broadband optical resonator subarray 118 can capture out-of-baseband signals of transmitted acoustic waves, such as superharmonics and subharmonics from tissue and / or contrast agents (as described, for example, in International Patent Application No. PCT / US2021 / 039551, incorporated above by reference). The highly sensitive optical resonator subarray 119 can capture signals from deeper regions both inside and outside the baseband.

[0053]

[0065] A non-optical subarray 113 may be operably coupled to a transmitter 142, which is operably coupled to a transmitting beamformer 145 that receives a transmitting beamformer setting 181. The non-optical subarray 113 transmits an acoustic signal toward a target and receives an acoustic echo in response to the acoustic signal. The non-optical subarray 113 may be operably further coupled to a non-optical receiver 143 and a non-optical receiving beamformer 146 in the front end 140 to generate a first signal in response to the acoustic echo received in the non-optical subarray 113. A non-optical backend processor 151 may analyze the first signal to generate a first image (non-optical image 182) that visualizes the target with conventional spatial resolution and imaging depth. A wideband optical resonator subarray 118 and an ultra-high sensitivity optical resonator subarray 119 may be operably coupled to an optical resonator receiver 144 and an optical resonator receiving beamformer 147. The optical resonator backend processor 152 can process signals from the two optical resonator subarrays 118 and 119 to produce one or more images (e.g., a fundamental frequency image, an ultra-harmonic image, a low-harmonic image, etc.) and one or more high-sensitivity images. For example, a second signal from the broadband optical resonator subarray 118 may be used to generate a second image (harmonic image 186), and / or a third signal from the ultra-high-sensitivity optical resonator subarray 119 may be used to generate a third image (high-sensitivity image 184). Thus, the image composite system 105 can simultaneously achieve improvements in spatial resolution and imaging depth.

[0054]

[0066] After the first, second, and / or third images are separately generated using the first, second, and / or third signals from the non-optical subarray 113, the broadband optical resonator subarray 118, and the ultra-high sensitivity optical resonator subarray 119, respectively, an image composite algorithm may be used to synthesize the first, second, and / or third images to produce a composite image, as will be further described below.

[0055]

[0067] Method for performing image composite Figures 6 to 10, described below, illustrate exemplary methods for performing image composites based on images received from the mixed array described above. While the methods are described primarily with reference to optical resonator sensors, it should be understood that they can be similarly performed using signals from other types of optical sensors (e.g., optical interferometers). The method for performing image composites may be carried out by an image composite computing device, which is, for example, part of the backend 150 illustrated and described with respect to Figures 1 to 5, and / or operably coupled to an image composite system (such as the image composite system 100 illustrated and described with respect to Figure 1). The image composite computing device may include a set of electronic circuits, such as a processor, memory, and communication interfaces. The processor may include, for example, a hardware-based integrated circuit (IC) or any other suitable device for operating or executing a set of instructions / code. For example, a processor may include a general-purpose processor, a central processing unit (CPU), an accelerated processing unit (APU), an application-specific integrated circuit (ASIC), a microprocessor, a field-programmable gate array (FPGA) chip, a graphics processing unit (GPU), a digital signal processing (DSP) chip, and the like. Memory may store code that contains instructions for the processor to perform one or more processes or functions (e.g., signal filtering, signal amplification, phase matching, noise reduction, aperture selection, etc.). Memory may include, for example, a memory buffer, random access memory (RAM), read-only memory (ROM), a flash drive, a secure digital (SD) memory card, and the like. Communication interfaces may include a Universal Serial Bus (USB) interface, a Peripheral Component Interconnection Express (PCIe) interface, or hardware components that are operably coupled to the processor and / or memory and enable communication with components of image composite computing devices and image composite systems, and / or, in some variations, with external devices and / or a network of devices (e.g., the Internet).

[0056]

[0068] Image composite computing devices may include applications as software stored in memory and executed by a processor. For example, an application may include code that causes the processor to perform tasks such as aperture selection, signal analysis, and image generation. Alternatively, applications may be implemented on hardware-based devices. For example, an application may include digital or analog circuits that cause the image composite computing device to filter signals, amplify signals, and / or delay signals.

[0057]

[0069] Figure 6 is a flowchart of an exemplary method 600 for performing image composite on images acquired by a mixed array. In some implementations, the method may be performed using a composite imaging system 102 (e.g., backend 150) as illustrated and described with respect to Figure 2. Method 600 may include starting image acquisition (601) (e.g., upon receiving an instruction to start acquisition). Method 600 may further include transmitting a non-optical signal (602), then receiving a non-optical signal (603), and receiving an optical resonator signal (or other optical sensor signal) (604). The method may repeat 602, 603, and / or 604 until all transmission steps are performed in which it is desired to transmit acoustic signals from all non-optical array elements, and all reception steps are performed in which acoustic echoes are received from all non-optical and optical array elements of the mixed array 110. Once all desired transmissions and receptions have been performed for at least one desired composite image (605), Method 600 may further include generating or forming a non-optical image (606) and generating or forming an optical resonator image (607) using the front-end 140 and back-end 150 of the composite imaging system 102. The back-end 150 may then apply image region filters to the non-optical image and the optical resonator image (608, 609). The image region filters may be specifically designed according to the image characteristics of each type of image. Method 600 may also include combining the non-optical image and the optical resonator image (610) using a composite algorithm (e.g., one described below) and creating a composite image (611). In general, in some variations, the composite image may be formed using a dynamically determined weight mask, along with composite coefficients indicating, for example, which features of the non-optical image and which features of the optical resonator image may be included in each composite image.

[0058]

[0070] As an addition or alternative, in some variations, the composite image may be formed using a static weight mask that is predetermined and stored for use during the subsequent image composite process. For example, if the image composite method is content-independent of the image (e.g., Method 700) or static, the weight mask may be pre-calculated and stored in the memory of the image composite system. Performing the image composite method based on a pre-calculated weight mask may be processed faster and more efficiently by the processor of the image composite system. Figure 7 is a flowchart of an exemplary Method 700 that performs image composite on images collected by a mixed array, where the image composite utilizes a pre-calculated weight mask along with composite coefficients.

[0059]

[0071] Method 700 may include steps 601-607, as illustrated and described with respect to Figure 6. However, Method 700 may further include taking a pre-calculated weight mask (708). Method 700 may then perform a weighted average of the non-optical image and the optical resonator image to generate a composite image (709). The weighted average may include arithmetic averaging, geometric averaging, depth-dependent weighting, region-based weighting, etc. Method 700 may further include filtering the composite image (710) and producing a composite image (711).

[0060]

[0072] Figure 8 is a flowchart of an exemplary method 800 for performing image composite on images acquired by a mixed array. In some implementations, method 800 may be performed using a composite imaging system 103 as illustrated and described with respect to Figure 3. Method 800 may include starting image acquisition (801) (for example, upon receiving an instruction to start acquisition). Method 800 may further include transmitting a non-optical signal (802) and subsequently receiving high-quality coefficient (high-Q) optical resonator and / or low-quality coefficient (low-Q) optical resonator signals (803). Method 800 may repeat 802 and 803 until all transmission steps are performed where it is desired to transmit acoustic signals from all non-optical array elements and all reception steps are performed where acoustic echoes are received from all high-Q optical resonator array elements and low-Q optical resonator array elements. Once all desired transmissions and receptions have been performed for at least one desired composite image (804), Method 800 may further include generating or forming a high-Q optical resonator image (also called a high-sensitivity image) (805) and generating or forming a low-Q optical resonator image (also called a high-dynamic-range image) (806) using the front-end 140 and back-end 150 of the composite imaging system 103. The back-end 150 may then filter the high-Q optical resonator image (807) and the low-Q optical resonator image (808). Method 800 may also include combining the high-Q optical resonator image and the low-Q optical resonator image (809) (e.g., using a composite algorithm) and creating a composite image (810). Similar to Method 700, in some variations (e.g., when Method 800 is content-independent or static), the weight mask may be pre-calculated and stored in the memory of the image composite system 103 for faster processing.

[0061]

[0073] Figure 9 is a flowchart of an exemplary method for performing image composite on images acquired by a mixed array. In some implementations, method 900 may be performed using a composite imaging system 104 as illustrated and described with respect to Figure 4. Method 900 may include starting image acquisition (901) (for example, upon receiving an instruction to start acquisition). Method 900 may further include transmitting a non-optical signal (902) and subsequently receiving an optical resonator signal from at least one tuneable optical resonator signal operating in high-Q mode (903). In some cases, the optical resonator may be operated in a high-Q setting by selecting the optical wavelength (of the light source) to match a resonant frequency with a high resonant quality coefficient of resonance. Method 900 may further include transmitting a non-optical signal (904) and subsequently receiving an optical resonator signal from at least one tuneable optical resonator signal operating in low-Q mode (905). The flowchart in Figure 9 shows that a signal is received from the optical resonator in high-Q mode before a signal is received from the optical resonator in low-Q mode; however, it should be understood that, alternatively, a signal from the optical resonator in low-Q mode may be received before a signal from the optical resonator in high-Q mode. Method 900 may repeat steps 902-905 until all transmission steps in which it is desired to transmit acoustic signals from all non-optical array elements and all reception steps in which acoustic echoes are received in all Q optical resonator array elements in low-Q and high-Q settings have been performed. Once all desired transmissions and receptions have been performed for at least one desired composite image (906), Method 900 may further include generating or forming a high-Q optical resonator image (907) and generating or forming a low-Q optical resonator image (908) using the front-end 140 and back-end 150 of the composite imaging system 104. The back-end 150 may then filter the high-Q optical resonator image (909) and filter the low-Q optical resonator image (910). Method 900 may include (for example, using a composite algorithm) combining a high-Q optical resonator image and a low-Q optical resonator image (911) to produce a composite image (912).Similar to methods 700 and 800, in some variations (for example, when method 900 is static), the weight mask may be pre-calculated and stored in the memory of the image composite system 104 for faster processing.

[0062]

[0074] Figure 10 is a flowchart of an exemplary method for performing image composite on images acquired by a mixed array. In some implementations, method 1000 may be performed using a composite imaging system 105 as illustrated and described with respect to Figure 5. Method 1000 may include starting image acquisition (1001) (for example, upon receiving an instruction to start acquisition). Method 1000 may further include transmitting a non-optical signal (1002), then receiving a non-optical signal (1003), and receiving an optical resonator signal (1004) (for example, from a broadband optical resonator subarray and / or an ultra-high sensitivity optical resonator subarray). Method 1000 may repeat steps 1002-1004 until all transmission steps are performed where it is desired to transmit acoustic signals from all non-optical array elements, and all reception steps are performed where acoustic echoes are received in all non-optical array elements and optical resonator array elements. Once all desired transmit and receive steps have been performed for at least one desired composite image (1005), method 1000 may further include using the front end 140 and back end 150 of the composite imaging system 105 to generate or form a non-optical image (1006), generate or form a harmonic optical resonator image (1007), and generate or form a high-sensitivity optical resonator image (1008). The back end 150 may then filter the non-optical image (1009), filter the harmonic optical resonator image (1010), and filter the high-sensitivity optical resonator image (1011). Filtering the harmonic optical resonator image (i.e., the low-Q optical resonator image) may include performing a set of bandpass filters and / or a set of one-dimensional signal filters to extract components in the low-harmonic and / or super-harmonic bands. These filtered signals are then used to form a harmonic image in each of the selected bands. Method 1000 may include (1012) combining a non-optical image, a harmonic optical resonator image, and a high-sensitivity optical resonator image (1013) (for example, using a composite algorithm).

[0063]

[0075] As described above, when forming a harmonic optical resonator image, the optical resonator signal may be processed using a filter bank containing one or more filters. Figures 11A to 11E show exemplary signals generated by a mixed array and the harmonic filtering of those signals. As shown in Figure 11A, the first signal 1101 is received by a broadband optical resonator. By performing a transformation such as a Fourier transform, the first signal 1101 can be transformed from the time domain to the frequency domain 1111. As shown by the solid line in Figure 11B, the first signal mainly contains a baseband component around 6 MHz with a bandwidth of approximately 87% (or 5.22 MHz). However, the spectrum of the first signal reveals the presence of a second harmonic component of -25 dB and a third harmonic component of -35 dB in the first signal. The first signal also has additive 1 / f pink noise of -35 dB.

[0064]

[0076] Figures 11C to 11E illustrate the extraction of harmonic components using suitable filters. For example, as shown in Figure 11D, a 101-tap finite impulse response (FIR) second harmonic bandpass filter may be applied to the first signal 1101 to extract a filtered second harmonic signal 1102. Furthermore, a third harmonic bandpass filter (dotted line in the lower right panel) may be applied to the first signal 1101 to extract a filtered third harmonic signal 1103. In some cases, the time signal (signal in the time domain) may be normalized, and the second and third harmonic signals may be much weaker than the baseband signal. This is because tissue-generated ultraharmonic signals are typically (e.g., several orders of magnitude) lower than the baseband signal. Furthermore, higher frequency signals suffer greater loss within biological tissue. Without broadband sensors such as optical resonators described herein, as well as methods and apparatus for composite imaging based on signals generated by optical resonators, harmonic imaging can be extremely difficult to achieve.

[0065]

[0077] Combined algorithms This specification describes exemplary composite algorithms for synthesizing multiple images based on signals from non-optical array elements and / or optical resonator array elements. In some cases, n images of m dimensions (mD) are synthesized (by image composite) to produce a single mD image computed as output (n and m are integers). When m is 2, the mD images are sometimes called “images,” and when m is 3, they are sometimes called “volumes.” The composite algorithms described can be applied to both images and volumes. In general, in some variant forms, the composite algorithm may produce composite coefficients (e.g., factors) that characterize which or how much of each feature (e.g., pixel intensity) of each distinct image (e.g., non-optical image, optical resonator image) can contribute to each composite image. The composite coefficients may be described in a weighted mask that can be applied to the images to extract the desired features for contribution to the composite image.

[0066]

[0078] In some variations, the composite algorithm may be or include arithmetic averaging. The concept behind arithmetic averaging for composite imaging based on signals received from a mixed array is to synthesize n input images into a single output image using direct pixel-by-pixel arithmetic averaging of pixel values.

number

[0067]

[0079] In some variations, the composite algorithm may be geometric averaging or include it. Similar to the arithmetic averaging method described above, geometric averaging is also a pixel-by-pixel method performed as follows:

number

[0068]

[0080] In some variations, a composite algorithm may be a transformation-domain composite or include one. This is a class of composite methods that rely on transforming an input image into a transformation domain that supports one-to-one forward and inverse transformations. One-to-one transformations may include, for example, Fourier transform, discrete wavelet transform (DWT), discrete cosine transform (DCT), wave-atomic transform, etc. After the transformation, a heuristic-based rule or a set of learned rules may be applied to obtain composite coefficients in the transformation domain. Then, an inverse transform may be performed to transform the composite coefficients back into the image domain. An example of this process is shown in Figure 12. An input image 1202 (a non-optical image and / or an optical resonator image) may undergo a transformation 1204, and coefficients 1206 may be produced. Coefficient composite rule 1208 may be applied to these coefficients to produce composite coefficients 1210 in the transformation domain. Then, the composite coefficients may be inversely transformed to transform the composite coefficients back into the image domain for use in generating a composite image 1214 (1212).

[0069]

[0081] In some variant forms, the transformation region composite may use transformations suitable for multiscale image analysis such as DWT. Exemplary examples of coefficient composite rules in the context of DWT include: • For the smallest scale among multiple scales, take the smallest coefficient among all image coefficients (e.g., non-optical image, high-Q optical image, low-Q optical image, etc.). This rule assumes that the smallest scale mainly contains noise and therefore should be minimized. • For the largest scale among multiple scales, the average of the coefficients across all input images is taken. This rule assumes that the largest scale represents the general shape of the object and should be consistent across the input images. • For all other scales (except the smallest and largest scales) among multiple scales, the maximum value of the coefficient is taken from all input images. This rule assumes that all other scales represent some detail of the target, and that different input images may be best suited to represent one or more aspects. By taking the maximum value, all details can be preserved.

[0070]

[0082] However, when the DWT method is applied to method 1000 as illustrated and described with respect to Figure 10, larger weights may be assigned to the smaller-scale coefficients of the ultraharmonic image and the larger-scale coefficients of the non-optical image.

[0071]

[0083] As an addition or alternative, a set of coefficient compounding rules (rules that can be learned, for example, via a suitable machine learning algorithm) can be predefined for different ultrasound frequencies (e.g., as a lookup table, as a function of ultrasound frequency, etc.). For example, a first compounding coefficient (or a first range of compounding coefficients) may be associated with images generated using high ultrasound frequencies (or a range of high ultrasound frequencies), and a second compounding coefficient (or a second range of compounding coefficients) may be associated with images generated using low ultrasound frequencies (or a range of low ultrasound frequencies). In general, in some variations, higher ultrasound frequencies attenuate more in far-field imaging, so the compounding coefficients may decrease as the imaging depth increases, so that images generated using high ultrasound frequencies are given less weight when creating the compound image.

[0072]

[0084] In some variations, the composite algorithm may be or include an Image Quality Factor (IQF) based composite, as shown in Figure 13. An IQF can be defined as a quantitative measure of image quality and may be represented or otherwise characterized by an IQF map for the image, at least in part. There are various IQFs developed for various purposes and applications. For example, signal-to-noise ratio (SNR), entropy, detail resolution, contrast resolution, and penetration depth, each and / or any combination thereof, can be used as an IQF. Different IQFs enhance different aspects of the ultrasound image. In some cases, one or more IQFs 1304 may be extracted from the input image 1302. The IQFs 1304 are then converted into a composite coefficient 1306. Composite image I f (x)1308 is input image I j It can be calculated by a weighted sum of (x).

number

[0073]

[0085] In some variations, the composite algorithm may be a local entropy weighted composite or include one. A local entropy weighted composite synthesizes input images by assigning a weight to each pixel of each input image based on the information content in its neighborhood. This can be done by calculating the entropy of the region surrounding each pixel of each input image. The local entropy of a pixel at coordinate x in the j-th image can be calculated as follows:

number

number

[0074]

[0086] Instead of this specific example, H x,j Many functions can be used to convert to non-negative values. A composite image can be represented as follows:

number

[0075]

[0087] In some variant forms, the composite algorithm may be or include a fast image content weighted composite. Faster linear filtering-based algorithms may also be used as an approximation of local entropy-based weighting. Instead of calculating the local entropy of the input image, which can be computationally expensive, W j [x] is calculated by applying a Gaussian difference (DoG) filter to the j-th image. The same formula as for local entropy weighted composites can be used to generate the composite image.

[0076]

[0088] In some variations, the composite algorithm may be or include depth-dependent weighting composites. A given depth-dependent weighting may be useful when the input images have a clearly defined characteristic that is depth-dependent. Depth-dependent weighting composites can be particularly useful when the optical resonator subarray includes or operates as an ultra-sensitive optical resonator (e.g., as shown in Figures 3 and 4), since some input images may have better quality in shallower regions and others in deeper regions. Many depth-weighting functions may be used, including but not limited to linear and gamma functions.

[0077]

[0089] In some variations, the composite algorithm may be or include saturation masking. When some input images are prone to signal saturation (e.g., images produced by a high-Q optical resonator) or other types of nonlinearity due to excessive signal amplitude, a saturation masking step may be introduced to these input images before they undergo the composite method. Signal saturation can be detected by comparing the moving average of the beamformed images to a predetermined threshold. When saturation is detected, saturated pixels in the input image under inspection may be assigned weights of 0 or near 0 so that their contribution to the composite image is small and one or more other non-saturated input images become dominant.

[0078]

[0090] While image composite methods and systems for mixed arrays are described in the context of ultrasound imaging, in some variations, image composite methods and systems may be used in applications other than ultrasound imaging. For example, in some cases, image composite methods and systems may be used in applications such as computed tomography, magnetic resonance imaging, metrology, signal processing, particle physics, teledetection, and aerospace. The image composite methods disclosed herein may also be applied to synthesize images generated using different imaging modalities to form a fused image. For example, ultrasound, CT, and MRI images of the same region of a patient may be fused to show more diagnostic information.

[0079]

[0091] In some of the variations described above, the tunable optical resonator is described as operating in a low-quality coefficient (low-Q) or high-quality coefficient (high-Q) operating mode. However, generally, a tunable optical resonator can operate in multiple operating modes (e.g., three operating modes, ten operating modes, or 100 operating modes). For example, a tunable optical resonator may operate in a low-Q operating mode for generating a first image with a high linear range, a high-Q operating mode for generating a second image with high sensitivity, and a medium-quality coefficient operating mode for generating a third image with a balance between sensitivity and linear range. The backend of the image composite system 100 may be configured to synthesize the first, second, and third images to produce a composite image that is better (e.g., resolution, depth, contrast, quality coefficient, etc.) than each of the first, second, or third images individually.

[0080]

[0092] The above description uses specific terminology to provide a complete understanding of the invention for illustrative purposes. However, it will become apparent to those skilled in the art that specific details are not necessary to practice the invention. Accordingly, the above description of specific embodiments of the invention is presented for illustrative and explanatory purposes. They are not intended to be exhaustive or to limit the invention to the exact form disclosed, and obviously, given the above teachings, many modifications and variations are possible. The embodiments are selected and described to illustrate the principles of the invention and examples of its practical application, thereby enabling others skilled in the art to utilize the invention and its various embodiments with various modifications to suit specific intended uses. The following claims and their equivalents are intended to define the scope of the invention.

Claims

1. Receiving a first signal from one or more array elements of a first type in a mixed transducer array, Receiving a second signal from one or more array elements of a second type in the mixed transducer array, wherein at least one of the first type and the second type is an optical sensor. The first image is generated from the first signal, and the second image is generated from the second signal. To generate a composite image, the first image and the second image are combined. An imaging method, including the imaging method.

2. The method according to claim 1, wherein the first type and the second type are optical resonators having different characteristics.

3. The method according to claim 2, wherein the first type is a high-Q optical resonator and the second type is a low-Q optical resonator.

4. The method according to claim 2, wherein the first type is a tunable optical resonator that operates as a high-Q optical resonator, and the second type is a tunable optical resonator that operates as a low-Q optical resonator.

5. The method according to claim 1, wherein the first type is a non-optical transducer and the second type is an optical sensor.

6. The method according to claim 5, wherein the non-optical transducer is a piezoelectric transducer, a single-crystal material transducer, a piezoelectric micromachine ultrasonic transducer (PMUT), or a capacitive micromachine ultrasonic transducer (CMUT).

7. The method according to claim 5, wherein the second type is a broadband optical sensor, and the method further comprises receiving a third signal from one or more array elements of a third type, wherein the third type is an ultra-high sensitivity optical sensor.

8. The method according to claim 7, comprising filtering the first signal, the second signal, and / or the third signal using one or more filters.

9. The method according to claim 8, wherein the one or more filters include a harmonic bandpass filter.

10. The method according to claim 1, wherein combining the first image and the second image is a means of determining the average of the first image and the second image.

11. The method according to claim 10, wherein combining the first image and the second image includes determining the arithmetic or geometric mean of the first image and the second image.

12. The method according to claim 10, wherein combining the first image and the second image includes determining a weighted average of the first image and the second image.

13. The method according to claim 12, further comprising determining one or more composite coefficients for the first image and the second image, and combining the first image and the second image based on the one or more composite coefficients.

14. Determining one or more composite coefficients for the first image and the second image is Converting the first image and the second image into a first transformation region image and a second transformation region image using at least one transformation operator, Determining one or more composite conversion region coefficients for the first conversion region image and the second conversion region image, In order to determine the one or more composite coefficients for the first image and the second image, the one or more composite coefficients of the transformation region are inversely transformed. The method according to claim 13, including the method described in claim 13.

15. The method according to claim 14, wherein determining one or more composite conversion region coefficients for the first conversion region image and the second conversion region image includes applying one or more composite coefficient rules to the first conversion region image and the second conversion region image.

16. The method according to claim 14, wherein the at least one transform operator includes a Fourier transform, a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a wave-atom transform.

17. Determining one or more composite coefficients for the first image and the second image is Determining a first image quality coefficient map for the first image and a second image quality coefficient map for the second image, Based on the first image quality coefficient map, a first composite coefficient is determined for the first image, and based on the second image quality coefficient map, a second composite coefficient is determined for the second image. The method according to claim 13, including the method described in claim 13.

18. The method according to claim 13, wherein determining one or more composite coefficients for the first image and the second image includes determining the local entropy of each pixel in the first image and the second image, and determining one or more composite coefficients based on the determined local entropy.

19. The method according to claim 13, wherein determining one or more composite coefficients for the first image and the second image includes applying a linear filter to each of the first image and the second image.

20. The method according to claim 19, wherein the linear filter includes a Gaussian difference filter.

21. The method according to claim 13, wherein determining one or more composite coefficients for the first image and the second image includes determining one or more composite coefficients as a function of imaging depth.

22. The method according to claim 12, wherein determining a weighted average of the first image and the second image includes applying a saturation mask that reduces the weights of at least a portion of the first image and / or the second image that exceed a predetermined saturation threshold.

23. The method according to claim 1, wherein the optical sensor is a WGM optical resonator.

24. The method according to claim 1, wherein the optical sensor is a microbubble optical resonator, a photonic integrated circuit (PIC) optical resonator, a microsphere resonator, a microtoroid resonator, a microring resonator, a microbottle resonator, a microcylinder resonator, or a microdisk optical resonator.

25. One or more non-optical transducers in the mixed transducer array transmit an acoustic signal at the fundamental frequency f. The method according to claim 1, wherein one or more array elements of the first type, the second type, or both of the first and second types are configured to produce one or more optical responses when they receive harmonic or subharmonic acoustic echoes corresponding to the transmitted acoustic signal, and the one or more array elements of the second type have a bandwidth ranging from at least f / M to Nf, where M and N are integers greater than 1.

26. The one or more non-optical transducers have a first fundamental frequency f １ and the second fundamental frequency f ２ The method according to claim 1, wherein an acoustic signal is transmitted.

27. One or more array elements of the second type are one or more linear combinations nf １ +mf ２ It is configured to produce one or more optical responses when it receives an acoustic echo corresponding to the frequency of nf, where n and m are nf １ +mf ２ The method according to claim 26, wherein is an integer such that is a positive number.

28. The method according to claim 1, wherein at least one of the first image and the second image is a harmonic image.

29. The method according to claim 28, wherein the harmonic image is a low-harmonic image or a super-harmonic image.

30. A device for imaging a target, One or more array elements of a first type configured to receive a first signal, One or more array elements of a second type configured to receive a second signal A mixed transducer array comprising, wherein at least one of the first type and the second type is an optical sensor, The first image is generated from the first signal, and the second image is generated from the second signal. To generate a composite image, the first image and the second image are combined. One or more processors configured to perform the following: A device equipped with the following features.

31. The apparatus according to claim 30, wherein the first type and the second type are optical resonators having different characteristics.

32. The apparatus according to claim 31, wherein the first type is a high-Q optical resonator and the second type is a low-Q optical resonator.

33. The apparatus according to claim 31, wherein the first type is a tunable optical resonator that operates as a high-Q optical resonator, and the second type is a tunable optical resonator that operates as a low-Q optical resonator.

34. The apparatus according to claim 30, wherein the first type is a non-optical transducer and the second type is an optical sensor.

35. The apparatus according to claim 34, wherein the non-optical transducer is a piezoelectric transducer, a single-crystal material transducer, a piezoelectric micromachine ultrasonic transducer (PMUT), or a capacitive micromachine ultrasonic transducer (CMUT).

36. The apparatus according to claim 34, wherein the second type is a broadband optical sensor, and the mixed transducer array further comprises one or more array elements of a third type configured to receive a third signal, wherein the third type is an ultra-high sensitivity optical sensor.

37. The apparatus according to claim 36, wherein one or more processors are configured to filter the first signal, the second signal, and / or the third signal using one or more filters.

38. The apparatus according to claim 37, wherein one or more of the filters include a harmonic bandpass filter.

39. The apparatus according to claim 30, wherein one or more processors are configured to synthesize the first image and the second image by determining the average of the first image and the second image, at least in part.

40. The apparatus according to claim 39, wherein one or more processors are configured to synthesize the first image and the second image by determining the arithmetic or geometric mean of the first image and the second image, at least in part.

41. The apparatus according to claim 39, wherein one or more processors are configured to synthesize the first image and the second image by determining a weighted average of the first image and the second image, at least in part.

42. The apparatus according to claim 41, wherein one or more processors are configured to determine one or more composite coefficients for the first image and the second image, and to synthesize the first image and the second image based on the one or more composite coefficients.

43. The one or more processors are at least partially Converting the first image and the second image into a first transformation region image and a second transformation region image using at least one transformation operator, Determining one or more composite conversion region coefficients for the first conversion region image and the second conversion region image, In order to determine the one or more composite coefficients for the first image and the second image, the one or more composite coefficients of the transformation region are inversely transformed. The apparatus according to claim 42, configured to determine one or more composite coefficients for the first image and the second image by means of the above.

44. The apparatus according to claim 43, wherein one or more processors are configured to determine one or more composite transformation region coefficients for the first transformation region image and the second transformation region image by applying one or more composite coefficient rules to the first transformation region image and the second transformation region image, at least in part.

45. The apparatus according to claim 43, wherein the at least one transform operator includes a Fourier transform, a discrete wavelet transform (DWT), a discrete cosine transform (DCT), or a wave-atom transform.

46. The one or more processors are at least partially Determining a first image quality coefficient map for the first image and a second image quality coefficient map for the second image, Based on the first image quality coefficient map, a first composite coefficient is determined for the first image, and based on the second image quality coefficient map, a second composite coefficient is determined for the second image. The apparatus according to claim 42, configured to determine one or more composite coefficients for the first image and the second image by means of the above.

47. The apparatus according to claim 42, wherein the one or more processors are configured to determine one or more composite coefficients for the first image and the second image by at least partially determining the local entropy of each pixel in the first image and the second image, and determining one or more composite coefficients based on the determined local entropy.

48. The apparatus according to claim 42, wherein one or more processors are configured to determine one or more composite coefficients for the first image and the second image by applying a linear filter to each of the first image and the second image, at least in part.

49. The apparatus according to claim 48, wherein the linear filter includes a Gaussian difference filter.

50. The apparatus according to claim 42, wherein the one or more processors are configured to determine one or more composite coefficients for the first image and the second image by determining one or more composite coefficients as a function of imaging depth, at least in part.

51. The apparatus according to claim 41, wherein one or more processors are configured to determine a weighted average of the first image and the second image by applying a saturation mask that reduces the weights of at least a portion of the first image and / or the second image that are at least partially exceeding a predetermined saturation threshold.

52. The apparatus according to claim 30, wherein the optical sensor is a WGM optical resonator.

53. The apparatus according to claim 30, wherein the optical sensor is a microbubble optical resonator, a photonic integrated circuit (PIC) optical resonator, a microsphere resonator, a microtoroid resonator, a microring resonator, a microbottle resonator, a microcylinder resonator, or a microdisk optical resonator.

54. One or more non-optical transducers in the mixed transducer array transmit an acoustic signal at the fundamental frequency f. The apparatus according to claim 30, wherein one or more array elements of the first type, the second type, or both of the first and second types are configured to produce one or more optical responses when they receive harmonic or subharmonic acoustic echoes corresponding to the transmitted acoustic signal, and the one or more array elements of the second type have a bandwidth ranging from at least f / M to Nf, where M and N are integers greater than 1.

55. The one or more non-optical transducers have a first fundamental frequency f １ and the second fundamental frequency f ２ The apparatus according to claim 30, which transmits an acoustic signal.

56. One or more array elements of the second type create one or more optical responses when receiving an acoustic echo corresponding to a frequency of １ nf + mf ２ , where n and m are integers such that １ nf + mf ２ is a positive number, the apparatus according to claim 55.

57. The apparatus according to claim 30, wherein at least one of the first image and the second image is a harmonic image.

58. The apparatus according to claim 57, wherein the harmonic image is a low-harmonic image or a super-harmonic image.

Images