Panoramic image capture for multispectral sensors

Abstract

The image capture device may include: a first spectral filter and a second spectral filter arranged such that a panoramic image capture operation captures light filtered by the first spectral filter and light filtered by the second spectral filter in a same region of a combined image; and one or more processors to: capture a plurality of images based on a panoramic image capture operation; extracting first information and second information from the plurality of images, where the first information is associated with a first spectral filter and the second information is associated with a second spectral filter; identifying an association between first information and second information based on features captured in the plurality of images by the first spectral filter and the second spectral filter; and storing or providing information based on an association between the first information and the second information.

Description

[0001] This application is a divisional application of the invention patent application with application number 202080027943.7, application date February 10, 2020, and invention name “Panoramic image capture for multispectral sensors”.

[0002] Related Applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 804,593, filed on February 12, 2019, entitled “PANORAMIC IMAGE CAPTURE FOR MULTISPECTRAL SENSOR,” and U.S. Non-Provisional Patent Application No. 16 / 784,777, filed on February 7, 2020, entitled “PANORAMIC IMAGE CAPTURE FOR MULTISPECTRAL SENSOR,” which are expressly incorporated herein by reference. Background Art

[0004] An image capture device may include an image sensor and various components associated with the image sensor, such as a lens, an aperture, a light source, etc. An example of an image capture device is a user device, such as a smartphone, a tablet computer, etc. The image capture device may provide various image capture modes, such as a portrait mode, a macro mode, a panoramic mode, etc. Summary of the invention

[0005] According to some implementations, an image capture device may include: a first spectral filter and a second spectral filter, wherein the first spectral filter and the second spectral filter are arranged so that a panoramic image capture operation captures light filtered by the first spectral filter and light filtered by the second spectral filter in the same area of ​​a combined image; one or more memories; a monolithic image sensor; and one or more processors, which are communicatively coupled to the one or more memories, to: capture multiple images based on the panoramic image capture operation; extract first information and second information from the multiple images, wherein the first information is associated with the first spectral filter and the second information is associated with the second spectral filter; identify an association between the first information and the second information based on features captured in the multiple images by the first spectral filter and the second spectral filter; and store or provide information based on the association between the first information and the second information.

[0006] According to some implementations, a method may include: capturing multiple images based on a panoramic image capture operation by an image capture device having multiple first spectral filters and second spectral filters, wherein the multiple first spectral filters and the second spectral filters are arranged so that the multiple images include first image data and second image data, wherein the first image data is for a region of a combined image and is based on the multiple first spectral filters, and wherein the second image data is for a region of the combined image and is based on the second spectral filters; aligning the multiple images with each other to generate an aligned image by the image capture device; extracting invisible range information and visible range information from the aligned image by the image capture device, wherein the invisible range information is associated with an invisible spectral range and is based on the first image data, and wherein the visible range information is associated with a visible spectral range and is based on the second image data; combining the invisible range information and the visible range information to generate a combined image by the image capture device; and providing the combined image for display by the image capture device.

[0007] According to some implementations, an image capture device may include multiple first spectral filters and second spectral filters, wherein the multiple first spectral filters and the second spectral filters are arranged so that a panoramic image capture operation captures light filtered by the multiple first spectral filters and light filtered by the second spectral filters with respect to the same combined image object of the panoramic image capture operation; and an image sensor, wherein the multiple first spectral filters and the second spectral filters are placed on or adjacent to the image sensor, and wherein the multiple first spectral filters are associated with a first area of ​​the image sensor and the second spectral filters are associated with a second area of ​​the image sensor, and wherein the image capture device is capable of performing a snapshot image capture operation via the second spectral filters. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a schematic diagram of an example of an image capture device for multispectral imaging using a panoramic image capture operation, as described herein.

[0009] Figure 2 is a schematic diagram of an example of multispectral imaging using a panoramic image capture operation, as described herein.

[0010] Figure 3 is a schematic diagram of another example of multispectral imaging using a panoramic image capture operation, as described herein.

[0011] Figure 4 is a schematic diagram of an example environment in which the systems and / or methods described herein may be implemented.

[0012] Figure 5 yes Figure 3 A schematic diagram of example components of one or more devices.

[0013] Figure 6 and Figure 7 is a flow chart of an example process for multispectral imaging using a panoramic image capture operation. DETAILED DESCRIPTION

[0014] The following detailed description of example implementations refers to the accompanying drawings.The same reference numbers may identify the same or similar elements in different drawings.

[0015] Multispectral imaging can be used to capture image data within a specific wavelength range of the entire electromagnetic spectrum. This image data can be used for various purposes, such as chemical composition analysis of materials, moisture content determination, vegetation cover determination, plant health, plant nutrition, human health assessment, etc. In some cases, hyperspectral imaging can be performed, which may use more spectral bands and / or tighter spectral bands than multispectral imaging. However, for the purposes of the implementations described herein, "multispectral" and "hyperspectral" are used interchangeably.

[0016] A user device (e.g., a smartphone, a tablet computer, etc.) may include a camera. The camera may use an image sensor (such as a silicon-based sensor) to capture images in the visible light range (e.g., for user consumption, for use with an application of the user device, etc.). Providing multispectral imaging (and / or analysis of multispectral imaging) via a user device may be beneficial. For example, conventional multispectral imaging devices may be complex, bulky, and expensive, so multispectral imaging via a user device may provide functionality associated with multispectral imaging (e.g., chemical composition analysis, etc.) without the corresponding size and expense, thereby enabling consumers to use multispectral imaging. However, it is challenging to implement a separate sensor for multispectral imaging in a user device. For example, a separate sensor configured to detect non-visible range light for multispectral imaging operations (e.g., in addition to an image sensor associated with a camera of a user device) is expensive, power-intensive, and bulky.

[0017] Some implementations described herein provide multispectral imaging using an image sensor of an image capture device (e.g., a user device, such as a smart phone, a tablet computer, etc.) based on a panoramic image capture operation of the image capture device. For example, a panoramic image capture operation (sometimes referred to as a "panoramic mode" or "panorama mode") can use multiple images to generate a combined image, and these images are captured when the image capture device "sweeps" across a scene. The implementations described herein provide a set of spectral filters for the image capture device, some of which are for the visible spectrum and some of which are for the invisible spectrum. The spectral filters for the invisible spectrum can be provided in a first area as strips, bands, columns, strips, rectangles, etc. that can be perpendicular to the direction of motion of the panoramic image capture operation. For example, the longer axis (e.g., the main axis) of the strips, bands, columns, strips, rectangles, etc. can be perpendicular or substantially perpendicular (e.g., closer to perpendicular than parallel) to the vector defining the direction of motion. The spectral filters for the visible region can be provided in a second region of the image sensor that is different from the first region. Therefore, the multiple images can include images of the measurement target via one or more visible spectral filters and via one or more invisible spectral filters. In some cases, a single spectral filter may be referenced herein. It should be understood that reference to a single spectral filter also contemplates the use of multiple spectral filters, unless otherwise explicitly stated. For example, reference to a spectral filter may be understood to mean "one or more spectral filters".

[0018] The image capture device may combine the invisible spectrum image data of the measurement target and the visible spectrum image data of the measurement target (e.g., using panoramic imaging technology) so that a combined image may be generated, which indicates the visible spectrum data and the invisible spectrum data. In some implementations, the image capture device may analyze the invisible spectrum data to determine the invisible spectrum information that may be provided in association with the combined image, such as chemical composition information, etc. Therefore, multi-spectral imaging may be implemented using a panoramic image capture operation. For example, the image capture device may use a panoramic image capture operation to achieve multi-spectral imaging while still being able to capture snapshot images using a visible range spectral filter. By performing multi-spectral imaging using a spectral filter that is swept across the panoramic image when the panoramic image is captured by the image sensor, the cost and size of the multi-spectral imaging device are reduced compared to having an independent image sensor for multi-spectral sensing. For example, the implementation described herein may reduce the cost, size, weight, and power consumption of the image capture device relative to an independent image sensor, and may reduce reliance on additional hardware, moving parts, auxiliary electronic devices, and the like.

[0019] Figure 11 is a schematic diagram of an example 100 of an image capture device for multispectral imaging using a panoramic image capture operation, as described herein. As indicated by reference numeral 110, example 100 includes an image capture device associated with an image sensor. For example, the image capture device may include a user device, and the image sensor may be associated with a camera. In some implementations, the image capture device may be a single camera system. In such a case, the image sensor may be a monolithic image sensor (e.g., using a monolithic silicon chip). In a single camera sensor, a single image sensor (e.g., a monolithic image sensor or another type of image sensor) may be paired with a lens system (e.g., a combination of lens elements along an optical axis).

[0020] As shown in reference numeral 120, example 100 includes a color spectrum filter. In some cases, the color spectrum filter can be used for light in the visible range. In some implementations, the color filter can be a Bayer filter, a red, green, blue (RGB) filter, a visible range filter in the range of about 380nm to 780nm, etc. As used herein, the visible range refers to a range of about 380nm to 780nm. The tiles of the color filter can be arranged in any way, and the color filter can use any combination of filter tiles. In some implementations, the color spectrum filter can be associated with an infrared cutoff filter (IRCF). The IRCF can filter out infrared wavelengths so that the color values ​​of the visible range are not distorted by infrared wavelengths. In some implementations, the color filter can be associated with a snapshot capability. For example, an image capture device can be able to capture a snapshot image via a color filter without performing a panoramic image capture operation. As used herein, a "snapshot" can refer to an image captured without sweeping the image capture device over the object of the image and having a multi-pixel width and a multi-pixel height.

[0021] As shown in reference numeral 130, spectral filters may be provided at the edge of the image sensor. Here, a plurality of spectral filters are provided at the left edge of the image sensor, some of which extend to a certain extent along the left edge. In addition, another set of spectral filters is provided at the right edge of the image sensor. This may mean that during multispectral imaging using a panoramic image capture operation, both the color spectral filter and the spectral filter shown in reference numeral 130 are swept over the measurement target, as shown in combination with Figure 2Described in more detail. The spectral filter shown in reference numeral 130 may correspond to a single pixel width or a multi-pixel width in a narrow direction. The spectral filter shown in reference numeral 130 may extend from a first side of the image sensor to a second side of the image sensor, or may extend from the first side to the second side to a certain extent (as shown on the left side of the image capture device). In some implementations, the spectral filter shown in reference numeral 130 may be provided in a central portion of the image sensor (e.g., along an edge on the image sensor or elsewhere). In some implementations, the spectral filter shown in reference numeral 130 may be arranged along an edge of the image capture device that is perpendicular to the direction of travel of the panoramic image capture operation. In some implementations, the spectral filter shown in reference numeral 130 may be arranged in a band perpendicular to the direction of travel of the panoramic image capture operation.

[0022] In some implementations, any number of spectral filters may be provided for the image sensor. For example, a single spectral filter may be provided (e.g., to enable measurement of a single spectral band), or multiple spectral filters may be provided (e.g., to enable measurement of a range of spectral bands). In some implementations, the spectral filter shown in reference numeral 130 may be used for non-visible spectral bands (e.g., near infrared bands, ultraviolet bands, etc.). For example, the spectral band shown in reference numeral 130 may pass light in the ultraviolet range or near infrared range. In some implementations, the spectral filter shown in reference numeral 130 may be associated with a spectral range different from the ultraviolet or near infrared range (such as a mid-infrared range, an infrared range, a specific portion of the visible range, etc.).

[0023] In some implementations, the spectral filter can be located within the color spectral filter (e.g., rather than at the edge of the image sensor and / or the color spectral filter) shown in reference numeral 120. For example, one or more pixels in the middle of the color spectral filter can be filtered to receive non-visible spectral bands, which can improve the flexibility of multi-spectral imaging of the image capture device.

[0024] In some implementations, the spectral filter (e.g., for the non-visible range) shown in reference numeral 130 may be a bandpass filter, a blocking filter, a long-wave pass filter, a short-wave pass filter, a dichroic filter, etc. In some implementations, the spectral filter shown in reference numeral 130 may represent an area implemented as lacking a filter. For example, a near-infrared blocking filter overlapping with a color spectral filter may not cover the area shown in reference numeral 130, thereby allowing near-infrared light to pass through the image sensor in the area shown in reference numeral 130. In some implementations, the spectral filter (e.g., shown by reference numeral 120 or 130) may be provided on the image sensor or adjacent to the image sensor (e.g., above, on top, sandwiched, etc.).

[0025] In some implementations, the IRCF is used to block unwanted infrared light from interfering with the color pixel operation of the image sensor, which would otherwise distort the color values, as described above. However, the IRCF may affect the operation of the spectral filters shown in reference numeral 130 because these spectral filters will allow near-infrared wavelengths to pass. In this case, a thin film filter that blocks unwanted wavelengths (e.g., infrared wavelengths in the color spectral filter area, color wavelengths in the multi-spectral area, etc.) can be used instead of the IRCF. Therefore, standard color imaging (e.g., for user consumption) as well as multi-spectral imaging can be supported using the implementations described herein. In some implementations, a patterned thin film IRCF can be applied in the area of ​​the color spectral filter shown in reference numeral 120 before placing the spectral filter shown in reference numeral 130. For example, the patterned thin film IRCF may not overlap with the spectral filter shown in reference numeral 130.

[0026] In this way, the image capture device can allow for full transmission windows for color imaging and for all wavelengths desired for multispectral imaging. In addition, by using spectral filters in bands or columns at the edge of the image sensor, the cost, size and complexity of the multispectral imaging device are reduced.

[0027] As mentioned above, Figure 1 are provided only as one or more examples. Other examples may be related to Figure 1 The examples described are different.

[0028] Figure 2 2 is a schematic diagram of an example 200 of multispectral imaging using a panoramic image capture operation, as described herein. As shown, Figure 2 An image capture device 210 (e.g., an image capture device associated with a camera system such as a single camera system) and a measurement target 220 are included. The measurement target 220 may correspond to any object for which a multispectral image is to be captured. The measurement target 220 is shown to illustrate how to image a specific area of ​​a combined image of a panoramic image capture operation using visible range light and non-visible range light. The direction of travel of the panoramic image capture operation is shown by reference numeral 230.

[0029] The spectral filters of the image sensor of the image capture device 210 are shown by reference numerals 240, 250, and 260. Here, the visible range spectral filter (e.g., color filter) is shown by reference numeral 250, and the invisible range spectral filter (e.g., near infrared filter, ultraviolet filter, etc.) is shown by reference numerals 240 and 260. For clarity, the invisible range spectral filters are referred to as filter A and filter B. In some implementations, filter A can include multiple different filters. In some implementations, filter B can include multiple different filters. In some implementations, only one of (multiple) filter A and (multiple) filter B can be included in the image capture device 210.

[0030] The position of the measurement target 220 is shown with respect to multiple images (e.g., Image 1, Image 2, and Image 3) of a panoramic image capture operation. As indicated by reference numeral 270, in Image 1, light from the measurement target 220 is filtered by filter B. Therefore, the image data regarding the measurement target 220 in Image 1 may represent the spectral range of filter B. As indicated by reference numeral 280, in Image 2, light from the measurement target 220 is filtered by a color filter. Therefore, the image data regarding the measurement target 220 in Image 2 may represent the visible light range of the color filter. As indicated by reference numeral 290, in Image 3, light from the measurement target 220 is filtered by filter A. Therefore, the image data regarding the measurement target 220 in Image 3 may represent the spectral range of filter A. In this manner, an image of the measurement target 220 is captured through each of filters A, B, and the color filter. As part of the panoramic image capture operation, the image capture device may combine these images to generate a combined image, such as a combination of filters A and B. Figure 3 Described in more detail.

[0031] As mentioned above, Figure 2 are provided only as one or more examples. Other examples may be related to Figure 2 The examples described are different.

[0032] Figure 3 is a schematic diagram of another example 300 of multispectral imaging using panoramic image capture operations, as described herein. The operations of example 300 may be performed by an image capture device (e.g., image capture device 210). In this example, the image capture device may include a combination of Figure 1 and Figure 2 The visible range filter and the invisible range filter are described so that multispectral imaging using a panoramic image capture operation can be performed. Figure 3As shown and as indicated by reference numeral 310, as part of a panoramic image capture operation, the image capture device can capture multiple images. Here, the multiple images depict fruit. In some implementations, the multiple images can partially overlap each other, such as in combination with Figure 2 For example, the image capture device may capture partially overlapping images so that both visible range data and invisible range data are captured for the overlapping regions of the images.

[0033] As shown at reference numeral 320, the image capture device may identify features of the plurality of images. For example, the plurality of images may include non-visible range (e.g., multispectral) image data and visible range image data. Features may include, for example, geometric features, people, faces, objects, etc. The image capture device may identify features so that the plurality of images can be aligned and combined so that the non-visible range image data and the visible range image data can be combined.

[0034] As shown in reference numeral 330, the image capture device can determine edge boundaries of multiple images based on features. For example, when a first image and a second image are combined to form a combined image, the image capture device can determine where the edge of the first image is located in the second image. The image capture device can determine edge boundaries based on features. For example, the image capture device can identify specific features in two images, and can identify the relative position of the edge boundaries of the two images based on the specific features. In some implementations, different methods can be used to determine edge boundaries and / or combine images, such as any method associated with panoramic image capture technology.

[0035] As shown in reference numeral 340, the image capture device can align multiple images with each other based on features and / or edge boundaries. For example, the image capture device can determine the corresponding coordinates of a specific feature in each of the multiple images, and can align the multiple images with each other based on the corresponding coordinates of the specific feature. As another example, the image capture device can determine the corresponding coordinates of the edge boundaries of the multiple images, and can align the multiple images with each other based on the corresponding coordinates of the edge boundaries. Thus, the image capture device enables the visible range information and the invisible range information of the multiple images to be combined, as described in more detail below. For example, returning to reference Figure 2 When the images 1, 2, and 3 are combined, the combined image may include spectral information corresponding to the filter A, the color filter, and the filter B used to measure the target 220.

[0036] In some implementations, the image capture device can extract information from multiple images without generating a combined image based on multiple images. For example, the image capture device can identify the association between invisible range information (e.g., first information) and visible range information (e.g., second information) based on the features captured in multiple images via the first spectral filter and the second spectral filter. In such a case, the image capture device can store, provide and / or display information based on the association between the first information and the second information. For example, the image capture device can provide an image based on visible range information for display, and can provide information based on invisible range information corresponding to a specific area of ​​the image. In such a case, the image capture device can receive an interaction with an area of ​​the image based on the visible range information, and can provide invisible range information associated with the area (e.g., chemical composition information, a visual representation of the chemical composition of the area of ​​the image, etc.). In this way, the image capture device can save processor resources that would otherwise be used to generate a combined image.

[0037] In some implementations, the image capture device may identify two or more selected images from a plurality of images that are to be combined to form a combined image. For example, the image capture device may identify the two or more selected images based on image analysis (e.g., based on clarity, brightness, focus, etc.). As another example, the image capture device may identify the two or more selected images based on edge boundaries of the two or more selected images. For example, the image capture device may identify an image subset of a plurality of images that has sufficient overlap so that a combined image may be formed that includes spectral content from each spectral filter of the image capture device. In some implementations, the image capture device may determine that one or more of the plurality of images will not be used to form a combined image (e.g., because data for the one or more images is already included in other overlapping images of the plurality of images), and may thereby save processing resources that would otherwise be used to combine a larger subset of images including the one or more images.

[0038] As shown in reference numeral 350, the image capture device may determine the invisible range information based on the aligned images and based on the spectral filter. For example, the image capture device may determine the invisible range information based on the invisible range image data. In some implementations, the image capture device may perform an analysis of the invisible range image data to determine the invisible range information. For example, the image capture device may determine the chemical composition of the object of the plurality of images based on the invisible range image data. Figure 3 In the embodiment of the present invention, an image capture device determines the sugar content of the fruit shown in the plurality of images.

[0039] In some implementations, the image capture device can apply a model to the invisible range image data to determine the invisible range information. The model can receive the invisible range image data as input and can output chemical composition information as the invisible range information. In some implementations, the model can be trained based on an algorithm, such as a machine learning algorithm, etc. For example, the model can be trained using a training set (e.g., a training set, a validation set, and / or a test set) of invisible range image data and corresponding chemical composition. In some implementations, the image capture device can apply the model locally. For example, the image capture device can store the model, and the model can be used to perform analysis of the invisible range image data. In some implementations, the image capture device can provide the invisible range image data to another device, and the other device can use the model to perform analysis of the invisible range image data. Applying the model locally on the image capture device can reduce the time delay associated with generating a combined image, which can support augmented reality applications, etc. Applying the model at the server device can save the battery and / or processor resources of the image capture device, and can enable the application of a more complex model than a model that would be feasible or efficient on the image capture device.

[0040] As shown in reference numeral 360, the image capture device can combine the invisible range information and the visible range information in the combined image, and as shown in reference numeral 370, the image capture device can provide the combined image for display. In some implementations, the image capture device can modify the combined image based on the invisible range information. For example, as shown in reference numeral 380, in this case, the image capture device superimposes an indication of the sugar content of each fruit (e.g., determined based on the chemical composition, which is determined using the above-mentioned invisible range information). In some implementations, the image capture device can superimpose other information, such as information indicating the maturity of fruits or vegetables, water depth, vegetation coverage, chemical composition of food, etc. Here, the superimposed information uses numbers to indicate the sugar content. However, the superimposed information can take any form, such as color, hatching, labels, etc. Therefore, the combined image of the visible range that the user can understand can be modified based on the invisible range information, thereby providing the invisible range information in a user-friendly interface, and saving image capture device resources (e.g., processing resources, display resources, battery resources, etc.), otherwise these resources will be used to provide a less intuitive interface or multiple different interfaces for the invisible range information.

[0041] In some implementations, the image capture device may use augmented reality technology to provide a combined image. For example, the image capture device may overlay invisible range information on the combined image. In some implementations, the image capture device may modify the combined image based on user interaction. For example, the image capture device may receive interaction with a specific portion of the combined image and may provide additional information about the specific portion. Figure 3 In the example of FIG. 1 , if the image capture device receives an interaction with one of the fruits, the image capture device may provide additional information about the chemical composition of the fruit (e.g., a more specific sugar content value, maturity, etc.). In this way, the image capture device may provide a user interface for digging deeper into the invisible range information and adjusting the combined image accordingly, thereby saving processor resources and / or battery that would otherwise be used to capture and generate another combined image to present adjustments to the combined image.

[0042] Thus, multispectral imaging can be implemented using a panoramic image capture operation. By performing multispectral imaging using a spectral filter that is swept across a panoramic image as the panoramic image is captured by an image sensor, the cost and size of a multispectral imaging device is reduced compared to having a separate image sensor for multispectral sensing. For example, implementations described herein can reduce the cost, size, weight, and power consumption of an image capture device relative to a separate image sensor, and can reduce reliance on additional hardware, moving parts, auxiliary electronics, and the like.

[0043] As mentioned above, Figure 3 are provided only as one or more examples. Other examples may be related to Figure 3 The examples described are different.

[0044] Figure 4 4 is a schematic diagram of an example environment 400 in which the systems and / or methods described herein may be implemented. Figure 4 As shown, environment 400 may include image capture device 410 including image sensor 420, server device 430, and network 440. Devices of environment 400 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections.

[0045] The image capture device 410 includes one or more devices capable of receiving, generating, storing, processing, and / or providing information associated with a panoramic image capture operation. For example, the image capture device 410 may include a communication and / or computing device, such as a smartphone, a tablet computer, a handheld computer, a wearable communication device (e.g., a smart watch, a pair of smart glasses, etc.), or a similar type of device. In some implementations, the image capture device 410 may be associated with a single camera system, such as including one or more image sensors 420 that are collectively aligned on an optical axis with a lens system.

[0046] The image capture device 410 includes an image sensor 420. The image sensor 420 includes a device capable of performing measurements of light directed toward the image sensor 420 (e.g., via one or more spectral filters). For example, the image sensor 420 can perform sensor measurements of light directed toward the image sensor 420. The image sensor 420 can use one or more sensor technologies, such as complementary metal oxide semiconductor (CMOS) technology, charge coupled device (CCD) technology, etc. The image sensor 420 can include a plurality of sensor elements (e.g., an array of sensor elements—referred to as a sensor array), each of which is configured to acquire information. Some sensor elements can be associated with a visible light range (e.g., and / or a visible light spectrum filter), and other sensor elements can be associated with an invisible light range (e.g., and / or an invisible light spectrum filter). In some implementations, the image sensor 420 can include a single image sensor, such as a monolithic image sensor.

[0047] Server device 430 includes one or more devices capable of storing, processing, and / or routing information associated with multispectral imaging using image capture device 410. In some implementations, server device 430 may include a communication interface that allows server device 430 to receive information from and / or send information to other devices in environment 400.

[0048] The network 440 includes one or more wired and / or wireless networks. For example, the network 440 may include a cellular network (e.g., a long term evolution (LTE) network, a code division multiple access (CDMA) network, a 4G network, a 5G network, another type of next generation network, etc.), a public land mobile network (PLMN), a local area network (LAN), a wide area network (WAN), a metropolitan area network (MAN), a telephone network (e.g., a public switched telephone network (PSTN)), a private network, an ad hoc network, an intranet, the Internet, a fiber-based network, a cloud computing network, etc., and / or a combination of these or other types of networks.

[0049] Figure 4 The number and arrangement of devices and networks shown are provided as examples. Figure 4 There may be additional devices and / or networks, fewer devices and / or networks, different devices and / or networks, or devices and / or networks arranged in a different manner than those shown in FIG. Figure 4 Two or more of the devices shown may be implemented in a single device, or Figure 4The single device shown may be implemented as multiple distributed devices. Additionally or alternatively, one set of devices (eg, one or more devices) of environment 400 may perform one or more functions described as being performed by another set of devices of environment 400.

[0050] Figure 5 5 is an illustration of example components of device 500. Device 500 may correspond to image capture device 410 and / or server device 430. In some implementations, image capture device 410 and / or server device 430 may include one or more devices 500 and / or one or more components of device 500. Figure 5 As shown, device 500 may include a bus 510 , a processor 520 , a memory 530 , a storage component 540 , an input component 550 , an output component 560 , and a communication interface 570 .

[0051] The bus 510 includes components that permit communication between multiple components of the device 500. The processor 520 is implemented in hardware, firmware, and / or a combination of hardware and software. The processor 520 is a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a microprocessor, a microcontroller, a digital signal processor (DSP), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), or another type of processing component. In some implementations, the processor 520 includes one or more processors that can be programmed to perform functions. The memory 530 includes a random access memory (RAM), a read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, and / or optical memory) that stores information and / or instructions for use by the processor 520.

[0052] Storage component 540 stores information and / or software related to the operation and use of device 500. For example, storage component 540 may include a hard disk (e.g., a magnetic disk, an optical disk, and / or a magneto-optical disk), a solid-state drive (SSD), a compact disk (CD), a digital versatile disk (DVD), a floppy disk, a cassette, a magnetic tape, and / or another type of non-transitory computer-readable medium, and a corresponding drive.

[0053] Input components 550 include components that permit device 500 to receive information, such as via user input (e.g., a touch screen display, a keyboard, a keypad, a mouse, buttons, switches, and / or a microphone). Additionally or alternatively, input components 550 may include components for determining location (e.g., a global positioning system (GPS) component) and / or sensors (e.g., an accelerometer, a gyroscope, an actuator, another type of positioning sensor or an environmental sensor, etc.). Output components 560 include components that provide output information from device 500 (via, for example, a display, a speaker, a tactile feedback component, an audio or visual indicator, etc.).

[0054] The communication interface 570 includes a transceiver-like component (e.g., a transceiver, a separate receiver, a separate transmitter, etc.) that enables the device 500 to communicate with other devices, such as via a wired connection, a wireless connection, or a combination of a wired connection and a wireless connection. The communication interface 570 can allow the device 500 to receive information from another device and / or provide information to another device. For example, the communication interface 570 can include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi interface, a cellular network interface, etc.

[0055] Device 500 can perform one or more processes described herein. Device 500 can perform these processes based on processor 520 executing software instructions stored by non-transient computer-readable media such as memory 530 and / or storage component 540. As used herein, the term "computer-readable medium" refers to a non-transient storage device. The memory device includes memory space within a single physical storage device or memory space spread across multiple physical storage devices.

[0056] The software instructions may be read into the memory 530 and / or storage component 540 from another computer-readable medium or from another device via the communication interface 570. When executed, the software instructions stored in the memory 530 and / or storage component 540 may cause the processor 520 to perform one or more processes described herein. Additionally or alternatively, hardware circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Therefore, the implementations described herein are not limited to any specific combination of hardware circuitry and software.

[0057] Figure 5 The number and arrangement of components shown are provided as examples. Figure 5Device 500 may include additional components, fewer components, different components, or components arranged in a different manner than the components shown. Additionally or alternatively, one set of components (e.g., one or more components) of device 500 may perform one or more functions described as being performed by another set of components of device 500.

[0058] Figure 6 is a flow chart of an example process 600 for multispectral imaging using a panoramic image capture operation. In some implementations, Figure 6 One or more process blocks of may be performed by an image capture device (e.g., image capture device 410). In some implementations, Figure 6 One or more process blocks of may be performed by another device or a group of devices separate from or including the image capture device, such as an image sensor (eg, image sensor 420), a server device (eg, server device 430), and the like.

[0059] The image capture device may include a first spectral filter and a second spectral filter. The first spectral filter and the second spectral filter may be arranged so that the panoramic image capture operation captures light filtered by the first spectral filter and light filtered by the second spectral filter in the same area of ​​the combined image.

[0060] like Figure 6 As shown, process 600 may include capturing multiple images based on a panoramic image capture operation (block 610). For example, an image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, input component 550, etc.) may capture multiple images based on a panoramic image capture operation, as described above. In some implementations, the image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, etc.) may identify features of the multiple images, as described above. In some implementations, the image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, etc.) may align the multiple images with each other based on the features to generate an aligned image, as described above.

[0061] like Figure 6 As further shown, process 600 may include extracting first information and second information from a plurality of images, wherein the first information is associated with a first spectral filter and the second information is associated with a second spectral filter (block 620). For example, an image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, etc.) may extract first information (e.g., invisible range information) and second information (e.g., visible range information) from a plurality of images, as described above. In some implementations, the first information is associated with a first spectral filter and the second information is associated with a second spectral filter.

[0062] like Figure 6 As further shown in FIG. 6 , process 600 may include identifying an association between the first information and the second information based on features captured in the plurality of images via the first spectral filter and the second spectral filter (block 630). For example, an image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, etc.) may identify an association between the first information and the second information based on features captured in the plurality of images via the first spectral filter and the second spectral filter.

[0063] like Figure 6 As further shown, process 600 may include storing or providing information based on the association between the first information and the second information (block 640). For example, an image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, input component 550, output component 560, and communication interface 570, etc.) may store or provide information based on the association between the first information and the second information, as described above.

[0064] Process 600 may include additional implementations, such as any single implementation or any combination of implementations described below and / or described in conjunction with one or more other processes described elsewhere herein.

[0065] In some implementations, the first spectral filter is arranged along an edge of the image capture device, which is perpendicular to the direction of travel of the panoramic image capture operation. In some implementations, the first spectral filter includes a plurality of spectral filters. In some implementations, the plurality of spectral filters are used for multi-spectral spectral imaging or hyperspectral spectral imaging. In some implementations, the first information is invisible range information associated with an invisible spectral range, and the second information is visible range information associated with a visible spectral range. In some implementations, the image capture device may determine chemical composition information based on the first information, and provide an indication of the chemical composition information for display in association with a combined image based on the first information and the second information. In some implementations, the image capture device may modify the combined image based on the indication of the chemical composition information. In some implementations, the first spectral filter includes a bandpass filter or a group of bandpass filters, and the second spectral filter includes a color filter or a group of color filters.

[0066] Although Figure 6 Example blocks of process 600 are shown, but in some implementations, Figure 6 Process 600 may include additional blocks, fewer blocks, different blocks, or blocks arranged in a different manner than the blocks shown in . Additionally or alternatively, two or more blocks of process 600 may be performed in parallel.

[0067] Figure 7 is a flow chart of an example process 700 for multispectral imaging using a panoramic image capture operation. In some implementations, Figure 7 One or more process blocks of may be performed by an image capture device (e.g., image capture device 410). In some implementations, Figure 7 One or more process blocks of may be performed by another device or a group of devices separate from or including the image capture device, such as an image sensor (eg, image sensor 420), a server device (eg, server device 430), and the like.

[0068] like Figure 7 As shown, process 700 may include: capturing multiple images based on a panoramic image capture operation by an image capture device having multiple first spectral filters and second spectral filters (box 710). For example, an image capture device (e.g., using an image sensor 420, a processor 520, a memory 530, a storage component 540, an input component 550, etc.) may capture multiple images based on a panoramic image capture operation. The image capture device may have multiple first spectral filters and second spectral filters. Multiple first spectral filters and second spectral filters may be arranged so that multiple images include first image data and second image data. The first image data may be for a region of a combined image and may be based on multiple first spectral filters. The second image data may be for a region of a combined image and may be based on a second spectral filter.

[0069] like Figure 7 As further shown, process 700 may include aligning the plurality of images with one another to generate an aligned image (block 720). For example, the image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, etc.) may align the plurality of images with one another based on the features to generate an aligned image, as described above.

[0070] like Figure 7 As further shown, process 700 may include extracting invisible range information and visible range information from the aligned image (block 730). For example, the image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, etc.) may extract invisible range information and visible range information from the aligned image, as described above. In some implementations, the invisible range information is associated with an invisible spectral range and is based on the first image data. In some implementations, the visible range information is associated with a visible spectral range and is based on the second image data.

[0071] like Figure 7As further shown, the process 700 may include combining the invisible range information and the visible range information to generate a combined image (block 740). For example, the image capture device (e.g., using the image sensor 420, the processor 520, the memory 530, the storage component 540, etc.) may combine the invisible range information and the visible range information to generate a combined image, as described above.

[0072] like Figure 7 As further shown, process 700 may include providing the combined image for display (block 750). For example, an image capture device (e.g., using image sensor 420, processor 520, memory 530, storage component 540, input component 550, output component 560, and communication interface 570, etc.) may provide the combined image for display, as described above.

[0073] Process 700 may include additional implementations, such as any single implementation or any combination of implementations described below and / or described in conjunction with one or more other processes described elsewhere herein.

[0074] In some implementations, the plurality of first spectral filters are arranged in a band perpendicular to the direction of travel of the panoramic image capture operation. In some implementations, the image capture device may determine the chemical composition information based on the invisible range information. In some implementations, the image capture device may provide an indication of the chemical composition information in association with the combined image for display. In some implementations, the image capture device may use augmented reality technology to provide an indication of the chemical composition information. In some implementations, the first spectral filter of the plurality of first spectral filters extends to a certain extent along the edge of the image sensor.

[0075] In some implementations, the plurality of first spectral filters are associated with respective spectral ranges for multispectral spectral imaging or hyperspectral spectral imaging. In some implementations, the image capture device may align the plurality of images based on respective features or edge boundaries of the plurality of images.

[0076] Although Figure 7 Example blocks of process 700 are shown, but in some implementations, Figure 7 Process 700 may include additional blocks, fewer blocks, different blocks, or blocks arranged in a different manner than shown. Additionally or alternatively, two or more blocks of process 700 may be performed in parallel.

[0077] The above disclosure provides illustration and description, but is not intended to be exhaustive or to limit the implementation to the precise form disclosed. Modifications and variations may be made in light of the above disclosure, and may also be acquired from practice of the implementation.

[0078] As used herein, the term "component" is intended to be broadly interpreted as hardware, firmware, and / or a combination of hardware and software.

[0079] Some user interfaces have been described in this article and / or are shown in the figures.User interface may include graphical user interface, non-graphical user interface, text-based user interface etc.User interface may provide information for display.In some implementations, the user may interact with the information, such as by providing input via the input assembly of equipment, and the equipment provides the user interface for display.In some implementations, the user interface may be configurable by equipment and / or user (for example, the user may change the size of the user interface, the information provided via the user interface, the position of the information provided by the user interface etc.).Additionally or alternatively, the user interface may be preconfigured to a standard configuration, based on the specific configuration of the type of the equipment showing the user interface, and / or based on the ability associated with the equipment displayed thereon with the user interface and / or a group of configurations of a specification.

[0080] It will be apparent that the systems and / or methods described herein can be implemented in various forms of hardware, firmware, or a combination of hardware and software. The actual dedicated control hardware or software code used to implement these systems and / or methods does not limit the implementation. Therefore, the operation and behavior of the systems and / or methods are described herein without reference to specific software code - it should be understood that software and hardware can be designed to implement the systems and / or methods described herein.

[0081] Although particular combinations of features are recited in the claims and / or disclosed in the specification, these combinations are not intended to limit the disclosure of the various implementations. In fact, many of these features may be combined in ways not specifically recited in the claims and / or disclosed in the specification. Although each of the attached dependent claims may be directly dependent on only one claim, the disclosure of the various implementations includes the combination of each dependent claim with every other claim in the claim set.

[0082] Any element, action or instruction used herein should not be interpreted as key or necessary, unless clearly stated. In addition, as used herein, the articles "one" and "an" are intended to include one or more projects, and can be used interchangeably with "one or more". Further, as used herein, the term "set" is intended to include one or more projects (e.g., related projects, unrelated projects, combinations of related projects and unrelated projects, etc.), and can be used interchangeably with "one or more". Where only one project is intended, the phrase "only one" or similar language is used. In addition, as used herein, the terms "has", "have", "having" etc. are intended to be open terms. In addition, unless otherwise clearly stated, the phrase "based on" is intended to mean "based at least in part on".

Claims

1. A method, include: identifying, by a device, features of a plurality of images, the plurality of images comprising non-visible range image data and visible range image data; determining, by the device, edge boundaries of the plurality of images based on the features; aligning, by the device, two or more images of the plurality of images based on the edge boundary; as well as The device extracts invisible range information from the two or more images based on the invisible range image data and based on aligning the two or more images without generating a combined image associated with the multiple images or before generating the combined image.

2. The method according to claim 1, further comprising: include: The plurality of images are captured as part of a panoramic image capture operation. 3 . The method of claim 2 , wherein the device comprises a set of spectral filters provided perpendicularly to a direction of motion of the panoramic image capture operation.

4. The method of claim 1, wherein the features include one or more of the following: Geometric features, people, face, or object.

5. The method according to claim 1, wherein the feature is identified include: Specific ones of the features in two of the two or more images are identified.

6. The method of claim 1, wherein determining the edge boundary include: Based on a particular one of the features in two of the two or more images, a relative position of one or more of the edge boundaries of the two images is identified.

7. The method of claim 1, wherein aligning the plurality of images include: determining corresponding coordinates of the edge boundaries; as well as The two or more images are aligned with each other based on the corresponding coordinates of the edge boundaries.

8. The method according to claim 1, further comprising: include: identifying a subset of the plurality of images having sufficient overlap to enable the combined image to be formed to include spectral content from each spectral filter of the device, Wherein the subset of images includes the two or more images.

9. The method according to claim 1, further comprising: include: Based on data of one or more images of the plurality of images being included in the two or more images, it is determined that the one or more images are not to be used in forming the combined image.

10. A device, include: one or more memories; as well as One or more processors, coupled to the one or more memories, the one or more processors configured to: identifying features of multiple images; determining edge boundaries of the plurality of images based on the features; aligning two or more images of the plurality of images based on the edge boundary; as well as After aligning the two or more images, and without generating a combined image associated with the plurality of images, or before generating the combined image, invisible range information is determined from the two or more images.

11. The device of claim 10, wherein the one or more processors are further configured to: The plurality of images are captured as part of a panoramic image capture operation.

12. The apparatus of claim 10, wherein the one or more processors are configured to: Specific ones of the features in two of the two or more images are identified.

13. The apparatus of claim 10, wherein the one or more processors are configured to: Based on a particular one of the features in two of the two or more images, a relative position of one or more of the edge boundaries of the two images is identified.

14. The apparatus of claim 10, wherein the one or more processors are configured to align the plurality of images by: determining corresponding coordinates of the edge boundaries; and The two or more images are aligned with each other based on the corresponding coordinates of the edge boundaries.

15. The device of claim 10, wherein the one or more processors are further configured to: identifying a subset of the plurality of images having sufficient overlap to enable the combined image to be formed to include spectral content from each spectral filter of the device, Wherein the subset of images includes the two or more images.

16. A non-transitory computer readable medium storing a set of instructions, the set of instructions include: One or more instructions that, when executed by one or more processors of a device, cause the device to: identifying a plurality of images including non-visible range image data and visible range image data; determining edge boundaries of the plurality of images; aligning two or more images of the plurality of images based on the edge boundary; as well as Without generating a combined image associated with the plurality of images or before generating the combined image, invisible range information is extracted from the two or more images based on the invisible range image data and based on the two or more images being aligned.

17. The non-transitory computer readable medium of claim 16, wherein the one or more instructions further cause the device to: The plurality of images are captured as part of a panoramic image capture operation.

18. The non-transitory computer readable medium of claim 17, wherein the device comprises a set of spectral filters provided perpendicular to a direction of motion of the panoramic image capture operation.

19. The non-transitory computer readable medium of claim 16, wherein the one or more instructions that cause the device to align the plurality of images cause the device to: determining corresponding coordinates of the edge boundaries; and The two or more images are aligned with each other based on the corresponding coordinates of the edge boundaries.

20. The non-transitory computer readable medium of claim 16, wherein the one or more instructions further cause the device to: identifying a subset of the plurality of images having sufficient overlap to enable the combined image to be formed to include spectral content from each spectral filter of the device, Wherein the subset of images includes the two or more images.

21. An image capturing device, include: a plurality of first spectral filters for the non-visible spectrum and provided as one or more shapes at a first edge of an image sensor of said image capture device, wherein one or more longer axes of the one or more shapes are perpendicular or substantially perpendicular to a vector defining a direction of motion of a panoramic image capture operation, wherein a first filter of the plurality of first spectral filters extends along a first portion of the first edge, and wherein a second filter of the plurality of first spectral filters extends along a second portion of the first edge; as well as one or more second spectral filters for the invisible spectrum and provided at a second edge of the image sensor, The second edge is located on a side of the image sensor opposite to the first edge.

22. The image capture device of claim 21, wherein the one or more shapes include one or more of: a bar, a strip, a column, a strip, or a rectangle.

23. The image capture device of claim 21, wherein the image sensor utilizes complementary metal oxide semiconductor (CMOS) technology or charge coupled device (CCD) technology.

24. The image capture device of claim 21 , wherein the plurality of first spectral filters comprises one or more of: Bandpass filter, Blocking filter, Long wave pass filter, Short pass filter, or Dichroic filter.

25. The image capture device of claim 21, further comprising: include: Lens system, Wherein the image sensor is one of one or more image sensors commonly aligned with the lens system on an optical axis.

26. The image capture device of claim 21, wherein the image capture device is a smartphone or a tablet computer.

27. The image capture device of claim 21, wherein the image sensor is a monolithic image sensor.

28. The image capture device of claim 21, further comprising: include: A color spectral filter in the area of ​​the image sensor.

29. The image capture device of claim 21, wherein the region of the image sensor is between the plurality of first spectral filters and the one or more second spectral filters.

30. The image capture device of claim 28, further comprising: include: a thin film filter configured to block infrared wavelengths in the region of the image sensor, The thin film filter is different from an infrared cut filter (IRCF).

31. The image capture device of claim 21 , wherein the color spectral filter is one or more of: Bayer filter, Red, Green, Blue (RGB) filter, or Visible range filters essentially in the range of 380nm to 780nm.

32. The image capture device of claim 21, wherein the invisible light spectrum is a near infrared band or an ultraviolet band.

33. A device, include: a plurality of first spectral filters located in a first band or first column at a first edge of an image sensor of the device, wherein the longer axis or axes of the first strips or the first columns are perpendicular to a vector defining a direction of motion of a panoramic image capture operation, wherein a first filter of the plurality of first spectral filters extends along a first portion of the first edge, and wherein a second filter of the plurality of first spectral filters extends along a second portion of the first edge; as well as One or more second spectral filters are located in a second band or second column at a second edge of the image sensor.

34. The apparatus of claim 33, wherein the second edge is located on an opposite side of the image sensor from the first edge.

35. The apparatus of claim 33, wherein the image sensor utilizes complementary metal oxide semiconductor (CMOS) technology or charge coupled device (CCD) technology.

36. The apparatus of claim 33, wherein the plurality of first spectral filters comprises one or more of: Bandpass filter, Blocking filter, Long wave pass filter, Short pass filter, or Dichroic filter.

37. An image sensor, include: a plurality of first spectral filters for the non-visible spectrum and provided as one or more shapes in a first region of the image sensor, wherein one or more longer axes of the one or more shapes are perpendicular or substantially perpendicular to a direction of motion of a panoramic image capture operation, wherein the first area is a first edge of the image sensor, wherein a first filter of the plurality of first spectral filters extends along a first portion of the first edge, and wherein a second filter of the plurality of first spectral filters extends along a second portion of the first edge; as well as One or more second spectral filters for the visible spectrum and provided in a second region of the image sensor.

38. The image sensor according to claim 37, wherein the second area is a second edge of the image sensor, and The second edge is located on a side of the image sensor opposite to the first edge.

39. The image sensor of claim 37, wherein the one or more shapes are a plurality of stripes.

40. The image sensor according to claim 37, further comprising: include: one or more third spectral filters located in a third region of the image sensor, The second area is located between the first area and the third area.

Images