Imaging systems and methods

Abstract

A system and method for extracting full-range hyperspectral data from one or more RGB images. The method includes preprocessing the one or more RGB images. Furthermore, the method includes estimating an illumination component associated with each preprocessed RGB image. The method then includes removing the illumination component from each preprocessed RGB image. Additionally, the method includes tracking the trajectory of one or more pixels across one or more frames associated with each preprocessed RGB image. The method then results in identifying the position of one or more pixels in one or more neighboring frames of the one or more frames based on blocks defined around the one or more pixels. Subsequently, the method includes extracting full-range hyperspectral data from each preprocessed RGB image based on at least one of the illumination component removal, the trajectory of the one or more pixels, and the position of the one or more pixels.

Description

Technical Field

[0001] This invention generally relates to the field of image processing, and more specifically to systems and methods for using imaging systems to extract full-range hyperspectral data from one or more RGB images to have at least one of one or more spectral and quantitative information to improve the accuracy of various applications in agriculture, health and other related fields. Background Technology

[0002] The following description of the prior art is intended to provide background information relevant to the field of this disclosure. This section may include certain aspects of the art that may be related to various features of this disclosure. However, it should be understood that this section is intended only to enhance the reader's understanding of this disclosure and is not an admission of prior art.

[0003] With the advancement of digital technology, image processing technology and imaging systems have also been greatly enhanced. Like many complex systems, camera phones / devices are the result of converged and enabling technologies. A camera phone is a smart / feature phone / device capable of taking photos and frequently recording videos using one or more built-in digital cameras, and can also send the generated images / videos via telephone. The main advantages of such devices are low cost, compact design, and the ability to use a touchscreen to focus the camera on a specific object in the field of view, providing even inexperienced users with a degree of focus control that experienced photographers would need to achieve with manual focus. Compared to camera phones / devices, consumer cameras in mobile phones require significantly less power and necessitate a higher level of camera electronics integration to achieve miniaturization. "Smart device or smart computing device or user equipment (UE) or user equipment" refers to any electrical, electronic, electromechanical computing device or device, or a combination of one or more of the above. In addition, a “smartphone” or “feature phone” is a “smart computing device” that refers to a mobile wireless cellular connection device that allows end users to use services on a cellular network, such as, but not limited to, 2G, 3G, 4G, 5G and / or similar mobile broadband network connections with an advanced mobile operating system that combines the functionality of a personal computer operating system with other features useful for mobile or handheld use.

[0004] Furthermore, today's widely deployed wireless networks, providing various communication services such as voice, video, data, advertising, content, messaging, and broadcasting, typically have multiple access networks, supporting multi-user communication by sharing available network resources. An example of such networks is the Evolved Universal Terrestrial Radio Access Network (E-UTRA), a radio access network standard designed to replace the UMTS and HSDPA / HSUPA technologies specified in 3GPP Release 5 and later. Unlike HSPA, LTE's E-UTRA is a completely new air interface system, unrelated to and incompatible with W-CDMA. It offers higher data rates, lower latency, and is optimized for packet data. Earlier UTRAN was the Radio Access Network (RAN), defined as part of the Universal Mobile Telecommunications System (UMTS), a third-generation (3G) mobile phone technology supported by the 3rd Generation Partnership Project (3GPP). UMTS is the successor to GSM (Global System for Mobile Communications) technology and currently supports various air interface standards, such as Wideband Code Division Multiple Access (W-CDMA), Time Division Code Division Multiple Access (TD-CDMA), and Time Division Synchronous Code Division Multiple Access (TD-SCDMA). UMTS also supports enhanced 3G data communication protocols, such as High-Speed ​​Packet Access (HSPA), which provides higher data transmission speeds and capacity for associated UMTS networks. As the demand for mobile data and voice access continues to increase, research and development is constantly driving technological advancements to not only meet the growing access demands but also improve and enhance the user experience of user equipment. Developed from technologies such as GSM / EDGE, UMTS / HSPA, CDMA2000 / EV-DO, and TD-SCDMA radio interfaces with 3GPP Release 8, e-UTRA aims to provide a single evolution path to improve data speeds and spectral efficiency while allowing for more functionalities.

[0005] 3GPP also introduced a new technology, NB-IoT, in Release 13. This technology can meet the needs of low-end IoT applications. It has been working to address the IoT market by completing the standardization of NB-IoT. NB-IoT technology is already implemented in licensed frequency bands. Licensed frequency bands for LTE are used for the development of this technology. This technology uses a minimum system bandwidth of 180kHz, which is allocated one PRB (Physical Resource Block). NB-IoT can be viewed as a standalone RAT (Radio Access Technology). NB-IoT can be deployed in three ways: "in-band deployment," "guardband deployment," and "standalone deployment." In "in-band" operation, resource blocks existing in the LTE carrier are used. Specific resource blocks are reserved for LTE signal synchronization and are not used for NB-IoT. In "guardband" operation, resource blocks between LTE carriers not used by any operator are used. In "standalone" operation, GSM frequencies are used, or potentially unused LTE frequency bands are used. Release 13 includes important improvements such as discontinuous reception (eDRX) and power-saving modes. PSM (Power Saving Mode) ensures battery life in version 12 and is handled by eDRX, making it suitable for devices that need to receive data more frequently.

[0006] Furthermore, artificial intelligence and machine learning technologies have seen significant advancements in recent years. Generative Adaptive Networks (GANs) are a class of machine learning techniques that, given a training set, learn to generate new data with the same statistical characteristics as the training set. For example, a GAN trained on a photograph can generate new photos that, to a human observer, at least superficially appear realistic, possessing many realistic features. While GANs were initially proposed as generative models for unsupervised learning, they have also proven applicable to semi-supervised, fully supervised, and reinforcement learning. Additionally, a Gaussian process is a stochastic process (a set of random variables indexed by time or space), and therefore each finite set of these random variables has a multivariate normal distribution, meaning each finite linear combination of them is normally distributed. The distribution of a Gaussian process is the joint distribution of all these (infinitely many) random variables; therefore, it is the distribution of a function with a continuous domain, such as time or space. Machine learning algorithms involving Gaussian processes use lazy learning and a measure of similarity between points (kernel functions) to predict the values ​​of unseen points in the training data. Prediction is not just an estimate of a point, but also contains uncertainty information—it is a one-dimensional Gaussian distribution (i.e., the marginal distribution at that point). For multi-output prediction, a multivariate Gaussian process is used, where the multivariate Gaussian distribution is the marginal distribution at each point.

[0007] Furthermore, artificial intelligence, cognitive modeling, and neural networks are information processing paradigms inspired by how biological nervous systems process data. Artificial intelligence and cognitive modeling attempt to simulate certain characteristics of biological neural networks. In the field of artificial intelligence, artificial neural networks have been successfully applied to areas such as speech recognition, image analysis, and adaptive control. A neural network (NN), in the case of artificial neurons called an artificial neural network (ANN) or simulated neural network (SNN), is a set of interconnected natural or artificial neurons that process information using mathematical or computational models based on connectionist computational methods. In most cases, an artificial neural network is an adaptive system that changes its structure based on external or internal information flowing through the network. In more practical terms, neural networks are nonlinear statistical data modeling or decision-making tools. They can be used to model complex relationships between inputs and outputs or to find patterns in data.

[0008] Furthermore, in the field of image processing, the RGB color model is a color model that reproduces various colors by adding red, green, and blue light in various ways. The model's name comes from the first letters of the three additive primary colors: red, green, and blue. This model is used to sense, represent, and display images in electronic systems such as mobile devices, smartphones, televisions, and computers, although it is also used in traditional / digital photography. This model is a device-dependent color model: different devices detect or reproduce a given RGB value differently because color elements (such as phosphors or dyes) and their responses to individual red, green, and blue levels vary from manufacturer to manufacturer, and even within the same device over time. Therefore, without some form of color management, RGB values ​​do not define the same colors across devices.

[0009] Furthermore, an image scanner is a device that optically scans an image (printed text, handwriting, or objects) and converts it into a digital image, which is then transmitted to a computer / smartphone, most of which support RGB color. Current scanners typically use charge-coupled devices (CCDs) or contact image sensors (CIS) as image sensors, while older drum scanners used photomultiplier tubes. One of the most serious problems with this technology was the potential for damage to the scanning film due to heating issues; this technology was later replaced by non-heated light sources, such as colored LEDs. In addition, accurate color reproduction is required, especially in professional environments, necessitating color management of all equipment involved in the production process, many of which use RGB. During a typical production cycle, color management involves multiple transparent conversions between device-independent and device-dependent color spaces (RGB and other colors, such as CMYK for color printing) to ensure color consistency throughout the process. Along with creative processing, such interventions in digital images can compromise color accuracy and image detail, especially with a reduced color gamut.

[0010] Furthermore, the human eye perceives visible light in most three wavelength bands (long wavelengths – perceived as red, medium wavelengths – perceived as green, short wavelengths – perceived as blue), and spectral imaging divides the spectrum into even more bands. This technique of segmenting images into bands can be extended beyond visible light. In hyperspectral imaging, the recorded spectrum has good wavelength resolution and covers a wide range of wavelengths. Unlike multispectral imaging, which measures intervals of spectral bands, hyperspectral imaging measures continuous spectral bands. Like other spectral imaging methods, hyperspectral imaging collects and processes information from the entire electromagnetic spectrum. The goal of hyperspectral imaging is to acquire the spectrum of every pixel in a scene image for purposes such as finding objects, identifying materials, or detecting processes. With the advent of modern acquisition techniques, hyperspectral imaging has also become an active research area. Unlike the aforementioned RGB (red-green-blue) or multispectral acquisition devices, the goal of hyperspectral imaging is to acquire the complete spectral features reflected from each observable point. Hyperspectral (HS) images have proven to be a greater source of information than RGB images. The utility of HS images varies across many fields, including agriculture, medicine, geology, astronomy, and security. For example, hyperspectral (HS) imaging can be used to measure plant phenotypic parameters. Plant phenotyping is an emerging science that links genomics with plant ecophysiology and agronomy. A functional plant body is formed during plant growth and development through the dynamic interaction between the genetic background and the physical world of plant development. According to various studies, large-scale experimental plant phenotyping is a key factor in breeding better crops that can feed a growing population and provide biomass energy while using less water, land, and fertilizer.

[0011] The richness of information associated with hyperspectral imaging is related to the high cost of sensors that capture the full spectral features. Hyperspectral imaging systems (HIS) facilitate many applications due to their information richness, but this comes at a cost, with a significant reduction in spatial or temporal resolution. The use of HIS is limited to fields and applications such as remote sensing, agriculture, geology, astronomy, earth sciences, and other fields and applications where these aspects of the signal (spatial resolution, but primarily temporal resolution) are not central. Even in these cases, HIS is often used for the preliminary analysis of observable signals to characterize the spectral portions that carry valuable information for the application. This information is then used to design multispectral devices (cameras with several spectral bands), which are optimized for those applications. Unlike their use in the aforementioned niche or specialized applications, the use of HIS in general computer vision, particularly in natural image analysis, is still in its infancy. The main obstacles are not only the spatial, spectral, and / or temporal resolution when acquiring a "cube" of hyperspectral images, but also the cost and physical size (weight and volume) of the hyperspectral devices used to acquire them, both of which are prohibitively high and impose strict limitations on most potential applications. Therefore, the high cost, bulky hardware systems, and complex sensors, including detection systems with temperature cooling units, severely limit the industrial application of hyperspectral technology in any field.

[0012] Furthermore, current imaging-related solutions face numerous limitations. For instance, agricultural systems (e.g., in India or other countries) are unique in their dispersed landholdings, field and in-field variations, and the temporal and spatial variability of agricultural input parameters. Satellite imagery suffers from low resolution and is best suited for weather data. Drone-based imaging technology remains a high-cost technology, driven by regulatory policies. Therefore, there is currently a significant opportunity to provide precise and low-cost ground-based solutions for large-scale sensing of diverse crop parameters, thereby offering farmers accurate, personalized, and timely advice.

[0013] Furthermore, there are currently no sufficiently accurate "all-in-one sensor" smart devices that can process captured images at low cost to measure various plant phenotypic parameters (such as nutrition, early disease prediction, harvest decisions, pesticide / insecticide use) to provide farmers with effective advice based on these parameters, offering the right solutions for planting crops. Field and in-field changes in agriculture or any such related field requiring image analysis need "differentiated processing," rather than the "uniform processing" of traditional management systems. Therefore, it increases the overall input costs for agricultural communities.

[0014] Another limitation of current technology is the scarcity of solutions for converting RGB to hyperspectral data. Currently, this conversion only occurs within the 400-700nm range, and there are no solutions for converting RGB to hyperspectral data / images beyond 700nm. Therefore, there is an inherent need for a method and system that can recover full-range hyperspectral data from RGB images using low-cost, high-resolution terrestrial equipment with the aid of artificial intelligence. AI can automatically construct and map parameters, such as plant phenotypes from existing RGB images to hyperspectral images. Furthermore, current technology is limited in its ability to upgrade RGB images captured by smartphone devices to hyperspectral images, enabling us to obtain accurate images at low cost. Hyperspectral images can perceive most crop parameters very accurately. Another limitation of current technology is the lack of methods and systems that can use low-cost smartphone camera technology to measure various crop input parameters for RGB image capture and conversion to hyperspectral images, thereby providing information output for farmers or agricultural enterprises and offering precise and decision-based agricultural support services. Additionally, one limitation of current technology is the lack of a method and system that can simultaneously provide high-precision hyperspectral-level crop type classification, crop growth stage classification, and disease identification using low-cost technology, independent of bulky hardware systems. Currently, there is no solution for "all-in-one sensor" smart devices to process captured images at low cost, measure various plant phenotypic parameters such as nutrients, early disease prediction, harvest decisions, pesticide / insecticide use, and provide farmers with the right solutions or effective advice for cultivating crops based on RGB images converted to hyperspectral images. Furthermore, there is currently no solution that can generate more features from fewer features.

[0015] Therefore, a new system and method are needed to extract full-range hyperspectral data from one or more RGB images, with at least one of one or more spectral and mathematical information, to improve accuracy in various applications such as agriculture, health and other related fields.

[0016] The foregoing examples and related limitations of the prior art are intended to be illustrative rather than exclusive. Further limitations of the prior art will be apparent to those skilled in the art upon reading the specification and studying the accompanying drawings. Summary of the Invention

[0017] This section provides a simplified overview of certain objects and aspects of the invention, which will be further described in detail in the following detailed description. This summary is not intended to identify key features or scope of the claimed subject matter.

[0018] To at least overcome some of the drawbacks mentioned in the previous section and those known to those skilled in the art, one object of the present invention is to provide a system and method for extracting full-range hyperspectral data from one or more RGB images. Another object of the present invention is to provide a method and system that can recover full-range hyperspectral data from RGB images using low-cost, high-resolution ground-based equipment and artificial intelligence assistance, wherein the AI ​​can automatically construct and map plant phenotypes from existing RGB images to hyperspectral images. Another object of the present invention is to provide a solution capable of providing accurate “all-in-one sensor” intelligent devices to process captured images at low cost for measuring various plant phenotypic parameters. Furthermore, one object of the present invention is to provide methods and systems for intelligently identifying and notifying field and field changes in agriculture or any such related fields, where image analysis and “differentiated treatment” are required, rather than the “uniform processing” scheme of traditional management systems. Another object of the present invention is to provide a solution that facilitates the conversion from RGB to hyperspectral data exceeding 700 nm with currents only in the 400-700 nm range. Furthermore, one object of the present invention is to provide a solution that can upgrade RGB images to hyperspectral images so that we can obtain accurate images at low cost, wherein the hyperspectral images can accurately sense most parameters, such as crop and health-related parameters. Another objective of this invention is to provide a solution that helps convert RGB images into hyperspectral images to provide information output to farmers or agricultural enterprises, thereby providing farmers with decision-based precision agriculture support services. Furthermore, an objective of this invention is to provide a mechanism that can simultaneously provide RGB-to-hyperspectral image conversion technology for crop type classification, crop growth stage classification, and disease identification. Another objective of this invention is to provide a device ecosystem that provides seamless enhancement of image analysis from RGB to hyperspectral images, thereby providing information output for precision and decision services in multi-SIM, multi-activity wireless devices. Another objective of this invention is to provide seamless enhancement of image analysis with hyperspectral images, thereby providing information output technology for precision and decision services in user devices regardless of whether the UE is 5G / 4G / 3G / EV-Do / eHRPD capable. Furthermore, an objective of this invention is to increase the service value to farmers and increase crop yield per hectare. Additionally, an objective of this invention is to provide value to mobile / smartphones and / or devices sold to agricultural enterprises (pesticide, seed, fertilizer companies, etc.) and industrial orchards through hyperspectral image enhancement and image analysis. Another objective of this invention is to add value to information- and technology-rich digital agriculture through abundant inputs, thereby achieving the dual goals of increasing productivity and reducing the ecological burden on soil with low input costs.Another objective of this invention is to upgrade RGB images to hyperspectral images so that we can obtain accurate images at low cost, wherein the hyperspectral images can accurately sense most parameters in order to provide accurate decision-based support services for any other industry, such as health or related fields.

[0019] To achieve the above objectives, the present invention provides a method and system for extracting full-range hyperspectral data from one or more RGB images.

[0020] One aspect of the present invention relates to a method for extracting full-range hyperspectral data from one or more RGB images. The method includes receiving one or more RGB images from one or more camera devices at a transceiver unit. The method then includes preprocessing the one or more RGB images by a processing unit. Furthermore, the method includes estimating by the processing unit an illumination component associated with each of the one or more preprocessed RGB images based on a first pre-trained dataset. The method then includes removing the illumination component from each preprocessed RGB image by the processing unit. Additionally, the method includes tracking by the processing unit the trajectory of one or more pixels on one or more frames associated with each preprocessed RGB image based on an optical flow model. The method then results in the processing unit identifying the position of one or more pixels in one or more neighboring frames of the one or more frames based on blocks defined around the one or more pixels. Subsequently, the method includes extracting full-range hyperspectral data from each preprocessed RGB image corresponding to the one or more RGB images by the processing unit based on at least one of the removal of the illumination component, the trajectory of the one or more pixels, and the position of the one or more pixels.

[0021] Another aspect of the invention relates to a system for extracting full-range hyperspectral data from one or more RGB images. The system includes a transceiver unit configured to receive one or more RGB images from one or more camera devices. The system then includes a processing unit configured to preprocess the one or more RGB images. Furthermore, the processing unit is configured to estimate an illumination component associated with each of the one or more preprocessed RGB images based on a first pre-trained dataset. The processing unit is then configured to remove the illumination component from each preprocessed RGB image. The processing unit is then configured to track the trajectory of one or more pixels on one or more frames associated with each preprocessed RGB image based on an optical flow model. The processing unit is also configured to identify the position of one or more pixels in one or more adjacent frames of the one or more frames based on blocks defined around the one or more pixels. The processing unit is further configured to extract full-range hyperspectral data from each preprocessed RGB image corresponding to the one or more RGB images based on at least one of the illumination component removal, the trajectory of the one or more pixels, and the position of the one or more pixels.

[0022] The following numbered paragraphs were also disclosed:

[0023] 1. A method for extracting full-range hyperspectral data from one or more RGB images, the method comprising:

[0024] - Receive one or more RGB images from one or more camera devices at the transceiver unit

[202] ;

[0025] - The processing unit

[204] preprocesses one or more RGB images;

[0026] - The processing unit

[204] estimates the illumination components associated with each of the one or more preprocessed RGB images based on the first pretrained dataset;

[0027] - The illumination component is removed from each preprocessed RGB image by the processing unit

[204] ;

[0028] - The processing unit

[204] tracks the trajectory of one or more pixels on one or more frames associated with each preprocessed RGB image based on an optical flow model;

[0029] - The processing unit

[204] identifies the position of one or more pixels in one or more adjacent frames of one or more frames based on a block defined around the one or more pixels; and

[0030] - The processing unit

[204] extracts full-range hyperspectral data from each preprocessed RGB image corresponding to one or more RGB images based on at least one of the following: removal of illumination components, trajectory of one or more pixels, and position of one or more pixels.

[0031] 2. The method as described in paragraph 1, wherein the preprocessing includes at least resizing one or more RGB images, denoising one or more RGB images, and enhancing the image quality of one or more RGB images.

[0032] 3. The method as described in paragraph 1, wherein one or more camera devices include one or more microelectromechanical systems (MEMS).

[0033] 4. The method as described in paragraph 1, wherein one or more RGB images are received at the transceiver unit

[202] via a master node associated with one or more camera devices.

[0034] 5. The method as described in paragraph 1, wherein the processing unit

[204] extracts full-range hyperspectral data based on a second pre-training dataset, wherein the second pre-training dataset includes multiple datasets trained by converting multiple RGB images frame by frame to the corresponding hyperspectral level resolution.

[0035] 6. The method as described in paragraph 1, wherein the first pre-training dataset comprises multiple datasets trained based on depth values ​​associated with each object captured in each of the multiple RGB images.

[0036] 7. The method as described in paragraph 6, wherein the estimation of the illumination components associated with each preprocessed RGB image by the processing unit

[204] further includes:

[0037] - The processing unit

[204] assigns a depth value to each RGB pixel based on the first pre-trained dataset, and each RGB pixel is associated with each pre-processed RGB image.

[0038] -The processing unit

[204] synthesizes one or more images of one or more objects captured in each preprocessed RGB image based on the depth value assigned to each RGB pixel under one or more lighting conditions, and

[0039] - The processing unit

[204] estimates the illumination component associated with each preprocessed RGB image based on one or more images of one or more synthesized objects, wherein the illumination component is estimated at a pixel-level scale.

[0040] 8. The method as described in paragraph 7, wherein the illumination components are further estimated by the processing unit

[204] based on one or more artificial intelligence techniques.

[0041] 9. The method as described in paragraph 1, wherein the method includes determining, by the processing unit

[204] , a target RGB value associated with each preprocessed RGB image under ideal conditions based on the following:

[0042] - The illumination component is removed from each preprocessed RGB image by the processing unit

[204] .

[0043] - The processing unit

[204] retrieves the original RGB pixel values ​​of one or more pixels of each preprocessed RGB image based on the removal of the illumination components, and

[0044] - The processing unit

[204] determines the target RGB value based on the original RGB pixel values ​​of one or more pixels of each preprocessed RGB image.

[0045] 10. The method as described in paragraph 1, further comprising performing pixel-level semantic segmentation of objects of interest present in full-range hyperspectral data by processing unit

[204] , the full-range hyperspectral data corresponding to each of one or more RGB images.

[0046] 11. The method as described in paragraph 10, wherein pixel-level semantic segmentation is performed based on one or more artificial intelligence techniques.

[0047] 12. The method described in paragraph 10, further comprising:

[0048] - The processing unit

[204] organizes the full-range hyperspectral data, which corresponds to each RGB image in one or more band subsets with similar spectral characteristics, and

[0049] - The processing unit

[204] extracts one or more spectral and one or more spatial features from the full-range hyperspectral data corresponding to each RGB image based on the organization.

[0050] 13. The method as described in paragraph 12, further comprising the processing unit

[204] determining one or more parameters related to at least one of the agricultural and health domains based on one or more extracted spectral features, one or more extracted spatial features, and a third pre-trained dataset.

[0051] 14. The method as described in paragraph 12, wherein the third pre-trained dataset comprises multiple data trained on hyperspectral data and RGB depth data associated with multiple events, which are associated with at least one in the fields of agriculture and health.

[0052] 15. A system for extracting full-range hyperspectral data from one or more RGB images, the system comprising:

[0053] - A transceiver unit

[202] configured to receive one or more RGB images from one or more camera devices; and

[0054] - A processing unit

[204] , configured as follows:

[0055] Preprocess one or more RGB images.

[0056] Based on the first pre-trained dataset, estimate the illumination components associated with each of the one or more pre-processed RGB images.

[0057] Remove the illumination component from each preprocessed RGB image.

[0058] Based on an optical flow model, the trajectory of one or more pixels is tracked across one or more frames associated with each preprocessed RGB image.

[0059] Based on the block defined around the one or more pixels, the location of one or more pixels in one or more adjacent frames of one or more frames is identified, and

[0060] Full-range hyperspectral data is extracted from each preprocessed RGB image corresponding to one or more RGB images, based on at least one of the following: removal of illumination components, trajectory of one or more pixels, and position of one or more pixels.

[0061] 16. The system as described in paragraph 15, wherein the processing unit

[204] is configured to preprocess one or more RGB images by at least resizing one or more RGB images, denoising one or more RGB images, and enhancing the image quality of one or more RGB images.

[0062] 17. The system as described in paragraph 15, wherein one or more camera devices include one or more microelectromechanical systems (MEMS).

[0063] 18. The system as described in paragraph 15, wherein one or more RGB images are received at the transceiver unit

[202] via a master node associated with one or more camera devices.

[0064] 19. The system as described in paragraph 15, wherein the extraction of full-range hyperspectral data is further based on a second pre-trained dataset, wherein the second pre-trained dataset comprises multiple datasets trained by converting multiple RGB images frame by frame to the corresponding hyperspectral level resolution.

[0065] 20. The system as described in paragraph 15, wherein the first pre-training dataset comprises multiple datasets trained based on depth values ​​associated with each object captured in each of the multiple RGB images.

[0066] 21. The system as described in paragraph 20, wherein the processing unit

[204] for estimating the illumination components associated with each preprocessed RGB image is configured as follows:

[0067] - Assign depth values ​​to each RGB pixel based on the first pre-trained dataset, with each RGB pixel associated with each pre-processed RGB image.

[0068] - Based on the depth value assigned to each RGB pixel, one or more images of one or more objects captured in each preprocessed RGB image are synthesized under one or more lighting conditions, and

[0069] - Estimate the illumination components associated with each preprocessed RGB image based on one or more images of one or more synthesized objects, where the illumination components are estimated at pixel-level scale.

[0070] 22. The system as described in paragraph 21, wherein the processing unit

[204] is further configured to estimate the lighting components based on one or more artificial intelligence techniques.

[0071] 23. The system as described in paragraph 15, wherein the processing unit

[204] is further configured to determine, under ideal conditions, the target RGB values ​​associated with each preprocessed RGB image based on the following:

[0072] - Remove the illumination component from each preprocessed RGB image.

[0073] - The processing unit

[204] retrieves the original RGB pixel values ​​of one or more pixels of each preprocessed RGB image based on the removal of the illumination components, and

[0074] - The processing unit

[204] determines the target RGB value based on the original RGB pixel values ​​of one or more pixels of each preprocessed RGB image.

[0075] 24. The system as described in paragraph 15, wherein the processing unit

[204] is further configured to perform pixel-level semantic segmentation on objects of interest present in full-range hyperspectral data, the full-range hyperspectral data corresponding to each of one or more RGB images.

[0076] 25. The system as described in paragraph 24, wherein pixel-level semantic segmentation is performed based on one or more artificial intelligence techniques.

[0077] 26. The system as described in paragraph 24, wherein the processing unit

[204] is further configured to:

[0078] - Organize full-range hyperspectral data, corresponding to each RGB image in one or more band subsets with similar spectral characteristics, and

[0079] -Based on the organization, one or more spectral and one or more spatial features are extracted from the full range of hyperspectral data corresponding to each RGB image.

[0080] 27. The system as described in paragraph 25, wherein the processing unit

[204] is further configured to determine one or more parameters related to at least one of the agricultural and health fields based on one or more extracted spectral features, one or more extracted spatial features, and a third pre-trained dataset.

[0081] 28. The system as described in paragraph 25, wherein the third pre-trained dataset comprises multiple data trained on hyperspectral data and RGB depth data associated with multiple events, which are associated with at least one in the fields of agriculture and health. Attached Figure Description

[0082] The accompanying drawings, which are incorporated herein and constitute a part of this disclosure, illustrate exemplary embodiments of the disclosed methods and systems, wherein the same reference numerals refer to the same components throughout all the different drawings. Components in the drawings are not necessarily drawn to scale, but rather the emphasis is on clearly illustrating the principles of this disclosure. Some drawings may use block diagrams to represent components and may not show the internal circuitry of each component. Those skilled in the art will understand that the disclosure of such drawings includes disclosure of electrical components, electronic components, or circuitry commonly used to implement such components.

[0083] Figure 1 An exemplary system architecture for RGB to hyperspectral conversion as an integrated part of a user equipment (UE) is shown according to an exemplary embodiment of the present invention

[100] .

[0084] Figure 2 An exemplary block diagram of a system

[200] for extracting full-range hyperspectral data from one or more RGB images according to an exemplary embodiment of the present invention is shown.

[0085] Figure 3 An example diagram

[300] is shown for collecting one or more RGB images to extract full-range hyperspectral data from one or more RGB images according to an exemplary embodiment of the present invention.

[0086] Figure 4 An exemplary method flowchart

[400] is shown according to an exemplary embodiment of the present invention, depicting a method for extracting full-range hyperspectral data from one or more RGB images.

[0087] Figure 5 (i.e.) Figure 5a 5b) illustrates an exemplary use case of extracting full-range hyperspectral data from one or more RGB images according to an exemplary embodiment of the present invention.

[0088] Figure 6 An exemplary process

[600] according to an exemplary embodiment of the present invention is shown, which indicates a use case for crop classification based on extracting full-range hyperspectral data from one or more RGB images.

[0089] The foregoing will become more apparent from the following specific embodiments of this disclosure. Detailed Implementation

[0090] In the following description, various specific details are set forth for purposes of explanation in order to fully understand embodiments of the present disclosure. However, it will be apparent that embodiments of the present disclosure can be practiced without these specific details. The several functions described below can be used independently or in any combination with other functions. A single function may not solve any of the problems described above, or may only solve some of the problems described above.

[0091] The following description provides exemplary embodiments only and is not intended to limit the scope, applicability, or configuration of this disclosure. Rather, the subsequent description of exemplary embodiments will provide those skilled in the art with enabling descriptions for implementing the exemplary embodiments. It should be understood that various changes can be made to the function and arrangement of the elements without departing from the spirit and scope of the invention as set forth.

[0092] Furthermore, it should be noted that each embodiment can be described as a process, which can be described as a flowchart, sequence diagram, data flow diagram, structure diagram, or block diagram. Although a flowchart can describe operations as a sequential process, many operations can be performed in parallel or simultaneously. Furthermore, the order of operations can be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the diagram. A process can correspond to a method, function, program, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0093] Furthermore, embodiments can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented as software, firmware, middleware, or microcode, program code or code segments (e.g., a computer program product) that perform the necessary tasks can be stored on a machine-readable medium. The processor can then perform the necessary tasks.

[0094] The terms "machine-readable storage medium" or "computer-readable storage medium" include, but are not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instructions and / or data. Machine-readable media can include non-transitory media in which data can be stored and which do not include carrier waves and / or transient electronic signals propagated wirelessly or via a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or magnetic tapes, optical storage media such as CDs or DVDs, flash memory, memory, or storage devices. Computer program products can include code and / or machine-executable instructions that can represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. Code segments can be coupled to another code segment or hardware circuitry by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted in any suitable manner, including memory sharing, message passing, token passing, network transmission, etc.

[0095] The terms “exemplary” and / or “illustrative” as used herein mean as examples, instances, or illustrations. For the avoidance of ambiguity, the subject matter disclosed herein is not limited to these examples. Furthermore, any aspect or design described herein as “exemplary” and / or “illustrative” is not necessarily to be construed as superior to other aspects or designs, nor does it imply the exclusion of equivalent exemplary structures and techniques known to those skilled in the art. Additionally, where the terms “comprising,” “having,” “including,” and other similar words are used in the embodiments or claims, such terms are intended to be encompassed in a similar manner to the term “comprising” as open-ended transitional terms—without excluding any additional or other elements.

[0096] Throughout this specification, references to "an embodiment" or "an embodiment" mean that a particular feature, structure, or characteristic described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrase "in an embodiment" appearing throughout this specification does not necessarily refer to the same embodiment. Furthermore, particular features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0097] The terminology used herein is for describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be understood that, when used in this specification, the terms “comprising” and / or “including” specify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0098] As used herein, the term "data" means any sign, signal, mark, symbol, field, set of symbols, representation, and any other physical form representing information, whether permanent or temporary, whether visible, audible, acoustic, electrical, magnetic, electromagnetic, or otherwise. The term "data" used to represent predetermined information in one physical form should be considered as encompassing any and all representations of corresponding information expressed in one or more different physical forms.

[0099] The terms “first,” “second,” “primary,” and “secondary” are used to distinguish an element, set, data, object, step, process, function, activity, or thing, and not to specify relative position, timing, or relative importance, unless otherwise expressly stated. As used herein, the terms “coupled” and “coupled to” refer to a relationship between two or more devices, apparatuses, documents, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, and / or devices that constitute (a) any one or more connections, whether direct or through one or more other devices, apparatuses, documents, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, or devices; (b) a communication relationship, whether direct or through one or more other devices, apparatuses, documents, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, or devices; and / or (c) a functional relationship in which any one or more devices, apparatuses, documents, circuits, elements, functions, operations, processes, procedures, media, components, networks, systems, subsystems, or devices are wholly or partially dependent on the operation of any one or more of them.

[0100] As used herein, the term "communication" includes both the transmission of data from a source to a destination and the transmission of data to a communication medium, system, channel, network, device, wire, cable, optical fiber, circuit, and / or link to a destination. The term "communication" as used herein refers to data so transmitted or transferred. The term "communication" as used herein includes one or more of the following: communication medium, system, channel, network, device, wire, cable, optical fiber, circuit, and link.

[0101] Furthermore, terms such as “User Equipment” (UE), “electronic device,” “mobile station,” “user equipment,” “mobile user station,” “access terminal,” “terminal,” “smartphone,” “intelligent computing device,” “mobile phone,” and similar terms refer to any electrical, electronic, electromechanical equipment, or a combination of one or more of the above. Intelligent computing devices may include, but are not limited to, mobile phones, smartphones, virtual reality (VR) devices, augmented reality (AR) devices, pagers, laptops, general-purpose computers, desktop computers, personal digital assistants, tablets, mainframe computers, or any other computing device obvious to those skilled in the art. Generally, an intelligent computing device is a digital, user-configurable, autonomously operating computer network device. An intelligent computing device is one of the appropriate systems for storing data and other private / sensitive information. The device operates at all seven levels of the ISO reference model, but its primary functions relate to the application layer as well as any additional functions such as networking, session and presentation layers, touchscreens, application ecosystems, physical and biometric security, etc. Furthermore, a "smartphone" is a "smart computing device" referring to a mobile wireless cellular connection device that allows end users to use services on 2G, 3G, 4G, 5G, and similar mobile broadband internet connections with an advanced mobile operating system, which combines the functionality of a personal computer operating system with other functions for mobile or handheld use. These smartphones can access the internet, have a touchscreen user interface, can run third-party applications, including the ability to host online applications and music players, and are camera phones with high-speed mobile broadband 4G LTE internet, featuring video calling, hotspot functionality, motion sensors, mobile payment mechanisms, and enhanced security features, and can issue alarms and alerts in emergencies. Mobile devices can include smartphones, wearable devices, smartwatches, smart bracelets, wearable enhancement devices, etc. For specific details, we use "mobile device" in this disclosure to refer to feature phones and smartphones, but this does not limit the scope of this disclosure and can be extended to any mobile device implementing the technical solutions. The aforementioned smart devices, including smartphones, and feature phones, including IoT devices, can enable on-device communication. Furthermore, the foregoing terms are used interchangeably in this specification and related drawings.

[0102] As used herein, "processor" or "processing unit" includes one or more processors, where a processor refers to any logic circuitry used to process instructions. A processor can be a general-purpose processor, a special-purpose processor, a conventional processor, a digital signal processor, multiple microprocessors, one or more microprocessors associated with a DSP core, a controller, a microcontroller, a low-end microcontroller, an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), any other type of integrated circuit, etc. Furthermore, the term "processor" as used herein includes, but is not limited to, one or more computers, hardwired circuitry, signal modification devices and systems, devices and machines for controlling systems, a central processing unit, a programmable device and system, a system-on-a-chip (SoC), a system composed of discrete components and / or circuitry, a state machine, a virtual machine, a data processor, a processing facility, and any combination thereof. A processor can perform signal-encoded data processing, input / output processing, and / or other functions to enable the operation of a system according to this disclosure. More specifically, a processor or processing unit is a hardware processor. As used herein, "processor" refers to a processing device, apparatus, program, circuit, component, system, or subsystem, whether implemented in hardware, tangibly embodied software, or both, and whether or not it is programmable.

[0103] As used herein, “memory cell,” “storage unit,” and / or “memory” refers to a machine- or computer-readable medium, including any mechanism for storing information in a form readable by a computer or similar machine. For example, computer-readable media include read-only memory (“ROM”), random access memory (“RAM”), disk storage media, optical storage media, flash memory devices, or other types of machine-accessible storage media. Memory cells, as used herein, are configured to temporarily or permanently retain data and to provide such retained data to various cells to perform their respective functions.

[0104] As used herein, a "transceiver unit" may include, but is not limited to, a transmitter that transmits data to one or more destinations and a receiver that receives data from one or more sources. Furthermore, a transceiver unit may include any other similar units that are obvious to those skilled in the art to achieve the functionality of this invention. A transceiver unit may convert data or information into signals and vice versa, for the purposes of transmission and reception, respectively.

[0105] As disclosed in the Background section, existing technologies have many limitations. To overcome at least some of these limitations of known solutions, this disclosure provides a solution for extracting full-range hyperspectral data from one or more RGB images for various applications in agriculture, health, and other related fields. In one implementation, the invention provides a solution for recovering full-range hyperspectral data from one or more RGB images using low-cost, high-resolution ground-based equipment, aided by artificial intelligence capable of automatically constructing and mapping parameters (e.g., plant phenotypic and / or health-related parameters) from existing RGB images to hyperspectral images. Furthermore, implementations based on the features of this invention achieve improved data accuracy and spectral resolution. For example, at least one of one or more spectral and mathematical information can be obtained from full-range hyperspectral data to improve accuracy in various applications in agriculture, health, and other related fields. Figure 1 An exemplary system architecture

[100] is described for converting RGB to hyperspectral data as an integrated part of a user equipment (UE). More specifically, Figure 1 An exemplary UE based on optical microelectromechanical systems (MEMS) spectral technology is described, wherein at least one image is captured using an RGB sensor, and then the image is converted into corresponding hyperspectral information. Furthermore, at

[102] , Figure 1 RGB objects, such as sample objects or scenes to be captured, are depicted. Furthermore, at

[104] and

[106] , an objective lens is shown together with the entire optical array in the RGB sensor and MEMS or related unit to capture at least one image of an RGB object. In one implementation, a MEMS UE initiates the capture of at least one image by a MEMS camera on the UE with supporting parameters. Subsequently,

[108] describes the conversion of at least one RGB image into corresponding hyperspectral data. More specifically, the MEMS UE transmits data capture information (i.e., at least one image) to a leader / master node of the local network of the MEMS node, which further uploads / publishes said information to a real-time database in a cloud server unit, at least for the conversion into hyperspectral data. The MEMS UE is configured to receive data corresponding to any changes occurring in the dataset at the cloud server unit, any changes including, but not limited to, the addition of copying / synchronization of information from one or more sensors on the MEMS UE, thus receiving hyperspectral data output at the MEMS UE based on the conversion

[110] . Subsequently,

[112] shows that improved data accuracy and spectral resolution are achieved based on the hyperspectral data output. More specifically, in one implementation, the MEMS UE periodically performs on-device calibration of the hyperspectral data output and displays one or more graphs. Thus, hyperspectral information is utilized to enhance image analysis to provide better accuracy and resolution.

[0106] In the following, exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement the present disclosure.

[0107] Reference Figure 2 An exemplary block diagram of a system for extracting full-range hyperspectral data from one or more RGB images according to an exemplary embodiment of the present invention is shown

[200] .

[0108] System

[200] includes at least one transceiver unit

[202] , at least one processing unit

[204] , and at least one storage unit

[208] . Furthermore, all components / units of system

[100] are assumed to be interconnected unless otherwise stated below. Figure 1 Only a few units are shown in the diagram; however, the system

[200] may include a plurality of such units or any number of such units, depending on the requirements for implementing the features of this disclosure.

[0109] The system

[200] is configured to extract full-range hyperspectral data from one or more RGB images by means of interconnection between its components / units.

[0110] To extract full-range hyperspectral data from one or more RGB images, the transceiver unit

[202] of the system

[200] is configured to receive one or more RGB images from one or more camera devices. Furthermore, each image from the one or more RGB images can be a single image, a series of images, or a short video captured by one or more camera devices. In one embodiment, the one or more camera devices can be one or more user devices (e.g., smartphones) with image / video capture capabilities, wherein the one or more user devices are configured to capture one or more RGB images. Thus, in such an embodiment, based on the features of the invention, the UI on one or more user devices (UEs) is configured to allow the user to capture one or more RGB images for processing, further providing high-resolution spectral features based on hyperspectral imaging, which can provide the desired services in various fields such as agriculture and healthcare. Similarly, in another embodiment, the one or more camera devices include one or more microelectromechanical systems (MEMS), wherein the one or more MEMS camera devices / MEMS UEs are configured to initiate the capture of images and / or videos by a MEMS camera / sensor on one or more MEMS UEs with supported parameters to capture one or more RGB images. Furthermore, in such an implementation, one or more RGB images are received at the transceiver unit

[202] via a master node associated with one or more camera devices / MEMS cameras. Figure 3An exemplary graph

[300] is shown for collecting one or more RGB images to extract full-range hyperspectral data from one or more RGB images according to an exemplary embodiment of the present invention. Figure 3 A MEMS node network is described, including one or more leader / master nodes

[302] and one or more MEMS sensors / cameras

[304] . One or more master nodes

[302] are connected to a cloud server unit

[306] , wherein the cloud server unit

[306] is further connected to user equipment

[308] . Figure 3 As shown, whenever one or more MEMS cameras

[304] with supported parameters capture an image, the one or more MEMS cameras, i.e., one or more camera devices

[304] , relay the data capture information, i.e., the captured image, to one or more leader / master nodes

[302] of the local network of the MEMS node

[300] . The one or more local leader nodes

[302] upload / publish the data capture information to a real-time database in the cloud server unit

[306] via a transceiver unit

[202] . More specifically, in a given embodiment, the transceiver unit

[202] is configured to receive the data capture information, i.e., the captured image, from one or more master nodes

[302] at a system

[200] configured at the cloud server unit

[306] . Furthermore, in one embodiment, the transceiver unit

[202] is also configured to transmit full-range hyperspectral data extracted by the system

[100] from the captured image at a user equipment

[308] .

[0111] Once one or more RGB images are received at the transceiver unit

[202] , the processing unit

[204] connected to the transceiver unit

[202] is first configured to preprocess the one or more RGB images in order to extract full-range hyperspectral data from the one or more RGB images. More specifically, the processing unit

[204] is configured to preprocess the one or more RGB images by at least resizing the one or more RGB images, denoising the one or more RGB images, and improving the image quality of the one or more RGB images. The preprocessing of the one or more RGB images to resize, denoise, and enhance the image quality is implemented because the one or more RGB images can be received from one or more different or identical camera devices of different sizes and resolutions, and are subject to different distortions due to the external environment.

[0112] Subsequently, the processing unit

[204] is configured to estimate the illumination component associated with each of the one or more preprocessed RGB images based on a first pretrained dataset. The first pretrained dataset includes multiple datasets trained based on depth values ​​associated with each object captured in each of the multiple RGB images. More specifically, the processing unit

[204] is configured to assign depth values ​​to each RGB pixel associated with each preprocessed RGB image based on the first pretrained dataset. Subsequently, the processing unit

[202] synthesizes one or more images of the one or more objects captured in each preprocessed RGB image under one or more lighting conditions based on the depth values ​​assigned to each RGB pixel. Furthermore, the processing unit

[204] is configured to estimate the illumination component associated with each preprocessed RGB image based on the synthesized one or more images of the one or more objects, wherein the illumination component is estimated at a pixel-level scale. In addition, the processing unit

[204] is also configured to estimate the illumination component based on one or more artificial intelligence techniques. In one implementation, one or more 3D point cloud representations of one or more objects are formed based on depth values ​​assigned to each RGB pixel associated with each preprocessed RGB image, wherein the one or more objects are captured in each preprocessed RGB image under one or more lighting conditions, in order to estimate the illumination components of each preprocessed RGB image at the pixel scale.

[0113] The processing unit

[204] is further configured to remove the illumination component from each preprocessed RGB image. Removing the illumination component from each preprocessed RGB image yields illumination-independent RGB pixel values ​​for one or more pixels of each preprocessed RGB image, so as to further obtain, under ideal conditions, the actual RGB value (i.e., the target RGB value) associated with each preprocessed RGB image. More specifically, the processing unit

[204] is configured to determine, under ideal conditions, the target RGB value associated with each preprocessed RGB image based on the removal of the illumination component from each preprocessed RGB image, the removal of the illumination component, and the original RGB pixel values ​​(i.e., illumination-independent) retrieved by the processing unit

[204] for one or more pixels of each preprocessed RGB image, and the target RGB value (i.e., the actual RGB value) determined by the processing unit

[204] based on the original RGB pixel values ​​of one or more pixels of each preprocessed RGB image.

[0114] Subsequently, the processing unit

[204] is configured to track the trajectories of one or more pixels on one or more frames associated with each preprocessed RGB image based on an optical flow model. In one embodiment, the processing unit

[204] is configured to estimate the trajectories of one or more pixels on different frames associated with each preprocessed RGB image using an optical flow model (dense estimation). For example, the processing unit

[204] for tracking pixel trajectories is configured to receive one or more frames and / or video of the preprocessed RGB image of the input image as input. Subsequently, the processing unit

[204] is configured to include the received one or more frames and / or video together in the same pixel based on the optical flow model to obtain a better / more accurate hyperspectral representation of the input image.

[0115] Furthermore, the processing unit

[204] is configured to identify the location of one or more pixels in one or more adjacent frames of one or more frames based on blocks defined around the one or more pixels. Additionally, the processing unit is configured to define blocks as pixel neighborhoods of one or more pixels to aggregate the motion of one or more pixels based on the same assumption that the optical flow of pixels in the neighborhood is a given. In one embodiment, the block-level aggregation of motion for identifying the location of one or more pixels is based on one or more artificial intelligence and machine learning techniques. In one embodiment, a model is trained based on cascaded motion and RGB information, using hyperspectral block pairs, so that a Long Short-Term Memory (LSTM) learns structure / normals from one or more temporal images (i.e., internal 3D representations of one or more objects), thereby providing better estimates compared to any standard neural architecture that only provides a single RGB image.

[0116] Subsequently, the processing unit

[204] is configured to extract full-range hyperspectral data from each preprocessed RGB image corresponding to one or more RGB images based on at least one of illumination component removal, the trajectory of one or more pixels, and the position of one or more pixels. In one embodiment, the extraction of full-range hyperspectral data is further based on a second pre-training dataset, wherein the second pre-training dataset includes multiple datasets trained on converting multiple RGB images frame-by-frame to corresponding hyperspectral horizontal resolutions. For example, the second pre-training dataset may include multiple datasets trained on converting multiple health-related RGB images frame-by-frame to corresponding hyperspectral horizontal resolutions, wherein in one embodiment, the processing unit is configured to map preprocessed RGB images indicating diseases to the second pre-training dataset to further extract the corresponding full-range hyperspectral data.

[0117] Furthermore, the processing unit

[204] is configured to perform pixel-level semantic segmentation on objects of interest present in the full range of hyperspectral data corresponding to one or more RGB images. Pixel-level semantic segmentation is performed based on one or more artificial intelligence techniques. Additionally, pixel-level semantic segmentation is performed to separate the objects of interest, i.e., foreground and background separation.

[0118] Once pixel-level semantic segmentation is performed, the processing unit

[204] is configured to organize the full-range hyperspectral data corresponding to each RGB image into one or more band subsets with similar spectral features. Furthermore, the processing unit

[204] is also configured to extract at least one or more spectral features and at least one or more spatial features from the full-range hyperspectral data corresponding to each RGB image based on the organization of the full-range hyperspectral data. The full-range hyperspectral data is organized into band subsets with similar spectral features because different parts of the object of interest may have different features, for example, different parts of a crop may have different chemical features, and one or more spectral and spatial features are extracted to make the next stage easier for the processing unit

[204] to determine, with one or more parameters associated with at least one of the agricultural, health, and similar fields.

[0119] Subsequently, the processing unit

[204] is further configured to determine one or more parameters related to at least one of the agricultural and health domains based on one or more extracted spectral features, one or more extracted spatial features, and a third pre-training dataset. The third pre-training dataset includes multiple datasets trained on hyperspectral data and RGB depth data associated with multiple events related to at least one of the agricultural and health domains. In one embodiment, the third pre-training dataset may include hyperspectral data and RGB depth data of multiple crops indicating different growth stages, crop health / nutrient mapping data, and disease progression mapping data. In such an embodiment, the processing unit

[204] is configured to accurately map the hyperspectral signal patterns of the crop's RGB image (i.e., one or more extracted spectral features and one or more extracted spatial features) to each crop parameter (e.g., a certain band pattern of nitrogen, nutrient stage, reproductive stage, fungal disease, bacterial disease, etc.) in the third pre-training dataset to determine one or more parameters related to the crop, thereby further predicting one or more crop conditions. Furthermore, in one instance, the processing unit

[204] is configured to use a computer vision generative adversarial network (GAN) to generate each hyperspectral signal pattern using RGB patterns, i.e., the RGB depth data contained in the third pre-training dataset. Therefore, one or more agricultural-related parameters determined by mapping hyperspectral signal patterns to crop parameters can provide farmers with a variety of services to achieve precision nutrition related to crop growth stages (to further reduce input costs and improve crop yield and soil quality), early disease prediction (to further reduce the use and cost of pesticides, fungicides, and insecticides), and seed quality assessment and harvest decisions (to further reduce input costs and extend the shelf life of vegetables and fruits).

[0120] Furthermore, once one or more parameters relating to at least one of agriculture, health, and other such fields are determined, the transceiver unit

[202] is configured to transmit the one or more parameters to the user equipment for further display of graphs indicating relevant information for one or more of the agriculture, health, and other such fields. In one embodiment, the transceiver unit

[202] is configured to observe / listen to any changes occurring in the storage unit

[206] in order to transmit updated information to the user equipment. Thus, any changes, including but not limited to adding information from one or more camera devices / MEMS sensors, are copied / synchronized on the user equipment. Additionally, in one embodiment, the user equipment periodically performs device calibration on readings collected from the transceiver unit

[202] and displays graphs.

[0121] refer to Figure 4An exemplary method flowchart

[400] according to an exemplary embodiment of the present invention is shown, depicting a method for extracting full-range hyperspectral data from one or more RGB images. In one embodiment, the method is performed by a system

[200] , and the system

[200] may be configured on a cloud server unit. Figure 4 As shown, the method begins at step

[402] .

[0122] In step

[404] , the method includes receiving one or more RGB images from one or more camera devices at a transceiver unit

[202] . Each RGB image from the one or more RGB images may be a single RGB image, a series of RGB images, or a short video captured by one or more camera devices. In one embodiment, the one or more camera devices may be one or more user devices (e.g., smartphones) with image / video capture capabilities, wherein the one or more user devices are configured to capture one or more RGB images. Thus, in such an embodiment, the method includes allowing one or more users to capture one or more RGB images via a UI on one or more user devices (UEs) to process the one or more RGB images according to an embodiment based on the features of the present invention, further providing high-resolution spectral features based on hyperspectral imaging, which can provide desired services in various fields such as agriculture, healthcare, etc. Also in another embodiment, the one or more camera devices include one or more microelectromechanical systems (MEMS), wherein in such an embodiment, the method includes being initiated by one or more MEMS camera devices / MEMS UEs to capture images and / or videos using MEMS cameras / sensors of one or more MEMS UEs with supporting parameters to capture one or more RGB images. Furthermore, in such an implementation, one or more RGB images are received at the transceiver unit

[202] via a master node associated with one or more camera devices, i.e., one or more MEMS cameras. More specifically, the method includes relaying one or more RGB images from one or more MEMS cameras to the master node, wherein the master node uploads / publishes one or more RGB images to a storage unit

[206] via the transceiver unit

[202] , the storage unit

[206] being configured in a cloud server unit.

[0123] Subsequently, in step

[406] , the method includes preprocessing one or more RGB images by the processing unit

[204] . The preprocessing includes at least resizing one or more RGB images, denoising one or more RGB images, and enhancing the image quality of one or more RGB images. The preprocessing of one or more RGB images to resize one or more RGB images, denoise one or more RGB images, and enhance the image quality of one or more RGB images is implemented because one or more RGB images may be received from one or more different or identical camera devices of different sizes and resolutions and are affected by different deformations due to the external environment.

[0124] Next, in step

[408] , the method includes having the processing unit

[204] estimate the illumination components associated with each of the one or more preprocessed RGB images based on a first pretrained dataset. The first pretrained dataset includes multiple datasets trained based on depth values ​​associated with each object captured in each of the multiple RGB images. Furthermore, the process by which the processing unit

[204] estimates the illumination components associated with each preprocessed RGB image first includes having the processing unit

[204] assign depth values ​​to each RGB pixel based on the first pretrained dataset, each RGB pixel being associated with each preprocessed RGB image. Subsequent processes result in the processing unit

[204] synthesizing one or more images of the one or more objects captured in each preprocessed RGB image under one or more illumination conditions based on the depth values ​​assigned to each RGB pixel. Subsequent processes include having the processing unit

[204] estimate the illumination components associated with each preprocessed RGB image based on the synthesized one or more images of the one or more objects, wherein the illumination components are estimated at a pixel-level scale. Furthermore, the process by which the processing unit

[204] estimates the illumination components associated with each preprocessed RGB image is based on one or more artificial intelligence techniques. In one implementation, one or more 3D point cloud representations of one or more objects are formed based on depth values ​​assigned to each RGB pixel associated with each preprocessed RGB image, wherein the one or more objects are captured in each preprocessed RGB image under one or more lighting conditions, to estimate the illumination components at the pixel-level scale for each preprocessed RGB image. Further considering an example, if the preprocessed RGB images contain a specific series of preprocessed RGB images, the method includes, by processing unit

[204] , assigning depth values ​​to each RGB pixel associated with the specific series of preprocessed RGB images based on a first pretrained dataset, wherein the first pretrained dataset comprises multiple datasets trained based on depth values ​​associated with the specificity captured in each of the multiple RGB images.

[0125] Subsequently, the method includes, by processing unit

[204] , synthesizing the one or more specific images captured in the series of preprocessed RGB images under one or more lighting conditions, based on depth values ​​assigned to each RGB pixel associated with the series of preprocessed RGB images. Furthermore, the method includes, by processing unit

[204] , estimating an illumination component associated with each image of the series of preprocessed RGB images based on the specific synthesized one or more images, wherein the illumination component is estimated at a pixel-level scale.

[0126] Furthermore, in step

[410] , the method includes removing the illumination component from each preprocessed RGB image by the processing unit

[204] . Removing the illumination component from each preprocessed RGB image yields illumination-independent RGB pixel values ​​for one or more pixels of each preprocessed RGB image, so as to obtain, under ideal conditions, the actual RGB value (i.e., the target RGB value) associated with each preprocessed RGB image. More specifically, the method includes determining, under ideal conditions, the target RGB value associated with each preprocessed RGB image by the processing unit

[204] based on the following: removing the illumination component from each preprocessed RGB image by the processing unit

[204] ; retrieving, by the processing unit

[204] , the original (i.e., illumination-independent) RGB pixel values ​​of one or more pixels of each preprocessed RGB image based on the removal of the illumination component; and determining, by the processing unit

[204] , the target RGB value (i.e., the actual RGB value) based on the original RGB pixel values ​​of one or more pixels of each preprocessed RGB image. Considering the example above, where one or more images based on a specific synthesis determine the illumination component associated with each image in a series of preprocessed RGB images, the method in the given example also includes removing the illumination component from all the specific preprocessed series of RGB images to retrieve the actual RGB pixel values ​​(i.e., the target RGB values) associated with the series of preprocessed RGB images.

[0127] Subsequently, in step

[412] , the method includes the processing unit

[204] tracking the trajectory of one or more pixels on one or more frames associated with each preprocessed RGB image based on an optical flow model. In one embodiment, the method includes the processing unit

[204] estimating the trajectory of one or more pixels on different frames associated with each preprocessed RGB image (i.e., dense estimation) using an optical flow model. For example, a method of tracking pixel trajectories includes the processing unit

[204] receiving one or more frames and / or video of a preprocessed RGB image of an input image as input. Subsequently, the method includes including the received one or more frames and / or video together in the same pixel based on an optical flow model to obtain a better / more accurate hyperspectral representation of the input image.

[0128] Next, in step

[414] , the method includes having the processing unit

[204] identify the location of one or more pixels in one or more neighboring frames of the one or more frames based on blocks defined around the one or more frames. Furthermore, the method includes having the processing unit define blocks as neighborhoods of one or more pixels based on the assumption that the optical flow of pixels in the neighborhood is the same, to aggregate the motion of one or more pixels based on the neighborhood. In one embodiment, the block-level aggregation of motion for identifying the location of one or more pixels is based on one or more artificial intelligence and machine learning techniques. In one embodiment, a model is trained based on cascaded motion and RGB information, hyperspectral block pairs, so that a Long Short-Term Memory (LSTM) learns structure / normals from one or more temporal images (i.e., internal 3D representations of one or more objects), thereby providing better estimates compared to any standard neural architecture that only provides a single RGB image.

[0129] Furthermore, in step

[416] , the method includes the processing unit

[204] extracting full-range hyperspectral data from each preprocessed RGB image corresponding to one or more RGB images based on at least one of the following: removal of illumination components, trajectories of one or more pixels, and positions of one or more pixels. Additionally, in one embodiment, the process by which the processing unit

[204] extracts the full-range hyperspectral data is also based on a second pre-training dataset, wherein the second pre-training dataset includes multiple datasets trained by converting multiple RGB images frame-by-frame to corresponding hyperspectral level resolutions. For example, the second pre-training dataset may include multiple datasets trained by converting multiple agricultural-related RGB images frame-by-frame to corresponding hyperspectral level resolutions, wherein in one embodiment, the method includes the processing unit

[204] mapping a preprocessed RGB image indicating a crop to the second pre-training dataset to further extract the corresponding full-range hyperspectral data.

[0130] Furthermore, the method includes performing pixel-level semantic segmentation on objects of interest in the full range of hyperspectral data corresponding to one or more RGB images by a processing unit

[204] . Pixel-level semantic segmentation is performed based on one or more artificial intelligence techniques. Additionally, pixel-level semantic segmentation is performed to separate objects of interest (i.e., foreground) from the background.

[0131] Once pixel-level semantic segmentation is performed, the method includes organizing full-range hyperspectral data corresponding to each RGB image in one or more band subsets with similar spectral features by a processing unit

[204] .

[0132] Furthermore, the method results in the processing unit

[204] extracting one or more spectral features and one or more spatial features from the full-range hyperspectral data corresponding to each RGB image based on the full-range hyperspectral data of the tissue. The full-range hyperspectral data is organized into band subsets with similar spectral features because different parts of the object of interest may have different features (e.g., different diseases have different parameters) and the extraction of one or more spectral and one or more spatial features makes it easier for the processing unit

[204] to determine one or more parameters related to at least one of the agricultural field, the health field, and similar fields in the next stage.

[0133] The method further includes the processing unit

[204] determining one or more parameters associated with at least one in the agricultural and health fields based on one or more extracted spectral features, one or more extracted spatial features, and a third pre-training dataset. The third pre-training dataset includes multiple datasets trained on hyperspectral and RGB depth data associated with multiple events associated with at least one in the agricultural and health fields. In one embodiment, the one or more parameters are determined based on one or more deep learning / ML-based recognition techniques. Furthermore, in an example, the third pre-training dataset may include hyperspectral and RGB depth data of multiple health conditions of humans / animals indicating different recovery stages, health / nutrition mapping data, disease progression mapping data, etc. In such an embodiment, the method includes the processing unit

[204] mapping one or more hyperspectral signal patterns (i.e., one or more extracted spectral features and one or more extracted spatial features) of an RGB image of a disease to each health condition parameter (e.g., body temperature pattern, oxygen level pattern, fungal disease, bacterial disease, etc.) in the third pre-training dataset to determine one or more parameters associated with the disease, thereby further predicting one or more health conditions. Furthermore, in one instance, the method includes a processing unit

[204] using a computer vision generative adversarial network (GAN) to generate each hyperspectral signal pattern using RGB patterns, i.e., RGB depth data contained in a third pre-trained dataset. Therefore, various services can be provided in the healthcare industry based on one or more disease-related parameters determined by mapping hyperspectral signal patterns to health status parameters.

[0134] Furthermore, once one or more parameters related to at least one of agriculture, health, and other such fields are determined, the method includes transmitting the one or more parameters by the transceiver unit

[202] to the user equipment to further display charts indicating information related to one or more of the agriculture, health, and other such fields. In one embodiment, the method includes the transceiver unit

[202] observing one or more changes occurring in the storage unit

[206] to transmit updated information to the user equipment. Thus, any changes, including but not limited to adding information from one or more camera devices / MEMS sensors, are copied / synchronized on the user equipment. Additionally, in one embodiment, the user equipment periodically performs device calibration on readings collected from the transceiver unit

[202] and displays charts.

[0135] The method then terminates at step

[418] .

[0136] Refer to Figure 5 (i.e.) Figure 5a Figures 5b) illustrate an exemplary use case of extracting full-range hyperspectral data from one or more RGB images according to an exemplary embodiment of the present invention.

[0137] More specifically, Figure 5a

[502] indicates the raw RGB image received from the camera device.

[0138] At

[504] , a preprocessed image corresponding to the received original image is shown, wherein the preprocessing is performed at least by cropping the received image to the region of interest.

[0139] Next, at

[506] , the illumination component is removed from the preprocessed image to obtain the actual / true color.

[0140] Furthermore, at

[508] , hyperspectral data corresponding to the preprocessed image is determined, and 18 bands of a ground-based hyperspectral 31-band image are shown as an example. More specifically, each band depicts a spectral channel from 400 nm to 700 nm, for example, 400, 410, 420, 430, 440, 450, 460, 470, 480, 490, 500, 510, 520, 530, 540, 550, 560, 570 nm...700 nm. Thus, each band has an increment of 10 nm.

[0141] Next, in Figure 5b At

[510] , 18 bands of the reconstructed hyperspectral 31-band image are shown as an example to extract one or more spectral features and one or more spatial features.

[0142] refer to Figure 6An exemplary process

[600] according to an exemplary embodiment of the present invention is shown, which indicates a use case for crop classification based on extracting full-range hyperspectral data from one or more RGB images.

[0143] In step

[602] , the method includes capturing one or more RGB images or burst videos of a tree by a user device.

[0144] Next, in step

[604] , two RGB images / frames containing the tree are depicted at times t=0 and t=1.

[0145] Furthermore, in step

[606] , the method includes estimating and removing the illumination component from the two RGB images of the tree.

[0146] Subsequently, in step

[608] , two RGB images / frames of illumination component removal are described.

[0147] Next, in step

[610] , the method includes performing pixel tracking across frames using an optical flow model.

[0148] In addition, in step

[612] , the method includes cropping and stacking two RGB images / frames.

[0149] Next, in step

[614] , two cropped RGB images of the object of interest, namely the tree, are indicated. Furthermore, block-level aggregation of motion is then implemented.

[0150] Furthermore, in step

[616] , the conversion from RGB to hyperspectral data is completed at least based on block-level aggregation of motion, pixel tracking across frames, and removal of illumination components.

[0151] Next, in step

[618] , hyperspectral data / images corresponding to the two RGB images / frames are depicted.

[0152] Furthermore, in step

[620] , the method includes performing pixel-level semantic segmentation on hyperspectral data corresponding to two RGB images.

[0153] Next, in step

[622] , one or more crop-related parameters are determined based on semantic segmentation and the pre-trained dataset (i.e., the third pre-trained dataset) to complete crop classification. After crop classification is completed, the results are transmitted to the user device.

[0154] As can be clearly seen from the above disclosure, this invention provides a novel solution for extracting full-range hyperspectral data from one or more RGB images using low-cost, high-resolution ground-based equipment with the assistance of artificial intelligence. The AI ​​can automatically construct and map parameters from existing RGB images to the hyperspectral images. Furthermore, this invention provides a solution for mapping full-range hyperspectral data corresponding to input RGB data to optimally estimated sensor values ​​to determine one or more parameters relevant to various fields. Such parameters are plotted in the user equipment to identify various conditions, such as in agriculture, including areas where sufficient fertilization has occurred, areas receiving high photosynthetically active radiation, humidity, temperature, and overall crop productivity.

[0155] While this document focuses on preferred embodiments, it should be understood that many embodiments and changes to the preferred embodiments can be made without departing from the principles of the invention. These and other variations in the preferred embodiments will be apparent to those skilled in the art from the disclosure herein, and it should be clearly understood that the foregoing description is illustrative only and not limiting.

Claims

1. A method for extracting full-range hyperspectral data from a plurality of RGB images, the method comprising: - receiving, at a transceiver unit (202), a plurality of RGB images associated with different frames from one or more camera devices; - pre-processing, by a processing unit (204), the plurality of RGB images, wherein the pre-processing comprises at least one of resizing the plurality of RGB images, denoising the plurality of RGB images; - estimating, by the processing unit (204), an illumination component associated with each pre-processed RGB image of the plurality of pre-processed RGB images based on a first pre-trained dataset; - removing, by the processing unit (204), the illumination component from the each pre- processed RGB image to determine actual RGB values of one or more pixels of each pre- processed RGB image, wherein the determination of the actual RGB values of one or more pixels comprises retrieving illumination-independent RGB pixel values of one or more pixels of each pre-processed RGB image and determining the actual RGB values of one or more pixels based on the illumination-independent RGB pixel values of one or more pixels of each pre-processed RGB image; - tracking, by the processing unit (204), a trajectory of the one or more pixels across a plurality of frames associated with the each pre-processed RGB image based on an optical flow model to obtain an accurate hyperspectral representation of each pre-processed RGB image; - identifying, by the processing unit (204), a location of the one or more pixels in one or more adjacent frames of the plurality of frames based on a block defined around the one or more pixels; and - extracting, by the processing unit (204), the full-range hyperspectral data from the each pre-processed RGB image of the plurality of RGB images based on at least one of the removal of the illumination component, the trajectory of the one or more pixels, and the location of the one or more pixels.

2. The method of claim 1, wherein, The one or more camera devices comprise one or more micro-electro-mechanical systems (MEMS).

3. The method of claim 1, wherein, The plurality of RGB images are received at the transceiver unit (202) via a master node associated with the one or more camera devices.

4. The method of claim 1, wherein, The extraction of the full-range hyperspectral data by the processing unit (204) is further based on a second pre-trained dataset, wherein the second pre-trained dataset comprises a plurality of data trained based on converting a plurality of RGB images frame-by-frame to a corresponding hyperspectral level resolution, wherein the extraction of the full-range hyperspectral data based on the second pre-trained dataset comprises mapping a pre-processed RGB image to the second pre-trained dataset to further extract a corresponding full-range hyperspectral data.

5. The method of claim 1, wherein, The first pre-trained dataset comprises a plurality of data trained based on a depth value associated with each object captured in each image of the plurality of RGB images.

6. The method of claim 5, wherein, The estimation, by the processing unit (204), of the illumination component associated with each pre-processed RGB image further comprises: - assigning, by the processing unit (204), a depth value to each RGB pixel associated with the each pre-processed RGB image based on the first pre-trained dataset, - synthesizing, by the processing unit (204), one or more images of one or more objects captured in the each pre-processed RGB image under one or more lighting conditions based on the depth value assigned to the each RGB pixel, and - estimating, by the processing unit (204), a lighting component associated with the each pre-processed RGB image based on the synthesized one or more images of the one or more objects, wherein the lighting component is estimated on a pixel-level scale.

7. The method of claim 6, wherein, The estimating, by the processing unit (204), the lighting component is further based on one or more artificial intelligence techniques.

8. The method of claim 1, wherein, The method further comprises performing, by the processing unit (204), a pixel-level semantic segmentation on an object of interest present in the full-range hyperspectral data corresponding to each RGB image of the plurality of RGB images.

9. The method of claim 8, wherein, The performing the pixel-level semantic segmentation is based on one or more artificial intelligence techniques.

10. The method of claim 8, wherein, The method further comprises: - organizing, by the processing unit (204), the full-range hyperspectral data corresponding to the each RGB image in one or more subsets of wavebands having similar spectral characteristics, and - extracting, by the processing unit (204), one or more spectral and one or more spatial features from the full-range hyperspectral data corresponding to the each RGB image based on the organizing.

11. The method of claim 10, wherein, The method further comprises determining, by the processing unit (204), one or more parameters related to at least one of the fields of agriculture and health based on the one or more extracted spectral features, the one or more extracted spatial features, and a third pre-trained dataset, wherein the third pre-trained dataset comprises a plurality of data trained based on hyperspectral data and RGB depth data associated with a plurality of events associated with at least one of the fields of agriculture and health.

12. A system for extracting full-range hyperspectral data from a plurality of RGB images, the system comprising: - a transceiver unit (202) configured to receive a plurality of RGB images associated with different frames from one or more camera devices; and - a processing unit (204) configured to: pre-process the plurality of RGB images, wherein the pre-processing comprises at least one of resizing the plurality of RGB images, denoising the plurality of RGB images, estimate a lighting component associated with each pre-processed RGB image of the plurality of pre-processed RGB images based on a first pre-trained dataset, removing the illumination component from the each pre-processed RGB image to determine actual RGB values of one or more pixels of each pre-processed RGB image, wherein the determination of the actual RGB values of one or more pixels comprises retrieving illumination-independent RGB pixel values of one or more pixels of each pre-processed RGB image and determining the actual RGB values of one or more pixels based on the illumination-independent RGB pixel values of one or more pixels of each pre-processed RGB image, tracking a trajectory of the one or more pixels across a plurality of frames associated with the each pre-processed RGB image based on an optical flow model to obtain an accurate hyperspectral representation of each pre-processed RGB image, identifying a location of the one or more pixels in one or more adjacent frames of the plurality of frames based on a block defined around the one or more pixels, and extracting the full-range hyperspectral data from the each pre-processed RGB image of the plurality of RGB images based on at least one of the removal of the illumination component, the trajectory of the one or more pixels, and the location of the one or more pixels.

Images