A method and electronic device for image recognition

Abstract

This application provides an image recognition method and electronic device. The method includes: acquiring a first image containing at least one target substance, the first image being an image of a first spectral band; the first spectral band being a low-resolution spectral band; reconstructing the spectral band of the first image to obtain a second image, the second image being an image of a second spectral band; the second spectral band being a high-resolution spectral band with wavelengths in the visible and infrared regions; recognizing the composition information of at least one target substance in the second image; and displaying the composition information of at least one target substance. In this method, sufficient and effective spectral information can be obtained through the reconstructed high-resolution spectral band second image, thus enabling more accurate identification of the composition of at least one target substance in the image when recognizing its composition information.

Description

Technical Field

[0001] This application relates to the field of image recognition technology, and in particular to an image recognition method and electronic device. Background Technology

[0002] Currently, images captured by ordinary cameras are generally unsuitable for identifying the composition of target substances within them, thus failing to meet current requirements. Therefore, in practical work or production, higher-resolution images are often needed to obtain compositional information of target substances, thereby identifying their components and content. For example, by identifying the compositional information of fruit in an image, one can determine the fruit's composition and sugar content.

[0003] Compared to RGB images acquired by ordinary cameras, hyperspectral images are multi-channel images subdivided across the wavelength domain. Therefore, when identifying components in hyperspectral images, more surface or intrinsic information about the target substance can be obtained. For example, by acquiring the spectral information of a hyperspectral image, the vibrational information of specific chemical groups (such as OH, CH, NH bonds) of the target substance in the image can be analyzed. However, the acquisition of hyperspectral images is often affected by factors such as the acquisition equipment and acquisition time, resulting in low-resolution spectral images. Sufficient and effective spectral information cannot be obtained from low-resolution images, making it difficult to accurately identify the composition of the target substance in the image. Therefore, hyperspectral reconstruction techniques are needed to reconstruct high-resolution spectral images from low-resolution images to obtain more effective spectral information.

[0004] Current hyperspectral reconstruction techniques primarily involve reconstructing RGB images to obtain hyperspectral images in the visible spectrum. In these reconstructed images, the wavelength range is mainly concentrated in the visible light region. However, reconstructing these images does not yield sufficiently effective spectral information. Therefore, when identifying the composition of target substances within these hyperspectral images, the problem of accurately identifying their components remains. Summary of the Invention

[0005] This application provides an image recognition method and electronic device for accurately identifying target substance composition information in an image.

[0006] In a first aspect, embodiments of the present invention provide an image recognition method applicable to electronic devices with image capture and recognition functions. The method involves acquiring a first image containing at least one target substance, wherein the first image is an image of a first spectral band; the first spectral band is a low-resolution spectral band; spectral band reconstruction is performed on the first image to obtain a second image, wherein the second image is an image of a second spectral band; the second spectral band is a high-resolution spectral band with wavelengths in the visible and infrared regions; the composition information of the at least one target substance is obtained by recognizing the second image; and the composition information of the at least one target substance is displayed.

[0007] This design first acquires a low-resolution hyperspectral image containing at least one target substance. Then, the low-resolution hyperspectral image is reconstructed to obtain a high-resolution hyperspectral image. The wavelength range of the high-resolution hyperspectral image is mainly concentrated in the visible and infrared regions. Therefore, sufficient spectral information can be obtained from the reconstructed high-resolution hyperspectral image. When identifying the composition of at least one target substance in the reconstructed high-resolution hyperspectral image, the composition of at least one target substance in the image can be identified more accurately.

[0008] In one possible design, spectral reconstruction of the first image to obtain a second image includes: acquiring a first spectral reconstruction model, and performing spectral reconstruction of the first image based on the first spectral reconstruction model to obtain a second image; wherein the first spectral reconstruction model is obtained by adding a penalty term of multi-channel supervision information from a color temperature sensor to a pre-trained second spectral reconstruction model; the second spectral reconstruction model is trained from multiple second images; the multiple third images are images acquired by a first acquisition device and a second acquisition device; the first acquisition device is an RGB camera, and the second acquisition device is a TOF camera and / or a color temperature sensor; the penalty term of the multi-channel supervision information from the color temperature sensor is used to adjust the channels of the second spectral reconstruction model.

[0009] This design reconstructs the spectral bands of a first low-resolution spectral image using the first spectral band reconstruction model, resulting in a second high-resolution spectral image. The first spectral band reconstruction model is obtained by adding a penalty term with multi-channel supervision information from a color temperature sensor to a pre-trained second spectral band reconstruction model. This pre-trained second spectral band reconstruction model is trained on images acquired by a first and a second acquisition device (an RGB camera and a TOF camera and / or a color temperature sensor). Therefore, the second spectral band reconstruction model trained on the images acquired by the first and second acquisition devices can be used to reconstruct a high-resolution hyperspectral image. Furthermore, by adding a penalty term with multi-channel supervision information from a color temperature sensor to the second spectral band reconstruction model, the first spectral band reconstruction model becomes more accurate, exhibiting higher reconstruction precision and better adaptability.

[0010] In one possible design, identifying the composition information of the at least one target substance by recognizing the second image includes: acquiring the spectral information of the second image, wherein the spectral information is the spectral information corresponding to the spectral segments of the second image; using a convolutional neural network model to identify the spectral information of the second image to determine the chemical bonds of the at least one target substance; and determining the composition information of the at least one target substance based on the chemical bonds of the at least one target substance.

[0011] This design obtains the spectral information of the reconstructed second image. By using a convolutional neural network model to identify the spectral information, the chemical bonds of at least one target substance in the second image can be determined. Based on the chemical bonds, the components of at least one target substance and related information about those components can be analyzed.

[0012] In one possible design, before using a convolutional neural network model to identify the spectral information of the second image, the method further includes: obtaining a first spectrum corresponding to a spectral segment of the second image and obtaining a second spectrum corresponding to a spectral segment of a fourth image based on the spectral information of the second image; wherein the fourth image is an image obtained by a color temperature sensor collecting data on the at least one target substance; and adjusting the spectral information in the second image when the error value between the first spectrum and the second spectrum is greater than a set threshold based on the first spectrum and the second spectrum.

[0013] This design involves collecting the spectrum of a color temperature sensor and comparing the spectrum of the second image with that of the color temperature sensor. If the error between the two spectra is large, it indicates that the reconstruction of the second image is inaccurate, and the spectrum of the reconstructed second image needs to be adjusted to ensure that the spectrum of the second image is consistent with that of the color temperature sensor.

[0014] In one possible design, adjusting the spectral information in the second image includes: obtaining N spectral values ​​corresponding to N channels in the second image based on the first spectrum; obtaining N spectral values ​​corresponding to N channels in the fourth image based on the second spectrum; wherein the value of N is greater than or equal to 1; wherein the N channels in the second image and the N channels in the fourth image have a one-to-one correspondence; and adjusting the N channels in the second image based on the N spectral values ​​corresponding to the N channels in the second image and the N spectral values ​​corresponding to the N channels in the fourth image.

[0015] With this design, when it is determined that the spectrum of the second image needs to be adjusted, the spectral values ​​of each channel in the second image and the spectral values ​​of each channel in the fourth image collected by the color temperature sensor are obtained. Then, based on the spectral values ​​of N channels in the two types of images, the N channels in the second image are adjusted to ensure the accuracy of the spectrum of each channel in the adjusted second image.

[0016] In one possible design, adjusting the N channels in the second image includes: for the i-th channel among the N channels, determining the spectral value corresponding to the i-th channel in the second image as L. i The spectral value corresponding to the i-th channel in the fourth image acquired by the color temperature sensor is l i , i takes any integer value from 1 to N; when the L is determined i With l i When the absolute value of the difference is greater than the set threshold, the i-th channel in the second image is adjusted.

[0017] Through this design, for any one of the N channels in the second image, the spectral value corresponding to that channel and the spectral value corresponding to that channel in the fourth image acquired by the color temperature sensor are determined. Then, if the error range between the two values ​​is greater than a set threshold, it indicates that the spectrum of that channel in the second image is inaccurate. The channel in the second image is then adjusted to ensure the accuracy of the spectrum of that channel.

[0018] Secondly, this application provides an image recognition apparatus that has the function of implementing the method described in the first aspect or any possible design of the first aspect. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-described function, such as a display unit, a communication unit, and a processing unit.

[0019] Thirdly, this application also provides a computer storage medium storing a software program, which, when read and executed by one or more processors, can implement the method provided in the first aspect or any of the designs described above.

[0020] Fourthly, embodiments of this application also provide a computer program product containing instructions that, when executed on a computer, cause the computer to perform the method provided in the first aspect or any of the designs described above.

[0021] Fifthly, embodiments of this application provide a chip system including a processor for supporting devices to implement the functions involved in the first aspect above.

[0022] In one possible design, the chip system further includes a memory for storing necessary program instructions and data. The chip system can be composed of chips or may include chips and other discrete devices.

[0023] In a sixth aspect, this application provides a chip system including a processor and an interface, the interface being used to acquire a program or instructions, and the processor being used to invoke the program or instructions to implement or support the device in implementing the functions involved in the first aspect, such as determining or processing at least one of the data and information involved in the above methods.

[0024] In one possible design, the chip system further includes a memory for storing program instructions and data necessary for the electronic device. The chip system can be composed of chips or may include chips and other discrete components.

[0025] In a seventh aspect, embodiments of this application also provide an interface, wherein the device has a display screen, a memory, and a processor, the processor being configured to execute a computer program stored in the memory, and the interface including an interface displayed when the device performs the method described in the first or third aspect.

[0026] The technical effects that can be achieved in the second to seventh aspects mentioned above can be referred to the description of the technical effects that can be achieved in the first aspect or any of the designs mentioned above, and will not be repeated here. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention;

[0028] Figure 2 This is an application scenario diagram illustrating the applicable scenarios for an image recognition method provided in this embodiment of the invention;

[0029] Figure 3 This is a schematic diagram of an image recognition method flow provided in an embodiment of the present invention;

[0030] Figure 4 This is a schematic diagram of an embodiment of an image recognition method provided by the present invention;

[0031] Figure 5 This is a schematic diagram showing the distribution of a target substance according to an embodiment of the present invention. Detailed Implementation

[0032] The technical solutions in the embodiments of this application will now be described with reference to the accompanying drawings.

[0033] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0034] 1) The target substance involved in the embodiments of this application refers to a specific physical object containing different components and contents, such as apples, bananas, etc.

[0035] 2) The images involved in the embodiments of this application include ordinary images and hyperspectral images. The ordinary images are images captured by ordinary cameras or the camera of electronic devices. The hyperspectral images are multi-channel images subdivided in the wavelength domain. For example, they are images of the target area simultaneously in multiple continuous and subdivided spectral bands in the ultraviolet, visible, near-infrared, and mid-infrared regions of the electromagnetic spectrum using hyperspectral sensors mounted on different space platforms, i.e., imaging spectrometers. Therefore, hyperspectral images can acquire more information about the surface or internal structure of materials, as well as spectral information.

[0036] 3) The reconstruction of high-spectral-band hyperspectral images involved in the embodiments of this application is mainly aimed at low-resolution high-spectral-band hyperspectral images. Since the spectral information in low-resolution high-spectral-band hyperspectral images is usually less and blurry, it cannot be used to accurately identify the material composition in the image. Thus, the low-resolution high-spectral-band hyperspectral image is reconstructed into a high-resolution high-spectral-band hyperspectral image. The spectral information in the high-resolution high-spectral-band hyperspectral image is relatively more and more accurate, and the material composition in the image can be identified more accurately.

[0037] 4) The RGB (Red, Green, Blue) camera involved in this application embodiment mainly uses three different cables to provide the three basic color components (red, green, and blue). Typically, such cameras use three independent charge-coupled device (CCD) image sensors to acquire the three color signals. Color images can be obtained through an RGB camera, but the resolution and clarity of the images obtained through an RGB camera are limited.

[0038] 5) The Time-of-Flight (TOF) camera involved in this application embodiment is a true-depth sensing lens used for 3D recognition. It utilizes the TOF measurement principle (TOF image sensor) to determine the distance between the camera and an object or the surrounding environment, and generates a depth image or 3D image based on the measured points. Specifically, the TOF measurement principle involves a modulated light emitter emitting high-frequency light, which reflects back after hitting an object. The receiver acquires the round-trip time and calculates the distance to the object. Light travels at different depths of field, and this time difference allows for the formation of a high-precision 3D image, which is then compared to complete the recognition. TOF cameras are commonly used in applications including laser-based non-scanning lidar imaging systems, motion sensing and tracking, object detection in machine vision and autonomous driving, and terrain mapping.

[0039] 6) The spectral information involved in the embodiments of this application, simply put, is the composition information of light. Light can be classified according to the energy corresponding to photons (usually represented by wavelength). From shortest to longest wavelength, there are three regions: ultraviolet light, visible light, and infrared light. Different colors of visible light correspond to different wavelengths. A spectrum is the proportional information of each wavelength component in light. In a typical representation, the horizontal axis represents wavelength, and the vertical axis represents relative intensity. Spectral information can be used to identify and determine the chemical composition and relative content of substances.

[0040] 7) The multiple mentioned in the embodiments of this application refers to two or more.

[0041] In addition, it should be understood that in the description of this application, the words "first" and "second" are used only for the purpose of distinguishing descriptions and should not be construed as indicating or implying relative importance or order.

[0042] This application provides an identification method applicable to various electronic devices, such as mobile phones, cameras, and tablet computers. Figure 1 A structural diagram of a possible electronic device is shown. (See also...) Figure 1As shown, the electronic device 100 includes components such as a radio frequency (RF) circuit 101, a power supply 102, a processor 103, a memory 104, an input unit 105, a display screen 106, a camera 107, a sensor 108, a communication interface 109, and a wireless fidelity (WiFi) module 110. Those skilled in the art will understand that... Figure 1 The structure of the electronic device shown in the figure does not constitute a limitation on the electronic device. The electronic device provided in the embodiments of this application may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0043] The following combination Figure 1 The various components of the electronic device 100 are described in detail below:

[0044] The RF circuit 101 can be used for receiving and transmitting data during communication or a call. Specifically, after receiving downlink data from the base station, the RF circuit 101 sends it to the processor 103 for processing; additionally, it sends uplink data to be transmitted to the base station. Typically, the RF circuit 101 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low-noise amplifier (LNA), a duplexer, etc.

[0045] In addition, the RF circuit 101 can also communicate wirelessly with networks and other devices. The wireless communication can use any communication standard or protocol, including but not limited to Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), email, and Short Messaging Service (SMS).

[0046] WiFi technology is a short-range wireless transmission technology. The electronic device 100 can connect to an access point (AP) via the WiFi module 110, thereby enabling access to the data network. The WiFi module 110 can be used for receiving and sending data during communication.

[0047] The electronic device 100 can physically connect to other devices through the communication interface 109. Optionally, the communication interface 109 can be connected to the communication interfaces of other devices via a cable to enable data transmission between the electronic device 100 and other devices.

[0048] Since the electronic device 100 in this embodiment can be used to implement communication services and send information to other electronic devices, the electronic device 100 needs to have data transmission capabilities, that is, the electronic device 100 needs to contain a communication module. Although Figure 1 The RF circuit 101, the WiFi module 110, and the communication interface 109 are shown, but it is understood that the electronic device 100 contains at least one of the above-mentioned components or other communication modules (such as a Bluetooth module) for data transmission.

[0049] For example, when the electronic device 100 is a mobile phone, the electronic device 100 may include the RF circuit 101 and the WiFi module 110; when the electronic device 100 is a computer, the electronic device 100 may include the communication interface 109 and the WiFi module 110; when the electronic device 100 is a tablet computer, the electronic device 100 may include the WiFi module.

[0050] The memory 104 can be used to store software programs and modules. The processor 103 executes various functional applications and data processing of the electronic device 100 by running the software programs and modules stored in the memory 104.

[0051] Optionally, the memory 104 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, various application programs, etc.; the data storage area may store multimedia files such as pictures and videos.

[0052] In addition, the memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0053] The input unit 105 can be used to receive user input of images, numbers or characters, or images captured by the camera 107, and to generate key signal inputs related to user settings and function control of the electronic device 100.

[0054] Optionally, the input unit 105 may include a touch panel 1051 and other input devices 1052.

[0055] The touch panel 1051, also known as a touch screen, can collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel 1051), and drive corresponding connection devices according to a pre-set program. Optionally, the touch panel 1051 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch position and the signal generated by the touch operation, and transmits the signal to the touch controller; the touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 103, and can also receive and execute commands from the processor 130. Furthermore, the touch panel 1051 can be implemented using various types of touch technologies, such as resistive, capacitive, infrared, and surface acoustic wave.

[0056] Optionally, the other input device 1052 may include, but is not limited to, one or more of the following: physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.

[0057] The display screen 106 can be used to display information input by the user or information provided to the user, as well as various menus of the electronic device 100. The display screen 106 is the display system of the electronic device 100, used to present the interface and realize human-computer interaction.

[0058] The display screen 106 may include a display panel 1061. Optionally, the display panel 1061 may be configured as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

[0059] Furthermore, the touch panel 1051 may cover the display panel 1061. When the touch panel 1051 detects a touch operation on or near it, it transmits the information to the processor 103 to determine the type of touch event. Subsequently, the processor 103 provides corresponding visual output on the display panel 1061 according to the type of touch event.

[0060] Although Figure 1 In this embodiment, the touch panel 1051 and the display panel 1061 are two independent components to realize the input and output functions of the electronic device 100. However, in some embodiments, the touch panel 1051 and the display panel 1061 can be integrated to realize the input and output functions of the electronic device 100.

[0061] The processor 103 is the control center of the electronic device 100. It connects various components through various interfaces and lines. By running or executing software programs and / or modules stored in the memory 104, and calling data stored in the memory 104, it performs various functions of the electronic device 100 and processes data, thereby realizing various services based on the electronic device.

[0062] Optionally, the processor 103 may include one or more processing units. Optionally, the processor 103 may integrate an application processor and a modem processor, wherein the application processor primarily handles the operating system, user interface, and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 103.

[0063] The camera 107 is used to enable the electronic device 100 to take pictures or videos. The camera 107 can also be used to enable the electronic device 100 to scan objects (QR codes / barcodes).

[0064] It should be noted that in this embodiment of the application, the camera 107 can be an RGB camera or a TOF camera.

[0065] RGB cameras are used to capture target materials and can obtain color images composed of RGB colors. TOF cameras can be used to capture target materials and generate depth images or 3D images.

[0066] The sensor 108 may include one or more sensors. For example, a touch sensor 1082, a color temperature sensor 1081, etc. In other embodiments, the sensor 108 may also include one or more of the following: a gyroscope, an accelerometer, a fingerprint sensor, an ambient light sensor, a distance sensor, a proximity sensor, a bone conduction sensor, a pressure sensor, a positioning sensor (such as a global positioning system (GPS) sensor), etc., without limitation.

[0067] The touch sensor 1082, also known as a "touch panel," can be used to collect user touch operations on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch sensor 1082), and drive corresponding connected devices according to a pre-set program. Optionally, the touch sensor 1082 may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 103. It can also receive and execute commands from the processor 103. Furthermore, the touch sensor 1082 can be implemented using various types, including resistive, capacitive, infrared, and surface acoustic wave sensors.

[0068] When the touch sensor 1082 is disposed on the display screen 106, the touch sensor 1082 and the display screen 106 constitute a touchscreen, also known as a "touchscreen". The touch sensor 1082 is used to detect touch operations applied to or near it. The touch sensor 1082 can transmit the detected touch operation to the application processor to determine the type of touch event, so that the electronic device 100 can provide visual output related to the touch operation through the display screen 106. For example, the electronic device 100 can switch interfaces in response to the touch sensor 1082 detecting a touch operation applied to or near it, and display the switched interface on the display screen 106. In some embodiments, the touch sensor 1082 may also be disposed on the surface of the electronic device 100, in a different position than the display screen 106.

[0069] A color temperature sensor 1081 is used to detect the ambient color temperature when the camera 107 captures a photo, obtain a color temperature signal, and convert the color temperature signal into an electrical signal. For example, the color temperature sensor 1081 can be located on the display screen 106. Based on the detected ambient light color temperature, the processor 103 receives the color temperature electrical signal and then performs color temperature adjustment to ensure that the color temperature is consistent with the ambient light level, thereby making the colors of the photos captured by the camera 107 more accurate.

[0070] The electronic device 100 also includes a power supply 102 (such as a battery) for supplying power to various components. Optionally, the power supply 102 can be logically connected to the processor 103 through a power management system, thereby enabling the power management system to manage functions such as charging, discharging, and power consumption.

[0071] It should be noted that, although not shown, the electronic device 100 may also include audio circuitry, etc., which will not be described in detail here.

[0072] Before introducing the image recognition method of the embodiments of this application, let's first introduce an application scenario to which the embodiments of this application are applicable, for example, Figure 2 A schematic diagram illustrating an application scenario of an embodiment of this application is shown.

[0073] like Figure 2 As shown, taking a mobile phone as an example, this phone is a smartphone capable of capturing hyperspectral images and spectra. The user turns on the smartphone's camera and takes a picture of a fruit, such as an apple, to obtain a hyperspectral image of the apple. The captured hyperspectral image of the apple is automatically displayed on the left side of the smartphone's screen. The hyperspectral image includes the image of the apple and its corresponding components and their contents. For example, the components include: water, sugar, protein (g), fiber (g), etc., with corresponding contents of 85%, 12.3%, 0.2g, 1.7g, etc.

[0074] Typically, by acquiring hyperspectral images of fruits and obtaining spectral information, it is possible to analyze the vibrational information of specific chemical groups in a substance (such as OH, CH, NH bonds) to determine the composition of the substance.

[0075] However, in actual collection of hyperspectral images of fruits, factors such as the acquisition equipment and time often affect the acquisition, resulting in low-resolution spectral images. These low-resolution images cannot provide sufficient and effective spectral information, leading to inaccurate or missing identification of substance composition and content. Therefore, hyperspectral reconstruction technology is needed to reconstruct high-resolution spectral images from low-resolution images to obtain more spectral information.

[0076] Existing hyperspectral image reconstruction techniques typically target single RGB images, employing either general convolutional neural networks or convolutional neural networks with added residual connections or dense modules for image reconstruction, or GAN-based generative techniques for hyperspectral image reconstruction.

[0077] During the reconstruction process, it is necessary to collect RGB-hyperspectral stereo (Hypercube) pairs of the same object to train the reconstruction network. Since the reconstruction network is usually based on convolution, residual connections or denser Dense modules are introduced to tightly fuse the features of each layer, thereby completing the mapping from a limited 3-channel to N-channel and completing the reconstruction.

[0078] However, in this hyperspectral reconstruction process, the reconstruction is mainly performed on a single RGB image. Since the wavelength range of this image is concentrated in the visible light region, while the effective information of more substances is concentrated in the near-infrared region, the reconstruction of a single RGB image cannot obtain sufficient substance information. Moreover, it is often limited by the initial acquisition dataset, which may lead to deviations in the actual reconstruction process, thus giving incorrect substance information (for example, when the object is a different substance of the same color).

[0079] For example, neural networks and near-infrared spectroscopy can be used to detect the composition of substances (e.g., the sugar content of fruits). This method includes: First step: Selecting a sample set of fruits of the same type, randomly dividing it into a calibration set and a prediction set; Second step: Collecting the raw near-infrared spectra of all samples, dividing the spectra into equal intervals, and summing the absorbance of each interval; Third step: Determining the sugar content in the samples using chemical analysis; Fourth step: Constructing a quantitative calibration model between the sugar content of the calibration set samples and the near-infrared characteristic spectra using a backpropagation (BP) neural network; Fifth step: Inputting the near-infrared spectral information of the prediction set samples into the model to obtain the sugar content of the prediction set samples.

[0080] However, in this hyperspectral reconstruction process, when detecting the composition of substances, since the current spectrum is concentrated in the near-infrared region, only one-dimensional spectra acquired at specific points can be analyzed, yielding single-point results rather than two-dimensional material distribution. Furthermore, the traditional back-propagation (BP) network has relatively weak feature extraction capabilities and only identifies components of a single substance, failing to classify and detect general chemical bonds.

[0081] Therefore, this application provides an image recognition method. The method first acquires a first image containing at least one target substance, where the first image is an image of a first spectral band; the first spectral band is a low-resolution spectral band. Then, spectral reconstruction is performed on the first image to obtain a second image, where the second image is an image of a second spectral band; the second spectral band is a high-resolution spectral band with wavelengths in the visible and infrared regions. Finally, the composition information of the at least one target substance is obtained by recognizing the second image and displaying the composition information of the at least one target substance. In this scheme, spectral reconstruction is performed on the low-resolution segment image containing at least one target substance to obtain a high-resolution spectral band image with wavelengths in the visible and infrared regions. Therefore, through this high-resolution spectral band image, more effective spectral information can be obtained, thereby accurately identifying the composition information of at least one target substance in the high-resolution spectral band image.

[0082] Please see Figure 3This is a flowchart illustrating an image recognition method provided in an embodiment of this application. The method described in this embodiment can be applied to... Figure 1 The electronic device 100 shown is applicable to Figure 2 In the scenario shown. For example... Figure 3 As shown, the process of this method includes:

[0083] S301: The electronic device acquires a first image containing at least one target substance, the first image being an image of a first spectral band; the first spectral band being a low-resolution spectral band.

[0084] In one embodiment, the first image is a low-resolution hyperspectral image, which can be acquired by a specific acquisition device, including: an RGB camera + a TOF camera + a color temperature sensor (at least two of which must include an RGB camera).

[0085] The target substance can be a substance containing different amounts of components, and this application does not impose specific limitations. For example, the target substance can be a certain fruit, such as an apple or a banana.

[0086] For example, when an electronic device receives a user's shooting instruction, it can open its own camera and capture a first image under the user's control. For instance, if a user takes a picture of a flower or a fruit with their phone, the phone can obtain a first image containing at least one target substance. As another example, the electronic device can also receive first images from other devices, such as through a WiFi module or RF circuit, thereby also obtaining a first image containing at least one target substance.

[0087] S302: The electronic device performs spectral reconstruction on the first image to obtain a second image, which is an image of the second spectral band; the second spectral band is a high-resolution spectral band with wavelengths in the visible light region and the infrared light region.

[0088] In one implementation, when performing step S302, a first spectral band reconstruction model may be obtained first, and the first image may be reconstructed based on the first spectral band reconstruction model to obtain a second image; wherein, the first spectral band reconstruction model is obtained by adding a penalty term of multi-channel supervision information of a color temperature sensor to a pre-trained second spectral band reconstruction model; the second spectral band reconstruction model is trained from multiple third images; the multiple third images are images acquired by a first acquisition device and a second acquisition device; the first acquisition device is an RGB camera, and the second acquisition device is a TOF camera and / or a color temperature sensor; the penalty term of the multi-channel supervision information of the color temperature sensor is used to adjust the channels of the second spectral band reconstruction model.

[0089] For example, a hyperspectral image containing at least one target substance is input into the electronic device (the hyperspectral image has a low spectral resolution). The first hyperspectral reconstruction model obtained by the electronic device is used to reconstruct the hyperspectral image containing at least one target substance to obtain a reconstructed hyperspectral image of the target substance (the reconstructed hyperspectral image has a high spectral resolution, and the wavelength range of the reconstructed hyperspectral image is concentrated in the visible light region and the infrared light region).

[0090] For example, the first hyperspectral reconstruction model (e.g., a neural network for hyperspectral reconstruction) is used to reconstruct a hyperspectral image of an unknown fruit object in the 400-1000 nm (wavelength range).

[0091] It is important to note that, typically, the first spectral band reconstruction model is the first hyperspectral reconstruction model, and the second spectral band reconstruction model is the trained second hyperspectral reconstruction model. These models are used to reconstruct the spectral bands of a low-resolution hyperspectral image, thereby obtaining a high-resolution hyperspectral image.

[0092] In one implementation, the second hyperspectral reconstruction model can be pre-trained from multiple third images, specifically including but not limited to the following operations:

[0093] M hyperspectral images (i.e., third images) of the target substance are acquired using a first acquisition device and a second acquisition device. The first acquisition device can be an RGB camera, and the second acquisition device can be any one or a combination of a TOF camera and a color temperature sensor, where M is an integer greater than 1. The M hyperspectral images (i.e., third images) of the target substance are trained to obtain a trained second hyperspectral reconstruction model. A color temperature penalty term is added to the trained second hyperspectral reconstruction model to obtain the first hyperspectral reconstruction model. The penalty term of the multi-channel supervision information of the color temperature sensor is used to adjust the channels of the first hyperspectral reconstruction model.

[0094] For example, multiple hyperspectral image pairs of the target substance are acquired using a first acquisition device and a second acquisition device (the first acquisition device being an RGB camera, and the second acquisition device being a TOF camera and / or a color temperature sensor). These pairs contain the correspondence between images acquired by the RGB camera, TOF camera, or color temperature sensor and the hyperspectral images acquired by the first and second acquisition devices. For instance, the images acquired by the RGB camera and TOF camera are raw 4-channel images, covering the visible to near-infrared light region. Further, the acquired hyperspectral image pairs of the target substance are used to train a second hyperspectral image reconstruction model (e.g., a hyperspectral reconstruction neural network).

[0095] Among them, multimodal information from an RGB camera, a TOF camera, and a color temperature sensor (at least two types, of which the RGB camera must be included) can be used to reconstruct hyperspectral images.

[0096] In addition, since hyperspectral images are multi-channel images subdivided in the wavelength domain, the first hyperspectral reconstruction model can be constructed by adding a penalty term for the multi-channel supervision information of the color temperature sensor on the basis of training the second hyperspectral image reconstruction model.

[0097] In this step, multiple (here, "multiple" can be many, such as 1000, etc., and the value can be set relatively large) hyperspectral image pairs of substances are acquired using a first acquisition device (RGB camera), a second acquisition device (TOF camera and / or color temperature sensor), and a high-precision device. A hyperspectral image reconstruction model is then trained. A penalty term for multi-channel supervision information from the color temperature sensor is added to the trained hyperspectral image reconstruction model (e.g., a neural network for hyperspectral reconstruction) to ensure that the final constructed first hyperspectral reconstruction model has higher accuracy and better adaptability. The first hyperspectral reconstruction model is then used to reconstruct the low-resolution spectral images of the target substance to be identified, obtaining high-resolution spectral images of the target substance. These high-resolution spectral segments include high-resolution spectral bands with wavelengths in the visible and infrared regions. Therefore, through these high-resolution spectral images, more effective spectral information can be obtained, leading to more accurate determination of the target substance's composition information.

[0098] S303: The electronic device identifies the second image to obtain the composition information of the at least one target substance and displays the composition information of the at least one target substance.

[0099] Before performing step S303, the spectral information in the second image can also be adjusted using a color temperature sensor. Specific implementations include, but are not limited to, the following:

[0100] The electronic device can also first obtain the first spectrum corresponding to the spectral segment of the second image based on the spectral information of the second image, and then obtain the second spectrum corresponding to the spectral segment of the fourth image; the fourth image is an image obtained by the color temperature sensor in the electronic device from the at least one target substance; the electronic device can further adjust the spectral information in the second image when it determines that the error value between the first spectrum and the second spectrum is greater than a set threshold based on the first spectrum and the second spectrum.

[0101] In one embodiment, the electronic device adjusts the spectral information in the second image, and specific embodiments include, but are not limited to, the following:

[0102] First, based on the first spectrum, obtain N spectral values ​​corresponding to N channels in the second image, and based on the second spectrum, obtain N spectral values ​​corresponding to N channels in the fourth image; the value of N is greater than or equal to 1; wherein, there is a one-to-one correspondence between the N channels in the second image and the N channels in the fourth image; further, adjust the N channels in the second image based on the N spectral values ​​corresponding to the N channels in the second image and the N spectral values ​​corresponding to the N channels in the fourth image.

[0103] Furthermore, since each channel in the fourth image corresponds to a one-dimensional spectrum, while each channel in the second image corresponds to a multi-dimensional spectrum, it is necessary to process the spectral dimensions corresponding to each channel in the second image to obtain a one-dimensional spectrum. For example, for each channel in the second image, global average pooling can be performed on each channel to obtain a one-dimensional spectrum.

[0104] For example, based on the first spectrum in the second image, N spectral values ​​corresponding to N channels in the second image are obtained, including but not limited to the following: after processing the spectral dimension of each channel in the second image to obtain a one-dimensional spectrum, the spectral value corresponding to the one-dimensional spectrum obtained in each channel is used as the spectral value of the corresponding channel.

[0105] In one implementation, the electronic device adjusts N channels in the second image, and specific implementations include, but are not limited to, the following:

[0106] For example, for the i-th channel among the N channels, the i-th channel is adjusted by first determining the spectral value corresponding to the i-th channel in the second image as L. i And determine the spectral value corresponding to the i-th channel in the fourth image acquired by the color temperature sensor as l. i , i takes any integer value from 1 to N; when the L is determined i With the l i When the absolute value of the difference is greater than the set threshold, the i-th channel in the second image is adjusted.

[0107] It should be noted that when the L is determined i With l i When the absolute value of the difference is greater than the set threshold, it indicates that the error between the i-th channel in the reconstructed second image and the i-th channel in the image acquired by the color temperature sensor is relatively large, meaning that the reconstruction of the i-th channel in the second image is inaccurate, and the electronic device needs to adjust that channel. When it is determined that L... i With l iWhen the absolute value of the difference is less than the set threshold, it means that the error between the i-th channel in the reconstructed second image and the i-th channel in the image acquired by the color temperature sensor is relatively small, which means that the i-th channel in the second image is accurately reconstructed, and the electronic device does not need to adjust the channel.

[0108] Therefore, by adjusting N channels in the second image through the above steps, the spectrum of the second image can be adjusted, thereby ensuring the accuracy of the spectral information in the second image.

[0109] After adjusting the second image based on the color temperature sensor, step S303 is then executed.

[0110] In one embodiment, the electronic device can first acquire the spectral information of the adjusted second image, wherein the spectral information is the spectral information corresponding to the spectral segments of the adjusted second image; then, a convolutional neural network model is used to identify the spectral information of the second image to determine the chemical bonds of the at least one target substance; finally, based on the chemical bonds of the at least one target substance, the composition information of the at least one target substance is determined.

[0111] For example, the electronic device can specifically use a convolutional neural network to perform classification regression on the spectral dimensions in the adjusted second image based on the spectral information of the second image, determine the chemical bonds of the at least one target substance, determine the composition information of the at least one target substance, and analyze the composition information of the at least one target substance to obtain a distribution map of the composition of the at least one target substance.

[0112] For example, when one of the target substances is fruit, the internal chemical bond information can be determined through step S303, thereby obtaining the distribution of soluble solids in the fruit, such as fructose and blood oxygen.

[0113] Based on the image recognition method provided in the above embodiments, this application also provides an embodiment of an image recognition method. This method can be executed by an electronic device capable of supporting image capture and display. For example... Figure 4 As shown, the specific process is as follows.

[0114] a401: Electronic device acquires a first hyperspectral image to be reconstructed containing at least one target substance.

[0115] For example, the electronic device acquires a hyperspectral image of at least one fruit to be reconstructed through its camera, or the electronic device acquires a hyperspectral image of at least one fruit to be reconstructed from other electronic devices.

[0116] a402: The electronic device acquires the adjusted hyperspectral image reconstruction network, reconstructs the first hyperspectral image, and obtains the second hyperspectral image.

[0117] For example, the first hyperspectral image is reconstructed using an adjusted hyperspectral image reconstruction network to obtain a reconstructed second hyperspectral image of the unknown fruit.

[0118] Since the first hyperspectral image containing the target substance is a low-resolution hyperspectral image, an adjusted hyperspectral image reconstruction network is used to reconstruct the spectral bands of the hyperspectral image containing the target substance, resulting in a second hyperspectral image containing the target substance with a high-resolution spectral band, the wavelength range of which is in the visible and infrared ranges. Therefore, more and more effective spectral information can be obtained through this high-resolution hyperspectral image.

[0119] The adjusted hyperspectral image reconstruction network can be obtained through the following steps:

[0120] B1: Multiple composite RGBX four-channel image pairs are obtained by capturing images using an RGB camera and a TOF camera. For example, the RGB camera is the primary image capturer, and the TOF camera is the secondary image capturer. When the RGB camera (wavelength range of 400-780nm) has a resolution greater than 24 pixels (MegaPixel), it uses up to 240P. The OF camera (X: 960nm) also uses up to 240P.

[0121] The photos are taken by an RGB camera and a TOF camera, and the resulting images are composite RGBX four-channel images. Multiple images can be taken, and there is no specific limit here.

[0122] B2: The hyperspectral image reconstruction network is obtained through training.

[0123] Specifically, the multiple RGBX four-channel images obtained in step B1 are trained to obtain a trained hyperspectral image reconstruction network (equivalent to a hyperspectral image reconstruction model).

[0124] B3: The hyperspectral image reconstruction network is trained and a penalty term is added to the multi-channel supervision information of the color temperature sensor to obtain the adjusted hyperspectral image reconstruction model.

[0125] Since hyperspectral images are multi-channel images subdivided in the wavelength domain, a penalty term for multi-channel supervision information from a color temperature sensor is added to the hyperspectral image reconstruction model obtained after training. This allows for real-time monitoring and adjustment of the trained hyperspectral image reconstruction model, resulting in higher accuracy and better adaptability.

[0126] Steps B1-B3 form the training and adjustment framework for the hyperspectral image reconstruction model, so that the first hyperspectral image to be reconstructed, containing at least one target substance, can be reconstructed subsequently.

[0127] a403: The electronic device acquires the spectral information of the N-channel of the reconstructed second hyperspectral image.

[0128] a404: Perform average pooling on each channel to obtain a one-dimensional spectrum and get the corresponding spectral values.

[0129] Since the spectrum corresponding to each channel in the second hyperspectral image is multidimensional, it is necessary to process the spectral dimensions corresponding to each channel in the second hyperspectral image to obtain a one-dimensional spectrum. For example, for each channel in the second hyperspectral image, global average pooling can be performed on each channel to obtain a one-dimensional spectrum.

[0130] For example, based on the first spectrum in the second hyperspectral image, N spectral values ​​corresponding to N channels in the second hyperspectral image are obtained, specifically including the following: after processing the spectral dimension of each channel in the second hyperspectral image to obtain a one-dimensional spectrum, the spectral value corresponding to the one-dimensional spectrum obtained in each channel is used as the spectral value of the corresponding channel.

[0131] a405: The electronic device acquires a hyperspectral image of the at least one target substance through a color temperature sensor and obtains the spectral value of the N channel in the hyperspectral image.

[0132] Since the hyperspectral image of the at least one target substance acquired by the color temperature sensor contains a one-dimensional spectrum for each channel, the spectral value corresponding to each channel spectrum can be directly obtained.

[0133] a406: The electronic device calculates the error between the spectral value of each channel in the second hyperspectral image and the spectral value of each channel in the hyperspectral image obtained by the color temperature sensor.

[0134] For example, for the i-th channel out of N channels in the second hyperspectral image, the spectral value corresponding to the i-th channel in the second hyperspectral image is determined to be L. i The spectral value corresponding to the i-th channel in the hyperspectral image acquired by the color temperature sensor is l i Let i take any integer value from 1 to N, and calculate the L. i With l i The absolute value of the difference.

[0135] a407(1): When the error is greater than the set threshold, the spectral channels are adjusted in real time online by the color temperature sensor to obtain the adjusted hyperspectral stereo image and high-precision N-channel hyperspectral stereo.

[0136] For example, when determining the L i With l i When the absolute value of the difference is greater than the set threshold, the i-th channel in the second hyperspectral image is adjusted.

[0137] a407(2): If the error is less than the set threshold, then output directly.

[0138] For example, when determining the L i With l i If the absolute value of the difference is less than the set threshold, then the i-th channel in the second hyperspectral image will not be adjusted.

[0139] When it is determined that the absolute value of the difference between the spectral value of each channel in the second hyperspectral image and the spectral value of each channel in the hyperspectral image acquired by the color temperature sensor is less than a set threshold, the electronic device performs step a408 on the second hyperspectral image.

[0140] a408: The electronic device uses a convolutional neural network to analyze the chemical bond information contained in the one-dimensional spectrum of each pixel, determine the composition of the substance, and obtain a distribution map of the substance.

[0141] After steps a403-a407(2), the adjusted second hyperspectral image is obtained. Further, the electronic device performs step a408 on the adjusted second hyperspectral image.

[0142] For example, the electronic device uses a convolutional neural network to analyze the chemical bond information contained in the one-dimensional spectrum of each pixel in the adjusted second hyperspectral image, determines the composition of the first substance, and obtains a distribution map of the substance. For example, if the first substance is an apple, it obtains the fructose distribution in the apple; or if the first substance is blood, it obtains the blood oxygen distribution in the blood. High-precision N-channel hyperspectral stereo, wavelength range: 400-1000nm, wavelength resolution: 10nm.

[0143] For example, by using the method of this application embodiment to identify an image containing an apple, the resulting identification image can be referred to. Figure 5 As shown.

[0144] In summary, this application provides an image recognition method. This method uses a hyperspectral image reconstruction model to reconstruct the low-resolution spectral image of the target substance, obtaining a high-resolution hyperspectral image. The hyperspectral image reconstruction model is a model obtained by adding a color temperature penalty term constraint to a pre-trained hyperspectral image reconstruction model. This pre-trained model is trained on multiple image datasets acquired by an acquisition device (which includes RGB cameras, RGB cameras, and / or TOF cameras in addition to RGB cameras), thus ensuring that the improved hyperspectral image reconstruction model has higher accuracy and better adaptability. Furthermore, a color temperature sensor is used to further adjust the reconstructed high-resolution hyperspectral image, making the spectral information of the high-resolution hyperspectral image more accurate. Therefore, the final high-resolution hyperspectral image not only acquires more spectral information but also ensures the accuracy of the spectral information, thereby ensuring that the target substance composition information determined by the highly accurate spectral information is more abundant and accurate, and thus enabling more precise identification of the target substance composition in the image.

[0145] Based on the same concept, this application provides an electronic device, the specific structure of which can be referred to... Figure 2 The schematic diagram of the hardware structure of an electronic device 100 shown includes at least one processor 103, memory 104, display screen 106, camera 107, and transceiver (e.g., RF circuit 101 and wireless communication module 110).

[0146] The at least one processor 103 is interconnected with the transceiver 104, the display screen 106, and the camera 107. Optionally, the at least one processor 103 can be interconnected with the transceiver 104, the display screen 106, and the camera 107 via a bus; the bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc.

[0147] The transceiver is used for communication and interaction with other devices. For example, when the electronic device 100 captures an image of its current scene through an external camera device, the electronic device 100 uses the transceiver to obtain the image captured by the other external camera device as a first image. Optionally, the transceiver can be a Bluetooth module, a WiFi module 110, an RF circuit 101, etc.

[0148] Thus, the at least one processor 103 can acquire an image of at least one target substance as the first image by using the camera 107 within the electronic device 100, or it can acquire an image containing at least one target substance sent by another electronic device as the first image by using a transceiver, such as the RF circuit 101 or the transceiver 110.

[0149] The at least processor 103 is used to implement the above-mentioned... Figure 3 The embodiments and examples shown provide an image recognition method, which can be found in the description of the above embodiments and will not be repeated here.

[0150] The display screen 106 is used to display the image captured by the camera 107, or to display a distribution map of at least one target substance composition information obtained after image recognition.

[0151] Optionally, the electronic device 100 may also include audio circuitry for receiving and transmitting voice signals.

[0152] The memory 104 is used to store program instructions and data (e.g., hyperspectral image reconstruction models, acquired hyperspectral images, etc.). Specifically, the program instructions may include program code, which includes instructions for computer operation. The memory 104 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage device. The at least one processor 103 executes the program stored in the memory 104 and, through the aforementioned components, implements the above-described functions, thereby ultimately realizing the method provided in the above embodiments.

[0153] Through the above description of the embodiments, those skilled in the art will clearly understand that the embodiments of this application can be implemented in hardware, firmware, or a combination thereof. When implemented in software, the above functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium. Computer-readable media include computer storage media and communication media, wherein communication media include any medium that facilitates the transfer of a computer program from one place to another. Storage media can be any available medium accessible to a computer. For example, but not limited to, computer-readable media can include RAM, ROM, electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disc storage, magnetic disk storage media, or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible to a computer. Furthermore, any connection can suitably be a computer-readable medium. For example, if the software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. As used in embodiments of this application, disks and discs include compact discs (CDs), laser discs, optical discs, digital video discs (DVDs), floppy disks, and Blu-ray discs, wherein disks typically magnetically copy data, while discs optically copy data using lasers. The combinations above should also be included within the scope of protection for computer-readable media.

[0154] In summary, the above descriptions are merely embodiments of this application and are not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the disclosure of this application should be included within the scope of protection of this application.

Claims

1. A method for image recognition, characterized in that, The method includes: Acquire a first image containing at least one target substance, wherein the first image is an image of a first spectral band; the first spectral band is a low-resolution spectral band. The first image is reconstructed by spectral band reconstruction to obtain a second image, which is an image of the second spectral band; the second spectral band is a high-resolution spectral band with wavelengths in the visible light and infrared light regions. The step of reconstructing the first image into a second image by performing spectral reconstruction includes: acquiring a first spectral reconstruction model; performing spectral reconstruction on the first image based on the first spectral reconstruction model to obtain the second image; the first spectral reconstruction model is obtained by adding a penalty term of multi-channel supervision information from a color temperature sensor to a pre-trained second spectral reconstruction model; the second spectral reconstruction model is trained from multiple third images; wherein the multiple third images are images acquired by a first acquisition device and a second acquisition device, the first acquisition device being an RGB camera, the second acquisition device being a TOF camera and / or a color temperature sensor, and the penalty term of the multi-channel supervision information from the color temperature sensor being used to adjust the channels of the second spectral reconstruction model; The composition information of at least one target substance is obtained by identifying the second image; Display the composition information of the at least one target substance.

2. The method as described in claim 1, characterized in that, Recognizing the second image to obtain the composition information of the at least one target substance includes: Obtain the spectral information of the second image, wherein the spectral information is the spectral information corresponding to the spectral bands of the second image; The spectral information of the second image is identified using a convolutional neural network model to determine the chemical bonds of the at least one target substance; The composition information of the at least one target substance is determined based on the chemical bonds of the at least one target substance.

3. The method as described in claim 2, characterized in that, Before using a convolutional neural network model to identify the spectral information of the second image, the method further includes: Based on the spectral information of the second image, a first spectrum corresponding to the spectral segment of the second image is obtained, and a second spectrum corresponding to the spectral segment of the fourth image is obtained; wherein, the fourth image is an image obtained by a color temperature sensor from the at least one target substance; If, based on the first spectrum and the second spectrum, the error value between the first spectrum and the second spectrum is greater than a set threshold, the spectral information in the second image is adjusted.

4. The method as described in claim 3, characterized in that, Adjusting the spectral information in the second image includes: Based on the first spectrum, obtain N spectral values ​​corresponding to N channels in the second image; Based on the second spectrum, obtain N spectral values ​​corresponding to N channels in the fourth image; the value of N is greater than or equal to 1; wherein, there is a one-to-one correspondence between the N channels in the second image and the N channels in the fourth image; Based on the N spectral values ​​corresponding to the N channels in the second image and the N spectral values ​​corresponding to the N channels in the fourth image, the N channels in the second image are adjusted.

5. The method as described in claim 4, characterized in that, Adjustments are made to N channels in the second image, including: For the i-th channel among the N channels, determine the spectral value corresponding to the i-th channel in the second image as L. i The spectral value corresponding to the i-th channel in the fourth image acquired by the color temperature sensor is l i i takes any integer value from 1 to N; When the L is determined i With l i When the absolute value of the difference is greater than the set threshold, the i-th channel in the second image is adjusted.

6. An electronic device, characterized in that, Includes a display screen, one or more processors, and one or more memory units; The display screen is used to display information; The one or more memories store one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the electronic device performs the following: Acquire a first image containing at least one target substance, wherein the first image is an image of a first spectral band; the first spectral band is a low-resolution spectral band. The first image is reconstructed into a second image, which is an image of a second spectral band. The second spectral band is a high-resolution spectral band with wavelengths in the visible and infrared regions. When the electronic device performs spectral reconstruction on the first image to obtain the second image, it specifically performs the following: acquiring a first spectral band reconstruction model, and reconstructing the first image into a second image based on the first spectral band reconstruction model. The first spectral band reconstruction model is obtained by adding a penalty term of multi-channel supervision information from a color temperature sensor to a pre-trained second spectral band reconstruction model. The second spectral band reconstruction model is trained from multiple third images. The multiple third images are images acquired by a first acquisition device and a second acquisition device. The first acquisition device is an RGB camera, and the second acquisition device is a TOF camera and / or a color temperature sensor. The penalty term of the multi-channel supervision information from the color temperature sensor is used to adjust the channels of the second spectral band reconstruction model. The composition information of at least one target substance is obtained by identifying the second image; The composition information of the at least one target substance is displayed on the display screen.

7. The electronic device as claimed in claim 6, characterized in that, When the instruction is executed by the processor, the electronic device performs the following: Obtain the spectral information of the second image, wherein the spectral information is the spectral information corresponding to the spectral bands of the second image; The spectral information of the second image is identified using a convolutional neural network model to determine the chemical bonds of the at least one target substance; The composition information of the at least one target substance is determined based on the chemical bonds of the at least one target substance.

8. The electronic device as claimed in claim 7, characterized in that, When the instruction is executed by the processor, the electronic device also performs: Before using a convolutional neural network model to identify the spectral information of the second image, a first spectrum corresponding to the spectral segment of the second image and a second spectrum corresponding to the spectral segment of the fourth image are obtained based on the spectral information of the second image; wherein, the fourth image is an image obtained by a color temperature sensor from the at least one target substance; If, based on the first spectrum and the second spectrum, the error value between the first spectrum and the second spectrum is greater than a set threshold, the spectral information in the second image is adjusted.

9. The electronic device as claimed in claim 8, characterized in that, When the instruction is executed by the processor, the electronic device performs the following: Based on the first spectrum, obtain N spectral values ​​corresponding to N channels in the second image; Based on the second spectrum, obtain N spectral values ​​corresponding to N channels in the fourth image; the value of N is greater than or equal to 1; wherein, there is a one-to-one correspondence between the N channels in the second image and the N channels in the fourth image; Based on the N spectral values ​​corresponding to the N channels in the second image and the N spectral values ​​corresponding to the N channels in the fourth image, the N channels in the second image are adjusted.

10. The electronic device as claimed in claim 9, characterized in that, When the instruction is executed by the processor, the electronic device performs the following: For the i-th channel among the N channels, determine the spectral value corresponding to the i-th channel in the second image as L. i The spectral value corresponding to the i-th channel in the fourth image acquired by the color temperature sensor is l i i takes any integer value from 1 to N; When the L is determined i With l i When the absolute value of the difference is greater than the set threshold, the i-th channel in the second image is adjusted.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed on the device, causes the device to perform the method as described in any one of claims 1-5.

12. A chip, characterized in that, The chip is used to read a computer program stored in a memory and execute the method as described in any one of claims 1-5.

Images