A method and device for processing images of multiple types of optical kit cameras

Abstract

The present invention discloses a method and device for image processing of multiple types of optical kit cameras. The method comprises: obtaining optical device information and sensor information; collecting polarized image data according to the sensor information and the optical device information; calculating infrared thermal imaging texture information according to a preset model construction algorithm and the polarized image data; and restoring an imaging image according to the infrared thermal imaging texture information. The present invention solves the technical problem that the thermal imaging ghosting problem in the prior art requires a large amount of computing resources to be consumed, the algorithm is highly complex, and the time consumption of several minutes for one image makes this method almost impossible to be implemented in the industry.

Description

Technical Field

[0001] The present invention relates to the field of infrared imaging correction, and in particular to a method and device for processing images of multiple types of optical kit cameras. Background Art

[0002] With the continuous development of intelligent technology, people are using more and more intelligent devices in their lives, work and study. The use of intelligent technology has improved the quality of people's lives and increased the efficiency of their study and work.

[0003] In the early 20th century, scientists first discovered infrared radiation, also known as thermal radiation. This is an invisible electromagnetic radiation. With the development of thermal sensors, thermal imaging tubes and infrared technology, infrared thermal imaging technology came into being. Infrared thermal imaging technology has had a significant impact in the fields of security, military and non-destructive testing. The main disadvantage of infrared thermal imaging is the "ghost effect": the principle of imaging in the visible light range is that the light from the visible light source is scattered to the object and then reflected to the receiving unit of the camera. After processing, the image is formed. These reflected lights carry texture information. The images obtained in the infrared band are different. The electromagnetic waves spontaneously radiated by each object will cover up the electromagnetic waves carrying its own texture information. However, so far, it is still a bit tricky to solve the ghost problem of infrared thermal imaging with high quality. Many people have tried to solve similar problems in history. Kristan P. Gurton first tried to process Stokes images in the far infrared band in 2010. Based on polarized optical imaging, they found that Stokes images can characterize details that cannot be seen in far infrared band thermal images. Based on this research, in 2014, Kristan P. Gurton et al. proposed a method to obtain facial details under thermal imaging based on polarization optics. However, the only drawback is that they used the Stokes formula to calculate the process diagram and used the polarization image to reflect the image details, which resulted in only the details of objects with drastic changes in polarization, while ignoring the texture details of a large number of other objects. This also limited the application scenarios of their method. With the development of computational imaging technology, in 2016, some people found that the polarization angle image of an object can be calculated with the help of polarization optics, and this polarization angle image can reflect the phase information of the object and provide richer process information for perfecting texture fusion. When processing infrared polarization process diagrams at different positions at multiple levels of the same original image, the concept of image feature fusion has developed with computational imaging technology. As early as 2014, Yue Z et al. proposed a manual fusion algorithm for polarization images based on the pyramid algorithm; with the development of AI, the use of machine learning or models for polarization image fusion has also begun to emerge. Recently, Fanglin Bao et al. proposed to establish an imaging model for infrared thermal imaging, and based on this model, designed a device for collecting hyperspectral images and a corresponding neural network model for processing hyperspectral thermal cubes, thereby obtaining the material, temperature and even inferring texture information of objects in infrared thermal imaging. This method caused a sensation in the industry, proving that the material and texture of infrared thermal imaging objects can be calculated, with good imaging effects. The only drawback is that this method requires a lot of computing resources, its algorithm is highly complex, and the several minutes it takes to process an image makes this method almost hopeless to be implemented in industry.

[0004] To this end, the present invention proposes an infrared thermal imaging object image texture restoration method, which also uses polarization optics. The difference between our method for restoring human faces and the method proposed by Kristan P. Gurton is that based on the infrared band object radiation model established by ourselves, we have a more complete object texture restoration theory and a wider range of texture restoration applications. Compared with the infrared thermal imaging imaging model proposed by Fanglin Bao et al., our imaging texture restoration method does not require complex calculations based on complex multi-step AI reasoning results. Instead, it uses the infrared band object radiation model established by ourselves and polarization optics to screen out pictures containing effective real textures for computational imaging, which has a faster calculation speed and a texture restoration effect comparable to their results. Summary of the invention

[0005] The embodiment of the present invention provides a method and device for processing images of multiple types of optical kit cameras, so as to at least solve the problem of thermal imaging ghosting in the prior art. The solution process requires a large amount of computing resources, the algorithm is highly complex, and it takes several minutes to process one image, which makes this method almost impossible to be implemented in industry.

[0006] According to one aspect of an embodiment of the present invention, a method for processing images of a multi-type optical kit camera is provided, including: acquiring optical device information and sensor information; collecting polarization image data according to the sensor information and the optical device information; calculating infrared thermal imaging texture information according to a preset model building algorithm and the polarization image data; and restoring an imaging image according to the infrared thermal imaging texture information.

[0007] Optionally, collecting polarization image data according to the sensing information and the optical device information includes: inputting the optical device information and the sensing information into a matrix

[0008]

[0009] The polarization image data set is obtained, wherein D1~Dn is an array of n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays.

[0010] Optionally, before calculating the infrared thermal imaging texture information according to the preset model building algorithm and the polarization image data, the method further includes: performing edge optimization processing on the polarization image data to obtain optimized data.

[0011] Optionally, restoring the imaging image according to the infrared thermal imaging texture information includes: constructing a restored binary array using the infrared thermal imaging texture information and original image data; and processing elements in the restored binary array according to a preset restoration rule to obtain corrected image data.

[0012] According to another aspect of an embodiment of the present invention, a multi-type optical kit camera image processing device is also provided, including: an acquisition module for acquiring optical device information and sensor information; an acquisition module for acquiring polarization image data based on the sensor information and the optical device information; a calculation module for calculating infrared thermal imaging texture information based on a preset model building algorithm and the polarization image data; and a restoration module for restoring an imaging image based on the infrared thermal imaging texture information.

[0013] Optionally, the acquisition module includes: a computing unit for inputting the optical device information and the sensor information into a matrix

[0014]

[0015] The polarization image data set is obtained, wherein D1~Dn is an array of n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays.

[0016] Optionally, the device further includes: an optimization module, configured to perform edge optimization processing on the polarization image data to obtain optimized data.

[0017] Optionally, the restoration module includes: a construction unit, used to construct a restored binary array from the infrared thermal imaging texture information and the original image data; and a processing unit, used to process the elements in the restored binary array according to preset restoration rules to obtain corrected image data.

[0018] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute a multi-type optical kit camera image processing method.

[0019] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute a multi-type optical kit camera image processing method when executed.

[0020] In an embodiment of the present invention, the optical device information and the sensor information are acquired; polarization image data is collected according to the sensor information and the optical device information; infrared thermal imaging texture information is calculated according to a preset model construction algorithm and the polarization image data; and an imaging image is restored according to the infrared thermal imaging texture information. This solves the technical problem that the thermal imaging ghost problem in the prior art needs to consume a large amount of computing resources, the algorithm is highly complex, and a picture takes several minutes to form, which makes this method almost impossible to be implemented in industry. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 is a schematic diagram of obtaining a polarized infrared thermal imaging image according to an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of S0 according to an embodiment of the present invention;

[0024] Figure 3 is a schematic diagram of some texture features calculated by S1 according to an embodiment of the present invention;

[0025] Figure 4 is another schematic diagram of texture features calculated by S2 according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of the phase change of an object obtained by Aop calculation according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of texture restoration results according to an embodiment of the present invention;

[0028] Figure 7 is a flowchart of image texture restoration according to an embodiment of the present invention;

[0029] Figure 8 is a flow chart of the fusion of texture features by a manual algorithm according to an embodiment of the present invention;

[0030] Fig. 9 is a flow chart of AI algorithm fusion texture features according to an embodiment of the present invention;

[0031] Fig.10 is a diagram of an unsupervised learning model architecture for image texture restoration according to an embodiment of the present invention;

[0032] Fig.11 is a flow chart of a method for processing images of a multi-type optical kit camera according to an embodiment of the present invention;

[0033] Fig.12 is a structural block diagram of a multi-type optical kit camera image processing device according to an embodiment of the present invention;

[0034] Fig.13 is a block diagram of a terminal device for executing a method according to an embodiment of the present invention;

[0035] Fig.14 It is a storage unit for holding or carrying a program code for implementing a method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0036] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0037] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0038] According to an embodiment of the present invention, a method embodiment of a method for processing images of a multi-type optical kit camera is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0039] Embodiment 1

[0040] Figure 1 is a flowchart of a multi-type optical kit camera image processing method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0041] Step S102, obtaining optical device information and sensor information.

[0042] Step S104: collecting polarization image data according to the sensing information and the optical device information.

[0043] Step S106, calculating infrared thermal imaging texture information according to a preset model building algorithm and the polarization image data.

[0044] Step S108: restoring the imaging image according to the infrared thermal imaging texture information.

[0045] Optionally, collecting polarization image data according to the sensing information and the optical device information includes: inputting the optical device information and the sensing information into a matrix

[0046]

[0047] The polarization image data set is obtained, wherein D1~Dn is an array of n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays.

[0048] Optionally, before calculating the infrared thermal imaging texture information according to the preset model building algorithm and the polarization image data, the method further includes: performing edge optimization processing on the polarization image data to obtain optimized data.

[0049] Optionally, restoring the imaging image according to the infrared thermal imaging texture information includes: constructing a restored binary array using the infrared thermal imaging texture information and original image data; and processing elements in the restored binary array according to a preset restoration rule to obtain corrected image data.

[0050] A multi-type optical kit camera image processing method of the present invention is shown in FIG. Figure 1 As shown, first there are electromagnetic waves generated by spontaneous radiation of objects (type I electromagnetic waves), and then the electromagnetic waves spontaneously radiated by each object are radiated to the surrounding objects and reflected to obtain electromagnetic waves (type II electromagnetic waves). Type I and type II electromagnetic waves carry the information of the object group and pass through the polarizer S&P controlled by the stepper motor and then enter the infrared thermal imaging device CA1. The pulse signal sent by the computer controls the rotation speed and angle of the stepper motor, and at the same time controls the infrared thermal imaging camera to obtain the required polarization image at the frequency corresponding to the rotation angular velocity.

[0051] Figure 7The image processing flow shown here is after the image is acquired. First, the acquired multiple original images are processed by some traditional manual algorithms to obtain pre-processed feature texture maps such as S0, S1, S2 and Aop. Then, with the help of targeted manual algorithms or AI processing, the features of multiple process maps rich in different texture features are organically and reasonably refined into one map. Finally, the texture map with multi-level features is superimposed on the original image to obtain an infrared thermal map with restored texture information.

[0052] Figure 8 The manual algorithm implementation method of the image texture feature fusion part in the image processing process is shown. The multiple multi-level texture feature maps obtained in the previous step are processed in blocks with the help of the algorithm, and the parts with high value weights are calculated. Then, this part is fused through the algorithm using a specific formula to obtain the final texture information fusion map.

[0053]

[0054] Among them, Top is the fusion result, S1, S2, and Aop are different pixel processing channels respectively.

[0055] The specific example process is as follows:

[0056] First, constrain the preprocessed images such as S1, S2 and Aop to the same size and channel, and use formula (1) to process the values ​​of the pixels at the same position to obtain the top image. Generally speaking, some texture information in the top image is hidden due to the non-universality of the formula. The following operation is to expand the hidden and weakened texture information.

[0057] Calculate the overall tolerance of the Top image and record it as σ G , the entire Top image is processed using a sliding window, the area within the sliding window is recorded as B, and then the tolerance σ within B is calculated B and mean u B , and then obtain a coefficient α=σ G / σ B , use this coefficient to calculate the generalized pixel value of the high-frequency component at the corresponding (i, j) position in B, that is, L(i, j) = u B +α(L B (i,j)-u B ), all L(i,j) are processed into a picture, which is the restored texture map.

[0058] Fig. 9 The AI ​​method for unsupervised multi-image texture fusion in the image texture feature fusion part of the image processing process is demonstrated. First, the multi-image texture fusion AI is trained, and then the pre-processed feature texture maps such as S0, S1, S2 and Aop are sent to the AI ​​to obtain the final texture information fusion map.

[0059] The specific example process is as follows:

[0060] Taking an unsupervised learning model as an example, its system architecture is based on Fig.10 For example:

[0061] It consists of three main components: feature extraction, fusion and reconstruction. Preprocessed feature texture maps such as S1, S2 and Aop are input into three channels. Channel A is composed of D 11 、E 11 、E 21 and E 31 Channel B consists of D 12 、E 12 、E 22 and E 32 Channel C consists of D 13 、E 13 、E 23 and E 33 blocks. These blocks are all used for feature extraction, and the blocks starting with E have the function of compressing features. In the feature fusion part, the deep features are directly connected, that is, the channels are superimposed, and the superposition scheme depends on formula (2). In formula (2), the superscript fe means the data after feature extraction, Norm represents normalization, and the image source formulas of S1 and S2 have a certain degree of similarity in mathematics, so they can be simply overlapped during fusion and use the same coefficient. The feature map of Aop needs to be added with a balance parameter γ separately. The result of the feature fusion layer is reconstructed from the fused features through another five convolutional layers (D2, D3, D4, D5 and D6).

[0062]

[0063] Note: The training loss function is defined according to formula (3), where the parameters are defined as follows:

[0064]

[0065] SSIM is an algorithm that measures the structural similarity of images. It combines the brightness, contrast and structure of the image to measure the image quality. TV stands for Total Variation, which is an indicator for measuring image noise. There is usually an order of magnitude difference between the two, so the parameter is introduced. To balance, usually

[0066] Based on polarization optical imaging technology, the present invention proposes a method and device for removing ghost images and restoring textures of infrared thermal imaging objects through physical mathematical modeling. The device includes a computer with certain computing power, an infrared camera, a stepper motor rotation mount, and a polarizer. The stepper motor is controlled by a computer to rotate the polarizer so that the infrared camera can obtain the required polarization image, and then the infrared image texture information is restored based on the derivative algorithm (or AI) of our physical mathematical model.

[0067] Through the above-mentioned embodiments, the technical problem that the thermal imaging ghosting problem in the prior art needs to consume a large amount of computing resources, the algorithm is highly complex, and it takes several minutes to form an image, which makes this method almost impossible to be implemented in the industry is solved.

[0068] Embodiment 2

[0069] Figure 2 is a structural block diagram of a multi-type optical kit camera image processing device according to an embodiment of the present invention, such as Figure 2 As shown, the device comprises:

[0070] The acquisition module 20 is used to acquire optical device information and sensor information.

[0071] The acquisition module 22 is used to acquire polarization image data according to the sensing information and the optical device information.

[0072] The calculation module 24 is used to calculate the infrared thermal imaging texture information according to a preset model building algorithm and the polarization image data.

[0073] The restoration module 26 is used to restore the imaging image according to the infrared thermal imaging texture information.

[0074] Optionally, the acquisition module includes: a computing unit for inputting the optical device information and the sensor information into a matrix

[0075]

[0076] The polarization image data set is obtained, wherein D1~Dn is an array of n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays.

[0077] Optionally, the device further includes: an optimization module, configured to perform edge optimization processing on the polarization image data to obtain optimized data.

[0078] Optionally, the restoration module includes: a construction unit, used to construct a restored binary array from the infrared thermal imaging texture information and the original image data; and a processing unit, used to process the elements in the restored binary array according to preset restoration rules to obtain corrected image data.

[0079] A multi-type optical kit camera image processing device of the present invention is shown in FIG. Figure 1 As shown, first there are electromagnetic waves generated by spontaneous radiation of objects (type I electromagnetic waves), and then the electromagnetic waves spontaneously radiated by each object are radiated to the surrounding objects and reflected to obtain electromagnetic waves (type II electromagnetic waves). Type I and type II electromagnetic waves carry the information of the object group and pass through the polarizer S&P controlled by the stepper motor and then enter the infrared thermal imaging device CA1. The pulse signal sent by the computer controls the rotation speed and angle of the stepper motor, and at the same time controls the infrared thermal imaging camera to obtain the required polarization image at the frequency corresponding to the rotation angular velocity.

[0080] Figure 7 The image processing flow shown here is after the image is acquired. First, the acquired multiple original images are processed by some traditional manual algorithms to obtain pre-processed feature texture maps such as S0, S1, S2 and Aop. Then, with the help of targeted manual algorithms or AI processing, the features of multiple process maps rich in different texture features are organically and reasonably refined into one map. Finally, the texture map with multi-level features is superimposed on the original image to obtain an infrared thermal map with restored texture information.

[0081] Figure 8 The manual algorithm implementation method of the image texture feature fusion part in the image processing process is shown. The multiple multi-level texture feature maps obtained in the previous step are processed in blocks with the help of the algorithm, and the parts with high value weights are calculated. Then, this part is fused through the algorithm using a specific formula to obtain the final texture information fusion map.

[0082]

[0083] Among them, Top is the fusion result, S1, S2, and Aop are different pixel processing channels respectively.

[0084] The specific example process is as follows:

[0085] First, constrain the preprocessed images such as S1, S2 and Aop to the same size and channel, and use formula (1) to process the values ​​of the pixels at the same position to obtain the top image. Generally speaking, some texture information in the top image is hidden due to the non-universality of the formula. The following operation is to expand the hidden and weakened texture information.

[0086] Calculate the overall tolerance of the Top image and record it as σ G, the entire Top image is processed using a sliding window, the area within the sliding window is recorded as B, and then the tolerance σ within B is calculated B and mean u B , and then obtain a coefficient α=σ G / σ B , use this coefficient to calculate the generalized pixel value of the high-frequency component at the corresponding (i, j) position in B, that is, L(i, j) = u B +α(L B (i,j)-u B ), all L(i,j) are processed into a picture, which is the restored texture map.

[0087] Fig. 9 The AI ​​method for unsupervised multi-image texture fusion in the image texture feature fusion part of the image processing process is demonstrated. First, the multi-image texture fusion AI is trained, and then the pre-processed feature texture maps such as S0, S1, S2 and Aop are sent to the AI ​​to obtain the final texture information fusion map.

[0088] The specific example process is as follows:

[0089] Taking an unsupervised learning model as an example, its system architecture is based on Fig.10 For example:

[0090] It consists of three main components: feature extraction, fusion and reconstruction. Preprocessed feature texture maps such as S1, S2 and Aop are input into three channels. Channel A is composed of D 11 、E 11 、E 21 and E 31 Channel B consists of D 12 、E 12 、E 22 and E 32 Channel C consists of D 13 、E 13 、E 23 and E 33 blocks. These blocks are all used for feature extraction, and the blocks starting with E have the function of compressing features. In the feature fusion part, the deep features are directly connected, that is, the channels are superimposed, and the superposition scheme depends on formula (2). In formula (2), the superscript fe means the data after feature extraction, Norm represents normalization, and the image source formulas of S1 and S2 have a certain degree of similarity in mathematics, so they can be simply overlapped during fusion and use the same coefficient. The feature map of Aop needs to be added with a balance parameter γ separately. The result of the feature fusion layer is reconstructed from the fused features through another five convolutional layers (D2, D3, D4, D5 and D6).

[0091]

[0092] Note: The training loss function is defined according to formula (3), where the parameters are defined as follows:

[0093]

[0094] SSIM is an algorithm that measures the structural similarity of images. It combines the brightness, contrast and structure of the image to measure the image quality. TV stands for Total Variation, which is an indicator for measuring image noise. There is usually an order of magnitude difference between the two, so the parameter is introduced. To balance, usually

[0095] Based on polarization optical imaging technology, the present invention proposes a method and device for removing ghost images and restoring textures of infrared thermal imaging objects through physical mathematical modeling. The device includes a computer with certain computing power, an infrared camera, a stepper motor rotation mount, and a polarizer. The stepper motor is controlled by a computer to rotate the polarizer so that the infrared camera can obtain the required polarization image, and then the infrared image texture information is restored based on the derivative algorithm (or AI) of our physical mathematical model.

[0096] Through the above-mentioned embodiments, the technical problem that the thermal imaging ghosting problem in the prior art needs to consume a large amount of computing resources, the algorithm is highly complex, and it takes several minutes to form an image, which makes this method almost impossible to be implemented in the industry is solved.

[0097] According to another aspect of an embodiment of the present invention, a non-volatile storage medium is provided, wherein the non-volatile storage medium includes a stored program, wherein when the program is executed, the device where the non-volatile storage medium is located is controlled to execute a multi-type optical kit camera image processing method.

[0098] Specifically, the method includes: obtaining optical device information and sensor information; collecting polarized image data according to the sensor information and the optical device information; calculating infrared thermal imaging texture information according to a preset model building algorithm and the polarized image data; and restoring an image according to the infrared thermal imaging texture information. Optionally, collecting polarized image data according to the sensor information and the optical device information includes: inputting the optical device information and the sensor information into a matrix

[0099]

[0100] , the polarization image data is obtained, wherein D1~Dn is an array summarizing n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays. Optionally, before the step of calculating the infrared thermal imaging texture information based on the preset model building algorithm and the polarization image data includes: performing edge optimization processing on the polarization image data to obtain optimized data. Optionally, the step of restoring the imaging image based on the infrared thermal imaging texture information includes: constructing a restored binary array with the infrared thermal imaging texture information and the original image data; processing the elements in the restored binary array according to preset restoration rules to obtain corrected image data.

[0101] According to another aspect of an embodiment of the present invention, an electronic device is provided, comprising a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute a multi-type optical kit camera image processing method when executed.

[0102] Specifically, the method includes: obtaining optical device information and sensor information; collecting polarized image data according to the sensor information and the optical device information; calculating infrared thermal imaging texture information according to a preset model building algorithm and the polarized image data; and restoring an image according to the infrared thermal imaging texture information. Optionally, collecting polarized image data according to the sensor information and the optical device information includes: inputting the optical device information and the sensor information into a matrix

[0103]

[0104] , the polarization image data is obtained, wherein D1~Dn is an array summarizing n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays. Optionally, before the step of calculating the infrared thermal imaging texture information based on the preset model building algorithm and the polarization image data includes: performing edge optimization processing on the polarization image data to obtain optimized data. Optionally, the step of restoring the imaging image based on the infrared thermal imaging texture information includes: constructing a restored binary array with the infrared thermal imaging texture information and the original image data; processing the elements in the restored binary array according to preset restoration rules to obtain corrected image data.

[0105] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0106] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] in addition, Fig.13 This is a schematic diagram of the hardware structure of a terminal device provided in an embodiment of the present application. Figure 3 As shown, the terminal device may include an input device 30, a processor 31, an output device 32, a memory 33 and at least one communication bus 34. The communication bus 34 is used to realize the communication connection between the components. The memory 33 may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory. Various programs can be stored in the memory 33 to complete various processing functions and implement the method steps of this embodiment.

[0110] Optionally, the processor 31 may be implemented as a central processing unit (CPU), an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing device (DSPD), a programmable logic device (PLD), a field programmable gate array (FPGA), a controller, a microcontroller, a microprocessor or other electronic components, and the processor 31 is coupled to the input device 30 and the output device 32 via a wired or wireless connection.

[0111] Optionally, the input device 30 may include multiple input devices, for example, it may include at least one of a user interface for users, a device interface for devices, a programmable interface for software, a camera, and a sensor. Optionally, the device interface for devices may be a wired interface for data transmission between devices, or a hardware insertion interface for data transmission between devices (such as a USB interface, a serial port, etc.); Optionally, the user interface for users may be, for example, a control button for users, a voice input device for receiving voice input, and a touch sensing device for users to receive user touch input (such as a touch screen with a touch sensing function, a touch pad, etc.); Optionally, the programmable interface for the software may be, for example, an entry for users to edit or modify programs, such as an input pin interface or an input interface of a chip; Optionally, the transceiver may be a radio frequency transceiver chip with communication function, a baseband processing chip, and a transceiver antenna, etc. Audio input devices such as microphones may receive voice data. The output device 32 may include output devices such as displays and speakers.

[0112] In this embodiment, the processor of the terminal device includes functions for executing each module of the data processing device in each device. The specific functions and technical effects can be referred to the above embodiments and will not be repeated here.

[0113] Fig.14 A schematic diagram of the hardware structure of a terminal device provided in another embodiment of the present application. Fig.14 Yes Fig.13 A specific embodiment in the implementation process. Fig.14 As shown, the terminal device of this embodiment includes a processor 41 and a memory 42.

[0114] The processor 41 executes the computer program code stored in the memory 42 to implement the method in the above embodiment.

[0115] The memory 42 is configured to store various types of data to support operations on the terminal device. Examples of such data include instructions for any application or method used to operate on the terminal device, such as messages, pictures, videos, etc. The memory 42 may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage.

[0116] Optionally, the processor 41 is provided in the processing component 40. The terminal device may further include: a communication component 43, a power component 44, a multimedia component 45, an audio component 46, an input / output interface 47 and / or a sensor component 48. The specific components included in the terminal device are set according to actual needs, and this embodiment does not limit this.

[0117] The processing component 40 generally controls the overall operation of the terminal device. The processing component 40 may include one or more processors 41 to execute instructions to complete all or part of the steps of the above method. In addition, the processing component 40 may include one or more modules to facilitate the interaction between the processing component 40 and other components. For example, the processing component 40 may include a multimedia module to facilitate the interaction between the multimedia component 45 and the processing component 40.

[0118] The power supply component 44 provides power to various components of the terminal device. The power supply component 44 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the terminal device.

[0119] The multimedia component 45 includes a display screen that provides an output interface between the terminal device and the user. In some embodiments, the display screen may include a liquid crystal display (LCD) and a touch panel (TP). If the display screen includes a touch panel, the display screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation.

[0120] The audio component 46 is configured to output and / or input audio signals. For example, the audio component 46 includes a microphone (MIC), and when the terminal device is in an operating mode, such as a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 42 or sent via the communication component 43. In some embodiments, the audio component 46 also includes a speaker for outputting audio signals.

[0121] The input / output interface 47 provides an interface between the processing component 40 and the peripheral interface modules, which may be click wheels, buttons, etc. These buttons may include but are not limited to: volume buttons, start buttons, and lock buttons.

[0122] The sensor assembly 48 includes one or more sensors for providing various aspects of status assessment for the terminal device. For example, the sensor assembly 48 can detect the open / closed state of the terminal device, the relative positioning of the components, and the presence or absence of contact between the user and the terminal device. The sensor assembly 48 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact, including detecting the distance between the user and the terminal device. In some embodiments, the sensor assembly 48 may also include a camera, etc.

[0123] The communication component 43 is configured to facilitate wired or wireless communication between the terminal device and other devices. The terminal device can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In one embodiment, the terminal device may include a SIM card slot for inserting a SIM card, so that the terminal device can log in to the GPRS network and establish communication with the service end through the Internet.

[0124] From the above, we can see that Fig.14 The communication component 43, the audio component 46, the input / output interface 47, and the sensor component 48 involved in the embodiment can all be used as Figure 3 Implementation method of the input device in the embodiment.

[0125] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0126] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0127] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0128] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0129] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A multi-type optical kit camera image processing method, characterized in that: include: Obtaining optical device information and sensor information; Collect polarization image data according to the sensing information and the optical device information; Calculating infrared thermal imaging texture information according to a preset model building algorithm and the polarization image data; The imaging image is restored according to the infrared thermal imaging texture information.

2. The method according to claim 1, characterized in that The collecting polarization image data according to the sensing information and the optical device information comprises: Input the optical device information and the sensor information into a matrix The polarization image data set is obtained, wherein D1~Dn is an array of n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays.

3. The method according to claim 1, characterized in that Before calculating the infrared thermal imaging texture information according to the preset model building algorithm and the polarization image data, the method further includes: The polarization image data is subjected to edge optimization processing to obtain optimized data.

4. The method according to claim 1, characterized in that: The restoring of the image according to the infrared thermal imaging texture information comprises: Constructing and restoring a binary array from the infrared thermal imaging texture information and the original image data; The elements in the restored binary array are processed according to a preset restoration rule to obtain corrected image data.

5. A multi-type optical kit camera image processing device, characterized in that: include: An acquisition module, used for acquiring optical device information and sensor information; An acquisition module, used for acquiring polarization image data according to the sensing information and the optical device information; A calculation module, used for calculating infrared thermal imaging texture information according to a preset model building algorithm and the polarization image data; A restoration module is used to restore the imaging image according to the infrared thermal imaging texture information.

6. The device according to claim 5, characterized in that The acquisition module comprises: A computing unit, used to input the optical device information and the sensor information into a matrix The polarization image data set is obtained, wherein D1~Dn is an array of n groups of sensor information and optical device information, and T1~Tn is the corresponding polarization image data corresponding to different sensor information and optical device information arrays.

7. The device according to claim 5, characterized in that The device also includes: The optimization module is used to perform edge optimization processing on the polarization image data to obtain optimized data.

8. The device according to claim 5, characterized in that The recovery module comprises: A construction unit, used to construct and restore a binary array from the infrared thermal imaging texture information and the original image data; The processing unit is used to process the elements in the restored binary array according to a preset restoration rule to obtain corrected image data.

9. A non-volatile storage medium, characterized in that: The non-volatile storage medium includes a stored program, wherein the program controls the device where the non-volatile storage medium is located to execute the method according to any one of claims 1 to 4 when the program is executed.

10. An electronic device, characterized in that: It comprises a processor and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein the computer-readable instructions execute the method described in any one of claims 1 to 4 when executed.

Images