Method and system for improving picture clarity based on infrared camera

Abstract

The present application provides a method and system for improving the clarity of a captured image based on an infrared camera, which belongs to the field of data processing technology and is used to improve the clarity of the captured image to ensure the user experience when the imaging of the infrared camera is disturbed. The method includes: the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera; if there is at least a partially interfered area in the infrared image collected by the infrared camera, the monitoring terminal uses the infrared image collected historically by the infrared camera to repair at least a part of the interfered area in the first infrared image currently collected by the infrared camera, and obtains a repaired second infrared image, wherein the clarity of at least a part of the interfered area in the first infrared image is less than the clarity of the corresponding area of ​​the at least part of the interfered area in the first infrared image in the second infrared image.

Description

Technical Field

[0001] The present application relates to the field of device control technology, and in particular to a method for improving the clarity of images shot by an infrared camera. Background Art

[0002] An infrared camera is a device that can capture thermal images. It obtains the temperature information of an object by emitting and receiving infrared radiation, thereby generating a thermal map or thermal imaging. Infrared cameras are widely used in security, medical, industrial and other fields. The working principle of infrared cameras is based on the characteristics of infrared radiation. Infrared radiation is a long-wavelength electromagnetic wave, with a wavelength range of 800 nanometers to 1000 nanometers. Since infrared radiation is not affected by the atmosphere, infrared cameras can provide reliable imaging in dark and harsh environmental conditions.

[0003] The detection and imaging technology of infrared cameras mainly includes the following aspects: 1. Infrared radiation detection: Infrared cameras detect infrared radiation emitted by objects through built-in infrared detectors. These detectors usually use technologies such as infrared Joule calorimeters or infrared cold junction calorimeters. 2. Signal processing: The detected infrared radiation signal needs to be preprocessed to improve the quality of imaging. This includes steps such as signal filtering, amplification, and digital signal conversion. 3. Image reconstruction: Through digital signal processing technology, infrared cameras can generate thermal maps or thermal imaging. These images usually have high spatial resolution and temporal resolution, and can provide detailed information on the temperature distribution of objects. However, the imaging of infrared cameras will also be interfered by some factors, mainly including: 1. Light source interference: Infrared cameras mainly detect infrared radiation emitted by objects, but in some cases, visible light or ultraviolet light emitted by light sources may interfere with the imaging of infrared cameras. 2. Electromagnetic signal radiation: The electromagnetic radiation generated by wireless signals may interfere with the imaging of infrared cameras. 3. Environmental factors: Environmental factors such as temperature, humidity, and atmosphere may affect the performance of infrared cameras. For example, when the temperature of an object is low, the intensity of infrared radiation will decrease, thereby affecting the quality of imaging. 4. Characteristics of the object itself: The intensity and distribution of infrared radiation emitted by different objects may be different, which will also affect the imaging effect of the infrared camera.

[0004] Therefore, when the imaging of the infrared camera is disturbed, how to improve the clarity of the captured image to ensure the user experience is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiments of the present application provide a method and system for improving the clarity of captured images based on an infrared camera, so as to ensure the user experience by improving the clarity of the captured images when the imaging of the infrared camera is disturbed.

[0006] In order to achieve the above objectives, this application adopts the following technical solutions:

[0007] In a first aspect, a method for improving the clarity of a captured image based on an infrared camera is provided, which is applied to a monitoring terminal, wherein the monitoring terminal is provided with an infrared camera, and the method comprises: the monitoring terminal determines whether there is at least a partially interfered area in an infrared image captured by the infrared camera; if there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal uses infrared images historically captured by the infrared camera to repair at least a partially interfered area in a first infrared image currently captured by the infrared camera, and obtains a repaired second infrared image, wherein the clarity of at least a partially interfered area in the first infrared image is less than the clarity of a corresponding area in the second infrared image of the at least partially interfered area in the first infrared image.

[0008] Optionally, the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera, including: the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera.

[0009] Optionally, the monitoring terminal is provided with an antenna panel, which can send M beams with different directions in the collection direction of the infrared camera, where M is an integer greater than 1, and the infrared image collected by the infrared camera can be divided into M grid-shaped areas, the M areas correspond one-to-one to the directions of the M beams, the M beams correspond one-to-one to M wireless channels, and the M wireless channels are used for communication of the monitoring terminal; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera, including: the monitoring terminal measures the M wireless channels to obtain the channel quality corresponding to each of the M wireless channels, a total of M channel qualities; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera based on the M channel qualities; wherein, if N channel qualities among the M channel qualities are less than the channel quality threshold, and the N channel qualities correspond to N beams among the M beams, then the monitoring terminal determines that the N areas corresponding one-to-one to the N beams are at least partially interfered areas in the infrared image collected by the infrared camera.

[0010] Optionally, the method also includes: when N areas are at least partially interfered areas in the infrared image captured by the infrared camera, the monitoring terminal performs clarity analysis on the N areas in each of the multiple consecutive infrared images captured by the infrared camera; accordingly, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered areas in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image, including: if more than a preset number of infrared images in the multiple consecutive infrared images have N areas whose clarity is lower than a clarity threshold, then in response to more than a preset number of infrared images in the multiple consecutive infrared images having N areas whose clarity is lower than a clarity threshold, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered areas in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

[0011] Optionally, the energy level of the uplink channel is lower than the energy level of the downlink channel. If the M wireless channels are M uplink wireless channels, the monitoring terminal can perform clarity analysis on N areas in each of the multiple consecutive infrared images captured by the infrared camera. Otherwise, if the M wireless channels are M downlink wireless channels, the monitoring terminal directly uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

[0012] Optionally, the infrared images historically captured by the infrared camera include K infrared images, where K is an integer greater than 1, and the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image, including: the monitoring terminal inputs images of N areas in each of the K infrared images and images of N areas in the first infrared image into a deep neural network model to obtain repaired images of N areas output by the deep neural network model; the monitoring terminal uses the repaired images of the N areas to replace the images of the N areas in the first infrared image to obtain a repaired second infrared image.

[0013] Optionally, when infrared image restoration is just beginning, the K infrared images are the number of infrared image frames between the ith infrared image and the i+1th infrared image, and the number of infrared image frames is positively correlated with the value of i, and i traverses from 1 to K-1; or, when the restored infrared image is used to restore subsequent infrared images, the K infrared images are the number of infrared image frames between the ith infrared image and the i+1th infrared image, and the number of infrared image frames is negatively correlated with the value of i, and when i+1=K, the Kth infrared image is the infrared image frame previous to the first infrared image.

[0014] Optionally, the monitoring terminal inputs the images of the N areas in each of the K infrared images and the images of the N areas in the first infrared image into a deep neural network model to obtain restored images of the N areas output by the deep neural network model, including: the monitoring terminal inputs the images of the target area in each of the K infrared images and the image of the target area in the first infrared image into the deep neural network model to obtain the restored images of the target area output by the deep neural network model, wherein the target area includes not only the images of the N areas but also partial images of areas adjacent to the N areas.

[0015] Optionally, the first infrared image currently captured by the infrared camera refers to each frame of infrared image captured by the infrared camera since the monitoring terminal determines that there is at least a partially interfered area in the infrared image captured by the infrared camera.

[0016] In a second aspect, a system for improving the clarity of captured images based on an infrared camera is provided, the system includes an application to a monitoring terminal, the monitoring terminal is provided with an infrared camera, and the system is configured as follows: the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera; if there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least a portion of the interfered area in the first infrared image currently captured by the infrared camera, and obtains a repaired second infrared image, wherein the clarity of the repaired at least portion of the interfered area in the second infrared image is higher than the clarity of the at least portion of the interfered area in the first infrared image.

[0017] Optionally, the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera, including: the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera.

[0018] Optionally, the monitoring terminal is provided with an antenna panel, which can send M beams with different directions in the collection direction of the infrared camera, where M is an integer greater than 1, and the infrared image collected by the infrared camera can be divided into M grid-shaped areas, the M areas correspond one-to-one to the directions of the M beams, the M beams correspond one-to-one to M wireless channels, and the M wireless channels are used for communication of the monitoring terminal; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera, including: the monitoring terminal measures the M wireless channels to obtain the channel quality corresponding to each of the M wireless channels, a total of M channel qualities; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera based on the M channel qualities; wherein, if N channel qualities among the M channel qualities are less than the channel quality threshold, and the N channel qualities correspond to N beams among the M beams, then the monitoring terminal determines that the N areas corresponding one-to-one to the N beams are at least partially interfered areas in the infrared image collected by the infrared camera.

[0019] Optionally, the system is also configured as follows: when N areas are at least partially interfered areas in the infrared image captured by the infrared camera, the monitoring terminal performs clarity analysis on the N areas in each of the multiple consecutive infrared images captured by the infrared camera; accordingly, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered areas in the first infrared image currently captured by the infrared camera, to obtain a repaired second infrared image, including: if more than a preset number of infrared images in the multiple consecutive infrared images have N areas whose clarity is lower than a clarity threshold, then in response to more than a preset number of infrared images in the multiple consecutive infrared images having N areas whose clarity is lower than a clarity threshold, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered areas in the first infrared image currently captured by the infrared camera, to obtain a repaired second infrared image.

[0020] Optionally, the energy level of the uplink channel is lower than the energy level of the downlink channel. If the M wireless channels are M uplink wireless channels, the monitoring terminal can perform clarity analysis on N areas in each of the multiple consecutive infrared images captured by the infrared camera. Otherwise, if the M wireless channels are M downlink wireless channels, the monitoring terminal directly uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

[0021] Optionally, the infrared images historically captured by the infrared camera include K infrared images, where K is an integer greater than 1, and the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image, including: the monitoring terminal inputs images of N areas in each of the K infrared images and images of N areas in the first infrared image into a deep neural network model to obtain repaired images of N areas output by the deep neural network model; the monitoring terminal uses the repaired images of the N areas to replace the images of the N areas in the first infrared image to obtain a repaired second infrared image.

[0022] Optionally, when infrared image restoration is just beginning, the K infrared images are the number of infrared image frames between the ith infrared image and the i+1th infrared image, and the number of infrared image frames is positively correlated with the value of i, and i traverses from 1 to K-1; or, when the restored infrared image is used to restore subsequent infrared images, the K infrared images are the number of infrared image frames between the ith infrared image and the i+1th infrared image, and the number of infrared image frames is negatively correlated with the value of i, and when i+1=K, the Kth infrared image is the infrared image frame previous to the first infrared image.

[0023] Optionally, the monitoring terminal inputs the images of the N areas in each of the K infrared images and the images of the N areas in the first infrared image into a deep neural network model to obtain restored images of the N areas output by the deep neural network model, including: the monitoring terminal inputs the images of the target area in each of the K infrared images and the image of the target area in the first infrared image into the deep neural network model to obtain the restored images of the target area output by the deep neural network model, wherein the target area includes not only the images of the N areas but also partial images of areas adjacent to the N areas.

[0024] Optionally, the first infrared image currently captured by the infrared camera refers to each frame of infrared image captured by the infrared camera since the monitoring terminal determines that there is at least a partially interfered area in the infrared image captured by the infrared camera.

[0025] In summary, the above method and system have the following technical effects:

[0026] When the monitoring terminal detects that there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal can use the infrared images historically captured by the infrared camera to repair at least a partially interfered area in the first infrared image currently captured by the infrared camera, and obtain a repaired second infrared image. Since the clarity of the repaired at least partially interfered area in the second infrared image is higher than the clarity of at least a partially interfered area in the first infrared image, that is, the clarity of at least a partially interfered area is improved through image restoration, so that when the imaging of the infrared camera is interfered with, the user experience can be guaranteed by improving the clarity of the captured image. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 A schematic diagram of the architecture of a control system provided in an embodiment of the present application;

[0028] Figure 2 A schematic diagram of a process for improving the clarity of a captured image based on an infrared camera provided in an embodiment of the present application;

[0029] Figure 3 A schematic diagram of an application scenario of a method for improving the clarity of a captured image based on an infrared camera provided in an embodiment of the present application;

[0030] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0031] The present application will present various aspects, embodiments or features around a system that may include multiple devices, components, modules, etc. It should be understood and appreciated that each system may include additional devices, components, modules, etc., and / or may not include all of the devices, components, modules, etc. discussed in conjunction with the figures. In addition, combinations of these schemes may also be used.

[0032] In addition, in the embodiments of the present application, words such as "exemplary" and "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of the word "exemplary" is intended to present concepts in a concrete way.

[0033] In the embodiments of the present application, "information", "signal", "message", "channel" and "signaling" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are matched. "of", "corresponding, relevant" and "corresponding" can sometimes be used interchangeably. It should be noted that when the distinction between them is not emphasized, the meanings they intend to express are matched. In addition, the " / " mentioned in the present application can be used to express an "or" relationship.

[0034] The network architecture and business scenarios described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. A person of ordinary skill in the art can appreciate that with the evolution of the network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0035] To facilitate understanding of the embodiments of the present application, first Figure 1 The control system shown in the figure is used as an example to describe in detail the control system applicable to the embodiments of the present application.

[0036] Figure 1 The schematic diagram of the architecture of a control system applicable to the method for improving the clarity of a picture shot by an infrared camera provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the control system includes: a monitoring terminal, and an infrared camera connected to the monitoring terminal for communication.

[0037] The infrared camera can be arranged on the monitoring terminal, that is, it can be arranged together with the monitoring terminal.

[0038] The monitoring terminal can be a terminal with transceiver functions, or a chip or chip system set in the terminal. The terminal can also be called user equipment (UE), access terminal, subscriber unit (subscriberunit), user station, mobile station (MS), mobile station, remote station, remote terminal, mobile device, user terminal, terminal, wireless communication equipment, user agent or user device. The terminal node in the embodiments of the present application can be a mobile phone, a cellular phone, a smart phone, a tablet computer, a wireless data card, a personal digital assistant (PDA), a wireless modem, a handset, a laptop computer, a machine type communication (MTC) terminal, a computer with wireless transceiver function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a smart home device (for example, a refrigerator, a television, an air conditioner, an electric meter, etc.), an intelligent robot, a robotic arm, a workshop equipment, a wireless terminal in unmanned driving, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in telemedicine, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, a wireless terminal in a smart home, a vehicle-mounted terminal, a road side unit with terminal function, and a wireless terminal in a smart city. unit, RSU), etc., flying equipment (for example, intelligent robots, hot air balloons, drones, airplanes), etc. The terminal node of the present application may also be an on-board module, on-board module, on-board component, on-board chip or on-board unit that is built into the vehicle as one or more components or units. The terminal node may also be other devices with terminal functions. For example, the terminal node may also be a device that serves as a terminal function in D2D communication. The embodiments of the present application do not limit the device form of the terminal node. The device for realizing the function of the terminal may be a terminal node; it may also be a device that can support the terminal to realize the function, such as a chip system. The device may be installed in the terminal or used in combination with the terminal. In the embodiments of the present application, the chip system may be composed of chips, or may include chips and other discrete devices.

[0039] The monitoring terminal is provided with an antenna panel (pannel), which can transmit or receive multiple beams in different directions, which are called multiple beams of the antenna panel.

[0040] A beam is a special directional transmission or reception effect formed by the transmitter or receiver of a network device or terminal through an antenna array, similar to a beam formed by a flashlight converging light in one direction. Sending and receiving signals in the form of a beam can effectively increase the transmission distance of the signal. The beam used for communication between terminals can also be called a side beam.

[0041] The beam may be a wide beam, a narrow beam, or other types of beams. The beam forming technology may be a beam forming technology or other technologies. The beam forming technology may specifically be a digital beam forming technology, an analog beam forming technology, or a hybrid digital / analog beam forming technology.

[0042] Beams generally correspond to resources. For example, when performing beam measurement, the network device measures different beams through different resources, and the terminal feeds back the measured resource quality, so that the network device can know the quality of the corresponding beam. During data transmission, the beam can also be indicated by its corresponding resource. For example, the network device indicates a transmission configuration indication-state through the transmission configuration index (TCI) field in the downlink control information (DCI), and the terminal determines the beam corresponding to the reference resource based on the reference resource contained in the TCI-state.

[0043] The beam can be specifically characterized as a digital beam, an analog beam, a spatial domain filter, a spatial filter, a spatial parameter, TCI, a TCI-state, etc. A beam used to send a signal can be called a transmission beam (or Tx beam), a spatial domain transmission filter, a spatial transmission filter, a spatial domain transmission parameter, a spatial transmission parameter, etc. A beam used to receive a signal can be called a reception beam (or Rx beam), a spatial domain reception filter, a spatial reception filter, a spatial domain reception parameter, a spatial reception parameter, etc.

[0044] It can be understood that the embodiments of the present application are uniformly described using beams, but beams can be replaced by other equivalent concepts and are not limited to the concepts mentioned above.

[0045] In this control system, when the monitoring terminal detects that there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal can use the infrared images historically captured by the infrared camera to repair at least a partially interfered area in the first infrared image currently captured by the infrared camera, and obtain a repaired second infrared image. Since the clarity of the repaired at least partially interfered area in the second infrared image is higher than the clarity of the at least partially interfered area in the first infrared image, that is, the clarity of at least a partially interfered area is improved through image restoration, so that when the imaging of the infrared camera is interfered, the user experience can be guaranteed by improving the clarity of the captured image.

[0046] For ease of understanding, the following will combine Figure 2 The method for improving the clarity of a captured image based on an infrared camera provided in an embodiment of the present application is specifically described.

[0047] For example, Figure 2The present invention provides a flowchart of a method for improving the clarity of a captured image based on an infrared camera. The method can be applied to the application function entity and the geographic location information system in the above control system.

[0048] like Figure 2 As shown, the process of the method for improving the clarity of the captured image based on the infrared camera is as follows:

[0049] S201, the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera.

[0050] The monitoring terminal can determine whether there is at least a partially interfered area in the infrared image collected by the infrared camera by monitoring the wireless signal in the collection direction of the infrared camera. For example, the monitoring terminal is provided with an antenna panel, and the antenna panel can send M beams with different directions in the collection direction of the infrared camera, where M is an integer greater than 1. The infrared image collected by the infrared camera can be divided into M grid-like areas, and the M areas correspond one-to-one to the directions of the M beams, that is, the infrared image can be divided into corresponding areas according to the spatial orientation of the beam. For example, Figure 3 As shown, the antenna panel can send 6 beams in different directions in the collection direction of the infrared camera, namely beams #1-#6. Each beam corresponds to an area in the coverage space, which is the area collected by the infrared camera in this direction. That is, beams #1-#6 correspond to 6 areas, namely areas #1-#6. M beams correspond one-to-one to M wireless channels, and M wireless channels can be used for communication of monitoring terminals.

[0051] On this basis, the monitoring terminal can measure the M wireless channels to obtain the channel quality corresponding to each of the M wireless channels, a total of M channel qualities. The monitoring terminal can determine whether there is at least a partially interfered area in the infrared image collected by the infrared camera based on the M channel qualities. Among them, if N channel qualities among the M channel qualities are less than the channel quality threshold, and the N channel qualities correspond to N beams among the M beams, the monitoring terminal determines that the N areas corresponding to the N beams are at least partially interfered areas in the infrared image collected by the infrared camera. In other words, the monitoring terminal can reuse its own communication performance to achieve interference measurement of infrared detection, such as measuring whether infrared detection is subject to electromagnetic interference from wireless communication. Channel quality can be channel quality indicator (CQI), which indicates the quality of the first channel, and its value range is 0 to 31. When the CQI value is 0, the channel quality is the worst; when the CQI value is 31, the channel quality is the best. When the channel quality is poor, it means that the wireless channel is attenuated due to strong electromagnetic interference, and this electromagnetic interference may also interfere with the imaging of infrared detection.

[0052] Optionally, when the N regions are at least partially interfered regions in the infrared image captured by the infrared camera, the monitoring terminal performs clarity analysis on the N regions in each of the multiple consecutive infrared images captured by the infrared camera. For example, the multiple consecutive infrared images may be 10 or 20 consecutive infrared images, and the specific values ​​are not limited. The clarity analysis may be implemented using existing algorithms, such as the Brenner gradient method, the Tenegrad gradient method, and the like. Accordingly, the monitoring terminal uses the infrared images collected historically by the infrared camera to repair at least part of the interfered area in the first infrared image currently collected by the infrared camera to obtain a repaired second infrared image, including: if there are more than a preset number (the specific value is not limited) of infrared images in a plurality of consecutive infrared images, the clarity of N areas (the average value or weighted value of the clarity of each of the N areas) is lower than the clarity threshold, then in response to the fact that there are more than a preset number of infrared images in a plurality of consecutive infrared images, the clarity of N areas is lower than the clarity threshold, the monitoring terminal uses the infrared images collected historically by the infrared camera to repair at least part of the interfered area in the first infrared image currently collected by the infrared camera to obtain a repaired second infrared image. In other words, the clarity of the infrared image is not entirely affected by the electromagnetic interference of wireless communication, and the infrared image may still remain clear when the electromagnetic interference of wireless communication is less affected. The monitoring terminal can perform image repair only when it is determined that the image is not clear due to interference through clarity analysis, avoiding redundancy caused by performing image repair when the infrared image is clear, resulting in waste of resources.

[0053] Optionally, in the case where the uplink channel and the downlink channel are not reciprocal, the energy level of the uplink channel is usually lower than the energy level of the downlink channel. If the M wireless channels are M uplink wireless channels, the monitoring terminal can perform clarity analysis on the N areas in each of the multiple consecutive infrared images captured by the infrared camera. In other words, since the energy level of the uplink channel is relatively small, the electromagnetic interference it receives is relatively small, and the interference it produces to the infrared detection imaging is also relatively small. Otherwise, if the M wireless channels are M downlink wireless channels, the monitoring terminal directly uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image. In other words, since the energy level of the downlink channel is relatively large, the electromagnetic interference it receives is relatively large, and the interference it produces to the infrared detection imaging is also relatively large.

[0054] It can be understood that measuring the wireless channel to detect whether interference is caused to the infrared detection imaging is an example. For example, the monitoring terminal can also directly determine through clarity analysis whether there are at least some interfered areas in the infrared image captured by the infrared camera with low clarity, and execute S202 when the clarity is lower than the threshold.

[0055] S202, if there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least a partially interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

[0056] The clarity of at least part of the disturbed area in the first infrared image is less than the clarity of the corresponding area of ​​at least part of the disturbed area in the first infrared image in the second infrared image. The first infrared image currently captured by the infrared camera refers to each frame of infrared image captured by the infrared camera after the monitoring terminal determines that there is at least part of the disturbed area in the infrared image captured by the infrared camera, that is, S202 can be repeatedly executed multiple times.

[0057] For example, the infrared images historically collected by the infrared camera include K infrared images, where K is an integer greater than 1. The monitoring terminal uses the infrared images historically collected by the infrared camera to repair at least part of the interfered area in the first infrared image currently collected by the infrared camera to obtain a repaired second infrared image, including: the monitoring terminal can input the images of N areas in each of the K infrared images and the images of N areas in the first infrared image into a deep neural network model (which can be a trained deep neural network model) to obtain the repaired images of N areas output by the deep neural network model. In this way, the monitoring terminal can use the repaired images of N areas to replace the images of N areas in the first infrared image to obtain the repaired second infrared image. The specific value of K is not limited and can be dynamically set according to actual conditions.

[0058] In the case of the initial infrared image restoration (such as the first execution of S202), the number of infrared image frames between the i-th infrared image and the i+1-th infrared image of the K infrared images is positively correlated with the value of i, and i traverses from 1 to K-1. For example, K=5, with a positive correlation step size of 2, an initial interval of 1, the number of infrared image frames between the first infrared image and the second infrared image is 1 frame, the number of infrared image frames between the second infrared image and the third infrared image is 3 (1+step size, i.e., 1+2) frames, the number of infrared image frames between the third infrared image and the fourth infrared image is 5 frames, and the number of infrared image frames between the fourth infrared image and the fifth infrared image is 7 frames. That is, the image used for restoration is used as early as possible, which may not be disturbed at this time, that is, the image that has not been disturbed is used as much as possible to ensure the restoration effect. Alternatively, in the case where the restored infrared image is used to restore the subsequent infrared image (S202 is executed for the 10th / 20th time), the number of infrared image frames between the i-th infrared image and the i+1-th infrared image is negatively correlated with the value of i, and when i+1=K, the K-th infrared image is the infrared image frame before the first infrared image. For example, K=5, with a positive correlation step size of 2, an initial interval of 7, the number of infrared image frames between the first infrared image and the second infrared image is 7, the number of infrared image frames between the second infrared image and the third infrared image is 5, the number of infrared image frames between the third infrared image and the fourth infrared image is 3, and the number of infrared image frames between the fourth infrared image and the fifth infrared image is 1. That is, the image used for restoration is the restored image as much as possible to ensure the restoration effect.

[0059] It can be understood that since the behavior of objects in infrared images is usually continuous, the restoration process is to use the historical behavior of the object to predict the current behavior of the object, and use the estimated current behavior of the object to improve the actual current behavior of the object to achieve the purpose of improving clarity. In addition, since the deep neural network model only processes images of N regions, compared with processing the entire frame of infrared images, it reduces processing overhead and improves performance.

[0060] Optionally, the monitoring terminal inputs the images of the N regions in each of the K infrared images and the images of the N regions in the first infrared image into a deep neural network model to obtain the restored images of the N regions output by the deep neural network model, including: the monitoring terminal inputs the images of the target regions in each of the K infrared images and the images of the target regions in the first infrared image into the deep neural network model to obtain the restored images of the target regions output by the deep neural network model, wherein the target regions include not only the images of the N regions but also partial images of regions adjacent to the N regions. For example, Figure 3As shown, region 2 is the disturbed region, and the target region may include parts of surrounding regions 1, 3, 4, 5, and 6. In this way, when N regions are replaced, the image separation between them and the surrounding regions is made smaller.

[0061] In summary, when the monitoring terminal detects that there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal can use the infrared images historically captured by the infrared camera to repair at least a partially interfered area in the first infrared image currently captured by the infrared camera, and obtain a repaired second infrared image. Since the clarity of the repaired at least partially interfered area in the second infrared image is higher than the clarity of at least a partially interfered area in the first infrared image, that is, the clarity of at least a partially interfered area is improved through image restoration, so that when the imaging of the infrared camera is interfered with, the user experience can be guaranteed by improving the clarity of the captured image.

[0062] Combination of the above Figure 2 The method for improving the clarity of a picture shot based on an infrared camera provided in an embodiment of the present application is described in detail. The following is a detailed description of a system for improving the clarity of a picture shot based on an infrared camera for executing the method for improving the clarity of a picture shot based on an infrared camera provided in an embodiment of the present application.

[0063] The system includes an application to a monitoring terminal, which is provided with an infrared camera. The system is configured as follows: the monitoring terminal determines whether there is at least a partially interfered area in an infrared image captured by the infrared camera; if there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal uses infrared images historically captured by the infrared camera to repair at least a partially interfered area in a first infrared image currently captured by the infrared camera, and obtains a repaired second infrared image, wherein the clarity of the repaired at least partially interfered area in the second infrared image is higher than the clarity of the at least partially interfered area in the first infrared image.

[0064] Optionally, the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera, including: the monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera.

[0065] Optionally, the monitoring terminal is provided with an antenna panel, which can send M beams with different directions in the collection direction of the infrared camera, where M is an integer greater than 1, and the infrared image collected by the infrared camera can be divided into M grid-shaped areas, the M areas correspond one-to-one to the directions of the M beams, the M beams correspond one-to-one to M wireless channels, and the M wireless channels are used for communication of the monitoring terminal; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera, including: the monitoring terminal measures the M wireless channels to obtain the channel quality corresponding to each of the M wireless channels, a total of M channel qualities; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera based on the M channel qualities; wherein, if N channel qualities among the M channel qualities are less than the channel quality threshold, and the N channel qualities correspond to N beams among the M beams, then the monitoring terminal determines that the N areas corresponding one-to-one to the N beams are at least partially interfered areas in the infrared image collected by the infrared camera.

[0066] Optionally, the system is also configured as follows: when N areas are at least partially interfered areas in the infrared image captured by the infrared camera, the monitoring terminal performs clarity analysis on the N areas in each of the multiple consecutive infrared images captured by the infrared camera; accordingly, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered areas in the first infrared image currently captured by the infrared camera, to obtain a repaired second infrared image, including: if more than a preset number of infrared images in the multiple consecutive infrared images have N areas whose clarity is lower than a clarity threshold, then in response to more than a preset number of infrared images in the multiple consecutive infrared images having N areas whose clarity is lower than a clarity threshold, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered areas in the first infrared image currently captured by the infrared camera, to obtain a repaired second infrared image.

[0067] Optionally, the energy level of the uplink channel is lower than the energy level of the downlink channel. If the M wireless channels are M uplink wireless channels, the monitoring terminal can perform clarity analysis on N areas in each of the multiple consecutive infrared images captured by the infrared camera. Otherwise, if the M wireless channels are M downlink wireless channels, the monitoring terminal directly uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

[0068] Optionally, the infrared images historically captured by the infrared camera include K infrared images, where K is an integer greater than 1, and the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image, including: the monitoring terminal inputs images of N areas in each of the K infrared images and images of N areas in the first infrared image into a deep neural network model to obtain repaired images of N areas output by the deep neural network model; the monitoring terminal uses the repaired images of the N areas to replace the images of the N areas in the first infrared image to obtain a repaired second infrared image.

[0069] Optionally, when infrared image restoration is just beginning, the K infrared images are the number of infrared image frames between the ith infrared image and the i+1th infrared image, and the number of infrared image frames is positively correlated with the value of i, and i traverses from 1 to K-1; or, when the restored infrared image is used to restore subsequent infrared images, the K infrared images are the number of infrared image frames between the ith infrared image and the i+1th infrared image, and the number of infrared image frames is negatively correlated with the value of i, and when i+1=K, the Kth infrared image is the infrared image frame previous to the first infrared image.

[0070] Optionally, the monitoring terminal inputs the images of the N areas in each of the K infrared images and the images of the N areas in the first infrared image into a deep neural network model to obtain restored images of the N areas output by the deep neural network model, including: the monitoring terminal inputs the images of the target area in each of the K infrared images and the image of the target area in the first infrared image into the deep neural network model to obtain the restored images of the target area output by the deep neural network model, wherein the target area includes not only the images of the N areas but also partial images of areas adjacent to the N areas.

[0071] Optionally, the first infrared image currently captured by the infrared camera refers to each frame of infrared image captured by the infrared camera since the monitoring terminal determines that there is at least a partially interfered area in the infrared image captured by the infrared camera.

[0072] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. For example, the electronic device may be a terminal device, or a chip (system) or other component or assembly that can be set in the terminal device. Figure 4 As shown, the electronic device 400 may include a processor 401. Optionally, the electronic device 400 may also include a memory 402 and / or a transceiver 403. The processor 401 is coupled to the memory 402 and the transceiver 403, such as being connected via a communication bus. In addition, the electronic device 400 may also be a chip, such as including the processor 401, in which case the transceiver may be an output and input interface of the chip.

[0073] Combine the following Figure 4 The components of the electronic device 400 are specifically introduced as follows:

[0074] The processor 401 is the control center of the electronic device 400, and may be a processor or a general term for multiple processing elements. For example, the processor 401 is one or more central processing units (CPUs), or may be application specific integrated circuits (ASICs), or may be one or more integrated circuits configured to implement the embodiments of the present application, such as one or more microprocessors (digital signal processors, DSPs), or one or more field programmable gate arrays (FPGAs).

[0075] Optionally, the processor 401 can execute various functions of the electronic device 400 by running or executing the software program stored in the memory 402 and calling the data stored in the memory 402, such as executing the above Figure 2 The method for improving the clarity of pictures taken with an infrared camera is shown.

[0076] In a specific implementation, as an embodiment, the processor 401 may include one or more CPUs, such as Figure 4 CPU0 and CPU1 are shown in FIG.

[0077] In a specific implementation, as an embodiment, the electronic device 400 may also include multiple processors. Each of these processors may be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). The processor here may refer to one or more devices, circuits, and / or processing cores for processing data (such as computer programs or instructions).

[0078] The memory 402 is used to store the software program for executing the solution of the present application, and the execution is controlled by the processor 401. The specific implementation method can refer to the above method embodiment, which will not be repeated here.

[0079] Optionally, the memory 402 may be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 402 may be integrated with the processor 401, or may exist independently and access the computer through the interface circuit ( Figure 4 (not shown) is coupled to the processor 401, which is not specifically limited in this embodiment of the present application.

[0080] The transceiver 403 is used for communication with other electronic devices. For example, if the electronic device 400 is a terminal device, the transceiver 403 can be used to communicate with a network device, or with another terminal device. For another example, if the electronic device 400 is a network device, the transceiver 403 can be used to communicate with a terminal device, or with another network device.

[0081] Optionally, the transceiver 403 may include a receiver and a transmitter ( Figure 4 The receiver is used to implement a receiving function, and the transmitter is used to implement a sending function.

[0082] Optionally, the transceiver 403 may be integrated with the processor 401, or may exist independently and communicate with the electronic device 400 through an interface circuit ( Figure 4 (not shown) is coupled to the processor 401, which is not specifically limited in this embodiment of the present application.

[0083] Understandably, Figure 4 The structure of the electronic device 400 shown in the figure does not constitute a limitation on the electronic device, and the actual electronic device may include more or fewer components than those shown in the figure, or combine certain components, or arrange the components differently.

[0084] In addition, the technical effects of the electronic device 400 can refer to the technical effects of the methods described in the above method embodiments, which will not be repeated here.

[0085] It should be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0086] It should also be understood that the memory in the embodiments of the present application may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of random access memory (RAM) are available, such as static RAM (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0087] The above embodiments may be implemented in whole or in part by software, hardware (such as circuits), firmware or any other combination thereof. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer program or instruction may be stored in a computer-readable storage medium, or may be transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer program or instruction may be transmitted from one website, computer, server or data center to another website, computer, server or data center by wired (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium may be a magnetic medium (for example, a floppy disk, a hard disk, a tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium may be a solid-state hard disk.

[0088] It should be understood that the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. A and B can be singular or plural. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship, but it may also indicate an "and / or" relationship. Please refer to the context for specific understanding.

[0089] In this application, "at least one" means one or more, and "plurality" means two or more. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0090] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0091] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0092] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0093] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0094] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0096] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, and other media that can store program codes.

[0097] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for improving the clarity of a picture shot by an infrared camera, characterized in that: Applied to a monitoring terminal, the monitoring terminal is provided with an infrared camera, and the method comprises: The monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera; If there is at least a partially interfered area in the infrared image captured by the infrared camera, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least a partially interfered area in the first infrared image currently captured by the infrared camera, to obtain a repaired second infrared image, wherein the clarity of at least a partially interfered area in the first infrared image is less than the clarity of the corresponding area of ​​at least a partially interfered area in the first infrared image in the second infrared image, and the first infrared image currently captured by the infrared camera refers to each frame of infrared image captured by the infrared camera since the monitoring terminal determined that there is at least a partially interfered area in the infrared image captured by the infrared camera; The monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera, including: The monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera; The monitoring terminal is provided with an antenna panel, and the antenna panel can send M beams with different directions in the collection direction of the infrared camera, where M is an integer greater than 1, and the infrared image collected by the infrared camera can be divided into M grid-shaped areas, and the M areas correspond to the directions of the M beams one-to-one, and the M beams correspond to M wireless channels one-to-one, and the M wireless channels are used for communication of the monitoring terminal; the monitoring terminal determines whether there is at least a partially interfered area in the infrared image collected by the infrared camera through the wireless signal monitored in the collection direction of the infrared camera, including: The monitoring terminal measures the M wireless channels to obtain channel qualities corresponding to the M wireless channels, a total of M channel qualities; The monitoring terminal determines whether there is at least a partially interfered area in the infrared image captured by the infrared camera based on the M channel qualities; wherein, if N channel qualities among the M channel qualities are less than a channel quality threshold, and the N channel qualities correspond to N beams among the M beams, then the monitoring terminal determines that the N areas corresponding one by one to the N beams are at least partially interfered areas in the infrared image captured by the infrared camera.

2. The method according to claim 1, characterized in that: The method further comprises: In a case where the N areas are at least partially interfered areas in the infrared image captured by the infrared camera, the monitoring terminal performs clarity analysis on the N areas in each of the plurality of consecutive infrared images captured by the infrared camera; Correspondingly, the monitoring terminal uses the infrared image historically collected by the infrared camera to repair at least part of the interfered area in the first infrared image currently collected by the infrared camera to obtain a repaired second infrared image, including: If more than a preset number of the consecutive infrared images have N areas whose clarity is lower than a clarity threshold, then in response to more than a preset number of the consecutive infrared images having N areas whose clarity is lower than a clarity threshold, the monitoring terminal uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

3. The method according to claim 2, characterized in that The energy level of the uplink channel is lower than the energy level of the downlink channel. If the M wireless channels are M uplink wireless channels, the monitoring terminal can perform clarity analysis on the N areas in each of the multiple consecutive infrared images captured by the infrared camera. Otherwise, if the M wireless channels are M downlink wireless channels, the monitoring terminal directly uses the infrared images historically captured by the infrared camera to repair at least part of the interfered area in the first infrared image currently captured by the infrared camera to obtain a repaired second infrared image.

4. The method according to claim 2, characterized in that: The infrared images historically collected by the infrared camera include K infrared images, where K is an integer greater than 1. The monitoring terminal uses the infrared images historically collected by the infrared camera to repair at least part of the interfered area in the first infrared image currently collected by the infrared camera to obtain a repaired second infrared image, including: The monitoring terminal inputs the images of the N regions in each of the K infrared images and the images of the N regions in the first infrared image into a deep neural network model to obtain restored images of the N regions output by the deep neural network model; The monitoring terminal uses the repaired images of the N areas to replace the images of the N areas in the first infrared image to obtain the repaired second infrared image.

5. The method according to claim 4, characterized in that The monitoring terminal inputs the images of the N regions in each of the K infrared images and the images of the N regions in the first infrared image into a deep neural network model to obtain restored images of the N regions output by the deep neural network model, including: The monitoring terminal inputs the image of the target area in each of the K infrared images and the image of the target area in the first infrared image into the deep neural network model to obtain a repaired image of the target area output by the deep neural network model, wherein the target area includes not only the images of the N areas but also partial images of areas adjacent to the N areas.

Images