Video display method, system and storage medium based on visible light and thermal imaging

Abstract

The present application proposes a video display method, system and storage medium based on visible light and thermal imaging, the method comprising: capturing a visible light video stream and a thermal imaging video stream with the same timestamp through a shooting device; obtaining display instructions input by a user, including a visible light mode, a thermal imaging mode and a splicing mode; in response to obtaining a display instruction of a visible light mode or a thermal imaging mode, receiving a visible light video stream or a thermal imaging video stream and displaying it on a control terminal; in response to obtaining a display instruction of a splicing mode: compressing the visible light video stream and / or the thermal imaging video stream; based on a timestamp reference, splicing the compressed visible light video stream and the thermal imaging video stream into a target video stream for synchronous display and displaying it on a control terminal. The present application can display two complete video screens at the same time by compressing and splicing the two video streams, and can switch between a single screen and a spliced ​​screen, thereby improving the flexibility of video display.

Description

Technical Field

[0001] The present application relates to the technical field of video image processing, and in particular to a video display method, system and storage medium based on visible light and thermal imaging. Background Art

[0002] A drone is an unmanned aircraft controlled by a radio remote control device and a self-contained program control device. It can carry a gimbal camera to shoot at various angles and postures and transmit the images back to the control terminal on the ground. With the rapid development of gimbal cameras, gimbal cameras can be equipped with two lenses to shoot two types of video images in the same scene, such as visible light video images and thermal imaging video images.

[0003] How to flexibly display the video images sent back by the two-lens PTZ camera on the control terminal to improve the user experience has become an important issue. The existing technology generally adopts the following two solutions:

[0004] (i) Crop the two video images separately, and then splice the two cropped video images for display. However, this method only retains the important video images, and will inevitably discard part of the video images, which may lead to the loss of video image information, such as the edge video image information of the video image, the temperature information of the thermal imaging video image, etc.;

[0005] (ii) Display one of the video images on the main screen, and display the other video image in a small window superimposed on the main screen. The two videos can be switched between the main screen and the small window. Although this method can preserve the video image information, has a high definition, and can bring convenience to users to a certain extent, for the video image displayed in the small window, detailed video image information cannot be seen, and any zoom / magnification operation cannot be performed.

[0006] Both of the above solutions cannot display two complete video screen information at the same time, and lack video display flexibility. Therefore, it is necessary to provide a video display method, system and storage medium based on visible light and thermal imaging to solve the above-mentioned technical problems. Summary of the invention

[0007] The present application proposes a video display method, system and storage medium based on visible light and thermal imaging, which are used to solve the above-mentioned technical problems existing in the prior art.

[0008] According to the first aspect of the present application, a video display method based on visible light and thermal imaging is proposed, comprising:

[0009] Capturing a visible light video stream and a thermal imaging video stream with the same timestamp by a shooting device;

[0010] Acquire a display instruction input by a user, wherein the display instruction is used to indicate a video display mode of the control terminal, wherein the video display mode includes a visible light mode, a thermal imaging mode, and a splicing mode;

[0011] In response to obtaining a display instruction of a visible light mode or a thermal imaging mode, receiving a visible light video stream or a thermal imaging video stream sent by the shooting device and displaying it on the control terminal;

[0012] In response to obtaining the display instruction of the splicing mode, the following operations are performed:

[0013] compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution;

[0014] Based on the timestamp reference, the compressed visible light video stream and thermal imaging video stream are spliced ​​into a target video stream for synchronous display;

[0015] The target video stream sent by the shooting device is received and displayed on the control terminal.

[0016] Preferably, the thermal imaging video stream includes a plurality of video frames and a plurality of temperature data frames with timestamp identifiers. In response to obtaining a display instruction of a thermal imaging mode, or in response to obtaining a display instruction of a splicing mode, the method includes:

[0017] Parsing the temperature data of the temperature data frame by the shooting device, and acquiring the point temperature value and point coordinates of the target temperature in the temperature data;

[0018] The control terminal receives the video frame sent by the shooting device and the corresponding point temperature value and point coordinates of the target temperature, and displays the point temperature value of the target temperature in the video frame through point coordinate mapping.

[0019] Preferably, in response to obtaining a display instruction of a splicing mode, compressing the thermal imaging video stream includes:

[0020] Compressing the video frames of the thermal imaging video stream, and recording a first timestamp corresponding to the latest compressed video frame;

[0021] The step of parsing the temperature data of the temperature data frame by the photographing device includes:

[0022] Based on the first timestamp reference, temperature data corresponding to the temperature data frame is parsed.

[0023] Preferably, the control terminal has a visible light display process and a thermal imaging display process in parallel, and both the visible light display process and the thermal imaging display process have a signal bit that communicates with the shooting device, and the signal bit has at least two states, and the state change of the signal bit is used to represent the display instruction input by the user.

[0024] Preferably, the signal bit has two states, a high state and a low state, and the state change of the signal bit is used to characterize the display instruction input by the user, including:

[0025] When the signal bit of the visible light display process is high and the signal bit of the thermal imaging display process is low, the display instruction is a visible light mode;

[0026] When the signal bit of the visible light display process is low and the signal bit of the thermal imaging display process is high, the display instruction is a thermal imaging mode;

[0027] When the signal bit of the visible light display process and the signal bit of the thermal imaging display process are both high, the display instruction is a splicing mode.

[0028] Preferably, when performing a zoom operation on the thermal imaging video stream, the method includes:

[0029] Adjusting the mapping relationship of the point coordinates of the target temperature on the control terminal according to the resolution and zoom factor of the thermal imaging video stream;

[0030] The point coordinate mapping of the target temperature changes as follows:

[0031] x = (x1 / k + w / 2) - (w / 2 / k)

[0032] y = (y1 / k+h / 2)-(h / 2 / k)

[0033] Among them, x and y are the horizontal and vertical coordinates before zooming, x1 and y1 are the horizontal and vertical coordinates after zooming, k is the zoom factor, w is the width value of the thermal imaging video stream resolution, and h is the height value of the thermal imaging video stream resolution.

[0034] Preferably, the resolutions of the visible light video stream and the thermal imaging video stream are in the same ratio, and the compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution includes:

[0035] The visible light video stream and / or the thermal imaging video stream are compressed by an interpolation algorithm so that the visible light video stream and the thermal imaging video stream have the same resolution.

[0036] According to a second aspect of the present application, a video display system based on visible light and thermal imaging is proposed, the system comprising a shooting device and a control terminal, wherein:

[0037] The shooting device is used to: capture a visible light video stream and a thermal imaging video stream having the same timestamp;

[0038] The control terminal is used to: obtain a display instruction input by a user, wherein the display instruction is used to indicate a video display mode of the control terminal, wherein the video display mode includes a visible light mode, a thermal imaging mode, and a splicing mode;

[0039] The photographing device is also used for:

[0040] In response to obtaining a display instruction of a visible light mode or a thermal imaging mode, sending a visible light video stream or a thermal imaging video stream to the control terminal;

[0041] In response to obtaining a display instruction of the splicing mode, performing the following operations: compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution, splicing the compressed visible light video stream and the thermal imaging video stream into a target video stream for synchronous display based on a timestamp reference, and sending the target video stream to the control terminal;

[0042] The control terminal is also used for:

[0043] In response to obtaining a display instruction of a visible light mode or a thermal imaging mode, receiving and displaying a visible light video stream or a thermal imaging video stream sent by the shooting device;

[0044] In response to obtaining a display instruction of the splicing mode, the target video stream sent by the shooting device is received and displayed.

[0045] Preferably, the photographing device is a gimbal camera, and the control terminal is a drone remote controller.

[0046] According to a third aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, a video display method provided in any embodiment of the first aspect is implemented.

[0047] The present application proposes a video display method, system and storage medium based on visible light and thermal imaging. For the dual-channel video streams shot by the dual-lens pan-tilt camera, the video display mode is divided into visible light mode, thermal imaging mode and splicing mode. When the display mode is the visible light mode and thermal imaging mode, the visible light video stream and the thermal imaging video stream are displayed on a single screen; when the display mode is the splicing mode, two complete video screens can be displayed at the same time by compressing the two video streams and splicing them for display, and simultaneous zooming and magnification can be achieved. For the thermal imaging video stream, temperature measurement information can also be displayed. It is also possible to switch between single screen and spliced ​​screen, which improves the flexibility of video display and enhances user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] The accompanying drawings are included to provide a further understanding of the embodiments and are incorporated into and constitute a part of this specification. The accompanying drawings illustrate the embodiments and are used together with the description to explain the principles of the present invention. It will be easy to recognize other embodiments and many expected advantages of the embodiments because they become better understood by reference to the following detailed description. The elements of the drawings are not necessarily to scale with each other. The same reference numerals refer to corresponding similar parts.

[0049] Figure 1 is a flow chart of a video display method based on visible light and thermal imaging according to an embodiment of the present application;

[0050] Figure 2 It is the protocol format of the private protocol according to the embodiment of the present application;

[0051] Figure 3 is a schematic diagram of a single-screen display interface of a control terminal according to an embodiment of the present application;

[0052] Figure 4 is a schematic diagram of a spliced ​​screen display interface of a control terminal according to an embodiment of the present application;

[0053] Figure 5 is a schematic diagram of a temperature measurement display interface of a control terminal according to an embodiment of the present application;

[0054] Figure 6 It is a system architecture diagram of a video display system based on visible light and thermal imaging according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In order to make the purpose, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below in conjunction with the accompanying drawings and Examples. It should be understood that the specific embodiments described herein are only configured to explain the present invention and are not configured to limit the present invention. For those skilled in the art, the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention.

[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "include..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0057] Figure 1 FIG. 4 shows a flow chart of a video display method based on visible light and thermal imaging according to an embodiment of the present application. Figure 1 The method comprises the following steps:

[0058] S100: Capture a visible light video stream and a thermal imaging video stream with the same timestamp using a shooting device.

[0059] The shooting device uses two image sensors to shoot the visible light video stream and thermal imaging video stream of the same scene in real time. In order to ensure the consistency of the subsequent display of the two video streams, the same timestamp is added to the two video streams.

[0060] It should be noted that the video display method provided in this embodiment is applied to a scenario where the shooting device includes two image sensors. In other embodiments, it can also be applied to a scenario where the shooting device includes more than two image sensors.

[0061] It should be noted that in the video display method provided in this embodiment, one of the two image sensors is a visible light image sensor and the other is a thermal imaging image sensor. In other embodiments, the two image sensors can also be any one of a visible light image sensor, a thermal imaging image sensor, an infrared image sensor, and a zoom image sensor.

[0062] It should be noted that, in the video display method provided in this embodiment, in order to prevent deformation of the target in the video when the video stream is subsequently compressed, the videos captured by the visible light image sensor and the thermal imaging image sensor have the same resolution ratio, for example, the resolution of the visible light video stream may be 1280*1024, and the resolution of the thermal imaging video stream may be 640*512. In other embodiments, the videos captured by the visible light image sensor and the thermal imaging image sensor may also have different resolution ratios.

[0063] S200: Obtain a display instruction input by a user, where the display instruction is used to indicate a video display mode of the control terminal, and the video display modes include a visible light mode, a thermal imaging mode, and a splicing mode.

[0064] In a specific implementation, the user may input the display instruction by pressing an operation button on the control terminal, or may input the display instruction by touching the display screen of the control terminal.

[0065] In one embodiment provided in the present application, after the control terminal obtains the display instruction input by the user, the display instruction is transmitted to the shooting device through the communication link between the control terminal and the shooting device. The communication link can transmit data through the private protocol of the network and the serial port.

[0066] Figure 2 The protocol format of the private protocol according to the embodiment of the present application is shown, such as Figure 2 As shown, the protocol format uses a header, data field length, command ID, header CRC32 check, data field and whole packet CRC32 check. The control terminal sends the corresponding data packet according to the private protocol definition, and the shooting device parses the data packet according to the protocol format after receiving the data packet; for example, the command ID 0xaa is defined as the command for the video display mode of the control terminal, 0x00 in the data field is the visible light mode, 0x01 is the thermal imaging mode, and 0x02 is the splicing mode. The shooting device receives the corresponding command and executes the corresponding action. At the same time, this embodiment adds a header CRC check and a whole packet CRC check to ensure the reliability and integrity of the data packet.

[0067] In another embodiment provided by the present application, after the control terminal obtains the display instruction input by the user, it sends the display instruction to the shooting device through the state change of its signal bit. The details are as follows:

[0068] The control terminal has a visible light display process and a thermal imaging display process in parallel. Both the visible light display process and the thermal imaging display process have a signal bit for communicating with the shooting device. The signal bit has at least two states. The state change of the signal bit can be used to represent the display instruction input by the user.

[0069] In this embodiment, the signal bit is a Boolean data variable, including two values ​​0 or 1, and the signal bit has two states, high and low, where 1 is high and 0 is low. In other embodiments, the signal bit can also be 0 for high and 1 for low.

[0070] The display command input by the user is represented by the state change of the signal bit. The specific settings are as follows:

[0071] When the signal bit of the visible light display process is high and the signal bit of the thermal imaging display process is low, the display instruction is the visible light mode; when the signal bit of the visible light display process is low and the signal bit of the thermal imaging display process is high, the display instruction is the thermal imaging mode; when the signal bit of the visible light display process and the signal bit of the thermal imaging display process are both high, the display instruction is the splicing mode.

[0072] S30a, in response to obtaining a display instruction of the visible light mode or the thermal imaging mode, receiving the visible light video stream or the thermal imaging video stream sent by the shooting device and displaying it on the control terminal.

[0073] Figure 3 A schematic diagram of a single-screen display interface of a control terminal according to an embodiment of the present application is shown. Figure 3 As shown, when the display instruction input by the user is a visible light mode or a thermal imaging mode, the shooting device directly transmits the corresponding visible light video stream or thermal imaging video stream to the control terminal after receiving the display instruction, that is, the visible light video stream or the thermal imaging video stream is displayed in a single screen.

[0074] In one embodiment provided in the present application, when the visible light video stream or the thermal imaging video stream is displayed in a single screen, the visible light video stream or the thermal imaging video stream can be stretched to cover the entire display screen of the control terminal.

[0075] S30b, in response to obtaining a display instruction of the stitching mode, performing the following operations: compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution; stitching the compressed visible light video stream and the thermal imaging video stream into a target video stream for synchronous display based on a timestamp reference; receiving the target video stream sent by the shooting device and displaying it on the control terminal.

[0076] When the display instruction input by the user is a splicing mode, the visible light video stream and / or the thermal imaging video stream are first compressed by an interpolation algorithm so that the visible light video stream and the thermal imaging video stream have the same resolution. In this embodiment, taking the resolution of the visible light video stream as 1280*1024 and the resolution of the thermal imaging video stream as 640*512 as an example, the resolution of the visible light video stream is compressed to 640*512 by an interpolation algorithm, and the compressed visible light video stream and the thermal imaging video stream are spliced ​​into a target video stream according to a timestamp reference to maintain the consistency of the two video streams. The shooting device then sends the target video stream to the control terminal for display.

[0077] Figure 4 FIG. 4 shows a schematic diagram of a spliced ​​screen display interface of a control terminal according to an embodiment of the present application. Figure 4 As shown, in the splicing mode, the visible light video stream and the thermal imaging video stream each occupy half of the screen on the display screen of the control terminal.

[0078] It should be noted that the video display method provided in this embodiment sets the resolution of the visible light video stream to be higher than that of the thermal imaging video stream. In other embodiments, the resolution of the thermal imaging video stream may also be set to be higher than that of the visible light video stream.

[0079] It should be noted that the video display method provided in this embodiment is to compress a visible light video stream with a higher resolution. In other embodiments, it is also possible to stretch a thermal imaging video stream with a lower resolution.

[0080] It should be noted that the video display method provided in this embodiment only compresses one of the video streams. In other embodiments, two video streams may also be compressed to the same resolution at the same time.

[0081] Furthermore, in this embodiment, the thermal imaging video stream includes multiple video frames and multiple temperature data frames with timestamp identifiers. Therefore, in response to obtaining a display instruction of the thermal imaging mode, or in response to obtaining a display instruction of the splicing mode, a temperature measurement display function is added to the thermal imaging video stream.

[0082] Specifically, when the shooting device receives a display instruction of a thermal imaging mode or a stitching mode, the shooting device parses the temperature data of the temperature data frame and obtains the point temperature value and point coordinates of the target temperature in the temperature data. The control terminal receives the video frame sent by the shooting device and the corresponding point temperature value and point coordinates of the target temperature, and displays the point temperature value of the target temperature in the video frame through point coordinate mapping.

[0083] In one embodiment provided in the present application, the target temperature is selected from the highest temperature and the lowest temperature. In other embodiments, the target temperature can be selected according to actual needs.

[0084] Figure 5 FIG. 4 shows a schematic diagram of a temperature measurement display interface of a control terminal according to an embodiment of the present application. Figure 5 As shown, in thermal imaging mode and stitching mode, for the thermal imaging video stream, the control terminal will map the points where the highest and lowest temperatures of the video frame are located and the temperature values ​​of the points on the screen.

[0085] In one embodiment provided in the present application, the photographing device transmits the point temperature value and point coordinates of the target temperature to the control terminal through the private protocol provided in step S200.

[0086] Further, in response to obtaining the display instruction of the splicing mode, compressing the thermal imaging video stream includes:

[0087] The video frames of the thermal imaging video stream are compressed, and the first timestamp corresponding to the latest compressed video frame is recorded.

[0088] The temperature data of the temperature data frame is parsed by the shooting device, including:

[0089] Based on the first timestamp reference, the temperature data of the corresponding temperature data frame is parsed.

[0090] In one embodiment provided in the present application, when compressing the thermal imaging video stream in the splicing mode, only the video frames in the thermal imaging video stream need to be compressed, and the temperature measurement and display function is directly realized by parsing the temperature data in the temperature data frame. Therefore, after compressing the video frames of the thermal imaging video stream, the first timestamp corresponding to the latest compressed video frame is recorded, and when parsing the temperature data in multiple temperature data frames later, the temperature data frames corresponding to the first timestamp can be directly parsed, and the temperature data frames corresponding to the first timestamp do not need to be parsed, thereby reducing part of the calculation amount.

[0091] Furthermore, when the thermal imaging video stream in the thermal imaging mode or the splicing mode is zoomed, the point coordinates of the target temperature on the display screen of the control terminal will also change. Therefore, in one embodiment provided by the present application, when the thermal imaging video stream is zoomed, the mapping relationship of the point coordinates of the target temperature on the control terminal is adjusted according to the resolution and zoom multiple of the thermal imaging video stream, thereby realizing the automatic mapping of the point coordinates of the target temperature on the control terminal during the zoom operation. Among them, the mapping changes of the point coordinates of the target temperature are as follows:

[0092] x = (x1 / k + w / 2) - (w / 2 / k)

[0093] y = (y1 / k+h / 2)-(h / 2 / k)

[0094] Among them, x and y are the horizontal and vertical coordinates before zooming, x1 and y1 are the horizontal and vertical coordinates after zooming, k is the zoom factor, w is the width value of the thermal imaging video stream resolution, and h is the height value of the thermal imaging video stream resolution.

[0095] Based on the above-mentioned video display method based on visible light and thermal imaging, the present application also proposes a video display system based on visible light and thermal imaging. Figure 6 FIG. 4 shows a system architecture diagram of a video display system based on visible light and thermal imaging according to an embodiment of the present application. Figure 6 As shown, the video display system includes a shooting device 10 and a control terminal 20, wherein:

[0096] The shooting device 10 is used to capture a visible light video stream and a thermal imaging video stream with the same timestamp.

[0097] The control terminal 20 is used to obtain a display instruction input by a user, where the display instruction is used to indicate a video display mode of the control terminal 20 , and the video display modes include a visible light mode, a thermal imaging mode, and a splicing mode.

[0098] The photographing device 10 is also used for:

[0099] In response to obtaining a display instruction of the visible light mode or the thermal imaging mode, a visible light video stream or a thermal imaging video stream is sent to the control terminal 20; in response to obtaining a display instruction of the splicing mode, the following operations are performed:

[0100] The visible light video stream and / or the thermal imaging video stream are compressed so that the visible light video stream and the thermal imaging video stream have the same resolution. Based on a timestamp reference, the compressed visible light video stream and the thermal imaging video stream are spliced ​​into a target video stream for synchronous display, and the target video stream is sent to the control terminal 20.

[0101] The control terminal 20 is also used for:

[0102] In response to obtaining a display instruction of the visible light mode or the thermal imaging mode, the visible light video stream or the thermal imaging video stream sent by the shooting device 10 is received and displayed; in response to obtaining a display instruction of the stitching mode, the target video stream sent by the shooting device 10 is received and displayed.

[0103] In one embodiment provided in the present application, the shooting device 10 is a pan-tilt camera, and the control terminal 20 is a drone remote controller. In other embodiments, the shooting device can be any device, apparatus, or terminal with a shooting function, and the control terminal can be any device, apparatus, or terminal with a communication control function, which is not limited here.

[0104] Based on the above-mentioned video display method based on visible light and thermal imaging, the present application also proposes a computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the video display method based on visible light and thermal imaging provided by any of the above-mentioned embodiments is implemented.

[0105] The present application proposes a video display method, system and storage medium based on visible light and thermal imaging. For the dual-channel video streams shot by the dual-lens pan-tilt camera, the video display mode is divided into visible light mode, thermal imaging mode and splicing mode. When the display instruction input by the user is the visible light mode or thermal imaging mode, the visible light video stream or thermal imaging video stream is displayed as a single screen on the display screen of the control terminal, and the temperature measurement information can also be displayed for the thermal imaging video stream; when the display instruction input by the user is the splicing mode, two complete video screens can be displayed at the same time by compressing the two video streams and splicing them for display. The two video screens can achieve simultaneous zooming and magnification, and the temperature measurement information can also be displayed for the thermal imaging video stream. And it is possible to switch between single screen and spliced ​​screen, which improves the flexibility of video display and enhances user experience.

[0106] In the embodiments of the present application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device / system / method embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0107] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0108] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0109] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0110] Obviously, those skilled in the art can make various modifications and changes to the embodiments of the present invention without departing from the spirit and scope of the present invention. In this way, if these modifications and changes are within the scope of the claims of the present invention and their equivalents, the present invention is also intended to cover these modifications and changes. The word "comprising" does not exclude the presence of other elements or steps not listed in the claims. The simple fact that certain measures are recorded in mutually different dependent claims does not indicate that the combination of these measures cannot be used to profit. Any reference numerals in the claims should not be considered to limit the scope.

Claims

1. A video display method based on visible light and thermal imaging, characterized in that: include: Capturing a visible light video stream and a thermal imaging video stream with the same timestamp by a shooting device, wherein the resolutions of the visible light video stream and the thermal imaging video stream are in the same ratio; Acquire a display instruction input by a user, wherein the display instruction is used to indicate a video display mode of the control terminal, wherein the video display mode includes a visible light mode, a thermal imaging mode, and a splicing mode; In response to obtaining a display instruction of a visible light mode or a thermal imaging mode, receiving a visible light video stream or a thermal imaging video stream sent by the shooting device and displaying it on the control terminal; In response to obtaining the display instruction of the splicing mode, the following operations are performed: compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution; Based on the timestamp reference, the compressed visible light video stream and thermal imaging video stream are spliced ​​into a target video stream for synchronous display; Receiving the target video stream sent by the shooting device and displaying it on the control terminal; The thermal imaging video stream includes a plurality of video frames and a plurality of temperature data frames with timestamp identifiers. In response to obtaining a display instruction of a thermal imaging mode, or in response to obtaining a display instruction of a splicing mode, the method includes: Parsing the temperature data of the temperature data frame by the shooting device, and acquiring the point temperature value and point coordinates of the target temperature in the temperature data; The control terminal receives the video frame sent by the shooting device and the corresponding point temperature value and point coordinates of the target temperature, and displays the point temperature value of the target temperature in the video frame through point coordinate mapping; When a zoom operation is performed on the thermal imaging video stream, the method includes: Adjusting the mapping relationship of the point coordinates of the target temperature on the control terminal according to the resolution and zoom factor of the thermal imaging video stream; The point coordinate mapping of the target temperature changes as follows: x = (x1 / k + w / 2) - (w / 2 / k) y = (y1 / k+h / 2)-(h / 2 / k) Among them, x and y are the horizontal and vertical coordinates before zooming, x1 and y1 are the horizontal and vertical coordinates after zooming, k is the zoom factor, w is the width value of the thermal imaging video stream resolution, and h is the height value of the thermal imaging video stream resolution.

2. The video display method based on visible light and thermal imaging according to claim 1, characterized in that: In response to obtaining a display instruction of the splicing mode, compressing the thermal imaging video stream includes: Compressing the video frames of the thermal imaging video stream, and recording a first timestamp corresponding to the latest compressed video frame; The step of parsing the temperature data of the temperature data frame by the photographing device includes: Based on the first timestamp reference, temperature data corresponding to the temperature data frame is parsed.

3. The video display method based on visible light and thermal imaging according to claim 1, characterized in that: The control terminal has a visible light display process and a thermal imaging display process in parallel. Both the visible light display process and the thermal imaging display process have a signal bit that communicates with the shooting device. The signal bit has at least two states. The state change of the signal bit is used to represent the display instruction input by the user.

4. The video display method based on visible light and thermal imaging according to claim 3, characterized in that: The signal bit has two states, a high state and a low state. The state change of the signal bit is used to represent the display instruction input by the user, including: When the signal bit of the visible light display process is high and the signal bit of the thermal imaging display process is low, the display instruction is a visible light mode; When the signal bit of the visible light display process is low and the signal bit of the thermal imaging display process is high, the display instruction is a thermal imaging mode; When the signal bit of the visible light display process and the signal bit of the thermal imaging display process are both high, the display instruction is a splicing mode.

5. The video display method based on visible light and thermal imaging according to claim 1, characterized in that: The resolutions of the visible light video stream and the thermal imaging video stream are in the same ratio, and the compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution includes: The visible light video stream and / or the thermal imaging video stream are compressed by an interpolation algorithm so that the visible light video stream and the thermal imaging video stream have the same resolution.

6. A video display system based on visible light and thermal imaging, characterized in that: It includes a shooting device and a control terminal, wherein: The shooting device is used to: capture a visible light video stream and a thermal imaging video stream with the same timestamp, and the resolutions of the visible light video stream and the thermal imaging video stream are in the same ratio; The control terminal is used to: obtain a display instruction input by a user, wherein the display instruction is used to indicate a video display mode of the control terminal, wherein the video display mode includes a visible light mode, a thermal imaging mode, and a splicing mode; The photographing device is also used for: In response to obtaining a display instruction of a visible light mode or a thermal imaging mode, sending a visible light video stream or a thermal imaging video stream to the control terminal; In response to obtaining a display instruction of the splicing mode, performing the following operations: compressing the visible light video stream and / or the thermal imaging video stream so that the visible light video stream and the thermal imaging video stream have the same resolution, splicing the compressed visible light video stream and the thermal imaging video stream into a target video stream for synchronous display based on a timestamp reference, and sending the target video stream to the control terminal; The thermal imaging video stream includes a plurality of video frames with timestamp identifiers and a plurality of temperature data frames. In response to obtaining a display instruction of a thermal imaging mode, or in response to obtaining a display instruction of a stitching mode, the shooting device parses the temperature data of the temperature data frame and obtains a point temperature value and a point coordinate of a target temperature in the temperature data; The control terminal is also used for: In response to obtaining a display instruction of a visible light mode or a thermal imaging mode, receiving and displaying a visible light video stream or a thermal imaging video stream sent by the shooting device; In response to obtaining a display instruction of the splicing mode, receiving and displaying the target video stream sent by the shooting device; In response to obtaining a display instruction of a thermal imaging mode, or in response to obtaining a display instruction of a stitching mode, receiving a video frame sent by the shooting device and a corresponding point temperature value and point coordinates of the target temperature, and displaying the point temperature value of the target temperature in the video frame through point coordinate mapping; When the thermal imaging video stream is zoomed, the mapping relationship of the point coordinates of the target temperature on the control terminal is adjusted according to the resolution and zoom factor of the thermal imaging video stream; The point coordinate mapping of the target temperature changes as follows: x = (x1 / k + w / 2) - (w / 2 / k) y = (y1 / k+h / 2)-(h / 2 / k) Among them, x and y are the horizontal and vertical coordinates before zooming, x1 and y1 are the horizontal and vertical coordinates after zooming, k is the zoom factor, w is the width value of the thermal imaging video stream resolution, and h is the height value of the thermal imaging video stream resolution.

7. The video display system based on visible light and thermal imaging according to claim 6, characterized in that: The shooting device is a pan-tilt camera, and the control terminal is a drone remote controller.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the video display method based on visible light and thermal imaging as described in any one of claims 1 to 5 is implemented.

Images