Image processing method, device and equipment

Abstract

The present application provides an image processing method, device and equipment, the method comprising: acquiring a visible light image and a first thermal imaging image corresponding to the visible light image; acquiring multiple regional images in the visible light image according to the visible light image and the first thermal imaging image, the absolute value of the difference between the integral image density of the regional image and the integral image density of the visible light image being less than or equal to a first threshold; determining the target distance corresponding to each pixel point in the visible light image according to the multiple regional images and the visible light image, the target distance being the distance between the photographed object corresponding to the pixel point and the photographing device; updating the first thermal imaging image according to the target distance to obtain a second thermal imaging image. The display accuracy of the thermal imaging image is improved.

Description

Technical Field

[0001] The present application relates to the field of thermal imaging technology, and in particular to an image processing method, device and equipment. Background Art

[0002] The dual-photothermal camera can generate thermal images based on the infrared light waves emitted by the object. However, infrared light waves will attenuate to varying degrees during long-distance transmission. Therefore, the distance between the object and the shooting device can be used to obtain the infrared light waves actually emitted by the object, and then determine the temperature of the object.

[0003] Currently, the distance between the object and the shooting device can be determined by manual calibration. For example, when the dual-photothermal camera generates a thermal image, the user can divide the thermal image into multiple display areas, and determine the distance between each display area and the dual-photothermal camera by manual measurement, and then correct the thermal image of each display area. However, the manual calibration method is relatively complicated, and the distance between different display objects in the display area and the dual-photothermal camera may also be different, which results in a low accuracy of the displayed thermal image. Summary of the invention

[0004] The present application provides an image processing method, device and equipment for solving the technical problem of low accuracy of thermal images displayed in the prior art.

[0005] In a first aspect, the present application provides an image processing method, the method comprising:

[0006] Acquire a visible light image and a first thermal imaging image corresponding to the visible light image;

[0007] Acquire a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein an absolute value of a difference between an integral image density of the regional images and an integral image density of the visible light image is less than or equal to a first threshold;

[0008] Determine, according to the multiple regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image, wherein the target distance is a distance between a photographing object corresponding to the pixel point and a photographing device;

[0009] The first thermal imaging image is updated according to the target distance to obtain a second thermal imaging image.

[0010] In a possible implementation, determining, according to the multiple regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image includes:

[0011] Determine, according to the multiple regional images, a first depth distance map for each regional image, wherein the first depth distance map is used to indicate distances corresponding to pixels in the regional image;

[0012] Determine, according to the visible light image, a second depth distance map of the visible light image, wherein the second depth distance map is used to indicate distances corresponding to pixels in the visible light image;

[0013] The target distance is determined according to the plurality of first depth distance maps and the second depth distance map.

[0014] In a possible implementation, for any one of the first depth distance maps; determining the target distance according to the multiple first depth distance maps and the second depth distance map includes:

[0015] Determining a target position of the first depth distance map in the second depth distance map;

[0016] According to the target position, acquiring an estimated distance map in the second depth distance map, wherein an area image corresponding to the estimated distance map is the same as an area image corresponding to the first depth distance map;

[0017] The first depth distance map and the estimated distance map are channel-merged to obtain a merged depth distance map, and the merged depth distance map is processed by a preset model to obtain a target distance corresponding to a pixel point in the regional image.

[0018] In a possible implementation, determining a second depth distance map of the visible light image according to the visible light image includes:

[0019] Increasing the pixel size of the visible light image to obtain a first target visible light image, and reducing the pixel size of the visible light image to obtain a second target visible light image;

[0020] Processing the first target visible light image by using a depth estimation model to obtain a first distance map;

[0021] Processing the second target visible light image by using the depth estimation model to obtain a second distance map;

[0022] Perform channel merging processing on the first distance map and the second distance map to obtain a merged distance map;

[0023] The combined distance map is processed by a preset model to obtain a second depth distance map of the visible light image.

[0024] In a possible implementation, acquiring a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image includes:

[0025] Acquire a first integral image density corresponding to the visible light image and a second integral image density corresponding to the first thermal imaging image;

[0026] Acquire a plurality of sub-region images in the visible light image;

[0027] The plurality of sub-region images are processed by using the first integral image density and the second integral image density to obtain the plurality of region images.

[0028] In a possible implementation, for any sub-region image, the plurality of sub-region images are processed by using the first integral image density and the second integral image density to obtain the plurality of region images, including:

[0029] Acquire a third integral image density of the sub-region image;

[0030] When the third integral image density is greater than or equal to the first integral image density, increasing the size of the sub-region image so that the absolute value of the difference between the third integral image density and the first integral image density is less than or equal to the first threshold, thereby obtaining a first sub-region image;

[0031] When the third integral image density is greater than or equal to the second integral image density, increasing the size of the sub-region image until the absolute value of the difference between the third integral image density and the second integral image density is less than or equal to the first threshold, thereby obtaining a second sub-region image;

[0032] The region image is obtained according to the first sub-region image and the second sub-region image.

[0033] In a possible implementation manner, obtaining the region image according to the first sub-region image and the second sub-region image includes:

[0034] Acquire an intersection area of ​​the first sub-region image and the second sub-region image, and a union area of ​​the first sub-region image and the second sub-region image;

[0035] When the ratio of the intersection area to the union area is greater than or equal to a second threshold, the first sub-area image and the second sub-area image are merged to obtain the area image.

[0036] In a second aspect, the present application provides an image processing device, including a first acquisition module, a second acquisition module, a determination module and an update module, wherein:

[0037] The first acquisition module is used to acquire a visible light image and a first thermal imaging image corresponding to the visible light image;

[0038] The second acquisition module is used to acquire a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein an absolute value of a difference between an integral image density of the regional images and an integral image density of the visible light image is less than or equal to a first threshold;

[0039] The determination module is used to determine, based on the multiple regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image, wherein the target distance is a distance between a photographing object corresponding to the pixel point and a photographing device;

[0040] The updating module is used to update the first thermal imaging image according to the target distance to obtain a second thermal imaging image.

[0041] In a possible implementation manner, the determining module is specifically configured to:

[0042] Determine, according to the multiple regional images, a first depth distance map for each regional image, wherein the first depth distance map is used to indicate distances corresponding to pixels in the regional image;

[0043] Determine, according to the visible light image, a second depth distance map of the visible light image, wherein the second depth distance map is used to indicate distances corresponding to pixels in the visible light image;

[0044] The target distance is determined according to the plurality of first depth distance maps and the second depth distance map.

[0045] In a possible implementation manner, the determining module is specifically configured to:

[0046] Determining a target position of the first depth distance map in the second depth distance map;

[0047] According to the target position, acquiring an estimated distance map in the second depth distance map, wherein an area image corresponding to the estimated distance map is the same as an area image corresponding to the first depth distance map;

[0048] The first depth distance map and the estimated distance map are channel-merged to obtain a merged depth distance map, and the merged depth distance map is processed by a preset model to obtain a target distance corresponding to a pixel point in the regional image.

[0049] In a possible implementation manner, the determining module is specifically configured to:

[0050] Increasing the pixel size of the visible light image to obtain a first target visible light image, and reducing the pixel size of the visible light image to obtain a second target visible light image;

[0051] Processing the first target visible light image by using a depth estimation model to obtain a first distance map;

[0052] Processing the second target visible light image by using the depth estimation model to obtain a second distance map;

[0053] Perform channel merging processing on the first distance map and the second distance map to obtain a merged distance map;

[0054] The combined distance map is processed by a preset model to obtain a second depth distance map of the visible light image.

[0055] In a possible implementation manner, the second acquisition module is specifically configured to:

[0056] Acquire a first integral image density corresponding to the visible light image and a second integral image density corresponding to the first thermal imaging image;

[0057] Acquire a plurality of sub-region images in the visible light image;

[0058] The plurality of sub-region images are processed by using the first integral image density and the second integral image density to obtain the plurality of region images.

[0059] In a possible implementation manner, the second acquisition module is specifically configured to:

[0060] Acquire a third integral image density of the sub-region image;

[0061] When the third integral image density is greater than or equal to the first integral image density, increasing the size of the sub-region image so that the absolute value of the difference between the third integral image density and the first integral image density is less than or equal to the first threshold, thereby obtaining a first sub-region image;

[0062] When the third integral image density is greater than or equal to the second integral image density, increasing the size of the sub-region image until the absolute value of the difference between the third integral image density and the second integral image density is less than or equal to the first threshold, thereby obtaining a second sub-region image;

[0063] The region image is obtained according to the first sub-region image and the second sub-region image.

[0064] In a possible implementation manner, the second acquisition module is specifically configured to:

[0065] Acquire an intersection area of ​​the first sub-region image and the second sub-region image, and a union area of ​​the first sub-region image and the second sub-region image;

[0066] When the ratio of the intersection area to the union area is greater than or equal to a second threshold, the first sub-area image and the second sub-area image are merged to obtain the area image.

[0067] In a third aspect, the present application provides a terminal device, including: a processor and a memory;

[0068] The memory stores computer-executable instructions;

[0069] The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method as described in the first aspect.

[0070] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement any of the aforementioned image processing methods when the computer-executable instructions are executed by a processor.

[0071] The present application provides an image processing method, device and equipment, which obtains a visible light image and a first thermal imaging image corresponding to the visible light image, obtains multiple regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein the absolute value of the difference between the integral density of the regional image and the integral density of the visible light image is less than or equal to a first threshold value, and determines the target distance corresponding to each pixel in the visible light image according to the multiple regional images and the visible light image, wherein the target distance is the distance between the photographed object corresponding to the pixel and the photographing device, and updates the first thermal imaging image according to the target distance to obtain a second thermal imaging image. In the above method, the thermal imaging device can obtain multiple regional images in the visible light image through the visible light image and the first thermal imaging image, and because the integral density of the regional image is similar to the integral density of the visible light image, the regional image can include more image detail information, therefore, according to the regional image and the visible light image, the distance corresponding to each pixel can be accurately determined, and then the thermal imaging image is updated according to the distance, thereby improving the accuracy of the thermal image display. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0073] Figure 2 A flowchart of an image processing method provided in an embodiment of the present application;

[0074] Figure 3A schematic diagram of a process for obtaining an image of a first sub-region provided in an embodiment of the present application;

[0075] Figure 4 A schematic diagram of a process for obtaining a regional image provided in an embodiment of the present application;

[0076] Figure 5 A schematic diagram of a method for determining a target distance provided in an embodiment of the present application;

[0077] Figure 6 A schematic diagram of a preset model provided in an embodiment of the present application;

[0078] Figure 7 A schematic diagram of an image processing method provided in an embodiment of the present application;

[0079] Figure 8 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0080] Fig. 9 A schematic diagram of the hardware structure of the terminal device provided for this application. DETAILED DESCRIPTION

[0081] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0082] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0083] In the related art, a dual-photothermal camera can generate a thermal image based on the infrared light waves emitted by an object. However, the infrared light waves will gradually attenuate during long-distance transmission. Therefore, it is necessary to accurately determine the distance between the photographed object and the shooting device, and then obtain the infrared light waves actually emitted by the photographed object to determine the temperature of the object. At present, the distance between the photographed object and the shooting device can be obtained by manual calibration. For example, when a dual-photothermal camera generates a thermal image, the user can first divide the thermal image into multiple different display areas, and then measure the distance between each display area and the dual-photothermal camera, and then correct the thermal image of each display area. However, the manual calibration method is more complicated, and the distance between different display objects in the display area and the dual-tube thermal camera may also be different, which leads to low display accuracy of the thermal image.

[0084] In order to solve the technical problem of low accuracy of the thermal image displayed in the related art, the embodiment of the present application provides an image processing method, which obtains a visible light image and a first thermal imaging image corresponding to the visible light image, obtains multiple regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein the absolute value of the difference between the integral image density of the regional image and the integral image density of the visible light image is less than or equal to a first threshold value, determines a first depth distance map of each regional image according to the multiple regional images, determines a second depth distance map of the visible light image according to the visible light image, determines a target distance according to the multiple first depth distance maps and the second depth distance maps, and updates the first thermal imaging image according to the target distance to obtain a second thermal imaging image. In this way, the thermal imaging device can accurately determine the target distance corresponding to each pixel in the visible light image through the first depth distance map corresponding to the visible light image and the multiple second depth distance maps corresponding to the multiple regional images in the visible light image, and updates the thermal imaging image according to the target distance, thereby improving the accuracy of the displayed thermal imaging image.

[0085] Next, combine Figure 1 , the application scenarios of the embodiments of the present application are described.

[0086] Figure 1 This is a schematic diagram of an application scenario provided by the embodiment of this application. Figure 1, including a thermal imaging device and a photographed object. The thermal imaging device photographs the photographed object to obtain a visible light image and a first thermal imaging image. The visible light image and the first thermal imaging image both include the photographed object. The thermal imaging device obtains a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image, and updates the first thermal imaging image according to the plurality of regional images and the visible light image to obtain a second thermal imaging image. In this way, since the integral density of the regional image is similar to that of the visible light image, the regional image can include more image detail information. Therefore, according to the regional image and the visible light image, the distance corresponding to each pixel point can be accurately determined, and then the thermal imaging image is updated according to the distance to improve the accuracy of the thermal image.

[0087] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0088] Figure 2 This is a flowchart of an image processing method provided in an embodiment of the present application. Figure 2 The method flow includes:

[0089] S201 : Acquire a visible light image and a first thermal imaging image corresponding to the visible light image.

[0090] The execution subject of the present application may be a thermal imaging device, or an image processing device arranged in the thermal imaging device, wherein the image processing device may be implemented by software, or by a combination of software and hardware.

[0091] The thermal imaging device may be any device having a thermal imaging function. For example, the thermal imaging device may be an infrared thermal imaging camera, a dual-light thermal imaging device, etc. A visible light image is an image including colors visible to the human eye. For example, a real image of an object captured by a camera is a visible light image. The first thermal imaging image is a thermal imaging image of a visible light image. For example, the captured object of the visible light image is the same as the captured object of the thermal imaging image, and the thermal imaging image is used to indicate the temperature of the captured object in the visible light image.

[0092] Optionally, the visible light image and the first thermal imaging image corresponding to the visible light image may be acquired by a thermal imaging device. For example, when a dual-light thermal imaging device captures a visible light image, it may acquire a thermal imaging image corresponding to the visible light image.

[0093] S202 . Acquire multiple regional images in the visible light image according to the visible light image and the first thermal imaging image.

[0094] The absolute value of the difference between the integral image density of the regional image and the integral image density of the visible light image is less than or equal to the first threshold value. The integral image is a matrix representation method that can describe the global information of the image, and the integral image density is the ratio of the sum of the pixel values ​​of the integral image to the number of pixels of the integral image. For example, if the sum of the pixel values ​​of the integral image corresponding to the visible light image is 100 and the number of pixels of the integral image is 10, then the integral image density of the visible light image is 10.

[0095] According to the following feasible implementation, multiple regional images can be obtained in the visible light image: a first integral image density corresponding to the visible light image and a second integral image density corresponding to the first thermal imaging image are obtained. Multiple sub-region images are obtained in the visible light image. The multiple sub-region images are processed by the first integral image density and the second integral image density to obtain multiple regional images.

[0096] Optionally, the first integral image density is the density of the integral image corresponding to the visible light image, and the second integral image density is the density of the integral image corresponding to the first thermal imaging image. For example, the first integral image density is the ratio of the sum of pixel values ​​of the integral image of the visible light image to the number of pixels, and the second integral image density is the ratio of the sum of pixel values ​​of the integral image of the first thermal imaging image to the number of pixels.

[0097] Optionally, the first integral image density may be obtained according to the following feasible implementation methods: obtaining an edge image corresponding to the visible light image. For example, the edge image corresponding to the visible light image may be obtained by a canny edge detection algorithm. Determine the integral image of the edge image of the visible light image. For example, the edge image of the visible light image may be processed by an integral image calculation method to obtain an integral image corresponding to the visible light image.

[0098] The first integral image density is obtained according to the integral image of the visible light image. For example, when the integral image of the visible light image is obtained, the pixel value of each pixel in the integral image can be obtained, and then the first integral image density is determined according to the sum of the pixel values ​​and the number of pixel values. For example, if the integral image of the visible light image includes 100 pixels and the sum of the pixel values ​​is 800, the first integral image density is 8.

[0099] Optionally, the second integral image density can be obtained according to the following feasible implementation method: obtain a pseudo-color image corresponding to the first thermal imaging image. Obtain an edge image corresponding to the pseudo-color image. For example, the pseudo-color image can be processed by a canny edge detection algorithm to obtain an edge image corresponding to the pseudo-color image. Determine the integral image of the edge image of the pseudo-color image and determine it as the integral image corresponding to the first thermal imaging image. For example, the edge image of the pseudo-color image can be processed by an integral image calculation method to obtain an integral image corresponding to the first thermal imaging image.

[0100] The density of the second integral image is obtained according to the integral image of the first thermal imaging image. For example, when the integral image of the first thermal imaging image is obtained, the pixel value of each pixel in the integral image can be obtained, and then the density of the second integral image is determined according to the sum of the pixel values ​​and the number of pixel values. For example, if the integral image of the first thermal imaging image includes 50 pixels and the sum of the pixel values ​​is 500, the density of the second integral image is 10.

[0101] Optionally, the sub-region image is a partial region image in the visible light image. Optionally, multiple sub-region images can be obtained according to the following feasible implementation method: multiple sub-region images are obtained in the visible light image by a sliding window method. The size of the sliding window can be a preset size, and the step length of the sliding window can be a preset step length. For example, the size of the sliding window can be 190px (190 pixels), and the sliding step length is 190 pixels. 0.8, through the parameters of the above sliding window, multiple sub-region images can be obtained in the visible light image. For example, a sub-region image (100 pixels) with a length and width of 10 pixels is intercepted in the visible light image, and then moved to the right by 8 pixels, and continued to intercept a sub-region image with a length and width of 10 pixels, and the above steps are repeated until every part of the visible light image is intercepted, thereby obtaining multiple sub-region images corresponding to the visible light image.

[0102] The plurality of sub-region images are processed by the first integral image density and the second integral image density to obtain the plurality of region images, specifically: for any sub-region image, the third integral image density of the sub-region image is obtained. For example, the edge image corresponding to the sub-region image is obtained, and the integral image corresponding to the edge image is calculated, and then the third integral image density of the sub-region image can be obtained according to the integral image. It should be noted that obtaining the integral image and the integral image density is a solution in the prior art, and the embodiments of the present application will not be described in detail.

[0103] When the third integral image density is greater than or equal to the first integral image density, the size of the sub-region image is increased so that the absolute value of the difference between the third integral image density and the first integral image density and the first integral image density is less than or equal to the first threshold value, thereby obtaining the first sub-region image. For example, when the third integral image density corresponding to the sub-region image is greater than the first integral image density of the visible light image, the range of the sub-region image is increased according to a preset pixel size, and if the third integral image density of the sub-region image after the size increase is less than or equal to the first integral image density (that is, the absolute value of the difference between the third integral image density and the first integral image density is less than or equal to the first threshold value), the sub-region image after the size increase is determined as the first sub-region image, and if the third integral image density of the sub-region image after the size increase is still greater than the first integral image density, the size of the sub-region image is continued to be increased according to the preset pixel size until the absolute value of the difference between the third integral image density of the sub-region image and the first integral image density is less than or equal to the first threshold value, thereby obtaining the first sub-region image.

[0104] Next, combine Figure 3 , the process of obtaining the first sub-region image is explained.

[0105] Figure 3 A schematic diagram of a process for obtaining a first sub-region image provided in an embodiment of the present application. Figure 3 , including: a sub-region image. The sub-region image includes part of the photographed object. By calculation, the third integral image density of the sub-region image is greater than the first integral image density, so the sub-region image is enlarged to obtain the first sub-region image. The first sub-region image includes more photographed objects, and the absolute value of the difference between the integral image density corresponding to the first sub-region image and the first integral image density of the visible light image is less than a first threshold. In this way, the first sub-region image can include more information about the photographed object, thereby improving the accuracy of thermal imaging.

[0106] When the third integral image density is greater than or equal to the second integral image density, the size of the sub-region image is increased until the absolute value of the difference between the third integral image density and the second integral image density is less than or equal to the first threshold, thereby obtaining the second sub-region image. For example, when the third integral image density corresponding to the sub-region image is greater than the second integral image density of the first thermal imaging image, the range of the sub-region image is increased according to a preset pixel size, and if the third integral image density of the sub-region image after the size increase is less than or equal to the second integral image density (that is, the absolute value of the difference between the third integral image density and the second integral image density is less than or equal to the first threshold), the sub-region image after the size increase is determined as the second sub-region image, and if the third integral image density of the sub-region image after the size increase is still greater than the second integral image density, the size of the sub-region image is continued to be increased according to the preset pixel size until the absolute value of the difference between the third integral image density and the second integral image density of the sub-region image is less than or equal to the first threshold, thereby obtaining the second sub-region image.

[0107] According to the first sub-region image and the second sub-region image, a region image is obtained. Optionally, the region image can be obtained according to the following feasible implementation method: the intersection region of the first sub-region image and the second sub-region image, and the union region of the first sub-region image and the second sub-region image are obtained. The intersection region is the region where the first sub-region image and the second sub-region image overlap. The union region is all regions included in the first sub-region image and the second sub-region image.

[0108] When the ratio of the intersection area to the union area is greater than or equal to the second threshold, the first sub-region image and the second sub-region image are merged to obtain a regional image. For example, if the ratio of the intersection area of ​​the first sub-region image and the second sub-region image to the union area is greater than 0.7, the first sub-region image and the second sub-region image are merged to obtain a regional image (the union of the first sub-region image and the second sub-region image).

[0109] Optionally, if there are multiple first sub-region images and multiple second sub-region images in the visible light image, it is necessary to obtain the ratio of the intersection area of ​​each first sub-region image and each second sub-region image to the union area to obtain the region image. For example, if the visible light image includes a first sub-region image A, a first sub-region image B, a second sub-region image A, and a second sub-region image B, it is necessary to determine the ratio of the intersection area of ​​the first sub-region image A and the second sub-region image A to the union area, the ratio of the intersection area of ​​the first sub-region image A and the second sub-region image B to the union area, the ratio of the intersection area of ​​the first sub-region image B and the second sub-region image A to the union area, and the ratio of the intersection area of ​​the first sub-region image B and the second sub-region image B to the union area, and then determine the region image according to the above ratios.

[0110] Next, combine Figure 4 , the process of obtaining the regional image is explained.

[0111] Figure 4 A schematic diagram of a process for obtaining a regional image provided in an embodiment of the present application. Figure 4 , including: a first sub-region image and a second sub-region image. The area where the first sub-region image and the second sub-region image intersect is the intersection area, and all areas included in the first sub-region image and the second sub-region image are the union area. When the ratio of the intersection area to the union area is greater than a second threshold, the first sub-region and the second sub-region are merged to obtain a region image. The region image includes all the contents of the first sub-region image and the second sub-region image.

[0112] S203 . Determine, according to the multiple regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image.

[0113] The target distance is the distance between the photographed object and the photographing device corresponding to the pixel. For example, 100 pixels in the visible light image constitute the photographed object, and the target distance of the 100 pixels is the distance between the photographed object and the photographing device.

[0114] The target distance corresponding to each pixel point in the visible light image can be determined according to the following feasible implementation method: a first depth distance map of each regional image is determined according to multiple regional images. The first depth distance map is used to indicate the distance corresponding to the pixel point in the regional image. For example, the pixel value in the first depth distance map is the distance from the shooting device to the object corresponding to each pixel in the regional image.

[0115] Optionally, the first depth distance map of each regional image can be obtained by a preset algorithm. For example, the first depth distance map corresponding to the regional image can be obtained by a network structure such as midasnet, srlnet, leres, etc. For example, the regional image is input into the midasnet algorithm, and midasnet can output the first depth distance map corresponding to the regional image. Since the regional image includes rich detailed information of the visible light image, the first depth distance map can accurately determine the distance corresponding to each pixel.

[0116] According to the visible light image, a second depth distance map of the visible light image is determined. The second depth distance map is used to indicate the distances corresponding to the pixels in the visible light image. For example, the pixel values ​​in the second depth distance map are the distances from the camera to each point in the scene. Optionally, the visible light image can be processed by a preset model to obtain a second depth distance map corresponding to the visible light image.

[0117] The target distance is determined according to the plurality of first depth distance maps and the second depth distance map. For example, the target distance corresponding to each pixel in the visible light image can be determined by model training.

[0118] S204: Update the first thermal imaging image according to the target distance to obtain a second thermal imaging image.

[0119] Optionally, the thermal imaging device may update the first thermal imaging image according to the target distance to obtain a second thermal imaging image. For example, after the thermal imaging device obtains the target distance corresponding to each pixel of the visible light image, it may compensate and correct the first thermal imaging image according to the target distance to obtain a second thermal imaging image.

[0120] Optionally, the second thermal imaging image may be obtained according to the following feasible implementation method: obtaining a first preset relationship. The first preset relationship includes at least one distance and a temperature compensation value corresponding to each distance. For example, the first preset relationship may be as shown in Table 1:

[0121] Table 1

[0122]

[0123] It should be noted that Table 1 is only an example to illustrate the first preset relationship, and is not a limitation to the first preset relationship.

[0124] The first thermal imaging image is updated according to the target distance and the first preset relationship. For example, if the target distance corresponding to the pixel is distance 1, the temperature compensation value corresponding to the target distance is temperature 1, and the first thermal imaging image is updated by the temperature of the pixel in the first thermal imaging image and temperature 1 to obtain the actual temperature of the pixel; if the target distance corresponding to the pixel is distance 2, the temperature compensation value corresponding to the target distance is temperature 2, and the first thermal imaging image is updated by the temperature of the pixel in the first thermal imaging image and temperature 1 to obtain the actual temperature of the pixel.

[0125] The embodiment of the present application provides an image processing method, which obtains a visible light image and a first thermal imaging image corresponding to the visible light image, obtains multiple regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein the absolute value of the difference between the integral image density of the regional image and the integral image density of the visible light image is less than or equal to a first threshold value, determines a first depth distance map of each regional image according to the multiple regional images, determines a second depth distance map of the visible light image according to the visible light image, determines a target distance according to the multiple first depth distance maps and the second depth distance maps, and updates the first thermal imaging image according to the target distance to obtain a second thermal imaging image. In this way, the thermal imaging device can accurately determine the target distance corresponding to each pixel in the visible light image through the second depth distance map corresponding to the visible light image and the multiple first depth distance maps corresponding to the multiple regional images in the visible light image, and updates the thermal imaging image according to the target distance, thereby improving the accuracy of the displayed thermal imaging image.

[0126] exist Figure 2 Based on the embodiment shown below, combined with Figure 5 , the method for determining the target distance corresponding to each pixel point in the visible light image is explained.

[0127] Figure 5 A flow chart of a method for determining target distance provided in an embodiment of the present application. Figure 5 The method flow includes:

[0128] S501 : Determine a first depth distance map of each regional image according to a plurality of regional images.

[0129] Optionally, the first depth distance map is used to indicate the distance corresponding to the pixel point in the regional image. For example, when the thermal imaging device obtains multiple regional images corresponding to the visible light image, the thermal imaging device can determine the first depth distance map corresponding to the regional image through a network such as midasnet, srlnet, leres, etc.

[0130] S502: Determine a second depth distance map of the visible light image according to the visible light image.

[0131] Optionally, the second depth distance map is used to indicate the distance corresponding to the pixel point in the visible light image. Optionally, the second depth distance map of the visible light image can be determined according to the following feasible implementation method: increasing the pixel size of the visible light image to obtain the first target visible light image, and reducing the pixel size of the visible light image to obtain the second target visible light image. For example, increasing the pixel resolution of the visible light image to obtain the first target visible light image, the first target visible light image is a high-resolution visible light image, and reducing the pixel resolution of the visible light image to obtain the second target visible light image, the second target visible light image is a low-resolution visible light image.

[0132] The first target visible light image is processed by the depth estimation model to obtain a first distance map, and the second target visible light image is processed by the depth estimation model to obtain a second distance map. For example, the depth estimation model can be a network model such as midasnet, srlnet, leres, etc. The high-resolution visible light image is processed by the depth estimation model to obtain the first distance map, and the low-resolution visible light image is processed by the depth estimation model to obtain the second distance map.

[0133] The first distance map and the second distance map are subjected to channel merging processing to obtain a merged distance map. For example, the first distance map and the second distance map may be merged according to the RGB channels of the first distance map and the RGB channels of the second distance map to obtain a merged distance map. In this way, since the first distance map is a high-precision distance map and the second distance map is a low-precision distance map, the merged distance map can accurately reflect the high-precision information and the low-precision information of the visible light image.

[0134] The merged distance map is processed by a preset model to obtain a second depth distance map of the visible light image. The preset model is learned from multiple groups of samples. The multiple groups of samples include multiple sample merged distance maps and depth distance maps corresponding to the sample merged distance maps. For example, a group of samples includes a sample merged distance map. Figure 1 and sample depth distance Figure 1 , using this method, multiple groups of samples can be obtained.

[0135] Optionally, the preset model includes a first convolution layer, a second convolution layer, a third convolution layer and a fourth convolution layer, wherein the first convolution layer is used to split image features, the second convolution layer is used to merge image features, the third convolution layer is used to split image features, and the fourth convolution layer is used to merge image features.

[0136] Next, combine Figure 6 , explaining the structure of the preset model.

[0137] Figure 6 A schematic diagram of a preset model provided in an embodiment of the present application. Figure 6 , including: a preset model. The preset model includes a first module, a second module, a third module and a fourth module. When the first convolution layer receives an image, it determines that the size of the convolution kernel is 7 7, the step size is 2, and it is normalized. The activation function is the relu function. The image features of the input image are split into 1 / 4 through block1 (for feature extraction), the image features of the input image are split into 1 / 8 through block2, the image features of the input image are split into 1 / 16 through block3, and the image features of the input image are split into 1 / 32 through block4, and the image features split into 1 / 32 are sent to the second module.

[0138] See also Figure 6 , the second module receives the 1 / 32 image features through DeBlock1, merges them into 1 / 16 image features through DeBlock2, and then merges them into 1 / 8 image features through DeBlock3. The second module obtains the input image and reduces the size of the input image to 1 / 8 to obtain a reduced image. The third module merges the reduced image and the 1 / 8 image features, splits the merged image into 1 / 16 image features through EnBlock1, splits the 1 / 16 image features into 1 / 32 image features through EnBlock2, and sends the merged image split into 1 / 32 image features to the fourth module.

[0139] See also Figure 6 The fourth module receives 1 / 32 of the image features through DeBlock4, and merges them into 1 / 16 of the image features through DeBlock5. DeBlock6 merges 1 / 16 of the image features into 1 / 8 of the image features, and DeBlock7 merges 1 / 8 of the image features into 1 / 4 of the image features. The fourth model uses a convolution kernel of size 4 4. A convolutional layer with a step size of 2 and an activation function of relu processes 1 / 4 of the image features and normalizes them to obtain a distance map.

[0140] S503: Determine the target distance according to the multiple first depth distance maps and the second depth distance map.

[0141] Optionally, for any first depth distance map, the target distance of the pixel point of the second depth distance map can be determined according to the following feasible implementation method: determine the target position of the first depth distance map in the second depth distance map. For example, obtain the bounding-box of the first depth distance map, and then determine the target position through the bounding-box.

[0142] According to the target position, an estimated distance map is obtained in the second depth distance map. The regional image corresponding to the estimated distance map is the same as the regional image corresponding to the first depth distance map. For example, the position of the regional image corresponding to the first depth distance map in the second depth distance map can be accurately determined by the target distance.

[0143] The first depth distance map and the estimated distance map are channel-merged to obtain a merged depth distance map. For example, the first depth distance map and the estimated distance map may be merged according to the RGB channels of the first depth distance map and the RGB channels of the estimated distance map to obtain a merged depth distance map.

[0144] The combined depth distance map is processed by a preset model to obtain the target distance corresponding to the pixel points in the regional image. For example, the combined depth distance map is input into the preset model, and the preset model can output a high-precision distance map corresponding to the combined depth distance map, and then the target distance corresponding to each pixel point in the visible light image is obtained through the high-precision distance map.

[0145] The embodiment of the present application provides a method for determining the target distance, which determines a first depth distance map of each regional image based on multiple regional images, determines a second depth distance map of the visible light image based on the visible light image, and determines the target distance based on the multiple first depth distance maps and the second depth distance map. In this way, since the regional image includes more image information of the visible light image, the high-precision distance map of the visible light image can be accurately determined based on the first depth distance map of the regional image and the second depth distance map of the visible light image, and then the target distance of each pixel point of the visible light image is accurately obtained through the high-precision distance map, thereby improving the accuracy of the displayed thermal imaging image.

[0146] Based on any of the above embodiments, Figure 7 , the process of the above-mentioned image processing method is explained.

[0147] Figure 7 A schematic diagram of an image processing method provided in an embodiment of the present application. Figure 7 , including: a thermal imaging device and a photographed object. The thermal imaging device photographs the photographed object to obtain a visible light image and a first thermal imaging image. The visible light image and the first thermal imaging image both include the photographed object. The thermal imaging device is based on the visible light image and the first thermal imaging image. Based on the visible light image and the first thermal imaging image, multiple regional images can be determined in the visible light image.

[0148] See also Figure 7, obtain a first depth image A, a first depth image B, a first depth image C corresponding to the multiple regional images, and a second depth image corresponding to the visible light image. According to the first depth image A, the first depth image B, the first depth image C and the second depth image, obtain a second thermal imaging image. In this way, the thermal imaging device can accurately determine the target distance corresponding to each pixel in the visible light image through the second depth distance map corresponding to the visible light image and the multiple first depth distance maps corresponding to the multiple regional images in the visible light image, and update the thermal imaging image through the target distance, thereby improving the accuracy of the displayed thermal imaging image.

[0149] Figure 8 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Figure 8 The image processing device 10 includes a first acquisition module 11, a second acquisition module 12, a determination module 13 and an update module 14, wherein:

[0150] The first acquisition module 11 is used to acquire a visible light image and a first thermal imaging image corresponding to the visible light image;

[0151] The second acquisition module 12 is used to acquire a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein an absolute value of a difference between an integral image density of the regional images and an integral image density of the visible light image is less than or equal to a first threshold;

[0152] The determination module 13 is used to determine, based on the multiple regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image, wherein the target distance is a distance between a photographing object corresponding to the pixel point and a photographing device;

[0153] The updating module 14 is used to update the first thermal imaging image according to the target distance to obtain a second thermal imaging image.

[0154] In a possible implementation manner, the determining module 13 is specifically configured to:

[0155] Determine, according to the multiple regional images, a first depth distance map for each regional image, wherein the first depth distance map is used to indicate distances corresponding to pixels in the regional image;

[0156] Determine, according to the visible light image, a second depth distance map of the visible light image, wherein the second depth distance map is used to indicate distances corresponding to pixels in the visible light image;

[0157] The target distance is determined according to the plurality of first depth distance maps and the second depth distance map.

[0158] In a possible implementation manner, the determining module 13 is specifically configured to:

[0159] Determining a target position of the first depth distance map in the second depth distance map;

[0160] According to the target position, acquiring an estimated distance map in the second depth distance map, wherein an area image corresponding to the estimated distance map is the same as an area image corresponding to the first depth distance map;

[0161] The first depth distance map and the estimated distance map are channel-merged to obtain a merged depth distance map, and the merged depth distance map is processed by a preset model to obtain a target distance corresponding to a pixel point in the regional image.

[0162] In a possible implementation manner, the determining module 13 is specifically configured to:

[0163] Increasing the pixel size of the visible light image to obtain a first target visible light image, and reducing the pixel size of the visible light image to obtain a second target visible light image;

[0164] Processing the first target visible light image by using a depth estimation model to obtain a first distance map;

[0165] Processing the second target visible light image by using the depth estimation model to obtain a second distance map;

[0166] Perform channel merging processing on the first distance map and the second distance map to obtain a merged distance map;

[0167] The combined distance map is processed by a preset model to obtain a second depth distance map of the visible light image.

[0168] In a possible implementation manner, the second acquisition module 12 is specifically configured to:

[0169] Acquire a first integral image density corresponding to the visible light image and a second integral image density corresponding to the first thermal imaging image;

[0170] Acquire a plurality of sub-region images in the visible light image;

[0171] The plurality of sub-region images are processed by using the first integral image density and the second integral image density to obtain the plurality of region images.

[0172] In a possible implementation manner, the second acquisition module 12 is specifically configured to:

[0173] Acquire a third integral image density of the sub-region image;

[0174] When the third integral image density is greater than or equal to the first integral image density, increasing the size of the sub-region image so that the absolute value of the difference between the third integral image density and the first integral image density is less than or equal to the first threshold, thereby obtaining a first sub-region image;

[0175] When the third integral image density is greater than or equal to the second integral image density, increasing the size of the sub-region image until the absolute value of the difference between the third integral image density and the second integral image density is less than or equal to the first threshold, thereby obtaining a second sub-region image;

[0176] The region image is obtained according to the first sub-region image and the second sub-region image.

[0177] In a possible implementation manner, the second acquisition module 12 is specifically configured to:

[0178] Acquire an intersection area of ​​the first sub-region image and the second sub-region image, and a union area of ​​the first sub-region image and the second sub-region image;

[0179] When the ratio of the intersection area to the union area is greater than or equal to a second threshold, the first sub-area image and the second sub-area image are merged to obtain the area image.

[0180] The image processing device provided in the embodiment of the present application can execute the technical solution shown in the above method embodiment, and its implementation principle and beneficial effects are similar, which will not be repeated here.

[0181] Fig. 9 The hardware structure diagram of the terminal device provided for this application. Fig. 9 The terminal device 20 may include: a processor 21 and a memory 22, wherein the processor 21 and the memory 22 can communicate; illustratively, the processor 21 and the memory 22 communicate via a communication bus 23, the memory 22 is used to store program instructions, and the processor 21 is used to call the program instructions in the memory to execute the image processing method shown in any of the above method embodiments.

[0182] Optionally, the terminal device 20 may further include a communication interface, which may include a transmitter and / or a receiver.

[0183] Optionally, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. The steps of the method disclosed in the present application may be directly implemented as being executed by a hardware processor, or may be implemented by a combination of hardware and software modules in the processor.

[0184] An embodiment of the present application provides a readable storage medium, on which a computer program is stored; the computer program is used to implement the image processing method described in any of the above embodiments.

[0185] An embodiment of the present application provides a computer program product, which includes instructions. When the instructions are executed, a computer executes the above-mentioned image processing method.

[0186] All or part of the steps of the above-mentioned method embodiments can be completed by hardware related to program instructions. The above-mentioned program can be stored in a readable memory. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned memory (storage medium) includes: read-only memory (English: read-only memory, abbreviated: ROM), RAM, flash memory, hard disk, solid state drive, magnetic tape (English: magnetic tape), floppy disk (English: floppy disk), optical disc (English: optical disc) and any combination thereof.

[0187] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable terminal device to generate a machine, so that the instructions executed by the processing unit of the computer or other programmable terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0188] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0189] These computer program instructions can also be loaded onto a computer or other programmable terminal device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0190] Obviously, those skilled in the art can make various changes and modifications to the embodiments of the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

[0191] In the present application, the term "include" and its variations may refer to non-restrictive inclusion; the term "or" and its variations may refer to "and / or". The terms "first", "second", etc. in the present application are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. In the present application, "plurality" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previously associated objects are in an "or" relationship.

Claims

1. An image processing method, characterized in that: include: Acquire a visible light image and a first thermal imaging image corresponding to the visible light image; According to the visible light image and the first thermal imaging image, a plurality of regional images are acquired in the visible light image, wherein an absolute value of a difference between an integral image density of the regional image and an integral image density of the visible light image is less than or equal to a first threshold; the integral image density is a ratio of a sum of pixel values ​​of the integral image to the number of pixels of the integral image; according to the plurality of regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image is determined, wherein the target distance is a distance between a photographing object corresponding to the pixel point and a photographing device; updating the first thermal imaging image according to the target distance to obtain a second thermal imaging image; Determining a target distance corresponding to each pixel point in the visible light image according to the multiple regional images and the visible light image includes: Determine, according to the multiple regional images, a first depth distance map for each regional image, wherein the first depth distance map is used to indicate distances corresponding to pixels in the regional image; Determine, according to the visible light image, a second depth distance map of the visible light image, wherein the second depth distance map is used to indicate distances corresponding to pixels in the visible light image; The target distance is determined according to the plurality of first depth distance maps and the second depth distance map.

2. The method according to claim 1, characterized in that For any one of the first depth distance maps; determining the target distance according to the plurality of first depth distance maps and the second depth distance map, comprises: Determining a target position of the first depth distance map in the second depth distance map; According to the target position, acquiring an estimated distance map in the second depth distance map, wherein an area image corresponding to the estimated distance map is the same as an area image corresponding to the first depth distance map; The first depth distance map and the estimated distance map are subjected to channel merging processing to obtain a merged depth distance map, and the merged depth distance map is processed by a preset model to obtain a target distance corresponding to a pixel point in the regional image.

3. The method according to claim 1 or 2, characterized in that: Determining a second depth distance map of the visible light image according to the visible light image includes: Increasing the pixel size of the visible light image to obtain a first target visible light image, and reducing the pixel size of the visible light image to obtain a second target visible light image; Processing the first target visible light image by using a depth estimation model to obtain a first distance map; Processing the second target visible light image by using the depth estimation model to obtain a second distance map; Perform channel merging processing on the first distance map and the second distance map to obtain a merged distance map; The combined distance map is processed by a preset model to obtain a second depth distance map of the visible light image.

4. The method according to claim 1 or 2, characterized in that: Acquiring a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image, including: Acquire a first integral image density corresponding to the visible light image and a second integral image density corresponding to the first thermal imaging image; Acquire a plurality of sub-region images in the visible light image; The plurality of sub-region images are processed by using the first integral image density and the second integral image density to obtain the plurality of region images.

5. The method according to claim 4, characterized in that For any sub-region image; the plurality of sub-region images are processed by the first integral image density and the second integral image density to obtain the plurality of region images, including: Acquire a third integral image density of the sub-region image; When the third integral image density is greater than or equal to the first integral image density, increasing the size of the sub-region image so that the absolute value of the difference between the third integral image density and the first integral image density is less than or equal to the first threshold, thereby obtaining a first sub-region image; When the third integral image density is greater than or equal to the second integral image density, increasing the size of the sub-region image until the absolute value of the difference between the third integral image density and the second integral image density is less than or equal to the first threshold, thereby obtaining a second sub-region image; The region image is obtained according to the first sub-region image and the second sub-region image.

6. The method according to claim 5, characterized in that Obtaining the region image according to the first sub-region image and the second sub-region image includes: Acquire an intersection area of ​​the first sub-region image and the second sub-region image, and a union area of ​​the first sub-region image and the second sub-region image; When the ratio of the intersection area to the union area is greater than or equal to a second threshold, the first sub-area image and the second sub-area image are merged to obtain the area image.

7. An image processing device, characterized in that: It includes a first acquisition module, a second acquisition module, a determination module and an update module, wherein: The first acquisition module is used to acquire a visible light image and a first thermal imaging image corresponding to the visible light image; The second acquisition module is used to acquire a plurality of regional images in the visible light image according to the visible light image and the first thermal imaging image, wherein an absolute value of a difference between an integral image density of the regional image and an integral image density of the visible light image is less than or equal to a first threshold; and the integral image density is a ratio of a sum of pixel values ​​of the integral image to the number of pixels of the integral image; The determination module is used to determine, based on the multiple regional images and the visible light image, a target distance corresponding to each pixel point in the visible light image, wherein the target distance is a distance between a photographing object corresponding to the pixel point and a photographing device; The updating module is used to update the first thermal imaging image according to the target distance to obtain a second thermal imaging image; The determination module is specifically configured to determine, based on the multiple regional images, a first depth distance map for each regional image, wherein the first depth distance map is used to indicate distances corresponding to pixels in the regional image; Determine, according to the visible light image, a second depth distance map of the visible light image, wherein the second depth distance map is used to indicate distances corresponding to pixels in the visible light image; The target distance is determined according to the plurality of first depth distance maps and the second depth distance map.

8. A terminal device, characterized in that: include: Processor, memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the image processing method according to any one of claims 1 to 6.

Images