Image processing device, learning device, edge device, image generating method, learning method of learning device, reasoning method, image generating program, machine learning program, and reasoning program

Abstract

To provide an image processing device capable of executing the image processing so that each dynamic range of multiple objects on a screen becomes larger for infrared images.SOLUTION: An image processing device 100 comprises a gradation conversion image generating unit 111 to perform the gradation conversion for the infrared image capturing two or more multiple objects using temperature information for each object to generate the gradation conversion image, and a temperature information storage processing unit 112 to assign the generated multiple gradation conversion images as a three-channel image to generate an extended image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, a learning apparatus, an edge device, an image generation method, a learning method of a learning apparatus, an inference method, an image generation program, a machine learning program, and an inference program.

Background Art

[0002] Conventionally, devices for monitoring an object using an infrared image generated using an infrared camera have been studied.

[0003] For example, as an invention using a temperature image related to an infrared image, Patent Document 1 discloses an image processing apparatus. Patent Document 1 discloses an "image processing apparatus including a conversion unit that performs gradation conversion on a temperature image using a conversion function based on data related to a target object" (see the abstract of Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] By the way, for example, thermal sensors often have a high bit depth, and it is necessary to perform gradation conversion in order to reduce the processing cost and data volume. In this case, the thermal sensor, for example, converts infrared rays emitted from an object into an electrical signal, assigns a luminance signal from 0 to 255, and creates an image.

[0006] When the object to be detected is a specific object (e.g., a person, an animal, a vehicle, ···) rather than an object of the entire image, the tone conversion is optimized only for a specific temperature range according to the object. Therefore, when there are multiple objects shown on the screen and the temperature ranges of these objects are extremely different, the information other than the object of interest is rounded off and data is lost.

[0007] Regarding such problems, in the present invention, it is an object to perform image processing on an infrared image so that the dynamic range of each of a plurality of objects shown on the screen becomes larger.

Means for Solving the Problems

[0008] That is, the above problems of the present invention are solved by the following configuration. (1) A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, and generates a tone conversion image; A temperature information storage processing unit that assigns the generated plurality of tone conversion images as three-channel images and generates an extended image; An image processing apparatus comprising: (2) The image processing apparatus according to (1), further comprising an image integration unit that generates an integrated image based on the generated extended image. (1) The image processing apparatus according to (1). (3) The tone conversion image generation unit generates the tone conversion image based on temperature information preset by a user. (1) The image processing apparatus according to (1). (4) The tone conversion image generation unit generates the tone conversion image that emphasizes an artificial temperature range based on temperature information that can occur in nature. (1) The image processing apparatus according to (1). (5) The tone conversion image generation unit continues to monitor the object, acquires a daily temperature range or an abnormal temperature range, and dynamically determines the temperature information based on the acquired temperature range. The image processing apparatus according to (1). (6) Further comprising a learning engineering unit that performs machine learning on the generated extended image as learning data for creating a machine learning model The image processing apparatus according to (1). (7) The image integration unit Performs predetermined weighting on each component with respect to the generated extended image The image processing apparatus according to (2). (8) The image integration unit Displays the integrated image on a display unit and monitors changes over time The image processing apparatus according to (2). (9) A tone conversion image generation unit that performs tone conversion on infrared images in which two or more objects are imaged, using the temperature information for each object to generate a tone conversion image, And a temperature information storage processing unit that assigns the generated plurality of tone conversion images as three-channel images to generate an extended image, and a learning engineering unit that performs machine learning on the generated extended image as learning data for creating a machine learning model Learning device (10) A tone conversion image generation unit that performs tone conversion on infrared images in which two or more objects are imaged, using the temperature information for each object to generate a tone conversion image, And a temperature information storage processing unit that assigns the generated plurality of tone conversion images as three-channel images to generate an extended image, And an inference engineering unit that applies the generated extended image as learning data for creating a machine learning model to the inference model learned by the image processing apparatus including a learning engineering unit that performs machine learning, And an inference engineering unit that applies the tone conversion image generated by the tone conversion image generation unit to the inference model to infer an object Edge device comprising (11) A step of performing tone conversion on infrared images in which two or more objects are imaged, using the temperature information for each object to generate a tone conversion image, Assigning the generated plurality of the tone-converted images as 3-channel images to generate an enlarged image; An image generation method for execution. (12) For an infrared image in which two or more objects are imaged, performing tone conversion using the temperature information for each object to generate a tone-converted image; a tone-converted image generation unit; Assigning the generated plurality of the tone-converted images as 3-channel images to generate an enlarged image; a temperature information storage processing unit; Machine learning the enlarged image generated by the image processing apparatus having the same as learning data for machine learning model creation A learning method of a learning apparatus for execution. (13) For an infrared image in which two or more objects are imaged, performing tone conversion using the temperature information for each object to generate a tone-converted image; a tone-converted image generation unit; Assigning the generated plurality of the tone-converted images as 3-channel images to generate an enlarged image; a temperature information storage processing unit; Using the generated enlarged image as learning data for machine learning model creation; a learning engineering unit for machine learning; For the inference model machine-learned by the image processing apparatus having the same, Applying the tone-converted image generated by the tone-converted image generation unit to infer an object; a step An inference method for execution. (14) For an infrared image in which two or more objects are imaged, performing tone conversion using the temperature information for each object to generate a tone-converted image; a procedure Assigning the generated plurality of the tone-converted images as 3-channel images to generate an enlarged image; a procedure An image generation program for causing a computer to execute the same. (15) For an infrared image in which two or more objects are imaged, performing tone conversion using the temperature information for each object to generate a tone-converted image; a tone-converted image generation unit; A temperature information storage processing unit that assigns the generated plurality of the tone conversion images as three-channel images to generate an extended image, and a procedure of performing machine learning on the extended image generated by the image processing apparatus as learning data for creating a machine learning model. A machine learning program for causing a computer to execute. (16) For an infrared image in which two or more objects are imaged, a tone conversion image generation unit that performs tone conversion using the temperature information for each object to generate a tone conversion image. A temperature information storage processing unit that assigns the generated plurality of the tone conversion images as three-channel images to generate an extended image. A learning engineering unit that performs machine learning using the generated extended image as learning data for creating a machine learning model, and a procedure of applying the tone conversion image generated by the tone conversion image generation unit to the inference model machine-learned by the image processing apparatus provided with the learning engineering unit to infer an object. A procedure of applying the tone conversion image generated by the tone conversion image generation unit to infer an object. An inference program for causing a computer to execute.

Advantages of the Invention

[0009] According to the present invention, it is possible to execute image processing such that the dynamic range of each of a plurality of objects shown on the screen becomes larger for an infrared image.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8A

Figure 8B

Figure 8C

Figure 9

Figure 10

Figure 11A

Figure 11B

Figure 11C

Figure 12A

Figure 12B

Figure 12C

Figure 13

Figure 14

Figure 15A

Figure 15B

Figure 15C

Figure 16A

Figure 16B

Figure 17A

Figure 17B

Figure 18A

Figure 18B

Figure 18C

Embodiments for Carrying Out the Invention

[0011] Hereinafter, embodiments for implementing the present invention will be described in detail. Note that the embodiments described below are examples for realizing the present invention. Therefore, it should be appropriately modified or changed according to the configuration of the apparatus to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments. Also, the same members are denoted by the same reference numerals, and the description will be omitted as appropriate.

[0012] <Description of the prior art> Figs. 16A and 16B are tone-converted images 900 and 910 showing comparative examples. Fig. 16A is a tone-converted image 900 in which the temperature range is set to 35[°C] to 100[°C]. On the other hand, Fig. 16B is a tone-converted image 910 in which the temperature range is set to 31[°C] to 42[°C]. Note that the tone-converted images 900 and 910 are images obtained by tone-converting an infrared image.

[0013] In the tone-converted image 900 shown in Fig. 16A, the set temperature range is wide, and the object 901 is faintly displayed. In this case, in the tone-converted image 900 of Fig. 16A, as a result of adjusting the temperature range so as to preferably display the object 901 in the tone-converted image 900, the visibility of parts other than the object 901 becomes extremely poor, and it is difficult to perform image recognition on parts other than the object 901.

[0014] On the other hand, in the tone-converted image 910 shown in Fig. 16B, by setting the temperature range shown in Fig. 16B, the entire garbage collection vehicle, which is the object 911, can be clearly recognized and extracted by image recognition. In this setting of the temperature range, the luminance values of the parts exceeding 42[°C] all become the maximum luminance value of 255.

[0015] In this case, since both the engine part of the garbage collection truck approaching 100[°C] and the tire part at about 80[°C] are uniformly displayed with the maximum luminance value of 255, the temperature of the engine part and the temperature of the tire part cannot be distinguished by the luminance value, resulting in the same information. Therefore, as shown in FIG. 16B, when the visibility of the entire garbage collection truck of the target 911 is improved, in the tone conversion image 910, there arises a problem that the temperature of the engine part (about 100[°C]) and the temperature of the tire part (about 80[°C]) cannot be distinguished.

[0016] FIGS. 17A and 17B are explanatory diagrams showing histograms of temperature data in image data. In the histograms shown in FIGS. 17A and 17B, the horizontal axis represents temperature [°C] and the vertical axis represents frequency.

[0017] In FIG. 17A, as the temperature range, the temperature range 500P is set to a temperature range of 0[°C] to 140[°C]. Also, the temperature range 501P is set to a temperature range of 20[°C] to 40[°C]. Further, the temperature range 502P is set to a temperature range of 50[°C] to 140[°C].

[0018] FIG. 17B is an enlarged view of the histogram of the region in FIG. 17A where the temperature range is 60[°C] to 120[°C]. As shown in FIG. 17B, for example, when there is a monitoring target device (monitoring target device) with a monitoring temperature of 60[°C] to 120[°C], there may be a local monitoring region in the region 920 around about 80[°C]. That is, when it is necessary to monitor 70[°C] to 100[°C] for the thermal runaway monitoring of the monitoring target device, only local changes may be extractable.

[0019] Here, in FIG. 17A, even when there are two or more user monitoring regions (temperature ranges), in the comparative example, only one region (temperature range) can be selected as the monitoring target. Therefore, the unselected monitoring regions will have the image data rounded off and discarded.

[0020] In particular, in an infrared image having temperature information, for example, it is composed of RAW information having a large amount of information such as 14 bits. When such rich 14-bit temperature information is subjected to tone conversion to 8 bits, information loss occurs.

[0021] Figs. 18A to 18C are explanatory diagrams showing infrared images 930 to 932 when the temperature ranges of Fig. 17A are assigned. That is, Fig. 18A shows an infrared image 930 in which a range from a minimum value to a maximum value is assigned by a temperature range 500P. Fig. 18B shows an infrared image 931 in which a range of a low temperature region (temperature range: 20[°C] to 40[°C]) is assigned by a temperature range 501P. Fig. 18C shows an infrared image 932 in which a range of a high temperature region (temperature range: 50[°C] to 140[°C]) is assigned by a temperature range 502P. In Figs. 18A to 18C, both facility equipment and people are targets for monitoring. Also, the infrared images 930 to 932 shall include tone-converted images.

[0022] In the infrared image 930 of Fig. 18A, the target facility equipment 930P can be seen, but the target person cannot be visually recognized. On the other hand, in the infrared image 931 of Fig. 18B, the target person 931P can be visually recognized, but the target facility equipment cannot be visually recognized. This is because the temperature information of the target facility equipment is saturated in the infrared image 931. On the other hand, in the infrared image 932 of Fig. 18C, the temperature state of the target facility equipment 932P can be seen, but the target person cannot be visually recognized.

[0023] In the embodiment according to the present invention, the visibility of objects in two or more temperature regions in an infrared image is improved, and information for each temperature range (temperature region) is assigned to a channel image to generate an extended image for machine learning. Specifically, improving the visibility achieves both temperature monitoring in temperature monitoring applications and person monitoring from the perspective of occupational safety and health. Also, the generation of the extended image creates a machine learning model by using the generated extended image as learning data for machine learning.

[0024] Accordingly, in the embodiment according to the present invention, it is possible to execute image processing such that the dynamic range of each of a plurality of objects shown on the screen becomes larger with respect to the infrared image.

[0025] <This embodiment> [Overall configuration of the image processing apparatus] FIG. 1 is a block diagram showing a main configuration example of an image processing system 200 according to this embodiment. The image processing system 200 according to this embodiment includes an image processing apparatus 100 and an infrared camera 180.

[0026] The image processing apparatus 100 includes a CPU (Central Processing Unit) 110, a storage unit 120, a ROM (Read Only Memory) 130, and a RAM (Random Access Memory) 140. The image processing apparatus 100 also includes an input unit 150, a display unit 160, a communication unit 170, and an internal bus 190. Further, the image processing apparatus 100 includes a learning data database 122, a learning engineering unit 310, and an inference model 320.

[0027] The CPU 110 is a central processing unit, and implements each process (functional block) shown in FIG. 2 by executing an image generation program 121 stored in the storage unit 120. Each process implemented by the CPU 110 will be described later with reference to FIG. 2. The CPU 110 also implements the learning engineering unit 310 by executing a machine learning program 123 stored in the storage unit 120.

[0028] The storage unit 120 is a large-capacity storage device, and is configured by, for example, a hard disk drive, a non-volatile memory, or the like. The storage unit 120 stores an image generation program 121 and a machine learning program 123. Note that the image generation program 121 and the machine learning program 123 may be stored in the ROM 130.

[0029] The RAM 140 functions as a work area that temporarily stores various programs, input data, output data, and parameters read from the ROM 130 and executable by the CPU 110.

[0030] The input unit 150 includes a keyboard having cursor keys, numeric input keys, and various function keys, and a pointing device such as a mouse. The input unit 150 outputs a key press signal of a key pressed on the keyboard or an operation signal by the mouse to the CPU 110 as an input signal. The CPU 110 executes various processes based on the operation signal from the input unit 150.

[0031] The display unit 160 includes, for example, a monitor such as a CRT (Cathode Ray Tube) or an LCD (Liquid Crystal Display). The display unit 160 displays various screens according to an instruction of a display signal input from the CPU 110. Note that the display unit 160 and the input unit 150 can also adopt a touch panel display.

[0032] The communication unit 170 includes a communication interface and communicates with an external device on a network. The communication unit 170 is connected to the infrared camera 180 and receives an infrared image captured by the infrared camera 180 as image data.

[0033] The infrared camera 180 is installed inside a factory or outdoors, and monitors the temperature of equipment to be monitored inside the factory and suspicious persons inside and outside the factory by capturing infrared images.

[0034] The internal bus 190 interconnects the components in the image processing apparatus 100.

[0035] The learning data database 122 is a database that stores the extended images generated by executing the image generation program 121. The extended images stored in the learning data database 122 are the image data of the infrared images captured by the infrared camera 180, which have been processed by the tone conversion image generation unit 111 and the temperature information storage processing unit 112 in FIG. 2. The extended images stored in the learning data database 122 are used for machine learning by the learning engineering department 310.

[0036] The learning engineering department 310 performs machine learning using the learning data (i.e., extended images) stored in the learning data database 122. Specifically, the learning engineering department 310 uses the extended images generated by the temperature information storage processing unit 112 in FIG. 2 as learning data for creating a machine learning model and performs machine learning. When the learning engineering department 310 performs machine learning using labels, it generates an inference model for detecting the target. When the learning engineering department 310 performs machine learning without using labels, it performs grouping to classify the tone conversion images or infrared images into groups.

[0037] The inference model 320 is the result of machine learning the extended images in the learning engineering department 310. By applying this inference model 320 to the inference engineering department 410 of the edge device 400 in FIG. 3, the edge device 400 detects the target from the tone conversion image or the infrared image.

[0038] [Functions of the Image Processing Apparatus] Next, the functions of the CPU 110 of the image processing apparatus 100 according to the present embodiment will be described with reference to FIG. 2.

[0039] FIG. 2 is a functional block diagram showing the functions of the CPU 110 of the image processing apparatus 100 according to the present embodiment. By executing the image generation program 121 in FIG. 1, the CPU 110 realizes the tone conversion image generation unit 111, the temperature information storage processing unit 112, and the image integration unit 113.

[0040] The gradation conversion image generation unit 111 performs gradation conversion on an infrared image in which two or more objects are imaged, using temperature information for each object, and generates a gradation conversion image. Further, the gradation conversion image generation unit 111 can also generate a gradation conversion image based on temperature information preset by the user. Further, the gradation conversion image generation unit 111 may generate the gradation conversion image that emphasizes an artificial temperature range based on temperature information that can occur in nature. Further, the gradation conversion image generation unit 111 may continue to monitor the object, obtain a daily temperature range or an abnormal temperature range, and dynamically determine temperature information based on the obtained temperature range.

[0041] The temperature information storage processing unit 112 assigns the generated plurality of gradation conversion images to an R (Red) component, a G (Green) component, and a B (Blue) component respectively as three-channel images, and generates an extended image.

[0042] The image integration unit 113 generates an integrated image based on the generated extended image. Here, the image integration unit 113 can perform predetermined weighting on each component of the generated extended image. Further, the image integration unit 113 may display the generated integrated image on the display unit 160 and monitor the change over time.

[0043] [Learning Device and Edge Device] FIG. 3 is a block diagram showing the configurations of the learning device 300 and the edge device 400. As shown in FIG. 3, the learning device 300 includes a learning data database 122, a learning engineering unit 310, and an inference model 320.

[0044] The learning device 300 functions as a learning device that performs machine learning by including the learning data database 122 and the learning engineering unit 310 provided in the image processing device 100. Thereby, the learning device 300 can obtain an inference model 320 obtained by performing machine learning on the extended image in the learning engineering unit 310.

[0045] Further, as shown in FIG. 3, the edge device 400 is configured to include an inference engineering unit 410 and an inference model 320.

[0046] The inference engineering unit 410 performs inference on the input image 250 (expanded image) using the inference model 320. By performing inference on the input image 250 using the inference model 320, the inference engineering unit 410 can obtain the recognition result 290. Note that a tone-converted image and an infrared image can be used as the input image 250.

[0047] Note that the edge device 400 can perform inference by an apparatus including the inference engineering unit 410 and the inference model 320. Therefore, in this embodiment, the image processing device 100 may further include the inference engineering unit 410. By further including the inference engineering unit 410, the image processing device 100 functions as an edge device.

[0048] [Hardware Configuration of Learning Device] FIG. 4 is an explanatory diagram showing a schematic hardware configuration of the learning device 300 according to this embodiment. The learning device 300 is configured by, for example, a server.

[0049] The learning device 300 includes a CPU 301, a storage unit 302, a ROM 303, a RAM 304, an input unit 305, a display unit 306, a communication unit 307, a learning data database 122, a learning engineering unit 310, and an inference model 320.

[0050] The CPU 301 reads out and executes the machine learning program 123 from the storage unit 302 to embody the learning engineering unit 310 and control the overall operation of the learning device 300. At this time, various data stored in the storage unit 302 are referred to. The storage unit 302 is configured by a large-capacity storage medium such as a semiconductor memory or a hard disk drive, for example.

[0051] The CPU 301 performs transmission and reception of various data with external devices connected to a communication network such as a LAN (Local Area Network) or WAN (Wide Area Network) via the communication unit 307.

[0052] The communication unit 307 is composed of a communication control card (communication interface) such as a NIC (Network Interface Card).

[0053] The input unit 305 is provided with various operation keys such as numeric keys and start keys. The input unit 305 receives various input operations by the user and outputs an operation signal as an input signal to the CPU 301.

[0054] The display unit 306 is, for example, a liquid crystal display, and displays characters, graphics, images, etc.

[0055] [Hardware Configuration of Edge Device] FIG. 5 is an explanatory diagram showing a schematic hardware configuration of the edge device 400 according to the present embodiment.

[0056] The edge device 400 is configured to include a CPU 401, a storage unit 402, a ROM 403, a RAM 404, an inference engineering unit 410, and an inference model 320.

[0057] The CPU 401 reads and executes the inference program 421 from the storage unit 402 to embody the inference engineering unit 410 and control the overall operation of the edge device 400. The inference model 320 of the edge device 400 applies the inference model 320 of the image processing device 100 shown in FIGS. 2 and 4, and the extended images, tone-converted images, infrared images, etc. stored in the storage unit 402 are referred to. The storage unit 402 is composed of a large-capacity storage medium such as a semiconductor memory or a hard disk drive, for example.

[0058] [First Embodiment] [Processing of Image Processing Apparatus] Next, the processing of the image processing apparatus 100 according to the first embodiment will be described. FIG. 6 is a flowchart showing a process of generating an extended image from an infrared image captured by the infrared camera 180 in the image processing apparatus 100 according to the first embodiment.

[0059] First, the image processing apparatus 100 receives various settings by a user operation via the input unit 150 (step S001). Specifically, the image processing apparatus 100 receives settings such as a temperature range, a weight, and the like.

[0060] FIG. 7 is an explanatory diagram showing a histogram of temperature data in infrared image data captured by the infrared camera 180. In the histogram shown in FIG. 7, the horizontal axis represents temperature [°C], and the vertical axis represents frequency.

[0061] In FIG. 7, as the temperature range, the temperature range 500 is set to a temperature range of 0 [°C] to 20 [°C]. The temperature range 500 is set to increase the dynamic range of a so-called normal-temperature object. Also, the temperature range 501 is set to a temperature range of 30 [°C] to 45 [°C]. The temperature range 501 is set to increase the dynamic range of an object heated to about human body temperature. Also, the temperature range 502 is set to a temperature range of 60 [°C] to 140 [°C]. The temperature range 502 is set to increase the dynamic range of an object heated to 60 [°C] or higher by reaction heat.

[0062] Returning to FIG. 6, the description will be continued. Next, the tone conversion image generation unit 111 of the image processing apparatus 100 acquires temperature data from the image data (step S003).

[0063] The gradation conversion image generation unit 111 extracts the image data according to the temperature range set in step S001 (step S007). In this case, the gradation conversion image generation unit 111 repeats for the set temperature range (step S005). Specifically, the gradation conversion image generation unit 111 performs gradation conversion on the infrared image using the temperature information for each target, and generates a gradation conversion image.

[0064] Figs. 8A to 8C are explanatory diagrams showing gradation conversion images obtained by performing gradation conversion for each temperature range. Fig. 8A shows a gradation conversion image 510 for the temperature range 500. Fig. 8B shows a gradation conversion image 511 for the temperature range 501. Fig. 8C shows a gradation conversion image 512 for the temperature range 502.

[0065] In the gradation conversion image 510 of Fig. 8A, it is shown that no target is photographed in the temperature range (0 [°C] to 20 [°C]) of the temperature range 500. In the gradation conversion image 511 of Fig. 8B, it is shown that the person 511P as the target can be easily extracted by image recognition processing in the temperature range (30 [°C] to 45 [°C]) of the temperature range 501. In the gradation conversion image 512 of Fig. 8C, it is shown that the monitored device 512P as the target can be easily extracted by image recognition processing in the temperature range (60 [°C] to 140 [°C]) of the temperature range 502.

[0066] Returning to Fig. 6, the description continues. Next, the temperature information storage processing unit 112 of the image processing apparatus 100 assigns the generated plurality of gradation conversion images 510 to 512 to the R component, G component, and B component respectively as 3-channel images, and generates an extended image (step S009).

[0067] Fig. 9 is a conceptual diagram showing the concept that the temperature information storage processing unit 112 assigns the generated plurality of gradation conversion images 510 to 512 to the R component, G component, and B component respectively as 3-channel images, and generates an extended image. As shown in Fig. 9, the temperature information storage processing unit 112 generates 3-channel images in which the plurality of gradation conversion images 510 to 512 are assigned to different channels.

[0068] In this way, the temperature information storage processing unit 112 generates an extended image by assigning each of the three-channel images as the R component, G component, and B component. Then, the generated extended image is used as an image for machine learning input.

[0069] Returning to FIG. 6, the description will be continued. In step S011, the temperature information storage processing unit 112 determines whether the generation of the extended image has been completed. If the generation of the extended image has not been completed (No in step S011), the temperature information storage processing unit 112 returns to step 003. Then, the processing after step S003 is continued.

[0070] On the other hand, if the generation of the extended image has been completed (Yes in step S011), the temperature information storage processing unit 112 ends the processing of FIG. 6.

[0071] As described above, the image processing apparatus 100 according to the first embodiment performs tone conversion using the temperature information for each target by the tone conversion image generation unit 111, and generates a tone conversion image. Further, the image processing apparatus 100 assigns the generated plurality of tone conversion images to the R component, G component, and B component as three-channel images respectively by the temperature information storage processing unit 112, and generates an extended image.

[0072] According to the first embodiment, the image processing apparatus 100 according to the first embodiment can generate an extended image in which a plurality of tone conversion images are assigned to the R component, G component, and B component as three-channel images respectively, so that the necessary processing can be executed for each different temperature range.

[0073] For example, the image processing apparatus 100 accumulates the generated extended image in the learning data database 122 and performs machine learning in the learning and engineering unit 310. In this case, the image processing apparatus 100 according to the first embodiment can efficiently perform machine learning, for example, by assigning information for each temperature range to the R component, the G component, and the B component as separate channels. Further, the image processing apparatus 100 can prevent loss of the input of the inference process of machine learning.

[0074] Furthermore, since the extraction process for each temperature range unit in the tone conversion image generation unit 111 is a reversible process, the tone conversion image can be generated while maintaining the original temperature information even when the temperature range is changed. As a result, the image processing apparatus 100 can easily increase the extended images used for machine learning.

[0075] <Second Embodiment> [Processing of Image Processing Apparatus] Next, the processing of the image processing apparatus 100 according to the second embodiment will be described. FIG. 10 is a flowchart showing a process of generating an integrated image from an infrared image captured by the infrared camera 180 in the image processing apparatus 100 according to the second embodiment.

[0076] The processing of the image processing apparatus 100 according to the second embodiment differs from the processing of the image processing apparatus 100 according to the first embodiment in that a process of integrating images is added to the flowchart of FIG. 6. That is, in the flowchart of FIG. 10, the same processing as that of the flowchart of FIG. 6 is performed from step S001 to step S009. Therefore, in the second embodiment, the description of step S013 shown in FIG. 10 will be given.

[0077] In step S013, the image integration unit 113 of the image processing apparatus 100 generates an integrated image based on the extended image generated by the temperature information storage processing unit 112. The image integration unit 113 can perform predetermined weighting on each component of the generated extended image. Further, the image integration unit 113 can display the integrated image on the display unit 160 and can also monitor changes over time.

[0078] As described above, the image processing apparatus 100 according to the second embodiment generates an integrated image based on the generated extended image by the image integration unit 113.

[0079] According to the second embodiment, since the image processing apparatus 100 according to the second embodiment can generate a combined image by the image integration unit 113, the visibility (monitoring visibility) of the user is improved. For example, the image integration unit 113 can perform an enhancement process on the information for each temperature range, assign it to the R component, G component, and B component respectively, and improve the visibility of the user.

[0080] In addition, the image integration unit 113 can also highlight (emphasis display) the temperature range that needs to be particularly noted. The image integration unit 113 can generate a combined image from the tone conversion image 511 in FIG. 8B and the tone conversion image 512 in FIG. 8C. In this case, the combined image can display the target person 511P and the target monitoring device 512P, and can be presented to the user with red indicating that the monitoring device 512P is at a high temperature.

[0081] In particular, the image processing apparatus 100 according to the second embodiment can be used for abnormality detection for each temperature range. For example, it can preferably monitor heat stroke alerts, approach of people to high-temperature equipment, thermal runaway of equipment, and ignition and smoke.

[0082] <Third Embodiment> The tone conversion image generation unit 111 of the image processing apparatus 100 can appropriately change the temperature range according to the usage state of the user.

[0083] FIGS. 11A to 11C are explanatory diagrams showing tone conversion images 520 to 522 obtained by changing the temperature range of the tone conversion image generation unit 111 to a predetermined range and performing tone conversion.

[0084] In the third embodiment, it is shown that the temperature range of the temperature range may include a temperature below 0 degrees Celsius, that is, a negative temperature. That is, the gradation conversion image generation unit 111 may perform gradation conversion using negative temperature information for each target and generate a gradation conversion image.

[0085] FIGS. 11A to 11C are explanatory diagrams showing gradation conversion images 520 to 522 obtained by changing the temperature range of the temperature range to a low temperature by the gradation conversion image generation unit 111 and performing gradation conversion.

[0086] The gradation conversion image 520 in FIG. 11A shows the case where the temperature range is set to -10 [°C] to -5 [°C]. The gradation conversion image 521 in FIG. 11B shows the case where the temperature range is set to -7 [°C] to 20 [°C]. The gradation conversion image 522 in FIG. 11C shows the case where the temperature range is set to 40 [°C] to 125 [°C].

[0087] In the case of the third embodiment, the image processing apparatus 100 can perform machine learning by the learning and engineering unit 310 while increasing the variations of the gradation conversion images using the gradation conversion images 520 to 522.

[0088] Further, since the image processing apparatus 100 can include the image integration unit 113, for example, when the temperature range is changed and a person is detected in the refrigerator, the person may be highlighted. Specifically, in the gradation conversion image 521 in FIG. 11B, when a person 521P as a target is detected in an infrared image showing the internal situation of the refrigerator, the image integration unit 113 highlights the person 521P as a target.

[0089] On the other hand, FIGS. 12A to 12C are explanatory diagrams showing gradation conversion images 530 to 532 obtained by changing the temperature range of the temperature range to a high temperature by the gradation conversion image generation unit 111 and performing gradation conversion.

[0090] The tone conversion image 530 in FIG. 12A shows the case where the temperature range is set to -5[°C] to +5[°C]. The tone conversion image 521 in FIG. 12B shows the case where the temperature range is set to 20[°C] to 40[°C]. The tone conversion image 532 in FIG. 12C shows the case where the temperature range is set to 40[°C] to 125[°C].

[0091] Similar to the tone conversion images 520 to 522 in FIGS. 11A to 11C, the tone conversion images 530 to 532 in FIGS. 12A to 12C can also be subjected to machine learning by the learning engineering unit 310.

[0092] In addition, since the image processing apparatus 100 can include an image integration unit 113, for example, when the temperature range is changed and a high-temperature facility device is detected, the high-temperature facility device may be highlighted.

[0093] That is, in the tone conversion image 531 in FIG. 12B, when a high-temperature facility device 531P is detected in the infrared image showing the internal situation of the warehouse, the image integration unit 113 highlights the high-temperature facility device 531P that is the monitoring target. Further, in the tone conversion image 532 in FIG. 12C, when a higher-temperature facility device 532P is detected in the infrared image showing the internal situation of the warehouse, the image integration unit 113 may highlight the higher-temperature facility device 532P even more.

[0094] <Fourth Embodiment> The tone conversion image generation unit 111 of the image processing apparatus 100 can perform reversible processing by recording a conversion formula regarding the assignment of the temperature range. In this case, the tone conversion image generation unit 111 records the conversion formula described later in the storage unit 15.

[0095] FIG. 13 is an explanatory diagram showing a histogram of certain temperature data. The horizontal axis of this histogram indicates temperature, and the vertical axis indicates the number of pixels. In FIG. 13, as the temperature range, the temperature range 700 has a temperature range set to 0[°C] to 10[°C]. The temperature range 700 is set to make the dynamic range of the slightly lower-temperature target larger. In addition, the temperature range 701 has a temperature range set from 25[°C] to 50[°C]. The temperature range 701 is set to increase the dynamic range of an object heated to about human body temperature. In addition, the temperature range 702 has a temperature range set from 60[°C] to 140[°C]. The temperature range 702 is set to increase the dynamic range of an object heated to 60[°C] or higher by reaction heat.

[0096] The gradation conversion image generation unit 111 receives the assignment of the predetermined temperature ranges 700 to 702, for example, based on Equation (1). Note that for T, a temperature range a corresponding to the temperature range 700, a temperature range b corresponding to the temperature range 701, and a temperature range c corresponding to the temperature range 702 are set.

[0097]

Equation

[0098] In this case, the gradation conversion image generation unit 111 calculates the luminance value v indicating the temperature in each range based on Equation (2). Note that the luminance value 0 indicates that the pixel does not correspond to the temperature range.

[0099]

Equation

[0100] That is, the temperature range 700 shown in FIG. 13 is from 0[°C] to 10[°C]. In addition, the temperature range 701 shown in FIG. 13 is from 25[°C] to 50[°C]. In addition, the temperature range 702 shown in FIG. 13 is from 60[°C] to 140[°C].

[0101] In this way, the tone conversion image generation unit 111 can reversibly process the generated tone conversion image by assigning a predetermined temperature range and recording it in the storage unit 15. Note that the luminance value v is rounded to 255 when it is equal to or higher than the temperature range. When the luminance value v is rounded to 255, the tone conversion image generation unit 111 cannot perform reversible processing.

[0102] <Fifth Embodiment> The image integration unit 113 of the image processing apparatus 100 can perform predetermined weighting on each component of the generated enlarged image.

[0103] FIG. 14 is an explanatory diagram showing the concept of performing predetermined weighting on each component of the generated enlarged image in a predetermined histogram. As shown in FIG. 14, since the frequency of the region 800 is high, it is a normal monitoring region.

[0104] On the other hand, since the frequency of the region 801 is not relatively high, there may be a case where, although it is local, the user wants it to be a monitoring region.

[0105] When monitoring the thermal runaway of the device to be monitored (monitoring target device), the image integration unit 113 performs weighting on the region 801 which is a special region. Specifically, the image integration unit 113 assigns the RGB components to a predetermined special region to emphasize the R (red) component, or weights the luminance to emphasize the luminance.

[0106] That is, if the high temperature range is the focus, the image integration unit 113 can assign the R component so that the high temperature range stands out and increase the weight of the luminance. Note that the image integration unit 113 is not limited to the RGB components and may be CMYK (Cyan Magenta Yellow Key plate) components.

[0107] Further, the image integration unit 113 may calculate a special area based on the occurrence temperature that can occur in nature, or may be arbitrarily set by the user. Furthermore, the image integration unit 113 may monitor the temperature for a predetermined period (constant period) and dynamically change the monitored temperature according to the frequency at the monitored temperature.

[0108] <Sixth Embodiment> The image integration unit 113 of the image processing apparatus 100 may display the integrated image on the display unit 160 and monitor changes over time.

[0109] FIGS. 15A to 15C are explanatory diagrams showing an example in which a predetermined luminance is assigned to a temperature range in an infrared image or a tone-converted image, and objects in two or more temperature ranges are displayed in a combined image.

[0110] FIG. 15A is an infrared image 514 in which the image integration unit 113 assigns 0 to 127 to a predetermined temperature range. FIG. 15B is an infrared image 515 in which the image integration unit 113 assigns 128 to 255 to a predetermined temperature range different from the temperature range of FIG. 15A. FIG. 15C is a combined image 516 obtained by combining the infrared image of FIG. 15A and the infrared image of FIG. 15B.

[0111] The image integration unit 113 can obtain the combined image 516 of FIG. 15C by combining the infrared image 514 of FIG. 15A and the infrared image 515 of FIG. 15B. Therefore, the image integration unit 113 can display each of the object in the temperature range of FIG. 15A and the object in the temperature range of FIG. 15B in the combined image 516.

[0112] Also, the image integration unit 113 may assign to the RBG components at the time of generating the combined image, or may assign to the YCrCb components, or the C (Cyan) component, M (Magenta) component, and Y (Yellow) component. Also, the RGB assignment may be four or more. In this case, for example, a part of the RGB components may overlap, and 128 to 255 of the R component may be assigned to a high temperature, and 0 to 127 of the R component may also be assigned to a predetermined low temperature range.

[0113] (Modification example) The present invention is not limited to the above-described embodiments, and can be implemented with modifications without departing from the spirit of the present invention. For example, there are the following (a) to (m).

[0114] (a) The tone conversion image generation unit 111 of the image processing apparatus 100 may generate a tone conversion image based on temperature information preset by the user. Further, the tone conversion image generation unit 111 may generate a tone conversion image that emphasizes an artificial temperature range based on temperature information that can occur in nature. Further, the tone conversion image generation unit 111 may continuously monitor the target monitoring, acquire a daily temperature range or an abnormal temperature range, and dynamically determine temperature information based on the acquired temperature range.

[0115] (b) The image processing apparatus 100 according to the first embodiment may further include a learning engineering unit 310 that performs machine learning using the generated extended image as learning data for creating a machine learning model.

[0116] (c) The image processing apparatus 100 according to the second embodiment may perform predetermined weighting on each component of the generated extended image by the image integration unit 113. Further, the image integration unit 113 may display the integrated image on the display unit 160 and monitor changes over time.

[0117] (d) The learning apparatus 300 may include a learning engineering unit 310 that performs machine learning using the extended image generated by the image processing apparatus 100 as learning data for creating a machine learning model.

[0118] (f) The edge device 400 may include an inference model 320 that has been machine-learned by the image processing apparatus 100 or the learning apparatus 300, and an inference engineering unit 410 that applies a tone conversion image or an infrared image to the inference model 320 to infer a target.

[0119] (g) The image processing system 200 according to the first embodiment may be a system that displays the temperature transition along the time axis. Further, the image processing system 200 may automatically detect or manually set a target device to be monitored and monitor the temperature distribution in a local area.

[0120] (h) The temperature range may be higher or lower than the temperature range in which warm-blooded animals exist. Thereby, for example, it is possible to monitor a person working in a low-temperature refrigerator or freezer, or a person working in a high-temperature environment. (i) The target (object) for setting the temperature range is not limited to a person, and may be a living thing such as a pet or livestock. Further, the target (object) may be a high-temperature equipment used in the production process, a tire of a vehicle in motion, or a construction vehicle body such as a forklift or a shovel car. For example, in the case of a construction vehicle body, not only does the temperature range become high, but also the body size of the vehicle is large, so the influence on the surroundings is large, and there is a particularly high need for monitoring. Therefore, for example, it is not limited to construction vehicles including caterpillars and wheel loaders, and emergency vehicles that can become hot, as well as large parking lots, fire departments, and other land and real estate may also be monitoring targets.

[0121] (j) The object to be machine-learned by the image processing device 100 or the learning device 300 may be human detection, animal detection, etc. It may also be a specific equipment, heavy machine body, engine, tire, etc. (k) The temperature range may be two or more and is not limited.

[0122] (l) The user may arbitrarily set which component of RGB to assign the value of the temperature range to. For example, for a high-temperature temperature range, it may be assigned to the R component so as to be conspicuous as the part of interest. (m) When there are four or more temperature ranges, the image integration unit 113 may arbitrarily set the assignment, such as assigning two or more to one component, when generating the combined image. Specifically, for the first temperature range, 0 to 127 is assigned to the R component. Also, for the second temperature range, 128 to 255 is assigned to the R component. Further, for the third temperature range, the G component is assigned. And for the fourth temperature range, the B component is assigned.

Explanation of Signs

[0123] 100 Image processing apparatus 110 CPU 111 Tone conversion image generation unit 112 Temperature information storage processing unit 113 Image integration unit 120 Storage unit 121 Image generation program 122 Learning data database 123 Machine learning program 130 ROM 140 RAM 150 Input unit 160 Display unit 170 Communication unit 180 Infrared camera 200 Image processing system 250 Input image 290 Recognition result 300 Learning device 301 CPU 302 Storage unit 303 ROM 304 RAM 305 Input unit 306 Display unit 307 Communication unit 310 Learning engineering department 320 Inference model 400 Edge device 401 CPU 402 Storage unit 403 ROM 404 RAM 410 Inference engineering department 421 Inference Program

Claims

1. A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, to generate a tone conversion image; A temperature information storage processing unit that assigns the generated plurality of tone conversion images as three-channel images to generate an extended image; An image processing apparatus comprising the above.

2. The image processing apparatus according to claim 1, further comprising an image integration unit that generates an integrated image based on the generated extended image. The image processing apparatus according to claim 1.

3. The tone conversion image generation unit: Generates the tone conversion image based on temperature information preset by a user. The image processing apparatus according to claim 1.

4. The tone conversion image generation unit: Generates the tone conversion image that emphasizes an artificial temperature range based on temperature information that can occur in nature. The image processing apparatus according to claim 1.

5. The tone conversion image generation unit: Continues to monitor the object, obtains a daily temperature range or an abnormal temperature range, and dynamically determines the temperature information based on the obtained temperature range. The image processing apparatus according to claim 1.

6. The image processing apparatus according to claim 1, further comprising a learning engineering department that performs machine learning on the generated extended image as learning data for creating a machine learning model. The image processing apparatus according to claim 1.

7. The image integration unit: Performs predetermined weighting on each component of the generated extended image. The image processing apparatus according to claim 2.

8. The image integration unit: Displays the integrated image on a display unit and monitors changes over time. The image processing apparatus according to claim 2.

9. A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, to generate a tone conversion image; A learning engineering department that performs machine learning on the extended image generated by an image processing apparatus comprising a temperature information storage processing unit that assigns the generated plurality of tone conversion images as three-channel images to generate an extended image, as learning data for creating a machine learning model; A learning apparatus.

10. A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, to generate a tone conversion image; A temperature information storage processing unit that assigns the generated plurality of tone conversion images as three-channel images to generate an extended image; A learning engineering department that performs machine learning using the generated extended image as learning data for creating a machine learning model, and an inference model obtained by performing machine learning using an image processing apparatus including the learning engineering department. An inference engineering department that applies the tone conversion image generated by the tone conversion image generation unit to the inference model to infer the object. An edge device comprising the above.

11. A step of performing tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, and generating a tone conversion image. A step of generating an extended image by assigning the generated plurality of tone conversion images as three-channel images. An image generation method for executing the above.

12. A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, and generates a tone conversion image. A step of performing machine learning on the extended image generated by an image processing apparatus including a temperature information storage processing unit that generates an extended image by assigning the generated plurality of tone conversion images as three-channel images, using the extended image as learning data for creating a machine learning model. A learning method for a learning apparatus for executing the above.

13. A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, and generates a tone conversion image. A temperature information storage processing unit that generates an extended image by assigning the generated plurality of tone conversion images as three-channel images. An inference model obtained by performing machine learning using an image processing apparatus including a learning engineering department that performs machine learning using the generated extended image as learning data for creating a machine learning model. A step of applying the tone conversion image generated by the tone conversion image generation unit to the inference model to infer the object. An inference method for executing the above.

14. A procedure for performing tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, and generating a tone conversion image. A procedure for generating an extended image by assigning the generated plurality of tone conversion images as three-channel images. An image generation program for causing a computer to execute the above.

15. A tone conversion image generation unit that performs tone conversion on an infrared image in which two or more objects are imaged, using the temperature information for each object, and generates a tone conversion image. A temperature information storage processing unit that assigns the plurality of generated gradation conversion images as three-channel images to generate an extended image, and a procedure for performing machine learning using the extended image generated by the image processing apparatus as learning data for creating a machine learning model A machine learning program that causes a computer to execute

16. For an infrared image in which two or more objects are imaged, a gradation conversion image generation unit that performs gradation conversion using the temperature information for each object to generate a gradation conversion image A temperature information storage processing unit that assigns the plurality of generated gradation conversion images as three-channel images to generate an extended image A learning engineering unit that performs machine learning using the generated extended image as learning data for creating a machine learning model, and a procedure for applying the gradation conversion image generated by the gradation conversion image generation unit to the inference model obtained by machine learning using the image processing apparatus provided with the learning engineering unit to infer an object A procedure for inferring an object by applying the gradation conversion image generated by the gradation conversion image generation unit to the inference model An inference program that causes a computer to execute

Images