Image processing device, edge device, image generation method, and image generation program

The image processing device improves infrared image object detection by generating extended images with enhanced contrast and noise robustness, addressing the challenges of poor infrared image quality and enhancing object detection performance through machine learning-based image processing.

WO2024232156A1PCT designated stage expired Publication Date: 2024-11-14KONICA MINOLTA INC
View PDF 4 Cites 0 Cited by

Patent Information

Application Number
PCT/JP2024/009263
Authority / Receiving Office
WO · WO
Patent Type
Applications
Current Assignee / Owner
Priority Date
2023-05-10
Filing Date
2024-03-11
Publication Date
2024-11-14

AI Technical Summary

Technical Problem

Infrared images generated by thermal sensors face challenges such as poor contrast, noise, and difficulty in distinguishing objects from backgrounds due to moisture and distance-related disturbances, making them unsuitable for reliable object detection.

Method used

An image processing device that creates extended images with enhanced contrast and noise robustness by using machine learning models, where infrared detection signals are used to generate training data, and the image expansion unit applies predetermined contrasts to specific object areas, generating multiple images to simulate various conditions, thereby improving object detection performance.

Benefits of technology

The solution significantly enhances object detection accuracy and robustness in infrared images by emphasizing target objects and reducing noise, improving the performance of machine learning models in recognizing objects across different scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure JP2024009263_14112024_PF_FP_ABST
    Figure JP2024009263_14112024_PF_FP_ABST
Patent Text Reader

Abstract

An image processing device (100) is provided with an image expansion unit (40) that creates, from an infrared detection signal obtained by imaging an object and on the basis of information about the object, expansion images (71-78) for use as training data for machine learning model creation, or for use as input images in object recognition processing by using a machine learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Image processing device, edge device, image generation method, and image generation program

[0001] The present invention relates to an image processing device, an edge device, an image generation method, and an image generation program.

[0002] It has been known for some time that images are generated using thermal sensors that detect heat. For example, a camera using a thermal sensor (called a thermal sensor camera) generates an infrared image by converting infrared light emitted from an object into an electrical signal. In this case, the thermal sensor camera normalizes the detection signal from its minimum value to its maximum value and assigns the normalized detection signal to a brightness signal ranging from 0 to 255, thereby generating an infrared image with a predetermined contrast.

[0003] However, the infrared light being detected is easily affected by moisture and prone to disturbances. Furthermore, the farther the distance, the stronger the influence of disturbances becomes, making the infrared light more susceptible to noise. Furthermore, the target to be detected is often a specific object, not the entire screen.

[0004] Here, Patent Document 1 discloses an object detection device as an invention related to infrared images, stating that "an image pre-processing unit 22 generates inverted image data by performing gradation conversion on the captured image data acquired by the far-infrared camera 10. An object classifier 24 performs object classification processing on each of the captured image data and the inverted image data based on the results of prior learning using learning data, and outputs classification results including information on the classification of the type of object in each of the captured image data and the inverted image data, and accuracy information indicating the reliability of the classification in each of the captured image data and the inverted image data. An image post-processing unit 26 then finally classifies the object in the captured image data based on the classification results by the object classifier 24" (see the abstract of Patent Document 1).

[0005] Furthermore, Patent Document 2 discloses a far-infrared image processing device including "an image information input unit (22a) that takes in image data acquired by a far-infrared camera (10); a histogram generation unit (22b) that generates a histogram of pixel values ​​contained in the image data; a histogram reconstruction unit (22c) that uses an upper threshold and a lower threshold to correct the frequency of the histogram generated by the histogram generation unit so that it becomes a value between the upper threshold and the lower threshold; a flattening processing unit (22d) that performs a histogram flattening process based on the corrected histogram and generates image data with a flattened histogram; and a processed image output unit (14a) that outputs the image data processed by the flattening processing unit" (see the abstract of Patent Document 2).

[0006] JP 2021-163222 A JP 2020-107097 A

[0007] However, when recognizing an object from an infrared image, the infrared image does not contain color information and therefore contains less information than a typical visible light image. Furthermore, the contrast between the object and the background in an infrared image is poor, making it difficult to recognize the object from the background. Furthermore, infrared images tend to have blurred edges and are prone to noise in rainy weather. Therefore, infrared images are not suitable for object detection.

[0008] In response to such problems, an object of the present invention is to provide an image processing device, an edge device, an image generation method, and an image generation program that improve the performance of detecting objects in infrared images.

[0009] That is, the above-described object of the present invention is achieved by the following configuration: (1) An image processing device including an image extension unit that generates an extended image based on information about an object from an infrared detection signal obtained by capturing an image of the object, the extended image being used as learning data for creating a machine learning model, or as an input image in a recognition process of the object by the machine learning model.

[0010] (2) The image processing device according to (1), wherein the object is a solid having a specific temperature.

[0011] (3) The image processing device according to (1), wherein the information about the object is a temperature range in which a specific object can exist.

[0012] (4) The image processing device according to (1), wherein the information on the object is an image area where a specific object exists.

[0013] (5) The image processing device according to (1), wherein the image extension unit, when creating the extended image, adds a predetermined contrast to a range of infrared information in which a specific object may exist.

[0014] (6) The image processing device according to (1), wherein the image extension unit generates a plurality of extended images, each of which has a different contrast, in a range of infrared information in which a specific object may exist.

[0015] (7) The image processing device described in (1) further includes an image generation unit that generates an infrared image from an infrared detection signal obtained by capturing an image of the object, and an identification unit that identifies an image area in which a specific object exists based on the infrared image, wherein the image extension unit adds a predetermined contrast to the image area in which the specific object exists when creating the extended image.

[0016] (8) The image processing device according to (7), wherein the image extension unit, when creating the extended image, uses the outside of the image region as a background image and imparts different contrast to the background image.

[0017] (9) The image processing device according to (1), further comprising: a learning process unit that performs machine learning using the learning data.

[0018] (10) The image processing device according to (1), wherein the image extension unit adds a predetermined contrast to a range of infrared information in which a specific object may exist when generating an input image used in the object recognition process.

[0019] (11) The image processing device according to (7), wherein the image extension unit adds a predetermined contrast to an image area where a specific object exists when generating an input image used in the object recognition process.

[0020] (12) An edge device comprising: an image extension unit that creates an extended image based on information about an object from an infrared detection signal obtained by capturing an image of the object, and the extended image is used as learning data for creating a machine learning model or as an input image in the object recognition process using the machine learning model; and an inference model created by machine learning using the learning data, the image extension unit being included in the image processing device; and an inference process unit that applies the extended image created in the image extension unit to the inference model and infers the object.

[0021] (13) An image generation method including: creating an augmented image based on information about an object from an infrared detection signal obtained by capturing an image of the object, the augmented image being used as learning data for creating a machine learning model, or as an input image in a recognition process of the object using the machine learning model.

[0022] (14) An image generation program for causing a computer to execute the following steps: based on information about an object, an infrared detection signal obtained by capturing an image of the object is used as learning data for creating a machine learning model, or an augmented image is created as an input image in the recognition process of the object using the machine learning model.

[0023] According to the present invention, it is possible to improve the performance of detecting an object in an infrared image.

[0024] 1 is a functional block diagram showing the functions of an image forming apparatus according to a first embodiment. FIG. 2 is a functional block diagram showing the functions of an image forming apparatus according to a second embodiment. FIG. 3 is a functional block diagram showing the functions of an image forming apparatus according to a third embodiment. FIG. 4 is a block diagram showing a concept of performing either a learning process or an inference process, or both. FIG. 5 is an explanatory diagram showing a schematic hardware configuration of an image forming apparatus according to the present embodiment. FIG. 6 is an explanatory diagram showing a schematic hardware configuration of an edge device according to the present embodiment. FIG. 7 is an explanatory diagram showing a concept of an image forming apparatus according to the first embodiment creating an augmented image to be used as learning data for creating a machine learning model from an infrared image of an object based on information about the object, from an infrared image of an object. FIG. 8 is an explanatory diagram showing a concept of an image forming apparatus according to the first embodiment creating an augmented image to be used as learning data for creating a machine learning model from an infrared image of an object based on information about the object, from an infrared image of an object. FIG. 9 is an explanatory diagram showing a concept of an image forming apparatus according to the first embodiment creating an augmented image to be used as learning data for creating a machine learning model from an infrared image of an object based on information about the object, from an infrared image of an object. FIG. 5 is an explanatory diagram (part 5) illustrating the concept of an image forming apparatus according to the first embodiment creating an augmented image to be used as learning data for creating a machine learning model from an infrared image of an object based on information about the object. FIG. 6 is an explanatory diagram (part 6) illustrating the concept of an image forming apparatus according to the first embodiment creating an augmented image to be used as learning data for creating a machine learning model from an infrared image of an object based on information about the object. FIG. 7 is an explanatory diagram (part 8) illustrating the concept of an image forming apparatus according to the first embodiment creating an augmented image to be used as learning data for creating a machine learning model from an infrared image of an object based on information about the object.FIG. 1 is a diagram illustrating a concept when a temperature range is set as a parameter for a human subject. FIG. 1 is an explanatory diagram illustrating a concept when an image forming apparatus according to a second embodiment creates an augmented image to be used as an input image in an object recognition process using a machine learning model (Part 1). FIG. 2 is an explanatory diagram illustrating a concept when an image forming apparatus according to a second embodiment creates an augmented image to be used as an input image in an object recognition process using a machine learning model (Part 2). FIG. 3 is an explanatory diagram illustrating a concept when an image forming apparatus according to a second embodiment creates an augmented image to be used as an input image in an object recognition process using a machine learning model (Part 4). FIG. 1 is a diagram illustrating a concept when a temperature range is set as a parameter for a vehicle outdoors in summer. FIG. 5 is an explanatory diagram illustrating a concept when an image forming apparatus according to a second embodiment creates an augmented image to be used as an input image in an object recognition process using a machine learning model (Part 6). FIG. 7 is an explanatory diagram illustrating a concept when an image forming apparatus according to a second embodiment creates an augmented image to be used as an input image in an object recognition process using a machine learning model (Part 7). 10B is an explanatory diagram (part 8) showing the concept of an image forming apparatus according to a second embodiment creating an augmented image to be used as an input image in object recognition processing using a machine learning model. This diagram shows the concept when a temperature range is set as a parameter outdoors in winter for a vehicle. This diagram shows an infrared image indoors, in which a high-temperature device serving as an object is displayed in a region. This diagram shows a histogram of the infrared image of FIG. 10A. This diagram shows an augmented image in which the target temperature range of the infrared image of FIG. 10A is normalized to approximately 20°C to approximately 50°C. This diagram shows a histogram in which the target temperature range of FIG. 10B is normalized to approximately 18°C ​​to approximately 55°C. This diagram shows an infrared image outdoors, in which a high-temperature part of a vehicle serving as an object is displayed in a region. This diagram shows a histogram of the infrared image of FIG. 11A.11B shows an expanded image in which the target temperature range of the infrared image of FIG. 11A is normalized to approximately 30°C to approximately 42°C. FIG. 11B shows a histogram in which the target temperature range of the infrared image of FIG. 11B is normalized to approximately 30°C to approximately 44°C. FIG. 12A shows a histogram of the infrared image. FIG. 12A shows an expanded image in which the target temperature range of the infrared image of FIG. 12A is normalized to approximately 10°C to approximately 31°C. FIG. 12B shows a histogram in which the target temperature range of the infrared image of FIG. 12B is normalized to approximately 9°C to approximately 33°C. FIG. 12B shows an example in which the image expansion unit of the image forming apparatus according to the third embodiment generates an infrared image in a rectangular area with contrast added in an image area where a specific object is present. FIG. 12C shows an example in which the image expansion unit of the image forming apparatus according to the third embodiment generates an infrared image in a rectangular area with contrast added in an image area where a specific object is present. FIG. 12D shows an example in which the image expansion unit of the image forming apparatus according to the third embodiment generates an infrared image in a rectangular area with contrast added in an image area where a specific object is present. FIG. 10 is an explanatory diagram (part 3) showing an example in which the image expansion unit of the image forming device according to the third embodiment generates an infrared image within a rectangle with contrast in an image area where a specific object is present.

[0025] The following describes in detail embodiments of the present invention. Note that the embodiments described below are merely examples for realizing the present invention, and should be appropriately modified or changed depending on the configuration of the device to which the present invention is applied and various conditions, and the present invention is not limited to the following embodiments.

[0026] <First embodiment> [Overall configuration of image processing device] Fig. 1 is a functional block diagram showing the functions of an image processing device 100 according to the first embodiment. The image processing device 100 is a device that generates and processes infrared images, and is connected to an infrared camera 300. Note that the image processing device 100 of the first embodiment processes infrared images targeting an object to be learned, and generates a combination of an augmented image and ground truth data to be used for machine learning.

[0027] The infrared camera 300 is a camera for visualizing infrared rays emitted from an object. The infrared camera 300 detects the temperature of the object as an infrared detection signal and sends it to the image processing device 100.

[0028] The image processing device 100 comprises an image generation unit 10 , a standard table 20 , an image extension unit 40 , a correct answer data generation processing unit 31 (identification unit), an object-specific table 50 , and a learning data database 60 .

[0029] The image generating unit 10 generates an infrared image (hereinafter also referred to as a standard image) based on the infrared detection signal acquired from the infrared camera 300 while referring to the standard table 20. In other words, the image generating unit 10 can generate an infrared image from the infrared detection signal obtained by capturing an image of an object with the infrared camera 300.

[0030] The standard table 20 stores parameters that are referenced when the image generating unit 10 generates a standard image.

[0031] The correct data generation processing unit 31 performs object recognition processing on the generated infrared image to detect objects. The correct data generation processing unit 31 sets labels of the detected objects in the generated infrared image. In this case, the correct data generation processing unit 31 sets labels as correct data for the objects in the infrared image.

[0032] The correct data generation processing unit 31 (identification unit) can also set a rectangle indicating the area of ​​the detected object. In this case, the rectangle indicating the area of ​​the object is set to the image area where the specific object exists. The correct data generation processing unit 31 may set the label and the rectangle manually or automatically. This allows the correct data generation processing unit 31 to identify the image area where the specific object exists based on the infrared image generated by the image generation unit 10.

[0033] When used during machine learning, the image extension unit 40 of the first embodiment creates an extended image to be used as learning data for creating a machine learning model based on the information about the object in the object-specific table 50 from the infrared detection signal obtained by capturing an image of the object.

[0034] When creating an augmented image, the image extension unit 40 imparts a predetermined contrast to the range of infrared information in which a specific object may exist. Here, the image extension unit 40 imparts the predetermined contrast by allocating the range of infrared information in which a specific object may exist to a predetermined brightness range (e.g., from 32 to 233). If the specific object is a person, the range of infrared information in which a person may exist refers to infrared information corresponding to, for example, a range of 10°C to 40°C. If the specific object is a vehicle, the range of infrared information in which a vehicle may exist refers to infrared information corresponding to, for example, a range of 10°C to 70°C. This enables the object appearance level to be emphasized, thereby improving the object detection performance of the machine learning model.

[0035] Furthermore, when creating an augmented image, the image extension unit 40 may generate a plurality of augmented images, each with a different contrast applied to a range of infrared information in which a specific object may exist. Here, the image extension unit 40 assigns the range of infrared information in which a specific object may exist to a plurality of brightness ranges (e.g., from 32 to 223 and from 64 to 191), thereby applying a different contrast to each of the ranges.

[0036] This allows the image extension unit 40 to generate multiple extended images that gradually enhance the object. For example, if the object is a vehicle, image variations corresponding to daytime and nighttime, and sunny and stormy weather can be extended. By applying machine learning to these multiple extended images, the object detection performance can be improved.

[0037] Furthermore, when creating an extended image, the image extension unit 40 may add a predetermined contrast to an image region where a specific object is present. That is, the image extension unit 40 acquires the minimum and maximum values ​​of the detection signal in the image region where the specific object is present, and assigns the minimum value of the detection signal to 0 and the maximum value of the detection signal to 255. This enables data extension that targets the object.

[0038] Furthermore, the image extension unit 40 may set the outside of the image area where the specific object is present as a background image and impart different contrast to the background image. That is, the image extension unit 40 acquires the maximum and minimum values ​​of the detection signal of this background image, and assigns the minimum value of the detection signal to a luminance signal of 64 and the maximum value of the detection signal to a luminance signal of 191. Furthermore, the image extension unit 40 may add a predetermined noise when creating the extended image. This can improve the robustness of the machine learning model that has learned the extended image.

[0039] The object-specific table 50 stores parameters referenced by the image extension unit 40 when generating an augmented image. These parameters include, for example, information about the temperature of the object when generating the augmented image, as well as temperature information related to the climate and season. Specifically, the object-specific table 50 corresponds to a temperature range for processing to emphasize the object or to processing to enhance variation. The object can be a living thing such as a person or an animal, or it can also be a solid object with a specific temperature, such as a vehicle.

[0040] In this embodiment, multiple augmented images are generated, each with a different contrast, within the range of infrared information where a specific object may exist. This is to reduce the cost of data collection and increase the variation of situations that cannot actually be captured through image processing. This increases the robustness of the machine learning model, allowing for better inference performance.

[0041] In addition, in this embodiment, the input image is processed to make it easier for the machine learning model to detect the target by adding a predetermined contrast to a predetermined temperature range of the infrared image.

[0042] 2 is a functional block diagram showing the functions of an image processing device 100 according to a second embodiment. The image processing device 100 of the second embodiment differs from the image processing device 100 of the first embodiment in that the function of the image extension unit 40 is different, that an inference unit 70 is provided instead of the learning data database 60, and that a supervised data generation processing unit 31 outputs a rectangle indicating the image area of ​​an object and label data indicating information about the object.

[0043] The other configurations are the same as those of the first embodiment, and therefore will not be described. Note that the second embodiment is intended to create more optimal images and improve inference capabilities even in scenes where infrared light is difficult to detect.

[0044] The image extension unit 40 according to the second embodiment outputs the generated extended image to the inference unit 70. The inference unit 70 uses this extended image as an input image and executes object recognition processing using a machine learning model.

[0045] The image extension unit 40 adds a predetermined contrast to a range of infrared information in which a specific object may exist when generating an input image to be used in the object recognition process by the inference unit 70. Furthermore, the image extension unit 40 may add a predetermined contrast to an image region in which a specific object exists when generating an input image to be used in the object recognition process by the inference unit 70.

[0046] The supervised data generation processing unit 31 according to the second embodiment outputs label data as supervised data indicating an object. The supervised data generation processing unit 31 also outputs a rectangle indicating an image area of ​​the object.

[0047] The image extension unit 40 may perform processing to sharpen the image within a specific rectangle. Furthermore, when capturing an image with the infrared camera 300, the rectangle may be generated using the results of object recognition processing that uses a visible light image captured simultaneously with a visible light camera (not shown).

[0048] The image extension unit 40 references the object-specific table 50 and generates an input image with contrast based on the detection signal from the infrared camera 300 and the rectangle information from the correct data generation processing unit 31. This allows the image extension unit 40 to refine the input image.

[0049] Therefore, when generating an input image to be used in the object recognition process, the image extension unit 40 can impart a predetermined contrast to the range of infrared information in which a specific object may exist. Also, when generating an input image to be used in the object recognition process, the image extension unit 40 may impart a predetermined contrast to the image region in which a specific object exists. Furthermore, the image extension unit 40 may thin out information in a range of infrared information in which a specific object may not exist. Specifically, the image extension unit 40 may thin out information in a temperature range in which a specific object may not exist.

[0050] The inference unit 70 performs inference using the input image generated by the image extension unit 40. This allows the image extension unit 40 to create a more optimal image even in scenes where infrared light is difficult to detect due to its characteristics, thereby improving the inference ability of the inference unit 70.

[0051] 3 is a functional block diagram showing the functions of an image processing device 100 according to a third embodiment. The image processing device 100 according to the third embodiment is a device equipped with the machine learning functions according to the first embodiment and the inference functions according to the second embodiment, and is configured to include a supervised data generation processing unit 31, an image extension unit 40, a training data database 60, and an inference unit 70.

[0052] FIG. 4 is a block diagram showing the concept of performing either the learning process or the inference process, or both.

[0053] As shown in Fig. 4, by including an image processing device 150 and an edge device 200, it is possible to realize functions equivalent to those of the image processing device 100 shown in Fig. 3. The image processing device 150 is configured to include a training data database 60, a training process unit 110, and an inference model 120. The training process unit 110 includes a data extension unit 111.

[0054] The learning process unit 110 performs machine learning using learning data (for example, standard images and augmented images) stored in the learning data database 60.

[0055] The data expansion unit 111 expands the variation in the training data to improve the robustness of the inference model. That is, the data expansion unit 111 generates expanded images (variation images) from a standard image using the function shown in the image expansion unit 40 in Figure 1. This allows the learning process unit 110 to perform machine learning by artificially increasing the variation of infrared images.

[0056] The inference model 120 is the result of machine learning of the augmented image and the variation image in the learning process unit 110. By applying this inference model 120 to the inference process unit 210 of the edge device 200, it is possible to detect an object with improved performance of the machine learning model.

[0057] The edge device 200 is configured to include an inference process unit 210 and an inference model 120. The inference process unit 210 includes a data processing unit 211.

[0058] The inference process unit 210 performs inference on the input data 280 using the inference model 120 .

[0059] The data processing unit 211 processes the input image to detect the object with better performance. That is, the data processing unit 211 generates an input image with contrast from the input data 280 using the function shown in the image extension unit 40 in FIG. 2. This allows the inference process unit 210 to lower the difficulty of recognizing the object and improve the recognition performance. Therefore, the inference process unit 210 can improve the recognition result 290.

[0060] The inference process unit 210 may detect the movement of the object based on the position of the label or the rectangle in the input data 280. For example, the inference process unit 210 can output the movement direction when the object moves from right to left in the input data 280 as the authentication result 290.

[0061] [Hardware Configuration of Image Forming Apparatus] FIG. 5 is an explanatory diagram showing a schematic hardware configuration of the image processing apparatus 100 according to this embodiment.

[0062] The image processing device 100 is configured to include a CPU (Central Processing Unit) 101, a memory unit 102, a ROM (Read Only Memory) 103, a RAM (Random Access Memory) 104, an operation unit 105, a display unit 106, a communication unit 107, a learning data database 60, a learning process unit 110, and an inference model 120.

[0063] The CPU 101 reads and executes programs from the storage unit 102 or the ROM 103, thereby realizing the learning process unit 110 and the inference model 120 and controlling the overall operation of the image processing device 100. At this time, various data stored in the storage unit 102 is referenced. The storage unit 102 is configured from a large-capacity storage medium such as a semiconductor memory or a hard disk drive, and configures the standard table 20 and the object-specific table 50.

[0064] The CPU 101 transmits and receives various data to and from external devices connected to a communication network such as a LAN (Local Area Network) or a WAN (Wide Area Network) via the communication unit 107. The CPU 101 receives, for example, a detection signal transmitted from the infrared camera 300, and generates an infrared image using the image generation unit 10 based on this detection signal (see FIG. 1).

[0065] The communication unit 107 is configured by a communication control card such as a network interface card (NIC).

[0066] The operation unit 105 includes various operation keys such as a numeric keypad, a start key, etc. The operation unit 105 accepts various input operations by the user and outputs operation signals to the CPU 101.

[0067] The display unit 106 is, for example, a liquid crystal display, and displays characters, figures, images, and the like.

[0068] [Hardware Configuration of Edge Device] FIG. 6 is an explanatory diagram showing a schematic hardware configuration of the edge device 200 according to this embodiment.

[0069] The edge device 200 is configured to include a CPU 201 , a storage unit 202 , a ROM 203 , a RAM 204 , an inference process unit 210 , and an inference model 120 .

[0070] The CPU 201 reads and executes a program from the storage unit 202 or the ROM 203, thereby realizing the inference process unit 210 and the inference model 120 and controlling the overall operation of the edge device 200. The inference model 120 of the edge device 200 is the inference model 120 of the image processing device 100 shown in Figures 4 and 5, and references various data stored in the storage unit 202. The storage unit 202 is configured, for example, by a large-capacity storage medium such as a semiconductor memory or a hard disk drive, and forms an object-specific table 50.

[0071] <Operation in Each Embodiment> [During Machine Learning of Image Forming Apparatus (First Embodiment)] Next, a description will be given of the processing of the image extension unit 40 of the image processing device 100 according to the first embodiment. Figures 7A to 7H are explanatory diagrams showing the concept of the image processing device 100 according to the first embodiment creating an extended image to be used as learning data for creating a machine learning model from an infrared image of an object captured by the infrared camera 300, based on information about the object.

[0072] The image extension unit 40 of the image processing device 100 refers to the object-specific table 50, changes the target range including the priority range, and generates eight extended images from the predetermined infrared detection signal. That is, the image extension unit 40 changes the dynamic range to generate extended images in which the range in which the target can be expressed in black and white is changed.

[0073] For example, the image extension unit 40 sets a temperature range of 31°C to 41°C, which corresponds to body temperature, as the priority range 79 for a human being, which is a warm-blooded animal, from the object-specific table 50. The image extension unit 40 then generates eight extension images 71 to 78 that include the temperature range of 31°C to 41°C as the priority range.

[0074] 7A shows an extended image 71 generated by setting the temperature parameter to a range of 5°C to 40°C. The image extension unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 5°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 40°C or higher. The image extension unit 40 then assigns proportional brightness values ​​to pixels whose infrared detection signals are between 5°C and 40°C.

[0075] 7B shows an expanded image 72 generated by setting the temperature parameter to a range of 15°C to 40°C. The image expansion unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 15°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 40°C or higher. The image expansion unit 40 then assigns proportional brightness values ​​to pixels whose infrared detection signals are between 15°C and 40°C.

[0076] 7C shows an extended image 73 generated by setting the temperature parameter to a range of 20°C to 40°C. The image extension unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 20°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 40°C or higher. The image extension unit 40 then assigns proportional brightness values ​​to pixels whose infrared detection signals are between 20°C and 40°C.

[0077] 7D shows an expanded image 74 generated by setting the temperature parameter to a range of 28°C to 40°C. The image expansion unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 28°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 40°C or higher. The image expansion unit 40 then assigns proportional brightness values ​​to pixels whose infrared detection signals are between 28°C and 40°C.

[0078] 7E shows an expanded image 75 generated by setting the temperature parameter to a range of 15°C to 50°C. The image expansion unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 15°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 50°C or higher. The image expansion unit 40 also assigns proportional brightness values ​​to pixels whose infrared detection signals are between 15°C and 50°C.

[0079] 7F shows an extended image 76 generated by setting the temperature parameter to a range of 20°C to 50°C. The image extension unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 20°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 50°C or higher. The image extension unit 40 also assigns proportional brightness values ​​to pixels whose infrared detection signals are between 20°C and 50°C.

[0080] 7G shows an expanded image 77 generated by setting the temperature parameter to a range of 25°C to 50°C. The image expansion unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 25°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 50°C or higher. The image expansion unit 40 also assigns proportional brightness values ​​to pixels whose infrared detection signals are between 25°C and 50°C.

[0081] 7H shows an expanded image 78 generated by setting the temperature parameter to a range of 25°C to 40°C. The image expansion unit 40 assigns a brightness value of 0 to pixels whose infrared detection signals are 25°C or lower, and assigns a brightness value of 255 to pixels whose infrared detection signals are 40°C or higher. The image expansion unit 40 then assigns proportional brightness values ​​to pixels whose infrared detection signals are between 25°C and 40°C.

[0082] FIG. 7I illustrates the concept of the temperature parameters of FIGS. 7A through 7H when displayed over a range of temperatures.

[0083] 7A to 7H all include a key range 79, and augmented images (variation images) with diverse appearances are generated within the visible dynamic range without compromising the temperature range in which the object appears. Furthermore, in Figures 7A to 7H, processing is performed on the entire image range, so augmented images 71 to 78 are generated in which processing has been performed on both the foreground and background.

[0084] [During inference by image forming device (second embodiment)] Next, a description will be given of the processing of the image extension unit 40 of the image processing device 100 according to the second embodiment. Figures 8A to 8E and 9A to 9E are explanatory diagrams showing the concept of the image processing device 100 according to the second embodiment creating an extended image to be used as an input image in object recognition processing using a machine learning model.

[0085] The image extension unit 40 of the image processing device 100 refers to the object-specific table 50, changes the target range including the priority range 89, and generates an input image with improved visibility of the target. That is, the image extension unit 40 changes the dynamic range to generate input images (extended images 81 to 84, 91 to 94) according to the situation of the operating environment.

[0086] For example, the image extension unit 40 adjusts the dynamic range by setting a temperature range of approximately 35°C to 53°C, which corresponds to the temperature of a vehicle, as the priority range 89 from the object-specific table 50. The image extension unit 40 adjusts the dynamic range as the target range depending on the installation environment (outdoor or indoor) or the seasonal environment.

[0087] FIG. 8A shows an augmented image 81 generated outdoors in summer with the temperature parameter set to a range of 0°C to 60°C. FIG. 8B shows an augmented image 82 generated outdoors in summer with the temperature parameter set to a range of 10°C to 60°C. FIG. 8C shows an augmented image 83 generated outdoors in summer with the temperature parameter set to a range of 15°C to 60°C. FIG. 8D shows an augmented image 84 generated outdoors in summer with the temperature parameter set to a range of 20°C to 60°C. FIG. 8E shows a concept of a vehicle outdoors in summer with the temperature parameters of FIGS. 8A to 8D set within the temperature ranges.

[0088] 8E, the extended images 81 to 84 include a priority range 89, and the visible dynamic range is adjusted. For example, in the case of outdoor scenes in hot weather, the low temperature range is increased to cover the high temperature range, with the upper limit being 60°C. The priority range 89 is also adjusted to approximately 35°C to 52°C.

[0089] In this case, the extended image 81 is not suitable as an input image because of its weak contrast, whereas the extended image 84 is suitable as an input image because of its strong contrast and emphasized edges.

[0090] FIG. 9A shows an augmented image 91 generated outdoors in winter with the temperature parameter set to a range of 0°C to 50°C. FIG. 9B shows an augmented image 92 generated outdoors in winter with the temperature parameter set to a range of 10°C to 50°C. FIG. 9C shows an augmented image 93 generated outdoors in winter with the temperature parameter set to a range of 15°C to 50°C. FIG. 9D shows an augmented image 94 generated outdoors in winter with the temperature parameter set to a range of 20°C to 50°C. FIG. 9E shows a concept of a vehicle outdoors in winter with the temperature parameters of FIGS. 9A to 9D set within a temperature range.

[0091] 9E, the extended images 91 to 94 include a priority range 89, and the visible dynamic range is adjusted. For example, in the case of outdoor use in cold weather, the high temperature side is lowered and the upper limit is set to 50°C to cover the low temperature range. The priority range 89 is also adjusted to approximately 30°C to 43°C.

[0092] In this case, the expanded image 94 is not suitable as an input image because of its weak contrast, whereas the expanded image 92 is suitable as an input image because of its strong contrast and emphasized edges.

[0093] [During machine learning / inference of image forming device (third embodiment)] Next, a description will be given of the processing of the image extension unit 40 of the image processing device 100 according to the third embodiment. Figures 10A to 10D are explanatory diagrams showing an example in which the image processing device 100 according to the third embodiment adjusts the target range using percentiles and detects a target indoors when generating an extended image.

[0094] Percentile is a method of normalizing the distribution of measured values ​​by sorting them from smallest to largest and displaying them as a percentage.

[0095] The image extension unit 40 of the third embodiment rearranges the infrared detection signals for each pixel and sets the target range between percentiles X and Y. Here, for example, percentile 10% to 90% means that the target range is from the smallest number, 10% to 90%, in the distribution of measurement values.

[0096] The image extension unit 40 of the image processing device 100 uses a histogram of the infrared detection signal to normalize the target range to percentiles X to Y. This removes outliers below X% and above Y%, improving visibility.

[0097] Fig. 10A shows an infrared image 450 in which a high-temperature device as an object is displayed in an indoor area 400. The infrared image 450 in Fig. 10A shows a temperature range from approximately 20°C to approximately 140°C.

[0098] 10B shows a histogram 453 of the infrared image 450 of FIG. 10A. The horizontal axis represents temperature [°C], and the vertical axis represents frequency [freq]. In the histogram 453 of FIG. 10B, the frequency of the detection signal increases in the range from approximately 18 [°C] to approximately 30 [°C]. Therefore, the image extension unit 40 can improve the visibility of the infrared image by adjusting the infrared image 450 using the range of percentiles X to Y as the target range.

[0099] Fig. 10C shows an augmented image 451 in which the target temperature range of the infrared image 450 in Fig. 10A has been normalized to approximately 20°C to approximately 55°C. In the augmented image 451 in Fig. 10C, specific objects are displayed in areas 400, 401, and 402. Area 400 displays a person carrying high-temperature equipment, area 401 displays a person carrying luggage, and area 402 displays a person standing in front of a shower room.

[0100] FIG. 10D shows a histogram 454 in which the target range of temperatures of FIG. 10B has been normalized to approximately 18° C. to approximately 55° C.

[0101] In this way, the image extension unit 40 creates a histogram of the detection signal and adjusts the target temperature range using percentiles X to Y. This makes it possible to remove outliers below X% and above Y%, making it easy to visually recognize information about the target object that would be buried by simple normalization.

[0102] In this embodiment, the values ​​of percentiles X to Y are the parameter range. The image extension unit 40 can increase the variety of extended images used in machine learning by setting the parameter range randomly or in a brute-force manner. Furthermore, during inference, the image extension unit 40 can set a low upper limit value Y of the parameter range, making it easier to detect objects with relatively low temperatures, for example, in an environment where high-temperature equipment is installed. In this way, the image extension unit 40 can adjust the temperature range of the target range in accordance with the parameter range.

[0103] Figures 11A to 11D are explanatory diagrams showing an example in which the image processing device 100 according to the third embodiment adjusts the target range using percentiles to detect targets outdoors when generating an extended image.

[0104] Similarly, outdoors, the image enhancement unit 40 of the image processing device 100 uses a histogram of the infrared detection signal to normalize the target range to percentiles X to Y. This removes outliers below X% and above Y%, improving visibility.

[0105] 11A shows an infrared image 460 taken outdoors, in which a high-temperature part of a vehicle, which is an object, is displayed in area 410. The infrared image 460 in FIG. 11A shows a temperature range from approximately 30°C to approximately 100°C.

[0106] 11B shows a histogram 463 of the infrared image of FIG. 11A. The horizontal axis represents temperature [°C], and the vertical axis represents frequency [freq]. In the histogram 463 of FIG. 11B, the frequency of the detection signal increases in the range from approximately 30 [°C] to approximately 40 [°C]. Therefore, the image extension unit 40 can improve the visibility of the infrared image by adjusting the infrared image 460 using the range of percentiles X to Y as the target range.

[0107] Fig. 11C shows an augmented image 461 in which the target temperature range of the infrared image 460 in Fig. 11A has been normalized to approximately 30°C to approximately 44°C. In the augmented image 461 in Fig. 11C, a specific target is displayed in an area 410. The engine part of a garbage truck is displayed in the area 410.

[0108] FIG. 11D shows a histogram 464 in which the target range of temperatures of FIG. 11B has been normalized to approximately 30° C. to approximately 44° C.

[0109] In this way, the image extension unit 40 creates a histogram of the detection signal, even outdoors, and adjusts the target temperature range using percentiles X to Y. This makes it possible to remove outliers below X% and above Y%, making it easy to view information about the target object that would be buried with simple normalization.

[0110] Figures 12A to 12D are explanatory diagrams showing an example in which the image processing device 100 according to the third embodiment adjusts the target range using percentiles to detect targets when generating an extended image in cold outdoor conditions.

[0111] Similarly, even in cold outdoor conditions, the image enhancement unit 40 of the image processing device 100 uses a histogram of the infrared detection signal to normalize it within the range of percentiles X to Y. This removes outliers below X% and above Y%, improving visibility.

[0112] 12A shows an infrared image 470 taken outdoors in cold weather, with a hot spot on a vehicle being the object displayed in area 420. The infrared image 470 in FIG. 12A shows a temperature range of approximately 10°C to approximately 90°C.

[0113] Fig. 12B shows a histogram 473 of the infrared image of Fig. 12A. The horizontal axis represents temperature [°C], and the vertical axis represents frequency [freq]. In the histogram 473 of Fig. 12B, the frequency of the detection signal increases in the range from approximately 10 [°C] to approximately 25 [°C].

[0114] Fig. 12C shows an expanded image 471 in which the temperature range of the infrared image 470 in Fig. 12A has been normalized to a range of approximately 10°C to approximately 35°C. In the expanded image 471 in Fig. 12C, specific objects are visible in regions 420, 421, and 422. A traveling vehicle is visible in region 420, an oncoming vehicle is visible in region 421, and a pedestrian is visible in region 422.

[0115] FIG. 12D shows a histogram 474 in which the target range of temperatures of FIG. 12B has been normalized to approximately 9° C. to approximately 33° C.

[0116] In this way, the image extension unit 40 uses the histogram of the infrared detection signal to change and normalize the target range of the temperature of the infrared image 470 in Fig. 12A within the range of percentiles X to Y. Similarly, even in the case of cold outdoors, the image extension unit 40 can remove outliers below X% and above Y% by adjusting the target range within the range of percentiles X to Y, making it possible to easily view information about the object that would be buried with simple normalization.

[0117] [During machine learning of the image forming device (third embodiment)] When creating an infrared image, the image extension unit 40 of the image processing device 100 according to the third embodiment adds a predetermined contrast to an image region where a specific object exists. In this case, the image extension unit 40 may use the area outside the image region as a background image and add different contrasts to each of the background images to generate multiple infrared images.

[0118] Figures 13A to 13C are explanatory diagrams showing an example in which the image expansion unit 40 of the image processing device 100 according to the third embodiment generates an infrared image within a rectangle with a predetermined contrast added to an image area in which a specific object is present.

[0119] 13A, the image extension unit 40 sets correct rectangles 430 to 433 of the object in the infrared image 480. Because the image processing device 100 includes the correct data generation processing unit 31, the image extension unit 40 can set rectangles as correct data (referred to as correct rectangles) using the correct data generation processing unit 31. Note that the target temperature range of the infrared image 480 is set to approximately -10°C to approximately 100°C.

[0120] The image extension unit 40 refers to the object-specific table 50 and generates an extended image 481 (infrared image) in a rectangle, in which a predetermined contrast is added to an image area where a specific object exists.

[0121] 13B, the image extension unit 40 adjusts the target temperature range of the infrared image 480 from approximately −12.5° C. to approximately 8° C. so that the extended image 481 of the important range in the correct rectangles 430 to 433 is displayed appropriately. That is, as shown in FIG. 13B, the image extension unit 40 appropriately sets the target temperature range, thereby generating the extended image 481 with a predetermined contrast in the correct rectangles 430 to 433.

[0122] The image extension unit 40 sets a target temperature range or a temperature range based on a predetermined percentile range, and generates a contrasted extended image 481. By referencing the object-specific table 50, the image extension unit 40 can generate an extended image 481 in which a temperature range corresponding to the target object in the correct rectangles 430 to 433 is set.

[0123] 13C, the image extension unit 40 generates an extended image by combining an extended image 482, in which various temperature ranges are set as a background image, with an extended image 481 in which contrast is added to the correct rectangles 430 to 433. The image extension unit 40 can set the outside of the rectangle as a background image and generate an extended image 482 in which multiple temperature ranges are set for the background image.

[0124] This allows the image extension unit 40 to combine the extended image 481 in the contrasted answer rectangles 430 to 433 with the extended image 482 as a background image with various variations, thereby generating various extended images to be used in machine learning.

[0125] <Functions and Effects> As described above, the image processing device 100 according to the first embodiment is configured to include the image extension unit 40. The image extension unit 40 creates an extended image from an infrared image of an object, based on information about the object in the standard table 20, to be used as learning data for creating a machine learning model.

[0126] As a result, the image processing device 100 according to the first embodiment can use an infrared image of an object to create an augmented image to be used as training data for creating a machine learning model, based on information about the object. Thus, the image processing device 100 according to the first embodiment can increase the variety of training data.

[0127] Furthermore, the image processing device 100 according to the first embodiment may impart a predetermined contrast to a range of infrared information in which a specific object may exist when creating an extended image using the image extension unit 40. Furthermore, the image processing device 100 according to the first embodiment may generate a plurality of infrared images in which different contrasts are imparted to ranges of infrared information in which a specific object may exist when creating an infrared image using the image extension unit 40.

[0128] The image processing device 100 according to the second embodiment is also configured to include an image extension unit 40. The image extension unit 40 creates an extended image to be used as an input image in object recognition processing using a machine learning model.

[0129] As a result, the image processing device 100 according to the second embodiment can create an augmented image to be used as an input image in the object recognition process using a machine learning model by referring to the object-specific table 50. Therefore, the image processing device 100 according to the second embodiment can refine the input image to be used in inference, thereby improving the performance of detecting objects in infrared images.

[0130] Furthermore, the image processing device 100 according to the second embodiment may impart a predetermined contrast to a range of infrared information in which a specific object may exist when generating an input image to be used in the object recognition process by the image extension unit 40. Furthermore, the image processing device 100 according to the second embodiment may impart a contrast to an image region in which a specific object exists when generating an input image to be used in the object recognition process by the image extension unit 40.

[0131] The image processing device 100 according to the third embodiment is configured to include an image extension unit 40, a training data database 60, and an inference unit .

[0132] The image processing apparatus 100 according to the third embodiment is an apparatus that combines the image processing apparatus 100 according to the first embodiment and the image processing apparatus 100 according to the second embodiment.

[0133] As a result, the image processing device 100 according to the third embodiment may impart a predetermined contrast to an image region where a specific object exists when creating an infrared image using the image extension unit 40. Furthermore, the image processing device 100 according to the third embodiment may use the outside of the image region as a background image when creating an infrared image using the image extension unit 40, and may impart different contrasts to the background image.

[0134] (Modifications) The present invention is not limited to the above-described embodiment, and modifications can be made without departing from the spirit of the present invention, for example, the following (a) to (e) are possible.

[0135] (a) The object in the first to third embodiments may be a solid having a specific temperature. For example, this may be a human, a pet, a warm-blooded animal such as livestock, or a vehicle. (b) The object information in the first to third embodiments may be a temperature range in which the specific object may exist. (c) The object information in the first to third embodiments may be an image area in which the specific object exists.

[0136] (d) The image processing device 100 according to the first to third embodiments may further include a learning process unit 110 that performs machine learning using learning data.

[0137] (e) The edge device 200 may include an inference model 120 that has been machine-learned by an image processing device 100 that includes an image extension unit 40 that creates an extended image from an infrared image of an object based on information about the object, and the extended image is used as learning data for creating a machine learning model or is used as an input image in the object recognition process using the machine learning model, and a learning process unit 110 that performs machine learning using the learning data, and an inference process unit 210 that applies the extended image created in the image extension unit 40 to the inference model 120 to infer the object.

[0138] In this case, the edge device 200 according to the third embodiment includes an inference process unit 210 and an inference model 120. The inference process unit 210 includes a data processing unit 211, and generates an input image with contrast from input data 280 using the function shown in the image extension unit 40 in Fig. 2. As a result, the inference process unit 210 performs inference on the input image with contrast using the inference model 120.

[0139] The inference model 120 is a model created by machine learning using an image processing device 100 that includes an image extension unit 40 that creates an extended image from an infrared image of an object based on information about the object, either to be used as learning data for creating a machine learning model or to be used as an input image in the object recognition process using a machine learning model, and a learning process unit 110 that performs machine learning using the learning data.

[0140] As a result, the edge device 200 according to the third embodiment is equipped with the inference process unit 210 and the inference model 120, thereby enabling improvement in the performance of detecting objects in infrared images.

[0141] REFERENCE SIGNS LIST 10 Image generation unit 20 Standard table 31 Correct answer data generation processing unit (identification unit) 40 Image expansion unit 50 Object-specific table 60 Learning data database 70 Inference unit 100, 150 Image processing device 110 Learning process unit 111 Data expansion unit 120 Inference model 200 Edge device 210 Inference process unit 211 Data processing unit 280 Input data 290 Recognition result 300 Infrared camera

Claims

1. An image processing device comprising: an image expansion unit that creates an expanded image based on information about an object from an infrared detection signal obtained by capturing an image of the object, and that creates an expanded image to be used as learning data for creating a machine learning model, or to be used as an input image in the recognition process of the object by the machine learning model.

2. The image processing device according to claim 1, wherein the object is a solid having a specific temperature.

3. The image processing device according to claim 1, wherein the information about the object is a temperature range in which a particular object may exist.

4. The image processing device according to claim 1, wherein the information on the object is an image area in which a specific object exists.

5. The image processing device according to claim 1, wherein the image extension unit, when creating the extended image, imparts a predetermined contrast to a range of infrared information in which a specific object may exist.

6. The image processing device according to claim 1, wherein the image extension unit generates a plurality of extended images, each of which has a different contrast applied to a range of infrared information in which a specific object may be present.

7. An image processing device as described in claim 1, further comprising: an image generation unit that generates an infrared image from an infrared detection signal obtained by capturing an image of the object; and an identification unit that identifies an image area in which a specific object exists based on the infrared image, wherein the image expansion unit imparts a predetermined contrast to the image area in which the specific object exists when creating the expanded image.

8. The image processing device according to claim 7, wherein when creating the extended image, the image extension section uses the outside of the image area as a background image and imparts different contrast to each of the background images.

9. The image processing device according to claim 1, further comprising a learning process unit that performs machine learning using the learning data.

10. The image processing device according to claim 1, wherein the image extension unit adds a predetermined contrast to a range of infrared information in which a specific object may exist when generating an input image to be used in the process of recognizing the object.

11. The image processing device according to claim 7, wherein the image extension unit adds a predetermined contrast to an image area in which a specific object exists when generating an input image to be used in the process of recognizing the object.

12. An edge device comprising: an image expansion unit that creates an extended image based on information about an object from an infrared detection signal that captures an image of the object, and the extended image is used as learning data for creating a machine learning model or as an input image in the object recognition process by the machine learning model; and an inference model created by machine learning using the learning data by an image processing device comprising: an image expansion unit that creates an extended image based on information about the object from an infrared detection signal that captures an image of the object, and the extended image is used as learning data for creating a machine learning model or as an input image in the object recognition process by the machine learning model; and an inference process unit that applies the extended image created by the image expansion unit to the inference model to infer the object.

13. An image generation method including a step of creating an augmented image based on information about an object from an infrared detection signal obtained by capturing an image of the object, the augmented image being used as learning data for creating a machine learning model, or as an input image in the recognition process of the object by the machine learning model.

14. An image generation program for causing a computer to execute the steps of: creating an augmented image based on information about an object from an infrared detection signal obtained by capturing an image of the object, and using the augmented image as learning data for creating a machine learning model, or as an input image in the recognition process of the object by the machine learning model.

Citation Information

Patent Citations

  • Far-infrared image processing device, far-infrared monitoring device including the same, and far-infrared image processing program

    JP2020107097A

  • Object detection apparatus

    JP2021163222A

  • Machine learning device, machine learning method, and machine learning program

    WO2020217537A1

  • Target region detection device, target region detection method, and target region detection program

    WO2021171411A1