Information processing method and information processing apparatus

Abstract

To provide an information processing method for improving the accuracy of a technique for determining the state of an object by using an image obtained by imaging the object.SOLUTION: Acquiring a thermo image IM1 obtained by imaging the subject S with the infrared light camera 21, acquiring an RGB image IM2 obtained by imaging the subject S with the RGB camera 22, displaying, on a screen, a synthetic image IM1 obtained by superimposing the thermo image IM2 and the RGB image IM0 on each other, acquiring distances from the infrared light camera 21 and the RGB camera 22 to the subject S, and correcting, on the basis of the acquired distances, a positional deviation between the thermo image IM1 and the RGB image IM2 when the images are superimposed on each other.SELECTED DRAWING: Figure 7A

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method and an information processing device. [Background technology]

[0002] Conventionally, there are known techniques for determining the condition of an object using an image of the object. For example, there is known a technique for determining tire damage. For example, Patent Document 1 discloses a tire damage detection system that detects the size of a damaged portion of a tire based on the diameter of a rim wheel, based on image data of the tire mounted on the rim wheel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2019-202729 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a demand for further improvement in the accuracy of technology for determining the condition of an object using an image of the object. For example, there is a demand for further improvement in the accuracy of technology for determining tire failure. For example, in the case of mining vehicles used at mining sites, if a sudden tire failure occurs, productivity decreases due to the need to transport the mining vehicle and replace the tire. In addition, regular tire inspections by workers to prevent sudden failures are a cost burden. Therefore, there is a demand for more accurate tire failure determination.

[0005] In view of the above circumstances, an object of the present disclosure is to provide an information processing method and an information processing device that improve the accuracy of a technique for determining the state of an object using an image of the object. [Means for solving the problem]

[0006] [1] An information processing method according to one embodiment of the present disclosure is an information processing method executed by a computer, and includes: acquiring a thermographic image of an object captured with an infrared camera; acquiring an RGB image of the object captured with an RGB camera; displaying a composite image of the thermographic image and the RGB image on a screen; acquiring distance data from the infrared camera and the RGB camera to the object; and correcting, based on the acquired distance data, any misalignment between the thermographic image and the RGB image when the images are superimposed on each other. According to an information processing method according to an embodiment of the present disclosure, it is possible to improve the accuracy of a technique for determining the state of an object using an image of the object.

[0007] [2] An information processing method according to one embodiment of the present disclosure is the information processing method described in [1] above, wherein acquiring the distance data may include acquiring the distance data using an AR function of an information processing device to which an imaging device having the infrared camera and the RGB camera is attached. According to the information processing method having such a configuration, the distance from the imaging device having an infrared camera and an RGB camera to the target object can be measured with high accuracy using the functions of the information processing device itself.

[0008] [3] An information processing method according to one embodiment of the present disclosure is the information processing method described in [2] above, wherein acquiring the distance data may include acquiring the distance data at a time interval longer than the update time interval of the distance data based on the AR function. According to the information processing method having such a configuration, it is possible to reduce the load of the calculation processing related to the acquired distance data.

[0009] [4] An information processing method according to one embodiment of the present disclosure is an information processing method described in any one of [1] to [3] above, wherein acquiring the distance data may include acquiring the distance data for multiple locations on the object and calculating statistical values ​​of the distance data for the multiple locations. According to the information processing method having such a configuration, it is possible to measure the distance from the imaging device to the object with higher accuracy.

[0010] [5] An information processing method according to one embodiment of the present disclosure is an information processing method described in any one of [1] to [4] above, wherein acquiring the distance data may include displaying an acquisition range of the distance data on the screen. According to the information processing method having such a configuration, a user viewing the screen can easily understand for which part of the object displayed on the screen distance data is being acquired.

[0011] [6] An information processing method according to one embodiment of the present disclosure is the information processing method described in [5] above, wherein acquiring the distance data may include changing at least one of the position and size of the acquired range on the screen based on a user's operation on the screen. According to the information processing method having such a configuration, it is possible to appropriately display on the screen the range for obtaining distance data according to at least one of the position and size desired by the user.

[0012] [7] An information processing method according to one embodiment of the present disclosure is the information processing method described in any one of [1] to [6] above, wherein the object may be a rubber product including a tire or a crawler. According to the information processing method having such a configuration, it is possible to generate and display a composite image in which the thermo image and the RGB image are aligned with high precision for a rubber product as an object.

[0013] [8] An information processing device according to one embodiment of the present disclosure includes a control unit, which acquires a thermal image of an object captured by an infrared camera, acquires an RGB image of the object captured by an RGB camera, displays a composite image on a screen in which the thermal image and the RGB image are superimposed on each other, acquires distance data from the infrared camera and the RGB camera to the object, and corrects misalignment between the images when the thermal image and the RGB image are superimposed on each other based on the acquired distance data. According to an information processing device according to an embodiment of the present disclosure, it is possible to improve the accuracy of a technique for determining the state of an object using an image of the object. [Effects of the Invention]

[0014] According to the present disclosure, it is possible to provide an information processing method and an information processing device that improve the accuracy of a technique for determining the state of an object using an image of the object. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a block diagram illustrating an example of a schematic configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing an example of a schematic configuration of the information processing device of FIG. 1. [Figure 3] 2 is a block diagram showing an example of a schematic configuration of the imaging device of FIG. 1. FIG. [Figure 4] 2 is a block diagram showing an example of a schematic configuration of a server in FIG. 1. FIG. [Figure 5] 3 is a flowchart showing an example of the operation of the information processing device in FIG. 2. [Figure 6A] FIG. 3 is a first schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. [Figure 6B] 10 is a second schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. FIG. [Figure 7A] 3 is a third schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. FIG. [Figure 7B]4 is a fourth schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0016] An information processing system 1 according to an embodiment of the present disclosure will be described below with reference to the drawings. Components and parts that are common to each drawing are denoted by the same reference numerals. Please note that the drawings are schematic, and the dimensional ratios and the like may differ from those of the actual drawings.

[0017] (Configuration of Information Processing System 1) An overview of an information processing system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of a schematic configuration of the information processing system 1 according to an embodiment of the present disclosure. As shown in Fig. 1, the information processing system 1 includes an information processing device 10, an imaging device 20, a server 30, and a measuring device 40.

[0018] For ease of explanation, FIG. 1 illustrates only one each of the information processing device 10, imaging device 20, server 30, and measuring device 40, but the number of each device included in the information processing system 1 is not limited to one. The information processing system 1 may include, for example, multiple information processing devices 10. The information processing system 1 may include, for example, multiple imaging devices 20. The information processing system 1 may include, for example, multiple servers 30. The information processing system 1 may include, for example, multiple measuring devices 40.

[0019] The information processing device 10, the server 30, and the measuring device 40 are each communicatively connected to, for example, a network 50. The imaging device 20 may be communicatively connected to the network 50 indirectly via the information processing device 10, or may be communicatively connected to the network 50 directly without via the information processing device 10.

[0020] The information processing device 10 and the imaging device 20 are connected to each other via, for example, a connector. The information processing device 10 is integrally connected to the imaging device 20 so that the distance between the imaging device 20 and an object imaged by the imaging device 20 can be accurately measured using the functions of the information processing device 10. However, the information processing device 10 is not limited to this, and does not need to be integrally connected to the imaging device 20 as long as the information processing device 10 can acquire distance data from the imaging device 20 to the object.

[0021] In the present disclosure, the term "object" includes, for example, any object for which the condition of a target portion of the object is to be estimated. For example, the object is a rubber product including a tire, a crawler, and the like. The "target portion of the object" includes, for example, a damaged portion of the object. The "condition of the target portion" includes, for example, the depth of the damaged portion of the object. As an example, the information processing device 10 performs image processing, which will be described later, as preprocessing in order to improve the estimation accuracy when the depth of the damaged portion of the tire is estimated by the server 30 or the like in tire failure determination.

[0022] The information processing device 10 includes any general-purpose electronic device including, for example, a smartphone, a tablet PC (Personal Computer), a wearable device such as a smart watch, and a PC. The information processing device 10 is not limited to these, and may include other electronic devices dedicated to the information processing system 1. As an example, the information processing device 10 functions as a terminal device for the server 30. The information processing device 10 is used by a user who wants the server 30 to estimate the state of a target location of an object.

[0023] The imaging device 20 is configured with a computer including multiple cameras. The imaging device 20 can, for example, acquire both thermal images and RGB images in parallel. The images acquired by the imaging device 20 may be still images such as photographs, or may be videos. The imaging device 20 generates thermal images and RGB images of objects such as tires, and transmits them to the information processing device 10. In one embodiment, the thermal image of a tire captures at least a portion of the tire. The RGB image of a tire captures at least a portion of the tire.

[0024] As described above, the imaging device 20 is, for example, connected to the information processing device 10 and is portable by a user along with the information processing device 10. However, the imaging device 20 is not limited to this. When the information processing device 10 and the imaging device 20 are not integrally connected to each other, the imaging device 20 may be, for example, a fixed device installed on a vehicle's travel route at a mining site. This allows the imaging device 20 to capture images of the vehicle traveling on the travel route, which is less likely to reduce the vehicle's operating rate and productivity at the mine. Alternatively, the imaging device 20 may be a device mounted on a drone.

[0025] The server 30 is configured with one or more computers. In one embodiment, the server 30 is described as being configured with one computer. However, the server 30 may be configured with multiple computers, such as in a cloud computing system. In the present disclosure, the server 30 functions as, for example, a tire malfunction determination device.

[0026] The measurement device 40 is configured with a computer including one or more sensors. The sensors may be, but are not limited to, a digital tachograph, a tire pressure monitoring system (TPMS), an electronic control unit (ECU), or a car navigation device. The measurement device 40 acquires time-series data related to tires mounted on a vehicle and transmits the data to the server 30. Therefore, the measurement device 40 may be installed on the vehicle or tires.

[0027] The time-series data on the tires mounted on a vehicle includes tire-related measurement values ​​and the dates and times of the measurements. For example, if the measurement device 40 includes a TPMS installed on the tires, the tire-related measurement values ​​may include tire condition information on the tires, such as tire internal pressure (air pressure), tire cavity temperature, and thermal history. The tire thermal history is the history of heat applied to the tire as the tire is used. The tire thermal history is used to evaluate how much energy has been applied to the tire since it was first used. Generally, the greater the thermal history, the more severe the tire's deterioration. The thermal history can be calculated, for example, by applying the tire cavity temperature to the Arrhenius equation. Furthermore, for example, if the measurement device 40 includes a digital tachograph installed on the vehicle, the tire-related measurement values ​​may include vehicle driving information, such as the vehicle's driving time, driving distance, speed, acceleration, and tire rotation count.

[0028] The network 50 is any communication network that allows mutual communication among the information processing device 10, the imaging device 20, the server 30, and the measuring device 40. The network 50 in one embodiment may be, for example, the Internet, a mobile communication network, a LAN (Local Area Network), or a combination thereof.

[0029] The information processing system 1 may function as a tire malfunction determination system centered around a server 30 serving as a tire malfunction determination device. In this case, the information processing system 1 is used to determine malfunctions in one or more tires. In the information processing system 1, the server 30 acquires from the information processing device 10 a composite image in which, for example, a thermographic image and an RGB image of the tire captured by the imaging device 20 are superimposed on each other. Based on the composite image, the server 30 acquires the surface temperature of at least one damaged portion of the tire.

[0030] The server 30 acquires time-series data of the tire cavity temperature from the measuring device 40. The server 30 estimates the depth of at least one damaged portion of the tire based on the surface temperature of at least one damaged portion of the tire in the composite image and the cavity temperature at the time the thermoimage and RGB image used in the composite image were acquired. In this way, the information processing system 1 can more accurately estimate the depth of the damaged portion by using the tire cavity temperature in addition to the surface temperature of at least one damaged portion of the tire in the composite image.

[0031] In the present disclosure, the tire is not particularly limited, but may be an OR (Off The Road) tire mounted on a mining vehicle such as a transport vehicle, a construction vehicle, a work vehicle, or a heavy machinery vehicle used at a mining site, etc. However, the tire may be a tire other than an OR tire.

[0032] In the present disclosure, the vehicle is, for example, a mining vehicle used at a mining site, etc. However, the vehicle is not limited to the above-mentioned mining vehicle, and may be any vehicle capable of being fitted with tires, such as a transport vehicle, a construction vehicle, a work vehicle, a heavy machinery vehicle, a bus, a passenger car, a motorcycle, a bicycle, or an airplane.

[0033] Fig. 2 is a block diagram showing an example of a schematic configuration of the information processing device 10 of Fig. 1. An example of the configuration of the information processing device 10 will be mainly described with reference to Fig. 2. The information processing device 10 has a communication unit 11, a storage unit 12, an acquisition unit 13, an input unit 14, an output unit 15, and a control unit 16. In the information processing device 10, the communication unit 11, the storage unit 12, the acquisition unit 13, the input unit 14, the output unit 15, and the control unit 16 are connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0034] The communication unit 11 includes a communication module for connecting to the network 50. The communication module is a communication module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation). The communication module may be a communication module compatible with standards such as wired LAN and wireless LAN. The communication module may be a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or infrared communication. In one embodiment, the information processing device 10 is communicatively connected to the network 50 via the communication unit 11. This allows the information processing device 10 to communicate with the server 30, the measurement device 40, other computers, etc.

[0035] The storage unit 12 includes, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The storage unit 12 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores any information used in the operation of the information processing device 10. For example, the storage unit 12 stores system programs, application programs, embedded software, databases, etc. The information stored in the storage unit 12 may be updatable with information obtained from the network 50 via the communication unit 11, for example.

[0036] The acquisition unit 13 includes a sensor that measures the distance between the information processing device 10 and an object imaged by the imaging device 20, and acquires distance data from the imaging device 20 to the object. The sensor includes, for example, a ranging module that can measure the distance from the information processing device 10 to the object. The ranging module includes, for example, a LiDAR (Light Detection And Ranging) and an ultrasonic element array using a plurality of ultrasonic elements.

[0037] The input unit 14 includes one or more input interfaces that detect user input and acquire input information based on the user's operation. The input interfaces include physical keys, capacitive keys, a touch screen that is integrated with the display of the output unit 15, an imaging module such as a camera, and a microphone that accepts audio input.

[0038] The output unit 15 includes one or more output interfaces that output information to notify the user. The output interfaces include a display that outputs information visually as an image, a speaker that outputs information audibly as sound, and a vibrator that outputs information tactilely as vibration.

[0039] The control unit 16 includes one or more processors. The processor may be, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a dedicated processor specialized for a specific process. The control unit 16 is not limited to a processor and may include one or more dedicated circuits. The dedicated circuits may be, for example, a FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 16 controls each component to realize the functions of the information processing device 10, including the functions of the components such as the communication unit 11, the storage unit 12, the acquisition unit 13, the input unit 14, and the output unit 15.

[0040] Fig. 3 is a block diagram showing an example of a schematic configuration of the imaging device 20 of Fig. 1. An example of the configuration of the imaging device 20 will be mainly described with reference to Fig. 3. The imaging device 20 has an infrared camera 21 and an RGB camera 22. In the imaging device 20, for example, the infrared camera 21 and the RGB camera 22 are arranged so that their positions along the optical axis are approximately the same.

[0041] The infrared camera 21 includes, for example, a thermal camera that detects the temperature of an object by detecting far-infrared rays. A thermal camera detects heat and visualizes the temperature. In the present disclosure, an image of an object captured by a thermal camera is called a thermal image. In a thermal image, high-temperature areas are usually displayed in red, and low-temperature areas in blue. Without being limited to this, the infrared camera 21 may be equipped with a near-infrared light-emitting diode (LED) that serves as a light source and detects near-infrared rays reflected from the object. Such an infrared camera 21 uses infrared rays like a thermal camera, but uses light of a completely different wavelength included in the infrared wavelength band and has a different configuration.

[0042] The infrared camera 21 is capable of acquiring a thermographic image. The thermographic image acquired by the infrared camera 21 may be a still image such as a photograph, or may be a video. The infrared camera 21 generates a thermographic image of an object such as a tire, and transmits the thermographic image to the information processing device 10. In one embodiment, the thermographic image of the tire includes at least a portion of the tire.

[0043] The RGB camera 22 includes, for example, a visible light camera that detects visual information of an object by detecting visible light. In the present disclosure, an image of an object captured by a visible light camera is referred to as an RGB image. The RGB camera 22 is capable of acquiring RGB images. The RGB images acquired by the RGB camera 22 may be still images such as photographs, or may be videos. The RGB camera 22 generates RGB images of objects such as tires, and transmits them to the information processing device 10. In one embodiment, the RGB images of tires include at least a portion of the tires.

[0044] Fig. 4 is a block diagram showing an example of a schematic configuration of the server 30 in Fig. 1. An example of the configuration of the server 30 will be mainly described with reference to Fig. 4. The server 30 has a communication unit 31, a storage unit 32, and a control unit 33. In the server 30, the communication unit 31, the storage unit 32, and the control unit 33 are connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0045] The communication unit 31 includes a communication module for connecting to the network 50. The communication module is a communication module compatible with mobile communication standards such as 4G and 5G. The communication module may be a communication module compatible with standards such as wired LAN and wireless LAN. The communication module may be a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or infrared communication. In one embodiment, the server 30 is communicatively connected to the network 50 via the communication unit 31. This allows the server 30 to communicate with the information processing device 10, the measurement device 40, other computers, etc.

[0046] The storage unit 32 includes, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The storage unit 32 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 32 stores any information used in the operation of the server 30. For example, the storage unit 32 stores system programs, application programs, embedded software, databases, etc. The information stored in the storage unit 32 may be updatable with information obtained from the network 50 via the communication unit 31, for example.

[0047] For example, the memory unit 32 may store tire identification information for one or more tires that are the subject of failure determination. In the present disclosure, tire identification information is also referred to as a tire ID (Identifier). The tire identification information for a tire is information that can uniquely identify the tire. The tire identification information for a tire is, for example, uniquely assigned by the server 30, but is not limited to this and may be the tire's serial number or the vehicle number of the vehicle on which the tire is mounted. Furthermore, the memory unit 32 may store information about the tire in association with the tire's tire identification information.

[0048] The tire-related information includes any information related to the tire. The tire-related information may include, for example, the above-mentioned time-series data related to the tire, tire damage information, tire configuration information, information on the vehicle on which the tire is mounted, or information on the position of the tire on the vehicle. The tire damage information may be information including, for example, the position, shape, depth, and registration date and time of damage previously sustained to the tire. The tire configuration information includes, for example, the tire type, model number, material properties, tread pattern, belt angle, size, and weight. The information on the vehicle on which the tire is mounted includes vehicle identification information, type, model number, engine displacement, number of tires mounted, and number of shafts.

[0049] The control unit 33 includes one or more processors. The processor may be, for example, a general-purpose processor such as a CPU, or a dedicated processor specialized for a specific process. The control unit 33 is not limited to a processor and may include one or more dedicated circuits. The dedicated circuits may be, for example, an FPGA or an ASIC. The control unit 33 controls each component to realize the functions of the server 30, including the functions of the components such as the communication unit 31 and the memory unit 32.

[0050] (Operation of information processing device 10) Fig. 5 is a flowchart showing an example of the operation of the information processing device 10 in Fig. 2. Fig. 5 shows each step of an information processing method executed by the information processing device 10 as a computer. With reference to Fig. 5, an example of the flow of processing executed by the control unit 16 of the information processing device 10 according to an embodiment of the present disclosure will be mainly described.

[0051] In step S101, the control unit 16 of the information processing device 10 acquires a thermo image of an object captured by the infrared camera 21 of the imaging device 20. The control unit 16 acquires the thermo image of the object captured by the infrared camera 21 from the infrared camera 21 of the imaging device 20, which is integrally connected to the information processing device 10 via, for example, a connector.

[0052] In step S102, the control unit 16 of the information processing device 10 acquires an RGB image of an object captured by the RGB camera 22 of the imaging device 20. The control unit 16 acquires the RGB image of the object captured by the RGB camera 22 from the RGB camera 22 of the imaging device 20, which is integrally connected to the information processing device 10 via, for example, a connector.

[0053] In step S103, the control unit 16 of the information processing device 10 acquires distance data from the infrared camera 21 and the RGB camera 22 to the object when the thermo image and the RGB image were acquired in step S101 and step S102, respectively. For example, the control unit 16 measures the distance from the information processing device 10 to the object using a distance measurement module in the acquisition unit 13 of the information processing device 10.

[0054] For example, if the imaging device 20 is integrally connected to the information processing device 10 via a connector or the like, the distance from the information processing device 10 to the object will be approximately the same as the distance from the imaging device 20 to the object. The distance from the information processing device 10 to the object will be approximately the same as the distance from the infrared camera 21 of the imaging device 20 to the object. The distance from the information processing device 10 to the object will be approximately the same as the distance from the RGB camera 22 of the imaging device 20 to the object. Therefore, the control unit 16 acquires the distance from the information processing device 10 to the object measured using the acquisition unit 13 as distance data from the imaging device 20 including the infrared camera 21 and RGB camera 22 to the object.

[0055] In step S104, the control unit 16 of the information processing device 10 corrects the misalignment between the thermo image acquired in step S101 and the RGB image acquired in step S102 when they are superimposed on each other, based on the distance data acquired in step S103. After correcting the misalignment between the images, the control unit 16 generates a composite image in which the thermo image and the RGB image are superimposed on each other.

[0056] In step S105, the control unit 16 of the information processing device 10 displays on a screen the composite image generated in step S104 by superimposing the thermo image and the RGB image on each other. For example, the control unit 16 displays the composite image generated in step S104 on the screen of the display of the output unit 15 of the information processing device 10 itself.

[0057] 6A to 7B, an example of the operation of the information processing device 10 described above will be described in more detail below using the flowchart shown in Fig. 5 based on an image diagram of an image displayed on the screen of the output unit 15. In the following, it is assumed that the information processing device 10 and the imaging device 20 are integrally connected to each other via a connector or the like.

[0058] Fig. 6A is a first schematic diagram for explaining an example of the operation of the information processing device 10 of Fig. 2. Fig. 6B is a second schematic diagram for explaining an example of the operation of the information processing device 10 of Fig. 2. Figs. 6A and 6B are each an image diagram of an image displayed on the screen of the output unit 15 of the information processing device 10. The image is, for example, an example of an image displayed on the screen of the output unit 15 when the user adjusts the imaging position of the object S using the information processing device 10 before the imaging device 20 images the object S and simultaneously acquires a thermo image and an RGB image.

[0059] For example, a composite image of a thermo image and an RGB image obtained by the imaging device 20 is displayed on the screen of the output unit 15 via a first application of the information processing device 10 linked to the imaging device 20. In addition, a camera button 151 is displayed as an icon at the bottom of the screen of the output unit 15. The user adjusts the position of the information processing device 10 relative to the object S while viewing the composite image obtained by the imaging device 20 via the first application on the screen of the output unit 15, and presses the camera button 151 on the input unit 14.

[0060] When the control unit 16 of the information processing device 10 receives an input operation of the user pressing the camera button 151 via the input unit 14, the control unit 16 controls the imaging device 20 via the first application to capture an image of the object S. The imaging device 20 operates the infrared camera 21 and the RGB camera 22 based on a control signal acquired from the control unit 16 of the information processing device 10, and acquires a thermo image and an RGB image of the object S, respectively. The imaging device 20 outputs the thermo image and the RGB image acquired by capturing the image of the object S to the control unit 16 of the information processing device 10. As a result, the control unit 16 of the information processing device 10 acquires a thermo image and an RGB image, respectively, as shown in steps S101 and S102 of FIG. 5.

[0061] 5, the control unit 16 of the information processing device 10 acquires distance data from the infrared camera 21 and the RGB camera 22 to the object S when the thermo image and the RGB image are acquired. For example, the control unit 16 acquires the distance data using the AR function of the information processing device 10 to which the imaging device 20 having the infrared camera 21 and the RGB camera 22 is attached. The control unit 16 acquires the distance data using the ranging module of the acquisition unit 13, for example, via a second application that provides the AR function of the information processing device 10 and is different from the first application.

[0062] Control unit 16 of information processing device 10 may acquire distance data at a time interval longer than the interval at which distance data is updated based on the AR function. If the normal update time interval for distance data in a second application that provides the AR function of information processing device 10 is, for example, 1 / 60 second, control unit 16 may acquire distance data by the first application in cooperation with the second application at intervals of, for example, 1 second, which is sufficiently longer than 1 / 60 second.

[0063] 6A and 6B, as an example, an operation screen of a first application of the information processing device 10 that operates in cooperation with the imaging device 20 is displayed on the output unit 15 of the information processing device 10. On the other hand, an operation screen of a second application that provides an AR function of the information processing device 10 is not displayed on the output unit 15. The second application that provides the AR function of the information processing device 10 operates by background processing.

[0064] The control unit 16 of the information processing device 10 displays, for example, the range within which distance data is acquired on the screen. As shown in FIGS. 6A and 6B, the control unit 16 displays, for example, a frame 152 indicating the range within which distance data is acquired on the operation screen of the first application on the output unit 15. The control unit 16 acquires distance data for a location on the object S corresponding to a center point 153 of the frame 152 displayed on the operation screen. The center point 153 is initially set to be located in the center of the screen of the output unit 15. The operation screen of the first application displayed on the output unit 15 also displays a slider 154 that reflects changes in the distance from the location on the object S corresponding to the center point 153, which is the center point of the screen of the output unit 15, to the information processing device 10 at predetermined time intervals. Directly above the slider 154, the measured distance value is displayed as "***m (meters)."

[0065] 6A and 6B, control unit 16 positions frame 152 and center point 153 at the center of the screen of output unit 15 according to the initial setting. Without being limited to this, control unit 16 may change at least one of the position and size on the screen of the distance data acquisition range based on a user operation on the screen when acquiring distance data in step S103 of FIG.

[0066] For example, when control unit 16 receives, via input unit 14, a user's operation of dragging frame 152 on the screen of output unit 15, control unit 16 may change the position on the screen of frame 152 indicating the distance data acquisition range in accordance with the user's dragging operation. For example, when control unit 16 receives, via input unit 14, a user's operation of pressing zoom button 155 on the screen of output unit 15, control unit 16 may change the size on the screen of frame 152 indicating the distance data acquisition range in accordance with the user's pressing operation.

[0067] In addition to the display configuration on the screen of the output unit 15 as described above, the control unit 16 of the information processing device 10 may additionally display, for example, temperature information 156 of the object S displayed on the screen. The temperature information 156 includes, for example, a maximum temperature, a minimum temperature, and an average temperature. The control unit 16 may additionally indicate on the screen a predetermined location on the object S that indicates the maximum temperature with a pointer 157 along with a numerical value of the maximum temperature, such as "***°C."

[0068] Fig. 7A is a third schematic diagram illustrating an example of the operation of the information processing device 10 of Fig. 2. Fig. 7B is a fourth schematic diagram illustrating an example of the operation of the information processing device 10 of Fig. 2. Figs. 7A and 7B are each an image diagram of an image displayed on the screen of the output unit 15 of the information processing device 10. The image is an example of a composite image IM0 generated by superimposing a thermo image IM1 and an RGB image IM2 on each other, for example, by the control unit 16 of the information processing device 10 correcting the positional deviation between the images in step S104 of Fig. 5.

[0069] Based on the acquired distance data, the control unit 16 of the information processing device 10 corrects the misalignment between the thermo image IM1 and the RGB image IM2 when the images are superimposed on each other. For example, the control unit 16 changes the scale between the thermo image IM1 and the RGB image IM2 and superimposes the thermo image IM1 and the RGB image IM2 on each other. For example, if the distances from the infrared camera 21 and the RGB camera 22 that acquired the thermo image IM1 and the RGB image IM2, respectively, to the object S are known based on the distance data, the control unit 16 can accurately align the thermo image IM1 and the RGB image IM2 simply by changing the scale between the images.

[0070] 7A and 7B, in the composite image IM0 generated by performing the alignment correction process, it is possible to display clear contours even in areas where the contours are unclear in the thermographic image IM1 alone. For example, the control unit 16 can display clear contours in the composite image IM0 of the letters "XYZ S," which are displayed unclearly in the thermographic image IM1.

[0071] (effect) The information processing device 10 and information processing method according to the embodiment described above can improve the accuracy of a technique for determining the state of an object using an image of the object. The information processing device 10 corrects misalignment between a thermographic image and an RGB image when the images are superimposed on each other based on acquired distance data. This allows the information processing device 10 to accurately align the thermographic image and the RGB image simply by changing the scale of the images based on the acquired distance data. Therefore, the information processing device 10 can clearly display the contour of a target area of ​​an object whose contour is unclear in the thermographic image alone by superimposing an RGB image. This allows the information processing device 10 to accurately identify the position, shape, size, etc. of the target area in the composite image. This improves the accuracy of determining the state of the object at the target area using the composite image. The information processing device 10 can automatically perform the above-described composite image generation and display processes in real time, all by itself.

[0072] For example, when the target location of the object includes a damaged portion of a tire, the information processing device 10 can clearly display the position, shape, size, etc. of the damaged portion of the tire on the composite image, enabling accurate identification. Therefore, the server 30, which acquires the composite image from the information processing device 10, can use the acquired composite image to accurately estimate the depth of the damaged portion of the tire, thereby improving the accuracy of tire failure determination.

[0073] The information processing device 10 acquires distance data using the AR function of the information processing device 10 to which the imaging device 20 having the infrared camera 21 and the RGB camera 22 is attached. This allows the information processing device 10 to accurately measure the distance from the imaging device 20 having the infrared camera 21 and the RGB camera 22 to an object using the function of the information processing device 10 itself.

[0074] The information processing device 10 acquires distance data at a time interval longer than the update time interval of the distance data based on the AR function, thereby enabling the information processing device 10 to reduce the load of calculation processing related to the acquired distance data.

[0075] Information processing device 10 displays the range in which distance data is acquired on the screen. This allows a user viewing the screen displayed on output unit 15 of information processing device 10 to easily understand the portion of the object displayed on the screen for which information processing device 10 is acquiring distance data.

[0076] The information processing device 10 changes at least one of the position and size of the acquisition range on the screen based on the user's operation on the screen, thereby enabling the information processing device 10 to appropriately display on the screen the acquisition range of distance data according to at least one of the position and size desired by the user.

[0077] The target object is a rubber product including a tire or a crawler. This allows the information processing device 10 to generate and display a composite image in which the thermo image and the RGB image are aligned with high precision for the rubber product as the target object.

[0078] For example, any method can be used to output the tire damage information as the object. For example, the control unit 33 of the server 30 may transmit a request to display the tire damage information to the information processing device 10 as a terminal device via the communication unit 31. In such a case, the information processing device 10 can display the tire damage information via the display of the output unit 15 or the like based on the request received from the server 30.

[0079] The tire damage information includes, for example, the depth of the damaged portion of the tire. However, the tire damage information is not limited to the depth of the damaged portion of the tire, and may include any information such as the location and number of damaged portions, and warning messages. As a result, a user of the information processing system 1 as a tire failure determination system can easily grasp the depth of the damaged portion on the outer surface of the tire, and can plan tire inspection, repair, replacement, etc. before the tire fails.

[0080] When tires are replaced, they may be replaced with different tires depending on the tire damage information. For example, if the surface temperature of the damaged portion of a tire is higher than the temperature limit set for the tire, the tire may be replaced with a different type of tire that is more heat resistant. Alternatively, if the number of damaged portions of a tire is greater than a predetermined number, the tire may be replaced with a different type of tire that is less susceptible to damaged portions. In this way, users of the tire failure determination system can select tires suitable for use at mining sites based on the tire damage information.

[0081] The server 30 can motivate the user to inspect more carefully the damaged portions of the tire in the composite image, particularly those with high surface temperatures. Conversely, the server 30 can also prompt the user to simplify or skip the inspection of the damaged portions of the tire in the composite image, particularly those with low surface temperatures. Therefore, the information processing system 1 can improve the usefulness of the technology for determining tire faults.

[0082] (Variation) Although the present disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, the configurations and functions included in each embodiment can be rearranged so as not to cause logical inconsistencies. Furthermore, the configurations and functions included in each embodiment can be combined with other embodiments, and multiple configurations and functions can be combined, divided, or partially omitted.

[0083] For example, an embodiment is possible in which a general-purpose computer functions as the information processing device 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the information processing device 10 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor or a non-transitory computer-readable medium storing the program. Non-transitory computer-readable media include, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory.

[0084] At least a part of the processing operations performed by the information processing device 10 in the above embodiment may be performed by the server 30. For example, the server 30 may perform the series of processes shown in the flowchart of FIG. 5 instead of the information processing device 10. That is, the server 30 may be the "information processing device" recited in the claims. In this case, the server 30 may transmit the composite image generated by the server 30 to the information processing device 10 and display it on the screen of the output unit 15 of the information processing device 10, as a process corresponding to step S105 shown in the flowchart of FIG. 5.

[0085] At least a part of the processing operations executed by the server 30 in the above embodiment may be executed by the information processing device 10. For example, the information processing device 10 itself may function as a tire malfunction determination device, instead of the server 30. The information processing device 10 may also execute a series of processes related to the tire malfunction determination described above, and the series of processes may also be completed by the information processing device 10 itself.

[0086] In the above embodiment, the information processing system 1 has been described as including the server 30 and the measuring device 40 in addition to the information processing device 10 and the imaging device 20, but is not limited to this. The information processing system 1 does not need to include at least one of the server 30 and the measuring device 40. For example, the information processing system 1 may be configured only with the information processing device 10 and the imaging device 20, without including both the server 30 and the measuring device 40.

[0087] In the above embodiment, the information processing device 10 is described as acquiring distance data using the AR function of the information processing device 10 to which the imaging device 20 having the infrared camera 21 and the RGB camera 22 is attached, but this is not limited to this. The information processing device 10 may acquire distance data using any other method. For example, the information processing device 10 may acquire distance data based on any other distance measurement function different from the AR function of the information processing device 10. For example, the information processing device 10 may acquire distance data using an independent module having a distance measurement function that is additionally attached to the imaging device 20 instead of or in addition to a distance measurement module such as the acquisition unit 13 that is included in the information processing device 10.

[0088] In the above embodiment, the information processing device 10 acquires distance data at a time interval longer than the interval at which distance data is updated based on the AR function, but this is not limiting. The information processing device 10 may acquire distance data at a time interval shorter than or equal to the interval at which distance data is updated based on the AR function.

[0089] In the above embodiment, the information processing device 10 has been described as acquiring distance data for a location on the object S corresponding to the center point 153 of the frame 152 displayed on the operation screen of the output unit 15, but this is not limited to this. The control unit 16 of the information processing device 10 may acquire distance data for multiple locations on the object S and calculate statistical values ​​of the distance data for the multiple locations. The statistical values ​​include, for example, a median and an average value. The multiple locations on the object S are, for example, locations corresponding to multiple pixels included in the frame 152. As a result, the information processing device 10 can more accurately measure the distance from the imaging device 20 to the object S.

[0090] In the above embodiment, the information processing device 10 has been described as displaying the range in which distance data is obtained on the screen, but this is not limiting. The information processing device 10 does not necessarily have to display the range in which distance data is obtained on the screen.

[0091] In the above embodiment, the information processing device 10 has been described as changing at least one of the position and size of the acquisition range on the screen based on a user operation on the screen, but this is not limiting. The information processing device 10 may not execute the process of changing the acquisition range based on a user operation on the screen, and may maintain both the position and size of the acquisition range on the screen constant.

[0092] In the above embodiment, the target object is described as a rubber product including a tire or a crawler, but is not limited to this. The target object may include any other object that is a target of estimation of the state of a target location by the information processing system 1. [Industrial Applicability]

[0093] According to the present disclosure, it is possible to provide an information processing method and an information processing device that improve the accuracy of a technique for determining the state of an object using an image of the object.

[0094] [Contribution to the United Nations-led Sustainable Development Goals (SDGs)] The SDGs have been proposed to realize a sustainable society. One embodiment of the present disclosure is believed to be a technology that can contribute to goals such as "No. 9 - Build infrastructure for industry and technological innovation," "No. 12 - Responsible consumption and production," and "No. 13 - Take concrete measures against climate change." [Explanation of symbols]

[0095] 1. Information Processing Systems 10. Information processing equipment 11 Communications Department 12 Storage section 13 Acquisition Department 14 Input section 15 Output section 151 Camera button 152 frames 153 Center point 154 Slider 155 Zoom button 156 Temperature information 157 Pointer 16 Control Unit 20 Imaging device 21 Infrared camera 22 RGB cameras 30 servers 31 Communications Department 32 Storage section 33 Control Unit 40 Measuring Equipment 50 Network IM0 composite image IM1 Thermo Image IM2 RGB image S Object

Claims

1. 1. A computer-implemented information processing method, comprising: Obtaining a thermal image of an object using an infrared camera; acquiring an RGB image of the object using an RGB camera; displaying a composite image on a screen in which the thermo image and the RGB image are superimposed on each other; acquiring distance data to the object from the infrared camera and the RGB camera; correcting a positional deviation between the thermo image and the RGB image when the images are superimposed on each other based on the acquired distance data; Including, Information processing methods.

2. 2. The information processing method according to claim 1, acquiring the distance data includes acquiring the distance data using an AR function of an information processing device to which an imaging device having the infrared camera and the RGB camera is attached; Information processing methods.

3. 3. The information processing method according to claim 2, acquiring the distance data includes acquiring the distance data at a time interval longer than an update time interval of the distance data based on the AR function; Information processing methods.

4. 4. The information processing method according to claim 1, acquiring the distance data includes acquiring the distance data for a plurality of locations on the object, and calculating statistics of the distance data for the plurality of locations. Information processing methods.

5. 4. The information processing method according to claim 1, acquiring the distance data includes displaying an acquisition range of the distance data on the screen; Information processing methods.

6. 6. The information processing method according to claim 5, acquiring the distance data includes changing at least one of a position and a size of the acquisition range on the screen based on a user operation on the screen. Information processing methods.

7. 4. The information processing method according to claim 1, The object is a rubber product including a tire or a crawler. Information processing methods.

8. An information processing device, A control unit is provided, the control unit A thermographic image of the object is captured using an infrared camera, An RGB image of the object is acquired by an RGB camera; A composite image obtained by superimposing the thermo image and the RGB image on each other is displayed on a screen. Acquire distance data to the object from the infrared camera and the RGB camera; correcting a positional deviation between the thermo image and the RGB image when the images are superimposed on each other based on the acquired distance data; Information processing device.

Images