Information processing method and information processing apparatus

Abstract

To provide an information processing method for improving the accuracy of a technique for determining the state of an object by using an image obtained by imaging the object.SOLUTION: The method includes acquiring a thermo image obtained by imaging a subject S with an infrared light camera 21, specifying a position of a target place in the subject S on the basis of the acquired thermo image, acquiring surface temperature of the target place on the basis of the acquired thermo image, acquiring a range from the infrared light camera 21 to the subject S, and correcting the surface temperature on the basis of the acquired range.SELECTED DRAWING: Figure 6A

Description

[Technical Field]

[0001] The present disclosure relates to an information processing method and an information processing device. [Background technology]

[0002] Conventionally, there are known techniques for determining the condition of an object using an image of the object. For example, there is known a technique for determining tire damage. For example, Patent Document 1 discloses a tire damage detection system that detects the size of a damaged portion of a tire based on the diameter of a rim wheel, based on image data of the tire mounted on the rim wheel. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2019-202729 Summary of the Invention [Problem to be solved by the invention]

[0004] However, there is a demand for further improvement in the accuracy of technology for determining the condition of an object using an image of the object. For example, there is a demand for further improvement in the accuracy of technology for determining tire failure. For example, in the case of mining vehicles used at mining sites, if a sudden tire failure occurs, productivity decreases due to the need to transport the mining vehicle and replace the tire. In addition, regular tire inspections by workers to prevent sudden failures are a cost burden. Therefore, there is a demand for more accurate tire failure determination.

[0005] In view of the above circumstances, an object of the present disclosure is to provide an information processing method and an information processing device that improve the accuracy of a technique for determining the state of an object using an image of the object. [Means for solving the problem]

[0006] [1] An information processing method according to one embodiment of the present disclosure is an information processing method executed by a computer, and includes: acquiring a thermographic image of an object captured with an infrared camera; identifying a position of a target location on the object based on the acquired thermographic image; acquiring a surface temperature of the target location based on the acquired thermographic image; acquiring a distance from the infrared camera to the object; and correcting the surface temperature based on the acquired distance. According to an information processing method according to an embodiment of the present disclosure, it is possible to improve the accuracy of a technique for determining the state of an object using an image of the object.

[0007] [2] An information processing method according to one embodiment of the present disclosure may be the information processing method described in [1] above, and may further include estimating the state of the target location based on the identified position of the target location and the corrected surface temperature. According to the information processing method having such a configuration, the accuracy of determining the state of an object at a target location using a thermo image is improved.

[0008] [3] An information processing method according to one embodiment of the present disclosure is the information processing method described in [2] above, wherein the target location includes a damaged portion in the object, and the condition of the target location may include the depth of the damaged portion. According to the information processing method having such a configuration, the surface temperature of the damaged portion of the object can be identified with high accuracy on the thermo image.

[0009] [4] An information processing method according to one embodiment of the present disclosure is the information processing method described in any one of [1] to [3] above, wherein correcting the surface temperature may include correcting the surface temperature using machine learning. According to the information processing method having such a configuration, the surface temperature can be corrected with higher accuracy.

[0010] [5] An information processing method according to one embodiment of the present disclosure is an information processing method described in any one of [1] to [4] above, wherein correcting the surface temperature may include correcting the surface temperature using at least one of a correction coefficient and a correction term determined based on a function that includes the distance as a variable. According to an information processing method having such a configuration, the load of the calculation processing required for the surface temperature correction processing can be reduced compared to when machine learning is used.

[0011] [6] An information processing method according to one embodiment of the present disclosure is an information processing method described in any one of [1] to [5] above, wherein acquiring the distance may include acquiring the distance using an AR function of an information processing device to which the infrared camera is attached. According to the information processing method having such a configuration, the distance from the infrared camera to the target object can be measured with high accuracy using the functions of the information processing device itself.

[0012] [7] An information processing method according to one embodiment of the present disclosure is the information processing method described in [6] above, and may further include displaying the corrected surface temperature on the screen of the information processing device in real time. According to the information processing method having such a configuration, the correction process and display process relating to the surface temperature can be completed by the information processing device alone and can be executed automatically in real time.

[0013] [8] An information processing method according to one embodiment of the present disclosure is the information processing method described in any one of [1] to [7] above, wherein the object may be a rubber product including a tire or a crawler. According to the information processing method having such a configuration, it is possible to display a thermographic image of a rubber product as the object, in which the surface temperature of the target location on the object has been accurately corrected.

[0014] [9] An information processing device according to one embodiment of the present disclosure includes a control unit that acquires a thermal image of an object captured by an infrared camera, identifies the position of a target location on the object based on the acquired thermal image, acquires the surface temperature of the target location based on the acquired thermal image, acquires the distance from the infrared camera to the object, and corrects the surface temperature based on the acquired distance. According to an information processing device according to an embodiment of the present disclosure, it is possible to improve the accuracy of a technique for determining the state of an object using an image of the object. [Effects of the Invention]

[0015] According to the present disclosure, it is possible to provide an information processing method and an information processing device that improve the accuracy of a technique for determining the state of an object using an image of the object. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a block diagram illustrating an example of a schematic configuration of an information processing system according to an embodiment of the present disclosure. [Figure 2] FIG. 2 is a block diagram showing an example of a schematic configuration of the information processing device of FIG. 1. [Figure 3] 2 is a block diagram showing an example of a schematic configuration of the imaging device of FIG. 1. FIG. [Figure 4] 2 is a block diagram showing an example of a schematic configuration of a server in FIG. 1. FIG. [Figure 5] 2 is a flowchart showing an example of the operation of the information processing system of FIG. 1. [Figure 6A] FIG. 3 is a first schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. [Figure 6B] 10 is a second schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. FIG. [Figure 7A] 3 is a third schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. FIG. [Figure 7B] 4 is a fourth schematic diagram for explaining an example of the operation of the information processing device in FIG. 2. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0017] An information processing system 1 according to an embodiment of the present disclosure will be described below with reference to the drawings. Components and parts that are common to each drawing are designated by the same reference numerals. Please note that the drawings are schematic, and the dimensional ratios and the like may differ from those of the actual drawings.

[0018] (Configuration of Information Processing System 1) An overview of an information processing system 1 according to an embodiment of the present disclosure will be described with reference to Fig. 1. Fig. 1 is a block diagram showing an example of a schematic configuration of the information processing system 1 according to an embodiment of the present disclosure. As shown in Fig. 1, the information processing system 1 includes an information processing device 10, an imaging device 20, a server 30, and a measuring device 40.

[0019] For ease of explanation, FIG. 1 illustrates only one each of the information processing device 10, imaging device 20, server 30, and measuring device 40, but the number of each device included in the information processing system 1 is not limited to one. The information processing system 1 may include, for example, multiple information processing devices 10. The information processing system 1 may include, for example, multiple imaging devices 20. The information processing system 1 may include, for example, multiple servers 30. The information processing system 1 may include, for example, multiple measuring devices 40.

[0020] The information processing device 10, the server 30, and the measuring device 40 are each communicatively connected to, for example, a network 50. The imaging device 20 may be communicatively connected to the network 50 indirectly via the information processing device 10, or may be communicatively connected to the network 50 directly without via the information processing device 10.

[0021] The information processing device 10 and the imaging device 20 are connected to each other via, for example, a connector. The information processing device 10 is integrally connected to the imaging device 20 so that the distance between the imaging device 20 and an object imaged by the imaging device 20 can be accurately measured using the functions of the information processing device 10. However, the information processing device 10 is not limited to this, and does not need to be integrally connected to the imaging device 20 as long as the information processing device 10 can acquire the distance from the imaging device 20 to the object.

[0022] In the present disclosure, the term "object" includes, for example, any object for which the condition of a target portion of the object is to be estimated. For example, the object is a rubber product including a tire, a crawler, and the like. The "target portion of the object" includes, for example, a damaged portion of the object. The "condition of the target portion" includes, for example, the depth of the damaged portion of the object. As an example, the information processing device 10 performs a surface temperature correction process, which will be described later, as preprocessing in order to improve the estimation accuracy when the depth of the damaged portion of the tire is estimated by the server 30 or the like in tire failure determination.

[0023] The information processing device 10 includes any general-purpose electronic device including, for example, a smartphone, a tablet PC (Personal Computer), a wearable device such as a smart watch, and a PC. The information processing device 10 is not limited to these, and may include other electronic devices dedicated to the information processing system 1. As an example, the information processing device 10 functions as a terminal device for the server 30. The information processing device 10 is used by a user who wants the server 30 to estimate the state of a target location of an object.

[0024] The imaging device 20 is configured with a computer including one or more cameras. The imaging device 20 is capable of acquiring, for example, thermal images. The images acquired by the imaging device 20 may be still images such as photographs, or may be videos. The imaging device 20 generates a thermal image of an object such as a tire and transmits it to the information processing device 10. In one embodiment, the thermal image of the tire captures at least a portion of the tire.

[0025] As described above, the imaging device 20 is, for example, connected to the information processing device 10 and is portable by a user along with the information processing device 10. However, the imaging device 20 is not limited to this. When the information processing device 10 and the imaging device 20 are not integrally connected to each other, the imaging device 20 may be, for example, a fixed device installed on a vehicle's travel route at a mining site. This allows the imaging device 20 to capture images of the vehicle traveling on the travel route, which is less likely to reduce the vehicle's operating rate and productivity at the mine. Alternatively, the imaging device 20 may be a device mounted on a drone.

[0026] The server 30 is configured with one or more computers. In one embodiment, the server 30 is described as being configured with one computer. However, the server 30 may be configured with multiple computers, such as in a cloud computing system. In the present disclosure, the server 30 functions as, for example, a tire malfunction determination device.

[0027] The measurement device 40 is configured with a computer including one or more sensors. The sensors may be, but are not limited to, a digital tachograph, a tire pressure monitoring system (TPMS), an electronic control unit (ECU), or a car navigation device. The measurement device 40 acquires time-series data related to tires mounted on a vehicle and transmits the data to the server 30. Therefore, the measurement device 40 may be installed on the vehicle or tires.

[0028] The time-series data on the tires mounted on a vehicle includes tire-related measurement values ​​and the dates and times of the measurements. For example, if the measurement device 40 includes a TPMS installed on the tires, the tire-related measurement values ​​may include tire condition information on the tires, such as tire internal pressure (air pressure), tire cavity temperature, and thermal history. The tire thermal history is the history of heat applied to the tire as the tire is used. The tire thermal history is used to evaluate how much energy has been applied to the tire since it was first used. Generally, the greater the thermal history, the more severe the tire's deterioration. The thermal history can be calculated, for example, by applying the tire cavity temperature to the Arrhenius equation. Furthermore, for example, if the measurement device 40 includes a digital tachograph installed on the vehicle, the tire-related measurement values ​​may include vehicle driving information, such as the vehicle's driving time, driving distance, speed, acceleration, and tire rotation count.

[0029] The network 50 is any communication network that allows mutual communication among the information processing device 10, the imaging device 20, the server 30, and the measuring device 40. The network 50 in one embodiment may be, for example, the Internet, a mobile communication network, a LAN (Local Area Network), or a combination thereof.

[0030] The information processing system 1 may function as a tire malfunction determination system centered around a server 30 serving as a tire malfunction determination device. In this case, the information processing system 1 is used to determine malfunctions in one or more tires. In the information processing system 1, the server 30 acquires, from the information processing device 10, a thermographic image of a tire that has been captured by an imaging device 20 and in which the surface temperature of the damaged portion has been corrected by the information processing device 10. The server 30 acquires the surface temperature of at least one damaged portion of the tire based on the thermographic image.

[0031] The server 30 acquires time-series data of the tire cavity temperature from the measuring device 40. The server 30 estimates the depth of at least one damaged portion of the tire based on the surface temperature of at least one damaged portion of the tire in the thermal image and the cavity temperature at the time the thermal image was acquired. In this way, the information processing system 1 can more accurately estimate the depth of the damaged portion by using the tire cavity temperature in addition to the surface temperature of at least one damaged portion of the tire in the thermal image.

[0032] In the present disclosure, the tire is not particularly limited, but may be an OR (Off The Road) tire mounted on a mining vehicle such as a transport vehicle, a construction vehicle, a work vehicle, or a heavy machinery vehicle used at a mining site, etc. However, the tire may be a tire other than an OR tire.

[0033] In the present disclosure, the vehicle is, for example, a mining vehicle used at a mining site, etc. However, the vehicle is not limited to the above-mentioned mining vehicle, and may be any vehicle capable of being fitted with tires, such as a transport vehicle, a construction vehicle, a work vehicle, a heavy machinery vehicle, a bus, a passenger car, a motorcycle, a bicycle, or an airplane.

[0034] Fig. 2 is a block diagram showing an example of a schematic configuration of the information processing device 10 of Fig. 1. An example of the configuration of the information processing device 10 will be mainly described with reference to Fig. 2. The information processing device 10 has a communication unit 11, a storage unit 12, an acquisition unit 13, an input unit 14, an output unit 15, and a control unit 16. In the information processing device 10, the communication unit 11, the storage unit 12, the acquisition unit 13, the input unit 14, the output unit 15, and the control unit 16 are connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0035] The communication unit 11 includes a communication module for connecting to the network 50. The communication module is a communication module compatible with mobile communication standards such as 4G (4th Generation) and 5G (5th Generation). The communication module may be a communication module compatible with standards such as wired LAN and wireless LAN. The communication module may be a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or infrared communication. In one embodiment, the information processing device 10 is communicatively connected to the network 50 via the communication unit 11. This allows the information processing device 10 to communicate with the server 30, the measurement device 40, other computers, etc.

[0036] The storage unit 12 includes, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The storage unit 12 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 12 stores any information used in the operation of the information processing device 10. For example, the storage unit 12 stores system programs, application programs, embedded software, databases, etc. The information stored in the storage unit 12 may be updatable with information obtained from the network 50 via the communication unit 11, for example.

[0037] The acquisition unit 13 includes a sensor that measures the distance between the information processing device 10 and an object imaged by the imaging device 20, and acquires the distance from the imaging device 20 to the object. The sensor includes, for example, a ranging module that can measure the distance from the information processing device 10 to the object. The ranging module includes, for example, a LiDAR (Light Detection And Ranging) and an ultrasonic element array using a plurality of ultrasonic elements.

[0038] The input unit 14 includes one or more input interfaces that detect user input and acquire input information based on the user's operation. The input interfaces include physical keys, capacitive keys, a touch screen that is integrated with the display of the output unit 15, an imaging module such as a camera, and a microphone that accepts audio input.

[0039] The output unit 15 includes one or more output interfaces that output information to notify the user. The output interfaces include a display that outputs information visually as an image, a speaker that outputs information audibly as sound, and a vibrator that outputs information tactilely as vibration.

[0040] The control unit 16 includes one or more processors. The processor may be, for example, a general-purpose processor such as a CPU (Central Processing Unit) or a dedicated processor specialized for a specific process. The control unit 16 is not limited to a processor and may include one or more dedicated circuits. The dedicated circuits may be, for example, a FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit). The control unit 16 controls each component to realize the functions of the information processing device 10, including the functions of the components such as the communication unit 11, the storage unit 12, the acquisition unit 13, the input unit 14, and the output unit 15.

[0041] Fig. 3 is a block diagram showing an example of a schematic configuration of the imaging device 20 of Fig. 1. The example of the configuration of the imaging device 20 will be mainly described with reference to Fig. 3. The imaging device 20 has an infrared camera 21.

[0042] The infrared camera 21 includes, for example, a thermal camera that detects the temperature of an object by detecting far-infrared rays. A thermal camera detects heat and visualizes the temperature. In the present disclosure, an image of an object captured by a thermal camera is called a thermal image. In a thermal image, high-temperature areas are usually displayed in red, and low-temperature areas in blue. Without being limited to this, the infrared camera 21 may be equipped with a near-infrared light-emitting diode (LED) that serves as a light source and detects near-infrared rays reflected from the object. Such an infrared camera 21 uses infrared rays like a thermal camera, but uses light of a completely different wavelength included in the infrared wavelength band and has a different configuration.

[0043] The infrared camera 21 is capable of acquiring a thermographic image. The thermographic image acquired by the infrared camera 21 may be a still image such as a photograph, or may be a video. The infrared camera 21 generates a thermographic image of an object such as a tire, and transmits the thermographic image to the information processing device 10. In one embodiment, the thermographic image of the tire includes at least a portion of the tire.

[0044] Without being limited to the above, the imaging device 20 may include any other camera, such as a visible light camera. A visible light camera detects visual information of an object by detecting, for example, visible light. In this disclosure, an image of an object captured by a visible light camera is referred to as an RGB image. A visible light camera can acquire an RGB image. The RGB image acquired by a visible light camera may be a still image such as a photograph, or may be a video. The visible light camera may generate an RGB image of an object such as a tire and transmit it to the information processing device 10. In one embodiment, the RGB image of a tire includes at least a portion of the tire.

[0045] Fig. 4 is a block diagram showing an example of a schematic configuration of the server 30 in Fig. 1. An example of the configuration of the server 30 will be mainly described with reference to Fig. 4. The server 30 has a communication unit 31, a storage unit 32, and a control unit 33. In the server 30, the communication unit 31, the storage unit 32, and the control unit 33 are connected to each other so as to be able to communicate with each other via wire or wirelessly.

[0046] The communication unit 31 includes a communication module for connecting to the network 50. The communication module is a communication module compatible with mobile communication standards such as 4G and 5G. The communication module may be a communication module compatible with standards such as wired LAN and wireless LAN. The communication module may be a communication module compatible with short-range wireless communication standards such as Wi-Fi (registered trademark), Bluetooth (registered trademark), or infrared communication. In one embodiment, the server 30 is communicatively connected to the network 50 via the communication unit 31. This allows the server 30 to communicate with the information processing device 10, the measurement device 40, other computers, etc.

[0047] The storage unit 32 includes, for example, a semiconductor memory, a magnetic memory, an optical memory, etc. The storage unit 32 functions, for example, as a main storage device, an auxiliary storage device, or a cache memory. The storage unit 32 stores any information used in the operation of the server 30. For example, the storage unit 32 stores system programs, application programs, embedded software, databases, etc. The information stored in the storage unit 32 may be updatable with information obtained from the network 50 via the communication unit 31, for example.

[0048] For example, the memory unit 32 may store tire identification information for one or more tires that are the subject of failure determination. In the present disclosure, tire identification information is also referred to as a tire ID (Identifier). The tire identification information for a tire is information that can uniquely identify the tire. The tire identification information for a tire is, for example, uniquely assigned by the server 30, but is not limited to this and may be the tire's serial number or the vehicle number of the vehicle on which the tire is mounted. Furthermore, the memory unit 32 may store information about the tire in association with the tire's tire identification information.

[0049] The tire-related information includes any information related to the tire. The tire-related information may include, for example, the above-mentioned time-series data related to the tire, tire damage information, tire configuration information, information on the vehicle on which the tire is mounted, or information on the position of the tire on the vehicle. The tire damage information may be information including, for example, the position, shape, depth, and registration date and time of damage previously sustained to the tire. The tire configuration information includes, for example, the tire type, model number, material properties, tread pattern, belt angle, size, and weight. The information on the vehicle on which the tire is mounted includes vehicle identification information, type, model number, engine displacement, number of tires mounted, and number of shafts.

[0050] The control unit 33 includes one or more processors. The processor may be, for example, a general-purpose processor such as a CPU, or a dedicated processor specialized for a specific process. The control unit 33 is not limited to a processor and may include one or more dedicated circuits. The dedicated circuits may be, for example, an FPGA or an ASIC. The control unit 33 controls each component to realize the functions of the server 30, including the functions of the components such as the communication unit 31 and the memory unit 32.

[0051] (Operation of information processing device 10) Fig. 5 is a flowchart showing an example of the operation of the information processing system 1 in Fig. 1. Fig. 5 shows each step of an information processing method executed by a computer. With reference to Fig. 5, an example of the flow of processing executed by the control unit 16 of the information processing device 10 or the control unit 33 of the server 30 according to an embodiment of the present disclosure will be mainly described.

[0052] In step S101, the control unit 16 of the information processing device 10 acquires a thermo image of an object captured by the infrared camera 21 of the imaging device 20. The control unit 16 acquires the thermo image of the object captured by the infrared camera 21 from the infrared camera 21 of the imaging device 20, which is integrally connected to the information processing device 10 via, for example, a connector.

[0053] In step S102, the control unit 16 of the information processing device 10 identifies the position of the target area in the object based on the thermoimage acquired in step S101. For example, the control unit 16 uses any method such as image analysis to identify the position of the target area, such as a damaged part of the object, contained in the thermoimage.

[0054] In step S103, the control unit 16 of the information processing device 10 acquires the surface temperature of the target location of the target object identified in step S102 based on the thermo image acquired in step S101.

[0055] In step S104, the control unit 16 of the information processing device 10 acquires the distance from the infrared camera 21 to the object when the thermo image was acquired in step S101. For example, the control unit 16 measures the distance from the information processing device 10 to the object using a distance measurement module in the acquisition unit 13 of the information processing device 10.

[0056] For example, if the imaging device 20 is integrally connected to the information processing device 10 via a connector or the like, the distance from the information processing device 10 to the object will be approximately the same as the distance from the imaging device 20 to the object. The distance from the information processing device 10 to the object will also be approximately the same as the distance from the infrared camera 21 of the imaging device 20 to the object. Therefore, the control unit 16 acquires the distance from the information processing device 10 to the object measured using the acquisition unit 13 as the distance from the imaging device 20 including the infrared camera 21 to the object.

[0057] In step S105, the control unit 16 of the information processing device 10 corrects the surface temperature acquired in step S103 based on the distance acquired in step S104.

[0058] In step S106, the control unit 16 of the information processing device 10 causes the surface temperature corrected in step S105 to be displayed on the screen of the output unit 15 of the information processing device 10 in real time.

[0059] In step S107, the control unit 33 of the server 30 estimates the state of the target location on the object based on the position of the target location identified by the information processing device 10 in step S102 and the surface temperature corrected by the information processing device 10 in step S105. For example, the control unit 33 of the server 30 receives the position of the target location identified in step S102 and the surface temperature corrected in step S105 together with the thermo image from the information processing device 10 via the network 50 and the communication unit 31. The control unit 33 of the server 30 estimates the state of the target location on the object based on the position and surface temperature of the target location acquired as information from the information processing device 10.

[0060] 6A to 7B, an example of the operation of the information processing device 10 described above will be described in more detail below using the flowchart shown in Fig. 5 based on an image diagram of an image displayed on the screen of the output unit 15. In the following, it is assumed that the information processing device 10 and the imaging device 20 are integrally connected to each other via a connector or the like.

[0061] Fig. 6A is a first schematic diagram for explaining an example of the operation of the information processing device 10 of Fig. 2. Fig. 6A shows a plurality of conceptual diagrams of images displayed on the screen of the output unit 15 of the information processing device 10, arranged according to the distance from the infrared camera 21 to the object. The image is, for example, an example of a thermo image obtained by capturing an image of the object S with the infrared camera 21 of the imaging device 20.

[0062] For example, a thermographic image obtained by the imaging device 20 is displayed on the screen of the output unit 15 via a first application of the information processing device 10 linked to the imaging device 20. For example, a user adjusts the position of the information processing device 10 relative to the object S while visually checking the thermographic image obtained by the imaging device 20 via the first application on the screen of the output unit 15, and performs an imaging operation such as pressing a camera button on the input unit 14.

[0063] When the control unit 16 of the information processing device 10 receives a user's imaging operation via the input unit 14, it controls the imaging device 20 via the first application to capture an image of the object S. The imaging device 20 operates the infrared camera 21 based on a control signal acquired from the control unit 16 of the information processing device 10, and acquires a thermographic image of the object S. The imaging device 20 outputs the thermographic image acquired by capturing an image of the object S to the control unit 16 of the information processing device 10. As a result, the control unit 16 of the information processing device 10 acquires the thermographic image, as shown in step S101 of FIG. 5 .

[0064] 5, the control unit 16 of the information processing device 10 acquires the distance from the infrared camera 21 to the object S when the thermo image was acquired. For example, the control unit 16 acquires the distance using the AR function of the information processing device 10 to which the infrared camera 21 is attached. The control unit 16 acquires the distance using the ranging module of the acquisition unit 13 via, for example, a second application that provides the AR function of the information processing device 10 and is different from the first application.

[0065] The control unit 16 of the information processing device 10 may acquire the distance at a time interval longer than the distance update time interval based on the AR function. If the normal distance update time interval in the second application that provides the AR function of the information processing device 10 is, for example, 1 / 60 second, the control unit 16 may acquire the distance by the first application in cooperation with the second application at an interval such as 1 second that is sufficiently longer than 1 / 60 second.

[0066] 6A, as an example, a screen of a first application of the information processing device 10 that operates in cooperation with the imaging device 20 is displayed on the output unit 15 of the information processing device 10. On the other hand, an operation screen of a second application that provides an AR function of the information processing device 10 is not displayed on the output unit 15. The second application that provides the AR function of the information processing device 10 operates by background processing.

[0067] In Figure 6A, the distance from the infrared camera 21 to the object S becomes closer as one moves from left to right. The object S in the thermographic image displayed on the screen of the output unit 15 also becomes larger as the distance changes from far to close. Of the four images IM1, IM2, IM3, and IM4, two damaged portions C11 and C12 are clearly visible on the tire in image IM3, which is relatively close. In image IM4, which is closest, damaged portion C12 is out of view, and only damaged portion C11 is displayed at a large magnification.

[0068] The control unit 16 of the information processing device 10 may display in real time the surface temperature of the damaged portion C11 of the object S in the image IM4, for example, on the screen of the output unit 15. The control unit 16 may additionally display the surface temperature of the damaged portion C11 of the object S on the screen by placing a maximum temperature value such as "***°C" close to the damaged portion C11.

[0069] Fig. 6B is a second schematic diagram for explaining an example of the operation of the information processing device 10 of Fig. 2. Fig. 6B is a graph plotting the surface temperature for each of the damaged portions C11 and C12 for each distance corresponding to the images IM1, IM2, IM3, and IM4 shown in Fig. 6A. The vertical axis of the graph in Fig. 6B indicates the maximum value of the surface temperature for each of the damaged portions C11 and C12. The horizontal axis of the graph in Fig. 6B indicates the distance corresponding to the images IM1, IM2, IM3, and IM4 shown in Fig. 6A.

[0070] Generally, light such as far-infrared rays attenuates in proportion to the square of the distance from the source. When atmospheric environmental factors such as air quality are taken into account, light such as far-infrared rays attenuates further the farther from the source. As shown in the graph in Figure 6B, the surface temperature decreases slightly as the horizontal axis moves from the plot in image IM4, which is close, to the plot in image IM1, which is farther away. Therefore, the farther the distance, the lower the apparent surface temperature of the target point on object S becomes relative to the actual value.

[0071] Therefore, in step S105 of the flowchart shown in FIG. 5, the control unit 16 of the information processing device 10 corrects the surface temperature based on the distance from the infrared camera 21 to the object.

[0072] At this time, the control unit 16 may correct the surface temperature using machine learning. For example, the control unit 16 may acquire in advance training data of at least one of a correction coefficient and a correction term to be added to the surface temperature depending on the distance, and construct a learning model. The control unit 16 may input the distance measured using the acquisition unit 13 to an input layer of the constructed learning model, and derive at least one of the correction coefficient and the correction term.

[0073] The control unit 16 may correct the surface temperature without using machine learning. In this case, the control unit 16 may correct the surface temperature using at least one of a correction coefficient and a correction term determined based on a function that includes distance as a variable, for example. In the present disclosure, the term "function" includes any function, such as a linear function or a nonlinear function.

[0074] 7A is a third schematic diagram for explaining an example of the operation of the information processing device 10 of FIG. 2. FIG. 7A shows a plurality of image diagrams displayed on the screen of the output unit 15 of the information processing device 10, arranged according to the distance from the infrared camera 21 to the object. The image is, for example, an example of a thermographic image obtained by capturing an image of the object S with the infrared camera 21 of the imaging device 20. FIG. 7A differs from FIG. 6A only in the image sample, and the same explanation as above applies to the points in common with FIG. 6A. The points different from FIG. 6A will be mainly explained.

[0075] 7A, the distance from the infrared camera 21 to the object S becomes closer as one moves from left to right. The object S in the thermographic image displayed on the screen of the output unit 15 also expands as the distance changes from far to close. Of the two images IM5 and IM6, three damaged areas C21, C22, and C23 are clearly visible on the tire in image IM6, which is the closest.

[0076] The control unit 16 of the information processing device 10 may display in real time on the screen of the output unit 15, for example, the surface temperature of each of the damaged portions C21, C22, and C23 of the object S in the image IM6. The control unit 16 may additionally display the surface temperature of each of the damaged portions C21, C22, and C23 of the object S on the screen by displaying a maximum temperature value such as "***°C" adjacent to each damaged portion.

[0077] Fig. 7B is a fourth schematic diagram for explaining an example of the operation of the information processing device 10 of Fig. 2. Fig. 7B is a graph plotting the surface temperature for each of damaged portions C21, C22, and C23 for each distance corresponding to images IM5 and IM6 shown in Fig. 7A. The vertical axis of the graph in Fig. 7B indicates the maximum surface temperature for each of damaged portions C21, C22, and C23. The horizontal axis of the graph in Fig. 7B indicates the distance corresponding to images IM5 and IM6 shown in Fig. 7A.

[0078] As in FIG. 6B, in the graph of FIG. 7B, the surface temperature decreases slightly as one moves along the horizontal axis from the plot in image IM6 when the distance is close to the plot in image IM5 when the distance is farther. Therefore, the surface temperature of the target location on object S appears lower than the actual value as the distance increases. Therefore, in step S105 of the flowchart shown in FIG. 5, control unit 16 of information processing device 10 corrects the surface temperature based on the distance from infrared camera 21 to the target. Specific details of the correction process are the same as those described using FIG. 6B.

[0079] (effect) The information processing device 10 and information processing method according to the embodiment described above can improve the accuracy of a technique for determining the state of an object using an image of the object. The information processing device 10 corrects the surface temperature of a target location on the object based on the acquired distance. This allows the information processing device 10 to accurately measure the surface temperature of the target location based on the acquired distance, independent of the distance. This allows the information processing device 10 to accurately identify the surface temperature of the target location in a thermographic image at various positions relative to the object. This improves the accuracy of determining the state of the object at the target location using a thermographic image. The information processing device 10 can automatically perform the above-described surface temperature correction and display processes in real time, all by itself.

[0080] For example, when the target location of the object includes a damaged portion of a tire, the information processing device 10 can accurately identify the surface temperature of the damaged portion of the tire on a thermographic image. Therefore, the server 30, which acquires the thermographic image together with the corrected surface temperature from the information processing device 10, can accurately estimate the depth of the damaged portion of the tire using the acquired thermographic image and the corrected surface temperature, thereby improving the accuracy of tire failure detection.

[0081] The server 30 estimates the state of the target location based on the position of the target location identified by the information processing device 10 and the corrected surface temperature. This improves the accuracy of determining the state of the object at the target location using the thermo image.

[0082] The target location includes a damaged portion of the object. The condition of the target location includes the depth of the damaged portion. This allows the information processing device 10 to accurately identify the surface temperature of the damaged portion of the object on the thermographic image. Therefore, the server 30, which acquires the thermographic image along with the corrected surface temperature from the information processing device 10, can accurately estimate the depth of the damaged portion of the object using the acquired thermographic image and the corrected surface temperature, thereby improving the accuracy of fault detection for the object.

[0083] The information processing device 10 corrects the surface temperature using machine learning, thereby enabling the information processing device 10 to correct the surface temperature with higher accuracy.

[0084] The information processing device 10 corrects the surface temperature using at least one of a correction coefficient and a correction term determined based on a function that includes distance as a variable. This allows the information processing device 10 to reduce the load of calculations required for the surface temperature correction process compared to when machine learning is used.

[0085] The information processing device 10 acquires the distance using the AR function of the information processing device 10 to which the infrared camera 21 is attached. This allows the information processing device 10 to accurately measure the distance from the infrared camera 21 to the target object using the function that the information processing device 10 itself has.

[0086] The information processing device 10 acquires the distance at a time interval longer than the interval for updating the distance based on the AR function, thereby enabling the information processing device 10 to reduce the load of calculation processing related to the acquired distance.

[0087] The information processing device 10 displays the corrected surface temperature in real time on the screen of the output unit 15 of the information processing device 10. As a result, the information processing device 10 can complete the correction process and display process related to the surface temperature by itself, and automatically execute them in real time.

[0088] The target object is a rubber product including a tire or a crawler. This allows the information processing device 10 to display a thermo image of the rubber product as the target object, in which the surface temperature of the target portion of the target object has been accurately corrected.

[0089] For example, any method can be used to output the tire damage information as the object. For example, the control unit 33 of the server 30 may transmit a request to display the tire damage information to the information processing device 10 as a terminal device via the communication unit 31. In such a case, the information processing device 10 can display the tire damage information via the display of the output unit 15 or the like based on the request received from the server 30.

[0090] The tire damage information includes, for example, the depth of the damaged portion of the tire. However, the tire damage information is not limited to the depth of the damaged portion of the tire, and may include any information such as the location and number of damaged portions, and warning messages. As a result, a user of the information processing system 1 as a tire failure determination system can easily grasp the depth of the damaged portion on the outer surface of the tire, and can plan tire inspection, repair, replacement, etc. before the tire fails.

[0091] When tires are replaced, they may be replaced with different tires depending on the tire damage information. For example, if the surface temperature of the damaged portion of a tire is higher than the temperature limit set for the tire, the tire may be replaced with a different type of tire that is more heat resistant. Alternatively, if the number of damaged portions of a tire is greater than a predetermined number, the tire may be replaced with a different type of tire that is less susceptible to damaged portions. In this way, users of the tire failure determination system can select tires suitable for use at mining sites based on the tire damage information.

[0092] The server 30 can motivate the user to inspect more carefully the damaged portions of the tire in the thermal image, especially those with high surface temperatures. Conversely, the server 30 can also prompt the user to simplify or skip the inspection of the damaged portions of the tire in the thermal image, especially those with low surface temperatures. Therefore, the information processing system 1 can improve the usefulness of the technology for determining tire faults.

[0093] (Variation) Although the present disclosure has been described based on the drawings and embodiments, it should be noted that those skilled in the art can make various modifications and alterations based on the present disclosure. Therefore, it should be noted that these modifications and alterations are within the scope of the present disclosure. For example, the configurations and functions included in each embodiment can be rearranged so as not to cause logical inconsistencies. Furthermore, the configurations and functions included in each embodiment can be combined with other embodiments, and multiple configurations and functions can be combined, divided, or partially omitted.

[0094] For example, an embodiment is possible in which a general-purpose computer functions as the information processing device 10 according to the above-described embodiment. Specifically, a program describing the processing content for realizing each function of the information processing device 10 according to the above-described embodiment is stored in the memory of the general-purpose computer, and the program is read and executed by a processor. Therefore, the present disclosure can also be realized as a program executable by a processor or a non-transitory computer-readable medium storing the program. Non-transitory computer-readable media include, for example, a magnetic recording device, an optical disk, a magneto-optical recording medium, and a semiconductor memory.

[0095] At least some of the processing operations performed by the information processing device 10 in the above embodiment may be performed by the server 30. For example, instead of the information processing device 10, the server 30 may perform the series of processes shown in the flowchart of FIG. 5. That is, the server 30 may be the "information processing device" recited in the claims. In this case, as a process corresponding to step S106 shown in the flowchart of FIG. 5, the server 30 may transmit a thermogram together with the surface temperature corrected by the server 30 to the information processing device 10, and display the corrected surface temperature together with the thermogram on the screen of the output unit 15 of the information processing device 10 in real time.

[0096] At least a part of the processing operations executed by the server 30 in the above embodiment may be executed by the information processing device 10. For example, the information processing device 10 itself may function as a tire malfunction determination device, instead of the server 30. The information processing device 10 may also execute a series of processes related to the tire malfunction determination described above, and the series of processes may also be completed by the information processing device 10 itself.

[0097] In the above embodiment, the information processing system 1 has been described as including the server 30 and the measuring device 40 in addition to the information processing device 10 and the imaging device 20, but is not limited to this. The information processing system 1 does not need to include at least one of the server 30 and the measuring device 40. For example, the information processing system 1 may be configured only with the information processing device 10 and the imaging device 20, without including both the server 30 and the measuring device 40.

[0098] In the above embodiment, the server 30 estimates the state of the target location based on the identified target location's position and the corrected surface temperature, but this is not limiting. As described above, the information processing device 10 may also perform such estimation processing.

[0099] In the above embodiment, the information processing device 10 corrects the surface temperature using at least one of a correction coefficient and a correction term determined based on machine learning or a function including distance as a variable, but is not limited to this. The information processing device 10 may also perform the surface temperature correction process based on the distance from the infrared camera 21 to the target object using any other method.

[0100] In the above embodiment, the information processing device 10 acquires the distance using the AR function of the information processing device 10 to which the infrared camera 21 is attached, but this is not limiting. The information processing device 10 may acquire the distance using any other method. For example, the information processing device 10 may acquire the distance based on any other distance measurement function different from the AR function of the information processing device 10. For example, the information processing device 10 may acquire the distance using an independent module additionally attached to the imaging device 20 and having a distance measurement function, instead of or in addition to a distance measurement module such as the acquisition unit 13 included in the information processing device 10.

[0101] In the above embodiment, the information processing device 10 acquires the distance at a time interval longer than the interval for updating the distance based on the AR function, but this is not limiting. The information processing device 10 may acquire the distance at a time interval shorter than or equal to the interval for updating the distance based on the AR function.

[0102] In the above embodiment, the information processing device 10 has been described as displaying the corrected surface temperature on the screen of the output unit 15 of the information processing device 10 in real time, but this is not limiting. The information processing device 10 does not have to display the corrected surface temperature on the screen of the output unit 15 of the information processing device 10 in real time. Alternatively, the information processing device 10 may display the corrected surface temperature on the screen of another device different from the information processing device 10 in real time.

[0103] In the above embodiment, the target object is described as a rubber product including a tire or a crawler, but is not limited to this. The target object may include any other object that is a target of estimation of the state of a target location by the information processing system 1. [Industrial Applicability]

[0104] According to the present disclosure, it is possible to provide an information processing method and an information processing device that improve the accuracy of a technique for determining the state of an object using an image of the object.

[0105] [Contribution to the United Nations-led Sustainable Development Goals (SDGs)] The SDGs have been proposed to realize a sustainable society. One embodiment of the present disclosure is believed to be a technology that can contribute to goals such as "No. 9 - Build infrastructure for industry and technological innovation," "No. 12 - Responsible consumption and production," and "No. 13 - Take concrete measures against climate change." [Explanation of symbols]

[0106] 1. Information Processing Systems 10. Information processing equipment 11 Communications Department 12 Storage section 13 Acquisition Department 14 Input section 15 Output section 16 Control Unit 20 Imaging device 21 Infrared camera 30 servers 31 Communications Department 32 Storage section 33 Control Unit 40 Measuring Equipment 50 Network C11, C12, C21, C22, C23 Damaged parts IM1, IM2, IM3, IM4, IM5, IM6 images S Object

Claims

1. A computer-implemented information processing method, comprising: Obtaining a thermal image of an object using an infrared camera; Identifying a position of a target portion of the object based on the acquired thermographic image; Obtaining a surface temperature of the target location based on the acquired thermographic image; Obtaining a distance from the infrared camera to the object; correcting the surface temperature based on the acquired distance; Including, Information processing methods.

2. 2. The information processing method according to claim 1, and estimating a state of the target location based on the identified position of the target location and the corrected surface temperature. Information processing methods.

3. 3. The information processing method according to claim 2, the target location includes a damaged portion of the target object; The condition of the target area includes the depth of the damaged portion. Information processing methods.

4. 4. The information processing method according to claim 1, correcting the surface temperature includes correcting the surface temperature using machine learning. Information processing methods.

5. 4. The information processing method according to claim 1, correcting the surface temperature includes correcting the surface temperature using at least one of a correction coefficient and a correction term determined based on a function including the distance as a variable; Information processing methods.

6. 4. The information processing method according to claim 1, acquiring the distance includes acquiring the distance using an AR function of an information processing device to which the infrared camera is attached. Information processing methods.

7. 7. The information processing method according to claim 6, further comprising displaying the corrected surface temperature on a screen of the information processing device in real time. Information processing methods.

8. 4. The information processing method according to claim 1, The object is a rubber product including a tire or a crawler. Information processing methods.

9. An information processing device, A control unit is provided, the control unit A thermographic image of the object is captured using an infrared camera, Identifying a position of a target portion of the object based on the acquired thermographic image; Obtaining a surface temperature of the target location based on the acquired thermographic image; Acquire the distance from the infrared camera to the object; correcting the surface temperature based on the acquired distance; Information processing device.

Images