Information processing method, information processing device, and computer program

Abstract

To provide an information processing method, an information processing device, and a computer program. [Solution] A computer executes a process of acquiring image data captured by an imaging device and measurement data measured by a measuring device communicatively connected to the imaging device, and storing the acquired measurement data in association with information about the measurement object identified based on the image data.

Description

[Technical Field]

[0001] The present invention relates to an information processing method, an information processing device, and a computer program. [Background technology]

[0002] Patent Document 1 discloses a measuring instrument reader that acquires an image obtained by photographing the identification features and measurement result display portion of a measuring instrument, and reads the model of the measuring instrument and the measurement results from the acquired image. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-152198 Summary of the Invention [Problem to be solved by the invention]

[0004] In the above-mentioned Patent Document 1, the model of the measuring instrument and the measurement results are identified from an image, but identification of the measurement target is not envisaged.

[0005] An object of the present disclosure is to provide an information processing method, an information processing device, and a computer program that associate and store measurement data measured by a measuring device with a measurement target identified by image data. [Means for solving the problem]

[0006] An information processing method according to a first aspect of the present disclosure includes a process performed by a computer to acquire image data captured by an imaging device and measurement data measured by a measuring device communicatively connected to the imaging device, and to store the acquired measurement data in association with information about a measurement object identified based on the image data.

[0007] An information processing method according to a second aspect of the present disclosure is the information processing method according to the first aspect, wherein the computer executes a process of storing information on the measurement object and information on measurement date and time in association with the measurement data.

[0008] An information processing method according to a third aspect of the present disclosure is the information processing method according to the first or second aspect, in which the computer executes a process of identifying a measurement target device or a measurement target location in the measurement target device as the measurement target.

[0009] An information processing method according to a fourth aspect of the present disclosure is an information processing method according to any one of the first to third aspects, in which the computer executes a process of identifying the type of the measuring device based on the image data and storing information on the identified type of measuring device in association with information on the object to be measured.

[0010] An information processing method according to a fifth aspect of the present disclosure is an information processing method according to any one of the first to fourth aspects, wherein the measurement object includes refrigeration and air conditioning-related equipment, and the measurement data includes at least one of measurement data of the refrigeration and air conditioning-related equipment: temperature, current, voltage, power, electrical resistance, air volume, air speed, pressure, vibration, frequency, humidity, and illuminance.

[0011] An information processing method according to a sixth aspect of the present disclosure is an information processing method according to any one of the first to fifth aspects, in which the computer executes a process of storing measurement data from the start of measurement by the measuring device to the end of measurement in association with information about the object to be measured.

[0012] An information processing method according to a seventh aspect of the present disclosure is the information processing method according to the sixth aspect, in which the computer executes a process of comparing the amount of change in the measurement data with a threshold value for the amount of change, and determining the measurement start time and the measurement end time based on the comparison result.

[0013] An information processing method according to an eighth aspect of the present disclosure is an information processing method according to the first to seventh aspects, in which a measurement target location to be measured by the measuring device is identified based on the image data, the accuracy of the identified measurement target location is determined, and if the measurement target location is determined to be accurate, the measurement data is stored in association with information about the measurement target identified based on the image data.

[0014] An information processing method according to a ninth aspect of the present disclosure is the information processing method according to the first to eighth aspects, in which the computer determines whether the measurement object needs to be repaired based on the measurement data, and if it is determined that repair is needed, outputs repair authorization information.

[0015] An information processing method according to a tenth aspect of the present disclosure is the information processing method according to the second aspect, in which the computer executes a process of generating a report by providing a language model with a prompt including stored information about the measurement object, the measurement data, information about the measurement date and time, and an instruction to create a report.

[0016] An information processing device relating to an eleventh aspect of the present disclosure includes at least one processing unit, which acquires image data captured by an imaging device and measurement data measured by a measuring device communicatively connected to the imaging device, and stores the acquired measurement data in association with information about a measurement target identified based on the image data.

[0017] A computer program according to a twelfth aspect of the present disclosure causes a computer to acquire image data captured by an imaging device and measurement data measured by a measuring device communicatively connected to the imaging device, and store the acquired measurement data in association with information about a measurement target identified based on the image data. [Effects of the Invention]

[0018] According to the present disclosure, measurement data measured by a measuring device can be stored in association with a measurement target identified by image data. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 2 is an explanatory diagram illustrating an outline of processing executed by the information processing system according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the internal configuration of the wearable device. [Figure 3] FIG. 2 is a block diagram showing the internal configuration of the measuring device. [Figure 4] FIG. 2 is a block diagram showing the internal configuration of the server device. [Figure 5] FIG. 1 is a schematic diagram illustrating an example of the configuration of a learning model. [Figure 6] FIG. 10 is a conceptual diagram illustrating an example of a database. [Figure 7] 4 is a flowchart illustrating a procedure of a process executed by a server device according to the first embodiment. [Figure 8] FIG. 10 is a schematic diagram illustrating an example of the configuration of a learning model that identifies the type of device being measured. [Figure 9] FIG. 10 is a conceptual diagram showing an example of a condition table that defines measurement conditions for a measurement object. [Figure 10] 10 is a flowchart illustrating a procedure of a process executed by a server device according to the second embodiment. [Figure 11] 11 is a flowchart illustrating a procedure of a process executed by a server device according to the third embodiment. [Figure 12] 10 is a flowchart illustrating a procedure of a process executed by a server device according to the fourth embodiment. [Figure 13] FIG. 10 is an explanatory diagram illustrating a method for creating a work report. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, an information processing system according to an embodiment will be specifically described with reference to the drawings. (Embodiment 1) 1 is an explanatory diagram outlining the processing executed by an information processing system 1 according to embodiment 1. The information processing system 1 according to the embodiment includes a wearable device 10 and a measuring device 20 carried by a worker, and a server device 30 communicably connected to at least one of the wearable device 10 and the measuring device 20.

[0021] In this embodiment, the worker is, for example, a worker who performs installation work, repair work, inspection work, etc. related to refrigeration and air conditioning related equipment. Refrigeration and air conditioning related equipment refers to equipment that is connected to refrigerant piping and constitutes a refrigeration cycle. Refrigeration and air conditioning related equipment includes equipment such as outdoor units, indoor units, ventilators, heat source equipment, and central control devices that constitute an air conditioning system, and equipment such as chillers, heat exchangers, and control devices that constitute a chiller system.

[0022] A worker wears a wearable device 10. The wearable device 10 is an example of an imaging device and has imaging and communication functions. The wearable device 10 is worn, for example, around the worker's neck. Alternatively, the wearable device 10 may be worn on the worker's head, or on another body part such as the shoulder or arm. The wearable device 10 may also be a goggle-type camera device, and any device having imaging and communication functions, such as a smartphone or action camera, may be used instead of the wearable device 10.

[0023] Wearable device 10 captures images of the work site while the worker is performing work (at least while measurement is being performed by measuring device 20). Measurements using measuring device 20 may be performed during work. In this case, the angle of view captured by wearable device 10 includes the measurement target measured by measuring device 20. The measurement target may be the refrigeration and air conditioning-related equipment itself, or a specific measurement location in the refrigeration and air conditioning-related equipment. For example, when measuring current, voltage, electrical resistance, power, etc., for indoor and outdoor units, the measurement target may be electrical components such as motors, thermistors, power supplies, and wiring that constitute the indoor and outdoor units. Furthermore, when measuring temperature, air volume, air speed, etc., for indoor and outdoor units, the measurement target may be a location slightly away from the air inlet or outlet of the indoor or outdoor unit.

[0024] The wearable device 10 captures an image of the work site to generate a captured image (image data) including the measurement target. The image captured by the wearable device 10 may be a video or a still image. When the captured image is a video, it is sufficient that the measurement target is included in at least some of the frames that make up the video.

[0025] The measuring device 20 measures a desired physical quantity of a measurement target and outputs measurement data indicating the measurement results. The physical quantities measured by the measuring device 20 include the aforementioned current, voltage, electrical resistance, power, temperature, air volume, and wind speed. The physical quantities measured by the measuring device 20 may also include pressure, vibration, frequency, humidity, and illuminance. In addition to the measurement function of measuring physical quantities, the measuring device 20 also has a communication function of communicating with the wearable device 10. The measuring device 20 outputs the measurement data obtained as a measurement result to the wearable device 10 via communication.

[0026] The wearable device 10 can communicate with the server device 30 via a communication network NW such as the Internet. The wearable device 10 uploads image data captured by the wearable device 10 and measurement data acquired from the measuring device 20 to the server device 30 via the communication network NW.

[0027] The server device 30 acquires image data captured by the wearable device 10 and measurement data measured by the measuring device 20 via the communication network NW. The server device 30 identifies the measurement object based on the image data received via the communication network NW, and stores the received measurement data and the identified measurement object in association with each other in the database DB.

[0028] In this embodiment, the measurement data measured by the measuring device 20 is stored in the database DB together with information about the object to be measured, so that the work manager can infer the measurement items by checking the contents stored in the database DB, and can easily understand which part of the refrigeration and air conditioning-related equipment was measured and what was measured.

[0029] In the present embodiment, the server device 30 is described as acquiring image data and measurement data from the wearable device 10, but the data acquisition path of the server device 30 is not limited to the above. For example, if the measuring device 20 has a communication function for communicating with the server device 30, the server device 30 may acquire image data from the wearable device 10 and acquire measurement data from the measuring device 20. The server device 30 may also acquire both image data and measurement data from the measuring device 20. Furthermore, the image data captured by the wearable device 10 and the measurement data measured by the measuring device 20 may be transferred to another terminal device (e.g., the worker's smartphone), and then the server device 30 may acquire the image data and measurement data via this terminal device.

[0030] 2 is a block diagram showing the internal configuration of wearable device 10. Wearable device 10 includes a processing unit 11, a memory unit 12, a first communication unit 13, a second communication unit 14, an imaging unit 15, a sound input unit 16, a sound output unit 17, a sensor unit 18, an operation unit 19, and the like.

[0031] The processing unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), etc. The ROM included in the processing unit 11 stores control programs and the like that control the operation of each hardware unit included in the wearable device 10. The CPU in the processing unit 11 reads and executes the control programs and the like stored in the ROM, and controls the operation of each hardware unit, thereby causing the entire device to function as the wearable device 10 of the present disclosure. The RAM included in the processing unit 11 temporarily stores data used during the execution of various processes.

[0032] The storage unit 12 includes an auxiliary storage device and stores image data generated by the imaging unit 15. An application program executed by the processing unit 11 may be installed in the storage unit 12. The application program may be installed in advance or may be installed after use has begun.

[0033] The first communication unit 13 includes a communication module for wireless communication with an external device such as the server device 30. The communication module of the first communication unit 13 is a communication module for wireless communication using a known mobile communication standard such as 3G, 4G, or 5G or a wireless LAN system such as WiFi (registered trademark). The first communication unit 13 communicates with the external device such as the server device 30 via the communication network NW, and transmits necessary data such as image data and receives appropriate data transmitted from the external device.

[0034] The second communication unit 14 includes a communication module for communicating with the measuring device 20. The communication module of the second communication unit 14 is a communication module for short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark). Alternatively, the communication module of the second communication unit 14 may be a communication module for wired communication such as RS485.

[0035] The imaging unit 15 includes an optical lens, an imaging element, a driver circuit, etc. A wide-angle lens is preferably used as the optical lens. The imaging element is a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge-Coupled Device), etc., and generates an electrical signal according to the intensity of light imaged through the optical lens. The driver circuit includes a timing generator (TG), etc., and sequentially reads out the electrical signals from the imaging element in synchronization with a clock signal output from the TG to generate image data. The image data generated by the imaging unit 15 is sent to the processing unit 11 and stored in the memory unit 12. Alternatively, the image data generated by the imaging unit 15 may be sequentially transmitted to the server device 30 via the first communication unit 13.

[0036] The sound input unit 16 includes a microphone for collecting sound, a processing circuit for converting the collected sound into a digital signal (sound data), etc. The sound data generated by the sound input unit 16 is sent to the processing unit 11, where appropriate processing such as noise removal is performed. The sound data generated by the sound input unit 16 is stored in the storage unit 12, or transmitted to the server device 30 via the first communication unit 13.

[0037] The sound output unit 17 includes a speaker that outputs sound. The sound output unit 17 outputs sound based on the acoustic data provided by the processing unit 11.

[0038] The sensor unit 18 includes a non-contact sensor for detecting the worker's fingers and the like. The sensors included in the sensor unit 18 include a proximity sensor, a gesture sensor, and the like. The proximity sensor detects, for example, when the worker's fingers approach within a predetermined range. The gesture sensor detects, for example, the movement of the worker's fingers. The detection result by the sensor unit 18 is notified to the processing unit 11. The processing unit 11 may issue an instruction to start capturing or stop capturing to the imaging unit 15 based on the detection result of the sensor unit 18.

[0039] The operation unit 19 has various operation buttons, operation switches, etc., and accepts operations by the operator. Operation information corresponding to the operation of the operation unit 19 is input to the processing unit 11. The processing unit 11 executes appropriate processing based on the operation information input from the operation unit 19. For example, the operation unit 19 may accept an operation to start measurement by the measuring device 20 or an operation to end measurement, and issue an instruction to start measurement or an instruction to end measurement to the processing unit 11.

[0040] 3 is a block diagram showing the internal configuration of the measuring device 20. The measuring device 20 includes a processing unit 21, a storage unit 22, a measuring unit 23, a communication unit 24, an operation unit 25, a display unit 26, and the like.

[0041] The processing unit 21 is a processing circuit configured with a CPU, ROM, RAM, etc., and controls the operation of each hardware unit of the device, thereby causing the entire device to function as the measurement device of the present disclosure. The storage unit 22 is a memory for temporarily storing measurement data obtained by the measurement unit 23.

[0042] The measuring unit 23 includes a sensor for measuring a physical quantity. The physical quantity measured by the measuring unit 23 varies depending on the type of the measuring device 20. For example, if the measuring device 20 is a device that measures current, the measuring unit 23 is provided with a sensor for measuring current, such as a Hall sensor, and if the measuring device 20 is a device that measures temperature, the measuring unit 23 is provided with a sensor for measuring temperature, such as a resistance thermometer or a thermocouple. The same applies when measuring other physical quantities.

[0043] The communication unit 24 includes a communication module for communicating with the wearable device 10. The communication module of the communication unit 24 is a communication module for short-range wireless communication such as Bluetooth (registered trademark) or ZigBee (registered trademark). Alternatively, the communication module of the communication unit 24 may be a communication module for wired communication such as RS485.

[0044] The operation unit 25 is equipped with various switches, buttons, etc. that accept operations by the operator. The operation unit 25 accepts instructions from the operator to start measurement, to end measurement, to send measurement data, etc. The display unit 26 is equipped with LED lamps, a liquid crystal display, etc. The display unit 26 displays status such as "measuring," "measurement completed," and "data sending," as well as the measurement results.

[0045] It should be noted that the operation unit 25 and the display unit 26 are not essential components of the measuring device 20. The measuring device 20 may automatically measure a target physical quantity and automatically transmit the obtained measurement data to the wearable device 10. When transmitting the measurement data to the wearable device 10, the measuring device 20 may also transmit information about the type of the measuring device 20 and identification information of the measuring device 20 to the wearable device 10.

[0046] 4 is a block diagram showing the internal configuration of the server device 30. The server device 30 is a dedicated or general-purpose server device, and includes a processing unit 31, a storage unit 32, a communication unit 33, an operation unit 34, a display unit 35, and the like.

[0047] The processing unit 31 includes a CPU, a ROM, a RAM, etc. The ROM included in the processing unit 31 stores a control program that controls the operation of each hardware unit included in the server device 30, etc. The CPU in the processing unit 31 reads and executes the control program stored in the ROM and a computer program (described below) stored in the storage unit 22, and executes processing to control the operation of each hardware unit, thereby causing the entire device to function as the server device 30 of the present disclosure. The RAM included in the processing unit 31 temporarily stores data used during the execution of various processes.

[0048] In the embodiment, the processing unit 31 is configured to include a CPU, a ROM, and a RAM, but the configuration of the processing unit 31 is not limited to the above. The processing unit 31 may be one or more processing circuits including, for example, a GPU (Graphics Processing Unit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), a quantum processor, volatile or non-volatile memory, etc. The processing unit 31 may also have functions such as a clock that outputs date and time information, a timer that measures the elapsed time from when a measurement start instruction is given until when a measurement end instruction is given, and a counter that counts numbers.

[0049] The storage unit 32 includes a storage device such as a hard disk drive (HDD), a solid state drive (SSD), etc. The storage unit 32 stores various computer programs executed by the processing unit 31 and various data acquired through the communication unit 33.

[0050] The computer program (program product) stored in the memory unit 32 includes an automatic labeling program PG1 that adds information about the measurement target to measurement data acquired through the wearable device 10 and stores the data in the memory unit 32. The automatic labeling program PG1 is a computer program that causes a computer to execute a process of acquiring image data captured by the wearable device 10 and measurement data measured by the measuring device 20, and storing the acquired measurement data in association with information about the measurement target identified based on the image data.

[0051] The computer programs stored in the memory unit 32 may further include an image recognition program PG2 for identifying the measurement target from the acquired image data, and a good measurement judgment program PG3 for judging whether the worker is measuring the correct measurement point based on the acquired image data.

[0052] The computer program including the automatic labeling program PG1 may be a single computer program or a group of programs consisting of multiple computer programs. Furthermore, the computer program including the automatic labeling program PG1 may be executed by a single computer or may be executed by multiple computers (for example, the wearable device 10 and the server device 30) working together.

[0053] A computer program including the automatic labeling program PG1 is provided by a non-transitory recording medium RM on which the computer program is readably recorded. The recording medium RM is a portable memory such as a CD-ROM, a USB memory, an SD card, a micro SD card, or a CompactFlash (registered trademark). The processing unit 31 reads various computer programs from the recording medium RM using a reading device (not shown) and stores the read various computer programs in the storage unit 32. The computer programs stored in the storage unit 32 may also be provided by communication. In this case, the processing unit 31 acquires the computer programs by communication via the communication unit 33 and stores the acquired computer programs in the storage unit 32.

[0054] The storage unit 32 may store a learning model MD1 used in the image recognition program PG2. FIG. 5 is a schematic diagram showing an example of the configuration of the learning model MD1. The learning model MD1 is a learning model based on machine learning, and when image data acquired from the wearable device 10 is input, it is trained to output information about the measurement target contained in the image. The learning model MD1 is constructed using a neural network for object detection, such as R-CNN (Region-based Convolutional Neural Networks), YOLO (You Only Look Once), or SSD (Single Shot Multi-Box Detector). Alternatively, the learning model MD1 may be constructed using any neural network capable of image segmentation, such as SegNet, U-Net (U-Shaped Network), or PSPNet (Pyramid Scene Parsing Network).

[0055] The learning model MD1 includes an input layer, an intermediate layer, and an output layer. Image data acquired by the wearable device 10 is input to the input layer. The intermediate layer includes a convolutional layer, a pooling layer, a fully connected layer, and the like. The calculations performed by the intermediate layer include extracting features from the input image, selecting candidate regions likely to be the detection target (in this embodiment, the measurement target of the measuring device 20) contained in the input image, and identifying the measurement target within the selected region. The output layer references the calculation results from the intermediate layer and outputs the detection results. The detection results include a bounding box surrounding the detected measurement target and an identification name (class name) identifying the measurement target within the bounding box. In this embodiment, the identification name is the name of an electrical component constituting a refrigeration / air-conditioning-related device, or the name of a measurement location such as an inlet or outlet. The detection results may also include information on the prediction reliability (certainty) of the detected measurement target.

[0056] The learning model MD1 is generated by preparing a dataset including image data containing various measurement targets, class names of the measurement targets contained in the image data, and coordinate values ​​of bounding boxes surrounding the measurement targets, and using the dataset as training data to perform machine learning with a predetermined algorithm. The machine learning may be performed inside the server device 30, or may be performed in an information processing device external to the server device 30. In the latter case, after performing machine learning in the external information processing device, the trained learning model MD1 may be acquired and stored in the memory unit 32. The machine learning algorithm itself is publicly known, so a detailed description thereof will be omitted.

[0057] In this embodiment, the measurement object is identified using the learning model MD1, but the measurement object may also be identified using an existing detection method that does not use a learning model, such as template matching.

[0058] The storage unit 32 further includes a database DB that stores measurement data measured by the measuring device 20 and information about the measurement target identified from the image data, in association with each other. FIG. 6 is a conceptual diagram showing an example of the database DB. The database DB stores measurement data measured by the measuring device 20 and information about the measurement target identified from the image data, in association with each other. In the example of FIG. 6, information about the target device that was the measurement target and information about the measurement location are stored as information about the measurement target, but either one of the pieces of information may be stored. In addition to the measurement data and information about the measurement target, the database DB may also store information about the device that was the measurement target, information about the measurement date and time, information about the type of measuring device 20, identification information for the wearable device 10 and the measuring device 20, the name of the worker who performed the work, and the like.

[0059] The measuring device 20 may be identified from image data, as with the measurement target, or may be notified from the wearable device 10. Information on the measurement date and time may be notified from the wearable device 10. Instead of information on the measurement date and time, information on the date and time when the measurement data was acquired may be stored in the database DB. Identification information for the wearable device 10 and the measuring device 20 may be notified from the wearable device 10. The worker's name may be stored in advance in the memory unit 32 in association with the identification information of the wearable device 10, and the processing unit 31 may read the worker's name from the memory unit 32 and register it in the database DB.

[0060] The communication unit 33 includes a communication module for wireless communication with an external device such as the wearable device 10. The communication module may be a communication module for wireless communication using a known mobile communication standard such as 3G, 4G, or 5G, or a wireless LAN system such as WiFi (registered trademark). The communication unit 33 communicates with an external device such as the wearable device 10 via the communication network NW, receives image data captured by the wearable device 10 and measurement data measured by the measuring device 20, and transmits appropriate data to be sent to the external device.

[0061] The operation unit 34 is equipped with operation devices such as a touch panel, a keyboard, and switches, and receives various inputs and operations from a work manager, etc. The processing unit 31 acquires information input through the operation unit 34 and performs appropriate control based on various operation information provided by the operation unit 34.

[0062] The display unit 35 includes a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor, and displays information to be notified to a work manager or the like in response to an instruction from the processing unit 31.

[0063] The server device 30 may be a single computer, or may be a computer system configured with multiple computers and peripheral devices, etc. The server device 30 may also be a virtual machine whose entity is virtualized, or may be a cloud.

[0064] The processing executed in the information processing system 1 will be described below. A worker who starts work at a work site puts on the wearable device 10 on his or her body. The wearable device 10 automatically starts capturing images at an appropriate timing after the worker puts on the wearable device 10. Alternatively, the wearable device 10 starts capturing images when instructed by the worker or the server device 30.

[0065] A worker performs installation, repair, inspection, etc. of refrigeration and air conditioning-related equipment, and, as necessary, performs measurements using measuring device 20. The physical quantities measured include at least one of the temperature, current, voltage, power, electrical resistance, air volume, air speed, pressure, vibration, frequency, humidity, and illuminance of the refrigeration and air conditioning-related equipment. Measuring device 20 transmits measurement data obtained by measuring the physical quantities to wearable device 10. Wearable device 10 transmits image data obtained by imaging unit 15 and measurement data received from measuring device 20 to server device 30. Wearable device 10 may transmit image data and measurement data to server device 30 in real time during work, or may transmit image data and measurement data to server device 30 after work is completed.

[0066] The processing unit 31 of the server device 30 reads out and executes programs such as an automatic labeling program PG1, an image recognition program PG2, and a good measurement determination program PG3 at appropriate timing from the storage unit 32. The processing unit 31 executes the following processes in accordance with these programs.

[0067] 7 is a flowchart illustrating the procedure of processing executed by server device 30 according to embodiment 1. Communication unit 33 of server device 30 receives image data and measurement data transmitted from wearable device 10 via communication network NW (step S101). Communication unit 33 outputs the received image data and measurement data to processing unit 31.

[0068] The processing unit 31 inputs the image data acquired through the communication unit 33 to the learning model MD1 and executes calculations using the learning model MD1 (step S102). Specifically, the processing unit 31 inputs the acquired image data to the input layer of the learning model MD1, and executes, in the intermediate layer, a process of extracting features from the input image, a process of selecting candidates for an area likely to be the detection target (the object to be measured by the measuring device 20) included in the input image, a process of identifying the measurement target within the selected area, and the like, and then outputs the detection results from the output layer.

[0069] The processing unit 31 identifies the measurement object based on the calculation result by the learning model MD1 (step S103). The learning model MD1 outputs, as the calculation result, information on the area surrounding the detected measurement object and an identification name (class name) that identifies the measurement object within that area. The processing unit 31 identifies the measurement object (at least one of the equipment that became the measurement object and the measurement location) based on the output of the learning model MD1.

[0070] The processing unit 31 associates the measurement data acquired through the communication unit 33 with the information on the measurement target identified in step S103 and stores them in the database DB (step S104). The database DB illustrated in FIG. 6 includes items for the target device, the measurement target, and the measurement result. Therefore, the processing unit 31 simply registers the information on the measurement target identified in step S103 in the items for the target device and the measurement target, and registers the acquired measurement data in the item for the measurement result. The measurement data may be a single measurement value or multiple measurement values ​​measured in time series. If the measurement data is created as a data file, a link to the data file may be registered in the database DB. The processing unit 31 may acquire information on the type of measuring device 20 and information on the measurement date and time via the wearable device 10 along with the measurement data, and register the acquired information in the database DB.

[0071] As described above, in embodiment 1, the measurement data measured by the measuring device 20 is stored in association with the measurement object identified from the image data captured by the wearable device 10, so that by checking the information registered in the database DB, the measurement items can be inferred and it is easy to understand which part of the refrigeration and air conditioning-related equipment was measured and what was measured.

[0072] In this embodiment, the measurement object is identified using learning model MD1, but it is also possible to prepare a first learning model for identifying the equipment that is the measurement object, such as an outdoor unit or indoor unit, from image data, and a second learning model for identifying the measurement location by measuring device 20 from the image in which the target equipment is identified, and use these two learning models to identify the measurement object.

[0073] In this embodiment, information on the type of measuring device 20 is acquired from the wearable device 10. However, since the angle of view captured by the wearable device 10 includes the measuring device 20, the type of measuring device 20 may be identified based on the image data. FIG. 8 is a schematic diagram showing an example configuration of a learning model MD2 that identifies the type of measuring device 20. The configuration of the learning model MD2 is similar to that of the learning model MD1, and is constructed using a neural network for object detection such as R-CNN or a neural network for image segmentation such as SegNet, and includes an input layer, an intermediate layer, and an output layer. When image data captured by the wearable device 10 is input, the learning model MD2 is trained to output information on the type of measuring device 20 included in the image.

[0074] In the present embodiment, the measurement target is identified based on image data captured by the wearable device 10. However, the measurement target may also be identified based on the voice of the worker input through the sound input unit 16. For example, when the voice of the worker is input through the sound input unit 16 saying, "I will now measure the intake temperature of the outdoor unit," the processing unit 11 can identify the target device as the "outdoor unit" and the measurement location as the "intake port of the outdoor unit" using existing voice recognition technology. The processing unit 11 may also identify the measuring device 20 based on the voice of the worker. In the above example, the processing unit 11 can recognize that temperature measurement will be performed, and therefore can identify the measuring device 20 as a temperature sensor.

[0075] (Embodiment 2) In the second embodiment, a configuration will be described in which it is determined whether the measurement location by the measuring device 20 is correct, and if it is determined that the measurement location is correct, the measurement data and information on the measurement target are associated and registered in the database DB.

[0076] In the embodiment, measurement conditions are defined for each measurement object. FIG. 9 is a conceptual diagram showing an example of a condition table TB in which measurement conditions for a measurement object are defined. In the condition table TB, measurement conditions are defined for each measurement content. For example, if the measurement content is "intake temperature of the outdoor unit," "within 10 cm from the intake port" is defined as the measurement condition. Also, if the measurement content is "outlet temperature of the outdoor unit," "within 30 cm from the outlet port" is defined as the measurement condition. Various measurement conditions for such measurement content are defined in the condition table TB, and are stored in the memory unit 32 of the server device 30.

[0077] The server device 30 determines whether the measurement location by the measuring device 20 is correct by referring to the condition table TB stored in the storage unit 32. Figure 10 is a flowchart explaining the procedure of the process executed by the server device 30 according to the second embodiment. The communication unit 33 of the server device 30 receives the image data and measurement data transmitted from the wearable device 10 via the communication network NW (step S201). The communication unit 33 outputs the received image data and measurement data to the processing unit 31.

[0078] The processing unit 31 inputs the image data acquired through the communication unit 33 into the learning model MD1 in the same procedure as in embodiment 1, and identifies the object to be measured by performing calculations using the learning model MD1 (steps S202 to S203).

[0079] Based on the measurement target identified in step S203, the processing unit 31 refers to the condition table TB and determines whether the measurement by the measuring device 20 was performed correctly (step S204). For example, if the measurement target identified in step S203 is the air inlet of an outdoor unit and the measurement data received in step S201 is temperature data, the measurement content can be determined to be "the air inlet temperature of the outdoor unit." Therefore, the processing unit 31 refers to the condition table TB and determines whether the measuring device 20 was within 10 cm of the air inlet. Based on the image data received together with the measurement data, the processing unit 31 can determine whether the measuring device 20 was within 10 cm of the air inlet. For example, if the dimensions of the measurement target, such as the outdoor unit, the air inlet, and the measuring device 20, are known, the processing unit 31 can estimate the distance between the air inlet and the measuring device 20 from the image. An existing method can be used to estimate the distance from the image.

[0080] If it is determined that the measurement by the measuring device 20 is not being performed correctly (S204: NO), the processing unit 31 notifies the wearable device 10 of this fact (step S205). When the wearable device 10 receives a notification from the server device 30 that the measurement is not being performed correctly, the sound output unit 17 notifies the worker of this fact by voice.

[0081] If it is determined that the measurement by the measuring device 20 is being performed correctly (S204: YES), the processing unit 31 associates the measurement data acquired through the communication unit 33 with the information on the measurement subject identified in step S203 and stores them in the database DB (step S206). Even if the processing unit 31 determines that the measurement by the measuring device 20 is being performed correctly, it may notify the wearable device 10.

[0082] As described above, in the second embodiment, registration in the database DB is performed only when the measurement is performed correctly, and if the measurement is not performed correctly, the operator is notified of this and prompted to perform the correct measurement.

[0083] (Embodiment 3) In the third embodiment, when the measurement data obtained by the measuring device 20 is time-series data, a configuration will be described in which the measurement data from the start of measurement to the end of measurement is registered in the database DB.

[0084] 11 is a flowchart illustrating the procedure of processing executed by server device 30 according to embodiment 3. Communication unit 33 of server device 30 receives image data and measurement data transmitted from wearable device 10 via communication network NW (step S301). Communication unit 33 outputs the received image data and measurement data to processing unit 31.

[0085] The processing unit 31 inputs the image data acquired through the communication unit 33 into the learning model MD1 in the same procedure as in embodiment 1, and identifies the object to be measured by performing calculations using the learning model MD1 (steps S302 to S303).

[0086] The processing unit 31 determines the measurement start time and measurement end time for the measurement data received in step S301 (step S304). For example, the processing unit 31 can examine changes in the measurement data and determine the time when the measurement value crosses the first set value as the measurement start time and the time when it crosses the second set value as the measurement end time. The first set value and the second set value are set appropriately depending on the measurement content. Alternatively, operations on the measuring device 20 may be recorded, and the time when the measurement button is pressed may be determined as the measurement start time and the time when the measurement button is released as the measurement end time. Furthermore, the time when an instruction to start measurement is given using the operation unit 19 of the wearable device 10 may be determined as the measurement start time and the time when an instruction to end measurement is given as the measurement end time.

[0087] The processing unit 31 associates the measurement data acquired through the communication unit 33 from the start of measurement to the end of measurement with the information on the measurement target identified in step S303 and stores it in the database DB (step S305).

[0088] As described above, in the third embodiment, only the net measurement data can be stored in the database DB, so that the amount of data stored in the database DB can be reduced.

[0089] (Fourth embodiment) In the fourth embodiment, a configuration will be described in which it is determined whether or not the measurement object needs to be repaired, and if repair is necessary, repair permission information is output.

[0090] 12 is a flowchart illustrating the procedure of processing executed by server device 30 according to embodiment 4. Communication unit 33 of server device 30 receives image data and measurement data transmitted from wearable device 10 via communication network NW (step S401). Communication unit 33 outputs the received image data and measurement data to processing unit 31.

[0091] The processing unit 31 inputs the image data acquired through the communication unit 33 into the learning model MD1 in the same procedure as in embodiment 1, and identifies the object to be measured by performing calculations using the learning model MD1 (steps S402 to S403).

[0092] The processing unit 31 determines whether the measurement object needs to be repaired based on the measurement data received in step S401 (step S404). The processing unit 31 determines whether repair is needed based on whether the measurement value measured by the measuring device 20 exceeds a threshold. The threshold can be set appropriately depending on the measurement object.

[0093] If the processing unit 31 determines that the measurement object needs to be repaired (S404: YES), it outputs repair permission information (step S405). The processing unit 31 notifies the wearable device 10 of the repair permission information, and notifies the worker via the wearable device 10 that repair permission has been granted. The repair permission information may be registered in the database DB.

[0094] If the processing unit 31 determines that the measurement object should not be repaired (S404: NO), or if it outputs repair permission information in step S405, it associates the measurement data acquired through the communication unit 33 with the information on the measurement object identified in step S403 and stores them in the database DB (step S406).

[0095] (Embodiment 5) In the fifth embodiment, a configuration for creating a work report based on data stored in a database DB will be described.

[0096] FIG. 13 is an explanatory diagram illustrating a method for creating a work report. A processing unit 31 of a server device 30 receives, from a worker or a work manager, a specification of a record to be used for creating a report from among records stored in a database DB, and generates a prompt that instructs the LLM server to create a report based on the specified record. The LLM server is an existing server computer that provides services using a large language model (LLM). The LLM server includes a language model that generates a response sentence to an input prompt. The language model is, for example, an existing large-scale language model such as GPT-4 (Generative Pretrained Transformer 4), LLaMA (Large Language Model Meta AI), or BERT (Bidirectional Encoder Representations from Transformers).

[0097] The example in Figure 13 shows an example in which a prompt is generated that instructs the user to create a work report using the record dated November 8, 2022, among the records stored in the database DB. The server device 30 creates the report by sending such a prompt to the LLM server. The text included in the report is generated by the LLM server and returned from the LLM server to the server device 30 as a response to the prompt.

[0098] As described above, in the fifth embodiment, a work report can be automatically created.

[0099] Furthermore, as an application example of the information processing system according to this embodiment, a work scene involving refrigeration and air conditioning-related equipment has been envisaged, but it goes without saying that the system can be applied to a variety of work scenes, not limited to work scenes involving refrigeration and air conditioning-related equipment, such as work scenes involving elevators, and work scenes relating to maintenance and inspection at chemical plants and power supply facilities.

[0100] The embodiments disclosed herein should be considered in all respects as illustrative and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims. [Explanation of symbols]

[0101] 10. Wearable Devices 20 Measuring Devices 30 Analysis Server 31 Processing section 32 Storage section 33 Communications Department 34 Control section 35 Display section PG1 Automated Labeling Program PG2 Image Recognition Program PG3 Good Measurement Judgment Program MD1 learning model DB Database

Claims

1. Acquire image data captured by an imaging device and measurement data measured by a measuring device communicably connected to the imaging device; The acquired measurement data is stored in association with information on the measurement target identified based on the image data. An information processing method for executing processing by a computer, comprising: The measurement object includes refrigeration and air conditioning related equipment, The measurement data includes at least one of temperature, current, voltage, power, electrical resistance, air volume, air speed, pressure, vibration, frequency, humidity, and illuminance of the refrigeration and air conditioning related equipment. Information processing methods.

2. The information on the measurement object and the measurement date and time are stored in association with the measurement data.

2. The information processing method according to claim 1, wherein the processing is executed by the computer.

3. As the measurement target, a measurement target device or a measurement target location in the measurement target device is identified.

2. The information processing method according to claim 1, wherein the processing is executed by the computer.

4. Identifying the type of the measuring device based on the image data; The information on the type of the specified measuring device is stored in association with the information on the measurement target.

2. The information processing method according to claim 1, wherein the processing is executed by the computer.

5. The measurement data from the start of measurement by the measuring device to the end of measurement is stored in association with information about the measurement object.

2. The information processing method according to claim 1, wherein the processing is executed by the computer.

6. comparing the amount of change in the measurement data with a threshold value for the amount of change; The measurement start time and the measurement end time are determined based on the comparison result.

6. The information processing method according to claim 5, wherein the processing is executed by the computer.

7. Identifying a measurement target location by the measuring device based on the image data; Determine whether the identified measurement target location is correct or not; If the measurement target location is determined to be correct, the measurement data is stored in association with information on the measurement target identified based on the image data. The information processing method according to claim 1 .

8. determining whether or not the measurement object needs to be repaired based on the measurement data; If it is determined that repair is necessary, repair authorization information is output.

2. The information processing method according to claim 1, wherein the processing is executed by the computer.

9. A report is generated by providing a language model with a prompt including the stored information on the measurement object, the measurement data, and the measurement date and time, as well as an instruction to generate a report.

3. The information processing method according to claim 2, wherein the processing is executed by the computer.

10. At least one processing unit; The processing unit Acquire image data captured by an imaging device and measurement data measured by a measuring device communicably connected to the imaging device; The acquired measurement data is stored in association with information on the measurement target identified based on the image data. An information processing device, The measurement object includes refrigeration and air conditioning related equipment, The measurement data includes at least one of temperature, current, voltage, power, electrical resistance, air volume, air speed, pressure, vibration, frequency, humidity, and illuminance of the refrigeration and air conditioning related equipment. Information processing device.

11. Acquire image data captured by an imaging device and measurement data measured by a measuring device communicably connected to the imaging device; The acquired measurement data is stored in association with information on the measurement target identified based on the image data. A computer program for causing a computer to execute a process, The measurement object includes refrigeration and air conditioning related equipment, The measurement data includes at least one of temperature, current, voltage, power, electrical resistance, air volume, air speed, pressure, vibration, frequency, humidity, and illuminance of the refrigeration and air conditioning related equipment. Computer program.

Images