Information processing method, information processing device, and computer program

Abstract

Providing information processing methods, information processing devices, and computer programs. [Solution] The system acquires image data captured by an imaging device and measurement data measured by a measuring device that is communicatively connected to the imaging device. A computer then performs a process to store the acquired measurement data in association with information about the object to be measured, which is identified based on the image data.

Description

Technical Field

[0001] The present invention relates to an information processing method, an information processing apparatus, and a computer program.

Background Art

[0002] Patent Document 1 discloses a measuring instrument reading device that acquires an image obtained by photographing the identification features of a measuring instrument and a measurement result display part, and reads the model of the measuring instrument and the measurement result from the acquired image.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the above Patent Document 1, the model of the measuring instrument and the measurement result are specified from the image, but specifying the measurement object is not assumed.

[0005] An object of the present disclosure is to provide an information processing method, an information processing apparatus, and a computer program that associate and store measurement data measured by a measurement device with a measurement object specified from image data.

Means for Solving the Problems

[0006] An information processing method according to a first aspect of the present disclosure acquires image data captured by an imaging device and measurement data measured by a measurement device communicably connected to the imaging device, and causes a computer to execute a process of associating and storing the acquired measurement data with information on a measurement object specified based on the image data.

[0007] It should be noted that there seems to be a mistake in the original text where "

特許文献

特許文献

特許文献1

Patent Documents

[0008] The information processing method relating to the third aspect of this disclosure is an information processing method relating to the first or second aspect, wherein the computer performs a process to identify the measurement target device or the measurement target location in the measurement target device as the measurement target.

[0009] The information processing method relating to the fourth aspect of this disclosure is an information processing method relating to any one of the first to third aspects, wherein the computer performs a process of identifying the type of measuring device based on the image data and storing the information of the identified type of measuring device in association with the information of the object to be measured.

[0010] The information processing method relating to the fifth aspect of this disclosure is an information processing method relating to any one of the first to fourth aspects, wherein the measurement target includes refrigeration and air conditioning equipment, and the measurement data includes at least one measurement data of temperature, current, voltage, power, electrical resistance, airflow rate, wind speed, pressure, vibration, frequency, humidity, and illuminance in the refrigeration and air conditioning equipment.

[0011] The information processing method relating to the sixth aspect of this disclosure is an information processing method relating to any one of the first to fifth aspects, wherein the computer performs a process to store measurement data from the start time of measurement by the measuring device to the end time of measurement, associating it with information of the object to be measured.

[0012] The information processing method relating to the seventh aspect of this disclosure includes, in the information processing method relating to the sixth aspect, a process in which the computer compares the amount of change in the measurement data with a threshold value for the amount of change, and determines the start time of the measurement and the end time of the measurement based on the comparison result.

[0013] The information processing method relating to the eighth aspect of this disclosure includes, in the information processing method relating to the first to seventh aspects, identifying a measurement target location by the measuring device based on the image data, determining whether the identified measurement target location is correct, and, if the measurement target location is determined to be correct, storing the measurement data in association with the information of the measurement target identified based on the image data.

[0014] The information processing method relating to the ninth aspect of this disclosure includes, in the information processing methods relating to the first to eighth aspects, the computer performs a process to determine whether the measured object needs to be repaired based on the measurement data, and if it is determined that repair is necessary, it outputs repair permission information.

[0015] The information processing method relating to the tenth aspect of this disclosure involves the computer performing a process to generate a report by providing a language model with the stored information of the object to be measured, the measurement data, the information of the measurement date and time, and a prompt including a command to create a report, in the information processing method relating to the second aspect.

[0016] An information processing device relating to the eleventh aspect of this disclosure comprises at least one processing unit, the processing unit acquires image data captured by an imaging device and measurement data measured by a measuring device communicably connected to the imaging device, and stores the acquired measurement data in association with information of a measurement target identified based on the image data.

[0017] A computer program relating to the twelfth aspect of this disclosure causes a computer to perform a process of acquiring image data captured by an imaging device and measurement data measured by a measuring device communicatively connected to the imaging device, and storing the acquired measurement data in association with information of a measurement target identified based on the image data. [Effects of the Invention]

[0018] According to this disclosure, measurement data measured by a measuring device can be associated with and stored in relation to the measurement target identified from the image data.

Brief Description of the Drawings

[0019] [Figure 1] It is an explanatory diagram for explaining the outline of the processing executed by the information processing system according to Embodiment 1. [Figure 2] It is a block diagram showing the internal configuration of the wearable device. [Figure 3] It is a block diagram showing the internal configuration of the measurement device. [Figure 4] It is a block diagram showing the internal configuration of the server device. [Figure 5] It is a schematic diagram showing a configuration example of the learning model. [Figure 6] It is a conceptual diagram showing an example of the database. [Figure 7] It is a flowchart for explaining the procedure of the processing executed by the server device according to Embodiment 1. [Figure 8] It is a schematic diagram showing a configuration example of the learning model for identifying the type of the measurement device. [Figure 9] It is a conceptual diagram showing an example of the condition table that defines the measurement conditions for the measurement target. <00OO095>It is a flowchart for explaining the procedure of the processing executed by the server device according to Embodiment 2. [Figure 11] It is a flowchart for explaining the procedure of the processing executed by the server device according to Embodiment 3. [Figure 12] It is a flowchart for explaining the procedure of the processing executed by the server device according to Embodiment 4. [Figure 13] It is an explanatory diagram for explaining the method of creating the work report.

Modes for Carrying Out the Invention

[0020] Hereinafter, the information processing system according to the embodiment will be specifically described based on the drawings. (Embodiment 1) Figure 1 is an explanatory diagram illustrating the outline of the processing performed by the information processing system 1 according to Embodiment 1. The information processing system 1 according to Embodiment 1 includes a wearable device 10 and a measuring device 20 carried by an operator, and a server device 30 that is communicatively connected to at least one of the wearable device 10 and the measuring device 20.

[0021] In this embodiment, the worker is, for example, a worker who performs installation, repair, and inspection work related to refrigeration and air conditioning equipment. Refrigeration and air conditioning equipment refers to equipment connected to refrigerant piping and constituting a refrigeration cycle. Refrigeration and air conditioning equipment includes equipment such as outdoor units, indoor units, ventilation systems, heat source equipment, and central control devices that constitute an air conditioning system, and equipment such as refrigerators, heat exchangers, and control devices that constitute a refrigeration system.

[0022] The worker is fitted with a wearable device 10. The wearable device 10 is an example of an imaging device and has imaging and communication functions. The wearable device 10 is fitted, for example, around the worker's neck. Alternatively, the wearable device 10 may be fitted to the worker's head, or to another body part such as the shoulder or arm. The wearable device 10 may also be a goggle-type camera device, or any device with imaging and communication functions, such as a smartphone or action camera, may be used instead of the wearable device 10.

[0023] The wearable device 10 captures images of the work site while the worker is performing their duties (at least while measurements are being taken by the measuring device 20). Measurements using the measuring device 20 may be performed during the work. In this case, the field of view captured by the wearable device 10 shall include the object to be measured by the measuring device 20. The object to be measured may be the refrigeration and air conditioning equipment itself, or it may be a specific measurement point on the refrigeration and air conditioning equipment. For example, when measuring current, voltage, electrical resistance, power, etc., for indoor and outdoor units, the object to be measured may be electrical components such as motors, thermistors, power supplies, and wiring that make up the indoor and outdoor units. Also, when measuring temperature, airflow, wind speed, etc., for indoor and outdoor units, the object to be measured may be a point slightly away from the intake or outlet of the indoor or outdoor unit.

[0024] The wearable device 10 generates captured images (image data) that include the object to be measured by capturing images of the work site. The images captured by the wearable device 10 may be videos or still images. If the captured images are videos, it is sufficient that the object to be measured is included in at least some of the frames that make up the video.

[0025] The measuring device 20 measures a desired physical quantity of the object to be measured and outputs measurement data showing the measurement result. The physical quantities measured by the measuring device 20 include the aforementioned current, voltage, electrical resistance, power, temperature, airflow, and wind speed. The physical quantities measured by the measuring device 20 may also include pressure, vibration, frequency, humidity, and illuminance. In addition to the measurement function for measuring physical quantities, the measuring device 20 has a communication function for communicating with the wearable device 10. The measuring device 20 outputs the measurement data obtained as a measurement result to the wearable device 10 via communication.

[0026] The wearable device 10 can communicate with the server device 30 via a communication network NW, such as the Internet. The wearable device 10 uploads image data captured by itself and measurement data acquired from the measurement device 20 to the server device 30 via the communication network NW.

[0027] The server device 30 acquires image data captured by the wearable device 10 and measurement data measured by the measurement device 20 via the communication network NW. Based on the image data received via the communication network NW, the server device 30 identifies the measurement target, associates the received measurement data with the identified measurement target, and stores the information in the database DB.

[0028] In this embodiment, the measurement data measured by the measuring device 20 is stored in the database DB along with information about the object being measured. Therefore, the work manager can infer the measurement items by checking the contents of the database DB, and easily understand which parts of the refrigeration and air conditioning equipment were measured and what was measured.

[0029] In this embodiment, the server device 30 is described as acquiring image data and measurement data from the wearable device 10, but the data acquisition path for the server device 30 is not limited to the above. For example, if the measurement device 20 has a communication function to communicate with the server device 30, the server device 30 may acquire image data from the wearable device 10 and measurement data from the measurement device 20. Alternatively, the server device 30 may acquire both image data and measurement data from the measurement device 20. Furthermore, the image data captured by the wearable device 10 and the measurement data measured by the measurement device 20 may be transferred to another terminal device (for example, the worker's smartphone), and then the server device 30 may acquire the image data and measurement data via this terminal device.

[0030] Figure 2 is a block diagram showing the internal configuration of the wearable device 10. The wearable device 10 includes a processing unit 11, a storage unit 12, a first communication unit 13, a second communication unit 14, an imaging unit 15, an audio input unit 16, an audio output unit 17, a sensor unit 18, an operation unit 19, and the like.

[0031] The processing unit 11 includes a CPU (Central Processing Unit), ROM (Read Only Memory), RAM (Random Access Memory), and the like. The ROM in the processing unit 11 stores control programs and the like that control the operation of each hardware component of the wearable device 10. The CPU in the processing unit 11 reads and executes the control programs and the like stored in the ROM, and controls the operation of the hardware components, thereby making the entire device function as the wearable device 10 of this disclosure. The RAM in the processing unit 11 temporarily stores data used during the execution of various processes.

[0032] The storage unit 12 is equipped with an auxiliary storage device and stores image data and other data generated by the imaging unit 15. The storage unit 12 may also have an application program installed that is executed by the processing unit 11. The application program may be pre-installed or installed after use begins.

[0033] The first communication unit 13 is equipped with a communication module for wireless communication with external devices such as the server device 30. The communication module of the first communication unit 13 utilizes a communication module for wireless communication using known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as WiFi (registered trademark). The first communication unit 13 communicates with external devices such as the server device 30 via a communication network NW, transmitting necessary data such as image data, and receiving appropriate data transmitted from the external devices.

[0034] The second communication unit 14 includes a communication module for communicating with the measuring device 20. The communication module of the second communication unit 14 may utilize a short-range wireless communication module such as Bluetooth® or ZigBee®. Alternatively, the communication module of the second communication unit 14 may utilize a wired communication module such as RS485.

[0035] The imaging unit 15 includes an optical lens, an image sensor, a driver circuit, and the like. A wide-angle lens is preferably used as the optical lens. The image sensor is a CMOS (Complementary Metal Oxide Semiconductor), a CCD (Charge-Coupled Device), etc., and generates electrical signals according to the intensity of light formed through the optical lens. The driver circuit includes a timing generator (TG), etc., and sequentially reads electrical signals from the image sensor in synchronization with the clock signal output from the TG to generate image data. The image data generated by the imaging unit 15 is sent to the processing unit 11 and stored in the storage unit 12. Alternatively, the image data generated by the imaging unit 15 may be transmitted sequentially to the server device 30 via the first communication unit 13.

[0036] The sound input unit 16 includes a microphone for collecting sound, a processing circuit for converting the collected sound into a digital signal (acoustic data), and the like. The acoustic data generated in the sound input unit 16 is sent to the processing unit 11, where appropriate processing such as noise reduction is performed. The acoustic data generated in the sound input unit 16 is also stored in the storage unit 12 or transmitted to the server device 30 via the first communication unit 13.

[0037] The sound output unit 17 is equipped with a speaker that outputs sound. The sound output unit 17 outputs sound based on acoustic data provided by the processing unit 11.

[0038] The sensor unit 18 is equipped with non-contact sensors for detecting the hands and fingers of an operator. The sensors in the sensor unit 18 include proximity sensors and gesture sensors. For example, the proximity sensor detects when an operator's hands and fingers come within a predetermined range. For example, the gesture sensor detects the movement of an operator's hands and fingers. The detection results from the sensor unit 18 are notified to the processing unit 11. Based on the detection results from the sensor unit 18, the processing unit 11 may give an instruction to start imaging or an instruction to stop imaging to the imaging unit 15.

[0039] The control unit 19 is equipped with various operation buttons, operation switches, etc., and receives operations from the operator. Operation information corresponding to the operation of the control unit 19 is input to the processing unit 11. The processing unit 11 performs appropriate processing based on the operation information input from the control unit 19. For example, the control unit 19 may receive an operation to start measurement by the measuring device 20 or an operation to end measurement, and give a measurement start instruction or measurement end instruction to the processing unit 11.

[0040] Figure 3 is a block diagram showing the internal configuration of the measuring device 20. The measuring device 20 includes a processing unit 21, a storage unit 22, a measuring unit 23, a communication unit 24, an operation unit 25, a display unit 26, and the like.

[0041] The processing unit 21 is a processing circuit composed of a CPU, ROM, RAM, etc., and controls the operation of each hardware component of the device, thereby enabling the entire device to function as a measuring device in this disclosure. The storage unit 22 is a memory for temporarily storing measurement data obtained from the measurement unit 23.

[0042] The measuring unit 23 is equipped with sensors for measuring physical quantities. The physical quantities measured by the measuring unit 23 vary depending on the type of measuring device 20. For example, if the measuring device 20 is a device that measures electric current, the measuring unit 23 is equipped with sensors for measuring electric current, such as a Hall sensor. If the measuring device 20 is a device that measures temperature, the measuring unit 23 is equipped with sensors for measuring temperature, such as a resistance thermometer or thermocouple. The same applies when measuring other physical quantities.

[0043] The communication unit 24 includes a communication module for communicating with the wearable device 10. The communication module of the communication unit 24 may utilize a short-range wireless communication module such as Bluetooth® or ZigBee®. Alternatively, the communication module of the communication unit 24 may utilize a wired communication module such as RS485.

[0044] The control unit 25 is equipped with various switches, buttons, etc., to receive operator input. The control unit 25 receives operator commands to start measurement, end measurement, transmit measurement data, etc. The display unit 26 is equipped with LED lamps, a liquid crystal display, etc. The display unit 26 displays statuses such as "measurement in progress," "measurement completed," and "data transmission in progress," as well as the measurement results.

[0045] Note that the operation unit 25 and the display unit 26 are not essential components of the measuring device 20. The measuring device 20 may automatically measure the target physical quantity and automatically transmit the obtained measurement data to the wearable device 10. In addition, when the measuring device 20 transmits the measurement data to the wearable device 10, it may also transmit information about the type of measuring device 20 and identification information of the measuring device 20 to the wearable device 10.

[0046] Figure 4 is a block diagram showing the internal configuration of the server device 30. The server device 30 is a dedicated or general-purpose server device and includes a processing unit 31, a storage unit 32, a communication unit 33, an operation unit 34, a display unit 35, and the like.

[0047] The processing unit 31 includes a CPU, ROM, RAM, etc. The ROM in the processing unit 31 stores control programs and the like that control the operation of each hardware component of the server device 30. The CPU in the processing unit 31 reads and executes the control programs stored in the ROM and the computer programs described later stored in the memory unit 22, and by executing processes that control the operation of the hardware components, the entire device functions as the server device 30 of this disclosure. The RAM in the processing unit 31 temporarily stores data used during the execution of various processes.

[0048] In this embodiment, the processing unit 31 is configured to include a CPU, ROM, and RAM, but the configuration of the processing unit 31 is not limited to the above. The processing unit 31 may be one or more processing circuits, for example, comprising a GPU (Graphics Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), quantum processor, volatile or non-volatile memory, etc. Furthermore, the processing unit 31 may include functions such as a clock that outputs date and time information, a timer that measures the elapsed time from the time a measurement start instruction is given to the time a measurement end instruction is given, and a counter that counts numbers.

[0049] The storage unit 32 is equipped with a storage device such as an HDD (Hard Disk Drive) or SSD (Solid State Drive). The storage unit 32 stores various computer programs executed by the processing unit 31 and various data acquired through the communication unit 33.

[0050] The computer program (program product) stored in the memory unit 32 includes an automatic labeling program PG1 that assigns information about the object to be measured to the measurement data acquired through the wearable device 10 and stores it in the memory unit 32. The automatic labeling program PG1 is a computer program that causes a computer to perform a process of acquiring image data captured by the wearable device 10 and measurement data measured by the measurement device 20, and storing the acquired measurement data in association with information about the object to be measured identified based on the image data.

[0051] The computer program stored in the memory unit 32 may further include an image recognition program PG2 for identifying the measurement target from the acquired image data, and a good measurement judgment program PG3 for determining whether the operator is measuring the correct measurement location based on the acquired image data.

[0052] The computer program including the automatic labeling program PG1 may be a single computer program or a group of programs composed of multiple computer programs. Furthermore, the computer program including the automatic labeling program PG1 may be executed on a single computer or executed collaboratively on multiple computers (for example, the wearable device 10 and the server device 30).

[0053] The computer program, including the automatic labeling program PG1, is provided on a non-temporary recording medium RM on which the computer program is recorded in a readable format. The recording medium RM is a portable memory such as a CD-ROM, USB memory, SD card, microSD card, or CompactFlash®. The processing unit 31 reads various computer programs from the recording medium RM using a reading device (not shown in the figure) and stores the read computer programs in the storage unit 32. The computer programs stored in the storage unit 32 may also be provided via communication. In this case, the processing unit 31 acquires the computer programs via communication through the communication unit 33 and stores the acquired computer programs in the storage unit 32.

[0054] The memory unit 32 may store the learning model MD1 used in the image recognition program PG2. Figure 5 is a schematic diagram showing an example of the configuration of the learning model MD1. The learning model MD1 is a machine learning model that, when image data acquired from the wearable device 10 is input, is learned to output information about the object to be measured contained in the image. The learning model MD1 is constructed using a neural network for object detection, such as R-CNN (Region-based Convolutional Neural Networks), YOLO (You Only Look Once), or SSD (Single Shot Multi-Box Detector). Alternatively, the learning model MD1 may be constructed using any neural network capable of image segmentation, such as SegNet, U-Net (U-Shaped Network), or PSPNet (Pyramid Scene Parsing Network).

[0055] The learning model MD1 comprises an input layer, an intermediate layer, and an output layer. Image data acquired by the wearable device 10 is input to the input layer. The intermediate layer comprises a convolutional layer, a pooling layer, a fully connected layer, etc. The calculations performed by the intermediate layer include extracting features from the input image, selecting candidate regions in the input image that are likely to be detection targets (in this embodiment, measurement targets by the measurement device 20), and identifying measurement targets within the selected regions. The output layer refers to the calculation results of the intermediate layer and outputs the detection result. The detection result includes a bounding box surrounding the detected measurement target and an identification name (class name) that identifies the measurement target within the bounding box. In this embodiment, the identification name is the name of an electrical component constituting the refrigeration and air conditioning equipment, or a name representing a measurement location such as an intake or outlet. The detection result may also include information on the prediction confidence level for the detected measurement target.

[0056] The learning model MD1 is generated by preparing a dataset containing image data including various measurement targets, the class names of the measurement targets included in the image data, and the coordinate values ​​of the bounding boxes surrounding the measurement targets, and then using this dataset as training data to perform machine learning using a predetermined algorithm. The machine learning may be performed inside the server device 30 or on an external information processing device. In the latter case, after performing machine learning on the external information processing device, the trained learning model MD1 can be obtained and stored in the storage unit 32. Since the machine learning algorithm itself is publicly known, a detailed explanation thereof will be omitted.

[0057] In this embodiment, the measurement target is identified using the learning model MD1, but the measurement target may also be identified using existing detection methods that do not use a learning model, such as template matching.

[0058] The storage unit 32 further includes a database DB that stores measurement data measured by the measuring device 20 in association with information about the object to be measured identified from the image data. Figure 6 is a conceptual diagram showing an example of the database DB. The database DB stores measurement data measured by the measuring device 20 in association with information about the object to be measured identified from the image data. In the example in Figure 6, the information about the object to be measured includes information about the target equipment being measured and information about the measurement location, but either one of these pieces of information may be stored. In addition to the measurement data and information about the object to be measured, the database DB may also store information about the equipment being measured, information about the measurement date and time, information about the type of measuring device 20, identification information of the wearable device 10 and the measuring device 20, the name of the worker who performed the work, etc.

[0059] The measuring device 20 may be identified from image data, similar to the object being measured, or it may be notified by the wearable device 10. The measurement date and time information may be notified by the wearable device 10. Instead of the measurement date and time information, the date and time the measurement data was acquired may be stored in the database DB. Identification information for the wearable device 10 and the measuring device 20 may be notified by the wearable device 10. The worker's name may be stored in advance in the storage unit 32 in association with the identification information of the wearable device 10, and the processing unit 31 can read the worker's name from the storage unit 32 and register it in the database DB.

[0060] The communication unit 33 includes a communication module for wireless communication with external devices such as the wearable device 10. The communication module is a communication module for wireless communication using known mobile communication standards such as 3G, 4G, and 5G, or wireless LAN methods such as WiFi (registered trademark). The communication unit 33 communicates with external devices such as the wearable device 10 via a communication network NW, receives image data captured by the wearable device 10 and measurement data measured by the measurement device 20, and transmits appropriate data to be sent to the external device.

[0061] The operation unit 34 is equipped with operating devices such as a touch panel, keyboard, and switches, and accepts various inputs and operations from the work manager, etc. The processing unit 31 acquires the information input through the operation unit 34 and performs appropriate control based on the various operation information provided by the operation unit 34.

[0062] The display unit 35 is equipped with a display device such as a liquid crystal monitor or an organic EL (Electro-Luminescence) monitor, and displays information that should be notified to the work manager, etc., in response to instructions from the processing unit 31.

[0063] The server device 30 may be a single computer, or it may be a computer system composed of multiple computers and peripheral devices. Furthermore, the server device 30 may be a virtualized virtual machine, or it may be a cloud.

[0064] The following describes the processes performed in Information Processing System 1. Workers begin work at the work site by attaching a wearable device 10 to their bodies. The wearable device 10 automatically starts imaging at an appropriate time after being attached to the worker. Alternatively, the wearable device 10 starts imaging when instructed to do so by the worker or the server device 30.

[0065] The worker performs installation, repair, and inspection work on refrigeration and air conditioning equipment, and, as necessary, performs measurements using the measuring device 20. The physical quantities to be measured include at least one of the following in the refrigeration and air conditioning equipment: temperature, current, voltage, power, electrical resistance, airflow, wind speed, pressure, vibration, frequency, humidity, and illuminance. The measuring device 20 transmits the measurement data obtained by measuring the physical quantities to the wearable device 10. The wearable device 10 transmits the image data obtained from the imaging unit 15 and the measurement data received from the measuring device 20 to the server device 30. The wearable device 10 may transmit the image data and measurement data to the server device 30 in real time during work, or it may transmit the image data and measurement data to the server device 30 after the work is completed.

[0066] The processing unit 31 of the server device 30 reads and executes programs such as the automatic labeling program PG1, the image recognition program PG2, and the good measurement judgment program PG3 from the storage unit 32 at appropriate timings. The processing unit 31 performs the following processes according to these programs.

[0067] Figure 7 is a flowchart illustrating the procedure of processing performed by the server device 30 according to Embodiment 1. The communication unit 33 of the server device 30 receives image data and measurement data transmitted from the wearable device 10 via the communication network NW (step S101). The communication unit 33 outputs the received image data and measurement data to the processing unit 31.

[0068] The processing unit 31 inputs the image data acquired through the communication unit 33 to the learning model MD1 and performs calculations using the learning model MD1 (step S102). Specifically, the processing unit 31 inputs the acquired image data to the input layer of the learning model MD1, and in the intermediate layer, it performs processes such as extracting features from the input image, selecting candidate regions that appear to be detection targets (measurement targets by the measurement device 20) included in the input image, and identifying measurement targets within the selected regions, and outputs the detection results from the output layer.

[0069] The processing unit 31 identifies the measurement target based on the calculation results of the learning model MD1 (step S103). The learning model MD1 outputs, as a calculation result, information about the region surrounding the detected measurement target and an identification name (class name) that identifies the measurement target within that region. Based on the output of the learning model MD1, the processing unit 31 identifies the measurement target (at least one of the equipment and measurement location that became the measurement target).

[0070] The processing unit 31 associates the measurement data acquired through the communication unit 33 with the information of the measurement target identified in step S103 and stores it in the database DB (step S104). The database DB illustrated in Figure 6 includes items for the target device, the measurement target, and the measurement result. Therefore, the processing unit 31 only needs to register the information of the measurement target identified in step S103 in the items for the target device and the measurement target, and register the acquired measurement data in the item for the measurement result. The measurement data may be a single measurement value or multiple measurement values ​​measured in a time series. If the measurement data is created as a data file, a link to the data file may be registered in the database DB. The processing unit 31 may also acquire information on the type of measurement device 20 and the measurement date and time via the wearable device 10 along with the measurement data, and register this acquired information in the database DB.

[0071] As described above, in Embodiment 1, the measurement data measured by the measurement device 20 and the measurement target identified from the image data captured by the wearable device 10 are associated and stored. Therefore, by checking the information registered in the database DB, the measurement items can be inferred, and it is easy to understand which part of the refrigeration and air conditioning equipment was measured and what was measured.

[0072] In this embodiment, the system uses the learning model MD1 to identify the measurement target. However, it is also possible to use a configuration in which a first learning model is prepared to identify the equipment to be measured, such as an outdoor unit or indoor unit, from image data, and a second learning model is prepared to identify the measurement location by the measurement device 20 from the image in which the target equipment has been identified, and these two learning models are used to identify the measurement target.

[0073] In this embodiment, information about the type of measurement device 20 is obtained from the wearable device 10. However, since the field of view captured by the wearable device 10 includes the measurement device 20, the type of measurement device 20 may be identified based on the image data. Figure 8 is a schematic diagram showing an example configuration of the learning model MD2 for identifying the type of measurement device 20. The configuration of the learning model MD2 is the same as that of the learning model MD1, and is constructed using a neural network for object detection such as R-CNN, or a neural network for image segmentation such as SegNet, and comprises an input layer, an intermediate layer, and an output layer. When the learning model MD2 is input image data captured by the wearable device 10, it is trained to output information about the type of measurement device 20 included in that image.

[0074] In this embodiment, the measurement target is identified based on image data captured by the wearable device 10. However, the measurement target may also be identified based on the worker's voice input through the sound input unit 16. For example, if the worker's voice saying "I will now measure the intake temperature of the outdoor unit" is input through the sound input unit 16, the processing unit 11 can use existing voice recognition technology to identify that the target device is an "outdoor unit" and that the measurement location is the "intake port of the outdoor unit". Alternatively, the processing unit 11 may identify the measurement device 20 based on the worker's voice. In the above example, since the processing unit 11 can recognize that temperature measurement is to be performed, it can identify the measurement device 20 as a temperature sensor.

[0075] (Embodiment 2) Embodiment 2 describes a configuration in which the measurement device 20 determines whether the measurement location is correct, and if it is determined that the measurement location is correct, it associates the measurement data and the information of the measurement target and registers them in a database DB.

[0076] In this embodiment, measurement conditions are defined for each measurement target. Figure 9 is a conceptual diagram showing an example of a condition table TB that defines measurement conditions for each measurement target. In the condition table TB, measurement conditions are defined for each measurement content. For example, if the measurement content is "outdoor unit intake temperature," then "within 10 cm from the intake port" is defined as the measurement condition. Also, if the measurement content is "outlet temperature of the outdoor unit," then "within 30 cm from the outlet port" is defined as the measurement condition. Various measurement conditions for such measurement content are defined in the condition table TB and stored in the storage unit 32 of the server device 30.

[0077] The server device 30 refers to the condition table TB stored in the storage unit 32 to determine whether the measurement location by the measurement device 20 is correct. Figure 10 is a flowchart illustrating the procedure of processing performed by the server device 30 according to Embodiment 2. The communication unit 33 of the server device 30 receives image data and measurement data transmitted from the wearable device 10 via the communication network NW (step S201). The communication unit 33 outputs the received image data and measurement data to the processing unit 31.

[0078] The processing unit 31, following the same procedure as in Embodiment 1, inputs the image data acquired through the communication unit 33 to the learning model MD1 and performs calculations using the learning model MD1 to identify the measurement target (steps S202 to S203).

[0079] The processing unit 31, based on the measurement target identified in step S203, refers to the condition table TB and determines whether the measurement by the measuring device 20 was performed correctly (step S204). For example, if the measurement target identified in step S203 is the air intake of the outdoor unit, and the measurement data received in step S201 is temperature data, the processing unit 31 can determine that the measurement content is "outdoor unit air intake temperature," and therefore refers to the condition table TB to determine whether the measuring device 20 was within 10 cm of the air intake. The processing unit 31 can also determine whether the measuring device 20 was within 10 cm of the air intake based on the image data received along with the measurement data. For example, if the dimensions of the outdoor unit, air intake, measuring device 20, etc., are known, the processing unit 31 can estimate the distance between the air intake and the measuring device 20 from the image. Existing methods are used to estimate the distance from the image.

[0080] If the measuring device 20 determines that the measurement is not being performed correctly (S204: NO), the processing unit 31 notifies the wearable device 10 of this (step S205). When the wearable device 10 receives notification from the server device 30 that the measurement is not being performed correctly, it notifies the worker of this fact by voice from the sound output unit 17.

[0081] If the processing unit 31 determines that the measurement by the measuring device 20 is being performed correctly (S204: YES), it associates the measurement data acquired through the communication unit 33 with the information of the measurement target identified in step S203 and stores it in the database DB (step S206). The processing unit 31 may also notify the wearable device 10 if it determines that the measurement by the measuring device 20 is being performed correctly.

[0082] As described above, in Embodiment 2, registration to the database DB is performed only when the measurement is performed correctly, and if the measurement is not performed correctly, the operator is notified to encourage correct measurement.

[0083] (Embodiment 3) In Embodiment 3, when the measurement data from the measurement device 20 is time-series data, a configuration is described in which the measurement data from the start of measurement to the end of measurement is registered in the database DB.

[0084] Figure 11 is a flowchart illustrating the procedure of processing performed by the server device 30 according to Embodiment 3. The communication unit 33 of the server device 30 receives image data and measurement data transmitted from the wearable device 10 via the communication network NW (step S301). The communication unit 33 outputs the received image data and measurement data to the processing unit 31.

[0085] The processing unit 31, following the same procedure as in Embodiment 1, inputs the image data acquired through the communication unit 33 to the learning model MD1 and performs calculations using the learning model MD1 to identify the measurement target (steps S302 to S303).

[0086] The processing unit 31 identifies the start and end points of measurement with respect to the measurement data received in step S301 (step S304). For example, the processing unit 31 can examine changes in the measurement data and identify the point in time when the measured value crosses a first set value as the start point of measurement, and the point in time when it crosses a second set value as the end point of measurement. The first and second set values ​​are set as appropriate according to the measurement content. Alternatively, the operation of the measuring device 20 may be recorded, and the point in time when the measurement button is pressed may be identified as the start point of measurement, and the point in time when the measurement button is released as the end point of measurement. Alternatively, the point in time when a measurement start instruction is given using the operation unit 19 of the wearable device 10 may be identified as the start point of measurement, and the point in time when a measurement end instruction is given as the end point of measurement.

[0087] The processing unit 31 stores the measurement data acquired through the communication unit 33, from the start of measurement to the end of measurement, in the database DB, associating it with the information of the measurement target identified in step S303 (step S305).

[0088] As described above, in Embodiment 3, only the net measurement data can be stored in the database DB, thus reducing the amount of data stored in the database DB.

[0089] (Embodiment 4) Embodiment 4 describes a configuration that determines whether the object to be measured is necessary and outputs repair permission information if it is deemed to require repair.

[0090] Figure 12 is a flowchart illustrating the procedure of processing performed by the server device 30 according to Embodiment 4. The communication unit 33 of the server device 30 receives image data and measurement data transmitted from the wearable device 10 via the communication network NW (step S401). The communication unit 33 outputs the received image data and measurement data to the processing unit 31.

[0091] The processing unit 31, following the same procedure as in Embodiment 1, inputs the image data acquired through the communication unit 33 to the learning model MD1 and performs calculations using the learning model MD1 to identify the measurement target (steps S402 to S403).

[0092] The processing unit 31 determines whether the object to be measured needs to be repaired based on the measurement data received in step S401 (step S404). The processing unit 31 determines whether the object to be measured needs to be repaired based on whether the measured value measured by the measuring device 20 exceeds a threshold. The threshold can be set appropriately depending on the object to be measured.

[0093] If the processing unit 31 determines that the object to be measured requires repair (S404: YES), it outputs repair permission information (step S405). The processing unit 31 notifies the wearable device 10 of the repair permission information and informs the worker via the wearable device 10 that the repair has been authorized. The repair permission information may be registered in the database DB.

[0094] If the processing unit 31 determines that the measurement target does not require repair (S404: NO), or if it outputs repair permission information in step S405, it associates the measurement data obtained through the communication unit 33 with the information of the measurement target identified in step S403 and stores it in the database DB (step S406).

[0095] (Embodiment 5) Embodiment 5 describes a configuration in which a work report is created based on data stored in a database DB.

[0096] Figure 13 is an explanatory diagram illustrating the method for creating work reports. The processing unit 31 of the server device 30 receives a request from the worker or work manager to specify the records to be used for creating the report from among the records stored in the database DB, and generates a prompt instructing the LLM server to create the report based on the specified records. The LLM server is an existing server computer that provides services using a Large Language Model (LLM). The LLM server is equipped with a language model that generates response sentences to input prompts. The language model is an existing large language model such as GPT-4 (Generative Pretrained Transformer 4), LLaMA (Large Language Model Meta AI), or BERT (Bidirectional Encoder Representations from Transformers).

[0097] The example in Figure 13 shows how a prompt is generated instructing the system to create a work report using the record from November 8, 2022, stored in the database DB. The server device 30 creates the report by sending such a prompt to the LLM server. The text included in the report is generated by the LLM server and sent back to the server device 30 as a response to the prompt.

[0098] As described above, in Embodiment 5, work reports can be automatically generated.

[0099] Furthermore, while the application example of the information processing system according to this embodiment is assumed to be a work scenario involving refrigeration and air conditioning equipment, it is of course applicable to a variety of work scenarios, not limited to work scenarios involving refrigeration and air conditioning equipment, but also including work scenarios related to elevators, maintenance and inspection work scenarios in chemical plants and power supply facilities, and so on.

[0100] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended. [Explanation of Symbols]

[0101] 10 Wearable Devices 20 measuring devices 30 Analysis Servers 31 Processing Unit 32 Storage section 33 Communications Department 34 Control section 35 Display section PG1 Automatic Labeling Program PG2 Image Recognition Program PG3 Good Measurement Judgment Program MD1 learning model DB Database

Claims

1. The system acquires image data captured by an imaging device and measurement data measured by a measuring device that is communicatively connected to the imaging device. Based on the acquired image data, the type of the measuring device is identified. The acquired measurement data and the information of the identified type of measurement device are stored in association with the information of the measurement target identified based on the image data. An information processing method in which processing is performed by a computer.

2. The system stores information about the object to be measured and information about the date and time of measurement in association with the measurement data. The information processing method according to claim 1, wherein the processing is performed by the computer.

3. The measurement target is to identify the device to be measured or the measurement target location within the device. The information processing method according to claim 1, wherein the processing is performed by the computer.

4. The aforementioned measurement targets include refrigeration and air conditioning related equipment, The measurement data includes at least one measurement of temperature, current, voltage, power, electrical resistance, airflow rate, wind speed, pressure, vibration, frequency, humidity, and illuminance in the refrigeration and air conditioning equipment. The information processing method according to claim 1.

5. The measurement data from the start to the end of measurement by the aforementioned measuring device is stored in association with the information of the object being measured. The information processing method according to claim 1, wherein the processing is performed by the computer.

6. The amount of change in the measurement data is compared with a threshold value for that amount of change. Based on the comparison results, the start and end times of the measurement are determined. The information processing method according to claim 5, wherein the processing is performed by the computer.

7. Based on the aforementioned image data, the measurement target location is identified by the measurement device. Determine whether the identified measurement target location is correct or incorrect. If the measurement target location is determined to be correct, the measurement data is stored in association with the information of the measurement target identified based on the image data. The information processing method according to claim 1.

8. Based on the aforementioned measurement data, a determination is made as to whether or not the object being measured needs to be repaired. If repairs are deemed necessary, repair permission information will be output. The information processing method according to claim 1, wherein the processing is performed by the computer.

9. The report is generated by providing the language model with the stored information on the object to be measured, the measurement data, the measurement date and time information, and a prompt including a command to create a report. The information processing method according to claim 2, wherein the processing is performed by the computer.

10. It comprises at least one processing unit, The aforementioned processing unit, The system acquires image data captured by an imaging device and measurement data measured by a measuring device that is communicatively connected to the imaging device. Based on the acquired image data, the type of the measuring device is identified. The acquired measurement data and the information of the identified type of measurement device are stored in association with the information of the measurement target identified based on the image data. Information processing device.

11. The system acquires image data captured by an imaging device and measurement data measured by a measuring device that is communicatively connected to the imaging device. Based on the acquired image data, the type of the measuring device is identified. The acquired measurement data and the information of the identified type of measurement device are stored in association with the information of the measurement target identified based on the image data. A computer program that causes a computer to perform a process.

Images